feat(ingest): per-window delta-base rotation for sketch deltas - #346
Merged
Merged
Conversation
The OTLP delta-apply path reconstructed per-series sketch state by
merging each window's delta onto a never-reset running base
(state(N) = state(N-1) ⊕ delta(N)). Because the edge tumbling window
resets per-series sketch state every window, each window's delta is
that window's marginal against an empty base — so accumulating forever
over-counts across windows (and for HLL the register-wise max base
holds the all-time max). See docs/delta-baseline-contract.md §3.
This adds per-window base rotation on the consumer side:
- Cache entry now carries the window start it was built for. The
per-series snapshot cache value changes from a bare boxed accumulator
to SnapshotCacheEntry { core, window_start }.
- The delta-apply branch detects a per-series window boundary by
comparing the data point's start_time_unix_nano against the cached
base's window_start. On a new window it resets the cached base to
empty (AggregateCore::reset_to_empty) BEFORE applying the new
window's delta, so within a window deltas accumulate and at a new
window the base starts fresh — state(N) is window N only.
- Full frames (PROTO_FULL) keep REPLACE semantics and set the stored
window_start.
- reset_to_empty is sketch-agnostic: implemented for the additive
delta-capable families (DDSketch, CMS, CountSketch, HLL), each
preserving its shape/config (dims / relative accuracy / register
width); default no-op elsewhere. KLL never deltas.
- The existing "no base yet → drop the delta" guard is unchanged.
Adds an ingest test that feeds full(win1) → delta(win1) → delta(win2):
within win1 the delta accumulates onto the full frame, and at the win2
boundary the base is rotated so the reconstructed state is win2 only,
not win1 + win2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 26, 2026
Align the backend's delta-apply with asap_sketchlib's updated wire format:
- HLL delta is now a varint-packed (index_delta, value) blob (was repeated
per-register sub-messages). Decode + apply via HllSketch::apply_delta_bytes
(single source of truth) instead of the vendored nested decode; vendored
hll_delta.proto updated to bytes packed_updates; HLL round-trip tests build
the packed blob.
- CountMinSketchDelta gained an hh_keys field upstream; pass an empty set on
the CountMin apply path (no TopK to rebuild here), mirroring CountSketch.
- Update the CMS/CountSketch zero-dims decode tests to assert the current
validate_sketch_dims message ('degenerate dims') surfaced by the bump.
DDSketch alignment (scalar-less wire) is already on main (#345/#346).
cargo build --workspace clean; cargo test -p data_plane --lib: 782 passed.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements per-window base rotation in the
ASAPQuery-backendingest delta-apply path so that applying per-window DDSketch (and other additive-family) deltas yields per-window reconstructed state instead of an all-time accumulation across windows.This is the consumer half of the delta–baseline contract (
docs/delta-baseline-contract.md§3).The problem
The OTLP delta-apply path (
data_plane/src/drivers/ingest/otel.rs) reconstructed per-series sketch state by merging each window's delta onto a never-reset running base:But the edge tumbling window resets per-series sketch state every window, so each window's delta is that window's marginal against an empty base. Accumulating forever therefore over-counts across windows. For HLL it is worse: the register-wise
maxbase has no inverse and holds the all-time max across every window seen, so per-window cardinality queries over-count by orders of magnitude.The change — per-window base rotation
SnapshotCacheEntry { core, window_start }(precompute_engine/ingest_handler.rs).start_time_unix_nanoagainst thewindow_startstored with the cached base for thatseries_key. When they differ, the cached base is reset to empty before applying the new window's delta — so within a window deltas accumulate, and at a new window the base starts fresh.state(N)is then window N only.PROTO_FULL) keep REPLACE semantics and set the storedwindow_start.AggregateCore::reset_to_empty(default no-op) implemented for the additive, delta-capable families — DDSketch, CMS, CountSketch, HLL — each preserving its shape/config (dims / relative accuracy / register width). KLL never deltas, so it never reaches this path.Test
Adds
delta_apply_rotates_per_series_base_at_window_boundary(drives the real ingest loop): for one series, full(win1) → delta(win1) (samewindow_start) → delta(win2) (a NEWwindow_start).Uses DDSketch (an additive family) so accumulation vs. reset is directly observable. All existing delta tests still pass.
Relationship to other work
count_over_time,quantile_over_time, distinct-count over time) aggregate across correct per-window buckets, and the contract's delta-ON == delta-OFF equality gate becomes achievable.🤖 Generated with Claude Code