Skip to content

feat(ingest): per-window delta-base rotation for sketch deltas - #346

Merged
zzylol merged 1 commit into
mainfrom
feat/per-window-base-rotation
May 26, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/per-window-base-rotation

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements per-window base rotation in the ASAPQuery-backend ingest delta-apply path so that applying per-window DDSketch (and other additive-family) deltas yields per-window reconstructed state instead of an all-time accumulation across windows.

This is the consumer half of the delta–baseline contract (docs/delta-baseline-contract.md §3).

The problem

The OTLP delta-apply path (data_plane/src/drivers/ingest/otel.rs) reconstructed per-series sketch state by merging each window's delta onto a never-reset running base:

state(N) = state(N-1) ⊕ delta(N)

But the edge tumbling window resets per-series sketch state every window, so each window's delta is that window's marginal against an empty base. Accumulating forever therefore over-counts across windows. For HLL it is worse: the register-wise max base has no inverse and holds the all-time max across every window seen, so per-window cardinality queries over-count by orders of magnitude.

The change — per-window base rotation

  • Cache entry now carries its window start. The per-series snapshot cache value changes from a bare boxed accumulator to SnapshotCacheEntry { core, window_start } (precompute_engine/ingest_handler.rs).
  • Window-boundary detection per series. The delta-apply branch compares the incoming data point's start_time_unix_nano against the window_start stored with the cached base for that series_key. When they differ, the cached base is reset to empty before applying the new window's delta — so within a window deltas accumulate, and at a new window the base starts fresh. state(N) is then window N only.
  • Full frames (PROTO_FULL) keep REPLACE semantics and set the stored window_start.
  • Sketch-agnostic reset. Adds AggregateCore::reset_to_empty (default no-op) implemented for the additive, delta-capable families — DDSketch, CMS, CountSketch, HLL — each preserving its shape/config (dims / relative accuracy / register width). KLL never deltas, so it never reaches this path.
  • The existing "no base yet → drop the delta" guard is unchanged.

Test

Adds delta_apply_rotates_per_series_base_at_window_boundary (drives the real ingest loop): for one series, full(win1) → delta(win1) (same window_start) → delta(win2) (a NEW window_start).

  • Within win1 the delta accumulates onto the full frame.
  • At the win2 boundary the base is rotated, so the reconstructed state is win2 only (count 20), not win1 + win2 (count 45).

Uses DDSketch (an additive family) so accumulation vs. reset is directly observable. All existing delta tests still pass.

Relationship to other work

  • Composes with the edge producer (true per-window deltas against an empty outbound base) in feat(delta): per-window DDSketch delta producer (delta-against-empty) ASAPCollector#456 — the edge emits the window's own sketch as the delta, this PR rotates the backend base to match.
  • Builds on chore(ddsketch): consume DDSketch wire format without metric scalars #345 (DDSketch wire format consumed without metric scalars), which landed the DDSketch delta consumer this rotation rides on.
  • Unblocks per-window-correct delta queries: per-window finalized state feeds the warm/ASAP-tier per-bucket store, so windowed-range reducers (count_over_time, quantile_over_time, distinct-count over time) aggregate across correct per-window buckets, and the contract's delta-ON == delta-OFF equality gate becomes achievable.

🤖 Generated with Claude Code

The OTLP delta-apply path reconstructed per-series sketch state by
merging each window's delta onto a never-reset running base
(state(N) = state(N-1) ⊕ delta(N)). Because the edge tumbling window
resets per-series sketch state every window, each window's delta is
that window's marginal against an empty base — so accumulating forever
over-counts across windows (and for HLL the register-wise max base
holds the all-time max). See docs/delta-baseline-contract.md §3.

This adds per-window base rotation on the consumer side:

- Cache entry now carries the window start it was built for. The
  per-series snapshot cache value changes from a bare boxed accumulator
  to SnapshotCacheEntry { core, window_start }.
- The delta-apply branch detects a per-series window boundary by
  comparing the data point's start_time_unix_nano against the cached
  base's window_start. On a new window it resets the cached base to
  empty (AggregateCore::reset_to_empty) BEFORE applying the new
  window's delta, so within a window deltas accumulate and at a new
  window the base starts fresh — state(N) is window N only.
- Full frames (PROTO_FULL) keep REPLACE semantics and set the stored
  window_start.
- reset_to_empty is sketch-agnostic: implemented for the additive
  delta-capable families (DDSketch, CMS, CountSketch, HLL), each
  preserving its shape/config (dims / relative accuracy / register
  width); default no-op elsewhere. KLL never deltas.
- The existing "no base yet → drop the delta" guard is unchanged.

Adds an ingest test that feeds full(win1) → delta(win1) → delta(win2):
within win1 the delta accumulates onto the full frame, and at the win2
boundary the base is rotated so the reconstructed state is win2 only,
not win1 + win2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 7a6b444 into main May 26, 2026
zzylol added a commit that referenced this pull request May 26, 2026
Align the backend's delta-apply with asap_sketchlib's updated wire format:

- HLL delta is now a varint-packed (index_delta, value) blob (was repeated
  per-register sub-messages). Decode + apply via HllSketch::apply_delta_bytes
  (single source of truth) instead of the vendored nested decode; vendored
  hll_delta.proto updated to bytes packed_updates; HLL round-trip tests build
  the packed blob.
- CountMinSketchDelta gained an hh_keys field upstream; pass an empty set on
  the CountMin apply path (no TopK to rebuild here), mirroring CountSketch.
- Update the CMS/CountSketch zero-dims decode tests to assert the current
  validate_sketch_dims message ('degenerate dims') surfaced by the bump.

DDSketch alignment (scalar-less wire) is already on main (#345/#346).
cargo build --workspace clean; cargo test -p data_plane --lib: 782 passed.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol deleted the feat/per-window-base-rotation branch July 17, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant