fix(data_plane): overlap-scan sketch reads for short/instant windows - #326
Merged
Merged
Conversation
The sketch read path (`SketchStore::query_range`) used strict CONTAINMENT (`w.0 >= start && w.1 <= end`) when scanning epochs, but the agent emits ~30s tumbling panes. A query window narrower than one pane cadence — a `quantile_over_time(...[30s])` range, or any instant selector whose freshest pane straddles `now` — fails to FULLY CONTAIN any pane, so the scan returned zero in-window samples. The #325 delta-stitching carry-in keys off the earliest in-window sample, so with no in-window samples it never fired, and the reducer yielded an empty series the engine surfaced as "No result for query". Two live gaps, one root cause: 1. `quantile_over_time(...[30s])` -> "No result" ([45s]+ worked because a 45s+ window can contain a 30s pane). 2. bare/instant `http_requests_total_latency_ms` / `quantile(...)` -> empty: per-window family + the straddling freshest pane invisible, so the instant projection (`samples.last()`) found nothing. Add `range_query_overlap_into` to MutableEpoch + SealedEpoch (half-open overlap `w.1 > start && w.0 < end`, matching the overlap semantics `range_query_into_grouped` already documents for this exact 30s-pane case) and switch only the sketch `query_range` reads to it. Exact-agg / rate / sum paths keep containment, so the working `sum by` shapes can't regress. The reducer's existing `w_end >= t0` per-window filter and cumulative `latest_end` projection still drop out-of-range values, so no carry-in / straddling-pane value leaks into the answer's time domain. Regression tests lock in both the fix and non-regression: epoch-level overlap-vs-containment + half-open boundary tests; reducer-level KLL short-window cumulative + per-window-instant tests (real proto sketch bytes through the live decode path); and a wide-window cumulative test asserting the already-working `[2m]+` shape is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 25, 2026
…#327) The warm sketch store grew unbounded (~0.74 GiB/min) under steady OTLP ingest, freezing a 251 GiB node. Root cause: SidStoreData is always built with epoch_capacity=None, so maybe_rotate_epoch returns early and nothing ever seals; the persistence flusher only evicts SEALED epochs, so current_epoch accumulated one ~19 KB pane per 30s tumbling window per sid forever. Memory was O(active_series x total_elapsed_time). Add a time-based retention horizon on SidStoreData: on each insert, evict windows whose END is older than newest_end - horizon from current_epoch (and any sealed_epochs), making memory O(active_series x horizon). Configurable via ASAP_SKETCH_RETENTION_MS (0 disables), default 2h. Does not regress the #323-#326 sketch read path: the horizon (2h) comfortably exceeds the ~30m max range-query window plus the delta-stitching carry-in's Full-base reach, and eviction keys on window-END so a straddling pane survives until fully behind the horizon. Recent unsealed windows stay queryable via the overlap scan (no force-seal). Regression tests prove (a) long-elapsed ingest keeps per-sid window count bounded and (b) a 30m range query within the horizon still resolves with its Full carry-in base intact. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 tasks
zzylol
added a commit
that referenced
this pull request
May 25, 2026
Compose hot (current_epoch) -> sealed (in-mem, pending flush) -> disk parts into a single tiered store so warm-sketch memory is bounded by flush-then-evict rather than #327's age-based drop. Reads union all three tiers across the requested range. - Sealing now fires under persistence: SidStoreData gains a seal_window_count cadence (default 20 windows ~= 10 min of 30s panes) so current_epoch rotates into sealed_epochs for the flusher to persist. max_epochs drop is disabled under persistence (the flusher owns sealed-epoch lifecycle). In-memory-only deploys keep #327. - query_range/union_disk_parts_into consult PartCache+Manifest for the evicted portion of the range, rebuilding the full label key->value map from the sid's group_by_keys and preserving the #323-#326 read contract (half-open overlap + delta-stitching carry-in) across the in-mem/on-disk boundary -- incl. a carry-in Full base that now lives on disk. Part format round-trips SketchEncoding via a repurposed v1 pad byte (legacy 0 decodes as Full). - enforce_retention is a no-op under persistence so retention never drops un-flushed sealed/current data; the disk-tier TTL bounds the durable copy. In-memory-only path is unchanged. - start_persistence installs a read handle + seal cadence and recovers the manifest+parts so a restart immediately serves recovered data. - New CLI flag --persistence-seal-window-count plumbs the cadence. Tests: seal-fires, flush+evict bounds memory, query-from-disk incl. disk carry-in base, restart recovery, and persistence-disabled non-regression. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the final two warm sketch-query gaps after #323/#324/#325.
SketchStore::query_rangescanned epochs with strict containment(
w.0 >= start && w.1 <= end), but the agent emits ~30s tumbling panes.A query window narrower than one pane cadence can't fully contain any
pane, so the scan returned zero in-window samples → the #325 carry-in
(which keys off the earliest in-window sample) never fired → empty
series surfaced as
"No result for query".quantile_over_time(...[30s])→ "No result" ([45s]+worked: a 45s+ window can contain a 30s pane).
http_requests_total_latency_ms/quantile(...)→ empty: per-window family + the straddling freshestpane invisible, so the instant projection (
samples.last()) foundnothing.
Fix
range_query_overlap_intotoMutableEpoch+SealedEpoch(half-open overlap
w.1 > start && w.0 < end, matching the overlapsemantics
range_query_into_groupedalready documents for this exact30s-pane case).
query_rangereads to overlap. Exact-agg /rate /
sum bypaths keep containment → working shapes can't regress.w_end >= t0per-window filter and cumulativelatest_endprojection still drop out-of-range values, so no carry-in/ straddling-pane value leaks into the answer's time domain.
Actual KLL ingest encoding
otel.rsassigns each datapoint'sSketchEncodingstraight from theOTLP
dp.encodingfield viaencoding_to_handle, defaulting unset(
0) toProtoFull(otel.rs:1267-1268). KLL deltas (ProtoDelta) areapplied as mergeable Full fragments through
decode_full(delta_apply.rs:157-171). The carry-in's
need_basecorrectly matchesProtoDelta | MsgpackDelta; the bug was upstream — containment starvedit of in-window samples.
Test plan
cargo test -p data_plane— 733 lib + integration tests pass, 0 failures[2m]+unchanged)cargo clippy -p data_plane— 0 new issues in changed files (one pre-existingapprox_constanterror in untouchedforward.rs:850remains on main)🤖 Generated with Claude Code