fix(data_plane): read delta-only sketch windows + count(HLL) - #325
Merged
Merged
Conversation
Two warm-tier sketch-query bugs that made quantile_over_time / count(HLL) return empty `asap_query` results (silent "No result", never failing over to the archive): 1. Delta-stitching carry-in. The agent emits a periodic Full snapshot then many cheap Delta frames, so a short query window (e.g. `[30s]`) routinely contains ONLY deltas — the Full landed earlier, outside the window. `query_range`'s strict containment filter dropped that Full, the delta-apply reducer couldn't establish a rolling base, and the engine returned `Ok(empty)` instead of a value. `query_range` now splices in the most-recent Full ending before `start` as a carry-in base; the reducer drops the out-of-range base from per-window output. 2. count(HLL) distinct-count idiom. `count(metric)` lifts the outer count into both `outer_agg=Count` and the `CardinalityApprox` capability while the bare-selector inner leaves the trace function empty. The engine passed the empty function to the reducer (UnsupportedFunction) AND re-applied the Count fold (collapsing the estimate to row-count 1). The engine now derives the reducer family from the capability when the function string is empty, and suppresses the already-consumed Count fold so the HLL distinct-count is returned directly. Fixes the previously-failing controller_plan_to_query_full_roundtrip_hll e2e and adds index-, reducer-, analyzer-, and engine-level regression tests for both shapes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
zzylol
added a commit
that referenced
this pull request
May 25, 2026
…326) The sketch read path (`SketchStore::query_range`) used strict CONTAINMENT (`w.0 >= start && w.1 <= end`) when scanning epochs, but the agent emits ~30s tumbling panes. A query window narrower than one pane cadence — a `quantile_over_time(...[30s])` range, or any instant selector whose freshest pane straddles `now` — fails to FULLY CONTAIN any pane, so the scan returned zero in-window samples. The #325 delta-stitching carry-in keys off the earliest in-window sample, so with no in-window samples it never fired, and the reducer yielded an empty series the engine surfaced as "No result for query". Two live gaps, one root cause: 1. `quantile_over_time(...[30s])` -> "No result" ([45s]+ worked because a 45s+ window can contain a 30s pane). 2. bare/instant `http_requests_total_latency_ms` / `quantile(...)` -> empty: per-window family + the straddling freshest pane invisible, so the instant projection (`samples.last()`) found nothing. Add `range_query_overlap_into` to MutableEpoch + SealedEpoch (half-open overlap `w.1 > start && w.0 < end`, matching the overlap semantics `range_query_into_grouped` already documents for this exact 30s-pane case) and switch only the sketch `query_range` reads to it. Exact-agg / rate / sum paths keep containment, so the working `sum by` shapes can't regress. The reducer's existing `w_end >= t0` per-window filter and cumulative `latest_end` projection still drop out-of-range values, so no carry-in / straddling-pane value leaks into the answer's time domain. Regression tests lock in both the fix and non-regression: epoch-level overlap-vs-containment + half-open boundary tests; reducer-level KLL short-window cumulative + per-window-instant tests (real proto sketch bytes through the live decode path); and a wide-window cumulative test asserting the already-working `[2m]+` shape is unchanged. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (the NEXT layer behind #323/#324)
Both layers fixed earlier (#323 metric-name canonicalization, #324 reconcile + Ghost short-circuit) were necessary but not sufficient. The KLL
quantile_over_timeand HLLcount(...)queries still returned an empty result taggeddata_source: asap_query(never failing over to the archive). The router treatsOk(empty)as success and does not fall over, so the client saw "No result".Two distinct, independent bugs caused the empty
Ok:1. Delta-stitching carry-in (the quantile bug)
The agent emits a periodic Full snapshot followed by many cheap Delta frames to save bandwidth. A short query window (e.g.
[30s]) routinely contains only deltas — the Full landed earlier, outside the window.SketchStore::query_range's strict containment filter (epoch_columnar.rs:w.0 >= start && w.1 <= end) dropped that out-of-window Full, so the delta-apply reducer (cumulative_evaluate) could not establish a rolling base and returnedOk(None). The reducer then pushed a series row with an empty sample vector, and sinceany_windowwas true (windows were seen) it did not raiseNoData→ the engine returnedOk(empty).Fix:
query_rangenow splices in the most-recent Full snapshot ending at/beforestartas a carry-in base (newcollect_ending_at_or_beforeon both epoch types). The reducer drops the out-of-range base from per-window output so it never leaks into the answer's time domain.2. count(HLL) distinct-count idiom (the cardinality bug)
count(metric)lifts the outercountinto bothouter_agg = Countandrequired_capability = CardinalityApprox, while the bare-selector inner leaves the analyzer's tracefunctionempty. The engine then (a) passed""toSketchReducer::evaluate→UnsupportedFunction, and (b) re-applied theCountfold, which would collapse the cardinality estimate to the row-count (1.0).Fix: the engine derives the reducer family from the typed
required_capabilitywhenfunctionis empty (effective_sketch_function), and suppresses the already-consumedCountfold forCardinalityApproxcandidates (outer_fold_already_consumed). The HLL distinct-count is returned directly.What this is / isn't
Tests
controller_plan_to_query_full_roundtrip_hlle2e.range_query_carries_in_latest_full_before_window,range_query_no_carry_in_when_window_leads_with_full), engine-level quantile carry-in (quantile_over_time_kll_full_before_window_carries_in_base), the genuine delta-only-no-base data gap (quantile_over_time_kll_delta_only_no_base_is_empty), engine-level HLL count (execute_count_hll_returns_cardinality_not_rowcount), and analyzer shape (analyze_count_bare_metric_trace_shape).data_plane(726 lib + all e2e) andcontrol_plane(771) suites green.Possible further layers
Ok(no base to stitch). Today it is a silent empty rather than aNoData→ archive failover; a follow-up could convert it to a capability-miss so the cold tier answers.Test plan
🤖 Generated with Claude Code