Skip to content

fix(data_plane): read delta-only sketch windows + count(HLL) - #325

Merged
zzylol merged 1 commit into
mainfrom
fix/sketch-query-delta-carry-in-and-count-hll
May 25, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/sketch-query-delta-carry-in-and-count-hll

Conversation

@zzylol

@zzylol zzylol commented May 25, 2026

Copy link
Copy Markdown
Contributor

Root cause (the NEXT layer behind #323/#324)

Both layers fixed earlier (#323 metric-name canonicalization, #324 reconcile + Ghost short-circuit) were necessary but not sufficient. The KLL quantile_over_time and HLL count(...) queries still returned an empty result tagged data_source: asap_query (never failing over to the archive). The router treats Ok(empty) as success and does not fall over, so the client saw "No result".

Two distinct, independent bugs caused the empty Ok:

1. Delta-stitching carry-in (the quantile bug)

The agent emits a periodic Full snapshot followed by many cheap Delta frames to save bandwidth. A short query window (e.g. [30s]) routinely contains only deltas — the Full landed earlier, outside the window. SketchStore::query_range's strict containment filter (epoch_columnar.rs: w.0 >= start && w.1 <= end) dropped that out-of-window Full, so the delta-apply reducer (cumulative_evaluate) could not establish a rolling base and returned Ok(None). The reducer then pushed a series row with an empty sample vector, and since any_window was true (windows were seen) it did not raise NoData → the engine returned Ok(empty).

Fix: query_range now splices in the most-recent Full snapshot ending at/before start as a carry-in base (new collect_ending_at_or_before on both epoch types). The reducer drops the out-of-range base from per-window output so it never leaks into the answer's time domain.

2. count(HLL) distinct-count idiom (the cardinality bug)

count(metric) lifts the outer count into both outer_agg = Count and required_capability = CardinalityApprox, while the bare-selector inner leaves the analyzer's trace function empty. The engine then (a) passed "" to SketchReducer::evaluateUnsupportedFunction, and (b) re-applied the Count fold, which would collapse the cardinality estimate to the row-count (1.0).

Fix: the engine derives the reducer family from the typed required_capability when function is empty (effective_sketch_function), and suppresses the already-consumed Count fold for CardinalityApprox candidates (outer_fold_already_consumed). The HLL distinct-count is returned directly.

What this is / isn't

  • It is a seal/unsealed read-path bug (carry-in) plus a query-shape-dispatch bug (empty-function + redundant fold) — NOT a reconcile/eviction bug. In-memory unsealed sketches are read correctly (matches the "可以in-memory, 不一定要seal" requirement).

Tests

  • Re-enables the previously-failing controller_plan_to_query_full_roundtrip_hll e2e.
  • New regression tests: index-level carry-in (range_query_carries_in_latest_full_before_window, range_query_no_carry_in_when_window_leads_with_full), engine-level quantile carry-in (quantile_over_time_kll_full_before_window_carries_in_base), the genuine delta-only-no-base data gap (quantile_over_time_kll_delta_only_no_base_is_empty), engine-level HLL count (execute_count_hll_returns_cardinality_not_rowcount), and analyzer shape (analyze_count_bare_metric_trace_shape).
  • Full data_plane (726 lib + all e2e) and control_plane (771) suites green.

Possible further layers

  • A delta-only window with no Full anywhere still returns empty Ok (no base to stitch). Today it is a silent empty rather than a NoData → archive failover; a follow-up could convert it to a capability-miss so the cold tier answers.
  • The instant-vector projection takes the latest window's value, so an empty "watermark" frame as the freshest window yields 0; benign for liveness but worth confirming against the live watermark cadence.

Test plan

  • Rebuild + redeploy the whole stack at this branch and re-run the 5 marquee queries (quantile_over_time KLL, request_size_bytes KLL, count(unique_users_per_min) HLL, plus the already-working sum-by and count cold-fallback).

🤖 Generated with Claude Code

Two warm-tier sketch-query bugs that made quantile_over_time / count(HLL)
return empty `asap_query` results (silent "No result", never failing over
to the archive):

1. Delta-stitching carry-in. The agent emits a periodic Full snapshot then
   many cheap Delta frames, so a short query window (e.g. `[30s]`) routinely
   contains ONLY deltas — the Full landed earlier, outside the window.
   `query_range`'s strict containment filter dropped that Full, the
   delta-apply reducer couldn't establish a rolling base, and the engine
   returned `Ok(empty)` instead of a value. `query_range` now splices in
   the most-recent Full ending before `start` as a carry-in base; the
   reducer drops the out-of-range base from per-window output.

2. count(HLL) distinct-count idiom. `count(metric)` lifts the outer count
   into both `outer_agg=Count` and the `CardinalityApprox` capability while
   the bare-selector inner leaves the trace function empty. The engine
   passed the empty function to the reducer (UnsupportedFunction) AND
   re-applied the Count fold (collapsing the estimate to row-count 1). The
   engine now derives the reducer family from the capability when the
   function string is empty, and suppresses the already-consumed Count fold
   so the HLL distinct-count is returned directly.

Fixes the previously-failing controller_plan_to_query_full_roundtrip_hll
e2e and adds index-, reducer-, analyzer-, and engine-level regression
tests for both shapes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit d3b0213 into main May 25, 2026
@zzylol
zzylol deleted the fix/sketch-query-delta-carry-in-and-count-hll branch May 25, 2026 04:53
zzylol added a commit that referenced this pull request May 25, 2026
…326)

The sketch read path (`SketchStore::query_range`) used strict CONTAINMENT
(`w.0 >= start && w.1 <= end`) when scanning epochs, but the agent emits
~30s tumbling panes. A query window narrower than one pane cadence — a
`quantile_over_time(...[30s])` range, or any instant selector whose
freshest pane straddles `now` — fails to FULLY CONTAIN any pane, so the
scan returned zero in-window samples. The #325 delta-stitching carry-in
keys off the earliest in-window sample, so with no in-window samples it
never fired, and the reducer yielded an empty series the engine surfaced
as "No result for query".

Two live gaps, one root cause:
  1. `quantile_over_time(...[30s])` -> "No result" ([45s]+ worked because
     a 45s+ window can contain a 30s pane).
  2. bare/instant `http_requests_total_latency_ms` / `quantile(...)` ->
     empty: per-window family + the straddling freshest pane invisible,
     so the instant projection (`samples.last()`) found nothing.

Add `range_query_overlap_into` to MutableEpoch + SealedEpoch (half-open
overlap `w.1 > start && w.0 < end`, matching the overlap semantics
`range_query_into_grouped` already documents for this exact 30s-pane
case) and switch only the sketch `query_range` reads to it. Exact-agg /
rate / sum paths keep containment, so the working `sum by` shapes can't
regress. The reducer's existing `w_end >= t0` per-window filter and
cumulative `latest_end` projection still drop out-of-range values, so no
carry-in / straddling-pane value leaks into the answer's time domain.

Regression tests lock in both the fix and non-regression: epoch-level
overlap-vs-containment + half-open boundary tests; reducer-level KLL
short-window cumulative + per-window-instant tests (real proto sketch
bytes through the live decode path); and a wide-window cumulative test
asserting the already-working `[2m]+` shape is unchanged.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant