feat(sketch-db): Phase 5f-a — BackfillRegistry::coverage() classifier - #34
Merged
Merged
Conversation
Adds the data-primitive the query path will consult to answer
"is this (agg_id, range) covered by backfill?" per §10.4 of the
design. Pure read-side method on `BackfillRegistry` — no I/O, no
wiring into SimpleEngine yet (that's Phase 5f-b, which depends on
Phase 3b-2's per-segment query dispatch landing first).
## What's landed
`BackfillRegistry::coverage(agg_id, range) -> Coverage`:
1. Collects every window from every `Complete` job for `agg_id`.
2. Sorts + merges overlapping intervals into a coverage set
(pure helper `range_covered_by`, tested independently).
3. If `range` is fully inside the coverage set → `Complete`.
4. Else, if any `Running` job's `time_range` overlaps → returns
`BackfillInProgress { job_id, pct }` with the job's progress
so the caller can decide whether to wait or fall back.
5. Otherwise → `Missing`.
Explicit non-contributors: `Cancelled` and `Failed` jobs don't
count toward coverage even if they wrote partial windows before
terminating. Only `Complete` jobs provide coverage signal.
Scope note: this method looks at backfill state only. Callers
that want the full "coverage including live" picture should split
the query range at `schema.created_at_ms` (§10.5 time-disjoint)
and ask this method only about the `[start, created_at)`
historical portion. The live `[created_at, end)` portion is
always `Complete` by construction.
## Test plan
14 new tests covering:
- `range_covered_by` interval math (empty, spanning, adjacent-and-
overlapping merge, gap, empty-target-trivially-true).
- `coverage` happy paths: missing when no jobs / only other-agg
jobs, complete when one job or merged windows cover range,
partial-completion, running-overlap with correct progress,
complete-takes-precedence-over-running.
- `Cancelled` / `Failed` jobs don't contribute to coverage.
- Empty / inverted query range is trivially Complete.
661 lib tests pass (up from 647); clippy + fmt clean.
## Next
Phase 5f-b wires this into the query path. Depends on Phase 3b-2
(per-segment dispatch via `combine_statistic`) landing first,
otherwise SimpleEngine has no natural hook to consult
`coverage()`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
…able stats (#48) Task #34 gap #1 of 3. The §7 schema-timeline primitive (`SchemaRegistry::timeline_for_metric`) and the cross-schema combiner (`engines::timeline_dispatch::combine_statistic`) landed in PRs #20 / #22 / #25, but `SimpleEngine::handle_query_promql` was still resolving a single `agg_id` via `resolve_agg_info_promql` and running the full query against it. A query whose time range spans a reconfigure boundary (old `agg_id` retired, new `agg_id` created) saw a data cliff for the pre-boundary slice. This PR wires the dispatcher: * New `SimpleEngine::try_handle_query_promql_via_timeline`: 1. Parse + pattern-match the query, extract metric name. 2. Build a probe `QueryExecutionContext` to read the resolved `[t1, t2]` + `Statistic`. 3. Call `timeline_for_query(metric, t1, t2)`. Bail out with `None` (fall-through to default single-agg path) if fewer than two segments, or if the statistic is non-combinable (quantile / topk / cardinality / rate / increase — those follow in PR B2 with a Partial HTTP response surface). 4. Per segment: reuse `build_query_execution_context_promql_for_agg_id` from PR #37 (the extracted forced-agg-id entry point), clip the store plan's `[start, end]` to the segment's bounds, execute, collect results. 5. Group by label-tuple and fold per-group per-segment scalars through `combine_statistic`. Emit the combined scalar as an `InstantVectorElement`. Purged segments or segments whose `agg_id` is no longer in the config go into `unresolved` so the combiner sees them. * `handle_query_promql` now tries the timeline path first; returns immediately on `Some`, falls through to the existing single-agg path on `None`. Zero behavior change when the timeline has 0–1 segments for the query's metric (the common case today). ## Scope Combinable stats only: Count / Sum / Min / Max. Non-combinable stats still take the single-agg path — PR B2 will surface `CombinedResult::Partial` on the HTTP response so users see `{covered, missing: [segments]}` explicitly instead of a silent data cliff. ## Validation - `cargo test -p query_engine_rust --lib` — 728 pass (baseline unchanged; the dispatcher stays dormant when tests only register one schema per metric). - `cargo clippy --all-targets -- -D warnings` — clean - `cargo fmt --all -- --check` — clean ## Follow-ups (explicit non-scope here) - **Integration test** seeding two agg_ids + cross-boundary Sum query. Requires the full `PrecomputeEngine` setup harness the existing e2e tests use; deferred as a dedicated PR so this one stays a focused dispatcher patch. - **PR B2**: Partial response surface for non-combinable stats on the HTTP adapter. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
Apr 20, 2026
Task #34 gap #2 of 3. #48's dispatcher activated per-segment execution for combinable stats (Count/Sum/Min/Max) but refused to run for non-combinable stats (Quantile/Topk/Cardinality/ Rate/Increase) — those still fell through to the single-agg path and saw the same data cliff the dispatcher was written to prevent. Reason: there was no way to tell the HTTP caller "this answer is Partial because the query spans a reconfigure boundary and the statistic can't be combined scalarly." Changes: - **`QueryResult::vector_with_warnings` + `warnings()` accessor.** `InstantVector` and `RangeVector` gain a `#[serde(default, skip_serializing_if = "Vec::is_empty")]` `warnings: Vec<String>` field. Default constructors (`QueryResult::vector`, `QueryResult::matrix`) stay wire- compatible — they still serialise without the field when empty, so every existing caller and snapshot test holds. - **Prometheus adapter** adds a top-level `warnings: []` on `PrometheusResponse` (also skip-if-empty), matching upstream's native API. `format_success_response` + `format_range_success_response` thread through any warnings the engine populated; absent → no field. - **Dispatcher** drops the combinable-only early return. The full loop now runs for every statistic whenever the timeline has ≥2 segments. `combine_statistic` still returns `Full(v)` for cleanly-combinable inputs and `Partial { covered, missing }` everywhere else — on Partial we accumulate the `covered` scalar (when present), flag `any_partial`, and build a human-readable warnings list with the metric, range, statistic, dropped-group count, and up to three unresolved segments (agg_id, clipped range, status, coverage). Over-three are summarised; the full set is still inspectable via `GET /api/v1/db/timeline`. 5 new unit tests: 3 in `engines::query_result::tests` covering the default-empty / with-warnings / matrix wire contract; 2 in the Prometheus adapter covering the response-side serialisation contract. 734 lib tests pass (+5), clippy clean, fmt clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pure read-side
BackfillRegistry::coverage(agg_id, range) -> Coverage— data primitive the query path will consult to classify a backfill range asComplete/BackfillInProgress { job_id, pct }/Missingper §10.4.Completejob windows foragg_idinto a coverage set.Complete.Runningjobs; overlap →BackfillInProgresswith current progress.Cancelled/Failedjobs don't contribute.Complete.Scope
Only backfill state is considered. Callers split at
schema.created_at_ms(§10.5 time-disjoint) and ask this method about the historical portion only; live[created_at, end)is alwaysCompleteby construction.Test plan
range_covered_byinterval math × 5,coveragebehaviour × 9 (missing/complete/partial/running/cancelled+failed exclusions/complete-precedes-running/empty-range).Next
Phase 5f-b wires this into SimpleEngine, depends on Phase 3b-2's per-segment dispatch landing first.
🤖 Generated with Claude Code