feat(sketch-db): Phase 5h — PrometheusReader (real RawSampleReader for backfill) - #35
Merged
Merged
Conversation
…r backfill)
First real `RawSampleReader` in-tree. Turns the `BackfillSource::Prometheus
{ url }` variant from "factory returns no-reader error" into an actual
working read path, so `--enable-backfill-worker` against a Prometheus-
backed deployment now produces real backfills instead of shadow-mode
Failed jobs.
## What's landed
`src/stores/sketch_db/prometheus_reader.rs`:
* `PrometheusReader::new(base_url) + with_step(d) + with_timeout(d)` —
HTTP client wrapper over `/api/v1/query_range`. Step defaults to 15s
(typical Prometheus scrape interval); overridable per-deployment.
* `impl RawSampleReader` that:
1. Translates `LabelFilter { metric, equality }` into a PromQL
matcher `metric{k1="v1",k2="v2"}` with stable key ordering and
proper escape for quotes/backslashes/newlines.
2. Issues a `GET /api/v1/query_range` with `query`, `start`, `end`,
`step` params.
3. Parses the `{status: "success", data: {resultType: "matrix",
result: [{metric, values}]}}` JSON shape into `Vec<RawSample>`.
4. Maps every failure into `RawSampleReaderError::{InvalidRange,
Upstream, Decode}` so the worker can surface the root cause on
`BackfillJob::error_message`.
* `ms_to_fractional_seconds` / `fractional_seconds_to_ms` helpers for
the ms ↔ Prometheus-fractional-seconds conversion.
* `render_series_key` rebuilds `metric{labels...}` from Prometheus's
response metric map so downstream grouping and `extract_group_key`
see the same shape they see for live samples.
`backfill_service::default_reader_factory()`:
* Routes `BackfillSource::Prometheus { url }` to `PrometheusReader::new`.
* Returns a clear "reader for <source> not yet implemented" error for
`S3Gorilla` / `ClickHouse` / `OtherSketch` so the controller sees
exactly which reader is missing.
`main.rs` now uses `default_reader_factory()` under
`--enable-backfill-worker` instead of `noop_reader_factory`, so the
service actually does work against Prometheus targets.
## On determinism (§10.5)
`/api/v1/query_range` with step = scrape_interval returns one point
per step per series. For deployments whose live ingest is Prometheus
remote write (backend-side sketch-core construction), this gives the
same sample sequence live saw, so the bit-identical parity test from
PR #32 still holds. For deployments whose live ingest goes through
DC's OTLP sketch-building path, approximate equivalence holds; exact
bit-identical would need `/api/v1/read` + sketchlib-go parity work.
Module doc flags that `/api/v1/read` (the protobuf remote-read
protocol) is the upgrade path if we ever need sub-scrape-interval
resolution; trait contract is unchanged.
## Test plan
- [x] 8 unit tests: `build_promql` ordering + escape + bare-metric,
`render_series_key` ordering + bare + fallback, fractional-seconds
round-trip, inverted-range → `InvalidRange`.
- [x] 8 integration tests against a spawned axum mock Prometheus:
happy-path multi-series parse + param propagation, PromQL matcher
synthesis, HTTP 5xx → Upstream, Prometheus `status: error` →
Upstream, malformed JSON → Decode, ingest-order preservation
per series, empty result, wrong `resultType` → Decode.
- [x] 677 lib tests pass (up from 661).
- [x] clippy `--workspace --all-targets --tests -- -D warnings` clean.
- [x] `cargo fmt -- --check` clean.
## What this unblocks
A Prometheus-backed deployment can now:
1. Create a backfill job via `POST /api/v1/db/backfill` with
`source: {Prometheus: {url: "http://prom:9090"}}`.
2. Start the backend with `--enable-backfill-worker`.
3. Watch the job transition `Queued → Running → Complete` in
`/api/v1/db/backfill/jobs/<id>`, with per-window precomputes
written to the store bit-identically to what live ingest would
have produced.
`S3Gorilla` and `ClickHouse` readers are the remaining Phase 5h-2 /
5h-3 work; `OtherSketch` (lossless widening from an existing sketch)
is Phase 5i.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First real
RawSampleReaderin-tree. TurnsBackfillSource::Prometheusfrom "factory returns no-reader error" into an actual working read path, so--enable-backfill-workeron a Prometheus-backed deployment now produces real backfills instead of shadow-modeFailedjobs.PrometheusReader::new(base_url)+with_step+with_timeout, wraps/api/v1/query_range.LabelFilter→ PromQL matcher with stable key ordering + quote/backslash escape.{status: success, data: {resultType: matrix, result: [{metric, values}]}}intoVec<RawSample>.RawSampleReaderError::{InvalidRange, Upstream, Decode}so the worker marks jobsFailedwith a clearerror_message.default_reader_factory()routesPrometheusto this reader; other variants still return "not yet implemented" per-variant.main.rsusesdefault_reader_factory()under--enable-backfill-worker(replacingnoop_reader_factory).Determinism
For raw-ingest deployments (live = Prometheus remote write), step=scrape_interval makes
query_rangereturn the same sample sequence live saw; the bit-identical parity test from PR #32 still holds. For DC-ingest deployments, approximate equivalence ε./api/v1/read(protobuf remote-read) is the upgrade path if we ever need sub-step resolution; trait contract unchanged.Test plan
What this unblocks
End-to-end backfill on Prometheus-backed deployments:
POST /api/v1/db/backfillwith{source: {Prometheus: {url}}}, start backend with--enable-backfill-worker, job goesQueued → Running → Completewith bit-identical per-window precomputes in the store.S3Gorilla / ClickHouse readers are Phase 5h-2 / 5h-3; OtherSketch is Phase 5i.
🤖 Generated with Claude Code