mvp step 2.3: backend HTTP-forward to thanos-query for Path A2 archive - #97
Merged
Merged
Conversation
…er slot
Step-2.3 core abstraction. `ThanosForwardEngine` is a `QueryEngine`
that POSTs PromQL to a `thanos-query` sidecar at
`${ASAP_THANOS_QUERY_URL}/api/v1/query`, parses the
Prometheus-format response, and wraps it in ASAP's
`QueryResult` shape with an exact accuracy envelope. Tolerates 5xx
/ network errors / timeouts as `ThanosForwardError::Unreachable`,
which the trait impl folds into `EngineError::Backend` so the
router's failover sequence falls through to the warm-tier sketch
on a `DoubleWrite` deploy.
`EngineRouter::register_aliased(id, engine)` registers an engine
under an explicit `data_source_id` instead of the one its
`capabilities()` reports. Step-2.3's binary wiring uses this to
register a single `ThanosForwardEngine` instance under both
`thanos_archive` (its native id, for explicit `X-ASAP-Engine`
overrides) and `gorilla_archive` (the legacy archive slot the
existing `compatible_storage_backends` failover sequence walks).
12 unit tests cover the forwarder (success path, capabilities,
unreachable upstream → 503-quirk, env-var parsing, payload
parsing); 2 unit tests cover the new router alias path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tests When `ASAP_THANOS_QUERY_URL` is set, both `main.rs` (full backend) and `bin/precompute_engine.rs` (deploy/docker image) register a `ThanosForwardEngine` under both `thanos_archive` (its native id) and `gorilla_archive` (the legacy archive slot the failover sequence walks). The legacy in-process `GorillaQueryEngine` is skipped in this mode. When the env var is unset, the binary falls back to the existing `ASAP_GORILLA_S3_*` registration path verbatim — dev environments without thanos still work. `HttpServer::with_query_engine_aliased(id, engine)` is the builder-side surface for the new `EngineRouter::register_aliased` slot; the binary calls it twice with the same `Arc<dyn QueryEngine>` to register under both ids. 3 HTTP integration tests pin the full wire path: * Failover dispatch → forwards to mock thanos → 2xx with `data_source: gorilla_archive` (Path A2 re-uses the archive tier slot). * Unreachable upstream → 5xx with `thanos_unreachable` marker in the error body. * `X-ASAP-Engine: thanos_archive` override → 2xx with `data_source: thanos_archive` annotation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
mvp/step2.3-backend-thanos-forward
branch
from
May 7, 2026 21:04
6256315 to
2001914
Compare
zzylol
added a commit
that referenced
this pull request
May 7, 2026
The repo is now the backend query engine of the ASAP observability system, not a standalone Prometheus-front query accelerator. Reflects the architecture that landed via: - Step 1 (#95) — engines/{simple,gorilla}/ tier-co-located layout, routing/ split out from drivers/, JSONL legacy code deleted - Phase α (#314 controller, #96 backend) — controller emits BackendStorageRouting JSON; backend hot-loads via HTTP push - Phase β (#315) — asap-planner-rs library's 5 PromQL patterns + 11 archive-only intents migrated into ASAPCollector controller's L3 intent_algebra + L4 sketch_algebra - Step 2.3 (#97) — ThanosForwardEngine in engines/gorilla/thanos_forward.rs; HTTP-forwards archive queries to thanos-query for full PromQL surface (Path A2) Key narrative changes: - Lead with two-engine architecture (warm sketch tier + archive via Thanos), not the single-tier query-accelerator framing - Explicit "Where the planner lives" section pointing at the ASAPCollector controller (was previously asap-planner-rs in this repo) - Repository layout reflects the new engines/{simple,gorilla}/ + routing/ split - Configuration via env vars (ASAP_THANOS_QUERY_URL, ASAP_GORILLA_S3_*, CONTROLLER_BACKEND_ENDPOINT) matching the current code - Query response infos shows the three data_source values (warm / gorilla_archive / thanos_archive) - "What's NOT in this repo" section enumerates the migrations and deletions for reviewers landing here from old docs - "What's still planned" tracks Phase δ (backend pure executor), per-tenant routing, hot-reload signal Drops: 100x latency reduction headline (was the asap-quickstart v0.1.0 framing — still true qualitatively but not the lede); the 4-component repo-structure diagram (most of those components either moved or have changed scope). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 7, 2026
The repo is now the backend query engine of the ASAP observability system, not a standalone Prometheus-front query accelerator. Reflects the architecture that landed via: - Step 1 (#95) — engines/{simple,gorilla}/ tier-co-located layout, routing/ split out from drivers/, JSONL legacy code deleted - Phase α (#314 controller, #96 backend) — controller emits BackendStorageRouting JSON; backend hot-loads via HTTP push - Phase β (#315) — asap-planner-rs library's 5 PromQL patterns + 11 archive-only intents migrated into ASAPCollector controller's L3 intent_algebra + L4 sketch_algebra - Step 2.3 (#97) — ThanosForwardEngine in engines/gorilla/thanos_forward.rs; HTTP-forwards archive queries to thanos-query for full PromQL surface (Path A2) Key narrative changes: - Lead with two-engine architecture (warm sketch tier + archive via Thanos), not the single-tier query-accelerator framing - Explicit "Where the planner lives" section pointing at the ASAPCollector controller (was previously asap-planner-rs in this repo) - Repository layout reflects the new engines/{simple,gorilla}/ + routing/ split - Configuration via env vars (ASAP_THANOS_QUERY_URL, ASAP_GORILLA_S3_*, CONTROLLER_BACKEND_ENDPOINT) matching the current code - Query response infos shows the three data_source values (warm / gorilla_archive / thanos_archive) - "What's NOT in this repo" section enumerates the migrations and deletions for reviewers landing here from old docs - "What's still planned" tracks Phase δ (backend pure executor), per-tenant routing, hot-reload signal Drops: 100x latency reduction headline (was the asap-quickstart v0.1.0 framing — still true qualitatively but not the lede); the 4-component repo-structure diagram (most of those components either moved or have changed scope). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
May 7, 2026
* mvp phase γ: delete asap-planner-rs workspace member + drop backend planner driver Phase α (#96) made the ASAPCollector controller the sole emitter of BackendStorageRouting; Phase β migrated all 5 PromQL pattern matchers and 11 archive-only intents into the controller; Step 2.3 (#97) wired the backend to forward archive queries to thanos-query. The `asap-planner-rs/` library + CLI in this repo is now redundant. Delete the entire `asap-planner-rs/` workspace member (49 files, ~5.1k LOC) and surgically remove its only remaining importer in `asap-query-engine`: - `src/planner_client.rs` — `LocalPlannerClient` was the in-process bridge that called `asap_planner::Controller::generate()` from the backend's query tracker. - `src/query_tracker/` — the passive auto-discovery loop that drained observed PromQL queries and called the planner client. Both are superseded by the controller observing queries via its own scrape side-channel and pushing plans / routing tables back via `POST /api/v1/streaming-config` and `POST /api/v1/storage_routing`. - `--enable-query-tracker` / `--tracker-observation-window-secs` CLI args and the `asap_planner` workspace dependency on `asap-query-engine` are dropped. - `HttpServer::new` loses its `Option<Arc<QueryTracker>>` arg; the `record_instant` / `record_range` hooks in instant- and range-query handlers are removed. - `asap-query-engine/Dockerfile` no longer COPYs `asap-planner-rs` sources or stubs them in the dependency cache layer. Backend builds clean (`cargo build --release`); lib tests show the same 34 pre-existing failures as origin/main with no new regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * mvp phase γ: refresh README + quickstart + docs after asap-planner-rs deletion - `README.md`: drop the `asap-planner-rs/` entry from the repo-tree diagram; refine the "Where the planner lives" prose to say the legacy crate was deleted (not "CLI retained"); point external scripters at the controller's HTTP / CLI surface. - `asap-quickstart/README.md`: rename the planner role to "Plan emitter (init container)" and note that the demo currently uses the published `ghcr.io/projectasap/asap-planner-rs:v0.2.0` image but will switch to a controller HTTP push under Phase δ. Update the "Edit the asap-planner-rs Config" subsection accordingly. - `asap-quickstart/docker-compose.dev.yml`: drop the local `asap-planner-rs` build override (the source no longer exists in this repo); the base compose still pins the published image. - `asap-quickstart/Dockerfile.queryengine-local`: drop the stale comment cross-referencing `asap-planner-rs/Cargo.toml`. - `asap-summary-ingest/README.md`: redirect references to the controller in `ASAPCollector/`. - `docs/`: refresh component indexes, architecture diagrams, the add-new-sketch guide, the bootstrap-config-from-query-log how-to, and the publishing-docker-images table to point at the ASAPCollector controller as the canonical planner location. Historical design docs that mention the old path are left as-is (they are records of past state). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
zzylol
added a commit
that referenced
this pull request
May 7, 2026
Adds a `PrometheusForwardEngine` (HTTP-forwarder to Prometheus's `/api/v1/query`) under the `prometheus_remote` engine id so the controller's Mode 3 (`RawAtEdgePrometheusArchive`) routing entries have a dispatchable target on the backend. Mirror of Step-2.3's `ThanosForwardEngine` (PR #97), pointed at Prometheus's standard HTTP API instead of a thanos-query sidecar. Wiring is conditional on `ASAP_PROMETHEUS_QUERY_URL`: when set, the binary registers the engine on the capability router (both `main.rs` and `bin/precompute_engine.rs`); when unset, the engine is not registered and a routing-table entry that references `prometheus_remote` surfaces a clean `NoEngineRegistered` 503 from the HTTP handler — the correct fail-loud behaviour for a misconfigured deploy. Phase ε.1 (controller, separate repo) emits `engine: prometheus_remote` in the `BackendStorageRouting` JSON; this PR closes the loop by accepting that string in `parse_engine_string` and resolving it to the new `StorageBackend::PrometheusRemote` variant. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the post-Step-1 backend to HTTP-forward archive-tier PromQL queries to a
thanos-querysidecar, gated byASAP_THANOS_QUERY_URL. This is the smallest piece left to complete the Path A2 dispatch chain (Step 2.1 PR #311 emits Prometheus TSDB blocks; Step 2.2 PR #310 adds the thanos sidecar to the demo overlay).ThanosForwardEngine(new fileengines/gorilla/thanos_forward.rs) — implementsQueryEngine, POSTs PromQL to${ASAP_THANOS_QUERY_URL}/api/v1/query, parses the Prometheus-format response, wraps it with an exact accuracy envelope. Tolerates 5xx / timeouts asThanosForwardError::Unreachable→EngineError::Backendso the router falls through to the warm-tier sketch on aDoubleWritedeploy.EngineRouter::register_aliased(id, engine)+HttpServer::with_query_engine_aliased— register a single engine instance under multipledata_source_ids. The binary uses this to registerThanosForwardEngineunder boththanos_archive(native id, for explicitX-ASAP-Engineoverrides) andgorilla_archive(the legacy archive slot the existing failover sequence walks).main.rs+bin/precompute_engine.rs) — env set → register thanos forwarder, skip legacy in-processGorillaQueryEngine. Env unset → fall back to existingASAP_GORILLA_S3_*registration verbatim. Dev environments without thanos still work.Engine ID naming
thanos_archivefor the native id (explicit overrides); aliased ontogorilla_archiveso the existingcompatible_storage_backends(...)failover sequence finds it without modifyingasap_typesorrouting/backend_storage_routing.rs. Path A2 re-uses the archive tier slot — only the engine answering changes.Test plan
engines/gorilla/thanos_forward.rs(forwarder success path, capabilities, alias override, 503-quirk, env-var parsing, payload parsing)routing/engine_router.rs(register_aliasedlookup + dispatch)drivers/query/servers/http.rs(full HTTP path → mock thanos sidecar → wrapped response; unreachable upstream → 5xx withthanos_unreachablemarker;X-ASAP-Engine: thanos_archiveoverride →data_source: thanos_archive)cargo test --release --lib --no-fail-fast—query_engine_rust873 passed / 33 failed (matches pre-existing 33-failure baseline; no new regressions)Step 2.4 (full e2e demo) hand-off notes
ASAP_THANOS_QUERY_URL(defaulthttp://thanos-query:10903per Step-2.2 overlay).thanos_archive(explicit override target),gorilla_archive(failover dispatch target — same engine instance).data_sourceannotations: failover-dispatch path →data_source: gorilla_archive(the tier); explicit override →data_source: thanos_archive(the engine implementation).GorillaQueryEngineregistered iffASAP_GORILLA_S3_*is also set; otherwise router serves warm-tier metrics only.🤖 Generated with Claude Code