mvp phase α: backend hot-loads controller-emitted BackendStorageRouting - #96
Merged
Merged
Conversation
Pairs with ASAPCollector#314 (controller emitter). Today the per-metric `BackendStorageRouting` table is hand-authored YAML that the backend reads at startup; after Phase α the controller emits it as part of its plan and the backend hot-loads it on push, closing the controller → backend feedback loop. Concretely: * `BackendStorageRouting::from_json_payload(&JsonValue)` — parses the controller-emitted JSON document (schema mirrors `controller/src/config/stage_config.rs::emit_backend_storage_routing`). Maps engine-name strings — `sketch_warm_tier`, `thanos_archive` (alias for `gorilla_s3_archive`), `gorilla_s3_archive`, `double_write` — into the existing `StorageBackend` variants. Unknown query-shape strings map to `QueryShape::Other` for forward-compat. Side fields (`warm_tier_native_shapes`) are ignored. * `QueryShape` extended with `HistogramQuantile`, `Delta`, `Deriv`, `Absent` — the controller's archive-eligible shape vocabulary. `classify_query_shape` updated to recognise the matching PromQL function names (`histogram_quantile`, `delta`/`increase`, `deriv`, `absent`/`absent_over_time`). * `HotReloadBackendStorageRouting` — `ArcSwap`-backed wrapper mirroring `HotReloadStreamingConfig`. Lets the swap handler atomically replace the routing table at runtime without restart. Cloneable; clones share the underlying `ArcSwap` so all holders see the same swaps. * `routing_table_hash(&BackendStorageRouting)` — stable short hash for the swap-response body. Lets the controller verify the backend installed exactly the bytes it pushed. * `POST /api/v1/storage_routing` HTTP endpoint — accepts the JSON document, validates via `from_json_payload`, atomically swaps the table via the hot-reload handle, returns 200 with the new table's hash + entry count. 400 on invalid JSON / schema; 503 when the backend wasn't built with a routing handle. * `GET /api/v1/storage_routing` — JSON snapshot for operator diagnostics (default engine, metrics count, table hash). * `HttpServer::with_backend_storage_routing` — now wraps the `Arc<BackendStorageRouting>` it receives in a hot-reload handle internally, so existing call sites keep their signature. The internal field type changed from `Option<Arc<BackendStorageRouting>>` to `Option<HotReloadBackendStorageRouting>`. The single read site in `resolve_metric_storage` snapshots the handle once per request, giving the dispatcher torn-read-free access to the current table. * `HttpServer::with_hot_reload_backend_storage_routing` — sibling builder that takes a pre-built handle, for callers that want to share it with other subsystems. * Bootstrap: `main.rs` and `bin/precompute_engine.rs` now always install a hot-reload routing handle — bootstrap from `--backend-storage-routing` YAML when set, an empty table otherwise. Operators can still hand-author the YAML for dev / standalone deployments; controller pushes overwrite it. Tests: * 14 new unit tests in `routing::backend_storage_routing` — `from_json_payload` happy path / unknown shape / invalid engine / empty targets / missing metrics / optional default / back-compat alias; `replace` in-place swap; hot-reload wrapper visibility + concurrent-reader torn-state guard; routing-table-hash stability. Plus `HistogramQuantile` / `Delta` / `Absent` classifier tests. * 4 new HTTP integration tests in `drivers::query::servers::http` — POST swaps atomically; POST rejects invalid JSON without partial swap; GET returns a snapshot whose hash matches the live table; swap is observed by subsequent dispatch lookups. Backend lib tests: 875 pass / 33 pre-existing fail / 9 ignored (vs origin/main: 856 pass / 33 fail / 9 ignored — +19 new passing tests, 0 new failures). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 7, 2026
The repo is now the backend query engine of the ASAP observability system, not a standalone Prometheus-front query accelerator. Reflects the architecture that landed via: - Step 1 (#95) — engines/{simple,gorilla}/ tier-co-located layout, routing/ split out from drivers/, JSONL legacy code deleted - Phase α (#314 controller, #96 backend) — controller emits BackendStorageRouting JSON; backend hot-loads via HTTP push - Phase β (#315) — asap-planner-rs library's 5 PromQL patterns + 11 archive-only intents migrated into ASAPCollector controller's L3 intent_algebra + L4 sketch_algebra - Step 2.3 (#97) — ThanosForwardEngine in engines/gorilla/thanos_forward.rs; HTTP-forwards archive queries to thanos-query for full PromQL surface (Path A2) Key narrative changes: - Lead with two-engine architecture (warm sketch tier + archive via Thanos), not the single-tier query-accelerator framing - Explicit "Where the planner lives" section pointing at the ASAPCollector controller (was previously asap-planner-rs in this repo) - Repository layout reflects the new engines/{simple,gorilla}/ + routing/ split - Configuration via env vars (ASAP_THANOS_QUERY_URL, ASAP_GORILLA_S3_*, CONTROLLER_BACKEND_ENDPOINT) matching the current code - Query response infos shows the three data_source values (warm / gorilla_archive / thanos_archive) - "What's NOT in this repo" section enumerates the migrations and deletions for reviewers landing here from old docs - "What's still planned" tracks Phase δ (backend pure executor), per-tenant routing, hot-reload signal Drops: 100x latency reduction headline (was the asap-quickstart v0.1.0 framing — still true qualitatively but not the lede); the 4-component repo-structure diagram (most of those components either moved or have changed scope). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 7, 2026
The repo is now the backend query engine of the ASAP observability system, not a standalone Prometheus-front query accelerator. Reflects the architecture that landed via: - Step 1 (#95) — engines/{simple,gorilla}/ tier-co-located layout, routing/ split out from drivers/, JSONL legacy code deleted - Phase α (#314 controller, #96 backend) — controller emits BackendStorageRouting JSON; backend hot-loads via HTTP push - Phase β (#315) — asap-planner-rs library's 5 PromQL patterns + 11 archive-only intents migrated into ASAPCollector controller's L3 intent_algebra + L4 sketch_algebra - Step 2.3 (#97) — ThanosForwardEngine in engines/gorilla/thanos_forward.rs; HTTP-forwards archive queries to thanos-query for full PromQL surface (Path A2) Key narrative changes: - Lead with two-engine architecture (warm sketch tier + archive via Thanos), not the single-tier query-accelerator framing - Explicit "Where the planner lives" section pointing at the ASAPCollector controller (was previously asap-planner-rs in this repo) - Repository layout reflects the new engines/{simple,gorilla}/ + routing/ split - Configuration via env vars (ASAP_THANOS_QUERY_URL, ASAP_GORILLA_S3_*, CONTROLLER_BACKEND_ENDPOINT) matching the current code - Query response infos shows the three data_source values (warm / gorilla_archive / thanos_archive) - "What's NOT in this repo" section enumerates the migrations and deletions for reviewers landing here from old docs - "What's still planned" tracks Phase δ (backend pure executor), per-tenant routing, hot-reload signal Drops: 100x latency reduction headline (was the asap-quickstart v0.1.0 framing — still true qualitatively but not the lede); the 4-component repo-structure diagram (most of those components either moved or have changed scope). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
May 7, 2026
* mvp phase γ: delete asap-planner-rs workspace member + drop backend planner driver Phase α (#96) made the ASAPCollector controller the sole emitter of BackendStorageRouting; Phase β migrated all 5 PromQL pattern matchers and 11 archive-only intents into the controller; Step 2.3 (#97) wired the backend to forward archive queries to thanos-query. The `asap-planner-rs/` library + CLI in this repo is now redundant. Delete the entire `asap-planner-rs/` workspace member (49 files, ~5.1k LOC) and surgically remove its only remaining importer in `asap-query-engine`: - `src/planner_client.rs` — `LocalPlannerClient` was the in-process bridge that called `asap_planner::Controller::generate()` from the backend's query tracker. - `src/query_tracker/` — the passive auto-discovery loop that drained observed PromQL queries and called the planner client. Both are superseded by the controller observing queries via its own scrape side-channel and pushing plans / routing tables back via `POST /api/v1/streaming-config` and `POST /api/v1/storage_routing`. - `--enable-query-tracker` / `--tracker-observation-window-secs` CLI args and the `asap_planner` workspace dependency on `asap-query-engine` are dropped. - `HttpServer::new` loses its `Option<Arc<QueryTracker>>` arg; the `record_instant` / `record_range` hooks in instant- and range-query handlers are removed. - `asap-query-engine/Dockerfile` no longer COPYs `asap-planner-rs` sources or stubs them in the dependency cache layer. Backend builds clean (`cargo build --release`); lib tests show the same 34 pre-existing failures as origin/main with no new regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * mvp phase γ: refresh README + quickstart + docs after asap-planner-rs deletion - `README.md`: drop the `asap-planner-rs/` entry from the repo-tree diagram; refine the "Where the planner lives" prose to say the legacy crate was deleted (not "CLI retained"); point external scripters at the controller's HTTP / CLI surface. - `asap-quickstart/README.md`: rename the planner role to "Plan emitter (init container)" and note that the demo currently uses the published `ghcr.io/projectasap/asap-planner-rs:v0.2.0` image but will switch to a controller HTTP push under Phase δ. Update the "Edit the asap-planner-rs Config" subsection accordingly. - `asap-quickstart/docker-compose.dev.yml`: drop the local `asap-planner-rs` build override (the source no longer exists in this repo); the base compose still pins the published image. - `asap-quickstart/Dockerfile.queryengine-local`: drop the stale comment cross-referencing `asap-planner-rs/Cargo.toml`. - `asap-summary-ingest/README.md`: redirect references to the controller in `ASAPCollector/`. - `docs/`: refresh component indexes, architecture diagrams, the add-new-sketch guide, the bootstrap-config-from-query-log how-to, and the publishing-docker-images table to point at the ASAPCollector controller as the canonical planner location. Historical design docs that mention the old path are left as-is (they are records of past state). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pairs with ASAPCollector#314 (controller emitter, already merged). Today the per-metric
BackendStorageRoutingtable is hand-authored YAML; after Phase α the controller emits it from its typed L5 stage_config and the backend hot-loads it on push, closing the controller → backend feedback loop.BackendStorageRouting::from_json_payload— parses the controller-emitted JSON document. Mapsthanos_archive(controller name) andgorilla_s3_archive(legacy YAML name) both toStorageBackend::GorillaS3Archive; rejects unknown engines.QueryShapeextended withHistogramQuantile/Delta/Deriv/Absent.classify_query_shapeupdated.HotReloadBackendStorageRouting—ArcSwap-backed atomic-swap wrapper, mirrorsHotReloadStreamingConfig.POST /api/v1/storage_routing— validates → swaps atomically → returns the new table's hash + entry count.GET /api/v1/storage_routing— JSON snapshot for operator diagnostics.HttpServer::with_backend_storage_routing— wraps internally in a hot-reload handle; existing call sites keep their signature.deploy/configs/backend-storage-routing.yaml. Controller pushes overwrite it.Test plan
Coordination
🤖 Generated with Claude Code