diff --git a/README.md b/README.md index 583aa4d9..85e85e28 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Unified control plane for the ASAP data-lifecycle stack. -This repo merges three previously-separate projects into a single workspace with a shared core and per-scenario plugin crates: +This repo merges three previously-separate projects into a single workspace with a shared core and per-deployment-model plugin crates: - `DataCollector/controller` — end-to-end lifecycle planner (collection → transmission → storage → analytics). - `ASAPQuery[-backend]/asap-planner-rs` — analytics-query-only planner (CLI, YAML in → YAML out). @@ -23,36 +23,37 @@ This repo merges three previously-separate projects into a single workspace with |---|---|---| | 1 | Query-language parsing (PromQL, SQL, DataFusion, ElasticDSL) | `core::query_language` | | 2 | Per-language logical plan (relational algebra tree) | `core::logical_plan` | -| 3 | Sketch algebra — language-, deployment-, AND data-model-independent IR (`QueryExpr` + `AggIntent`, intent only). Supports both time-series and tabular data via a `Source` sum inside `Scan`. | `core::sketch_algebra` | -| 4 | Cost-aware optimizer — rule engine + shared rule library; scenarios pick rules | **framework in `core::optimizer`**; choices in each scenario | -| 5 | Physical plan — stage allocation + emit to wire format | **framework in `core::physical`**; topology + emitter in each scenario | +| 3 | Intent algebra — language-, deployment-, AND data-model-independent IR (`QueryExpr` + `AggIntent`, intent only — no sketch type, no params). Supports both time-series and tabular data via a `Source` sum inside `Scan`. | `core::intent_algebra` | +| 4 | Cost-aware optimizer — rule engine + shared rule library; deployment models pick rules. Produces the **sketch algebra** IR `SketchExpr` (sketch-bound: kind + params committed). | **framework in `core::optimizer`**; sketch-bound IR in `core::sketch_algebra`; rule choices in each deployment model | +| 5 | Physical plan — stage allocation + emit to wire format | **framework in `core::physical`**; topology + emitter in each deployment model | -**Core owns the shared infrastructure across all 5 layers**, not just L1-3. Scenarios are thin: they pick which rules fire (L4), declare their deployment topology (L5), and provide an emitter for their output format. Typical scenario crate: **500-2500 LOC**. +**Core owns the shared infrastructure across all 5 layers**, not just L1-3. Deployment models are thin: they pick which rules fire (L4), declare their deployment topology (L5), and provide an emitter for their output format. Typical deployment model crate: **500-2500 LOC**. ``` crates/ core/ # all 5 layers of shared infrastructure; no I/O query_language/ # L1 — parsers logical_plan/ # L2 — per-language algebra trees - sketch_algebra/ # L3 — QueryExpr + AggIntent (intent only, ~25 variants superset) + intent_algebra/ # L3 IR — QueryExpr + AggIntent (intent only, ~25 variants superset) + sketch_algebra/ # L4 IR — SketchExpr (sketch-bound: kind + params committed) lower/ # L1→L2→L3 passes - optimizer/ # L4 framework — rule engine + shared rule library + cost traits + optimizer/ # L4 framework — rule engine + shared rule library + cost traits; produces SketchExpr physical/ # L5 framework — PhysicalPlanner + stage allocator + topology + sketch catalogue - pipeline/ # L1→…→L5 driver, parameterized on scenario + pipeline/ # L1→…→L5 driver, parameterized on deployment model runtime/ # HTTP / OpAMP / replanner / store — service skeleton - scenario-lifecycle # DC-specific rules + 3-stage topology + OTel/backend emitters - scenario-query # precompute-engine rules + 1-stage + StreamingConfig/InferenceConfig YAML emitters - scenario-fusion # DataFusion rewrites + 0-stage (in-process) + LogicalPlan emit + deployment-model-asaplifecycle # DC-specific rules + 3-stage topology + OTel/backend emitters + deployment-model-asapquery # precompute-engine rules + 1-stage + StreamingConfig/InferenceConfig YAML emitters + deployment-model-asapfusion # DataFusion rewrites + 0-stage (in-process) + LogicalPlan emit control-proto # OpAMP proto + internal proto (tonic/prost) testing # shared fixtures bin/ - asap-controller # long-running service, all scenarios - asap-query # one-shot CLI (scenario-query only) - asap-lifecycle # OPTIONAL: standalone service with only scenario-lifecycle - asap-fusion-bench # OPTIONAL: benchmark harness over scenario-fusion + asap-controller # long-running service, all deployment models + asap-query # one-shot CLI (deployment-model-asapquery only) + asap-lifecycle # OPTIONAL: standalone service with only deployment-model-asaplifecycle + asap-fusion-bench # OPTIONAL: benchmark harness over deployment-model-asapfusion ``` -A new scenario lands by adding one crate with `rules.rs` (pick L4 rules) + `topology.rs` + an emitter, plus one line in `bin/asap-controller/main.rs`. No changes to core or runtime. +A new deployment model lands by adding one crate with `rules.rs` (pick L4 rules) + `topology.rs` + an emitter, plus one line in `bin/asap-controller/main.rs`. No changes to core or runtime. ## Consumption modes @@ -60,7 +61,7 @@ Three ways downstream can use ASAPController — mix as needed: | Mode | Use case | How | |---|---|---| -| **Rust library** | In-process use of the IR or a specific scenario (e.g. asap-fusion benchmarks) | `Cargo.toml`: `asap-control-core = { git = "...", tag = "v0.1.0" }` or any individual `scenario-*` crate. Per-crate dep isolation keeps dep trees small (scenario-fusion pulls DataFusion; scenario-query pulls PromQL/SQL parsers; neither pulls axum/OpAMP). | +| **Rust library** | In-process use of the IR or a specific deployment model (e.g. asap-fusion benchmarks) | `Cargo.toml`: `asap-control-core = { git = "...", tag = "v0.1.0" }` or any individual `deployment-model-*` crate. Per-crate dep isolation keeps dep trees small (deployment-model-asapfusion pulls DataFusion; deployment-model-asapquery pulls PromQL/SQL parsers; neither pulls axum/OpAMP). | | **HTTP service sidecar** | Production control plane — e.g. ASAPQuery-backend POSTing QuerySpec on capability-miss | Run `asap-controller` binary, POST to `/api/v1/plan`. Same contract DC controller speaks today. | | **CLI / Docker image** | One-shot init container (e.g. docker-compose init job that writes `streaming_config.yaml`) | `asap-controller plan --workload ... --output-dir ...` or the dedicated `asap-query` binary | @@ -68,23 +69,23 @@ Three ways downstream can use ASAPController — mix as needed: ASAPController is the **control plane only**. The data plane — OTel collectors, ASAPQuery-backend's query engine, asap-fusion users' DataFusion runtimes — stays in its original repo. Communication is always over wire (OpAMP, HTTP, Prometheus scrape). This boundary is unchanged by the merger. -## Scenario placement: flexible +## Deployment model placement: flexible -Because core now owns the L4/L5 infrastructure (not just L1-3), scenario crates are small and largely self-contained. A scenario can live either: +Because core now owns the L4/L5 infrastructure (not just L1-3), deployment model crates are small and largely self-contained. A deployment model can live either: -- **Inside ASAPController workspace** (lockstep release with core, one-PR cross-scenario changes) +- **Inside ASAPController workspace** (lockstep release with core, one-PR cross-deployment-model changes) - **In its own downstream repo** (independent release cadence, depends on published `asap-control-core`) -Both produce functionally identical artefacts. The placement is a deployment/team-ownership decision, not an architectural fork. Default: -- `scenario-lifecycle` + `scenario-query` in ASAPController workspace (share a YAML emitter). -- `scenario-fusion` in the `asap-fusion` repo (research cadence, independent release). +Both produce functionally identical artifacts. The placement is a deployment/team-ownership decision, not an architectural fork. Default: +- `deployment-model-asaplifecycle` + `deployment-model-asapquery` in ASAPController workspace (share a YAML emitter). +- `deployment-model-asapfusion` in the `asap-fusion` repo (research cadence, independent release). ## Key design references - 5-layer pipeline: `docs/design.md` §3 - Core internals (L1-3 + L4 framework + L5 framework): `docs/design.md` §6 -- Scenario internals (each's rules + topology + emitter): `docs/design.md` §8 -- Extension point for future scenarios: `docs/design.md` §9 +- Deployment model internals (each's rules + topology + emitter): `docs/design.md` §8 +- Extension point for future deployment models: `docs/design.md` §9 - Consumption modes + dependency isolation: `docs/design.md` §11 -- Open questions (incl. L2-tree contract, scenario placement): `docs/design.md` §12 +- Open questions (incl. L2-tree contract, deployment model placement): `docs/design.md` §12 - Phase-by-phase migration: `docs/migration-plan.md` diff --git a/docs/design.md b/docs/design.md index b5d95fca..5b8b1d8e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -11,8 +11,8 @@ Merges three existing codebases into one: ## 1. Goals 1. **One repo, one workspace.** One `Cargo.toml` at root, one place to file issues, one release cadence. -2. **Common core, pluggable scenarios.** The three existing codebases each solve a *different* planning problem. Factor out what's actually shared; keep the scenario-specific parts as crates so each scenario can evolve independently. -3. **Extensible for future scenarios** (4th, 5th, …). A new scenario should land as a new crate that implements well-defined traits — not by touching the core or the runtime. +2. **Common core, pluggable deployment models.** The three existing codebases each solve a *different* planning problem. Factor out what's actually shared; keep the deployment-model-specific parts as crates so each deployment model can evolve independently. +3. **Extensible for future deployment models** (4th, 5th, …). A new deployment model should land as a new crate that implements well-defined traits — not by touching the core or the runtime. 4. **Preserve the two deployment shapes:** - **Service**: long-running HTTP+OpAMP process that accepts live `QuerySpec`s, replans on SLA violations, pushes configs to agents/backends. (DC controller today.) - **CLI**: one-shot "read workload YAML → emit `streaming_config.yaml` + `inference_config.yaml`". (`asap-planner-rs` today.) @@ -23,7 +23,7 @@ Merges three existing codebases into one: ## 2. Non-goals - **Not** redesigning the `Plan` IR end-to-end. DC's `algebra/` + fusion's `translator/optimizer/executor` stay where they are semantically; we merge them into a shared `Plan` crate with a clean trait seam, not a green-field rewrite. -- **Not** unifying DataFusion's `LogicalPlan` with DC's custom algebra at the type level. Those are different universes. Scenarios own their IR; core owns the *staged dispatch contract*. +- **Not** unifying DataFusion's `LogicalPlan` with DC's custom algebra at the type level. Those are different universes. Deployment models own their IR; core owns the *staged dispatch contract*. - **Not** in-scope: changing the existing wire protocol between controller and backend. ASAPQuery-backend keeps consuming `streaming_config.yaml` and keeps responding on `/api/v1/plan`. ## 3. The organizing spine: DC controller's 5-layer pipeline @@ -34,13 +34,13 @@ DataCollector/controller already documents its query→sketch translation as a 5 |---|-------|--------------|-------------------| | 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql,mod}.rs`; asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | | 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/algebra/lower.rs`; asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | -| 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent` (~25 intent variants). Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters.** Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | -| 4 | **Sketch Optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 and emit sketch-bound L3+. ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library; scenarios **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | -| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artefact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; scenarios supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | +| 3 | **Intent algebra** | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | +| 4 | **Sketch algebra + optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 (`QueryExpr`) and emit the sketch-bound IR (`SketchExpr`). ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library + the sketch-bound IR `core::sketch_algebra::SketchExpr`; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | +| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; deployment models supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | -**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every scenario reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same sketch-algebra IR (intent only, no sketch binding). +**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every deployment model reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same intent-algebra IR (intent only, no sketch binding). -**L4 and L5 also have substantial common infrastructure.** Initially we assumed scenarios owned L4/L5 wholesale; on closer inspection what's actually scenario-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes scenarios significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. +**L4 and L5 also have substantial common infrastructure.** Initially we assumed deployment models owned L4/L5 wholesale; on closer inspection what's actually deployment-model-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes deployment models significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. ### Sketch binding lives in L4, not L3 @@ -53,25 +53,25 @@ Today: - asap-fusion: correctly separated — `SketchConfigRule` at L4 fills `SketchConfig::NULL` with concrete `CountMinSketch{5,4096}` / `KLL{k=200,m=8}`. - asap-planner-rs: **L3+L4 fused today**. `map_statistic_to_precompute_operator` jumps from `Statistic` straight to `AggregationType::DatasketchesKLL{k=200}` in one call. **Phase 4 splits this** — `Statistic → AggIntent` at L3, `AggIntent + DeploymentConstraints → AggregationType + SketchParams` at L4. -### Intent vocabulary: DC's `AggIntent` is a superset; scenarios use subsets +### Intent vocabulary: DC's `AggIntent` is a superset; deployment models use subsets -DC has ~25 variants. Planner uses 9 (`Count, Sum, Cardinality, Increase, Rate, Min, Max, Quantile, Topk`). Fusion uses 3 (`Count, Sum, Quantile`). A scenario's L4+L5 only has to handle its own subset — but it reads and writes the **same `AggIntent` enum**. Adding a new intent (e.g. stddev) is a core change that scenarios can then opt in to. +DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because language-flavored synonyms (`QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core is 9 (`Count, Sum, Min, Max, Quantile, TopK, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps directly: `Topk` keeps its own intent (heavy-hitter sketches like SpaceSaving / CMS-with-heap compute it as a single primitive, so the intent earns L3 visibility). Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that deployment models opt into. ### Data-model support: both time-series and tabular ASAPQuery-backend / DC controller operate on time-series data (metrics + labels + timestamp); asap-fusion operates on tabular data (DataFusion `LogicalPlan` over relations) that may or may not be time-indexed. These two data models differ fundamentally in their leaf shape (`metric + labels + time` vs. `table + columns`), but they share everything above the leaf — filter semantics, aggregation semantics, sketches themselves. Core handles this with: -- **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Scenarios' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. -- **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase`, `QuantileOverTime`), or tabular-only (future additions for joins, correlated subqueries). +- **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Deployment models' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. +- **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Min`, `Max`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase` — both carry PromQL counter-reset semantics), or tabular-only (future additions for joins, correlated subqueries). - **Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values; that stream can come from a time-series window or a table column, the sketch does not know or care. So `BindKllOnQuantile` and siblings work uniformly across both. Practically, this means: -- `scenario-query` + `scenario-lifecycle` lower into `QueryExpr` with `Source::TimeSeries` leaves. -- `scenario-fusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). -- A hypothetical OLAP scenario that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. +- `deployment-model-asapquery` + `deployment-model-asaplifecycle` lower into `QueryExpr` with `Source::TimeSeries` leaves. +- `deployment-model-asapfusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). +- A hypothetical OLAP deployment model that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. -See §6 `core::sketch_algebra` for the concrete type sketches. +See §6 `core::intent_algebra` for the concrete type sketches. ### Scope: start single-query, grow into workload-aware @@ -81,36 +81,66 @@ The initial implementation can operate on **one query at a time** — L4 picks p ### L2 is mandatory; the tree shape is an evolvable contract -Every scenario must produce an L2 tree, even when the source language didn't originally come as one. asap-planner-rs's current approach (PromQL pattern catalogue → `IntermediateAggConfig`) skips L2; Phase 4 will reverse-engineer the five PromQL pattern shapes into a `PromqlLogicalPlan` tree so the L1→L2→L3 pipeline is uniform. +Every deployment model must produce an L2 tree, even when the source language didn't originally come as one. asap-planner-rs's current approach (PromQL pattern catalogue → `IntermediateAggConfig`) skips L2; Phase 4 will reverse-engineer the five PromQL pattern shapes into a `PromqlLogicalPlan` tree so the L1→L2→L3 pipeline is uniform. -A future scenario whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. +A future deployment model whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. -This drives the core/scenario split: +### System I/O contract — what the controller takes in, what it emits + +The controller is a **planner**, not an executor. It does not run queries; it decides where each piece of a query runs. The data plane (OTel collectors / ASAPQuery-backend / DataFusion `SessionContext`) runs them. + +**System input — what the controller takes in.** A `QueryWorkload` (one or more `QuerySpec`s) plus deployment context (available executors, their capabilities, current SLA targets, telemetry of recent violations). The `QuerySpec` carries the raw query string in its source language (PromQL / SQL / DataFusion / ElasticDSL) and the accuracy / latency / cost target. Workloads can arrive via four entry points (HTTP `POST /plan`, OpAMP capability-miss callback, YAML file for the CLI shell, query-log replay); they all normalise to `QueryWorkload` before L1. + +**Workload features that the controller cares about, beyond the queries themselves:** +- Execution model: **batch** (one-shot YAML, query-log replay) vs **streaming** (live pipeline that must keep producing results as data arrives). +- Data input source: time-series scrape, tabular relation, query log replay, etc. Drives `Source` variant choice in L3. +- Reuse opportunity: ≥2 queries planned together → `CostModel::workload_cost` credits shared sub-expressions. + +**System output — what the controller emits.** For each registered executor in the deployment, a sub-DAG of the optimized plan plus the configuration that lets that executor run it. Concretely: + +| Output | Consumer | Wire shape | +|---|---|---| +| Per-executor sub-DAG assignment | The executor (edge agent / gateway / backend / DataFusion session) | OpAMP `RemoteConfig` (OTel YAML) / `streaming_config.yaml` POST / rewritten `LogicalPlan` | +| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge / compute-from-raw over the precompute engine, …) | Implied by the per-executor configs; not a separate artifact | +| Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | + +**Premise behind the staged dispatch.** Two distinct concepts: + +- **Stage** (`StageId`) — a categorical *tier* in the data lifecycle (edge / gateway / backend / in-process). The topology declares which stages exist (3-stage / 1-stage / 0-stage). Stages are roles, not instances. +- **Executor** (`Executor`, defined in `core::physical::executor`) — a *concrete runtime instance* that occupies a stage. Carries `id`, `stage: StageId`, `capabilities`, and `address` (OpAMP agent / HTTP endpoint / in-process handle). One stage may have N executors — e.g. a 50-host edge fleet is 50 `Executor`s all with `stage = StageId("edge")`; a singleton backend is one `Executor` at `stage = StageId("backend")`. + +Every stage is in principle capable of running the entire query tree — all stages speak the same physical operators. The controller's job is to decide *which stage* runs *which sub-tree / sub-DAG* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by `StageId`, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring at stage granularity; the per-deployment-model `PhysicalPlanner` then materialises each stage's sub-DAG into one config per `Executor` at that stage (varying only by per-executor connection / identity details). The executor list comes from `DeploymentConstraints::executors()`. + +This is what makes the topology a *parameter* rather than an axis of code: the same `SketchExpr` plus a different `TopologyDescriptor` produces edge-only / 1-stage / 3-stage / 0-stage placements without rewriting the plan. + +**Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialize into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. + +This drives the core/deployment model split: - **`crates/core/`** owns all 5 layers of **shared infrastructure**: L1-3 end-to-end (parsers + lowering + intent-only IR), plus L4's rule engine driver + rule library + cost-model traits, plus L5's stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. -- **Each scenario is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any scenario-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. +- **Each deployment model is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any deployment-model-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. ## 4. Principles -### P1. Core owns shared infrastructure across all 5 layers; scenarios own choices +### P1. Core owns shared infrastructure across all 5 layers; deployment models own choices -See §3. Core is NOT just types + trait stubs — it ships working parsers + lowering passes (L1-3), a rule engine + rule library + cost-model traits (L4 framework), and a stage allocator + physical-plan framework + sketch catalogue (L5 framework). What scenarios plug in is **which rules fire** (picking from core's library + adding their own), **deployment topology** (how many stages; DC=3, query=1, fusion=0), and an **emitter** for the output format. A scenario that accepts all core's default L4 rules and uses `core::physical::single_stage_topology` is maybe 200 lines of code. +See §3. Core is NOT just types + trait stubs — it ships working parsers + lowering passes (L1-3), a rule engine + rule library + cost-model traits (L4 framework), and a stage allocator + physical-plan framework + sketch catalogue (L5 framework). What deployment models plug in is **which rules fire** (picking from core's library + adding their own), **deployment topology** (how many stages; DC=3, query=1, fusion=0), and an **emitter** for the output format. A deployment model that accepts all core's default L4 rules and uses `core::physical::single_stage_topology` is maybe 200 lines of code. ### P2. Core has no I/O -No HTTP, no OpAMP, no YAML, no Prometheus scrape, no `tokio::spawn`. Pure algorithms over in-memory data. This is what makes scenarios unit-testable without running the runtime. +No HTTP, no OpAMP, no YAML, no Prometheus scrape, no `tokio::spawn`. Pure algorithms over in-memory data. This is what makes deployment models unit-testable without running the runtime. -### P3. Runtime is a thin binary, scenarios are libraries +### P3. Runtime is a thin binary, deployment models are libraries -The `asap-controller` binary is assembled from: runtime (HTTP/OpAMP/replanner/store) + N scenario crates registered as plugins. Swapping scenarios is a build-time feature flag or a runtime registry entry. No scenario may reach directly into another. +The `asap-controller` binary is assembled from: runtime (HTTP/OpAMP/replanner/store) + N deployment model crates registered as plugins. Swapping deployment models is a build-time feature flag or a runtime registry entry. No deployment model may reach directly into another. ### P4. One input boundary, one output boundary **Input**: everything that enters the controller (HTTP `QuerySpec`, Prometheus query log replay, YAML workload, capability-miss callback) normalizes into a single `QueryWorkload` type in core — a collection of `QuerySpec`s each feeding L1→L2→L3. -**Output**: L5 emitters (OpAMP `RemoteConfig`, backend `StreamingConfig` POST, one-shot YAML file, rewritten DataFusion `LogicalPlan`) all implement a `PlanEmitter` trait. Scenarios supply emitter implementations; core doesn't know which emitters exist. +**Output**: L5 emitters (OpAMP `RemoteConfig`, backend `StreamingConfig` POST, one-shot YAML file, rewritten DataFusion `LogicalPlan`) all implement a `PlanEmitter` trait. Deployment models supply emitter implementations; core doesn't know which emitters exist. -This is the "extension point for future scenarios" — a new scenario adds an L4 rule set + an L5 emitter and registers them. +This is the "extension point for future deployment models" — a new deployment model adds an L4 rule set + an L5 emitter and registers them. ### P5. No feature-flag spaghetti @@ -125,7 +155,7 @@ ASAPController/ ├── docs/ │ ├── design.md # this file │ ├── migration-plan.md # the companion file -│ ├── scenarios/ # per-scenario design notes +│ ├── deployment-models/ # per-deployment-model design notes │ └── adr/ # architecture decision records ├── proto/ │ ├── asap_control.proto # Plan IR + QueryWorkload on the wire @@ -134,9 +164,10 @@ ASAPController/ │ ├── core/ # Shared infrastructure across all 5 layers; no I/O │ │ ├── query_language/ # L1: per-language parsers — promql/, sql/, datafusion/, elasticdsl/ │ │ ├── logical_plan/ # L2: per-language algebra tree (Aggregate/Window/Filter/…) -│ │ ├── sketch_algebra/ # L3: QueryExpr + AggIntent (~25 variants) + sketch directory +│ │ ├── intent_algebra/ # L3 IR: QueryExpr + AggIntent + Schema/HasSchema (intent only) +│ │ ├── sketch_algebra/ # L4 IR: SketchExpr (sketch-bound — kind + params committed) │ │ ├── lower/ # L1→L2→L3 lowering passes (one entry per language) -│ │ ├── optimizer/ # L4 framework: +│ │ ├── optimizer/ # L4 framework — produces SketchExpr from QueryExpr: │ │ │ ├── engine/ # rule driver — fixed-point iteration, cycle detection, priority │ │ │ ├── trait/ # OptimizerRule + RuleCategory (PushDown / Fusion / Elim / Bind) │ │ │ ├── rules/ # shared rule library (e.g. sketch-binding rules; stream-vs-batch picker) @@ -145,11 +176,12 @@ ASAPController/ │ │ │ ├── planner_trait/ # PhysicalPlanner trait │ │ │ ├── stage_allocator/ # generic topology-driven allocator │ │ │ ├── topology/ # Topology descriptor types (edge/gateway/backend, single, zero) +│ │ │ ├── executor/ # Executor type — concrete runtime instance occupying a stage │ │ │ └── sketch_catalog/ # candidate sketch types + parameter constraints -│ │ ├── pipeline/ # orchestrates L1→L2→L3→L4→L5, parameterized on scenario +│ │ ├── pipeline/ # orchestrates L1→L2→L3→L4→L5, parameterized on deployment model │ │ ├── workload/ # QueryWorkload wrapper around QuerySpecs -│ │ ├── emit/ # PlanEmitter trait — implemented by scenario L5 -│ │ ├── registry/ # ScenarioRegistry, ScenarioId +│ │ ├── emit/ # PlanEmitter trait — implemented by deployment model L5 +│ │ ├── registry/ # DeploymentModelRegistry, DeploymentModelId │ │ └── telemetry/ # tracing macros, metric names (no exporter) │ ├── runtime/ # service skeleton — HTTP, OpAMP, replanner │ │ ├── http/ # axum surface — /plan, /replan, /metrics, /status @@ -158,41 +190,41 @@ ASAPController/ │ │ ├── replan/ # Replanner — SLA + expiry triggers │ │ ├── store/ # PlanStore, WorkloadStore │ │ └── backend_client/ # HTTP client (pushes to ASAPQuery-backend, etc.) -│ ├── scenario-lifecycle/ # thin — picks rules, 3-stage topology, OTel/backend emitters +│ ├── deployment-model-asaplifecycle/ # thin — picks rules, 3-stage topology, OTel/backend emitters │ │ ├── rules.rs # L4 rule selection + DC-specific rules (stage-aware push-down) │ │ ├── topology.rs # 3-stage: edge / gateway / backend -│ │ ├── cost.rs # scenario-specific cost impls — delta / online / pareto / tco +│ │ ├── cost.rs # deployment-model-specific cost impls — delta / online / pareto / tco │ │ └── emit/ # OpAmpRemoteConfig + AsapqueryBackendConfig emitters -│ ├── scenario-query/ # thin — rules, 1-stage topology, YAML emitters + query-log input +│ ├── deployment-model-asapquery/ # thin — rules, 1-stage topology, YAML emitters + query-log input │ │ ├── rules.rs # L4 rule selection + sketch-binding rule (split from map_statistic_*) │ │ ├── topology.rs # 1-stage: backend-only │ │ ├── emit/ # StreamingConfig.yaml + InferenceConfig.yaml │ │ ├── query_log/ # extra L1 input: Prometheus query-log replay │ │ └── schema/ # PromQLSchema discovery from Prometheus (feeds L1) -│ ├── scenario-fusion/ # thin — DF-flavored rules, 0-stage, in-process emit +│ ├── deployment-model-asapfusion/ # thin — DF-flavored rules, 0-stage, in-process emit │ │ ├── rules.rs # L4 rules for DataFusion LogicalPlan (sketch-aware rewrites) │ │ ├── topology.rs # 0-stage: in-process │ │ ├── emit/ # rewritten DataFusion LogicalPlan │ │ ├── executor/ # DataFusion SessionContext wrapper (library-mode execution) │ │ └── sketch_support/ # asap_sketchlib-backed rewrites │ ├── control-proto/ # generated from proto/ (tonic/prost) -│ └── testing/ # test fixtures + harness shared across scenarios +│ └── testing/ # test fixtures + harness shared across deployment models └── bin/ - ├── asap-controller/ # long-running service with all scenarios - │ └── main.rs # axum + OpAMP + replanner + registered scenarios - ├── asap-query/ # one-shot CLI for scenario-query (what asap-planner-rs is today) + ├── asap-controller/ # long-running service with all deployment models + │ └── main.rs # axum + OpAMP + replanner + registered deployment models + ├── asap-query/ # one-shot CLI for deployment-model-asapquery (what asap-planner-rs is today) │ └── main.rs # clap — read workload YAML, emit two YAMLs - ├── asap-lifecycle/ # OPTIONAL: standalone service with only scenario-lifecycle + ├── asap-lifecycle/ # OPTIONAL: standalone service with only deployment-model-asaplifecycle │ └── main.rs # slimmer image — no DataFusion, no query YAML emitters - └── asap-fusion-bench/ # OPTIONAL: benchmark harness over scenario-fusion + └── asap-fusion-bench/ # OPTIONAL: benchmark harness over deployment-model-asapfusion └── main.rs # criterion entry; used by researchers ``` -**Per-scenario standalone binaries** are first-class. Each `bin//` is a thin shell that `use`s only the scenario crates it needs — so `bin/asap-lifecycle/` doesn't pull `datafusion` into its dep tree, and `bin/asap-fusion-bench/` doesn't pull `axum`/`opamp`. Feature flags on the workspace root let you `cargo build -p asap-lifecycle` and get a minimal binary. +**Per-deployment-model standalone binaries** are first-class. Each `bin//` is a thin shell that `use`s only the deployment model crates it needs — so `bin/asap-lifecycle/` doesn't pull `datafusion` into its dep tree, and `bin/asap-fusion-bench/` doesn't pull `axum`/`opamp`. Feature flags on the workspace root let you `cargo build -p asap-lifecycle` and get a minimal binary. -### Why three scenario crates today, not one monolithic `scenarios/` +### Why three deployment model crates today, not one monolithic `deployment-models/` -Each scenario has a different problem shape: +Each deployment model has a different problem shape: | | **lifecycle** | **query** | **fusion** | |---|---|---|---| @@ -202,7 +234,7 @@ Each scenario has a different problem shape: | Trigger | QuerySpec, SLA violation, expiry | one-shot CLI invocation | a DataFusion session constructing a query | | Cost model | accuracy × latency × $ staged | single-query accuracy/latency | operator selectivity / sketch feasibility | -Mashing them into one crate means the union of all their dependencies (sqlparser + promql-parser + DataFusion + OpAMP proto + Prometheus client) bleeds into every downstream user. Separating them means a user who only needs `scenario-fusion` (an offline query-optimization benchmark, say) can depend on it without pulling OpAMP. +Mashing them into one crate means the union of all their dependencies (sqlparser + promql-parser + DataFusion + OpAMP proto + Prometheus client) bleeds into every downstream user. Separating them means a user who only needs `deployment-model-asapfusion` (an offline query-optimization benchmark, say) can depend on it without pulling OpAMP. ## 6. Core crate details (layers 1–3 + driver) @@ -223,9 +255,22 @@ Each returns a language-flavored AST type. No sketch awareness. A **per-language** algebra tree — one `enum LogicalPlan` per language. Preserves language-specific semantics (PromQL instant vs range vector, SQL window frames, Elastic buckets) that would be lossy to collapse this early. Types are symmetric: `Aggregate { AggFunc }`, `Window`, `Filter`, `Sort`, `Limit`. No sketch names yet. -### `core::sketch_algebra` — Layer 3 +### `core::intent_algebra` — Layer 3 + +The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `HistogramQuantile`, no `PromQLSubquery` — those are L2 PromQL nodes), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP deployment models) via a `Source` sum type inside `QueryExpr::Scan`. + +#### Design rules for L3 + +1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). The exception is when an "intent" is its own physical primitive: heavy-hitter top-k is served by sketches (SpaceSaving, CMS-with-heap) as a single operation, so `AggIntent::TopK` is a first-class intent at L3 — distinct from the generic `Sort + Limit` operator pair, which still appears in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). +2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artifact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. +3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. +4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. +5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding context. The shape is a DAG, not a tree, because **a producer node can have multiple parents that share its precomputed intermediate state** — instead of recomputing the same sub-expression once per consumer, the consumers fan in to a single node and read its output. Three sources of fan-in: + 1. **Explicit, in-query.** SQL CTEs (`WITH name AS (expr) SELECT ... FROM name JOIN name AS n2 ON ...`) and PromQL recording rules name a sub-expression and reference it N times; each reference becomes a `QueryExpr::Ref(name)` parent of the named producer. + 2. **L4 reuse rules.** When two queries planned together both need (e.g.) a p99 quantile of the same series, the optimizer can introduce a shared sketch node whose output feeds both — even though neither query author wrote a CTE. + 3. **L4 stage / shard structure.** A pre-aggregate computed once on the edge can feed multiple downstream gateway-stage operators. -The language- and deployment-independent IR. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP scenarios) via a `Source` sum type inside `QueryExpr::Scan`. + `CostModel::workload_cost` credits a shared producer's build cost once across all consumers, which is what makes reuse a Pareto win. Tree IRs lose this — they have to duplicate the producer for every consumer. ```rust pub enum QueryExpr { @@ -237,149 +282,321 @@ pub enum QueryExpr { // ── Filtering & projection ──────────────────────────────────────────── /// σ — row-level filter (WHERE / PromQL label matchers). - Filter { child: Box, pred: Predicate }, + Filter { child: Box, pred: Predicate }, /// π — column projection (SELECT list). - Project { child: Box, cols: Vec }, - - // ── Aggregation ─────────────────────────────────────────────────────── - /// γ + α — GROUP BY + aggregates, with optional HAVING. Exact-agg path. - Aggregate { child: Box, by: Vec, aggs: Vec, - having: Option }, - /// γ + α specialised for a single sketch aggregation (one sketch per node). - /// Kept separate from `Aggregate` so the L4 allocator can bind a sketch - /// without re-parsing `AggFunc` variants. - SketchAgg { child: Box, intent: AggIntent, col: ColumnRef }, - /// Windowed sketch aggregation. Bundles window + intent because the window - /// defines the sketch lifecycle (flush / reset boundaries). - WindowedAgg { child: Box, intent: AggIntent, window: WindowSpec, - col: ColumnRef }, - - // ── Time / streaming ────────────────────────────────────────────────── - /// ψ — tumbling / sliding window (PromQL `[5m]`, SQL windowed aggregation). - Window { child: Box, duration: Duration, slide: Option }, - - // ── Distributed / multi-stage operators ────────────────────────────── + Project { child: Box, cols: Vec }, + + // ── Aggregation (logical, intent-only) ──────────────────────────────── + /// γ + α — GROUP BY + aggregate intents, with optional HAVING. + /// `aggs` carry `AggIntent`; concrete sketch / non-sketch operator + /// is chosen by L4 and lives in the L4-extended IR (`SketchExpr`). + Aggregate { child: Box, by: Vec, + aggs: Vec, having: Option }, + + // ── Time / streaming windows ────────────────────────────────────────── + /// ψ — tumbling / sliding / session window over the time axis. Defines + /// the lifecycle (flush / reset bounds) of any aggregate in its sub-tree / sub-DAG. + /// PromQL `[5m]` and streaming windows lower here. SQL `OVER (...)` + /// analytic frames are a different node — see `WindowFunc` below. + Window { child: Box, kind: WindowKind, + size: Duration, slide: Option }, + + // ── Distributed-execution structure ─────────────────────────────────── /// Partition the stream by key tuple (`GROUP BY` / PromQL `by (dims)`). Partition { child: Box, keys: PartitionKeys }, - /// δ — deduplicate on `col` before sketch ingestion. - Dedup { child: Box, col: String }, - /// τ — retain only top-K heavy hitters. - TopK { child: Box, k: u64, by: Vec }, - /// ⊕ — merge sketches from independent branches (distributed union). + /// δ — SQL `DISTINCT` / row deduplication on `cols`. + Distinct { child: Box, cols: Vec }, + /// ⊕ — union of sub-results from independent stages or shards (the + /// exact-merge case). Sketch unions are a separate node in `SketchExpr` + /// because they carry sketch-family / params type constraints. Merge { children: Vec }, - // ── Joins ───────────────────────────────────────────────────────────── - Join { kind: JoinKind, left: Box, right: Box, - pred: Option }, - /// Sketch-aware join push-down: pre-aggregate on inner side, then merge. - JoinSketch { outer: Box, inner: Box, join_key: String }, + // ── Joins (logical) ─────────────────────────────────────────────────── + /// Logical join. L4 picks the physical alternative — `HashJoin` / + /// `SortMergeJoin` / `SketchJoin` (e.g. KMV / theta-sketch / join-sample) + /// — based on selectivity, memory budget, and accuracy target. The + /// sketch-aware variant lives in `SketchExpr::SketchJoin`. + Join { kind: JoinKind, left: Box, right: Box, + pred: Option }, // ── Set operators ───────────────────────────────────────────────────── /// UNION / INTERSECT / EXCEPT, with or without ALL. - SetOp { kind: SetOpKind, all: bool, left: Box, right: Box }, + SetOp { kind: SetOpKind, all: bool, + left: Box, right: Box }, // ── Ordering & limiting ─────────────────────────────────────────────── + /// Generic order-by — survives L3 for non-heavy-hitter cases + /// (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`). Sort { child: Box, keys: Vec }, + /// `LIMIT n OFFSET k`. The heavy-hitter shape (`ORDER BY count DESC + /// LIMIT k`, PromQL `topk(k, …)`) is recognised at L1→L2→L3 lowering + /// and produces `AggIntent::TopK` rather than generic `Sort + Limit`, + /// so heavy-hitter sketches (SpaceSaving, CMS-with-heap) bind on the + /// intent. Generic `Sort + Limit` flows through unchanged. Limit { child: Box, n: u64, offset: u64 }, // ── Subquery / CTE ──────────────────────────────────────────────────── Subquery { child: Box, alias: String }, - /// SQL `WITH name AS (expr) IN body`, PromQL recording-rule binding. + /// SQL `WITH name AS (expr) IN body`; lowering target for PromQL + /// recording-rule bindings. LetBinding { name: String, expr: Box, body: Box }, // ── Analytic (OVER) window functions ────────────────────────────────── + /// SQL `OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)`. + /// Distinct from `Window` above — that is a streaming/tumbling window + /// over the time axis; this is an analytic frame over already-grouped rows. WindowFunc { child: Box, func: WindowFuncKind, partition_by: Vec, order_by: Vec, frame: Option }, - // ── PromQL-specific ─────────────────────────────────────────────────── - /// `histogram_quantile(φ, )` — HLL / histogram sketch → quantile. - HistogramQuantile { child: Box, phi: f64 }, - /// PromQL sub-query: `[range:resolution]`. - PromQLSubquery { child: Box, range: Duration, - resolution: Option }, - /// PromQL / SQL binary op between two relational sub-expressions - /// (arithmetic, comparison, `and`/`or`/`unless`). + // ── Binary composition ──────────────────────────────────────────────── + /// Arithmetic / comparison / boolean composition between two relational + /// sub-expressions (PromQL binary ops including `and`/`or`/`unless`, + /// SQL boolean composition). BinaryOp { op: BinaryOpKind, lhs: Box, rhs: Box, vector_match: Option }, } +pub enum WindowKind { Tumbling, Sliding, Session } + pub enum Source { - /// Time-series input — scenario-query / scenario-lifecycle shape. - /// `metric` identifies a metric family; `time` bounds the window; - /// `labels` constrains label-value combinations. - TimeSeries { - metric: MetricRef, - time: TimeRange, - labels: LabelFilter, - }, - /// Tabular input — scenario-fusion / future-OLAP shape. - /// `table_ref` identifies the table; `columns` projects the subset - /// in use. Join / subquery composition nests `Source` values. - Table { - table_ref: TableRef, - columns: Vec, - }, - Join { - left: Box, - right: Box, - on: JoinKey, - }, - // Future: WindowedStream, Subquery — added by scenarios that need them. + /// Time-series input — deployment-model-asapquery / deployment-model-asaplifecycle shape. + TimeSeries { metric: MetricRef, time: TimeRange, labels: LabelFilter }, + /// Tabular input — deployment-model-asapfusion / future-OLAP shape. + Table { table_ref: TableRef, columns: Vec }, + /// Join over Sources composes leaf shapes recursively. + Join { left: Box, right: Box, on: JoinKey }, + // Future: WindowedStream, Subquery — added by deployment models that need them. } -pub enum DataModel { - TimeSeries, - Tabular, - Any, -} +pub enum DataModel { TimeSeries, Tabular, Any } impl Source { - pub fn data_model(&self) -> DataModel { - match self { - Self::TimeSeries { .. } => DataModel::TimeSeries, - Self::Table { .. } | Self::Join { .. } => DataModel::Tabular, - } - } + pub fn data_model(&self) -> DataModel { /* … */ } + /// Output schema produced by this leaf (see "Schema flow"). + pub fn schema(&self, catalog: &SchemaCatalog) -> Schema { /* … */ } +} +``` + +#### What was removed during cleanup, and why + +| Removed | Reason | Replacement | +|---|---|---| +| `TopK { k, by }` *as a `QueryExpr` node* | A `QueryExpr`-level top-k operator collapses two distinct concepts: (a) the *intent* of "compute heavy hitters", which has its own sketch primitive, and (b) the generic operator pair `Sort + Limit`, which doesn't. Splitting them puts heavy-hitter logic at the right level | Heavy-hitter intent → `AggIntent::TopK` (L3, recognised by L1→L2→L3 lowering of `ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`); generic ordering+limit → `Sort + Limit` `QueryExpr` nodes (unchanged). Both retained — they describe different things | +| `WindowedAgg { intent, window }` | Equivalent to `Window` over `Aggregate`; redundant. Tumbling/sliding kind moves onto `Window::kind` | `Window { kind: …, … } → Aggregate { aggs: [intent] }` | +| `SketchAgg { intent, col }` | Sketch-bound; L3 must be intent-only | `Aggregate { aggs: [intent] }` at L3; L4 emits `SketchExpr::SketchAgg` | +| `JoinSketch { outer, inner, key }` | Sketch-bound physical alternative; L3 must be intent-only. Several papers describe sketch-of-join (KMV, theta, join-sample) — picking one is an L4 cost decision, not an L3 surface choice | `Join { … }` at L3; L4 emits `SketchExpr::SketchJoin` when a `Bind*OnJoin` rule fires | +| `HistogramQuantile { phi }` | PromQL artifact — consumes Prometheus's specific bucketed-histogram format. Language-specific | Lowers in PromQL L1→L2 to bucket reads + `Aggregate { aggs: [Quantile{q: phi, …}] }` over the bucket counts | +| `PromQLSubquery { range, resolution }` | Driver construct — asks the engine to evaluate the inner expression at N timestamps; not a single DAG node | Expanded by PromQL L1→L2 lowering into a set of independent `QueryExpr` instances, not preserved at L3 | +| `Dedup { col }` | Single-column-only spelling of SQL `DISTINCT` | Renamed to `Distinct { cols }`, generalised to N columns | + +#### Schema flow — every L3 edge carries a typed schema + +Every node has a derivable output schema given its input schemas and parameters. The DAG is type-checked: a `Filter` whose predicate references a column not in its child's output schema fails at plan time. + +```rust +pub struct Schema { + pub fields: Vec, + /// Index into `fields` for the time axis, if any. PromQL leaves carry one; + /// SQL leaves may or may not. + pub time_index: Option, + /// Optional metadata for reuse-aware planning. Each inner `Vec` is + /// a set of column indices that together uniquely identify rows; the outer + /// `Vec` allows multiple unique-key sets (e.g. primary key + another unique + /// constraint). + /// + /// **Populated by**: the per-node input/output spec (e.g. `Aggregate { by, .. }` + /// emits `unique_keys = [by]`; `Distinct { cols }` adds `cols`; `Project` + /// carries forward the retained columns; most other nodes pass through). + /// + /// **Consumed by**: `CostModel::workload_cost` only — the reuse-aware path + /// that credits shared sub-expressions across multiple queries. Single-query + /// plans, the `Bind*` rules, push-down rules, and L5 emitters do not read + /// this field. If the reuse path is deferred (see §13 future work), this + /// field is dead weight; it lives here so the metadata is available the + /// moment workload-aware planning lands without requiring an L3-wide + /// schema change. + pub unique_keys: Vec>, } +pub struct Field { + pub name: String, + pub dtype: DataType, // Int64 / Float64 / Utf8 / Map / … + pub nullable: bool, +} + +pub trait HasSchema { + fn input_schemas(&self) -> Vec<&Schema>; + fn output_schema(&self, inputs: &[&Schema], cat: &SchemaCatalog) -> Schema; +} +``` + +Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/intent_algebra/schema.rs`). Each row reads independently: every column position, type, and constraint is named explicitly rather than carried by a shorthand like `S` or `L` / `R`. + +| Node | Input schemas | Output schema | +|---|---|---| +| `Scan { source }` | none — leaf node | `source.schema(catalog)` — derived from the source's catalog metadata (TimeSeries metric labels + value + timestamp; or Table columns from `information_schema`; or recursive `Source::Join`) | +| `Ref(name)` | none — pointer node | the output schema of the `LetBinding` whose `name` matches; resolved at plan time | +| `Filter { pred }` | one input schema (the child's output) | the child's input schema unchanged — `Filter` is a row-level refinement; no columns added, removed, or re-typed | +| `Project { cols }` | one input schema | the input schema projected to `cols` — fields filtered and reordered to match `cols`; `time_index` and `unique_keys` carried over for retained columns | +| `Aggregate { by, aggs }` | one input schema | the `by` columns (carried verbatim from input) followed by one new column per entry in `aggs`, each named and typed by `AggIntent::output_type(input_field)`; `unique_keys = [by]` | +| `Window { kind, size, slide }` | one input schema, **must** contain a `time_index` field | the input schema extended with synthetic `window_id` and `window_start` / `window_end` metadata fields | +| `Partition { keys }` | one input schema | the input schema unchanged — logical-only marker; carries a sharding hint for L5's stage allocator | +| `Distinct { cols }` | one input schema | the input schema with `unique_keys` tightened to include `cols`; field types unchanged | +| `Merge` | N input schemas, all union-compatible (same field names, same types, same nullability, same `time_index` position if any) | the first input's schema (representative; checked union-compatible with the rest) | +| `Join { kind, pred }` | two input schemas — left and right children | the concatenation of left's fields and right's fields, minus columns that USING / NATURAL deduplicates; nullability widened on the OUTER side for outer joins | +| `SetOp { kind, all }` | two input schemas, must be union-compatible (as for `Merge`) | the left input's schema | +| `Sort { keys }` | one input schema; every field referenced in `keys` must be present in it | the input schema unchanged | +| `Limit { n, offset }` | one input schema | the input schema unchanged | +| `Subquery { alias }` | one input schema | the input schema with `alias` applied as the table alias to all field names | +| `LetBinding { name, expr, body }` | `expr` produces an intermediate schema bound to `name`; `body` consumes it (and any other in-scope bindings) | the `body`'s output schema | +| `WindowFunc { func, partition_by, order_by, frame }` | one input schema; every field in `partition_by` / `order_by` must be present | the input schema extended with one new column carrying the analytic-function output (named after `func`, typed per `func`) | +| `BinaryOp { op, vector_match }` | two input schemas — left and right operands. PromQL vector-match constraints (`on`/`ignoring` + `group_left`/`group_right`) govern label-set compatibility | for arithmetic/comparison `op`: the left input's schema with the value column re-typed to the result of `op`; for boolean `op` (`and`, `or`, `unless`): the left input's schema with a boolean value column | + +#### DAG schema, DB schema, sketch catalog — three distinct metadata sources + +These three are sometimes conflated and shouldn't be. Only the first two are *schemas* (descriptions of stream / table shape); the sketch catalog is a *registry* of available primitives, not a description of a stream: + +| Source | Where it lives | What it describes | Who reads it | +|---|---|---|---| +| **DAG schema** | On every edge of the L3 / L4 / L5 DAG (`Schema` above) | Columns + types flowing between operators | L4 rules (selectivity estimation, push-down legality), L5 emitter | +| **DB / source schema** | The query target (Prometheus TSDB metric metadata, SQL `information_schema`, DataFusion catalog) | What metrics / tables / columns exist in the data plane, with their types and indexing | `core::lower::*` to resolve names during L1→L2; exposed through a `SchemaCatalog` interface | +| **Sketch catalog** | `core::physical::sketch_catalog` (built at startup; static) | What sketches the runtime can build; what intents each one serves; mergeability, accuracy / confidence guarantees, supported aggregation keys, parameter ranges | L4 binding rules to choose a sketch for an `AggIntent`; L5 to instantiate the sketch | + +L1→L2 lowering reads the **DB schema** to resolve symbols. L3 onward, every edge carries a **DAG schema** that is type-checked locally. L4 binding rules consult the **sketch catalog** to map an intent to a concrete sketch under the deployment's constraints. They are three separate inputs to three distinct decisions. + +#### `AggIntent` — what to compute, not how + +```rust pub enum AggIntent { - // Data-model-agnostic — work on TimeSeries AND Tabular - Count { accuracy: AccuracyTarget }, + // Data-model-agnostic + Count { accuracy: AccuracyTarget }, Sum, Min, Max, - Quantile { q: f64, accuracy: AccuracyTarget }, - TopK { k: usize, accuracy: AccuracyTarget }, + Quantile { q: f64, accuracy: AccuracyTarget }, + /// Heavy-hitter top-k. Distinct from generic `Sort + Limit` because a + /// dedicated sketch primitive (SpaceSaving, CMS-with-heap, Misra-Gries) + /// computes it as a single operation. L1→L2→L3 lowering produces this + /// when it recognises a heavy-hitter shape (`ORDER BY count DESC LIMIT k`, + /// PromQL `topk(k, …)`); other ordering+limit cases stay as + /// `QueryExpr::Sort + QueryExpr::Limit`. + TopK { k: usize, by: Vec, accuracy: AccuracyTarget }, Cardinality { accuracy: AccuracyTarget }, - // Time-series only - Rate { window: Duration }, + // Time-series streaming derivatives — specific operations, not just + // "Sum / Count over a Window". `Rate` is the per-second average derivative + // computed with PromQL's counter-reset adjustment, not a generic windowed + // mean. Kept distinct because (a) they have counter-reset semantics that + // exact `Sum` does not, and (b) sketch backends specialised for + // derivatives (e.g. delta-set aggregator) bind on these intents directly. + Rate { window: Duration }, Increase { window: Duration }, - QuantileOverTime { q: f64, window: Duration, accuracy: AccuracyTarget }, - // Tabular / OLAP only — added as scenarios demand - // CorrelatedSubqueryCount { ... }, - // ApproxJoinCardinality { ... }, - // ~25 variants total + // Tabular / OLAP — added as deployment models demand + // CorrelatedSubqueryCount { … }, ApproxJoinCardinality { … }, } impl AggIntent { /// Which data-model this intent semantically requires. L4 rules /// consult this to skip non-applicable intents (e.g. `Rate` over /// a `Source::Table` is nonsense). - pub fn requires(&self) -> DataModel { - match self { - Self::Count{..} | Self::Sum | Self::Min | Self::Max - | Self::Quantile{..} | Self::TopK{..} | Self::Cardinality{..} - => DataModel::Any, - Self::Rate{..} | Self::Increase{..} | Self::QuantileOverTime{..} - => DataModel::TimeSeries, - } - } + pub fn requires(&self) -> DataModel { /* … */ } + /// Output column type — used by L3 schema derivation for `Aggregate`. + pub fn output_type(&self, input: &Field) -> DataType { /* … */ } + /// Which sketch families in the catalog can serve this intent. + /// Read by L4 binding rules. + pub fn candidate_sketches(&self) -> &'static [SketchKind] { /* … */ } +} +``` + +**Why `TopK` *is* an intent (and `Sort + Limit` is not collapsed into it).** Heavy-hitter top-k has a dedicated sketch primitive — SpaceSaving / CMS-with-heap / Misra-Gries compute it in a single pass with sub-linear memory. L4 binding rules want to fire on the *intent* "give me the top-k frequent items" rather than on a syntactic shape, the same argument that makes `Quantile` an intent rather than a `Sort` + "pick the φ-th element" pattern. So `AggIntent::TopK` lives at L3. Generic `QueryExpr::Sort + QueryExpr::Limit` *also* survives, because not every order-by-limit query is heavy-hitter (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`); these have no sketch alternative and stay as generic operators. The canonical-form invariant is preserved because L1→L2→L3 lowering picks one or the other deterministically based on whether it recognises a heavy-hitter pattern. + +**Why no `QuantileOverTime` intent?** It duplicated `Quantile` over a `Window`. The window — its kind, its size, its slide — is fully captured by the surrounding `Window { … }` node; the quantile *operation* is the same regardless of whether the input was a windowed time-series or a row-grouped table. PromQL's `quantile_over_time(0.99, m[5m])` lowers cleanly to `Window{size=5m} → Aggregate{aggs:[Quantile{q=0.99}]}`. One intent halves the L4 rule surface (one bind rule per operation, not per language-flavor of an operation). + +**Why `Rate` and `Increase` survive that argument.** They are not "Sum / Count over a Window with a different name" — they include PromQL's counter-reset adjustment, which is a non-trivial transformation an exact `Sum` does not perform. They earn distinct intent variants because they parameterise different physical operators (delta-set aggregators bind on these intents directly). If a non-PromQL streaming language has the same notion (e.g. SQL `RATE() OVER (RANGE)`), it lowers to the same intent — the intent vocabulary names the operation, not the language. + +### `core::sketch_algebra` — Layer 4 IR (`SketchExpr`) + +L4 binding rules consume L3 `QueryExpr` (in `core::intent_algebra`) and produce `SketchExpr` (in `core::sketch_algebra`). This is the IR L5 emitters consume. The two-IR split — intent-only L3 (`QueryExpr`) and sketch-bound L4 (`SketchExpr`), in two separate modules — gives L4 rule application a clean type signature: `fn apply(&QueryExpr, &Constraints) -> Option`, and the boundary cannot be silently violated. + +```rust +pub enum SketchExpr { + /// Any logical L3 node passes through unchanged when no L4 rule rewrote + /// it — a `Filter` doesn't need a sketch counterpart. + Logical(QueryExpr), + + /// Sketch aggregation. L4 picked the sketch type and parameters from the + /// catalog given the `AggIntent` and `DeploymentConstraints`. + SketchAgg { + child: Box, + sketch: SketchKind, // Kll, Cms, Hll, DDSketch, CmsWithHeap, … + params: SketchParams, // catalog-validated + col: ColumnRef, + by: Vec, + }, + + /// Sketch-aware join (KMV / theta-sketch for join cardinality; + /// join-sample for join sampling). Emitted only when a `Bind*OnJoin` + /// rule fires — L3 always presents the logical `Join` for L4 to choose. + SketchJoin { + outer: Box, + inner: Box, + key: ColumnRef, + sketch: SketchKind, + params: SketchParams, + }, + + /// Subtract one sketch from another. Valid only for sketches with a + /// linear-inverse property (CMS, theta, count-based). Lets the planner + /// compute "all-A minus all-B" cardinality / count without re-scanning. + SketchSubtract { left: Box, right: Box }, + + /// Delete a key from a sketch (CMS update with -1, deletable Bloom + /// filter, …). Valid only for deletion-supporting sketches. + SketchDelete { sketch_input: Box, key: ColumnRef }, + + /// Read out a query result from a built sketch. Inverse of `SketchAgg`. + /// `query` says what to extract — quantile φ, count for key k, cardinality. + SketchEstimate { sketch_input: Box, query: SketchQuery }, + + /// ⊕ — union of sketches across stages / shards. Distinct from L3 + /// `Merge` because sketch union has type constraints (same family, + /// same params). L5 stage allocator emits this when distributing. + SketchMerge { children: Vec }, +} +``` + +The optimizer's job is to selectively replace logical aggregates / joins with their sketch-bound variants when a binding rule fires; everything else stays inside `SketchExpr::Logical(…)`. + +#### Per-node input/output spec for `SketchExpr` + +L4 introduces a new field type into `Schema`: + +```rust +pub enum DataType { + // … the L3 types (Int64, Float64, Utf8, Map<…>, …) … + /// Sketch state. Carries the sketch family + params so the type system + /// rejects merges of incompatible sketches at plan time. + Sketch(SketchKind, SketchParams), } ``` -`AggIntent` describes **what** to compute (a quantile, a topk, a count) + what accuracy to hit — not **how** (KLL vs DDSketch vs CMS). Sketch binding happens at L4. +A "sketch-state schema" is a regular `Schema` whose value-bearing field has a `DataType::Sketch(...)` dtype. Reading rules: + +| Node | Input schemas | Output schema | +|---|---|---| +| `Logical(qe)` | whatever the inner L3 node `qe` consumes (per the L3 table above) | whatever `qe` produces — straight pass-through | +| `SketchAgg { child, sketch, params, col, by }` | one input schema; must contain `col` (the column being summarised) and every field referenced in `by` (group keys) | the `by` columns carried over verbatim, followed by one synthetic field of dtype `Sketch(sketch, params)` carrying the partial sketch state per group; `unique_keys = [by]` | +| `SketchJoin { outer, inner, key, sketch, params }` | two input schemas (outer + inner); both must contain `key` with compatible types | one field of dtype `Sketch(sketch, params)` carrying the join-cardinality / join-sample state — read out by a downstream `SketchEstimate` | +| `SketchSubtract { left, right }` | two input schemas, each with exactly one `Sketch(s, p)` field; **`s` and `p` must match** between the two inputs (catalog rejects mismatches at plan time); the `s` family must have `subtractable = true` in the catalog | one `Sketch(s, p)` field carrying the subtracted state (same family + params as inputs) | +| `SketchDelete { sketch_input, key }` | one input schema with a `Sketch(s, p)` field whose catalog entry has `deletable = true`; plus the `key` column to delete | the input schema unchanged in type — sketch state is mutated logically (the key's contribution removed) but the field's `(sketch, params)` signature is preserved | +| `SketchEstimate { sketch_input, query }` | one input schema with a `Sketch(s, p)` field; `query` (e.g. `Quantile(φ)`, `PointCount(k)`, `Cardinality`) must appear in the catalog entry's `supported_intents` for `s` | a regular row-shaped schema carrying the answer — `Float64` for quantile, `Int64` for count / cardinality, an array of `(key, count)` for top-k. The `Sketch(...)` field type does *not* propagate downstream of an Estimate | +| `SketchMerge { children }` | N input schemas, each with a `Sketch(s, p)` field; **all N must agree on `(s, p)`**; the `s` family must have `mergeable = true` in the catalog | one `Sketch(s, p)` field carrying the unioned state (same family + params as inputs) | + +Two type-system invariants make L4 robust: + +1. **Sketch-family mismatch is a plan-time error.** `SketchSubtract` over `Sketch(KLL, …)` and `Sketch(CMS, …)` fails type-checking before L5 ever sees it. +2. **Catalog capability flags gate which nodes can fire.** `SketchSubtract` requires `subtractable`, `SketchDelete` requires `deletable`, `SketchMerge` requires `mergeable`. The catalog (see §6 `core::physical::sketch_catalog`) is the single source of truth for these flags; binding rules consult it before producing the node. **Why a `Source` sum instead of two parallel `QueryExpr` trees:** most `QueryExpr` nodes (`Filter`, `Aggregate`) are data-model-agnostic — filter semantics are the same whether the input is a time-series window or a table scan. Only the leaf `Scan` differs. Keeping one tree with a polymorphic leaf means L4 rules like `BindKllOnQuantile` work uniformly across both data models; rules that care about data-model specifics (stage-aware push-down for TS; join-selectivity for tabular) gate on `source.data_model()` + `intent.requires()`. @@ -387,7 +604,7 @@ impl AggIntent { ### `core::lower` — L1 → L2 → L3 passes -One pass per language, each producing the same `sketch_algebra::QueryExpr`: +One pass per language, each producing the same `intent_algebra::QueryExpr`: ```rust pub fn lower_promql(ast: PromqlAst, schema: &MetricSchema) -> Result; @@ -396,35 +613,35 @@ pub fn lower_datafusion(ast: DfAst, ctx: &DfContext) -> Result; pub fn lower_elasticdsl(ast: EsAst, schema: &IndexSchema) -> Result; ``` -Once a query hits L3 it's language-agnostic. All scenarios downstream see the same IR. +Once a query hits L3 it's language-agnostic. All deployment models downstream see the same IR. ### `core::pipeline` — orchestration -The L1→…→L5 driver. Parameterised on a scenario's optimizer rules (L4) + emitter (L5): +The L1→…→L5 driver. Parameterised on a deployment model's optimizer rules (L4) + emitter (L5): ```rust -pub struct Pipeline { - scenario: S, +pub struct Pipeline { + deployment_model: S, } -impl Pipeline { +impl Pipeline { pub fn run(&self, workload: &QueryWorkload) -> Result { let l3: Vec = workload.queries() .iter() .map(|q| self.parse_and_lower(q)) // L1→L2→L3 .collect::>()?; - let l4 = self.scenario.optimizer().optimize(l3)?; // L4 - let l5 = self.scenario.physical().lower(l4)?; // L5 - self.scenario.emitter().emit(&l5) // L5 → bytes/protobuf/DF plan + let l4 = self.deployment_model.optimizer().optimize(l3)?; // L4 + let l5 = self.deployment_model.physical().lower(l4)?; // L5 + self.deployment_model.emitter().emit(&l5) // L5 → bytes/protobuf/DF plan } } ``` -Core owns the driver. Scenarios own what goes into each scenario-specific seam. +Core owns the driver. Deployment models own what goes into each deployment-model-specific seam. ### `core::optimizer` — Layer 4 framework -Core ships the **rule engine + trait surface + a shared rule library**. Scenarios pick which rules to enable. +Core ships the **rule engine + trait surface + a shared rule library**. Deployment models pick which rules to enable. ```rust // trait (in core) @@ -443,7 +660,7 @@ impl RuleEngine { -> Result, OptError>; } -// shared rule library (in core::optimizer::rules) — opt-in from scenarios +// shared rule library (in core::optimizer::rules) — opt-in from deployment models pub mod rules { pub struct BindKllOnQuantile; // AggIntent::Quantile → bind KLL(k by accuracy) pub struct BindCmsOnCount; // AggIntent::Count → bind CMS(w, d) @@ -455,10 +672,10 @@ pub mod rules { } ``` -Scenarios compose rule sets by picking from the shared library + adding their own: +Deployment models compose rule sets by picking from the shared library + adding their own: ```rust -// in scenario-lifecycle +// in deployment-model-asaplifecycle use asap_control_core::optimizer::{RuleEngine, rules::*}; fn rule_set() -> Vec> { vec![ @@ -472,7 +689,7 @@ fn rule_set() -> Vec> { } ``` -Core also ships `DeploymentConstraints` as a trait object; each scenario supplies a concrete impl with its deployment's memory budgets, network topology, available sketch backends. +Core also ships `DeploymentConstraints` as a trait object; each deployment model supplies a concrete impl with its deployment's memory budgets, network topology, available sketch backends, and the registered `Executor` list (one entry per concrete runtime instance, each tagged with its `StageId`). The executor list is populated from the deployment model's discovery channel — OpAMP for DC lifecycle, static config for single-backend query, the in-process `SessionContext` itself for fusion. ### `core::physical` — Layer 5 framework @@ -495,8 +712,34 @@ pub mod topology { pub struct ZeroStage; /* in-process */ } +/// Identifier for a stage role (categorical tier in the data lifecycle). +pub struct StageId(pub String); // "edge" / "gateway" / "backend" / "in-process" + +/// A **concrete runtime instance** that executes a sub-DAG of the plan. +/// One stage may have many executors — e.g. a 50-host edge fleet is 50 +/// executors all with `stage = StageId("edge")`; a singleton backend is +/// one executor at `stage = StageId("backend")`. The stage allocator +/// works at stage granularity; the deployment-model planner / emitter +/// materialises each stage's sub-DAG into one concrete config per executor. +pub struct Executor { + pub id: ExecutorId, // stable identifier across replans + pub stage: StageId, // which stage this executor occupies + pub capabilities: ExecutorCaps, // memory budget, available sketch backends, network neighbours + pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle +} + +pub struct ExecutorId(pub String); + +pub enum ExecutorAddr { + OpAmpAgent(AgentId), // OTel collector managed via OpAMP (DC lifecycle) + HttpEndpoint(Url), // ASAPQuery-backend, etc. + InProcess, // asap-fusion library-mode SessionContext +} + // generic stage allocator — given a QueryExpr tree + a topology, decide which -// ops land on which stage subject to constraints +// ops land on which stage subject to constraints. Stage-level only; per-executor +// fan-out happens in the deployment model's PhysicalPlanner using the executor +// list from `DeploymentConstraints::executors()`. pub struct StageAllocator; impl StageAllocator { pub fn allocate( @@ -504,22 +747,58 @@ impl StageAllocator { ) -> Result, PlanError>; } -// sketch catalogue — what sketches exist, what params they accept -pub struct SketchCatalog { /* built at startup; queried by L4 binding rules + L5 */ } +// sketch catalogue — what sketches exist, what they support, what params they +// accept. Built at startup from the registered sketch backends; queried by L4 +// binding rules to map an `AggIntent` → `SketchKind` + `SketchParams`, and by +// L5 to instantiate the chosen sketch. +pub struct SketchCatalog { + pub entries: Vec, +} + +pub struct SketchEntry { + pub kind: SketchKind, // Kll, Cms, Hll, DDSketch, KMV, Theta, … + /// Which `AggIntent` variants this sketch can serve. + pub supported_intents: &'static [IntentTag], // Quantile, Count, Cardinality, JoinCardinality, … + /// Mergeability — sketches built on disjoint inputs combine without re-scan. + /// (CMS / HLL / KLL / theta = mergeable; SpaceSaving = approximately; + /// some heavy-hitter variants = no.) + pub mergeable: Mergeability, + /// Whether the sketch supports point deletion (CMS update with -1, + /// deletable Bloom filter) — gates `SketchExpr::SketchDelete`. + pub deletable: bool, + /// Whether the sketch admits a linear inverse (CMS, theta, count-based) + /// — gates `SketchExpr::SketchSubtract`. + pub subtractable: bool, + /// Accuracy / confidence model: error bounds as a function of params. + /// `(eps, delta)` for randomised sketches; absolute error for KLL; etc. + pub accuracy: AccuracyModel, + /// Aggregation keys this sketch supports natively. Some sketches are + /// keyed (CMS over (label, value)); others are unkeyed (HLL). + pub aggregated_keys: KeyShape, // Unkeyed | KeyedScalar | KeyedTuple + /// Parameter ranges + defaults. Catalog rejects out-of-range params at + /// L4 bind time so L5 never sees an unsupported configuration. + pub param_ranges: ParamRanges, + /// Memory + CPU model used by `CostModel`. Function of params. + pub cost_model: SketchCostModel, +} ``` -Scenarios use these pieces: +Deployment models use these pieces: ```rust -// in scenario-lifecycle -use asap_control_core::physical::{StageAllocator, topology::ThreeStage}; +// in deployment-model-asaplifecycle +use asap_control_core::physical::{StageAllocator, topology::ThreeStage, Executor}; impl PhysicalPlanner for LifecyclePlanner { type Topology = ThreeStage; - type Output = (StagedExprs, SketchBindings); + type Output = Vec<(ExecutorId, ExecutorPlan)>; // one entry per executor fn lower(&self, l4: Vec, t: &ThreeStage) -> Result<_, _> { + // 1. Stage-level: colour the DAG by StageId. let assignments = StageAllocator.allocate(&l4, t, &self.constraints)?; - // scenario-specific post-processing: DC's delta_cost logic, + // 2. Per-executor fan-out: for each Executor in the deployment, + // pick the sub-DAG assigned to its stage and produce its config. + let executors: &[Executor] = self.constraints.executors(); + // deployment-model-specific post-processing: DC's delta_cost logic, // backend_client push preparation, etc. Ok(/* ... */) } @@ -529,7 +808,7 @@ impl PhysicalPlanner for LifecyclePlanner { ### `core::plan` — shared traits bridging layers ```rust -pub trait Scenario { +pub trait DeploymentModel { type Topology: TopologyDescriptor; type EmitterOutput; fn rules(&self) -> Vec>; @@ -557,11 +836,11 @@ pub enum QueryWorkload { pub struct QuerySpec { /* metric_name, aggregations, time_window, SLAs, ... */ } ``` -All three scenarios take `&QueryWorkload`. Same type, different planners. +All three deployment models take `&QueryWorkload`. Same type, different planners. ### `core::cost` -Extracted from DC's `planner/*` (delta, online, pareto, tco). Scenarios pick the models they need. +Extracted from DC's `planner/*` (delta, online, pareto, tco). Deployment models pick the models they need. ```rust pub trait CostModel { @@ -603,33 +882,221 @@ pub trait PlanEmitter { } ``` -Concrete emitters live in scenario crates: +Concrete emitters live in deployment model crates: -- `scenario-query::yaml::StreamingConfigEmitter` → `streaming_config.yaml` bytes -- `scenario-query::yaml::InferenceConfigEmitter` → `inference_config.yaml` bytes -- `scenario-lifecycle::emit::OpAmpRemoteConfigEmitter` → `opamp::RemoteConfig` protobuf -- `scenario-fusion::emit::DataFusionPlanEmitter` → rewritten `datafusion::LogicalPlan` +- `deployment-model-asapquery::yaml::StreamingConfigEmitter` → `streaming_config.yaml` bytes +- `deployment-model-asapquery::yaml::InferenceConfigEmitter` → `inference_config.yaml` bytes +- `deployment-model-asaplifecycle::emit::OpAmpRemoteConfigEmitter` → `opamp::RemoteConfig` protobuf +- `deployment-model-asapfusion::emit::DataFusionPlanEmitter` → rewritten `datafusion::LogicalPlan` ### `core::registry` The extension point. The runtime binary does: ```rust -let mut reg = ScenarioRegistry::new(); -reg.register::(); -reg.register::(); -reg.register::(); +let mut reg = DeploymentModelRegistry::new(); +reg.register::(); +reg.register::(); +reg.register::(); // add more... ``` -Registration is by-type; the runtime looks up by `ScenarioId` (either from the `QuerySpec.scenario` field or from an HTTP route). A 4th scenario is added by: +Registration is by-type; the runtime looks up by `DeploymentModelId` (either from the `QuerySpec.deployment_model` field or from an HTTP route). A 4th deployment model is added by: -1. New crate `scenario-/` -2. Implement `Scenario` trait (wraps a `Planner` + its `PlanEmitter`s) +1. New crate `deployment-model-/` +2. Implement `DeploymentModel` trait (wraps a `Planner` + its `PlanEmitter`s) 3. Register in `bin/asap-controller/main.rs` No core change. +### End-to-end example — one query through all five layers + +A worked trace of a single PromQL query as it flows L1 → L5. The query is intentionally simple (one metric, one window, one aggregate) so the IR shapes stay readable; production workloads have larger DAGs but the per-layer transformation is the same. + +**Input.** A `QuerySpec` arrives at `POST /plan` carrying: + +```text +quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m]) +``` + +with `accuracy: AccuracyTarget::Epsilon(0.01)` and `language: PromQL`. + +#### L1 — query language (parse to `PromqlAst`) + +`core::query_language::promql::parse` wraps `promql-parser`. Output (sketch — actual fields come from the upstream crate): + +```rust +PromqlAst::Call { + func: BuiltinFn::QuantileOverTime, + args: vec![ + Expr::Number(0.99), + Expr::MatrixSelector { + name: "http_request_duration_seconds", + matchers: vec![LabelMatcher::Eq("service", "api")], + range: Duration::from_secs(300), + }, + ], +} +``` + +Pure language-level parse — no schema lookup, no sketch awareness. Same call returns the same AST regardless of deployment model. + +#### L2 — language logical plan (`PromqlLogicalPlan` tree) + +`core::lower::promql::lower_to_logical` walks the AST against a `MetricSchema` resolved from Prometheus's `/api/v1/labels`. The result preserves PromQL semantics — `quantile_over_time` is a range-vector aggregate, distinct from a generic `Aggregate { Quantile }` over a `Window`: + +```rust +PromqlLogicalPlan::RangeAggregate { + func: PromqlRangeAggFunc::QuantileOverTime { q: 0.99 }, + range: Duration::from_secs(300), + matrix: Box::new(PromqlLogicalPlan::MatrixSelector { + metric: "http_request_duration_seconds", + matchers: vec![("service", LabelOp::Eq, "api")], + }), +} +``` + +Per language: the SQL form `SELECT approx_percentile(latency, 0.99) FROM events WHERE service='api' AND ts > now() - INTERVAL '5 min'` lands in `SqlLogicalPlan` with a different shape. L2 is per-language; the shapes converge at L3. + +#### L3 — intent algebra (`QueryExpr` + `AggIntent`) + +`core::lower::promql::lower_to_intent` strips PromQL-specific shapes. `quantile_over_time` does **not** survive — the canonical L3 form for a windowed quantile is `Window` over `Aggregate{Quantile}` (see §6 design rule 1, "no `WindowedAgg`"; see §6 "Why no `QuantileOverTime` intent?"). The accuracy target threads through from the `QuerySpec`: + +```rust +QueryExpr::Aggregate { + by: vec![], + aggs: vec![AggIntent::Quantile { + q: 0.99, + accuracy: AccuracyTarget::Epsilon(0.01), + }], + having: None, + child: Box::new(QueryExpr::Window { + kind: WindowKind::Sliding, + size: Duration::from_secs(300), + slide: None, + child: Box::new(QueryExpr::Scan { + source: Source::TimeSeries { + metric: MetricRef::from("http_request_duration_seconds"), + time: TimeRange::default(), // supplied by QuerySpec at evaluation + labels: LabelFilter::eq("service", "api"), + }, + predicates: vec![], + }), + }), +} +``` + +Per-edge schemas (derived per the §6 input/output spec): + +| Edge | Schema | +|---|---| +| `Scan` → `Window` | `{value: Float64, ts: Int64@time_index, service: Utf8}` | +| `Window` → `Aggregate` | the above + `{window_id: Int64, window_start: Int64, window_end: Int64}` | +| `Aggregate` → root | `{p99_value: Float64}` (one column per `AggIntent::Quantile`, typed via `output_type`) | + +This IR is identical across deployment models — same DAG, same intent, same accuracy target. Below this point each deployment model picks its own L4 rules. + +#### L4 — sketch algebra (`SketchExpr`) + +The shared rule `core::optimizer::rules::BindKllOnQuantile` matches `Aggregate { aggs: [Quantile{q, accuracy}] }`, consults the sketch catalog (KLL has `supported_intents: [Quantile]`, is mergeable, satisfies `ε=0.01` at `k=200`), and rewrites the matched sub-DAG into a `SketchAgg` wrapped in a `SketchEstimate` (the readout). Everything not rewritten passes through in `SketchExpr::Logical(...)`: + +```rust +SketchExpr::SketchEstimate { + sketch_input: Box::new(SketchExpr::SketchAgg { + child: Box::new(SketchExpr::Logical(/* the L3 Window+Scan subtree */)), + sketch: SketchKind::Kll, + params: SketchParams::Kll(KllParams { k: 200 }), + col: ColumnRef::value(), + by: vec![], + }), + query: SketchQuery::Quantile { q: 0.99 }, +} +``` + +Catalog-derived edge schemas: + +| Edge | Schema | +|---|---| +| inner `Logical(Window)` → `SketchAgg` | as L3 above (logical pass-through) | +| `SketchAgg` → `SketchEstimate` | `{kll: Sketch(Kll, KllParams{k:200})}` | +| `SketchEstimate` → root | `{p99_value: Float64}` (the `Sketch(...)` field type does not propagate past `SketchEstimate`) | + +The `Sketch(Kll, KllParams{k:200})` field type is the L4 type-system invariant — a downstream `SketchMerge` over a mismatched `Sketch(Cms, …)` input would fail at plan time, before L5 ever sees it. `BindKllOnQuantile` lives once in core and fires for all three deployment models; deployment-model-specific rules (lifecycle's `StageAwarePushDown`, fusion's `HashModeRule`) chain before or after. + +#### L5 — physical plan (stage allocation + emission, per deployment model) + +L4 chose the sketch family + params. L5 colors the DAG by `StageId` and emits per-executor configs. Same `SketchExpr` input; topology and emitter differ per deployment model. + +##### deployment-model-asaplifecycle (3-stage, OpAMP + backend POST) + +`StageAllocator` colors against `topology::ThreeStage`: + +| Node | StageId | Why | +|---|---|---| +| `Scan` + `Window` + `SketchAgg` | `"edge"` | scrape happens on the agent host; KLL build is mergeable so it's safe to run early | +| `SketchMerge` (inserted on cut edge) | `"gateway"` | catalog says `KLL.mergeable = true`; reduces N edge streams to 1 | +| `SketchEstimate` | `"backend"` | readout where users query | + +`PhysicalPlanner` then fans out via `DeploymentConstraints::executors()`. Given a deployment with 3 edge agents + 1 gateway + 1 backend: + +```rust +vec![ + (ExecutorId("edge-001"), OpAmpRemoteConfig { /* OTel YAML: scrape http_request_duration_seconds{service=api}, + sliding 5m window, KLL k=200, forward to gateway-001 */ }), + (ExecutorId("edge-002"), OpAmpRemoteConfig { /* identical OTel YAML; differs only by agent_id */ }), + (ExecutorId("edge-003"), OpAmpRemoteConfig { /* … */ }), + (ExecutorId("gateway-001"), OpAmpRemoteConfig { /* receive 3 KLL streams, SketchMerge, forward to backend */ }), + (ExecutorId("backend-001"), AsapqueryBackendConfig { /* read merged KLL, SketchEstimate q=0.99 */ }), +] +``` + +The 3 edge configs are byte-identical except for `agent_id` — that's the §3 "one stage may have N executors all materialised from the same per-stage sub-DAG" property. The `AsapqueryBackendConfig` is produced by calling `deployment-model-asapquery::yaml::StreamingConfigEmitter` (see §8 asymmetric dep). + +##### deployment-model-asapquery (1-stage, YAML) + +`StageAllocator` against `topology::SingleStage` returns everything on `"backend"`. `PhysicalPlanner` produces one config per backend executor (typically one): + +```yaml +# streaming_config.yaml — emitted by deployment-model-asapquery::yaml::StreamingConfigEmitter +operators: + - name: kll_p99_request_duration + kind: DatasketchesKLL + params: { k: 200 } + input: + metric: http_request_duration_seconds + label_filter: { service: api } + window: { kind: sliding, size: 5m } +estimates: + - sketch: kll_p99_request_duration + query: { kind: quantile, q: 0.99 } +``` + +This is the byte-for-byte format ASAPQuery-backend already consumes — `inference_config.yaml` is emitted alongside by `InferenceConfigEmitter` for the readout side. Both are wire invariants (see §10). + +##### deployment-model-asapfusion (0-stage, in-process `LogicalPlan`) + +`StageAllocator` against `topology::ZeroStage` returns the whole DAG in-process. `PhysicalPlanner` rewrites the caller's `datafusion::LogicalPlan`, replacing the original `Aggregate(quantile(0.99, ...))` node with an `Extension` wrapping the KLL sketch op: + +```rust +LogicalPlan::Extension(Extension { + node: Arc::new(SketchAggExt { + kind: SketchKind::Kll, + params: KllParams { k: 200 }, + input: /* original Window+Filter+Scan subtree, unchanged */, + estimate: SketchQuery::Quantile { q: 0.99 }, + }), +}) +``` + +Returned to the caller's `SessionContext` for execution by `asap-fusion`'s in-process operator. No wire format, no external executor; the data plane is the caller's process. Note also that `asap-fusion`'s entry point is `Source::Table` (not `Source::TimeSeries`) — this PromQL example is only illustrative for the fusion deployment model, which in practice consumes a pre-built DataFusion `LogicalPlan` and skips L1 (see §8). + +#### What this example demonstrates + +- **L1 → L3 are deployment-model-independent.** All three deployment models see the same `QueryExpr` for this query. +- **L4 binding is a shared rule.** `BindKllOnQuantile` lives once in `core::optimizer::rules` and fires for all three. +- **L5 is where deployment models diverge.** Same `SketchExpr` → three topologies → three emitter outputs. The topology is a parameter, not an axis of code (§3). +- **The sketch path is selected, not mandated.** If the `QuerySpec` had `accuracy: AccuracyTarget::Exact`, no `Bind*` rule would fire; L4 would pass `SketchExpr::Logical(QueryExpr::Aggregate{…})` through, and L5 would target an exact `HashAgg` (§1 goal 6). + ## 7. Runtime crate details Lifted from DC controller with zero semantic change: @@ -637,65 +1104,65 @@ Lifted from DC controller with zero semantic change: - `http/` — axum server on `:8080`. Routes: `POST /plan`, `POST /replan`, `GET /plans/:id`, `GET /status`, `GET /metrics` - `opamp/` — WebSocket OpAMP server on `:4320`. Same protocol DC speaks today. - `monitor/` — `Scraper` polls Prometheus, emits `Violation` -- `replan/` — subscribes to violations + expiry ticks, re-invokes the scenario registry +- `replan/` — subscribes to violations + expiry ticks, re-invokes the deployment model registry - `store/` — in-memory `PlanStore` + `WorkloadStore`. Pluggable backend later. -- `backend_client/` — pushes `StreamingConfig` YAML to ASAPQuery-backend's `/api/v1/streaming-config` endpoint. Factored out so new scenarios can push to other backends. +- `backend_client/` — pushes `StreamingConfig` YAML to ASAPQuery-backend's `/api/v1/streaming-config` endpoint. Factored out so new deployment models can push to other backends. -Runtime depends on `core` but NOT on any scenario crate directly. It talks to scenarios via the registry. +Runtime depends on `core` but NOT on any deployment model crate directly. It talks to deployment models via the registry. -## 8. Scenario crate details (each owns its L4 + L5) +## 8. Deployment model crate details (each owns its L4 + L5) -Each scenario crate is a library with: +Each deployment model crate is a library with: -- A `Scenario` impl that registers its planner, its emitters, and its HTTP routes (if any). +- A `DeploymentModel` impl that registers its planner, its emitters, and its HTTP routes (if any). - Its own config types — no shared config crate. - Its own cost model — maybe using `core::cost` primitives, maybe not. - Its own integration tests. -### `scenario-lifecycle` (thin) +### `deployment-model-asaplifecycle` (thin) - **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **L4 rules**: picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules that require stage awareness (`StageAwarePushDown`, `TransmissionCostRewrite`). Adding a stage-specific rule = a new file in `scenario-lifecycle/src/rules.rs`, impl `OptimizerRule`. Unchanged rules come from core. -- **L5 topology**: `core::physical::topology::ThreeStage` (edge → gateway → backend). No scenario-specific allocator logic — `StageAllocator` handles the tree walk; lifecycle only declares what the topology looks like. -- **Cost models**: `delta / online-EMA / Pareto / TCO` — these live in `scenario-lifecycle/src/cost.rs` because they're specific to the DC deployment's network/compute assumptions. Implement `core::optimizer::cost::CostModel`. -- **L5 emitters**: `OpAmpRemoteConfigEmitter` (per-role OTel YAML over OpAMP WebSocket) and `AsapqueryBackendConfigEmitter` (calls `scenario-query`'s `StreamingConfigEmitter` for the YAML bytes, then POSTs to backend). +- **L4 rules**: picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules that require stage awareness (`StageAwarePushDown`, `TransmissionCostRewrite`). Adding a stage-specific rule = a new file in `deployment-model-asaplifecycle/src/rules.rs`, impl `OptimizerRule`. Unchanged rules come from core. +- **L5 topology**: `core::physical::topology::ThreeStage` (edge → gateway → backend). No deployment-model-specific allocator logic — `StageAllocator` handles the tree walk; lifecycle only declares what the topology looks like. +- **Cost models**: `delta / online-EMA / Pareto / TCO` — these live in `deployment-model-asaplifecycle/src/cost.rs` because they're specific to the DC deployment's network/compute assumptions. Implement `core::optimizer::cost::CostModel`. +- **L5 emitters**: `OpAmpRemoteConfigEmitter` (per-role OTel YAML over OpAMP WebSocket) and `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery`'s `StreamingConfigEmitter` for the YAML bytes, then POSTs to backend). - **HTTP route**: `POST /plan` for full-lifecycle planning, `POST /replan` for SLA-triggered. - **Size estimate**: ~1500 LOC (was ~5000 pre-refactor). Cost models are the bulk; rule selection + topology + emitter are each a few hundred lines. -### `scenario-query` (thin) +### `deployment-model-asapquery` (thin) - **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. - **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. - **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. -- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::sketch_algebra::AggIntent` subset. -- **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this scenario covers most sketch types) + a scenario-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. +- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::intent_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). +- **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this deployment model covers most sketch types) + a deployment-model-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. - **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. -- **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `scenario-lifecycle` calls them when it needs to POST to ASAPQuery-backend. -- **Extra L1 inputs**: `query_log/` for Prometheus-query-log replay (unique to this scenario); `schema/` for `PromQLSchema` discovery from a live Prometheus URL. +- **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `deployment-model-asaplifecycle` calls them when it needs to POST to ASAPQuery-backend. +- **Extra L1 inputs**: `query_log/` for Prometheus-query-log replay (unique to this deployment model); `schema/` for `PromQLSchema` discovery from a live Prometheus URL. - **HTTP route**: `POST /plan/query` (JSON `QuerySpec` in, YAML stream out). Also backs the `bin/asap-query` one-shot CLI. - **Size estimate**: ~2500 LOC (was ~6000 pre-refactor — saved by picking from shared rule library; still pays the L2 tree + L3/L4 split refactor cost, which is one-time). -### `scenario-fusion` (thin) +### `deployment-model-asapfusion` (thin) - **Data model**: tabular. L2→L3 lowering produces `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when fusion extends to multi-table). Crucially **not time-indexed** — fusion works on arbitrary DataFusion relations; time is just another column if present. -- **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode scenario. -- **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so scenarios that want to take a DataFusion plan as input have a canonical name for it. -- **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::sketch_algebra::AggIntent` subset. -- **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + scenario-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. +- **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode deployment model. +- **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so deployment models that want to take a DataFusion plan as input have a canonical name for it. +- **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::intent_algebra::AggIntent` subset. +- **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + deployment-model-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. - **L5 topology**: `core::physical::topology::ZeroStage` (in-process). -- **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this scenario is library-mode. -- **Executor**: `ASAPExecutor` wraps a DataFusion `SessionContext`. Users construct `scenario-fusion` in-process. +- **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this deployment model is library-mode. +- **Executor**: `ASAPExecutor` wraps a DataFusion `SessionContext`. Users construct `deployment-model-asapfusion` in-process. - **HTTP route**: none by default. -- **Conformance cost**: near zero. Fusion's `SketchConfigRule` logic is replaced with picks from `core::optimizer::rules::*`; its `HashModeRule` stays scenario-specific. +- **Conformance cost**: near zero. Fusion's `SketchConfigRule` logic is replaced with picks from `core::optimizer::rules::*`; its `HashModeRule` stays deployment-model-specific. - **Size estimate**: ~1800 LOC (was ~3000 pre-refactor; the `SketchConfigRule` code folds into core's shared rule library). -The sketch microbenchmarks (KLL/CMS) move with the crate and keep running. The TODO items from `asap-fusion/TODO.md` (batch/multi-query execution, time semantics, distributed model) remain open but are now filed against `scenario-fusion/TODO.md` in-repo. +The sketch microbenchmarks (KLL/CMS) move with the crate and keep running. The TODO items from `asap-fusion/TODO.md` (batch/multi-query execution, time semantics, distributed model) remain open but are now filed against `deployment-model-asapfusion/TODO.md` in-repo. ### Data plane communication Control plane (ASAPController) and data plane (OTel agents / ASAPQuery-backend / DataFusion runtimes) always talk **over wire**, never via in-process calls. This is unchanged from today: -| Scenario | Data plane lives in | Wire protocol | +| Deployment model | Data plane lives in | Wire protocol | |---|---|---| | lifecycle | DataCollector (OTel collectors, agent + backend roles) | OpAMP WebSocket (config push) + HTTP POST (`StreamingConfig` → ASAPQuery-backend) + Prometheus scrape (metrics in) | | query | ASAPQuery-backend (query engine, SimpleMapStore) | HTTP POST `/api/v1/streaming-config` + `/api/v1/plan` (capability-miss callback in) + YAML file on disk (init-container mode) | @@ -703,34 +1170,34 @@ Control plane (ASAPController) and data plane (OTel agents / ASAPQuery-backend / **Data plane code stays in its original repo.** ASAPController only owns the control plane. The merger doesn't move OTel collectors out of DataCollector, doesn't move the query engine out of ASAPQuery-backend, and doesn't move DataFusion out of asap-fusion's users. Each data plane keeps its own release cadence. -### Scenario placement: in-repo or out-of-repo +### Deployment model placement: in-repo or out-of-repo -Because core owns the L4/L5 infrastructure (not just L1-3), a scenario crate is small and largely self-contained. That means scenarios can live either: +Because core owns the L4/L5 infrastructure (not just L1-3), a deployment model crate is small and largely self-contained. That means deployment models can live either: -- **Inside ASAPController workspace** — `crates/scenario-/`, lockstep release with core, cross-scenario changes are one PR. +- **Inside ASAPController workspace** — `crates/deployment-model-/`, lockstep release with core, cross-deployment-model changes are one PR. - **In their own downstream repo** — declares `asap-control-core` as a git-tagged dep, releases independently, owns its own CI. -Both produce functionally identical artefacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. +Both produce functionally identical artifacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. Default recommendation: -- **scenario-lifecycle** and **scenario-query** in ASAPController (they share `scenario-query`'s YAML emitter via a workspace `path = "../scenario-query"` dep — trivial in-workspace). -- **scenario-fusion** out-of-tree in `asap-fusion` repo (research project with independent benchmark cadence; depends on `asap-control-core` + `asap-control-optimizer` as published git tags). +- **deployment-model-asaplifecycle** and **deployment-model-asapquery** in ASAPController (they share `deployment-model-asapquery`'s YAML emitter via a workspace `path = "../deployment-model-asapquery"` dep — trivial in-workspace). +- **deployment-model-asapfusion** out-of-tree in `asap-fusion` repo (research project with independent benchmark cadence; depends on `asap-control-core` + `asap-control-optimizer` as published git tags). -Future scenarios choose whichever placement fits the team that owns them. +Future deployment models choose whichever placement fits the team that owns them. ### Asymmetric dependency -`scenario-lifecycle` depends on `scenario-query` because the `StreamingConfig` YAML emitter is authoritative there. When both live in ASAPController workspace this is a trivial `path =` dep. If one is ever moved out-of-tree, we lift the emitter into core to avoid a cross-repo Cargo dep. +`deployment-model-asaplifecycle` depends on `deployment-model-asapquery` because the `StreamingConfig` YAML emitter is authoritative there. When both live in ASAPController workspace this is a trivial `path =` dep. If one is ever moved out-of-tree, we lift the emitter into core to avoid a cross-repo Cargo dep. -## 9. Extension point — a 4th scenario +## 9. Extension point — a 4th deployment model -With the L4/L5 framework in core, adding a new scenario is mostly picking + a bit of glue. A hypothetical "edge caching" scenario that decides which queries to cache at the edge vs. backend would land as: +With the L4/L5 framework in core, adding a new deployment model is mostly picking + a bit of glue. A hypothetical "edge caching" deployment model that decides which queries to cache at the edge vs. backend would land as: ``` -crates/scenario-edge-cache/ # or your own repo +crates/deployment-model-edge-cache/ # or your own repo ├── Cargo.toml # depends on asap-control-core ├── src/ -│ ├── lib.rs # impl Scenario for EdgeCacheScenario +│ ├── lib.rs # impl DeploymentModel for EdgeCacheDeploymentModel │ ├── rules.rs # pick core rules + add EdgeCacheBindRule │ ├── topology.rs # TwoStage { edge, backend } TopologyDescriptor │ ├── cost.rs # impl CostModel — hit ratio × bandwidth @@ -742,21 +1209,21 @@ crates/scenario-edge-cache/ # or your own repo Total touch outside the new crate: - 1 line in `bin/asap-controller/main.rs` to register (if in-workspace) OR a published binary in your own repo - 1 line in workspace `Cargo.toml` (if in-workspace) -- optional: add `ScenarioId::EdgeCache` to `core::registry` +- optional: add `DeploymentModelId::EdgeCache` to `core::registry` -Typical size: ~500-2000 LOC depending on how scenario-specific the rules / cost model / emitter are. If the scenario accepts core's defaults everywhere, ~300 LOC is realistic. +Typical size: ~500-2000 LOC depending on how deployment-model-specific the rules / cost model / emitter are. If the deployment model accepts core's defaults everywhere, ~300 LOC is realistic. -### Standalone binary for one scenario +### Standalone binary for one deployment model -If you want a slim binary that runs only one scenario: +If you want a slim binary that runs only one deployment model: ``` bin/asap-edge-cache/ -├── Cargo.toml # depends only on runtime + scenario-edge-cache, not other scenarios -└── src/main.rs # registers only EdgeCacheScenario; no DataFusion, no query YAML +├── Cargo.toml # depends only on runtime + deployment-model-edge-cache, not other deployment models +└── src/main.rs # registers only EdgeCacheDeploymentModel; no DataFusion, no query YAML ``` -Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql-parser` unless this scenario needs it. Useful when a deployment is dedicated to one scenario (e.g. an edge-caching-only service). +Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql-parser` unless this deployment model needs it. Useful when a deployment is dedicated to one deployment model (e.g. an edge-caching-only service). ## 10. Wire protocols — what changes, what doesn't @@ -767,9 +1234,9 @@ Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql- - Prometheus scrape format **Internal** (controller's own surface, still HTTP+JSON for now): -- `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artefacts (URIs). `/api/v1/plan` proxies to this. +- `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artifacts (URIs). `/api/v1/plan` proxies to this. - `GET /plans/:id` — plan inspection -- `GET /status` — runtime + scenario health +- `GET /status` — runtime + deployment model health **New** (internal-only, proto): - `proto/asap_control.proto` defines `Plan`, `PlanNode`, `Expr` for persistence + store-internal serialization. Not on the wire between services. Optional; initial migration skips this and persists via `serde_json`. @@ -784,9 +1251,9 @@ resolver = "2" members = [ "crates/core", "crates/runtime", - "crates/scenario-lifecycle", - "crates/scenario-query", - "crates/scenario-fusion", + "crates/deployment-model-asaplifecycle", + "crates/deployment-model-asapquery", + "crates/deployment-model-asapfusion", "crates/control-proto", "crates/testing", "bin/asap-controller", @@ -803,40 +1270,54 @@ serde = { version = "1", features = ["derive"] } serde_yaml = "0.9" tracing = "0.1" thiserror = "1" -# scenario-specific deps kept in scenario crates +# deployment-model-specific deps kept in deployment model crates ``` Core depends only on `serde`, `tracing`, `thiserror`, and small utility crates. No HTTP, no DataFusion, no OpAMP. Runtime depends on core + `axum` + `reqwest` + `tokio-tungstenite` + `prost` (OpAMP proto). -Scenarios depend on core. Scenario-query also pulls in `promql-parser`, `sqlparser`. Scenario-fusion pulls in `datafusion` + `arrow`. Scenario-lifecycle pulls in `sqlparser` + `promql-parser` + the cost-model math crates. +Deployment models depend on core. deployment-model-asapquery also pulls in `promql-parser`, `sqlparser`. deployment-model-asapfusion pulls in `datafusion` + `arrow`. deployment-model-asaplifecycle pulls in `sqlparser` + `promql-parser` + the cost-model math crates. -This matters: a user who only wants `scenario-fusion` (e.g., an offline benchmark) gets DataFusion but NOT axum/OpAMP. +This matters: a user who only wants `deployment-model-asapfusion` (e.g., an offline benchmark) gets DataFusion but NOT axum/OpAMP. ## 12. Open questions -1. **Do we need a cross-scenario cost model?** Today DC's lifecycle planner and asap-planner-rs's query planner have overlapping but not identical cost models. Answer for now: keep them separate in scenario crates; let them re-converge organically. If a third scenario needs the same model, lift at that point. +1. **Do we need a cross-deployment-model cost model?** Today DC's lifecycle planner and asap-planner-rs's query planner have overlapping but not identical cost models. Answer for now: keep them separate in deployment model crates; let them re-converge organically. If a third deployment model needs the same model, lift at that point. -2. **Where does the backend's `ControllerClient.create_plan` call land?** Initially: HTTP `POST /api/v1/plan` on the controller, handled by `scenario-lifecycle` (same as today). Long-term: could route by `QuerySpec.scenario` to a different planner, but that's a follow-up. +2. **Where does the backend's `ControllerClient.create_plan` call land?** Initially: HTTP `POST /api/v1/plan` on the controller, handled by `deployment-model-asaplifecycle` (same as today). Long-term: could route by `QuerySpec.deployment_model` to a different planner, but that's a follow-up. -3. **Which scenario owns `StreamingConfig` YAML emission?** Both `scenario-lifecycle` and `scenario-query` emit it today (DC's `config/generate_streaming_config_yaml` and `output/generator.rs` in `asap-planner-rs`). Plan: **one emitter in `scenario-query`**, called from both. This is why `scenario-lifecycle` depends on `scenario-query` in the dependency graph (asymmetric — query does not depend on lifecycle). +3. **Which deployment model owns `StreamingConfig` YAML emission?** Both `deployment-model-asaplifecycle` and `deployment-model-asapquery` emit it today (DC's `config/generate_streaming_config_yaml` and `output/generator.rs` in `asap-planner-rs`). Plan: **one emitter in `deployment-model-asapquery`**, called from both. This is why `deployment-model-asaplifecycle` depends on `deployment-model-asapquery` in the dependency graph (asymmetric — query does not depend on lifecycle). -4. **Do we vendor DataFusion's IR into core?** No. `scenario-fusion` owns its DataFusion-flavored plan; `core::plan::Plan` stays an enum with a variant that wraps a `datafusion::LogicalPlan` behind a feature flag. Core itself never reaches into DataFusion types. +4. **Do we vendor DataFusion's IR into core?** No. `deployment-model-asapfusion` owns its DataFusion-flavored plan; `core::plan::Plan` stays an enum with a variant that wraps a `datafusion::LogicalPlan` behind a feature flag. Core itself never reaches into DataFusion types. 5. **OpAMP proto: vendored, or from crates.io?** Today DC vendors. Recommend: keep vendored in `proto/opamp.proto`, generate via `prost-build` in `crates/control-proto`. Same thing DC does today, just moved. -6. **What happens to ASAPQuery-backend's `asap-planner-rs` directory post-migration?** Deleted. ASAPQuery-backend's docker-compose drops the `asap-planner-rs` init container; the controller's `scenario-query` now runs in-process (service mode) or as the `asap-query` CLI (one-shot mode). The ASAPQuery-backend repo shrinks by two directories. +6. **What happens to ASAPQuery-backend's `asap-planner-rs` directory post-migration?** Deleted. ASAPQuery-backend's docker-compose drops the `asap-planner-rs` init container; the controller's `deployment-model-asapquery` now runs in-process (service mode) or as the `asap-query` CLI (one-shot mode). The ASAPQuery-backend repo shrinks by two directories. + +7. **Versioning?** Start at `0.1.0` on the workspace. Deployment models can rev independently later via per-crate versions, but initially lockstep. -7. **Versioning?** Start at `0.1.0` on the workspace. Scenarios can rev independently later via per-crate versions, but initially lockstep. +8. **What if a future deployment model can't fit a tree L2?** L2 is currently a per-language tree, mandatory for every deployment model. asap-planner-rs's Phase 4 conformance cost (reverse-engineering PromQL pattern templates into a `PromqlLogicalPlan` tree) was taken deliberately to keep the architecture uniform. If a future deployment model's source language doesn't map naturally onto a tree (e.g. a constraint-based or dataflow-graph query language), that's the moment to re-examine the L2 contract — the `core::logical_plan` module is a single Rust trait + per-language types, not a deep assumption baked across the codebase. Until then, L2 = tree. -8. **What if a future scenario can't fit a tree L2?** L2 is currently a per-language tree, mandatory for every scenario. asap-planner-rs's Phase 4 conformance cost (reverse-engineering PromQL pattern templates into a `PromqlLogicalPlan` tree) was taken deliberately to keep the architecture uniform. If a future scenario's source language doesn't map naturally onto a tree (e.g. a constraint-based or dataflow-graph query language), that's the moment to re-examine the L2 contract — the `core::logical_plan` module is a single Rust trait + per-language types, not a deep assumption baked across the codebase. Until then, L2 = tree. +9. **Deployment model placement — in ASAPController workspace, or in own repo?** Both supported, same architecture either way (see §8 "Deployment model placement"). Default: `deployment-model-asaplifecycle` and `deployment-model-asapquery` in ASAPController (easy cross-deployment-model changes, shared YAML emitter); `deployment-model-asapfusion` in the `asap-fusion` repo (research cadence). A new deployment model picks based on team ownership and release cadence preferences. -9. **Scenario placement — in ASAPController workspace, or in own repo?** Both supported, same architecture either way (see §8 "Scenario placement"). Default: `scenario-lifecycle` and `scenario-query` in ASAPController (easy cross-scenario changes, shared YAML emitter); `scenario-fusion` in the `asap-fusion` repo (research cadence). A new scenario picks based on team ownership and release cadence preferences. +10. **How far do shared L4/L5 rules live in core before they become deployment-model-specific?** The rule of thumb: if ≥2 current deployment models would use it, it lives in `core::optimizer::rules::*`. If only 1 deployment model uses it *and* it depends on deployment-model-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the deployment model crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `deployment-model-asaplifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the deployment model; lift to core once a second deployment model wants it. Don't pre-emptively generalize. -10. **How far do shared L4/L5 rules live in core before they become scenario-specific?** The rule of thumb: if ≥2 current scenarios would use it, it lives in `core::optimizer::rules::*`. If only 1 scenario uses it *and* it depends on scenario-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the scenario crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `scenario-lifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the scenario; lift to core once a second scenario wants it. Don't pre-emptively generalize. +11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP deployment models)?** Already handled — see §3 "Data-model support" and §6 `core::intent_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP deployment model picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. -11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP scenarios)?** Already handled — see §3 "Data-model support" and §6 `core::sketch_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP scenario picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. +### Resolved during the L3 IR cleanup (see §6 `core::intent_algebra`) + +The following questions came up during review of the previous `QueryExpr` draft and are now settled. Listed here so the trail is visible: + +- **Output of `SketchAgg` vs `WindowedAgg`?** Neither node exists at L3 anymore. `SketchAgg` is an L4 sketch-bound node (`SketchExpr::SketchAgg`) emitted by binding rules; `WindowedAgg` was redundant with `Window` over `Aggregate` and removed. Output of `SketchAgg` is a sketch-typed column carrying the partial state; `SketchEstimate` reads it out into a scalar / vector. +- **Is `TopK` different from `Sort + Limit`?** They overlap on heavy-hitter queries but are different concepts. `TopK` is now an `AggIntent` at L3 (heavy-hitter intent — SpaceSaving / CMS-with-heap / Misra-Gries serve it as a single primitive), while generic `Sort + Limit` survives as `QueryExpr` operators for non-heavy-hitter cases (`ORDER BY name LIMIT 10`). L1→L2→L3 lowering picks the intent form when it recognises a heavy-hitter pattern (`ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`) and the operator form otherwise. +- **What is `JoinSketch`?** A sketch-aware join (KMV / theta / join-sample). Moved to L4 (`SketchExpr::SketchJoin`) so the choice "exact join vs sketch join" is an L4 cost decision against `Join`, not a competing L3 surface. +- **`HistogramQuantile` and `PromQLSubquery` feel out of place** — they were. Both removed from L3. `histogram_quantile` lowers in PromQL L1→L2 to bucket reads + a `Quantile` intent. `[range:resolution]` is a driver that expands into multiple queries during PromQL L1→L2 lowering, not a DAG node. +- **Sketch subtract / delete / estimate operators?** Added: `SketchExpr::SketchSubtract`, `SketchDelete`, `SketchEstimate`. Catalog flags (`subtractable`, `deletable`) gate which sketch families admit them. +- **What's `Dedup` exactly?** SQL `DISTINCT`. Renamed to `Distinct { cols }` and generalised from one column to N. +- **Why differentiate `Quantile` and `QuantileOverTime`?** No reason — the surrounding `Window` already encodes the temporal axis. `QuantileOverTime` removed from `AggIntent`. +- **`WindowedAgg` vs `Window` + `Agg`?** Equivalent; `WindowedAgg` removed. `WindowKind::{Tumbling, Sliding, Session}` lives on the `Window` node so the streaming-window kind is explicit; SQL analytic `OVER (...)` stays on the separate `WindowFunc` node. +- **Need input/output specs more fine-grained than `QueryExpr`?** Done: every L3 edge carries a typed `Schema` (fields + time index + unique-key sets), and §6 lists per-node input/output schemas. ## 13. Future work @@ -852,13 +1333,53 @@ The L4 rule engine + `OptimizerRule` trait are **primitive-agnostic** — nothin The guiding principle: **same framework, different rule.** A new primitive class is never a translation-layer change — only a new rule, a new cost-model entry, and (if it introduces a new runtime op) a new physical operator in L5. -## 14. Success criteria +## 14. Glossary + +Terms that are project-specific or that get conflated. Where the term has a Rust counterpart, the type is shown in backticks. + +### Architecture roles + +- **Deployment model** — A concrete bundle of (L4 rule choices) + (L5 topology) + (emitter), packaged as a `deployment-model-*` crate. Three exist today: `asaplifecycle`, `asapquery`, `asapfusion`. +- **Stage** (`StageId`) — A *categorical tier* in the data lifecycle: edge / gateway / backend / in-process. Roles, not instances. Declared by the deployment model's `TopologyDescriptor`. +- **Executor** (`Executor`) — A *concrete runtime instance* occupying a stage: a specific OTel agent, the ASAPQuery-backend process, a DataFusion `SessionContext`. One stage may have N executors (e.g. 50 edge agents). Carries `id`, `stage: StageId`, `capabilities`, `address`. +- **Topology** (`TopologyDescriptor`) — Declares which stages exist + how they connect (3-stage / 1-stage / 0-stage). Categorical, not instance-level. +- **DeploymentConstraints** — Trait object owned by each deployment model. Carries memory budgets, network topology, available sketch backends, and the registered `Executor` list. Threaded through L4 rules and the stage allocator. + +### Layer drivers + +- **RuleEngine** — Generic L4 driver in core (fixed-point iteration + cycle detection + priority ordering). One implementation; same code for every deployment model. +- **OptimizerRule** — Trait for an L4 rewrite. Categories: `PushDown | Fusion | Elim | Bind | StageRouting`. Deployment models pick which rules from `core::optimizer::rules` + add their own. +- **CostModel** — Trait scoring a plan's accuracy / latency / dollars. Generic implementations in core; deployment-model-specific cost models (DC's delta / online / pareto / TCO) live in their crates. +- **StageAllocator** — Generic L5 algorithm in core. Colors the L4-bound DAG by `StageId`. **Stage-granularity only — no executor knowledge.** One implementation; same code for every deployment model. +- **PhysicalPlanner** — Per-deployment-model L5 *driver* (one impl per deployment model). Calls `StageAllocator`, then fans the per-stage sub-DAGs out to executors via `DeploymentConstraints::executors()`, then produces the deployment-specific output type. Compare to allocator: planner = whole L5 driver, allocator = the stage-coloring step it uses. +- **PlanEmitter** — Per-deployment-model trait that serialises the planner's output to its wire format (OpAMP `RemoteConfig` / `streaming_config.yaml` / rewritten DataFusion `LogicalPlan`). + +### IR by layer + +- **L1 — query language** — Raw query string + parser (PromQL / SQL / DataFusion / ElasticDSL). +- **L2 — logical plan** — Per-language relational algebra tree (`PromqlLogicalPlan`, `SqlLogicalPlan`, etc.). +- **L3 — intent algebra** (`QueryExpr`, `core::intent_algebra`) — Symbolic, intent-only IR. No sketch type, no params. `AggIntent` carries accuracy targets only. The "algebra" is the operator surface (`Scan`, `Filter`, `Aggregate`, `Window`, …); the algebra is *intent-only* because no sketches have been bound yet. +- **L4 — sketch algebra** (`SketchExpr`, `core::sketch_algebra`) — Same DAG shape as L3, but sketches now committed (kind + params). Produced by L4 binding rules; consumed by L5 emitters. Note the naming inversion vs. earlier drafts: "sketch algebra" is L4 (where sketches actually live), not L3. +- **L5 — physical plan** — Stage-assigned, executor-targeted, ready to serialize. Produced by `PhysicalPlanner`, written out by `PlanEmitter`. + +### Metadata sources (see §6 "DAG schema, DB schema, sketch catalog") + +- **DAG schema** (`Schema`) — Columns + types + `unique_keys` carried on every L3/L4/L5 edge. Type-checked locally at each node. +- **DB schema** / **source schema** (`SchemaCatalog`) — Data-plane metadata (Prometheus TSDB / SQL `information_schema` / DataFusion catalog). Read by `core::lower::*` for L1→L2 symbol resolution. +- **Sketch catalog** (`SketchCatalog`) — Runtime registry of available primitives: what sketches exist, mergeability, deletability, parameter ranges, accuracy model. Consulted by L4 binding rules. **Not a schema** — a catalog of available primitives, not a description of a stream. + +### Workload + identity + +- **QueryWorkload** — Top-level controller input: one or more `QuerySpec`s plus workload features (batch vs streaming, reuse opportunity, data source). Arrives via HTTP POST, OpAMP capability-miss callback, YAML file, or query-log replay. +- **QuerySpec** — A single query: source-language string + accuracy / latency / cost target. + +## 15. Success criteria The migration is done when: 1. `asap-controller` binary runs and passes DC controller's existing integration tests (OpAMP push, backend config POST, SLA replan). 2. `asap-query` binary takes the same YAML input asap-planner-rs does today and produces byte-identical `streaming_config.yaml` + `inference_config.yaml` (fuzz-test against a corpus of fixtures). 3. ASAPQuery-backend's docker-compose no longer starts `asap-planner-rs`; the controller handles both shapes. -4. `asap-fusion`'s microbenchmarks still run under `scenario-fusion` with identical numbers. +4. `asap-fusion`'s microbenchmarks still run under `deployment-model-asapfusion` with identical numbers. 5. The `DataCollector/controller/`, `ASAPQuery/asap-planner-rs/`, `ASAPQuery-backend/asap-planner-rs/`, and `asap-fusion/` directories are deletable (or already deleted) without breaking any currently-running deployment. -6. A new hypothetical scenario can be added with zero changes outside its crate + one line in `bin/asap-controller/main.rs`. +6. A new hypothetical deployment model can be added with zero changes outside its crate + one line in `bin/asap-controller/main.rs`. diff --git a/docs/migration-plan.md b/docs/migration-plan.md index 1cbe13b4..ba8918cb 100644 --- a/docs/migration-plan.md +++ b/docs/migration-plan.md @@ -36,28 +36,28 @@ Ordered by risk, lowest first. Each phase is a single PR unless noted. ### Phase 1 — Core extraction: all-5-layer shared infrastructure -**Scope:** Lift the DC controller's **layers 1–3** (query parsing + per-language algebra + sketch-algebra IR) into `crates/core/` verbatim, AND lift L4 and L5 **shared infrastructure**: rule engine, shared rule library, cost-model traits, `PhysicalPlanner` trait, stage allocator framework, sketch catalogue. Scenarios in later phases pick from this common base and only add scenario-specific pieces. +**Scope:** Lift the DC controller's **layers 1–3** (query parsing + per-language algebra + intent-algebra IR) into `crates/core/` verbatim, AND lift L4 and L5 **shared infrastructure**: rule engine, shared rule library, cost-model traits, `PhysicalPlanner` trait, stage allocator framework, sketch catalogue, sketch-bound L4 IR (`SketchExpr` in `core::sketch_algebra`). Deployment models in later phases pick from this common base and only add deployment-model-specific pieces. -Per design §3, these five layers' infrastructure is query-language-independent AND scenario-independent; the concrete rule set, topology, and emitter are scenario-specific. +Per design §3, these five layers' infrastructure is query-language-independent AND deployment-model-independent; the concrete rule set, topology, and emitter are deployment-model-specific. **Work — L1-3 (straight lift from DC):** - **`core::query_language`** (L1): lift `DataCollector/controller/src/query_parser/{promql,sql,mod}.rs` → `core/src/query_language/`. One module per language. - **`core::logical_plan`** (L2): lift per-language algebra tree definitions → `core/src/logical_plan/`. One `enum LogicalPlan` per language. Add `core::logical_plan::datafusion` as a thin re-export of `datafusion::logical_expr::LogicalPlan` for fusion's L2. -- **`core::sketch_algebra`** (L3): lift `algebra/{expr,directory}.rs` → `core/src/sketch_algebra/`. Enforce **intent-only** — `AggIntent` carries accuracy target, never sketch type. **Generalize `QueryExpr::Scan` over data model**: introduce a `Source` sum (`TimeSeries` / `Table` / `Join` variants) inside `Scan`, and a `DataModel` enum (`TimeSeries` / `Tabular` / `Any`). DC's current scan logic becomes `Source::TimeSeries`; `Source::Table` is new (stub impl fine for Phase 1 — scenario-fusion fills it in Phase 3). Add `AggIntent::requires() -> DataModel` so L4 rules can skip non-applicable intents. See design §3 "Data-model support" and §6 `core::sketch_algebra`. ~300 LOC on top of the DC lift. +- **`core::intent_algebra`** (L3): lift `algebra/{expr,directory}.rs` → `core/src/intent_algebra/`. Enforce **intent-only** — `AggIntent` carries accuracy target, never sketch type. **Generalize `QueryExpr::Scan` over data model**: introduce a `Source` sum (`TimeSeries` / `Table` / `Join` variants) inside `Scan`, and a `DataModel` enum (`TimeSeries` / `Tabular` / `Any`). DC's current scan logic becomes `Source::TimeSeries`; `Source::Table` is new (stub impl fine for Phase 1 — deployment-model-asapfusion fills it in Phase 3). Add `AggIntent::requires() -> DataModel` so L4 rules can skip non-applicable intents. See design §3 "Data-model support" and §6 `core::intent_algebra`. ~300 LOC on top of the DC lift. - **`core::lower`**: lift `algebra/lower.rs` (L1→L2→L3 passes) → `core/src/lower/`. One `lower_` entry point per language. Each lowering produces the appropriate `Source` variant (PromQL / SQL → `Source::TimeSeries`; DataFusion → `Source::Table`). **Work — L4 framework (NEW):** - **`core::optimizer::engine`**: rule driver — fixed-point iteration, cycle detection, priority ordering. Generic over `OptimizerRule`. - **`core::optimizer::rule` trait**: `OptimizerRule` + `RuleCategory` enum (`PushDown | Fusion | Elim | Bind | StageRouting`). Priority u16. `apply(&expr, &constraints) -> Option`. -- **`core::optimizer::rules`**: shared rule library. Lift the **scenario-agnostic rules** from DC's `algebra/optimizer.rs`: +- **`core::optimizer::rules`**: shared rule library. Lift the **deployment-model-agnostic rules** from DC's `algebra/optimizer.rs`: - `BindKllOnQuantile` (intent → KLL with params from accuracy) - `BindCmsOnCount` (+ `BindCmsOnSum`) - `BindHllOnCardinality` - `BindDdsketchOnQuantile` (alternative to KLL) - `FusionPassthrough` (Aggregate over Filter → push Filter under) - `ElimNoopFilter`, `ElimNoopSort`, `ElimNoopLimit` - - Scenario-specific rules (stage-aware push-down, TCO-aware deferral) STAY in `scenario-lifecycle` — not lifted. -- **`core::optimizer::cost`**: `CostModel` trait + `Accuracy` / `Latency` / `Dollars` value types. Lift DC's **generic** cost functions (memory budget, accuracy degradation curve). Scenario-specific cost models (delta, online, pareto, tco) STAY in `scenario-lifecycle`. + - DeploymentModel-specific rules (stage-aware push-down, TCO-aware deferral) STAY in `deployment-model-asaplifecycle` — not lifted. +- **`core::optimizer::cost`**: `CostModel` trait + `Accuracy` / `Latency` / `Dollars` value types. Lift DC's **generic** cost functions (memory budget, accuracy degradation curve). DeploymentModel-specific cost models (delta, online, pareto, tco) STAY in `deployment-model-asaplifecycle`. - **`core::plan`**: `DeploymentConstraints` trait — memory budgets, network topology descriptor, available sketch backends. **Work — L5 framework (NEW):** @@ -70,7 +70,7 @@ Per design §3, these five layers' infrastructure is query-language-independent - **`core::pipeline`**: the L1→…→L5 driver. ~300 LOC. - **`core::workload`**: lift `QuerySpec` + add `QueryWorkload` sum type. - **`core::emit`**: `PlanEmitter` trait + `EmitError`. -- **`core::registry`**: `ScenarioRegistry`, `ScenarioId` enum. +- **`core::registry`**: `DeploymentModelRegistry`, `DeploymentModelId` enum. **Testing:** - Port DC's existing `query_parser/` + `algebra/` unit tests verbatim. All pass after the move. @@ -83,7 +83,7 @@ Per design §3, these five layers' infrastructure is query-language-independent **Decision points:** 1. `asap_types` — publish to a private git tag in Phase 0. 2. `promql-parser` / `sqlparser` versions — adopt DC's newer versions. -3. **Which DC rules go into `core::optimizer::rules`?** Rule of thumb: if it depends only on `QueryExpr` + `DeploymentConstraints` (no DC-specific types), lift. If it reaches into DC's stage graph, backend_client, or OTel YAML shape, it stays in `scenario-lifecycle`. Default: lift `Bind*`, `Elim*`, `FusionPassthrough`; keep `StageAwarePushDown`, `TransmissionCostRewrite` in scenario-lifecycle. +3. **Which DC rules go into `core::optimizer::rules`?** Rule of thumb: if it depends only on `QueryExpr` + `DeploymentConstraints` (no DC-specific types), lift. If it reaches into DC's stage graph, backend_client, or OTel YAML shape, it stays in `deployment-model-asaplifecycle`. Default: lift `Bind*`, `Elim*`, `FusionPassthrough`; keep `StageAwarePushDown`, `TransmissionCostRewrite` in deployment-model-asaplifecycle. **Exit criteria:** `cargo build` green; DC's parser + algebra + subset of optimizer unit tests pass from `core/`; `StageAllocator` unit-tested against all pre-baked topologies; golden-file corpus locked in. @@ -109,43 +109,43 @@ Per design §3, these five layers' infrastructure is query-language-independent --- -### Phase 3 — `scenario-fusion`: first thin scenario +### Phase 3 — `deployment-model-asapfusion`: first thin deployment model -**Scope:** Land `scenario-fusion` as a thin scenario that picks rules from core's shared library. Lowest-risk because fusion's existing `SketchConfigRule` maps directly onto core's `BindKllOnQuantile` + `BindCmsOnCount` + `BindHllOnCardinality` that Phase 1 already lifted. +**Scope:** Land `deployment-model-asapfusion` as a thin deployment model that picks rules from core's shared library. Lowest-risk because fusion's existing `SketchConfigRule` maps directly onto core's `BindKllOnQuantile` + `BindCmsOnCount` + `BindHllOnCardinality` that Phase 1 already lifted. -**Decision — where does `scenario-fusion` live?** Default: **in the `asap-fusion` repo**, as a crate that depends on `asap-control-core` + `asap-control-optimizer`. asap-fusion is a research project with independent release cadence; keeping it in-tree in its own repo preserves that. If the team prefers lockstep, the alternative is `crates/scenario-fusion/` in ASAPController workspace — architecture is identical either way (see design §8). +**Decision — where does `deployment-model-asapfusion` live?** Default: **in the `asap-fusion` repo**, as a crate that depends on `asap-control-core` + `asap-control-optimizer`. asap-fusion is a research project with independent release cadence; keeping it in-tree in its own repo preserves that. If the team prefers lockstep, the alternative is `crates/deployment-model-asapfusion/` in ASAPController workspace — architecture is identical either way (see design §8). This plan assumes the in-repo-at-asap-fusion placement. If in-workspace is chosen, swap paths accordingly. **Work:** -- In `asap-fusion/`: add new crate `asap-fusion-scenario/` that depends on `asap-control-core`. Move the translator + optimizer + executor here. -- **Fill in `Source::Table` lowering**: Phase 1 stubbed this; now implement the L2 (DataFusion `LogicalPlan`) → L3 (`QueryExpr` with `Source::Table`) lowering pass in `core::lower::datafusion` — or, if fusion prefers, keep the lowering inside the scenario crate and use the `Source::Table` type directly. Either way, Phase 1's stub becomes concrete. -- `impl Scenario for FusionScenario`: +- In `asap-fusion/`: add new crate `deployment-model-asapfusion/` that depends on `asap-control-core`. Move the translator + optimizer + executor here. +- **Fill in `Source::Table` lowering**: Phase 1 stubbed this; now implement the L2 (DataFusion `LogicalPlan`) → L3 (`QueryExpr` with `Source::Table`) lowering pass in `core::lower::datafusion` — or, if fusion prefers, keep the lowering inside the deployment model crate and use the `Source::Table` type directly. Either way, Phase 1's stub becomes concrete. +- `impl DeploymentModel for ASAPFusionDeploymentModel`: - `rules()` picks `BindKllOnQuantile`, `BindCmsOnCount`, `BindHllOnCardinality` from `core::optimizer::rules::*` (drops asap-fusion's `SketchConfigRule` in favor of core's rules). - - Adds scenario-specific `HashModeRule` (stays local — depends on DataFusion-specific types). + - Adds deployment-model-specific `HashModeRule` (stays local — depends on DataFusion-specific types). - `topology()` returns `core::physical::topology::ZeroStage`. - `emitter()` returns the DataFusion `LogicalPlan` rewriter. -- Port the microbenchmarks to `asap-fusion-scenario/benches/`. Verify numbers within 5% of pre-migration baseline. +- Port the microbenchmarks to `deployment-model-asapfusion/benches/`. Verify numbers within 5% of pre-migration baseline. - Legacy `asap-fusion` crate becomes a thin shim that re-exports the new crate, so in-flight benchmark harnesses keep compiling. **Risk:** low — no service consumes asap-fusion today; only downstream is its own benches. The rule-lift cleanup (`SketchConfigRule` → core rules) is the main change; golden-file test over the 3 existing rewrites catches any divergence. -**Exit criteria:** `cargo bench -p asap-fusion-scenario` produces numbers within 5% of pre-migration; 3 rewrites documented in `asap-fusion/TESTS.md` produce identical output. +**Exit criteria:** `cargo bench -p deployment-model-asapfusion` produces numbers within 5% of pre-migration; 3 rewrites documented in `asap-fusion/TESTS.md` produce identical output. **PR size:** ~1800 LOC (asap-fusion's ~3k lines minus what folds into core's shared rule library). --- -### Phase 4 — `scenario-query`: migrate asap-planner-rs's newer copy +### Phase 4 — `deployment-model-asapquery`: migrate asap-planner-rs's newer copy -**Scope:** Move `ASAPQuery-backend/asap-planner-rs/` (the newer copy) into `crates/scenario-query/`, **and refactor it to fit the 5-layer model**: reverse-engineer PromQL pattern templates into an L2 tree, and split L3 intent from L4 sketch binding. Delete the older copy in `ASAPQuery/`. +**Scope:** Move `ASAPQuery-backend/asap-planner-rs/` (the newer copy) into `crates/deployment-model-asapquery/`, **and refactor it to fit the 5-layer model**: reverse-engineer PromQL pattern templates into an L2 tree, and split L3 intent from L4 sketch binding. Delete the older copy in `ASAPQuery/`. This phase is larger than a straight lift because two structural conformance changes are bundled in. **Work — the lift:** - Diff the two `asap-planner-rs` copies. Newer (backend) copy wins. **Port only the newer; discard the older.** -- Copy `ASAPQuery-backend/asap-planner-rs/src/{lib,main}.rs` + `src/{planner,output,query_log,prometheus_client}/` → `crates/scenario-query/src/`. -- `impl Scenario for QueryScenario` — registers YAML emitters (`StreamingConfigEmitter` + `InferenceConfigEmitter`) and HTTP route `POST /plan/query`. +- Copy `ASAPQuery-backend/asap-planner-rs/src/{lib,main}.rs` + `src/{planner,output,query_log,prometheus_client}/` → `crates/deployment-model-asapquery/src/`. +- `impl DeploymentModel for ASAPQueryDeploymentModel` — registers YAML emitters (`StreamingConfigEmitter` + `InferenceConfigEmitter`) and HTTP route `POST /plan/query`. - Port the CLI: `bin/asap-query/main.rs` with clap flags mirroring `asap-planner-rs`'s current CLI. **Work — L2 tree conformance (NEW):** @@ -156,15 +156,15 @@ This phase is larger than a straight lift because two structural conformance cha - **This is the conformance cost the user explicitly accepted for architectural uniformity.** Budget accordingly. **Work — L3 / L4 split (NEW):** -- In `core::lower::promql_to_sketch_algebra`: `PromqlLogicalPlan → QueryExpr` producing **intent-only** L3 (no sketch names). +- In `core::lower::promql_to_intent_algebra`: `PromqlLogicalPlan → QueryExpr` producing **intent-only** L3 (no sketch names). - `Statistic` enum → `AggIntent` subset (9 of DC's 25 variants). No sketch type, no sketch params at L3. -- In `scenario-query/src/rules.rs`: picks core's `BindKllOnQuantile`, `BindCmsOnCount`, etc. from Phase 1's shared rule library + adds a scenario-specific `PrecomputeEngineBindRule` (handles the precompute-engine-specific flavor: which sketches are available in ASAPQuery-backend's accumulator set, what params match the engine's config schema, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). -- Update `build_agg_configs_for_statistics` to call `core::optimizer::engine::RuleEngine::run()` with the scenario's rule set instead of the inline `map_statistic_to_precompute_operator` fusion. +- In `deployment-model-asapquery/src/rules.rs`: picks core's `BindKllOnQuantile`, `BindCmsOnCount`, etc. from Phase 1's shared rule library + adds a deployment-model-specific `PrecomputeEngineBindRule` (handles the precompute-engine-specific flavor: which sketches are available in ASAPQuery-backend's accumulator set, what params match the engine's config schema, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). +- Update `build_agg_configs_for_statistics` to call `core::optimizer::engine::RuleEngine::run()` with the deployment model's rule set instead of the inline `map_statistic_to_precompute_operator` fusion. - `IntermediateAggConfig` loses its inline sketch binding; it's now an L4 output type produced by `PrecomputeEngineBindRule`. **Work — topology + emitter:** -- `scenario-query/src/topology.rs`: declare `core::physical::topology::SingleStage` (backend-only). -- `scenario-query/src/emit/`: `StreamingConfigEmitter` + `InferenceConfigEmitter`. Lift from asap-planner-rs's `output/generator.rs`. These are the authoritative YAML emitters (scenario-lifecycle calls them). +- `deployment-model-asapquery/src/topology.rs`: declare `core::physical::topology::SingleStage` (backend-only). +- `deployment-model-asapquery/src/emit/`: `StreamingConfigEmitter` + `InferenceConfigEmitter`. Lift from asap-planner-rs's `output/generator.rs`. These are the authoritative YAML emitters (deployment-model-asaplifecycle calls them). **Testing:** - **Golden-file test**: capture a corpus of today's `asap-planner-rs` inputs → outputs. The new CLI must produce byte-identical output. Put under `bin/asap-query/tests/golden/`. This is the non-negotiable safety net for the refactor. @@ -186,28 +186,28 @@ This phase is larger than a straight lift because two structural conformance cha --- -### Phase 5 — `scenario-lifecycle`: DC's remaining scenario-specific pieces +### Phase 5 — `deployment-model-asaplifecycle`: DC's remaining deployment-model-specific pieces -**Scope:** Land `scenario-lifecycle` as a thin scenario. L1-3 + L4 shared rules + L5 framework already moved in Phase 1; only DC's **scenario-specific** L4/L5 pieces remain: stage-aware rewrite rules, cost models, the 3-stage `ThreeStage` topology's DC-specific constraints, OpAMP + backend emitters, SLA replanner wiring. +**Scope:** Land `deployment-model-asaplifecycle` as a thin deployment model. L1-3 + L4 shared rules + L5 framework already moved in Phase 1; only DC's **deployment-model-specific** L4/L5 pieces remain: stage-aware rewrite rules, cost models, the 3-stage `ThreeStage` topology's DC-specific constraints, OpAMP + backend emitters, SLA replanner wiring. -**Work — scenario crate:** -- **Rule selection**: `scenario-lifecycle/src/rules.rs` picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules: +**Work — deployment model crate:** +- **Rule selection**: `deployment-model-asaplifecycle/src/rules.rs` picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules: - `StageAwarePushDown` — pushes ops toward edge stage when data size + edge memory budget permit - `TransmissionCostRewrite` — uses the TCO model to defer aggregation when it saves bandwidth - `DeltaRewrite` — replaces raw sends with delta-encoded sends when the online cost model says it pays off Each impls `core::optimizer::rule::OptimizerRule`. -- **Cost models**: `scenario-lifecycle/src/cost.rs` — lift `DataCollector/controller/src/planner/{delta_cost_model,online_cost_model,pareto,tco,rules,baseline,cost_model}.rs`. Each impls `core::optimizer::cost::CostModel`. These stay scenario-specific because they encode DC's deployment assumptions (edge memory, WAN bandwidth, multi-tenant). -- **Topology**: `scenario-lifecycle/src/topology.rs` — uses `core::physical::topology::ThreeStage`; only scenario-specific bit is declaring which stage roles map to which OpAMP `AgentRole` enum values. -- **Emitters**: `scenario-lifecycle/src/emit/` — `OpAmpRemoteConfigEmitter` (per-role OTel YAML, lifted from `DataCollector/controller/src/config/generate_{agent,backend}_config*.rs`) + `AsapqueryBackendConfigEmitter` (calls `scenario-query::StreamingConfigEmitter` for the YAML bytes, then POSTs via `runtime::backend_client`). -- `impl Scenario for LifecycleScenario` — registers HTTP routes `POST /plan` and `POST /replan`. +- **Cost models**: `deployment-model-asaplifecycle/src/cost.rs` — lift `DataCollector/controller/src/planner/{delta_cost_model,online_cost_model,pareto,tco,rules,baseline,cost_model}.rs`. Each impls `core::optimizer::cost::CostModel`. These stay deployment-model-specific because they encode DC's deployment assumptions (edge memory, WAN bandwidth, multi-tenant). +- **Topology**: `deployment-model-asaplifecycle/src/topology.rs` — uses `core::physical::topology::ThreeStage`; only deployment-model-specific bit is declaring which stage roles map to which OpAMP `AgentRole` enum values. +- **Emitters**: `deployment-model-asaplifecycle/src/emit/` — `OpAmpRemoteConfigEmitter` (per-role OTel YAML, lifted from `DataCollector/controller/src/config/generate_{agent,backend}_config*.rs`) + `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery::StreamingConfigEmitter` for the YAML bytes, then POSTs via `runtime::backend_client`). +- `impl DeploymentModel for ASAPLifecycleDeploymentModel` — registers HTTP routes `POST /plan` and `POST /replan`. **Work — runtime wiring:** - **OpAMP server**: move `DataCollector/controller/src/opamp/` → `runtime/opamp/`. OpAMP is a transport, not a plan-type, so it lives in runtime. - **Replanner + monitor**: move `DataCollector/controller/src/{replan,monitor}.rs` → `runtime/{replan,monitor}/`. -- **Dispatch**: `runtime/http/` routes `POST /plan` through `ScenarioRegistry` instead of calling DC's planner directly. +- **Dispatch**: `runtime/http/` routes `POST /plan` through `DeploymentModelRegistry` instead of calling DC's planner directly. **Integration test:** -- Send a `QuerySpec` over HTTP → core runs L1-3 → `LifecycleScenario`'s rule set fires (core rules + DC-specific rules) → `StageAllocator` with `ThreeStage` topology produces staged plan → emitters produce OTel YAML + `StreamingConfig` → fake backend asserts POST body matches golden file. Port DC's existing test of this shape. +- Send a `QuerySpec` over HTTP → core runs L1-3 → `ASAPLifecycleDeploymentModel`'s rule set fires (core rules + DC-specific rules) → `StageAllocator` with `ThreeStage` topology produces staged plan → emitters produce OTel YAML + `StreamingConfig` → fake backend asserts POST body matches golden file. Port DC's existing test of this shape. **Risk:** medium. Cost-model code (`delta/online/pareto/tco`) is ~40KB of subtle logic; port mechanically — do not touch the logic. The generic pieces already in core (stage allocator, bind rules) have been unit-tested in Phase 1, reducing this phase's surface area meaningfully. @@ -230,7 +230,7 @@ This phase is larger than a straight lift because two structural conformance cha - Capability-miss callback in `SimpleEngine` continues to `POST /api/v1/plan` on the controller — no code change, same endpoint. - **ASAPQuery PR**: delete `asap-planner-rs/` (the older copy). Any downstream refs in ASAPQuery update to point at ASAPController's `asap-query` binary. - **DataCollector PR**: delete `controller/` directory. Anyone who ran `cargo build -p controller` now runs `cargo build -p asap-controller` against the new repo. Update DC's README to redirect. -- **asap-fusion PR**: convert the repo to a thin shim that re-exports `scenario-fusion` OR archive the repo outright if no external user depends on it. Recommendation: **archive**. +- **asap-fusion PR**: convert the repo to a thin shim that re-exports `deployment-model-asapfusion` OR archive the repo outright if no external user depends on it. Recommendation: **archive**. **Risk:** medium. Deletion is final but reversible via `git revert`; actual blast radius is low because Phase 5 has already proven the new binary works. @@ -243,7 +243,7 @@ This phase is larger than a straight lift because two structural conformance cha --- -### Phase 7 — Core refactor: tighten traits now that 3 scenarios exist +### Phase 7 — Core refactor: tighten traits now that 3 deployment models exist **Scope:** Look at the shape of the `Planner` / `Optimizer` / `PlanRewriteRule` traits with 3 real implementations, tighten anything awkward. @@ -266,13 +266,13 @@ This phase is larger than a straight lift because two structural conformance cha | 0. Skeleton | 1 | ~200 | 1 day | | 1. Core — L1-3 + L4/L5 framework + shared rules + `Source`/`DataModel` | 1–2 | ~6500 | 1.5 weeks | | 2. Runtime HTTP+store | 1 | ~7700 | 3 days | -| 3. scenario-fusion (thin; rules fold into core library) | 1 | ~9500 | 1 week | -| 4. scenario-query (L2 tree + L3/L4 split + YAML + CLI) | 1 | ~14000 | 2 weeks | -| 5. scenario-lifecycle (cost models + scenario-specific rules + OpAMP wiring) | 2 | ~17500 | 1 week | +| 3. deployment-model-asapfusion (thin; rules fold into core library) | 1 | ~9500 | 1 week | +| 4. deployment-model-asapquery (L2 tree + L3/L4 split + YAML + CLI) | 1 | ~14000 | 2 weeks | +| 5. deployment-model-asaplifecycle (cost models + deployment-model-specific rules + OpAMP wiring) | 2 | ~17500 | 1 week | | 6. Cutover (delete in source repos) | 3–4 | ~18000 | 1 week | | 7. Core refactor | 1 | ~18500 | 3 days | -**Total**: ~6.5 weeks for one engineer. Phase 1 grew (+0.5w) because it now also lifts L4/L5 shared infrastructure. Phases 3 and 5 shrank correspondingly (-1w combined) because fusion's `SketchConfigRule`-style bindings and lifecycle's `StageAllocator` now come from core's shared library. Net: slightly faster total, much cleaner per-scenario crates (each ~1500-2500 LOC of scenario-specific code rather than 5000+). +**Total**: ~6.5 weeks for one engineer. Phase 1 grew (+0.5w) because it now also lifts L4/L5 shared infrastructure. Phases 3 and 5 shrank correspondingly (-1w combined) because fusion's `SketchConfigRule`-style bindings and lifecycle's `StageAllocator` now come from core's shared library. Net: slightly faster total, much cleaner per-deployment-model crates (each ~1500-2500 LOC of deployment-model-specific code rather than 5000+). Parallelisable if two people: lifecycle (Phase 5) can start as soon as Phase 1's L4/L5 framework lands (before Phase 4's L2 tree work). @@ -282,14 +282,14 @@ Parallelisable if two people: lifecycle (Phase 5) can start as soon as Phase 1's |---|---|---|---| | YAML emission byte-differences between old and new planner | Medium | Breaks ASAPQuery-backend's config consumer | Golden-file test corpus in Phase 4; fuzz harness over the YAML schema | | OpAMP protocol detail drifts during port | Low | Breaks agent config push | Port opamp/ verbatim in Phase 2's runtime crate; don't touch until it's proven to work | -| DataFusion version skew in scenario-fusion | Low | Benchmarks drift | Pin DataFusion version in workspace `Cargo.toml`; measure benches before + after | +| DataFusion version skew in deployment-model-asapfusion | Low | Benchmarks drift | Pin DataFusion version in workspace `Cargo.toml`; measure benches before + after | | Two asap-planner-rs copies have subtle diff we miss | Medium | Behaviour regression | Explicit diff + reconcile step in Phase 4; one authoritative copy (the backend's newer one) | | ASAPQuery-backend deploys break during cutover | Low | Production outage | Phase 6 only after Phase 5 fully validates; staged rollout of docker-compose change | | Scope creep — "while we're at it, redesign the cost model" | High | Doubles timeline | Explicit non-goal list in design doc §2; defer all rewrites to post-migration phases | -| Core traits are wrong; need to rev after scenarios land | Medium | Phase 7 PR larger than expected | Accept; Phase 7 is planned for exactly this. Don't over-design Phase 1. | +| Core traits are wrong; need to rev after deployment models land | Medium | Phase 7 PR larger than expected | Accept; Phase 7 is planned for exactly this. Don't over-design Phase 1. | | PromQL L2 tree reverse-engineering loses a pattern-match case planner relied on | Medium | Generated YAML diverges from legacy | Golden-file corpus covers all 5 pattern shapes + generic fallthrough; tree-shape inspection at L3 lowering is a direct 1:1 translation of today's pattern-match logic, not a reinterpretation | | L3/L4 split misses a case in `map_statistic_to_precompute_operator` | Medium | Wrong sketch selected for some `(Statistic, Treatment)` combos | Every branch of the current function must round-trip via the new L3→L4 path on the golden-file corpus; add table-driven unit tests enumerating every `Statistic × Treatment` combination | -| `AggIntent` as a 25-variant superset grows unbounded as scenarios add needs | Low | Core churn | Variants added only for intent shapes used by ≥1 shipped scenario; never speculative. | +| `AggIntent` as a 25-variant superset grows unbounded as deployment models add needs | Low | Core churn | Variants added only for intent shapes used by ≥1 shipped deployment model; never speculative. | ## Decision log (during migration) @@ -297,42 +297,42 @@ These are the questions that will come up mid-migration. Pre-decide as many as p 1. **Which version of `asap_types`?** ASAPQuery-backend's current one, pinned to a git SHA. 2. **Which `promql-parser` + `sqlparser` versions?** DC controller's (newer). ASAPQuery-backend's planner uses older versions; port it to the newer ones during Phase 4. -3. **How is `scenario-lifecycle`'s `StreamingConfig` YAML emitter related to `scenario-query`'s?** One emitter in `scenario-query`, called from `scenario-lifecycle`. Asymmetric dependency. -4. **Does `asap-controller` have feature flags to disable scenarios at build time?** Yes, one `--features lifecycle,query,fusion` with all three enabled by default. A minimal CLI binary (`asap-query`) disables lifecycle and fusion. +3. **How is `deployment-model-asaplifecycle`'s `StreamingConfig` YAML emitter related to `deployment-model-asapquery`'s?** One emitter in `deployment-model-asapquery`, called from `deployment-model-asaplifecycle`. Asymmetric dependency. +4. **Does `asap-controller` have feature flags to disable deployment models at build time?** Yes, one `--features lifecycle,query,fusion` with all three enabled by default. A minimal CLI binary (`asap-query`) disables lifecycle and fusion. 5. **Who owns the Cargo.lock?** ASAPController. Downstream repos (ASAPQuery-backend, DataCollector) do NOT depend on ASAPController as a Cargo path dep — they pull the published binary via Docker or pin a git SHA. -6. **Does `ControllerClient` on the ASAPQuery-backend side need changes?** No. It keeps POSTing to `/api/v1/plan`; the controller's routing layer dispatches to `scenario-lifecycle` (same as DC controller does today). -7. **L2 tree for asap-planner-rs — mandatory or optional?** Mandatory. planner's current template-catalogue approach is replaced with a proper `PromqlLogicalPlan` tree in Phase 4. The extra conformance work is taken to keep the 5-layer model uniform. See design doc §12 Q8 for the escape hatch if a future scenario genuinely can't fit a tree. +6. **Does `ControllerClient` on the ASAPQuery-backend side need changes?** No. It keeps POSTing to `/api/v1/plan`; the controller's routing layer dispatches to `deployment-model-asaplifecycle` (same as DC controller does today). +7. **L2 tree for asap-planner-rs — mandatory or optional?** Mandatory. planner's current template-catalogue approach is replaced with a proper `PromqlLogicalPlan` tree in Phase 4. The extra conformance work is taken to keep the 5-layer model uniform. See design doc §12 Q8 for the escape hatch if a future deployment model genuinely can't fit a tree. 8. **Sketch binding — L3 or L4?** L4. DC and fusion already do this correctly; planner's `map_statistic_to_precompute_operator` conflates L3+L4 and is split during Phase 4. `AggIntent` at L3 carries intent + accuracy only; concrete `AggregationType` / `SketchParams` are produced by an L4 rule. -9. **Intent vocabulary when scenarios disagree in width?** Core's `AggIntent` is the superset (DC's ~25 variants). Each scenario only uses / produces / accepts the subset it needs (planner 9, fusion 3). Adding a new intent variant is a core change that scenarios opt into. +9. **Intent vocabulary when deployment models disagree in width?** Core's `AggIntent` is the superset (DC's ~25 variants). Each deployment model only uses / produces / accepts the subset it needs (planner 9, fusion 3). Adding a new intent variant is a core change that deployment models opt into. 10. **Tabular vs time-series data models?** Handled in core from Phase 1 via the `Source` sum (`TimeSeries` / `Table` / `Join`) inside `QueryExpr::Scan` and the `DataModel` tag on `AggIntent`. ASAPQuery produces `Source::TimeSeries`, asap-fusion produces `Source::Table`. `Source::Table` is stubbed in Phase 1 and filled in by Phase 3 (fusion). Sketches themselves are data-model-agnostic; the generalization is purely at the IR leaf. -## What a new scenario looks like post-migration +## What a new deployment model looks like post-migration -This is the test of the architecture. Adding a 4th scenario (e.g. "edge-caching") — typical size ~500-2000 LOC: +This is the test of the architecture. Adding a 4th deployment model (e.g. "edge-caching") — typical size ~500-2000 LOC: -1. `cargo new --lib crates/scenario-edge-cache` (in ASAPController workspace) OR a new crate in your own repo that depends on `asap-control-core = { git = "...", tag = "v0.1.0" }`. +1. `cargo new --lib crates/deployment-model-edge-cache` (in ASAPController workspace) OR a new crate in your own repo that depends on `asap-control-core = { git = "...", tag = "v0.1.0" }`. 2. Four files in `src/`: - - `lib.rs` — `impl Scenario for EdgeCacheScenario` - - `rules.rs` — pick rules from `core::optimizer::rules::*` + add scenario-specific rules + - `lib.rs` — `impl DeploymentModel for EdgeCacheDeploymentModel` + - `rules.rs` — pick rules from `core::optimizer::rules::*` + add deployment-model-specific rules - `topology.rs` — declare a `TopologyDescriptor` (e.g. `TwoStage { edge, backend }`) or pick a pre-baked core one - `emit/edge_agent_config.rs` — `impl PlanEmitter` -3. Optionally: `cost.rs` if the scenario needs its own cost model; otherwise use `core::optimizer::cost::*`. +3. Optionally: `cost.rs` if the deployment model needs its own cost model; otherwise use `core::optimizer::cost::*`. 4. If landing in ASAPController workspace: 1 line in `bin/asap-controller/main.rs` to register + 1 line in workspace `Cargo.toml`. If out-of-tree: publish as a binary in your own repo. -**No changes** to runtime, core, or other scenarios. That's the extension-point test passing. +**No changes** to runtime, core, or other deployment models. That's the extension-point test passing. -### Why post-migration scenarios are much smaller than pre-migration code +### Why post-migration deployment models are much smaller than pre-migration code -Pre-migration: each scenario ships its own rule engine, cost traits, allocator, topology handling, sketch catalogue. Total per scenario: 3000-8000 LOC. +Pre-migration: each deployment model ships its own rule engine, cost traits, allocator, topology handling, sketch catalogue. Total per deployment model: 3000-8000 LOC. -Post-migration: those all come from `core::optimizer::*` + `core::physical::*` as imports. Per-scenario code is only: +Post-migration: those all come from `core::optimizer::*` + `core::physical::*` as imports. Per-deployment-model code is only: - Which rules to enable (usually a dozen lines — pick from `core::optimizer::rules`) - Topology declaration (a dozen lines — often reusing a pre-baked `core::physical::topology::*`) -- Scenario-specific rules that truly can't be shared (100-500 LOC each) -- Scenario-specific cost models (if different from core's generic) +- DeploymentModel-specific rules that truly can't be shared (100-500 LOC each) +- DeploymentModel-specific cost models (if different from core's generic) - Emitter(s) for the output format (100-1000 LOC) -Typical post-migration scenario: **500-2500 LOC**. Extremely easy to understand, review, and evolve. +Typical post-migration deployment model: **500-2500 LOC**. Extremely easy to understand, review, and evolve. ## Rollback strategy