From 8983bb24319d0a100acf4124fdba742426cb9879 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Sat, 2 May 2026 10:28:43 -0400 Subject: [PATCH 01/10] docs: clean up L3 QueryExpr; add schema flow + L4 SketchExpr + system I/O Address review feedback on the QueryExpr/AggIntent design: - Drop redundant L3 nodes (TopK = Sort+Limit; WindowedAgg = Window+Aggregate) - Move sketch-bound nodes (SketchAgg, JoinSketch) out of L3 into a new L4 IR (SketchExpr) with SketchSubtract/Delete/Estimate/Merge added - Drop language-shaped nodes from L3 (HistogramQuantile, PromQLSubquery lower in PromQL L1->L2 instead) - Rename Dedup -> Distinct{cols} (SQL DISTINCT, generalised to N cols) - Drop QuantileOverTime/TopK from AggIntent; surrounding Window covers it - Add Schema/Field/HasSchema with per-node input/output spec table - Distinguish three schemas: DAG schema vs DB/source schema vs sketch catalog metadata - Expand SketchCatalog with mergeability, accuracy, supported intents, aggregated-key shape, parameter ranges - New "System I/O contract" subsection: input is QueryWorkload, output is per-executor sub-DAG assignments under the premise that every executor has full capability and the controller chooses placement - Refresh resolved-questions list in Section 12 Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design.md | 395 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 296 insertions(+), 99 deletions(-) diff --git a/docs/design.md b/docs/design.md index b5d95fca..2e262680 100644 --- a/docs/design.md +++ b/docs/design.md @@ -34,7 +34,7 @@ DataCollector/controller already documents its query→sketch translation as a 5 |---|-------|--------------|-------------------| | 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql,mod}.rs`; asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | | 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/algebra/lower.rs`; asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | -| 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent` (~25 intent variants). Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters.** Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | +| 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `TopK` (use `Sort + Limit`), no `WindowedAgg` (use `Window` over `Aggregate`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | | 4 | **Sketch Optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 and emit sketch-bound L3+. ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library; scenarios **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | | 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artefact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; scenarios supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | @@ -55,7 +55,7 @@ Today: ### Intent vocabulary: DC's `AggIntent` is a superset; scenarios use subsets -DC has ~25 variants. Planner uses 9 (`Count, Sum, Cardinality, Increase, Rate, Min, Max, Quantile, Topk`). Fusion uses 3 (`Count, Sum, Quantile`). A scenario's L4+L5 only has to handle its own subset — but it reads and writes the **same `AggIntent` enum**. Adding a new intent (e.g. stddev) is a core change that scenarios can then opt in to. +DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because shapes that decompose into other operators (`TopK → Sort + Limit`; `QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core today is ~7 (`Count, Sum, Min, Max, Quantile, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps onto this set: `TopK` is no longer an intent — it lowers to `Sort + Limit` over `Aggregate{Count}` (or whatever ranking key), and the heavy-hitter sketch binds at L4 by matching that shape. Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that scenarios opt into. ### Data-model support: both time-series and tabular @@ -63,7 +63,7 @@ ASAPQuery-backend / DC controller operate on time-series data (metrics + labels Core handles this with: - **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Scenarios' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. -- **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase`, `QuantileOverTime`), or tabular-only (future additions for joins, correlated subqueries). +- **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Min`, `Max`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase` — both carry PromQL counter-reset semantics), or tabular-only (future additions for joins, correlated subqueries). - **Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values; that stream can come from a time-series window or a table column, the sketch does not know or care. So `BindKllOnQuantile` and siblings work uniformly across both. Practically, this means: @@ -85,6 +85,31 @@ Every scenario must produce an L2 tree, even when the source language didn't ori A future scenario whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. +### System I/O contract — what the controller takes in, what it emits + +The controller is a **planner**, not an executor. It does not run queries; it decides where each piece of a query runs. The data plane (OTel collectors / ASAPQuery-backend / DataFusion `SessionContext`) runs them. + +**System input — what the controller takes in.** A `QueryWorkload` (one or more `QuerySpec`s) plus deployment context (available executors, their capabilities, current SLA targets, telemetry of recent violations). The `QuerySpec` carries the raw query string in its source language (PromQL / SQL / DataFusion / ElasticDSL) and the accuracy / latency / cost target. Workloads can arrive via four entry points (HTTP `POST /plan`, OpAMP capability-miss callback, YAML file for the CLI shell, query-log replay); they all normalise to `QueryWorkload` before L1. + +**Workload features that the controller cares about, beyond the queries themselves:** +- Execution model: **batch** (one-shot YAML, query-log replay) vs **streaming** (live pipeline that must keep producing results as data arrives). +- Data input source: time-series scrape, tabular relation, query log replay, etc. Drives `Source` variant choice in L3. +- Reuse opportunity: ≥2 queries planned together → `CostModel::workload_cost` credits shared sub-expressions. + +**System output — what the controller emits.** For each registered executor in the deployment, a sub-DAG of the optimized plan plus the configuration that lets that executor run it. Concretely: + +| Output | Consumer | Wire shape | +|---|---|---| +| Per-executor sub-DAG assignment | The executor (edge agent / gateway / backend / DataFusion session) | OpAMP `RemoteConfig` (OTel YAML) / `streaming_config.yaml` POST / rewritten `LogicalPlan` | +| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge over Kafka, …) | Implied by the per-executor configs; not a separate artefact | +| Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | + +**Premise behind the per-executor split.** Every executor (edge collector, gateway, backend, DataFusion runtime) is in principle capable of running the entire query tree — they all speak the same physical operators. The controller's job is to decide *which* executor runs *which sub-tree* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by executor, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring; the `PhysicalPlanner` per scenario emits the per-executor configs. + +This is what makes the topology a *parameter* rather than an axis of code: the same `SketchExpr` plus a different `TopologyDescriptor` produces edge-only / 1-stage / 3-stage / 0-stage placements without rewriting the plan. + +**Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialise into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. + This drives the core/scenario split: - **`crates/core/`** owns all 5 layers of **shared infrastructure**: L1-3 end-to-end (parsers + lowering + intent-only IR), plus L4's rule engine driver + rule library + cost-model traits, plus L5's stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. @@ -134,7 +159,7 @@ ASAPController/ │ ├── core/ # Shared infrastructure across all 5 layers; no I/O │ │ ├── query_language/ # L1: per-language parsers — promql/, sql/, datafusion/, elasticdsl/ │ │ ├── logical_plan/ # L2: per-language algebra tree (Aggregate/Window/Filter/…) -│ │ ├── sketch_algebra/ # L3: QueryExpr + AggIntent (~25 variants) + sketch directory +│ │ ├── sketch_algebra/ # L3: QueryExpr + AggIntent + Schema/HasSchema + sketch directory │ │ ├── lower/ # L1→L2→L3 lowering passes (one entry per language) │ │ ├── optimizer/ # L4 framework: │ │ │ ├── engine/ # rule driver — fixed-point iteration, cycle detection, priority @@ -225,7 +250,15 @@ A **per-language** algebra tree — one `enum LogicalPlan` per language. Preserv ### `core::sketch_algebra` — Layer 3 -The language- and deployment-independent IR. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP scenarios) via a `Source` sum type inside `QueryExpr::Scan`. +The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `HistogramQuantile`, no `PromQLSubquery` — those are L2 PromQL nodes), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP scenarios) via a `Source` sum type inside `QueryExpr::Scan`. + +#### Design rules for L3 + +1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Sort + Limit` is the canonical top-k shape; there is no separate `TopK`. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). +2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artefact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. +3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. +4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. +5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding tree. CTEs / let-bindings produce reuse edges (multiple consumers). ```rust pub enum QueryExpr { @@ -237,149 +270,267 @@ pub enum QueryExpr { // ── Filtering & projection ──────────────────────────────────────────── /// σ — row-level filter (WHERE / PromQL label matchers). - Filter { child: Box, pred: Predicate }, + Filter { child: Box, pred: Predicate }, /// π — column projection (SELECT list). - Project { child: Box, cols: Vec }, - - // ── Aggregation ─────────────────────────────────────────────────────── - /// γ + α — GROUP BY + aggregates, with optional HAVING. Exact-agg path. - Aggregate { child: Box, by: Vec, aggs: Vec, - having: Option }, - /// γ + α specialised for a single sketch aggregation (one sketch per node). - /// Kept separate from `Aggregate` so the L4 allocator can bind a sketch - /// without re-parsing `AggFunc` variants. - SketchAgg { child: Box, intent: AggIntent, col: ColumnRef }, - /// Windowed sketch aggregation. Bundles window + intent because the window - /// defines the sketch lifecycle (flush / reset boundaries). - WindowedAgg { child: Box, intent: AggIntent, window: WindowSpec, - col: ColumnRef }, - - // ── Time / streaming ────────────────────────────────────────────────── - /// ψ — tumbling / sliding window (PromQL `[5m]`, SQL windowed aggregation). - Window { child: Box, duration: Duration, slide: Option }, - - // ── Distributed / multi-stage operators ────────────────────────────── + Project { child: Box, cols: Vec }, + + // ── Aggregation (logical, intent-only) ──────────────────────────────── + /// γ + α — GROUP BY + aggregate intents, with optional HAVING. + /// `aggs` carry `AggIntent`; concrete sketch / non-sketch operator + /// is chosen by L4 and lives in the L4-extended IR (`SketchExpr`). + Aggregate { child: Box, by: Vec, + aggs: Vec, having: Option }, + + // ── Time / streaming windows ────────────────────────────────────────── + /// ψ — tumbling / sliding / session window over the time axis. Defines + /// the lifecycle (flush / reset bounds) of any aggregate in its subtree. + /// PromQL `[5m]` and streaming windows lower here. SQL `OVER (...)` + /// analytic frames are a different node — see `WindowFunc` below. + Window { child: Box, kind: WindowKind, + size: Duration, slide: Option }, + + // ── Distributed-execution structure ─────────────────────────────────── /// Partition the stream by key tuple (`GROUP BY` / PromQL `by (dims)`). Partition { child: Box, keys: PartitionKeys }, - /// δ — deduplicate on `col` before sketch ingestion. - Dedup { child: Box, col: String }, - /// τ — retain only top-K heavy hitters. - TopK { child: Box, k: u64, by: Vec }, - /// ⊕ — merge sketches from independent branches (distributed union). + /// δ — SQL `DISTINCT` / row deduplication on `cols`. + Distinct { child: Box, cols: Vec }, + /// ⊕ — union of sub-results from independent stages or shards (the + /// exact-merge case). Sketch unions are a separate node in `SketchExpr` + /// because they carry sketch-family / params type constraints. Merge { children: Vec }, - // ── Joins ───────────────────────────────────────────────────────────── - Join { kind: JoinKind, left: Box, right: Box, - pred: Option }, - /// Sketch-aware join push-down: pre-aggregate on inner side, then merge. - JoinSketch { outer: Box, inner: Box, join_key: String }, + // ── Joins (logical) ─────────────────────────────────────────────────── + /// Logical join. L4 picks the physical alternative — `HashJoin` / + /// `SortMergeJoin` / `SketchJoin` (e.g. KMV / theta-sketch / join-sample) + /// — based on selectivity, memory budget, and accuracy target. The + /// sketch-aware variant lives in `SketchExpr::SketchJoin`. + Join { kind: JoinKind, left: Box, right: Box, + pred: Option }, // ── Set operators ───────────────────────────────────────────────────── /// UNION / INTERSECT / EXCEPT, with or without ALL. - SetOp { kind: SetOpKind, all: bool, left: Box, right: Box }, + SetOp { kind: SetOpKind, all: bool, + left: Box, right: Box }, // ── Ordering & limiting ─────────────────────────────────────────────── Sort { child: Box, keys: Vec }, + /// `LIMIT n OFFSET k`. `Sort` followed by `Limit` is the canonical + /// top-k / heavy-hitters shape — no separate `TopK` node, since one + /// canonical representation per plan keeps L4 rule matching unambiguous. + /// Heavy-hitter sketches (CMS-with-heap, SpaceSaving) bind at L4 by + /// pattern-matching `Sort + Limit` on top of `Aggregate`. Limit { child: Box, n: u64, offset: u64 }, // ── Subquery / CTE ──────────────────────────────────────────────────── Subquery { child: Box, alias: String }, - /// SQL `WITH name AS (expr) IN body`, PromQL recording-rule binding. + /// SQL `WITH name AS (expr) IN body`; lowering target for PromQL + /// recording-rule bindings. LetBinding { name: String, expr: Box, body: Box }, // ── Analytic (OVER) window functions ────────────────────────────────── + /// SQL `OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)`. + /// Distinct from `Window` above — that is a streaming/tumbling window + /// over the time axis; this is an analytic frame over already-grouped rows. WindowFunc { child: Box, func: WindowFuncKind, partition_by: Vec, order_by: Vec, frame: Option }, - // ── PromQL-specific ─────────────────────────────────────────────────── - /// `histogram_quantile(φ, )` — HLL / histogram sketch → quantile. - HistogramQuantile { child: Box, phi: f64 }, - /// PromQL sub-query: `[range:resolution]`. - PromQLSubquery { child: Box, range: Duration, - resolution: Option }, - /// PromQL / SQL binary op between two relational sub-expressions - /// (arithmetic, comparison, `and`/`or`/`unless`). + // ── Binary composition ──────────────────────────────────────────────── + /// Arithmetic / comparison / boolean composition between two relational + /// sub-expressions (PromQL binary ops including `and`/`or`/`unless`, + /// SQL boolean composition). BinaryOp { op: BinaryOpKind, lhs: Box, rhs: Box, vector_match: Option }, } +pub enum WindowKind { Tumbling, Sliding, Session } + pub enum Source { /// Time-series input — scenario-query / scenario-lifecycle shape. - /// `metric` identifies a metric family; `time` bounds the window; - /// `labels` constrains label-value combinations. - TimeSeries { - metric: MetricRef, - time: TimeRange, - labels: LabelFilter, - }, + TimeSeries { metric: MetricRef, time: TimeRange, labels: LabelFilter }, /// Tabular input — scenario-fusion / future-OLAP shape. - /// `table_ref` identifies the table; `columns` projects the subset - /// in use. Join / subquery composition nests `Source` values. - Table { - table_ref: TableRef, - columns: Vec, - }, - Join { - left: Box, - right: Box, - on: JoinKey, - }, + Table { table_ref: TableRef, columns: Vec }, + /// Join over Sources composes leaf shapes recursively. + Join { left: Box, right: Box, on: JoinKey }, // Future: WindowedStream, Subquery — added by scenarios that need them. } -pub enum DataModel { - TimeSeries, - Tabular, - Any, -} +pub enum DataModel { TimeSeries, Tabular, Any } impl Source { - pub fn data_model(&self) -> DataModel { - match self { - Self::TimeSeries { .. } => DataModel::TimeSeries, - Self::Table { .. } | Self::Join { .. } => DataModel::Tabular, - } - } + pub fn data_model(&self) -> DataModel { /* … */ } + /// Output schema produced by this leaf (see "Schema flow"). + pub fn schema(&self, catalog: &SchemaCatalog) -> Schema { /* … */ } +} +``` + +#### What was removed during cleanup, and why + +| Removed | Reason | Replacement | +|---|---|---| +| `TopK { k, by }` | Decomposes into `Sort + Limit`; two ways to spell the same plan break canonical-form invariant | `Sort { keys: by } → Limit { n: k }` | +| `WindowedAgg { intent, window }` | Equivalent to `Window` over `Aggregate`; redundant. Tumbling/sliding kind moves onto `Window::kind` | `Window { kind: …, … } → Aggregate { aggs: [intent] }` | +| `SketchAgg { intent, col }` | Sketch-bound; L3 must be intent-only | `Aggregate { aggs: [intent] }` at L3; L4 emits `SketchExpr::SketchAgg` | +| `JoinSketch { outer, inner, key }` | Sketch-bound physical alternative; L3 must be intent-only. Several papers describe sketch-of-join (KMV, theta, join-sample) — picking one is an L4 cost decision, not an L3 surface choice | `Join { … }` at L3; L4 emits `SketchExpr::SketchJoin` when a `Bind*OnJoin` rule fires | +| `HistogramQuantile { phi }` | PromQL artefact — consumes Prometheus's specific bucketed-histogram format. Language-specific | Lowers in PromQL L1→L2 to bucket reads + `Aggregate { aggs: [Quantile{q: phi, …}] }` over the bucket counts | +| `PromQLSubquery { range, resolution }` | Driver construct — asks the engine to evaluate the inner expression at N timestamps; not a single DAG node | Expanded by PromQL L1→L2 lowering into a set of independent `QueryExpr` instances, not preserved at L3 | +| `Dedup { col }` | Single-column-only spelling of SQL `DISTINCT` | Renamed to `Distinct { cols }`, generalised to N columns | + +#### Schema flow — every L3 edge carries a typed schema + +Every node has a derivable output schema given its input schemas and parameters. The DAG is type-checked: a `Filter` whose predicate references a column not in its child's output schema fails at plan time. + +```rust +pub struct Schema { + pub fields: Vec, + /// Index into `fields` for the time axis, if any. PromQL leaves carry one; + /// SQL leaves may or may not. + pub time_index: Option, + /// Functional dependencies / unique-key sets. L4 reuses these to recognise + /// when two sub-expressions produce identical streams (sketch-reuse driver). + pub unique_keys: Vec>, +} + +pub struct Field { + pub name: String, + pub dtype: DataType, // Int64 / Float64 / Utf8 / Map / … + pub nullable: bool, +} + +pub trait HasSchema { + fn input_schemas(&self) -> Vec<&Schema>; + fn output_schema(&self, inputs: &[&Schema], cat: &SchemaCatalog) -> Schema; } +``` + +Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/sketch_algebra/schema.rs`): + +| Node | Inputs | Output | +|---|---|---| +| `Scan { source }` | — | `source.schema(catalog)` | +| `Ref(name)` | — | resolved from `LetBinding` named `name` | +| `Filter { pred }` | one `S` | `S` (predicate is a refinement; no schema change) | +| `Project { cols }` | one `S` | projection of `S` to `cols` | +| `Aggregate { by, aggs }` | one `S` | `by` columns + one column per `agg` (named & typed by `AggIntent::output_type(input_field)`) | +| `Window { kind, size, slide }` | one `S` (must have `time_index`) | `S` extended with window-id / window-bounds metadata fields | +| `Partition { keys }` | one `S` | `S` (logical-only; physical sharding hint for L5) | +| `Distinct { cols }` | one `S` | `S` with `unique_keys` tightened to include `cols` | +| `Merge` | N (all `S_i`); rule: schemas must be union-compatible | `S_0` (representative) | +| `Join { kind, pred }` | two `L`, `R` | union of `L` + `R` columns minus duplicates removed by USING/NATURAL | +| `SetOp { kind, all }` | two (must be union-compatible) | left's schema | +| `Sort { keys }` / `Limit` | one `S` | `S` | +| `Subquery { alias }` | one `S` | `S` with table alias | +| `LetBinding { name, expr, body }` | `body` consumes; `expr` bound by name | body's output schema | +| `WindowFunc { func, partition_by, order_by, frame }` | one `S` | `S` extended with the analytic-function output column | +| `BinaryOp { op, vector_match }` | two `L`, `R` (PromQL vector-match constraints apply) | element-wise op → schema with op-typed value column; boolean → boolean column | + +#### Three distinct schemas — DAG vs DB vs sketch catalog + +These three are sometimes conflated and shouldn't be: + +| Schema | Where it lives | What it describes | Who reads it | +|---|---|---|---| +| **DAG schema** | On every edge of the L3 / L4 / L5 DAG (`Schema` above) | Columns + types flowing between operators | L4 rules (selectivity estimation, push-down legality), L5 emitter | +| **DB / source schema** | The query target (Prometheus TSDB metric metadata, SQL `information_schema`, DataFusion catalog) | What metrics / tables / columns exist in the data plane, with their types and indexing | `core::lower::*` to resolve names during L1→L2; exposed through a `SchemaCatalog` interface | +| **Sketch catalog metadata** | `core::physical::sketch_catalog` (built at startup; static) | What sketches the runtime can build; what intents each one serves; mergeability, accuracy / confidence guarantees, supported aggregation keys, parameter ranges | L4 binding rules to choose a sketch for an `AggIntent`; L5 to instantiate the sketch | + +L1→L2 lowering reads the **DB schema** to resolve symbols. L3 onward, every edge carries a **DAG schema** that is type-checked locally. L4 binding rules consult the **sketch catalog** to map an intent to a concrete sketch under the deployment's constraints. They are three separate inputs to three distinct decisions. +#### `AggIntent` — what to compute, not how + +```rust pub enum AggIntent { - // Data-model-agnostic — work on TimeSeries AND Tabular - Count { accuracy: AccuracyTarget }, + // Data-model-agnostic + Count { accuracy: AccuracyTarget }, Sum, Min, Max, - Quantile { q: f64, accuracy: AccuracyTarget }, - TopK { k: usize, accuracy: AccuracyTarget }, + Quantile { q: f64, accuracy: AccuracyTarget }, Cardinality { accuracy: AccuracyTarget }, - // Time-series only - Rate { window: Duration }, + // Time-series streaming derivatives — specific operations, not just + // "Sum / Count over a Window". `Rate` is the per-second average derivative + // computed with PromQL's counter-reset adjustment, not a generic windowed + // mean. Kept distinct because (a) they have counter-reset semantics that + // exact `Sum` does not, and (b) sketch backends specialised for + // derivatives (e.g. delta-set aggregator) bind on these intents directly. + Rate { window: Duration }, Increase { window: Duration }, - QuantileOverTime { q: f64, window: Duration, accuracy: AccuracyTarget }, - // Tabular / OLAP only — added as scenarios demand - // CorrelatedSubqueryCount { ... }, - // ApproxJoinCardinality { ... }, - // ~25 variants total + // Tabular / OLAP — added as scenarios demand + // CorrelatedSubqueryCount { … }, ApproxJoinCardinality { … }, } impl AggIntent { /// Which data-model this intent semantically requires. L4 rules /// consult this to skip non-applicable intents (e.g. `Rate` over /// a `Source::Table` is nonsense). - pub fn requires(&self) -> DataModel { - match self { - Self::Count{..} | Self::Sum | Self::Min | Self::Max - | Self::Quantile{..} | Self::TopK{..} | Self::Cardinality{..} - => DataModel::Any, - Self::Rate{..} | Self::Increase{..} | Self::QuantileOverTime{..} - => DataModel::TimeSeries, - } - } + pub fn requires(&self) -> DataModel { /* … */ } + /// Output column type — used by L3 schema derivation for `Aggregate`. + pub fn output_type(&self, input: &Field) -> DataType { /* … */ } + /// Which sketch families in the catalog can serve this intent. + /// Read by L4 binding rules. + pub fn candidate_sketches(&self) -> &'static [SketchKind] { /* … */ } } ``` -`AggIntent` describes **what** to compute (a quantile, a topk, a count) + what accuracy to hit — not **how** (KLL vs DDSketch vs CMS). Sketch binding happens at L4. +**Why no `TopK` intent?** `Sort + Limit` over `Aggregate { aggs: [Count] }` (or any ranking key) is the canonical heavy-hitters shape. Heavy-hitter sketches (CMS-with-heap, SpaceSaving) bind at L4 by pattern-matching that shape, not on a dedicated intent. Same canonical-form argument as the L3 enum cleanup. + +**Why no `QuantileOverTime` intent?** It duplicated `Quantile` over a `Window`. The window — its kind, its size, its slide — is fully captured by the surrounding `Window { … }` node; the quantile *operation* is the same regardless of whether the input was a windowed time-series or a row-grouped table. PromQL's `quantile_over_time(0.99, m[5m])` lowers cleanly to `Window{size=5m} → Aggregate{aggs:[Quantile{q=0.99}]}`. One intent halves the L4 rule surface (one bind rule per operation, not per language-flavor of an operation). + +**Why `Rate` and `Increase` survive that argument.** They are not "Sum / Count over a Window with a different name" — they include PromQL's counter-reset adjustment, which is a non-trivial transformation an exact `Sum` does not perform. They earn distinct intent variants because they parameterise different physical operators (delta-set aggregators bind on these intents directly). If a non-PromQL streaming language has the same notion (e.g. SQL `RATE() OVER (RANGE)`), it lowers to the same intent — the intent vocabulary names the operation, not the language. + +#### L4 sketch-bound IR — `SketchExpr` + +L4 binding rules consume L3 `QueryExpr` and produce `SketchExpr`. This is the IR L5 emitters consume. The two-IR split — pure-logical L3 (`QueryExpr`) and sketch-bound L4 (`SketchExpr`) — gives L4 rule application a clean type signature: `fn apply(&QueryExpr, &Constraints) -> Option`, and the boundary cannot be silently violated. + +```rust +pub enum SketchExpr { + /// Any logical L3 node passes through unchanged when no L4 rule rewrote + /// it — a `Filter` doesn't need a sketch counterpart. + Logical(QueryExpr), + + /// Sketch aggregation. L4 picked the sketch type and parameters from the + /// catalog given the `AggIntent` and `DeploymentConstraints`. + SketchAgg { + child: Box, + sketch: SketchKind, // Kll, Cms, Hll, DDSketch, CmsWithHeap, … + params: SketchParams, // catalog-validated + col: ColumnRef, + by: Vec, + }, + + /// Sketch-aware join (KMV / theta-sketch for join cardinality; + /// join-sample for join sampling). Emitted only when a `Bind*OnJoin` + /// rule fires — L3 always presents the logical `Join` for L4 to choose. + SketchJoin { + outer: Box, + inner: Box, + key: ColumnRef, + sketch: SketchKind, + params: SketchParams, + }, + + /// Subtract one sketch from another. Valid only for sketches with a + /// linear-inverse property (CMS, theta, count-based). Lets the planner + /// compute "all-A minus all-B" cardinality / count without re-scanning. + SketchSubtract { left: Box, right: Box }, + + /// Delete a key from a sketch (CMS update with -1, deletable Bloom + /// filter, …). Valid only for deletion-supporting sketches. + SketchDelete { sketch_input: Box, key: ColumnRef }, + + /// Read out a query result from a built sketch. Inverse of `SketchAgg`. + /// `query` says what to extract — quantile φ, count for key k, cardinality. + SketchEstimate { sketch_input: Box, query: SketchQuery }, + + /// ⊕ — union of sketches across stages / shards. Distinct from L3 + /// `Merge` because sketch union has type constraints (same family, + /// same params). L5 stage allocator emits this when distributing. + SketchMerge { children: Vec }, +} +``` + +The optimizer's job is to selectively replace logical aggregates / joins with their sketch-bound variants when a binding rule fires; everything else stays inside `SketchExpr::Logical(…)`. **Why a `Source` sum instead of two parallel `QueryExpr` trees:** most `QueryExpr` nodes (`Filter`, `Aggregate`) are data-model-agnostic — filter semantics are the same whether the input is a time-series window or a table scan. Only the leaf `Scan` differs. Keeping one tree with a polymorphic leaf means L4 rules like `BindKllOnQuantile` work uniformly across both data models; rules that care about data-model specifics (stage-aware push-down for TS; join-selectivity for tabular) gate on `source.data_model()` + `intent.requires()`. @@ -504,8 +655,40 @@ impl StageAllocator { ) -> Result, PlanError>; } -// sketch catalogue — what sketches exist, what params they accept -pub struct SketchCatalog { /* built at startup; queried by L4 binding rules + L5 */ } +// sketch catalogue — what sketches exist, what they support, what params they +// accept. Built at startup from the registered sketch backends; queried by L4 +// binding rules to map an `AggIntent` → `SketchKind` + `SketchParams`, and by +// L5 to instantiate the chosen sketch. +pub struct SketchCatalog { + pub entries: Vec, +} + +pub struct SketchEntry { + pub kind: SketchKind, // Kll, Cms, Hll, DDSketch, KMV, Theta, … + /// Which `AggIntent` variants this sketch can serve. + pub supported_intents: &'static [IntentTag], // Quantile, Count, Cardinality, JoinCardinality, … + /// Mergeability — sketches built on disjoint inputs combine without re-scan. + /// (CMS / HLL / KLL / theta = mergeable; SpaceSaving = approximately; + /// some heavy-hitter variants = no.) + pub mergeable: Mergeability, + /// Whether the sketch supports point deletion (CMS update with -1, + /// deletable Bloom filter) — gates `SketchExpr::SketchDelete`. + pub deletable: bool, + /// Whether the sketch admits a linear inverse (CMS, theta, count-based) + /// — gates `SketchExpr::SketchSubtract`. + pub subtractable: bool, + /// Accuracy / confidence model: error bounds as a function of params. + /// `(eps, delta)` for randomised sketches; absolute error for KLL; etc. + pub accuracy: AccuracyModel, + /// Aggregation keys this sketch supports natively. Some sketches are + /// keyed (CMS over (label, value)); others are unkeyed (HLL). + pub aggregated_keys: KeyShape, // Unkeyed | KeyedScalar | KeyedTuple + /// Parameter ranges + defaults. Catalog rejects out-of-range params at + /// L4 bind time so L5 never sees an unsupported configuration. + pub param_ranges: ParamRanges, + /// Memory + CPU model used by `CostModel`. Function of params. + pub cost_model: SketchCostModel, +} ``` Scenarios use these pieces: @@ -667,7 +850,7 @@ Each scenario crate is a library with: - **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. - **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. - **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. -- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::sketch_algebra::AggIntent` subset. +- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::sketch_algebra::AggIntent` subset. The mapping is not 1:1 — planner's `Topk` lowers to `Sort + Limit` over `Aggregate{aggs:[Count]}`, not to a dedicated intent (see §6 cleanup notes). - **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this scenario covers most sketch types) + a scenario-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. - **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. - **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `scenario-lifecycle` calls them when it needs to POST to ASAPQuery-backend. @@ -838,6 +1021,20 @@ This matters: a user who only wants `scenario-fusion` (e.g., an offline benchmar 11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP scenarios)?** Already handled — see §3 "Data-model support" and §6 `core::sketch_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP scenario picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. +### Resolved during the L3 IR cleanup (see §6 `core::sketch_algebra`) + +The following questions came up during review of the previous `QueryExpr` draft and are now settled. Listed here so the trail is visible: + +- **Output of `SketchAgg` vs `WindowedAgg`?** Neither node exists at L3 anymore. `SketchAgg` is an L4 sketch-bound node (`SketchExpr::SketchAgg`) emitted by binding rules; `WindowedAgg` was redundant with `Window` over `Aggregate` and removed. Output of `SketchAgg` is a sketch-typed column carrying the partial state; `SketchEstimate` reads it out into a scalar / vector. +- **Is `TopK` different from `Sort + Limit`?** No — same plan, two spellings. Removed; `Sort + Limit` is canonical, and heavy-hitter sketches bind on that shape at L4. +- **What is `JoinSketch`?** A sketch-aware join (KMV / theta / join-sample). Moved to L4 (`SketchExpr::SketchJoin`) so the choice "exact join vs sketch join" is an L4 cost decision against `Join`, not a competing L3 surface. +- **`HistogramQuantile` and `PromQLSubquery` feel out of place** — they were. Both removed from L3. `histogram_quantile` lowers in PromQL L1→L2 to bucket reads + a `Quantile` intent. `[range:resolution]` is a driver that expands into multiple queries during PromQL L1→L2 lowering, not a DAG node. +- **Sketch subtract / delete / estimate operators?** Added: `SketchExpr::SketchSubtract`, `SketchDelete`, `SketchEstimate`. Catalog flags (`subtractable`, `deletable`) gate which sketch families admit them. +- **What's `Dedup` exactly?** SQL `DISTINCT`. Renamed to `Distinct { cols }` and generalised from one column to N. +- **Why differentiate `Quantile` and `QuantileOverTime`?** No reason — the surrounding `Window` already encodes the temporal axis. `QuantileOverTime` removed from `AggIntent`. +- **`WindowedAgg` vs `Window` + `Agg`?** Equivalent; `WindowedAgg` removed. `WindowKind::{Tumbling, Sliding, Session}` lives on the `Window` node so the streaming-window kind is explicit; SQL analytic `OVER (...)` stays on the separate `WindowFunc` node. +- **Need input/output specs more fine-grained than `QueryExpr`?** Done: every L3 edge carries a typed `Schema` (fields + time index + unique-key sets), and §6 lists per-node input/output schemas. + ## 13. Future work ### Extending the primitive set beyond sketches From 88e48c1704c241bd69595c5134f7397b89907ff8 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Sat, 2 May 2026 12:31:21 -0400 Subject: [PATCH 02/10] =?UTF-8?q?docs:=20spelling=20=E2=80=94=20artefact?= =?UTF-8?q?=20->=20artifact?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 2 +- docs/design.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 583aa4d9..adac56cc 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ Because core now owns the L4/L5 infrastructure (not just L1-3), scenario crates - **Inside ASAPController workspace** (lockstep release with core, one-PR cross-scenario changes) - **In its own downstream repo** (independent release cadence, depends on published `asap-control-core`) -Both produce functionally identical artefacts. The placement is a deployment/team-ownership decision, not an architectural fork. Default: +Both produce functionally identical artifacts. The placement is a deployment/team-ownership decision, not an architectural fork. Default: - `scenario-lifecycle` + `scenario-query` in ASAPController workspace (share a YAML emitter). - `scenario-fusion` in the `asap-fusion` repo (research cadence, independent release). diff --git a/docs/design.md b/docs/design.md index 2e262680..c1ba328a 100644 --- a/docs/design.md +++ b/docs/design.md @@ -36,7 +36,7 @@ DataCollector/controller already documents its query→sketch translation as a 5 | 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/algebra/lower.rs`; asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | | 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `TopK` (use `Sort + Limit`), no `WindowedAgg` (use `Window` over `Aggregate`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | | 4 | **Sketch Optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 and emit sketch-bound L3+. ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library; scenarios **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | -| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artefact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; scenarios supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | +| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; scenarios supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | **The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every scenario reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same sketch-algebra IR (intent only, no sketch binding). @@ -101,7 +101,7 @@ The controller is a **planner**, not an executor. It does not run queries; it de | Output | Consumer | Wire shape | |---|---|---| | Per-executor sub-DAG assignment | The executor (edge agent / gateway / backend / DataFusion session) | OpAMP `RemoteConfig` (OTel YAML) / `streaming_config.yaml` POST / rewritten `LogicalPlan` | -| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge over Kafka, …) | Implied by the per-executor configs; not a separate artefact | +| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge over Kafka, …) | Implied by the per-executor configs; not a separate artifact | | Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | **Premise behind the per-executor split.** Every executor (edge collector, gateway, backend, DataFusion runtime) is in principle capable of running the entire query tree — they all speak the same physical operators. The controller's job is to decide *which* executor runs *which sub-tree* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by executor, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring; the `PhysicalPlanner` per scenario emits the per-executor configs. @@ -255,7 +255,7 @@ The language- and deployment-independent IR. **Pure intent at this layer**: no l #### Design rules for L3 1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Sort + Limit` is the canonical top-k shape; there is no separate `TopK`. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). -2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artefact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. +2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artifact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. 3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. 4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. 5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding tree. CTEs / let-bindings produce reuse edges (multiple consumers). @@ -372,7 +372,7 @@ impl Source { | `WindowedAgg { intent, window }` | Equivalent to `Window` over `Aggregate`; redundant. Tumbling/sliding kind moves onto `Window::kind` | `Window { kind: …, … } → Aggregate { aggs: [intent] }` | | `SketchAgg { intent, col }` | Sketch-bound; L3 must be intent-only | `Aggregate { aggs: [intent] }` at L3; L4 emits `SketchExpr::SketchAgg` | | `JoinSketch { outer, inner, key }` | Sketch-bound physical alternative; L3 must be intent-only. Several papers describe sketch-of-join (KMV, theta, join-sample) — picking one is an L4 cost decision, not an L3 surface choice | `Join { … }` at L3; L4 emits `SketchExpr::SketchJoin` when a `Bind*OnJoin` rule fires | -| `HistogramQuantile { phi }` | PromQL artefact — consumes Prometheus's specific bucketed-histogram format. Language-specific | Lowers in PromQL L1→L2 to bucket reads + `Aggregate { aggs: [Quantile{q: phi, …}] }` over the bucket counts | +| `HistogramQuantile { phi }` | PromQL artifact — consumes Prometheus's specific bucketed-histogram format. Language-specific | Lowers in PromQL L1→L2 to bucket reads + `Aggregate { aggs: [Quantile{q: phi, …}] }` over the bucket counts | | `PromQLSubquery { range, resolution }` | Driver construct — asks the engine to evaluate the inner expression at N timestamps; not a single DAG node | Expanded by PromQL L1→L2 lowering into a set of independent `QueryExpr` instances, not preserved at L3 | | `Dedup { col }` | Single-column-only spelling of SQL `DISTINCT` | Renamed to `Distinct { cols }`, generalised to N columns | @@ -893,7 +893,7 @@ Because core owns the L4/L5 infrastructure (not just L1-3), a scenario crate is - **Inside ASAPController workspace** — `crates/scenario-/`, lockstep release with core, cross-scenario changes are one PR. - **In their own downstream repo** — declares `asap-control-core` as a git-tagged dep, releases independently, owns its own CI. -Both produce functionally identical artefacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. +Both produce functionally identical artifacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. Default recommendation: - **scenario-lifecycle** and **scenario-query** in ASAPController (they share `scenario-query`'s YAML emitter via a workspace `path = "../scenario-query"` dep — trivial in-workspace). @@ -950,7 +950,7 @@ Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql- - Prometheus scrape format **Internal** (controller's own surface, still HTTP+JSON for now): -- `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artefacts (URIs). `/api/v1/plan` proxies to this. +- `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artifacts (URIs). `/api/v1/plan` proxies to this. - `GET /plans/:id` — plan inspection - `GET /status` — runtime + scenario health From 661ca8c2e5724ee445df214cb9053436746bf68f Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Sat, 2 May 2026 12:36:13 -0400 Subject: [PATCH 03/10] docs: bring TopK back as AggIntent; small text fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - TopK is a first-class AggIntent at L3 (heavy-hitter sketches like SpaceSaving / CMS-with-heap / Misra-Gries serve it as a single primitive). Generic QueryExpr::Sort + QueryExpr::Limit stays for non-heavy-hitter cases (ORDER BY name LIMIT 10). L1→L2→L3 lowering picks the intent form when it recognises a heavy-hitter pattern. - Clarify CTE / let-binding "reuse edges" wording: one producer + N Ref(name) consumer edges, which is what makes the IR a DAG. - Per-executor split: sub-tree -> sub-tree / sub-DAG. - Cut-edge transports: drop Kafka mention, replace with "sketch-merge / compute-from-raw over the precompute engine". - serialise -> serialize. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design.md | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/docs/design.md b/docs/design.md index c1ba328a..132abf10 100644 --- a/docs/design.md +++ b/docs/design.md @@ -34,7 +34,7 @@ DataCollector/controller already documents its query→sketch translation as a 5 |---|-------|--------------|-------------------| | 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql,mod}.rs`; asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | | 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/algebra/lower.rs`; asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | -| 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `TopK` (use `Sort + Limit`), no `WindowedAgg` (use `Window` over `Aggregate`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | +| 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | | 4 | **Sketch Optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 and emit sketch-bound L3+. ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library; scenarios **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | | 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; scenarios supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | @@ -55,7 +55,7 @@ Today: ### Intent vocabulary: DC's `AggIntent` is a superset; scenarios use subsets -DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because shapes that decompose into other operators (`TopK → Sort + Limit`; `QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core today is ~7 (`Count, Sum, Min, Max, Quantile, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps onto this set: `TopK` is no longer an intent — it lowers to `Sort + Limit` over `Aggregate{Count}` (or whatever ranking key), and the heavy-hitter sketch binds at L4 by matching that shape. Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that scenarios opt into. +DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because language-flavored synonyms (`QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core is 9 (`Count, Sum, Min, Max, Quantile, TopK, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps directly: `Topk` keeps its own intent (heavy-hitter sketches like SpaceSaving / CMS-with-heap compute it as a single primitive, so the intent earns L3 visibility). Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that scenarios opt into. ### Data-model support: both time-series and tabular @@ -101,14 +101,14 @@ The controller is a **planner**, not an executor. It does not run queries; it de | Output | Consumer | Wire shape | |---|---|---| | Per-executor sub-DAG assignment | The executor (edge agent / gateway / backend / DataFusion session) | OpAMP `RemoteConfig` (OTel YAML) / `streaming_config.yaml` POST / rewritten `LogicalPlan` | -| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge over Kafka, …) | Implied by the per-executor configs; not a separate artifact | +| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge / compute-from-raw over the precompute engine, …) | Implied by the per-executor configs; not a separate artifact | | Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | -**Premise behind the per-executor split.** Every executor (edge collector, gateway, backend, DataFusion runtime) is in principle capable of running the entire query tree — they all speak the same physical operators. The controller's job is to decide *which* executor runs *which sub-tree* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by executor, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring; the `PhysicalPlanner` per scenario emits the per-executor configs. +**Premise behind the per-executor split.** Every executor (edge collector, gateway, backend, DataFusion runtime) is in principle capable of running the entire query tree — they all speak the same physical operators. The controller's job is to decide *which* executor runs *which sub-tree / sub-DAG* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by executor, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring; the `PhysicalPlanner` per scenario emits the per-executor configs. This is what makes the topology a *parameter* rather than an axis of code: the same `SketchExpr` plus a different `TopologyDescriptor` produces edge-only / 1-stage / 3-stage / 0-stage placements without rewriting the plan. -**Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialise into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. +**Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialize into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. This drives the core/scenario split: @@ -254,11 +254,11 @@ The language- and deployment-independent IR. **Pure intent at this layer**: no l #### Design rules for L3 -1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Sort + Limit` is the canonical top-k shape; there is no separate `TopK`. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). +1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). The exception is when an "intent" is its own physical primitive: heavy-hitter top-k is served by sketches (SpaceSaving, CMS-with-heap) as a single operation, so `AggIntent::TopK` is a first-class intent at L3 — distinct from the generic `Sort + Limit` operator pair, which still appears in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). 2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artifact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. 3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. 4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. -5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding tree. CTEs / let-bindings produce reuse edges (multiple consumers). +5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding tree. SQL CTEs (`WITH name AS (expr) SELECT ... FROM name JOIN name AS n2 ...`) and PromQL recording rules let a single named sub-expression be referenced from N downstream operators; the L3 representation is one producer node with N `QueryExpr::Ref(name)` consumer edges pointing back at it. That fan-out — multiple consumers of one producer — is what makes the IR a DAG rather than a tree. L4 reuse rules and `CostModel::workload_cost` use it to credit the producer's build cost once across all consumers. ```rust pub enum QueryExpr { @@ -283,7 +283,7 @@ pub enum QueryExpr { // ── Time / streaming windows ────────────────────────────────────────── /// ψ — tumbling / sliding / session window over the time axis. Defines - /// the lifecycle (flush / reset bounds) of any aggregate in its subtree. + /// the lifecycle (flush / reset bounds) of any aggregate in its sub-tree / sub-DAG. /// PromQL `[5m]` and streaming windows lower here. SQL `OVER (...)` /// analytic frames are a different node — see `WindowFunc` below. Window { child: Box, kind: WindowKind, @@ -313,12 +313,14 @@ pub enum QueryExpr { left: Box, right: Box }, // ── Ordering & limiting ─────────────────────────────────────────────── + /// Generic order-by — survives L3 for non-heavy-hitter cases + /// (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`). Sort { child: Box, keys: Vec }, - /// `LIMIT n OFFSET k`. `Sort` followed by `Limit` is the canonical - /// top-k / heavy-hitters shape — no separate `TopK` node, since one - /// canonical representation per plan keeps L4 rule matching unambiguous. - /// Heavy-hitter sketches (CMS-with-heap, SpaceSaving) bind at L4 by - /// pattern-matching `Sort + Limit` on top of `Aggregate`. + /// `LIMIT n OFFSET k`. The heavy-hitter shape (`ORDER BY count DESC + /// LIMIT k`, PromQL `topk(k, …)`) is recognised at L1→L2→L3 lowering + /// and produces `AggIntent::TopK` rather than generic `Sort + Limit`, + /// so heavy-hitter sketches (SpaceSaving, CMS-with-heap) bind on the + /// intent. Generic `Sort + Limit` flows through unchanged. Limit { child: Box, n: u64, offset: u64 }, // ── Subquery / CTE ──────────────────────────────────────────────────── @@ -368,7 +370,7 @@ impl Source { | Removed | Reason | Replacement | |---|---|---| -| `TopK { k, by }` | Decomposes into `Sort + Limit`; two ways to spell the same plan break canonical-form invariant | `Sort { keys: by } → Limit { n: k }` | +| `TopK { k, by }` *as a `QueryExpr` node* | A `QueryExpr`-level top-k operator collapses two distinct concepts: (a) the *intent* of "compute heavy hitters", which has its own sketch primitive, and (b) the generic operator pair `Sort + Limit`, which doesn't. Splitting them puts heavy-hitter logic at the right level | Heavy-hitter intent → `AggIntent::TopK` (L3, recognised by L1→L2→L3 lowering of `ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`); generic ordering+limit → `Sort + Limit` `QueryExpr` nodes (unchanged). Both retained — they describe different things | | `WindowedAgg { intent, window }` | Equivalent to `Window` over `Aggregate`; redundant. Tumbling/sliding kind moves onto `Window::kind` | `Window { kind: …, … } → Aggregate { aggs: [intent] }` | | `SketchAgg { intent, col }` | Sketch-bound; L3 must be intent-only | `Aggregate { aggs: [intent] }` at L3; L4 emits `SketchExpr::SketchAgg` | | `JoinSketch { outer, inner, key }` | Sketch-bound physical alternative; L3 must be intent-only. Several papers describe sketch-of-join (KMV, theta, join-sample) — picking one is an L4 cost decision, not an L3 surface choice | `Join { … }` at L3; L4 emits `SketchExpr::SketchJoin` when a `Bind*OnJoin` rule fires | @@ -445,6 +447,13 @@ pub enum AggIntent { Sum, Min, Max, Quantile { q: f64, accuracy: AccuracyTarget }, + /// Heavy-hitter top-k. Distinct from generic `Sort + Limit` because a + /// dedicated sketch primitive (SpaceSaving, CMS-with-heap, Misra-Gries) + /// computes it as a single operation. L1→L2→L3 lowering produces this + /// when it recognises a heavy-hitter shape (`ORDER BY count DESC LIMIT k`, + /// PromQL `topk(k, …)`); other ordering+limit cases stay as + /// `QueryExpr::Sort + QueryExpr::Limit`. + TopK { k: usize, by: Vec, accuracy: AccuracyTarget }, Cardinality { accuracy: AccuracyTarget }, // Time-series streaming derivatives — specific operations, not just @@ -473,7 +482,7 @@ impl AggIntent { } ``` -**Why no `TopK` intent?** `Sort + Limit` over `Aggregate { aggs: [Count] }` (or any ranking key) is the canonical heavy-hitters shape. Heavy-hitter sketches (CMS-with-heap, SpaceSaving) bind at L4 by pattern-matching that shape, not on a dedicated intent. Same canonical-form argument as the L3 enum cleanup. +**Why `TopK` *is* an intent (and `Sort + Limit` is not collapsed into it).** Heavy-hitter top-k has a dedicated sketch primitive — SpaceSaving / CMS-with-heap / Misra-Gries compute it in a single pass with sub-linear memory. L4 binding rules want to fire on the *intent* "give me the top-k frequent items" rather than on a syntactic shape, the same argument that makes `Quantile` an intent rather than a `Sort` + "pick the φ-th element" pattern. So `AggIntent::TopK` lives at L3. Generic `QueryExpr::Sort + QueryExpr::Limit` *also* survives, because not every order-by-limit query is heavy-hitter (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`); these have no sketch alternative and stay as generic operators. The canonical-form invariant is preserved because L1→L2→L3 lowering picks one or the other deterministically based on whether it recognises a heavy-hitter pattern. **Why no `QuantileOverTime` intent?** It duplicated `Quantile` over a `Window`. The window — its kind, its size, its slide — is fully captured by the surrounding `Window { … }` node; the quantile *operation* is the same regardless of whether the input was a windowed time-series or a row-grouped table. PromQL's `quantile_over_time(0.99, m[5m])` lowers cleanly to `Window{size=5m} → Aggregate{aggs:[Quantile{q=0.99}]}`. One intent halves the L4 rule surface (one bind rule per operation, not per language-flavor of an operation). @@ -850,7 +859,7 @@ Each scenario crate is a library with: - **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. - **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. - **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. -- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::sketch_algebra::AggIntent` subset. The mapping is not 1:1 — planner's `Topk` lowers to `Sort + Limit` over `Aggregate{aggs:[Count]}`, not to a dedicated intent (see §6 cleanup notes). +- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::sketch_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). - **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this scenario covers most sketch types) + a scenario-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. - **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. - **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `scenario-lifecycle` calls them when it needs to POST to ASAPQuery-backend. @@ -1026,7 +1035,7 @@ This matters: a user who only wants `scenario-fusion` (e.g., an offline benchmar The following questions came up during review of the previous `QueryExpr` draft and are now settled. Listed here so the trail is visible: - **Output of `SketchAgg` vs `WindowedAgg`?** Neither node exists at L3 anymore. `SketchAgg` is an L4 sketch-bound node (`SketchExpr::SketchAgg`) emitted by binding rules; `WindowedAgg` was redundant with `Window` over `Aggregate` and removed. Output of `SketchAgg` is a sketch-typed column carrying the partial state; `SketchEstimate` reads it out into a scalar / vector. -- **Is `TopK` different from `Sort + Limit`?** No — same plan, two spellings. Removed; `Sort + Limit` is canonical, and heavy-hitter sketches bind on that shape at L4. +- **Is `TopK` different from `Sort + Limit`?** They overlap on heavy-hitter queries but are different concepts. `TopK` is now an `AggIntent` at L3 (heavy-hitter intent — SpaceSaving / CMS-with-heap / Misra-Gries serve it as a single primitive), while generic `Sort + Limit` survives as `QueryExpr` operators for non-heavy-hitter cases (`ORDER BY name LIMIT 10`). L1→L2→L3 lowering picks the intent form when it recognises a heavy-hitter pattern (`ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`) and the operator form otherwise. - **What is `JoinSketch`?** A sketch-aware join (KMV / theta / join-sample). Moved to L4 (`SketchExpr::SketchJoin`) so the choice "exact join vs sketch join" is an L4 cost decision against `Join`, not a competing L3 surface. - **`HistogramQuantile` and `PromQLSubquery` feel out of place** — they were. Both removed from L3. `histogram_quantile` lowers in PromQL L1→L2 to bucket reads + a `Quantile` intent. `[range:resolution]` is a driver that expands into multiple queries during PromQL L1→L2 lowering, not a DAG node. - **Sketch subtract / delete / estimate operators?** Added: `SketchExpr::SketchSubtract`, `SketchDelete`, `SketchEstimate`. Catalog flags (`subtractable`, `deletable`) gate which sketch families admit them. From 22538983dfdb2636eccc291c3c83b903d0716efc Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Sat, 2 May 2026 12:40:59 -0400 Subject: [PATCH 04/10] =?UTF-8?q?docs:=20tighten=20DAG-vs-tree=20framing?= =?UTF-8?q?=20=E2=80=94=20multiple=20parents=20share=20state?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reframe "DAG, not tree" around the operational distinction: a producer node can have multiple parents that share its precomputed intermediate state. List the three fan-in sources: explicit CTE / recording-rule references, L4 reuse rules introducing shared producers across queries, and L4 stage / shard structure. The cost-model implication (credit build cost once across consumers) is what makes the DAG shape a Pareto win over a tree. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/design.md b/docs/design.md index 132abf10..739ed34d 100644 --- a/docs/design.md +++ b/docs/design.md @@ -258,7 +258,12 @@ The language- and deployment-independent IR. **Pure intent at this layer**: no l 2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artifact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. 3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. 4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. -5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding tree. SQL CTEs (`WITH name AS (expr) SELECT ... FROM name JOIN name AS n2 ...`) and PromQL recording rules let a single named sub-expression be referenced from N downstream operators; the L3 representation is one producer node with N `QueryExpr::Ref(name)` consumer edges pointing back at it. That fan-out — multiple consumers of one producer — is what makes the IR a DAG rather than a tree. L4 reuse rules and `CostModel::workload_cost` use it to credit the producer's build cost once across all consumers. +5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding context. The shape is a DAG, not a tree, because **a producer node can have multiple parents that share its precomputed intermediate state** — instead of recomputing the same sub-expression once per consumer, the consumers fan in to a single node and read its output. Three sources of fan-in: + 1. **Explicit, in-query.** SQL CTEs (`WITH name AS (expr) SELECT ... FROM name JOIN name AS n2 ON ...`) and PromQL recording rules name a sub-expression and reference it N times; each reference becomes a `QueryExpr::Ref(name)` parent of the named producer. + 2. **L4 reuse rules.** When two queries planned together both need (e.g.) a p99 quantile of the same series, the optimizer can introduce a shared sketch node whose output feeds both — even though neither query author wrote a CTE. + 3. **L4 stage / shard structure.** A pre-aggregate computed once on the edge can feed multiple downstream gateway-stage operators. + + `CostModel::workload_cost` credits a shared producer's build cost once across all consumers, which is what makes reuse a Pareto win. Tree IRs lose this — they have to duplicate the producer for every consumer. ```rust pub enum QueryExpr { From 5ef19da10f38ebb542a1c5f1483827c46bcf00d4 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Sat, 2 May 2026 12:48:57 -0400 Subject: [PATCH 05/10] docs: make per-node I/O table self-contained Drop the S / L / R / S_i shorthand; spell out each input and output in plain language so every row reads independently. Also split Sort and Limit into separate rows now that each gets its own constraint description (Sort requires keys' fields be present in input; Limit has no field-level constraint). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design.md | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/docs/design.md b/docs/design.md index 739ed34d..cdf6358b 100644 --- a/docs/design.md +++ b/docs/design.md @@ -410,26 +410,27 @@ pub trait HasSchema { } ``` -Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/sketch_algebra/schema.rs`): +Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/sketch_algebra/schema.rs`). Each row reads independently: every column position, type, and constraint is named explicitly rather than carried by a shorthand like `S` or `L` / `R`. -| Node | Inputs | Output | +| Node | Input schemas | Output schema | |---|---|---| -| `Scan { source }` | — | `source.schema(catalog)` | -| `Ref(name)` | — | resolved from `LetBinding` named `name` | -| `Filter { pred }` | one `S` | `S` (predicate is a refinement; no schema change) | -| `Project { cols }` | one `S` | projection of `S` to `cols` | -| `Aggregate { by, aggs }` | one `S` | `by` columns + one column per `agg` (named & typed by `AggIntent::output_type(input_field)`) | -| `Window { kind, size, slide }` | one `S` (must have `time_index`) | `S` extended with window-id / window-bounds metadata fields | -| `Partition { keys }` | one `S` | `S` (logical-only; physical sharding hint for L5) | -| `Distinct { cols }` | one `S` | `S` with `unique_keys` tightened to include `cols` | -| `Merge` | N (all `S_i`); rule: schemas must be union-compatible | `S_0` (representative) | -| `Join { kind, pred }` | two `L`, `R` | union of `L` + `R` columns minus duplicates removed by USING/NATURAL | -| `SetOp { kind, all }` | two (must be union-compatible) | left's schema | -| `Sort { keys }` / `Limit` | one `S` | `S` | -| `Subquery { alias }` | one `S` | `S` with table alias | -| `LetBinding { name, expr, body }` | `body` consumes; `expr` bound by name | body's output schema | -| `WindowFunc { func, partition_by, order_by, frame }` | one `S` | `S` extended with the analytic-function output column | -| `BinaryOp { op, vector_match }` | two `L`, `R` (PromQL vector-match constraints apply) | element-wise op → schema with op-typed value column; boolean → boolean column | +| `Scan { source }` | none — leaf node | `source.schema(catalog)` — derived from the source's catalog metadata (TimeSeries metric labels + value + timestamp; or Table columns from `information_schema`; or recursive `Source::Join`) | +| `Ref(name)` | none — pointer node | the output schema of the `LetBinding` whose `name` matches; resolved at plan time | +| `Filter { pred }` | one input schema (the child's output) | the child's input schema unchanged — `Filter` is a row-level refinement; no columns added, removed, or re-typed | +| `Project { cols }` | one input schema | the input schema projected to `cols` — fields filtered and reordered to match `cols`; `time_index` and `unique_keys` carried over for retained columns | +| `Aggregate { by, aggs }` | one input schema | the `by` columns (carried verbatim from input) followed by one new column per entry in `aggs`, each named and typed by `AggIntent::output_type(input_field)`; `unique_keys = [by]` | +| `Window { kind, size, slide }` | one input schema, **must** contain a `time_index` field | the input schema extended with synthetic `window_id` and `window_start` / `window_end` metadata fields | +| `Partition { keys }` | one input schema | the input schema unchanged — logical-only marker; carries a sharding hint for L5's stage allocator | +| `Distinct { cols }` | one input schema | the input schema with `unique_keys` tightened to include `cols`; field types unchanged | +| `Merge` | N input schemas, all union-compatible (same field names, same types, same nullability, same `time_index` position if any) | the first input's schema (representative; checked union-compatible with the rest) | +| `Join { kind, pred }` | two input schemas — left and right children | the concatenation of left's fields and right's fields, minus columns that USING / NATURAL deduplicates; nullability widened on the OUTER side for outer joins | +| `SetOp { kind, all }` | two input schemas, must be union-compatible (as for `Merge`) | the left input's schema | +| `Sort { keys }` | one input schema; every field referenced in `keys` must be present in it | the input schema unchanged | +| `Limit { n, offset }` | one input schema | the input schema unchanged | +| `Subquery { alias }` | one input schema | the input schema with `alias` applied as the table alias to all field names | +| `LetBinding { name, expr, body }` | `expr` produces an intermediate schema bound to `name`; `body` consumes it (and any other in-scope bindings) | the `body`'s output schema | +| `WindowFunc { func, partition_by, order_by, frame }` | one input schema; every field in `partition_by` / `order_by` must be present | the input schema extended with one new column carrying the analytic-function output (named after `func`, typed per `func`) | +| `BinaryOp { op, vector_match }` | two input schemas — left and right operands. PromQL vector-match constraints (`on`/`ignoring` + `group_left`/`group_right`) govern label-set compatibility | for arithmetic/comparison `op`: the left input's schema with the value column re-typed to the result of `op`; for boolean `op` (`and`, `or`, `unless`): the left input's schema with a boolean value column | #### Three distinct schemas — DAG vs DB vs sketch catalog From 5ef0547a7c68a3eb6f9cde0364200ffa5e134465 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Sat, 2 May 2026 12:52:40 -0400 Subject: [PATCH 06/10] docs: per-node input/output spec for SketchExpr (L4 IR) Adds a self-contained table covering the seven SketchExpr variants (Logical pass-through, SketchAgg, SketchJoin, SketchSubtract, SketchDelete, SketchEstimate, SketchMerge) with input and output schemas, and introduces DataType::Sketch(SketchKind, SketchParams) as the L4 field type that carries sketch state. Two type-system invariants spelled out: sketch-family mismatch is a plan-time error, and catalog capability flags (subtractable, deletable, mergeable) gate which nodes can fire. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/design.md b/docs/design.md index cdf6358b..ace6ba4b 100644 --- a/docs/design.md +++ b/docs/design.md @@ -547,6 +547,36 @@ pub enum SketchExpr { The optimizer's job is to selectively replace logical aggregates / joins with their sketch-bound variants when a binding rule fires; everything else stays inside `SketchExpr::Logical(…)`. +##### Per-node input/output spec for `SketchExpr` + +L4 introduces a new field type into `Schema`: + +```rust +pub enum DataType { + // … the L3 types (Int64, Float64, Utf8, Map<…>, …) … + /// Sketch state. Carries the sketch family + params so the type system + /// rejects merges of incompatible sketches at plan time. + Sketch(SketchKind, SketchParams), +} +``` + +A "sketch-state schema" is a regular `Schema` whose value-bearing field has a `DataType::Sketch(...)` dtype. Reading rules: + +| Node | Input schemas | Output schema | +|---|---|---| +| `Logical(qe)` | whatever the inner L3 node `qe` consumes (per the L3 table above) | whatever `qe` produces — straight pass-through | +| `SketchAgg { child, sketch, params, col, by }` | one input schema; must contain `col` (the column being summarised) and every field referenced in `by` (group keys) | the `by` columns carried over verbatim, followed by one synthetic field of dtype `Sketch(sketch, params)` carrying the partial sketch state per group; `unique_keys = [by]` | +| `SketchJoin { outer, inner, key, sketch, params }` | two input schemas (outer + inner); both must contain `key` with compatible types | one field of dtype `Sketch(sketch, params)` carrying the join-cardinality / join-sample state — read out by a downstream `SketchEstimate` | +| `SketchSubtract { left, right }` | two input schemas, each with exactly one `Sketch(s, p)` field; **`s` and `p` must match** between the two inputs (catalog rejects mismatches at plan time); the `s` family must have `subtractable = true` in the catalog | one `Sketch(s, p)` field carrying the subtracted state (same family + params as inputs) | +| `SketchDelete { sketch_input, key }` | one input schema with a `Sketch(s, p)` field whose catalog entry has `deletable = true`; plus the `key` column to delete | the input schema unchanged in type — sketch state is mutated logically (the key's contribution removed) but the field's `(sketch, params)` signature is preserved | +| `SketchEstimate { sketch_input, query }` | one input schema with a `Sketch(s, p)` field; `query` (e.g. `Quantile(φ)`, `PointCount(k)`, `Cardinality`) must appear in the catalog entry's `supported_intents` for `s` | a regular row-shaped schema carrying the answer — `Float64` for quantile, `Int64` for count / cardinality, an array of `(key, count)` for top-k. The `Sketch(...)` field type does *not* propagate downstream of an Estimate | +| `SketchMerge { children }` | N input schemas, each with a `Sketch(s, p)` field; **all N must agree on `(s, p)`**; the `s` family must have `mergeable = true` in the catalog | one `Sketch(s, p)` field carrying the unioned state (same family + params as inputs) | + +Two type-system invariants make L4 robust: + +1. **Sketch-family mismatch is a plan-time error.** `SketchSubtract` over `Sketch(KLL, …)` and `Sketch(CMS, …)` fails type-checking before L5 ever sees it. +2. **Catalog capability flags gate which nodes can fire.** `SketchSubtract` requires `subtractable`, `SketchDelete` requires `deletable`, `SketchMerge` requires `mergeable`. The catalog (see §6 `core::physical::sketch_catalog`) is the single source of truth for these flags; binding rules consult it before producing the node. + **Why a `Source` sum instead of two parallel `QueryExpr` trees:** most `QueryExpr` nodes (`Filter`, `Aggregate`) are data-model-agnostic — filter semantics are the same whether the input is a time-series window or a table scan. Only the leaf `Scan` differs. Keeping one tree with a polymorphic leaf means L4 rules like `BindKllOnQuantile` work uniformly across both data models; rules that care about data-model specifics (stage-aware push-down for TS; join-selectivity for tabular) gate on `source.data_model()` + `intent.requires()`. **Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values. That stream can come from a time-series window (`Source::TimeSeries`) or a table column (`Source::Table`); the sketch doesn't know or care. So `BindKllOnQuantile` works identically regardless of `Source`. From 4a55e0ba1bf976c18ce686be07c7c5d569702774 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 4 May 2026 17:02:09 -0400 Subject: [PATCH 07/10] =?UTF-8?q?docs:=20rename=20scenario=E2=86=92deploym?= =?UTF-8?q?ent=20model;=20introduce=20Executor=20type;=20add=20glossary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename scenario→deployment model across all three docs. Crate names get an asap- prefix (deployment-model-asaplifecycle / -asapquery / -asapfusion). "Scenario" understated the bundle; "deployment model" names the choice — rule selection + L5 topology + emitter all reflect deployment constraints. - Distinguish stage from executor as first-class concepts. Stage (StageId) is a categorical lifecycle tier; Executor is a concrete runtime instance occupying a stage with its own id / address / capabilities. StageAllocator stays stage-granularity; PhysicalPlanner fans out per executor via DeploymentConstraints::executors(). - Rename §6 "Three distinct schemas" → "DAG schema, DB schema, sketch catalog — three distinct metadata sources" and clarify the sketch catalog is a registry of available primitives, not a schema. - Add §14 glossary covering architecture roles, layer drivers, IR by layer, metadata sources, and workload identity. Renumber Success criteria to §15. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 44 ++--- docs/design.md | 374 +++++++++++++++++++++++++---------------- docs/migration-plan.md | 124 +++++++------- 3 files changed, 309 insertions(+), 233 deletions(-) diff --git a/README.md b/README.md index adac56cc..5ca75ad7 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Unified control plane for the ASAP data-lifecycle stack. -This repo merges three previously-separate projects into a single workspace with a shared core and per-scenario plugin crates: +This repo merges three previously-separate projects into a single workspace with a shared core and per-deployment-model plugin crates: - `DataCollector/controller` — end-to-end lifecycle planner (collection → transmission → storage → analytics). - `ASAPQuery[-backend]/asap-planner-rs` — analytics-query-only planner (CLI, YAML in → YAML out). @@ -24,10 +24,10 @@ This repo merges three previously-separate projects into a single workspace with | 1 | Query-language parsing (PromQL, SQL, DataFusion, ElasticDSL) | `core::query_language` | | 2 | Per-language logical plan (relational algebra tree) | `core::logical_plan` | | 3 | Sketch algebra — language-, deployment-, AND data-model-independent IR (`QueryExpr` + `AggIntent`, intent only). Supports both time-series and tabular data via a `Source` sum inside `Scan`. | `core::sketch_algebra` | -| 4 | Cost-aware optimizer — rule engine + shared rule library; scenarios pick rules | **framework in `core::optimizer`**; choices in each scenario | -| 5 | Physical plan — stage allocation + emit to wire format | **framework in `core::physical`**; topology + emitter in each scenario | +| 4 | Cost-aware optimizer — rule engine + shared rule library; deployment models pick rules | **framework in `core::optimizer`**; choices in each deployment model | +| 5 | Physical plan — stage allocation + emit to wire format | **framework in `core::physical`**; topology + emitter in each deployment model | -**Core owns the shared infrastructure across all 5 layers**, not just L1-3. Scenarios are thin: they pick which rules fire (L4), declare their deployment topology (L5), and provide an emitter for their output format. Typical scenario crate: **500-2500 LOC**. +**Core owns the shared infrastructure across all 5 layers**, not just L1-3. Deployment models are thin: they pick which rules fire (L4), declare their deployment topology (L5), and provide an emitter for their output format. Typical deployment model crate: **500-2500 LOC**. ``` crates/ @@ -38,21 +38,21 @@ crates/ lower/ # L1→L2→L3 passes optimizer/ # L4 framework — rule engine + shared rule library + cost traits physical/ # L5 framework — PhysicalPlanner + stage allocator + topology + sketch catalogue - pipeline/ # L1→…→L5 driver, parameterized on scenario + pipeline/ # L1→…→L5 driver, parameterized on deployment model runtime/ # HTTP / OpAMP / replanner / store — service skeleton - scenario-lifecycle # DC-specific rules + 3-stage topology + OTel/backend emitters - scenario-query # precompute-engine rules + 1-stage + StreamingConfig/InferenceConfig YAML emitters - scenario-fusion # DataFusion rewrites + 0-stage (in-process) + LogicalPlan emit + deployment-model-asaplifecycle # DC-specific rules + 3-stage topology + OTel/backend emitters + deployment-model-asapquery # precompute-engine rules + 1-stage + StreamingConfig/InferenceConfig YAML emitters + deployment-model-asapfusion # DataFusion rewrites + 0-stage (in-process) + LogicalPlan emit control-proto # OpAMP proto + internal proto (tonic/prost) testing # shared fixtures bin/ - asap-controller # long-running service, all scenarios - asap-query # one-shot CLI (scenario-query only) - asap-lifecycle # OPTIONAL: standalone service with only scenario-lifecycle - asap-fusion-bench # OPTIONAL: benchmark harness over scenario-fusion + asap-controller # long-running service, all deployment models + asap-query # one-shot CLI (deployment-model-asapquery only) + asap-lifecycle # OPTIONAL: standalone service with only deployment-model-asaplifecycle + asap-fusion-bench # OPTIONAL: benchmark harness over deployment-model-asapfusion ``` -A new scenario lands by adding one crate with `rules.rs` (pick L4 rules) + `topology.rs` + an emitter, plus one line in `bin/asap-controller/main.rs`. No changes to core or runtime. +A new deployment model lands by adding one crate with `rules.rs` (pick L4 rules) + `topology.rs` + an emitter, plus one line in `bin/asap-controller/main.rs`. No changes to core or runtime. ## Consumption modes @@ -60,7 +60,7 @@ Three ways downstream can use ASAPController — mix as needed: | Mode | Use case | How | |---|---|---| -| **Rust library** | In-process use of the IR or a specific scenario (e.g. asap-fusion benchmarks) | `Cargo.toml`: `asap-control-core = { git = "...", tag = "v0.1.0" }` or any individual `scenario-*` crate. Per-crate dep isolation keeps dep trees small (scenario-fusion pulls DataFusion; scenario-query pulls PromQL/SQL parsers; neither pulls axum/OpAMP). | +| **Rust library** | In-process use of the IR or a specific deployment model (e.g. asap-fusion benchmarks) | `Cargo.toml`: `asap-control-core = { git = "...", tag = "v0.1.0" }` or any individual `deployment-model-*` crate. Per-crate dep isolation keeps dep trees small (deployment-model-asapfusion pulls DataFusion; deployment-model-asapquery pulls PromQL/SQL parsers; neither pulls axum/OpAMP). | | **HTTP service sidecar** | Production control plane — e.g. ASAPQuery-backend POSTing QuerySpec on capability-miss | Run `asap-controller` binary, POST to `/api/v1/plan`. Same contract DC controller speaks today. | | **CLI / Docker image** | One-shot init container (e.g. docker-compose init job that writes `streaming_config.yaml`) | `asap-controller plan --workload ... --output-dir ...` or the dedicated `asap-query` binary | @@ -68,23 +68,23 @@ Three ways downstream can use ASAPController — mix as needed: ASAPController is the **control plane only**. The data plane — OTel collectors, ASAPQuery-backend's query engine, asap-fusion users' DataFusion runtimes — stays in its original repo. Communication is always over wire (OpAMP, HTTP, Prometheus scrape). This boundary is unchanged by the merger. -## Scenario placement: flexible +## Deployment model placement: flexible -Because core now owns the L4/L5 infrastructure (not just L1-3), scenario crates are small and largely self-contained. A scenario can live either: +Because core now owns the L4/L5 infrastructure (not just L1-3), deployment model crates are small and largely self-contained. A deployment model can live either: -- **Inside ASAPController workspace** (lockstep release with core, one-PR cross-scenario changes) +- **Inside ASAPController workspace** (lockstep release with core, one-PR cross-deployment-model changes) - **In its own downstream repo** (independent release cadence, depends on published `asap-control-core`) Both produce functionally identical artifacts. The placement is a deployment/team-ownership decision, not an architectural fork. Default: -- `scenario-lifecycle` + `scenario-query` in ASAPController workspace (share a YAML emitter). -- `scenario-fusion` in the `asap-fusion` repo (research cadence, independent release). +- `deployment-model-asaplifecycle` + `deployment-model-asapquery` in ASAPController workspace (share a YAML emitter). +- `deployment-model-asapfusion` in the `asap-fusion` repo (research cadence, independent release). ## Key design references - 5-layer pipeline: `docs/design.md` §3 - Core internals (L1-3 + L4 framework + L5 framework): `docs/design.md` §6 -- Scenario internals (each's rules + topology + emitter): `docs/design.md` §8 -- Extension point for future scenarios: `docs/design.md` §9 +- Deployment model internals (each's rules + topology + emitter): `docs/design.md` §8 +- Extension point for future deployment models: `docs/design.md` §9 - Consumption modes + dependency isolation: `docs/design.md` §11 -- Open questions (incl. L2-tree contract, scenario placement): `docs/design.md` §12 +- Open questions (incl. L2-tree contract, deployment model placement): `docs/design.md` §12 - Phase-by-phase migration: `docs/migration-plan.md` diff --git a/docs/design.md b/docs/design.md index ace6ba4b..4b767ded 100644 --- a/docs/design.md +++ b/docs/design.md @@ -11,8 +11,8 @@ Merges three existing codebases into one: ## 1. Goals 1. **One repo, one workspace.** One `Cargo.toml` at root, one place to file issues, one release cadence. -2. **Common core, pluggable scenarios.** The three existing codebases each solve a *different* planning problem. Factor out what's actually shared; keep the scenario-specific parts as crates so each scenario can evolve independently. -3. **Extensible for future scenarios** (4th, 5th, …). A new scenario should land as a new crate that implements well-defined traits — not by touching the core or the runtime. +2. **Common core, pluggable deployment models.** The three existing codebases each solve a *different* planning problem. Factor out what's actually shared; keep the deployment-model-specific parts as crates so each deployment model can evolve independently. +3. **Extensible for future deployment models** (4th, 5th, …). A new deployment model should land as a new crate that implements well-defined traits — not by touching the core or the runtime. 4. **Preserve the two deployment shapes:** - **Service**: long-running HTTP+OpAMP process that accepts live `QuerySpec`s, replans on SLA violations, pushes configs to agents/backends. (DC controller today.) - **CLI**: one-shot "read workload YAML → emit `streaming_config.yaml` + `inference_config.yaml`". (`asap-planner-rs` today.) @@ -23,7 +23,7 @@ Merges three existing codebases into one: ## 2. Non-goals - **Not** redesigning the `Plan` IR end-to-end. DC's `algebra/` + fusion's `translator/optimizer/executor` stay where they are semantically; we merge them into a shared `Plan` crate with a clean trait seam, not a green-field rewrite. -- **Not** unifying DataFusion's `LogicalPlan` with DC's custom algebra at the type level. Those are different universes. Scenarios own their IR; core owns the *staged dispatch contract*. +- **Not** unifying DataFusion's `LogicalPlan` with DC's custom algebra at the type level. Those are different universes. Deployment models own their IR; core owns the *staged dispatch contract*. - **Not** in-scope: changing the existing wire protocol between controller and backend. ASAPQuery-backend keeps consuming `streaming_config.yaml` and keeps responding on `/api/v1/plan`. ## 3. The organizing spine: DC controller's 5-layer pipeline @@ -35,12 +35,12 @@ DataCollector/controller already documents its query→sketch translation as a 5 | 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql,mod}.rs`; asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | | 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/algebra/lower.rs`; asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | | 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | -| 4 | **Sketch Optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 and emit sketch-bound L3+. ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library; scenarios **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | -| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; scenarios supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | +| 4 | **Sketch Optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 and emit sketch-bound L3+. ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | +| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; deployment models supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | -**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every scenario reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same sketch-algebra IR (intent only, no sketch binding). +**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every deployment model reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same sketch-algebra IR (intent only, no sketch binding). -**L4 and L5 also have substantial common infrastructure.** Initially we assumed scenarios owned L4/L5 wholesale; on closer inspection what's actually scenario-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes scenarios significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. +**L4 and L5 also have substantial common infrastructure.** Initially we assumed deployment models owned L4/L5 wholesale; on closer inspection what's actually deployment-model-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes deployment models significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. ### Sketch binding lives in L4, not L3 @@ -53,23 +53,23 @@ Today: - asap-fusion: correctly separated — `SketchConfigRule` at L4 fills `SketchConfig::NULL` with concrete `CountMinSketch{5,4096}` / `KLL{k=200,m=8}`. - asap-planner-rs: **L3+L4 fused today**. `map_statistic_to_precompute_operator` jumps from `Statistic` straight to `AggregationType::DatasketchesKLL{k=200}` in one call. **Phase 4 splits this** — `Statistic → AggIntent` at L3, `AggIntent + DeploymentConstraints → AggregationType + SketchParams` at L4. -### Intent vocabulary: DC's `AggIntent` is a superset; scenarios use subsets +### Intent vocabulary: DC's `AggIntent` is a superset; deployment models use subsets -DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because language-flavored synonyms (`QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core is 9 (`Count, Sum, Min, Max, Quantile, TopK, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps directly: `Topk` keeps its own intent (heavy-hitter sketches like SpaceSaving / CMS-with-heap compute it as a single primitive, so the intent earns L3 visibility). Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that scenarios opt into. +DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because language-flavored synonyms (`QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core is 9 (`Count, Sum, Min, Max, Quantile, TopK, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps directly: `Topk` keeps its own intent (heavy-hitter sketches like SpaceSaving / CMS-with-heap compute it as a single primitive, so the intent earns L3 visibility). Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that deployment models opt into. ### Data-model support: both time-series and tabular ASAPQuery-backend / DC controller operate on time-series data (metrics + labels + timestamp); asap-fusion operates on tabular data (DataFusion `LogicalPlan` over relations) that may or may not be time-indexed. These two data models differ fundamentally in their leaf shape (`metric + labels + time` vs. `table + columns`), but they share everything above the leaf — filter semantics, aggregation semantics, sketches themselves. Core handles this with: -- **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Scenarios' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. +- **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Deployment models' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. - **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Min`, `Max`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase` — both carry PromQL counter-reset semantics), or tabular-only (future additions for joins, correlated subqueries). - **Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values; that stream can come from a time-series window or a table column, the sketch does not know or care. So `BindKllOnQuantile` and siblings work uniformly across both. Practically, this means: -- `scenario-query` + `scenario-lifecycle` lower into `QueryExpr` with `Source::TimeSeries` leaves. -- `scenario-fusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). -- A hypothetical OLAP scenario that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. +- `deployment-model-asapquery` + `deployment-model-asaplifecycle` lower into `QueryExpr` with `Source::TimeSeries` leaves. +- `deployment-model-asapfusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). +- A hypothetical OLAP deployment model that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. See §6 `core::sketch_algebra` for the concrete type sketches. @@ -81,9 +81,9 @@ The initial implementation can operate on **one query at a time** — L4 picks p ### L2 is mandatory; the tree shape is an evolvable contract -Every scenario must produce an L2 tree, even when the source language didn't originally come as one. asap-planner-rs's current approach (PromQL pattern catalogue → `IntermediateAggConfig`) skips L2; Phase 4 will reverse-engineer the five PromQL pattern shapes into a `PromqlLogicalPlan` tree so the L1→L2→L3 pipeline is uniform. +Every deployment model must produce an L2 tree, even when the source language didn't originally come as one. asap-planner-rs's current approach (PromQL pattern catalogue → `IntermediateAggConfig`) skips L2; Phase 4 will reverse-engineer the five PromQL pattern shapes into a `PromqlLogicalPlan` tree so the L1→L2→L3 pipeline is uniform. -A future scenario whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. +A future deployment model whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. ### System I/O contract — what the controller takes in, what it emits @@ -104,38 +104,43 @@ The controller is a **planner**, not an executor. It does not run queries; it de | Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge / compute-from-raw over the precompute engine, …) | Implied by the per-executor configs; not a separate artifact | | Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | -**Premise behind the per-executor split.** Every executor (edge collector, gateway, backend, DataFusion runtime) is in principle capable of running the entire query tree — they all speak the same physical operators. The controller's job is to decide *which* executor runs *which sub-tree / sub-DAG* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by executor, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring; the `PhysicalPlanner` per scenario emits the per-executor configs. +**Premise behind the staged dispatch.** Two distinct concepts: + +- **Stage** (`StageId`) — a categorical *tier* in the data lifecycle (edge / gateway / backend / in-process). The topology declares which stages exist (3-stage / 1-stage / 0-stage). Stages are roles, not instances. +- **Executor** (`Executor`, defined in `core::physical::executor`) — a *concrete runtime instance* that occupies a stage. Carries `id`, `stage: StageId`, `capabilities`, and `address` (OpAMP agent / HTTP endpoint / in-process handle). One stage may have N executors — e.g. a 50-host edge fleet is 50 `Executor`s all with `stage = StageId("edge")`; a singleton backend is one `Executor` at `stage = StageId("backend")`. + +Every stage is in principle capable of running the entire query tree — all stages speak the same physical operators. The controller's job is to decide *which stage* runs *which sub-tree / sub-DAG* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by `StageId`, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring at stage granularity; the per-deployment-model `PhysicalPlanner` then materialises each stage's sub-DAG into one config per `Executor` at that stage (varying only by per-executor connection / identity details). The executor list comes from `DeploymentConstraints::executors()`. This is what makes the topology a *parameter* rather than an axis of code: the same `SketchExpr` plus a different `TopologyDescriptor` produces edge-only / 1-stage / 3-stage / 0-stage placements without rewriting the plan. **Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialize into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. -This drives the core/scenario split: +This drives the core/deployment model split: - **`crates/core/`** owns all 5 layers of **shared infrastructure**: L1-3 end-to-end (parsers + lowering + intent-only IR), plus L4's rule engine driver + rule library + cost-model traits, plus L5's stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. -- **Each scenario is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any scenario-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. +- **Each deployment model is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any deployment-model-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. ## 4. Principles -### P1. Core owns shared infrastructure across all 5 layers; scenarios own choices +### P1. Core owns shared infrastructure across all 5 layers; deployment models own choices -See §3. Core is NOT just types + trait stubs — it ships working parsers + lowering passes (L1-3), a rule engine + rule library + cost-model traits (L4 framework), and a stage allocator + physical-plan framework + sketch catalogue (L5 framework). What scenarios plug in is **which rules fire** (picking from core's library + adding their own), **deployment topology** (how many stages; DC=3, query=1, fusion=0), and an **emitter** for the output format. A scenario that accepts all core's default L4 rules and uses `core::physical::single_stage_topology` is maybe 200 lines of code. +See §3. Core is NOT just types + trait stubs — it ships working parsers + lowering passes (L1-3), a rule engine + rule library + cost-model traits (L4 framework), and a stage allocator + physical-plan framework + sketch catalogue (L5 framework). What deployment models plug in is **which rules fire** (picking from core's library + adding their own), **deployment topology** (how many stages; DC=3, query=1, fusion=0), and an **emitter** for the output format. A deployment model that accepts all core's default L4 rules and uses `core::physical::single_stage_topology` is maybe 200 lines of code. ### P2. Core has no I/O -No HTTP, no OpAMP, no YAML, no Prometheus scrape, no `tokio::spawn`. Pure algorithms over in-memory data. This is what makes scenarios unit-testable without running the runtime. +No HTTP, no OpAMP, no YAML, no Prometheus scrape, no `tokio::spawn`. Pure algorithms over in-memory data. This is what makes deployment models unit-testable without running the runtime. -### P3. Runtime is a thin binary, scenarios are libraries +### P3. Runtime is a thin binary, deployment models are libraries -The `asap-controller` binary is assembled from: runtime (HTTP/OpAMP/replanner/store) + N scenario crates registered as plugins. Swapping scenarios is a build-time feature flag or a runtime registry entry. No scenario may reach directly into another. +The `asap-controller` binary is assembled from: runtime (HTTP/OpAMP/replanner/store) + N deployment model crates registered as plugins. Swapping deployment models is a build-time feature flag or a runtime registry entry. No deployment model may reach directly into another. ### P4. One input boundary, one output boundary **Input**: everything that enters the controller (HTTP `QuerySpec`, Prometheus query log replay, YAML workload, capability-miss callback) normalizes into a single `QueryWorkload` type in core — a collection of `QuerySpec`s each feeding L1→L2→L3. -**Output**: L5 emitters (OpAMP `RemoteConfig`, backend `StreamingConfig` POST, one-shot YAML file, rewritten DataFusion `LogicalPlan`) all implement a `PlanEmitter` trait. Scenarios supply emitter implementations; core doesn't know which emitters exist. +**Output**: L5 emitters (OpAMP `RemoteConfig`, backend `StreamingConfig` POST, one-shot YAML file, rewritten DataFusion `LogicalPlan`) all implement a `PlanEmitter` trait. Deployment models supply emitter implementations; core doesn't know which emitters exist. -This is the "extension point for future scenarios" — a new scenario adds an L4 rule set + an L5 emitter and registers them. +This is the "extension point for future deployment models" — a new deployment model adds an L4 rule set + an L5 emitter and registers them. ### P5. No feature-flag spaghetti @@ -150,7 +155,7 @@ ASAPController/ ├── docs/ │ ├── design.md # this file │ ├── migration-plan.md # the companion file -│ ├── scenarios/ # per-scenario design notes +│ ├── deployment-models/ # per-deployment-model design notes │ └── adr/ # architecture decision records ├── proto/ │ ├── asap_control.proto # Plan IR + QueryWorkload on the wire @@ -170,11 +175,12 @@ ASAPController/ │ │ │ ├── planner_trait/ # PhysicalPlanner trait │ │ │ ├── stage_allocator/ # generic topology-driven allocator │ │ │ ├── topology/ # Topology descriptor types (edge/gateway/backend, single, zero) +│ │ │ ├── executor/ # Executor type — concrete runtime instance occupying a stage │ │ │ └── sketch_catalog/ # candidate sketch types + parameter constraints -│ │ ├── pipeline/ # orchestrates L1→L2→L3→L4→L5, parameterized on scenario +│ │ ├── pipeline/ # orchestrates L1→L2→L3→L4→L5, parameterized on deployment model │ │ ├── workload/ # QueryWorkload wrapper around QuerySpecs -│ │ ├── emit/ # PlanEmitter trait — implemented by scenario L5 -│ │ ├── registry/ # ScenarioRegistry, ScenarioId +│ │ ├── emit/ # PlanEmitter trait — implemented by deployment model L5 +│ │ ├── registry/ # DeploymentModelRegistry, DeploymentModelId │ │ └── telemetry/ # tracing macros, metric names (no exporter) │ ├── runtime/ # service skeleton — HTTP, OpAMP, replanner │ │ ├── http/ # axum surface — /plan, /replan, /metrics, /status @@ -183,41 +189,41 @@ ASAPController/ │ │ ├── replan/ # Replanner — SLA + expiry triggers │ │ ├── store/ # PlanStore, WorkloadStore │ │ └── backend_client/ # HTTP client (pushes to ASAPQuery-backend, etc.) -│ ├── scenario-lifecycle/ # thin — picks rules, 3-stage topology, OTel/backend emitters +│ ├── deployment-model-asaplifecycle/ # thin — picks rules, 3-stage topology, OTel/backend emitters │ │ ├── rules.rs # L4 rule selection + DC-specific rules (stage-aware push-down) │ │ ├── topology.rs # 3-stage: edge / gateway / backend -│ │ ├── cost.rs # scenario-specific cost impls — delta / online / pareto / tco +│ │ ├── cost.rs # deployment-model-specific cost impls — delta / online / pareto / tco │ │ └── emit/ # OpAmpRemoteConfig + AsapqueryBackendConfig emitters -│ ├── scenario-query/ # thin — rules, 1-stage topology, YAML emitters + query-log input +│ ├── deployment-model-asapquery/ # thin — rules, 1-stage topology, YAML emitters + query-log input │ │ ├── rules.rs # L4 rule selection + sketch-binding rule (split from map_statistic_*) │ │ ├── topology.rs # 1-stage: backend-only │ │ ├── emit/ # StreamingConfig.yaml + InferenceConfig.yaml │ │ ├── query_log/ # extra L1 input: Prometheus query-log replay │ │ └── schema/ # PromQLSchema discovery from Prometheus (feeds L1) -│ ├── scenario-fusion/ # thin — DF-flavored rules, 0-stage, in-process emit +│ ├── deployment-model-asapfusion/ # thin — DF-flavored rules, 0-stage, in-process emit │ │ ├── rules.rs # L4 rules for DataFusion LogicalPlan (sketch-aware rewrites) │ │ ├── topology.rs # 0-stage: in-process │ │ ├── emit/ # rewritten DataFusion LogicalPlan │ │ ├── executor/ # DataFusion SessionContext wrapper (library-mode execution) │ │ └── sketch_support/ # asap_sketchlib-backed rewrites │ ├── control-proto/ # generated from proto/ (tonic/prost) -│ └── testing/ # test fixtures + harness shared across scenarios +│ └── testing/ # test fixtures + harness shared across deployment models └── bin/ - ├── asap-controller/ # long-running service with all scenarios - │ └── main.rs # axum + OpAMP + replanner + registered scenarios - ├── asap-query/ # one-shot CLI for scenario-query (what asap-planner-rs is today) + ├── asap-controller/ # long-running service with all deployment models + │ └── main.rs # axum + OpAMP + replanner + registered deployment models + ├── asap-query/ # one-shot CLI for deployment-model-asapquery (what asap-planner-rs is today) │ └── main.rs # clap — read workload YAML, emit two YAMLs - ├── asap-lifecycle/ # OPTIONAL: standalone service with only scenario-lifecycle + ├── asap-lifecycle/ # OPTIONAL: standalone service with only deployment-model-asaplifecycle │ └── main.rs # slimmer image — no DataFusion, no query YAML emitters - └── asap-fusion-bench/ # OPTIONAL: benchmark harness over scenario-fusion + └── asap-fusion-bench/ # OPTIONAL: benchmark harness over deployment-model-asapfusion └── main.rs # criterion entry; used by researchers ``` -**Per-scenario standalone binaries** are first-class. Each `bin//` is a thin shell that `use`s only the scenario crates it needs — so `bin/asap-lifecycle/` doesn't pull `datafusion` into its dep tree, and `bin/asap-fusion-bench/` doesn't pull `axum`/`opamp`. Feature flags on the workspace root let you `cargo build -p asap-lifecycle` and get a minimal binary. +**Per-deployment-model standalone binaries** are first-class. Each `bin//` is a thin shell that `use`s only the deployment model crates it needs — so `bin/asap-lifecycle/` doesn't pull `datafusion` into its dep tree, and `bin/asap-fusion-bench/` doesn't pull `axum`/`opamp`. Feature flags on the workspace root let you `cargo build -p asap-lifecycle` and get a minimal binary. -### Why three scenario crates today, not one monolithic `scenarios/` +### Why three deployment model crates today, not one monolithic `deployment-models/` -Each scenario has a different problem shape: +Each deployment model has a different problem shape: | | **lifecycle** | **query** | **fusion** | |---|---|---|---| @@ -227,7 +233,7 @@ Each scenario has a different problem shape: | Trigger | QuerySpec, SLA violation, expiry | one-shot CLI invocation | a DataFusion session constructing a query | | Cost model | accuracy × latency × $ staged | single-query accuracy/latency | operator selectivity / sketch feasibility | -Mashing them into one crate means the union of all their dependencies (sqlparser + promql-parser + DataFusion + OpAMP proto + Prometheus client) bleeds into every downstream user. Separating them means a user who only needs `scenario-fusion` (an offline query-optimization benchmark, say) can depend on it without pulling OpAMP. +Mashing them into one crate means the union of all their dependencies (sqlparser + promql-parser + DataFusion + OpAMP proto + Prometheus client) bleeds into every downstream user. Separating them means a user who only needs `deployment-model-asapfusion` (an offline query-optimization benchmark, say) can depend on it without pulling OpAMP. ## 6. Core crate details (layers 1–3 + driver) @@ -250,7 +256,7 @@ A **per-language** algebra tree — one `enum LogicalPlan` per language. Preserv ### `core::sketch_algebra` — Layer 3 -The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `HistogramQuantile`, no `PromQLSubquery` — those are L2 PromQL nodes), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP scenarios) via a `Source` sum type inside `QueryExpr::Scan`. +The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `HistogramQuantile`, no `PromQLSubquery` — those are L2 PromQL nodes), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP deployment models) via a `Source` sum type inside `QueryExpr::Scan`. #### Design rules for L3 @@ -353,13 +359,13 @@ pub enum QueryExpr { pub enum WindowKind { Tumbling, Sliding, Session } pub enum Source { - /// Time-series input — scenario-query / scenario-lifecycle shape. + /// Time-series input — deployment-model-asapquery / deployment-model-asaplifecycle shape. TimeSeries { metric: MetricRef, time: TimeRange, labels: LabelFilter }, - /// Tabular input — scenario-fusion / future-OLAP shape. + /// Tabular input — deployment-model-asapfusion / future-OLAP shape. Table { table_ref: TableRef, columns: Vec }, /// Join over Sources composes leaf shapes recursively. Join { left: Box, right: Box, on: JoinKey }, - // Future: WindowedStream, Subquery — added by scenarios that need them. + // Future: WindowedStream, Subquery — added by deployment models that need them. } pub enum DataModel { TimeSeries, Tabular, Any } @@ -432,15 +438,15 @@ Per-node input/output spec — the stable contract for L3 nodes (full implementa | `WindowFunc { func, partition_by, order_by, frame }` | one input schema; every field in `partition_by` / `order_by` must be present | the input schema extended with one new column carrying the analytic-function output (named after `func`, typed per `func`) | | `BinaryOp { op, vector_match }` | two input schemas — left and right operands. PromQL vector-match constraints (`on`/`ignoring` + `group_left`/`group_right`) govern label-set compatibility | for arithmetic/comparison `op`: the left input's schema with the value column re-typed to the result of `op`; for boolean `op` (`and`, `or`, `unless`): the left input's schema with a boolean value column | -#### Three distinct schemas — DAG vs DB vs sketch catalog +#### DAG schema, DB schema, sketch catalog — three distinct metadata sources -These three are sometimes conflated and shouldn't be: +These three are sometimes conflated and shouldn't be. Only the first two are *schemas* (descriptions of stream / table shape); the sketch catalog is a *registry* of available primitives, not a description of a stream: -| Schema | Where it lives | What it describes | Who reads it | +| Source | Where it lives | What it describes | Who reads it | |---|---|---|---| | **DAG schema** | On every edge of the L3 / L4 / L5 DAG (`Schema` above) | Columns + types flowing between operators | L4 rules (selectivity estimation, push-down legality), L5 emitter | | **DB / source schema** | The query target (Prometheus TSDB metric metadata, SQL `information_schema`, DataFusion catalog) | What metrics / tables / columns exist in the data plane, with their types and indexing | `core::lower::*` to resolve names during L1→L2; exposed through a `SchemaCatalog` interface | -| **Sketch catalog metadata** | `core::physical::sketch_catalog` (built at startup; static) | What sketches the runtime can build; what intents each one serves; mergeability, accuracy / confidence guarantees, supported aggregation keys, parameter ranges | L4 binding rules to choose a sketch for an `AggIntent`; L5 to instantiate the sketch | +| **Sketch catalog** | `core::physical::sketch_catalog` (built at startup; static) | What sketches the runtime can build; what intents each one serves; mergeability, accuracy / confidence guarantees, supported aggregation keys, parameter ranges | L4 binding rules to choose a sketch for an `AggIntent`; L5 to instantiate the sketch | L1→L2 lowering reads the **DB schema** to resolve symbols. L3 onward, every edge carries a **DAG schema** that is type-checked locally. L4 binding rules consult the **sketch catalog** to map an intent to a concrete sketch under the deployment's constraints. They are three separate inputs to three distinct decisions. @@ -471,7 +477,7 @@ pub enum AggIntent { Rate { window: Duration }, Increase { window: Duration }, - // Tabular / OLAP — added as scenarios demand + // Tabular / OLAP — added as deployment models demand // CorrelatedSubqueryCount { … }, ApproxJoinCardinality { … }, } @@ -592,35 +598,35 @@ pub fn lower_datafusion(ast: DfAst, ctx: &DfContext) -> Result; pub fn lower_elasticdsl(ast: EsAst, schema: &IndexSchema) -> Result; ``` -Once a query hits L3 it's language-agnostic. All scenarios downstream see the same IR. +Once a query hits L3 it's language-agnostic. All deployment models downstream see the same IR. ### `core::pipeline` — orchestration -The L1→…→L5 driver. Parameterised on a scenario's optimizer rules (L4) + emitter (L5): +The L1→…→L5 driver. Parameterised on a deployment model's optimizer rules (L4) + emitter (L5): ```rust -pub struct Pipeline { - scenario: S, +pub struct Pipeline { + deployment_model: S, } -impl Pipeline { +impl Pipeline { pub fn run(&self, workload: &QueryWorkload) -> Result { let l3: Vec = workload.queries() .iter() .map(|q| self.parse_and_lower(q)) // L1→L2→L3 .collect::>()?; - let l4 = self.scenario.optimizer().optimize(l3)?; // L4 - let l5 = self.scenario.physical().lower(l4)?; // L5 - self.scenario.emitter().emit(&l5) // L5 → bytes/protobuf/DF plan + let l4 = self.deployment_model.optimizer().optimize(l3)?; // L4 + let l5 = self.deployment_model.physical().lower(l4)?; // L5 + self.deployment_model.emitter().emit(&l5) // L5 → bytes/protobuf/DF plan } } ``` -Core owns the driver. Scenarios own what goes into each scenario-specific seam. +Core owns the driver. Deployment models own what goes into each deployment-model-specific seam. ### `core::optimizer` — Layer 4 framework -Core ships the **rule engine + trait surface + a shared rule library**. Scenarios pick which rules to enable. +Core ships the **rule engine + trait surface + a shared rule library**. Deployment models pick which rules to enable. ```rust // trait (in core) @@ -639,7 +645,7 @@ impl RuleEngine { -> Result, OptError>; } -// shared rule library (in core::optimizer::rules) — opt-in from scenarios +// shared rule library (in core::optimizer::rules) — opt-in from deployment models pub mod rules { pub struct BindKllOnQuantile; // AggIntent::Quantile → bind KLL(k by accuracy) pub struct BindCmsOnCount; // AggIntent::Count → bind CMS(w, d) @@ -651,10 +657,10 @@ pub mod rules { } ``` -Scenarios compose rule sets by picking from the shared library + adding their own: +Deployment models compose rule sets by picking from the shared library + adding their own: ```rust -// in scenario-lifecycle +// in deployment-model-asaplifecycle use asap_control_core::optimizer::{RuleEngine, rules::*}; fn rule_set() -> Vec> { vec![ @@ -668,7 +674,7 @@ fn rule_set() -> Vec> { } ``` -Core also ships `DeploymentConstraints` as a trait object; each scenario supplies a concrete impl with its deployment's memory budgets, network topology, available sketch backends. +Core also ships `DeploymentConstraints` as a trait object; each deployment model supplies a concrete impl with its deployment's memory budgets, network topology, available sketch backends, and the registered `Executor` list (one entry per concrete runtime instance, each tagged with its `StageId`). The executor list is populated from the deployment model's discovery channel — OpAMP for DC lifecycle, static config for single-backend query, the in-process `SessionContext` itself for fusion. ### `core::physical` — Layer 5 framework @@ -691,8 +697,34 @@ pub mod topology { pub struct ZeroStage; /* in-process */ } +/// Identifier for a stage role (categorical tier in the data lifecycle). +pub struct StageId(pub String); // "edge" / "gateway" / "backend" / "in-process" + +/// A **concrete runtime instance** that executes a sub-DAG of the plan. +/// One stage may have many executors — e.g. a 50-host edge fleet is 50 +/// executors all with `stage = StageId("edge")`; a singleton backend is +/// one executor at `stage = StageId("backend")`. The stage allocator +/// works at stage granularity; the deployment-model planner / emitter +/// materialises each stage's sub-DAG into one concrete config per executor. +pub struct Executor { + pub id: ExecutorId, // stable identifier across replans + pub stage: StageId, // which stage this executor occupies + pub capabilities: ExecutorCaps, // memory budget, available sketch backends, network neighbours + pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle +} + +pub struct ExecutorId(pub String); + +pub enum ExecutorAddr { + OpAmpAgent(AgentId), // OTel collector managed via OpAMP (DC lifecycle) + HttpEndpoint(Url), // ASAPQuery-backend, etc. + InProcess, // asap-fusion library-mode SessionContext +} + // generic stage allocator — given a QueryExpr tree + a topology, decide which -// ops land on which stage subject to constraints +// ops land on which stage subject to constraints. Stage-level only; per-executor +// fan-out happens in the deployment model's PhysicalPlanner using the executor +// list from `DeploymentConstraints::executors()`. pub struct StageAllocator; impl StageAllocator { pub fn allocate( @@ -736,18 +768,22 @@ pub struct SketchEntry { } ``` -Scenarios use these pieces: +Deployment models use these pieces: ```rust -// in scenario-lifecycle -use asap_control_core::physical::{StageAllocator, topology::ThreeStage}; +// in deployment-model-asaplifecycle +use asap_control_core::physical::{StageAllocator, topology::ThreeStage, Executor}; impl PhysicalPlanner for LifecyclePlanner { type Topology = ThreeStage; - type Output = (StagedExprs, SketchBindings); + type Output = Vec<(ExecutorId, ExecutorPlan)>; // one entry per executor fn lower(&self, l4: Vec, t: &ThreeStage) -> Result<_, _> { + // 1. Stage-level: colour the DAG by StageId. let assignments = StageAllocator.allocate(&l4, t, &self.constraints)?; - // scenario-specific post-processing: DC's delta_cost logic, + // 2. Per-executor fan-out: for each Executor in the deployment, + // pick the sub-DAG assigned to its stage and produce its config. + let executors: &[Executor] = self.constraints.executors(); + // deployment-model-specific post-processing: DC's delta_cost logic, // backend_client push preparation, etc. Ok(/* ... */) } @@ -757,7 +793,7 @@ impl PhysicalPlanner for LifecyclePlanner { ### `core::plan` — shared traits bridging layers ```rust -pub trait Scenario { +pub trait DeploymentModel { type Topology: TopologyDescriptor; type EmitterOutput; fn rules(&self) -> Vec>; @@ -785,11 +821,11 @@ pub enum QueryWorkload { pub struct QuerySpec { /* metric_name, aggregations, time_window, SLAs, ... */ } ``` -All three scenarios take `&QueryWorkload`. Same type, different planners. +All three deployment models take `&QueryWorkload`. Same type, different planners. ### `core::cost` -Extracted from DC's `planner/*` (delta, online, pareto, tco). Scenarios pick the models they need. +Extracted from DC's `planner/*` (delta, online, pareto, tco). Deployment models pick the models they need. ```rust pub trait CostModel { @@ -831,29 +867,29 @@ pub trait PlanEmitter { } ``` -Concrete emitters live in scenario crates: +Concrete emitters live in deployment model crates: -- `scenario-query::yaml::StreamingConfigEmitter` → `streaming_config.yaml` bytes -- `scenario-query::yaml::InferenceConfigEmitter` → `inference_config.yaml` bytes -- `scenario-lifecycle::emit::OpAmpRemoteConfigEmitter` → `opamp::RemoteConfig` protobuf -- `scenario-fusion::emit::DataFusionPlanEmitter` → rewritten `datafusion::LogicalPlan` +- `deployment-model-asapquery::yaml::StreamingConfigEmitter` → `streaming_config.yaml` bytes +- `deployment-model-asapquery::yaml::InferenceConfigEmitter` → `inference_config.yaml` bytes +- `deployment-model-asaplifecycle::emit::OpAmpRemoteConfigEmitter` → `opamp::RemoteConfig` protobuf +- `deployment-model-asapfusion::emit::DataFusionPlanEmitter` → rewritten `datafusion::LogicalPlan` ### `core::registry` The extension point. The runtime binary does: ```rust -let mut reg = ScenarioRegistry::new(); -reg.register::(); -reg.register::(); -reg.register::(); +let mut reg = DeploymentModelRegistry::new(); +reg.register::(); +reg.register::(); +reg.register::(); // add more... ``` -Registration is by-type; the runtime looks up by `ScenarioId` (either from the `QuerySpec.scenario` field or from an HTTP route). A 4th scenario is added by: +Registration is by-type; the runtime looks up by `DeploymentModelId` (either from the `QuerySpec.deployment_model` field or from an HTTP route). A 4th deployment model is added by: -1. New crate `scenario-/` -2. Implement `Scenario` trait (wraps a `Planner` + its `PlanEmitter`s) +1. New crate `deployment-model-/` +2. Implement `DeploymentModel` trait (wraps a `Planner` + its `PlanEmitter`s) 3. Register in `bin/asap-controller/main.rs` No core change. @@ -865,65 +901,65 @@ Lifted from DC controller with zero semantic change: - `http/` — axum server on `:8080`. Routes: `POST /plan`, `POST /replan`, `GET /plans/:id`, `GET /status`, `GET /metrics` - `opamp/` — WebSocket OpAMP server on `:4320`. Same protocol DC speaks today. - `monitor/` — `Scraper` polls Prometheus, emits `Violation` -- `replan/` — subscribes to violations + expiry ticks, re-invokes the scenario registry +- `replan/` — subscribes to violations + expiry ticks, re-invokes the deployment model registry - `store/` — in-memory `PlanStore` + `WorkloadStore`. Pluggable backend later. -- `backend_client/` — pushes `StreamingConfig` YAML to ASAPQuery-backend's `/api/v1/streaming-config` endpoint. Factored out so new scenarios can push to other backends. +- `backend_client/` — pushes `StreamingConfig` YAML to ASAPQuery-backend's `/api/v1/streaming-config` endpoint. Factored out so new deployment models can push to other backends. -Runtime depends on `core` but NOT on any scenario crate directly. It talks to scenarios via the registry. +Runtime depends on `core` but NOT on any deployment model crate directly. It talks to deployment models via the registry. -## 8. Scenario crate details (each owns its L4 + L5) +## 8. Deployment model crate details (each owns its L4 + L5) -Each scenario crate is a library with: +Each deployment model crate is a library with: -- A `Scenario` impl that registers its planner, its emitters, and its HTTP routes (if any). +- A `DeploymentModel` impl that registers its planner, its emitters, and its HTTP routes (if any). - Its own config types — no shared config crate. - Its own cost model — maybe using `core::cost` primitives, maybe not. - Its own integration tests. -### `scenario-lifecycle` (thin) +### `deployment-model-asaplifecycle` (thin) - **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **L4 rules**: picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules that require stage awareness (`StageAwarePushDown`, `TransmissionCostRewrite`). Adding a stage-specific rule = a new file in `scenario-lifecycle/src/rules.rs`, impl `OptimizerRule`. Unchanged rules come from core. -- **L5 topology**: `core::physical::topology::ThreeStage` (edge → gateway → backend). No scenario-specific allocator logic — `StageAllocator` handles the tree walk; lifecycle only declares what the topology looks like. -- **Cost models**: `delta / online-EMA / Pareto / TCO` — these live in `scenario-lifecycle/src/cost.rs` because they're specific to the DC deployment's network/compute assumptions. Implement `core::optimizer::cost::CostModel`. -- **L5 emitters**: `OpAmpRemoteConfigEmitter` (per-role OTel YAML over OpAMP WebSocket) and `AsapqueryBackendConfigEmitter` (calls `scenario-query`'s `StreamingConfigEmitter` for the YAML bytes, then POSTs to backend). +- **L4 rules**: picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules that require stage awareness (`StageAwarePushDown`, `TransmissionCostRewrite`). Adding a stage-specific rule = a new file in `deployment-model-asaplifecycle/src/rules.rs`, impl `OptimizerRule`. Unchanged rules come from core. +- **L5 topology**: `core::physical::topology::ThreeStage` (edge → gateway → backend). No deployment-model-specific allocator logic — `StageAllocator` handles the tree walk; lifecycle only declares what the topology looks like. +- **Cost models**: `delta / online-EMA / Pareto / TCO` — these live in `deployment-model-asaplifecycle/src/cost.rs` because they're specific to the DC deployment's network/compute assumptions. Implement `core::optimizer::cost::CostModel`. +- **L5 emitters**: `OpAmpRemoteConfigEmitter` (per-role OTel YAML over OpAMP WebSocket) and `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery`'s `StreamingConfigEmitter` for the YAML bytes, then POSTs to backend). - **HTTP route**: `POST /plan` for full-lifecycle planning, `POST /replan` for SLA-triggered. - **Size estimate**: ~1500 LOC (was ~5000 pre-refactor). Cost models are the bulk; rule selection + topology + emitter are each a few hundred lines. -### `scenario-query` (thin) +### `deployment-model-asapquery` (thin) - **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. - **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. - **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. - **L3 intent**: maps `Statistic` enum (9 variants) onto `core::sketch_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). -- **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this scenario covers most sketch types) + a scenario-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. +- **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this deployment model covers most sketch types) + a deployment-model-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. - **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. -- **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `scenario-lifecycle` calls them when it needs to POST to ASAPQuery-backend. -- **Extra L1 inputs**: `query_log/` for Prometheus-query-log replay (unique to this scenario); `schema/` for `PromQLSchema` discovery from a live Prometheus URL. +- **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `deployment-model-asaplifecycle` calls them when it needs to POST to ASAPQuery-backend. +- **Extra L1 inputs**: `query_log/` for Prometheus-query-log replay (unique to this deployment model); `schema/` for `PromQLSchema` discovery from a live Prometheus URL. - **HTTP route**: `POST /plan/query` (JSON `QuerySpec` in, YAML stream out). Also backs the `bin/asap-query` one-shot CLI. - **Size estimate**: ~2500 LOC (was ~6000 pre-refactor — saved by picking from shared rule library; still pays the L2 tree + L3/L4 split refactor cost, which is one-time). -### `scenario-fusion` (thin) +### `deployment-model-asapfusion` (thin) - **Data model**: tabular. L2→L3 lowering produces `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when fusion extends to multi-table). Crucially **not time-indexed** — fusion works on arbitrary DataFusion relations; time is just another column if present. -- **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode scenario. -- **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so scenarios that want to take a DataFusion plan as input have a canonical name for it. +- **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode deployment model. +- **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so deployment models that want to take a DataFusion plan as input have a canonical name for it. - **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::sketch_algebra::AggIntent` subset. -- **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + scenario-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. +- **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + deployment-model-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. - **L5 topology**: `core::physical::topology::ZeroStage` (in-process). -- **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this scenario is library-mode. -- **Executor**: `ASAPExecutor` wraps a DataFusion `SessionContext`. Users construct `scenario-fusion` in-process. +- **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this deployment model is library-mode. +- **Executor**: `ASAPExecutor` wraps a DataFusion `SessionContext`. Users construct `deployment-model-asapfusion` in-process. - **HTTP route**: none by default. -- **Conformance cost**: near zero. Fusion's `SketchConfigRule` logic is replaced with picks from `core::optimizer::rules::*`; its `HashModeRule` stays scenario-specific. +- **Conformance cost**: near zero. Fusion's `SketchConfigRule` logic is replaced with picks from `core::optimizer::rules::*`; its `HashModeRule` stays deployment-model-specific. - **Size estimate**: ~1800 LOC (was ~3000 pre-refactor; the `SketchConfigRule` code folds into core's shared rule library). -The sketch microbenchmarks (KLL/CMS) move with the crate and keep running. The TODO items from `asap-fusion/TODO.md` (batch/multi-query execution, time semantics, distributed model) remain open but are now filed against `scenario-fusion/TODO.md` in-repo. +The sketch microbenchmarks (KLL/CMS) move with the crate and keep running. The TODO items from `asap-fusion/TODO.md` (batch/multi-query execution, time semantics, distributed model) remain open but are now filed against `deployment-model-asapfusion/TODO.md` in-repo. ### Data plane communication Control plane (ASAPController) and data plane (OTel agents / ASAPQuery-backend / DataFusion runtimes) always talk **over wire**, never via in-process calls. This is unchanged from today: -| Scenario | Data plane lives in | Wire protocol | +| Deployment model | Data plane lives in | Wire protocol | |---|---|---| | lifecycle | DataCollector (OTel collectors, agent + backend roles) | OpAMP WebSocket (config push) + HTTP POST (`StreamingConfig` → ASAPQuery-backend) + Prometheus scrape (metrics in) | | query | ASAPQuery-backend (query engine, SimpleMapStore) | HTTP POST `/api/v1/streaming-config` + `/api/v1/plan` (capability-miss callback in) + YAML file on disk (init-container mode) | @@ -931,34 +967,34 @@ Control plane (ASAPController) and data plane (OTel agents / ASAPQuery-backend / **Data plane code stays in its original repo.** ASAPController only owns the control plane. The merger doesn't move OTel collectors out of DataCollector, doesn't move the query engine out of ASAPQuery-backend, and doesn't move DataFusion out of asap-fusion's users. Each data plane keeps its own release cadence. -### Scenario placement: in-repo or out-of-repo +### Deployment model placement: in-repo or out-of-repo -Because core owns the L4/L5 infrastructure (not just L1-3), a scenario crate is small and largely self-contained. That means scenarios can live either: +Because core owns the L4/L5 infrastructure (not just L1-3), a deployment model crate is small and largely self-contained. That means deployment models can live either: -- **Inside ASAPController workspace** — `crates/scenario-/`, lockstep release with core, cross-scenario changes are one PR. +- **Inside ASAPController workspace** — `crates/deployment-model-/`, lockstep release with core, cross-deployment-model changes are one PR. - **In their own downstream repo** — declares `asap-control-core` as a git-tagged dep, releases independently, owns its own CI. Both produce functionally identical artifacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. Default recommendation: -- **scenario-lifecycle** and **scenario-query** in ASAPController (they share `scenario-query`'s YAML emitter via a workspace `path = "../scenario-query"` dep — trivial in-workspace). -- **scenario-fusion** out-of-tree in `asap-fusion` repo (research project with independent benchmark cadence; depends on `asap-control-core` + `asap-control-optimizer` as published git tags). +- **deployment-model-asaplifecycle** and **deployment-model-asapquery** in ASAPController (they share `deployment-model-asapquery`'s YAML emitter via a workspace `path = "../deployment-model-asapquery"` dep — trivial in-workspace). +- **deployment-model-asapfusion** out-of-tree in `asap-fusion` repo (research project with independent benchmark cadence; depends on `asap-control-core` + `asap-control-optimizer` as published git tags). -Future scenarios choose whichever placement fits the team that owns them. +Future deployment models choose whichever placement fits the team that owns them. ### Asymmetric dependency -`scenario-lifecycle` depends on `scenario-query` because the `StreamingConfig` YAML emitter is authoritative there. When both live in ASAPController workspace this is a trivial `path =` dep. If one is ever moved out-of-tree, we lift the emitter into core to avoid a cross-repo Cargo dep. +`deployment-model-asaplifecycle` depends on `deployment-model-asapquery` because the `StreamingConfig` YAML emitter is authoritative there. When both live in ASAPController workspace this is a trivial `path =` dep. If one is ever moved out-of-tree, we lift the emitter into core to avoid a cross-repo Cargo dep. -## 9. Extension point — a 4th scenario +## 9. Extension point — a 4th deployment model -With the L4/L5 framework in core, adding a new scenario is mostly picking + a bit of glue. A hypothetical "edge caching" scenario that decides which queries to cache at the edge vs. backend would land as: +With the L4/L5 framework in core, adding a new deployment model is mostly picking + a bit of glue. A hypothetical "edge caching" deployment model that decides which queries to cache at the edge vs. backend would land as: ``` -crates/scenario-edge-cache/ # or your own repo +crates/deployment-model-edge-cache/ # or your own repo ├── Cargo.toml # depends on asap-control-core ├── src/ -│ ├── lib.rs # impl Scenario for EdgeCacheScenario +│ ├── lib.rs # impl DeploymentModel for EdgeCacheDeploymentModel │ ├── rules.rs # pick core rules + add EdgeCacheBindRule │ ├── topology.rs # TwoStage { edge, backend } TopologyDescriptor │ ├── cost.rs # impl CostModel — hit ratio × bandwidth @@ -970,21 +1006,21 @@ crates/scenario-edge-cache/ # or your own repo Total touch outside the new crate: - 1 line in `bin/asap-controller/main.rs` to register (if in-workspace) OR a published binary in your own repo - 1 line in workspace `Cargo.toml` (if in-workspace) -- optional: add `ScenarioId::EdgeCache` to `core::registry` +- optional: add `DeploymentModelId::EdgeCache` to `core::registry` -Typical size: ~500-2000 LOC depending on how scenario-specific the rules / cost model / emitter are. If the scenario accepts core's defaults everywhere, ~300 LOC is realistic. +Typical size: ~500-2000 LOC depending on how deployment-model-specific the rules / cost model / emitter are. If the deployment model accepts core's defaults everywhere, ~300 LOC is realistic. -### Standalone binary for one scenario +### Standalone binary for one deployment model -If you want a slim binary that runs only one scenario: +If you want a slim binary that runs only one deployment model: ``` bin/asap-edge-cache/ -├── Cargo.toml # depends only on runtime + scenario-edge-cache, not other scenarios -└── src/main.rs # registers only EdgeCacheScenario; no DataFusion, no query YAML +├── Cargo.toml # depends only on runtime + deployment-model-edge-cache, not other deployment models +└── src/main.rs # registers only EdgeCacheDeploymentModel; no DataFusion, no query YAML ``` -Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql-parser` unless this scenario needs it. Useful when a deployment is dedicated to one scenario (e.g. an edge-caching-only service). +Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql-parser` unless this deployment model needs it. Useful when a deployment is dedicated to one deployment model (e.g. an edge-caching-only service). ## 10. Wire protocols — what changes, what doesn't @@ -997,7 +1033,7 @@ Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql- **Internal** (controller's own surface, still HTTP+JSON for now): - `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artifacts (URIs). `/api/v1/plan` proxies to this. - `GET /plans/:id` — plan inspection -- `GET /status` — runtime + scenario health +- `GET /status` — runtime + deployment model health **New** (internal-only, proto): - `proto/asap_control.proto` defines `Plan`, `PlanNode`, `Expr` for persistence + store-internal serialization. Not on the wire between services. Optional; initial migration skips this and persists via `serde_json`. @@ -1012,9 +1048,9 @@ resolver = "2" members = [ "crates/core", "crates/runtime", - "crates/scenario-lifecycle", - "crates/scenario-query", - "crates/scenario-fusion", + "crates/deployment-model-asaplifecycle", + "crates/deployment-model-asapquery", + "crates/deployment-model-asapfusion", "crates/control-proto", "crates/testing", "bin/asap-controller", @@ -1031,40 +1067,40 @@ serde = { version = "1", features = ["derive"] } serde_yaml = "0.9" tracing = "0.1" thiserror = "1" -# scenario-specific deps kept in scenario crates +# deployment-model-specific deps kept in deployment model crates ``` Core depends only on `serde`, `tracing`, `thiserror`, and small utility crates. No HTTP, no DataFusion, no OpAMP. Runtime depends on core + `axum` + `reqwest` + `tokio-tungstenite` + `prost` (OpAMP proto). -Scenarios depend on core. Scenario-query also pulls in `promql-parser`, `sqlparser`. Scenario-fusion pulls in `datafusion` + `arrow`. Scenario-lifecycle pulls in `sqlparser` + `promql-parser` + the cost-model math crates. +Deployment models depend on core. deployment-model-asapquery also pulls in `promql-parser`, `sqlparser`. deployment-model-asapfusion pulls in `datafusion` + `arrow`. deployment-model-asaplifecycle pulls in `sqlparser` + `promql-parser` + the cost-model math crates. -This matters: a user who only wants `scenario-fusion` (e.g., an offline benchmark) gets DataFusion but NOT axum/OpAMP. +This matters: a user who only wants `deployment-model-asapfusion` (e.g., an offline benchmark) gets DataFusion but NOT axum/OpAMP. ## 12. Open questions -1. **Do we need a cross-scenario cost model?** Today DC's lifecycle planner and asap-planner-rs's query planner have overlapping but not identical cost models. Answer for now: keep them separate in scenario crates; let them re-converge organically. If a third scenario needs the same model, lift at that point. +1. **Do we need a cross-deployment-model cost model?** Today DC's lifecycle planner and asap-planner-rs's query planner have overlapping but not identical cost models. Answer for now: keep them separate in deployment model crates; let them re-converge organically. If a third deployment model needs the same model, lift at that point. -2. **Where does the backend's `ControllerClient.create_plan` call land?** Initially: HTTP `POST /api/v1/plan` on the controller, handled by `scenario-lifecycle` (same as today). Long-term: could route by `QuerySpec.scenario` to a different planner, but that's a follow-up. +2. **Where does the backend's `ControllerClient.create_plan` call land?** Initially: HTTP `POST /api/v1/plan` on the controller, handled by `deployment-model-asaplifecycle` (same as today). Long-term: could route by `QuerySpec.deployment_model` to a different planner, but that's a follow-up. -3. **Which scenario owns `StreamingConfig` YAML emission?** Both `scenario-lifecycle` and `scenario-query` emit it today (DC's `config/generate_streaming_config_yaml` and `output/generator.rs` in `asap-planner-rs`). Plan: **one emitter in `scenario-query`**, called from both. This is why `scenario-lifecycle` depends on `scenario-query` in the dependency graph (asymmetric — query does not depend on lifecycle). +3. **Which deployment model owns `StreamingConfig` YAML emission?** Both `deployment-model-asaplifecycle` and `deployment-model-asapquery` emit it today (DC's `config/generate_streaming_config_yaml` and `output/generator.rs` in `asap-planner-rs`). Plan: **one emitter in `deployment-model-asapquery`**, called from both. This is why `deployment-model-asaplifecycle` depends on `deployment-model-asapquery` in the dependency graph (asymmetric — query does not depend on lifecycle). -4. **Do we vendor DataFusion's IR into core?** No. `scenario-fusion` owns its DataFusion-flavored plan; `core::plan::Plan` stays an enum with a variant that wraps a `datafusion::LogicalPlan` behind a feature flag. Core itself never reaches into DataFusion types. +4. **Do we vendor DataFusion's IR into core?** No. `deployment-model-asapfusion` owns its DataFusion-flavored plan; `core::plan::Plan` stays an enum with a variant that wraps a `datafusion::LogicalPlan` behind a feature flag. Core itself never reaches into DataFusion types. 5. **OpAMP proto: vendored, or from crates.io?** Today DC vendors. Recommend: keep vendored in `proto/opamp.proto`, generate via `prost-build` in `crates/control-proto`. Same thing DC does today, just moved. -6. **What happens to ASAPQuery-backend's `asap-planner-rs` directory post-migration?** Deleted. ASAPQuery-backend's docker-compose drops the `asap-planner-rs` init container; the controller's `scenario-query` now runs in-process (service mode) or as the `asap-query` CLI (one-shot mode). The ASAPQuery-backend repo shrinks by two directories. +6. **What happens to ASAPQuery-backend's `asap-planner-rs` directory post-migration?** Deleted. ASAPQuery-backend's docker-compose drops the `asap-planner-rs` init container; the controller's `deployment-model-asapquery` now runs in-process (service mode) or as the `asap-query` CLI (one-shot mode). The ASAPQuery-backend repo shrinks by two directories. -7. **Versioning?** Start at `0.1.0` on the workspace. Scenarios can rev independently later via per-crate versions, but initially lockstep. +7. **Versioning?** Start at `0.1.0` on the workspace. Deployment models can rev independently later via per-crate versions, but initially lockstep. -8. **What if a future scenario can't fit a tree L2?** L2 is currently a per-language tree, mandatory for every scenario. asap-planner-rs's Phase 4 conformance cost (reverse-engineering PromQL pattern templates into a `PromqlLogicalPlan` tree) was taken deliberately to keep the architecture uniform. If a future scenario's source language doesn't map naturally onto a tree (e.g. a constraint-based or dataflow-graph query language), that's the moment to re-examine the L2 contract — the `core::logical_plan` module is a single Rust trait + per-language types, not a deep assumption baked across the codebase. Until then, L2 = tree. +8. **What if a future deployment model can't fit a tree L2?** L2 is currently a per-language tree, mandatory for every deployment model. asap-planner-rs's Phase 4 conformance cost (reverse-engineering PromQL pattern templates into a `PromqlLogicalPlan` tree) was taken deliberately to keep the architecture uniform. If a future deployment model's source language doesn't map naturally onto a tree (e.g. a constraint-based or dataflow-graph query language), that's the moment to re-examine the L2 contract — the `core::logical_plan` module is a single Rust trait + per-language types, not a deep assumption baked across the codebase. Until then, L2 = tree. -9. **Scenario placement — in ASAPController workspace, or in own repo?** Both supported, same architecture either way (see §8 "Scenario placement"). Default: `scenario-lifecycle` and `scenario-query` in ASAPController (easy cross-scenario changes, shared YAML emitter); `scenario-fusion` in the `asap-fusion` repo (research cadence). A new scenario picks based on team ownership and release cadence preferences. +9. **Deployment model placement — in ASAPController workspace, or in own repo?** Both supported, same architecture either way (see §8 "Deployment model placement"). Default: `deployment-model-asaplifecycle` and `deployment-model-asapquery` in ASAPController (easy cross-deployment-model changes, shared YAML emitter); `deployment-model-asapfusion` in the `asap-fusion` repo (research cadence). A new deployment model picks based on team ownership and release cadence preferences. -10. **How far do shared L4/L5 rules live in core before they become scenario-specific?** The rule of thumb: if ≥2 current scenarios would use it, it lives in `core::optimizer::rules::*`. If only 1 scenario uses it *and* it depends on scenario-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the scenario crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `scenario-lifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the scenario; lift to core once a second scenario wants it. Don't pre-emptively generalize. +10. **How far do shared L4/L5 rules live in core before they become deployment-model-specific?** The rule of thumb: if ≥2 current deployment models would use it, it lives in `core::optimizer::rules::*`. If only 1 deployment model uses it *and* it depends on deployment-model-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the deployment model crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `deployment-model-asaplifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the deployment model; lift to core once a second deployment model wants it. Don't pre-emptively generalize. -11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP scenarios)?** Already handled — see §3 "Data-model support" and §6 `core::sketch_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP scenario picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. +11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP deployment models)?** Already handled — see §3 "Data-model support" and §6 `core::sketch_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP deployment model picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. ### Resolved during the L3 IR cleanup (see §6 `core::sketch_algebra`) @@ -1094,13 +1130,53 @@ The L4 rule engine + `OptimizerRule` trait are **primitive-agnostic** — nothin The guiding principle: **same framework, different rule.** A new primitive class is never a translation-layer change — only a new rule, a new cost-model entry, and (if it introduces a new runtime op) a new physical operator in L5. -## 14. Success criteria +## 14. Glossary + +Terms that are project-specific or that get conflated. Where the term has a Rust counterpart, the type is shown in backticks. + +### Architecture roles + +- **Deployment model** — A concrete bundle of (L4 rule choices) + (L5 topology) + (emitter), packaged as a `deployment-model-*` crate. Three exist today: `asaplifecycle`, `asapquery`, `asapfusion`. +- **Stage** (`StageId`) — A *categorical tier* in the data lifecycle: edge / gateway / backend / in-process. Roles, not instances. Declared by the deployment model's `TopologyDescriptor`. +- **Executor** (`Executor`) — A *concrete runtime instance* occupying a stage: a specific OTel agent, the ASAPQuery-backend process, a DataFusion `SessionContext`. One stage may have N executors (e.g. 50 edge agents). Carries `id`, `stage: StageId`, `capabilities`, `address`. +- **Topology** (`TopologyDescriptor`) — Declares which stages exist + how they connect (3-stage / 1-stage / 0-stage). Categorical, not instance-level. +- **DeploymentConstraints** — Trait object owned by each deployment model. Carries memory budgets, network topology, available sketch backends, and the registered `Executor` list. Threaded through L4 rules and the stage allocator. + +### Layer drivers + +- **RuleEngine** — Generic L4 driver in core (fixed-point iteration + cycle detection + priority ordering). One implementation; same code for every deployment model. +- **OptimizerRule** — Trait for an L4 rewrite. Categories: `PushDown | Fusion | Elim | Bind | StageRouting`. Deployment models pick which rules from `core::optimizer::rules` + add their own. +- **CostModel** — Trait scoring a plan's accuracy / latency / dollars. Generic implementations in core; deployment-model-specific cost models (DC's delta / online / pareto / TCO) live in their crates. +- **StageAllocator** — Generic L5 algorithm in core. Colors the L4-bound DAG by `StageId`. **Stage-granularity only — no executor knowledge.** One implementation; same code for every deployment model. +- **PhysicalPlanner** — Per-deployment-model L5 *driver* (one impl per deployment model). Calls `StageAllocator`, then fans the per-stage sub-DAGs out to executors via `DeploymentConstraints::executors()`, then produces the deployment-specific output type. Compare to allocator: planner = whole L5 driver, allocator = the stage-coloring step it uses. +- **PlanEmitter** — Per-deployment-model trait that serialises the planner's output to its wire format (OpAMP `RemoteConfig` / `streaming_config.yaml` / rewritten DataFusion `LogicalPlan`). + +### IR by layer + +- **L1 — query language** — Raw query string + parser (PromQL / SQL / DataFusion / ElasticDSL). +- **L2 — logical plan** — Per-language relational algebra tree (`PromqlLogicalPlan`, `SqlLogicalPlan`, etc.). +- **L3 — sketch algebra** (`QueryExpr`) — Symbolic, intent-only IR. No sketch type, no params. `AggIntent` carries accuracy targets only. +- **L4 — sketch-bound IR** (`SketchExpr`) — Same DAG shape, sketches now committed (kind + params). +- **L5 — physical plan** — Stage-assigned, executor-targeted, ready to serialize. Produced by `PhysicalPlanner`, written out by `PlanEmitter`. + +### Metadata sources (see §6 "DAG schema, DB schema, sketch catalog") + +- **DAG schema** (`Schema`) — Columns + types + `unique_keys` carried on every L3/L4/L5 edge. Type-checked locally at each node. +- **DB schema** / **source schema** (`SchemaCatalog`) — Data-plane metadata (Prometheus TSDB / SQL `information_schema` / DataFusion catalog). Read by `core::lower::*` for L1→L2 symbol resolution. +- **Sketch catalog** (`SketchCatalog`) — Runtime registry of available primitives: what sketches exist, mergeability, deletability, parameter ranges, accuracy model. Consulted by L4 binding rules. **Not a schema** — a catalog of available primitives, not a description of a stream. + +### Workload + identity + +- **QueryWorkload** — Top-level controller input: one or more `QuerySpec`s plus workload features (batch vs streaming, reuse opportunity, data source). Arrives via HTTP POST, OpAMP capability-miss callback, YAML file, or query-log replay. +- **QuerySpec** — A single query: source-language string + accuracy / latency / cost target. + +## 15. Success criteria The migration is done when: 1. `asap-controller` binary runs and passes DC controller's existing integration tests (OpAMP push, backend config POST, SLA replan). 2. `asap-query` binary takes the same YAML input asap-planner-rs does today and produces byte-identical `streaming_config.yaml` + `inference_config.yaml` (fuzz-test against a corpus of fixtures). 3. ASAPQuery-backend's docker-compose no longer starts `asap-planner-rs`; the controller handles both shapes. -4. `asap-fusion`'s microbenchmarks still run under `scenario-fusion` with identical numbers. +4. `asap-fusion`'s microbenchmarks still run under `deployment-model-asapfusion` with identical numbers. 5. The `DataCollector/controller/`, `ASAPQuery/asap-planner-rs/`, `ASAPQuery-backend/asap-planner-rs/`, and `asap-fusion/` directories are deletable (or already deleted) without breaking any currently-running deployment. -6. A new hypothetical scenario can be added with zero changes outside its crate + one line in `bin/asap-controller/main.rs`. +6. A new hypothetical deployment model can be added with zero changes outside its crate + one line in `bin/asap-controller/main.rs`. diff --git a/docs/migration-plan.md b/docs/migration-plan.md index 1cbe13b4..147f203e 100644 --- a/docs/migration-plan.md +++ b/docs/migration-plan.md @@ -36,28 +36,28 @@ Ordered by risk, lowest first. Each phase is a single PR unless noted. ### Phase 1 — Core extraction: all-5-layer shared infrastructure -**Scope:** Lift the DC controller's **layers 1–3** (query parsing + per-language algebra + sketch-algebra IR) into `crates/core/` verbatim, AND lift L4 and L5 **shared infrastructure**: rule engine, shared rule library, cost-model traits, `PhysicalPlanner` trait, stage allocator framework, sketch catalogue. Scenarios in later phases pick from this common base and only add scenario-specific pieces. +**Scope:** Lift the DC controller's **layers 1–3** (query parsing + per-language algebra + sketch-algebra IR) into `crates/core/` verbatim, AND lift L4 and L5 **shared infrastructure**: rule engine, shared rule library, cost-model traits, `PhysicalPlanner` trait, stage allocator framework, sketch catalogue. Deployment models in later phases pick from this common base and only add deployment-model-specific pieces. -Per design §3, these five layers' infrastructure is query-language-independent AND scenario-independent; the concrete rule set, topology, and emitter are scenario-specific. +Per design §3, these five layers' infrastructure is query-language-independent AND deployment-model-independent; the concrete rule set, topology, and emitter are deployment-model-specific. **Work — L1-3 (straight lift from DC):** - **`core::query_language`** (L1): lift `DataCollector/controller/src/query_parser/{promql,sql,mod}.rs` → `core/src/query_language/`. One module per language. - **`core::logical_plan`** (L2): lift per-language algebra tree definitions → `core/src/logical_plan/`. One `enum LogicalPlan` per language. Add `core::logical_plan::datafusion` as a thin re-export of `datafusion::logical_expr::LogicalPlan` for fusion's L2. -- **`core::sketch_algebra`** (L3): lift `algebra/{expr,directory}.rs` → `core/src/sketch_algebra/`. Enforce **intent-only** — `AggIntent` carries accuracy target, never sketch type. **Generalize `QueryExpr::Scan` over data model**: introduce a `Source` sum (`TimeSeries` / `Table` / `Join` variants) inside `Scan`, and a `DataModel` enum (`TimeSeries` / `Tabular` / `Any`). DC's current scan logic becomes `Source::TimeSeries`; `Source::Table` is new (stub impl fine for Phase 1 — scenario-fusion fills it in Phase 3). Add `AggIntent::requires() -> DataModel` so L4 rules can skip non-applicable intents. See design §3 "Data-model support" and §6 `core::sketch_algebra`. ~300 LOC on top of the DC lift. +- **`core::sketch_algebra`** (L3): lift `algebra/{expr,directory}.rs` → `core/src/sketch_algebra/`. Enforce **intent-only** — `AggIntent` carries accuracy target, never sketch type. **Generalize `QueryExpr::Scan` over data model**: introduce a `Source` sum (`TimeSeries` / `Table` / `Join` variants) inside `Scan`, and a `DataModel` enum (`TimeSeries` / `Tabular` / `Any`). DC's current scan logic becomes `Source::TimeSeries`; `Source::Table` is new (stub impl fine for Phase 1 — deployment-model-asapfusion fills it in Phase 3). Add `AggIntent::requires() -> DataModel` so L4 rules can skip non-applicable intents. See design §3 "Data-model support" and §6 `core::sketch_algebra`. ~300 LOC on top of the DC lift. - **`core::lower`**: lift `algebra/lower.rs` (L1→L2→L3 passes) → `core/src/lower/`. One `lower_` entry point per language. Each lowering produces the appropriate `Source` variant (PromQL / SQL → `Source::TimeSeries`; DataFusion → `Source::Table`). **Work — L4 framework (NEW):** - **`core::optimizer::engine`**: rule driver — fixed-point iteration, cycle detection, priority ordering. Generic over `OptimizerRule`. - **`core::optimizer::rule` trait**: `OptimizerRule` + `RuleCategory` enum (`PushDown | Fusion | Elim | Bind | StageRouting`). Priority u16. `apply(&expr, &constraints) -> Option`. -- **`core::optimizer::rules`**: shared rule library. Lift the **scenario-agnostic rules** from DC's `algebra/optimizer.rs`: +- **`core::optimizer::rules`**: shared rule library. Lift the **deployment-model-agnostic rules** from DC's `algebra/optimizer.rs`: - `BindKllOnQuantile` (intent → KLL with params from accuracy) - `BindCmsOnCount` (+ `BindCmsOnSum`) - `BindHllOnCardinality` - `BindDdsketchOnQuantile` (alternative to KLL) - `FusionPassthrough` (Aggregate over Filter → push Filter under) - `ElimNoopFilter`, `ElimNoopSort`, `ElimNoopLimit` - - Scenario-specific rules (stage-aware push-down, TCO-aware deferral) STAY in `scenario-lifecycle` — not lifted. -- **`core::optimizer::cost`**: `CostModel` trait + `Accuracy` / `Latency` / `Dollars` value types. Lift DC's **generic** cost functions (memory budget, accuracy degradation curve). Scenario-specific cost models (delta, online, pareto, tco) STAY in `scenario-lifecycle`. + - DeploymentModel-specific rules (stage-aware push-down, TCO-aware deferral) STAY in `deployment-model-asaplifecycle` — not lifted. +- **`core::optimizer::cost`**: `CostModel` trait + `Accuracy` / `Latency` / `Dollars` value types. Lift DC's **generic** cost functions (memory budget, accuracy degradation curve). DeploymentModel-specific cost models (delta, online, pareto, tco) STAY in `deployment-model-asaplifecycle`. - **`core::plan`**: `DeploymentConstraints` trait — memory budgets, network topology descriptor, available sketch backends. **Work — L5 framework (NEW):** @@ -70,7 +70,7 @@ Per design §3, these five layers' infrastructure is query-language-independent - **`core::pipeline`**: the L1→…→L5 driver. ~300 LOC. - **`core::workload`**: lift `QuerySpec` + add `QueryWorkload` sum type. - **`core::emit`**: `PlanEmitter` trait + `EmitError`. -- **`core::registry`**: `ScenarioRegistry`, `ScenarioId` enum. +- **`core::registry`**: `DeploymentModelRegistry`, `DeploymentModelId` enum. **Testing:** - Port DC's existing `query_parser/` + `algebra/` unit tests verbatim. All pass after the move. @@ -83,7 +83,7 @@ Per design §3, these five layers' infrastructure is query-language-independent **Decision points:** 1. `asap_types` — publish to a private git tag in Phase 0. 2. `promql-parser` / `sqlparser` versions — adopt DC's newer versions. -3. **Which DC rules go into `core::optimizer::rules`?** Rule of thumb: if it depends only on `QueryExpr` + `DeploymentConstraints` (no DC-specific types), lift. If it reaches into DC's stage graph, backend_client, or OTel YAML shape, it stays in `scenario-lifecycle`. Default: lift `Bind*`, `Elim*`, `FusionPassthrough`; keep `StageAwarePushDown`, `TransmissionCostRewrite` in scenario-lifecycle. +3. **Which DC rules go into `core::optimizer::rules`?** Rule of thumb: if it depends only on `QueryExpr` + `DeploymentConstraints` (no DC-specific types), lift. If it reaches into DC's stage graph, backend_client, or OTel YAML shape, it stays in `deployment-model-asaplifecycle`. Default: lift `Bind*`, `Elim*`, `FusionPassthrough`; keep `StageAwarePushDown`, `TransmissionCostRewrite` in deployment-model-asaplifecycle. **Exit criteria:** `cargo build` green; DC's parser + algebra + subset of optimizer unit tests pass from `core/`; `StageAllocator` unit-tested against all pre-baked topologies; golden-file corpus locked in. @@ -109,43 +109,43 @@ Per design §3, these five layers' infrastructure is query-language-independent --- -### Phase 3 — `scenario-fusion`: first thin scenario +### Phase 3 — `deployment-model-asapfusion`: first thin deployment model -**Scope:** Land `scenario-fusion` as a thin scenario that picks rules from core's shared library. Lowest-risk because fusion's existing `SketchConfigRule` maps directly onto core's `BindKllOnQuantile` + `BindCmsOnCount` + `BindHllOnCardinality` that Phase 1 already lifted. +**Scope:** Land `deployment-model-asapfusion` as a thin deployment model that picks rules from core's shared library. Lowest-risk because fusion's existing `SketchConfigRule` maps directly onto core's `BindKllOnQuantile` + `BindCmsOnCount` + `BindHllOnCardinality` that Phase 1 already lifted. -**Decision — where does `scenario-fusion` live?** Default: **in the `asap-fusion` repo**, as a crate that depends on `asap-control-core` + `asap-control-optimizer`. asap-fusion is a research project with independent release cadence; keeping it in-tree in its own repo preserves that. If the team prefers lockstep, the alternative is `crates/scenario-fusion/` in ASAPController workspace — architecture is identical either way (see design §8). +**Decision — where does `deployment-model-asapfusion` live?** Default: **in the `asap-fusion` repo**, as a crate that depends on `asap-control-core` + `asap-control-optimizer`. asap-fusion is a research project with independent release cadence; keeping it in-tree in its own repo preserves that. If the team prefers lockstep, the alternative is `crates/deployment-model-asapfusion/` in ASAPController workspace — architecture is identical either way (see design §8). This plan assumes the in-repo-at-asap-fusion placement. If in-workspace is chosen, swap paths accordingly. **Work:** -- In `asap-fusion/`: add new crate `asap-fusion-scenario/` that depends on `asap-control-core`. Move the translator + optimizer + executor here. -- **Fill in `Source::Table` lowering**: Phase 1 stubbed this; now implement the L2 (DataFusion `LogicalPlan`) → L3 (`QueryExpr` with `Source::Table`) lowering pass in `core::lower::datafusion` — or, if fusion prefers, keep the lowering inside the scenario crate and use the `Source::Table` type directly. Either way, Phase 1's stub becomes concrete. -- `impl Scenario for FusionScenario`: +- In `asap-fusion/`: add new crate `deployment-model-asapfusion/` that depends on `asap-control-core`. Move the translator + optimizer + executor here. +- **Fill in `Source::Table` lowering**: Phase 1 stubbed this; now implement the L2 (DataFusion `LogicalPlan`) → L3 (`QueryExpr` with `Source::Table`) lowering pass in `core::lower::datafusion` — or, if fusion prefers, keep the lowering inside the deployment model crate and use the `Source::Table` type directly. Either way, Phase 1's stub becomes concrete. +- `impl DeploymentModel for ASAPFusionDeploymentModel`: - `rules()` picks `BindKllOnQuantile`, `BindCmsOnCount`, `BindHllOnCardinality` from `core::optimizer::rules::*` (drops asap-fusion's `SketchConfigRule` in favor of core's rules). - - Adds scenario-specific `HashModeRule` (stays local — depends on DataFusion-specific types). + - Adds deployment-model-specific `HashModeRule` (stays local — depends on DataFusion-specific types). - `topology()` returns `core::physical::topology::ZeroStage`. - `emitter()` returns the DataFusion `LogicalPlan` rewriter. -- Port the microbenchmarks to `asap-fusion-scenario/benches/`. Verify numbers within 5% of pre-migration baseline. +- Port the microbenchmarks to `deployment-model-asapfusion/benches/`. Verify numbers within 5% of pre-migration baseline. - Legacy `asap-fusion` crate becomes a thin shim that re-exports the new crate, so in-flight benchmark harnesses keep compiling. **Risk:** low — no service consumes asap-fusion today; only downstream is its own benches. The rule-lift cleanup (`SketchConfigRule` → core rules) is the main change; golden-file test over the 3 existing rewrites catches any divergence. -**Exit criteria:** `cargo bench -p asap-fusion-scenario` produces numbers within 5% of pre-migration; 3 rewrites documented in `asap-fusion/TESTS.md` produce identical output. +**Exit criteria:** `cargo bench -p deployment-model-asapfusion` produces numbers within 5% of pre-migration; 3 rewrites documented in `asap-fusion/TESTS.md` produce identical output. **PR size:** ~1800 LOC (asap-fusion's ~3k lines minus what folds into core's shared rule library). --- -### Phase 4 — `scenario-query`: migrate asap-planner-rs's newer copy +### Phase 4 — `deployment-model-asapquery`: migrate asap-planner-rs's newer copy -**Scope:** Move `ASAPQuery-backend/asap-planner-rs/` (the newer copy) into `crates/scenario-query/`, **and refactor it to fit the 5-layer model**: reverse-engineer PromQL pattern templates into an L2 tree, and split L3 intent from L4 sketch binding. Delete the older copy in `ASAPQuery/`. +**Scope:** Move `ASAPQuery-backend/asap-planner-rs/` (the newer copy) into `crates/deployment-model-asapquery/`, **and refactor it to fit the 5-layer model**: reverse-engineer PromQL pattern templates into an L2 tree, and split L3 intent from L4 sketch binding. Delete the older copy in `ASAPQuery/`. This phase is larger than a straight lift because two structural conformance changes are bundled in. **Work — the lift:** - Diff the two `asap-planner-rs` copies. Newer (backend) copy wins. **Port only the newer; discard the older.** -- Copy `ASAPQuery-backend/asap-planner-rs/src/{lib,main}.rs` + `src/{planner,output,query_log,prometheus_client}/` → `crates/scenario-query/src/`. -- `impl Scenario for QueryScenario` — registers YAML emitters (`StreamingConfigEmitter` + `InferenceConfigEmitter`) and HTTP route `POST /plan/query`. +- Copy `ASAPQuery-backend/asap-planner-rs/src/{lib,main}.rs` + `src/{planner,output,query_log,prometheus_client}/` → `crates/deployment-model-asapquery/src/`. +- `impl DeploymentModel for ASAPQueryDeploymentModel` — registers YAML emitters (`StreamingConfigEmitter` + `InferenceConfigEmitter`) and HTTP route `POST /plan/query`. - Port the CLI: `bin/asap-query/main.rs` with clap flags mirroring `asap-planner-rs`'s current CLI. **Work — L2 tree conformance (NEW):** @@ -158,13 +158,13 @@ This phase is larger than a straight lift because two structural conformance cha **Work — L3 / L4 split (NEW):** - In `core::lower::promql_to_sketch_algebra`: `PromqlLogicalPlan → QueryExpr` producing **intent-only** L3 (no sketch names). - `Statistic` enum → `AggIntent` subset (9 of DC's 25 variants). No sketch type, no sketch params at L3. -- In `scenario-query/src/rules.rs`: picks core's `BindKllOnQuantile`, `BindCmsOnCount`, etc. from Phase 1's shared rule library + adds a scenario-specific `PrecomputeEngineBindRule` (handles the precompute-engine-specific flavor: which sketches are available in ASAPQuery-backend's accumulator set, what params match the engine's config schema, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). -- Update `build_agg_configs_for_statistics` to call `core::optimizer::engine::RuleEngine::run()` with the scenario's rule set instead of the inline `map_statistic_to_precompute_operator` fusion. +- In `deployment-model-asapquery/src/rules.rs`: picks core's `BindKllOnQuantile`, `BindCmsOnCount`, etc. from Phase 1's shared rule library + adds a deployment-model-specific `PrecomputeEngineBindRule` (handles the precompute-engine-specific flavor: which sketches are available in ASAPQuery-backend's accumulator set, what params match the engine's config schema, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). +- Update `build_agg_configs_for_statistics` to call `core::optimizer::engine::RuleEngine::run()` with the deployment model's rule set instead of the inline `map_statistic_to_precompute_operator` fusion. - `IntermediateAggConfig` loses its inline sketch binding; it's now an L4 output type produced by `PrecomputeEngineBindRule`. **Work — topology + emitter:** -- `scenario-query/src/topology.rs`: declare `core::physical::topology::SingleStage` (backend-only). -- `scenario-query/src/emit/`: `StreamingConfigEmitter` + `InferenceConfigEmitter`. Lift from asap-planner-rs's `output/generator.rs`. These are the authoritative YAML emitters (scenario-lifecycle calls them). +- `deployment-model-asapquery/src/topology.rs`: declare `core::physical::topology::SingleStage` (backend-only). +- `deployment-model-asapquery/src/emit/`: `StreamingConfigEmitter` + `InferenceConfigEmitter`. Lift from asap-planner-rs's `output/generator.rs`. These are the authoritative YAML emitters (deployment-model-asaplifecycle calls them). **Testing:** - **Golden-file test**: capture a corpus of today's `asap-planner-rs` inputs → outputs. The new CLI must produce byte-identical output. Put under `bin/asap-query/tests/golden/`. This is the non-negotiable safety net for the refactor. @@ -186,28 +186,28 @@ This phase is larger than a straight lift because two structural conformance cha --- -### Phase 5 — `scenario-lifecycle`: DC's remaining scenario-specific pieces +### Phase 5 — `deployment-model-asaplifecycle`: DC's remaining deployment-model-specific pieces -**Scope:** Land `scenario-lifecycle` as a thin scenario. L1-3 + L4 shared rules + L5 framework already moved in Phase 1; only DC's **scenario-specific** L4/L5 pieces remain: stage-aware rewrite rules, cost models, the 3-stage `ThreeStage` topology's DC-specific constraints, OpAMP + backend emitters, SLA replanner wiring. +**Scope:** Land `deployment-model-asaplifecycle` as a thin deployment model. L1-3 + L4 shared rules + L5 framework already moved in Phase 1; only DC's **deployment-model-specific** L4/L5 pieces remain: stage-aware rewrite rules, cost models, the 3-stage `ThreeStage` topology's DC-specific constraints, OpAMP + backend emitters, SLA replanner wiring. -**Work — scenario crate:** -- **Rule selection**: `scenario-lifecycle/src/rules.rs` picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules: +**Work — deployment model crate:** +- **Rule selection**: `deployment-model-asaplifecycle/src/rules.rs` picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules: - `StageAwarePushDown` — pushes ops toward edge stage when data size + edge memory budget permit - `TransmissionCostRewrite` — uses the TCO model to defer aggregation when it saves bandwidth - `DeltaRewrite` — replaces raw sends with delta-encoded sends when the online cost model says it pays off Each impls `core::optimizer::rule::OptimizerRule`. -- **Cost models**: `scenario-lifecycle/src/cost.rs` — lift `DataCollector/controller/src/planner/{delta_cost_model,online_cost_model,pareto,tco,rules,baseline,cost_model}.rs`. Each impls `core::optimizer::cost::CostModel`. These stay scenario-specific because they encode DC's deployment assumptions (edge memory, WAN bandwidth, multi-tenant). -- **Topology**: `scenario-lifecycle/src/topology.rs` — uses `core::physical::topology::ThreeStage`; only scenario-specific bit is declaring which stage roles map to which OpAMP `AgentRole` enum values. -- **Emitters**: `scenario-lifecycle/src/emit/` — `OpAmpRemoteConfigEmitter` (per-role OTel YAML, lifted from `DataCollector/controller/src/config/generate_{agent,backend}_config*.rs`) + `AsapqueryBackendConfigEmitter` (calls `scenario-query::StreamingConfigEmitter` for the YAML bytes, then POSTs via `runtime::backend_client`). -- `impl Scenario for LifecycleScenario` — registers HTTP routes `POST /plan` and `POST /replan`. +- **Cost models**: `deployment-model-asaplifecycle/src/cost.rs` — lift `DataCollector/controller/src/planner/{delta_cost_model,online_cost_model,pareto,tco,rules,baseline,cost_model}.rs`. Each impls `core::optimizer::cost::CostModel`. These stay deployment-model-specific because they encode DC's deployment assumptions (edge memory, WAN bandwidth, multi-tenant). +- **Topology**: `deployment-model-asaplifecycle/src/topology.rs` — uses `core::physical::topology::ThreeStage`; only deployment-model-specific bit is declaring which stage roles map to which OpAMP `AgentRole` enum values. +- **Emitters**: `deployment-model-asaplifecycle/src/emit/` — `OpAmpRemoteConfigEmitter` (per-role OTel YAML, lifted from `DataCollector/controller/src/config/generate_{agent,backend}_config*.rs`) + `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery::StreamingConfigEmitter` for the YAML bytes, then POSTs via `runtime::backend_client`). +- `impl DeploymentModel for ASAPLifecycleDeploymentModel` — registers HTTP routes `POST /plan` and `POST /replan`. **Work — runtime wiring:** - **OpAMP server**: move `DataCollector/controller/src/opamp/` → `runtime/opamp/`. OpAMP is a transport, not a plan-type, so it lives in runtime. - **Replanner + monitor**: move `DataCollector/controller/src/{replan,monitor}.rs` → `runtime/{replan,monitor}/`. -- **Dispatch**: `runtime/http/` routes `POST /plan` through `ScenarioRegistry` instead of calling DC's planner directly. +- **Dispatch**: `runtime/http/` routes `POST /plan` through `DeploymentModelRegistry` instead of calling DC's planner directly. **Integration test:** -- Send a `QuerySpec` over HTTP → core runs L1-3 → `LifecycleScenario`'s rule set fires (core rules + DC-specific rules) → `StageAllocator` with `ThreeStage` topology produces staged plan → emitters produce OTel YAML + `StreamingConfig` → fake backend asserts POST body matches golden file. Port DC's existing test of this shape. +- Send a `QuerySpec` over HTTP → core runs L1-3 → `ASAPLifecycleDeploymentModel`'s rule set fires (core rules + DC-specific rules) → `StageAllocator` with `ThreeStage` topology produces staged plan → emitters produce OTel YAML + `StreamingConfig` → fake backend asserts POST body matches golden file. Port DC's existing test of this shape. **Risk:** medium. Cost-model code (`delta/online/pareto/tco`) is ~40KB of subtle logic; port mechanically — do not touch the logic. The generic pieces already in core (stage allocator, bind rules) have been unit-tested in Phase 1, reducing this phase's surface area meaningfully. @@ -230,7 +230,7 @@ This phase is larger than a straight lift because two structural conformance cha - Capability-miss callback in `SimpleEngine` continues to `POST /api/v1/plan` on the controller — no code change, same endpoint. - **ASAPQuery PR**: delete `asap-planner-rs/` (the older copy). Any downstream refs in ASAPQuery update to point at ASAPController's `asap-query` binary. - **DataCollector PR**: delete `controller/` directory. Anyone who ran `cargo build -p controller` now runs `cargo build -p asap-controller` against the new repo. Update DC's README to redirect. -- **asap-fusion PR**: convert the repo to a thin shim that re-exports `scenario-fusion` OR archive the repo outright if no external user depends on it. Recommendation: **archive**. +- **asap-fusion PR**: convert the repo to a thin shim that re-exports `deployment-model-asapfusion` OR archive the repo outright if no external user depends on it. Recommendation: **archive**. **Risk:** medium. Deletion is final but reversible via `git revert`; actual blast radius is low because Phase 5 has already proven the new binary works. @@ -243,7 +243,7 @@ This phase is larger than a straight lift because two structural conformance cha --- -### Phase 7 — Core refactor: tighten traits now that 3 scenarios exist +### Phase 7 — Core refactor: tighten traits now that 3 deployment models exist **Scope:** Look at the shape of the `Planner` / `Optimizer` / `PlanRewriteRule` traits with 3 real implementations, tighten anything awkward. @@ -266,13 +266,13 @@ This phase is larger than a straight lift because two structural conformance cha | 0. Skeleton | 1 | ~200 | 1 day | | 1. Core — L1-3 + L4/L5 framework + shared rules + `Source`/`DataModel` | 1–2 | ~6500 | 1.5 weeks | | 2. Runtime HTTP+store | 1 | ~7700 | 3 days | -| 3. scenario-fusion (thin; rules fold into core library) | 1 | ~9500 | 1 week | -| 4. scenario-query (L2 tree + L3/L4 split + YAML + CLI) | 1 | ~14000 | 2 weeks | -| 5. scenario-lifecycle (cost models + scenario-specific rules + OpAMP wiring) | 2 | ~17500 | 1 week | +| 3. deployment-model-asapfusion (thin; rules fold into core library) | 1 | ~9500 | 1 week | +| 4. deployment-model-asapquery (L2 tree + L3/L4 split + YAML + CLI) | 1 | ~14000 | 2 weeks | +| 5. deployment-model-asaplifecycle (cost models + deployment-model-specific rules + OpAMP wiring) | 2 | ~17500 | 1 week | | 6. Cutover (delete in source repos) | 3–4 | ~18000 | 1 week | | 7. Core refactor | 1 | ~18500 | 3 days | -**Total**: ~6.5 weeks for one engineer. Phase 1 grew (+0.5w) because it now also lifts L4/L5 shared infrastructure. Phases 3 and 5 shrank correspondingly (-1w combined) because fusion's `SketchConfigRule`-style bindings and lifecycle's `StageAllocator` now come from core's shared library. Net: slightly faster total, much cleaner per-scenario crates (each ~1500-2500 LOC of scenario-specific code rather than 5000+). +**Total**: ~6.5 weeks for one engineer. Phase 1 grew (+0.5w) because it now also lifts L4/L5 shared infrastructure. Phases 3 and 5 shrank correspondingly (-1w combined) because fusion's `SketchConfigRule`-style bindings and lifecycle's `StageAllocator` now come from core's shared library. Net: slightly faster total, much cleaner per-deployment-model crates (each ~1500-2500 LOC of deployment-model-specific code rather than 5000+). Parallelisable if two people: lifecycle (Phase 5) can start as soon as Phase 1's L4/L5 framework lands (before Phase 4's L2 tree work). @@ -282,14 +282,14 @@ Parallelisable if two people: lifecycle (Phase 5) can start as soon as Phase 1's |---|---|---|---| | YAML emission byte-differences between old and new planner | Medium | Breaks ASAPQuery-backend's config consumer | Golden-file test corpus in Phase 4; fuzz harness over the YAML schema | | OpAMP protocol detail drifts during port | Low | Breaks agent config push | Port opamp/ verbatim in Phase 2's runtime crate; don't touch until it's proven to work | -| DataFusion version skew in scenario-fusion | Low | Benchmarks drift | Pin DataFusion version in workspace `Cargo.toml`; measure benches before + after | +| DataFusion version skew in deployment-model-asapfusion | Low | Benchmarks drift | Pin DataFusion version in workspace `Cargo.toml`; measure benches before + after | | Two asap-planner-rs copies have subtle diff we miss | Medium | Behaviour regression | Explicit diff + reconcile step in Phase 4; one authoritative copy (the backend's newer one) | | ASAPQuery-backend deploys break during cutover | Low | Production outage | Phase 6 only after Phase 5 fully validates; staged rollout of docker-compose change | | Scope creep — "while we're at it, redesign the cost model" | High | Doubles timeline | Explicit non-goal list in design doc §2; defer all rewrites to post-migration phases | -| Core traits are wrong; need to rev after scenarios land | Medium | Phase 7 PR larger than expected | Accept; Phase 7 is planned for exactly this. Don't over-design Phase 1. | +| Core traits are wrong; need to rev after deployment models land | Medium | Phase 7 PR larger than expected | Accept; Phase 7 is planned for exactly this. Don't over-design Phase 1. | | PromQL L2 tree reverse-engineering loses a pattern-match case planner relied on | Medium | Generated YAML diverges from legacy | Golden-file corpus covers all 5 pattern shapes + generic fallthrough; tree-shape inspection at L3 lowering is a direct 1:1 translation of today's pattern-match logic, not a reinterpretation | | L3/L4 split misses a case in `map_statistic_to_precompute_operator` | Medium | Wrong sketch selected for some `(Statistic, Treatment)` combos | Every branch of the current function must round-trip via the new L3→L4 path on the golden-file corpus; add table-driven unit tests enumerating every `Statistic × Treatment` combination | -| `AggIntent` as a 25-variant superset grows unbounded as scenarios add needs | Low | Core churn | Variants added only for intent shapes used by ≥1 shipped scenario; never speculative. | +| `AggIntent` as a 25-variant superset grows unbounded as deployment models add needs | Low | Core churn | Variants added only for intent shapes used by ≥1 shipped deployment model; never speculative. | ## Decision log (during migration) @@ -297,42 +297,42 @@ These are the questions that will come up mid-migration. Pre-decide as many as p 1. **Which version of `asap_types`?** ASAPQuery-backend's current one, pinned to a git SHA. 2. **Which `promql-parser` + `sqlparser` versions?** DC controller's (newer). ASAPQuery-backend's planner uses older versions; port it to the newer ones during Phase 4. -3. **How is `scenario-lifecycle`'s `StreamingConfig` YAML emitter related to `scenario-query`'s?** One emitter in `scenario-query`, called from `scenario-lifecycle`. Asymmetric dependency. -4. **Does `asap-controller` have feature flags to disable scenarios at build time?** Yes, one `--features lifecycle,query,fusion` with all three enabled by default. A minimal CLI binary (`asap-query`) disables lifecycle and fusion. +3. **How is `deployment-model-asaplifecycle`'s `StreamingConfig` YAML emitter related to `deployment-model-asapquery`'s?** One emitter in `deployment-model-asapquery`, called from `deployment-model-asaplifecycle`. Asymmetric dependency. +4. **Does `asap-controller` have feature flags to disable deployment models at build time?** Yes, one `--features lifecycle,query,fusion` with all three enabled by default. A minimal CLI binary (`asap-query`) disables lifecycle and fusion. 5. **Who owns the Cargo.lock?** ASAPController. Downstream repos (ASAPQuery-backend, DataCollector) do NOT depend on ASAPController as a Cargo path dep — they pull the published binary via Docker or pin a git SHA. -6. **Does `ControllerClient` on the ASAPQuery-backend side need changes?** No. It keeps POSTing to `/api/v1/plan`; the controller's routing layer dispatches to `scenario-lifecycle` (same as DC controller does today). -7. **L2 tree for asap-planner-rs — mandatory or optional?** Mandatory. planner's current template-catalogue approach is replaced with a proper `PromqlLogicalPlan` tree in Phase 4. The extra conformance work is taken to keep the 5-layer model uniform. See design doc §12 Q8 for the escape hatch if a future scenario genuinely can't fit a tree. +6. **Does `ControllerClient` on the ASAPQuery-backend side need changes?** No. It keeps POSTing to `/api/v1/plan`; the controller's routing layer dispatches to `deployment-model-asaplifecycle` (same as DC controller does today). +7. **L2 tree for asap-planner-rs — mandatory or optional?** Mandatory. planner's current template-catalogue approach is replaced with a proper `PromqlLogicalPlan` tree in Phase 4. The extra conformance work is taken to keep the 5-layer model uniform. See design doc §12 Q8 for the escape hatch if a future deployment model genuinely can't fit a tree. 8. **Sketch binding — L3 or L4?** L4. DC and fusion already do this correctly; planner's `map_statistic_to_precompute_operator` conflates L3+L4 and is split during Phase 4. `AggIntent` at L3 carries intent + accuracy only; concrete `AggregationType` / `SketchParams` are produced by an L4 rule. -9. **Intent vocabulary when scenarios disagree in width?** Core's `AggIntent` is the superset (DC's ~25 variants). Each scenario only uses / produces / accepts the subset it needs (planner 9, fusion 3). Adding a new intent variant is a core change that scenarios opt into. +9. **Intent vocabulary when deployment models disagree in width?** Core's `AggIntent` is the superset (DC's ~25 variants). Each deployment model only uses / produces / accepts the subset it needs (planner 9, fusion 3). Adding a new intent variant is a core change that deployment models opt into. 10. **Tabular vs time-series data models?** Handled in core from Phase 1 via the `Source` sum (`TimeSeries` / `Table` / `Join`) inside `QueryExpr::Scan` and the `DataModel` tag on `AggIntent`. ASAPQuery produces `Source::TimeSeries`, asap-fusion produces `Source::Table`. `Source::Table` is stubbed in Phase 1 and filled in by Phase 3 (fusion). Sketches themselves are data-model-agnostic; the generalization is purely at the IR leaf. -## What a new scenario looks like post-migration +## What a new deployment model looks like post-migration -This is the test of the architecture. Adding a 4th scenario (e.g. "edge-caching") — typical size ~500-2000 LOC: +This is the test of the architecture. Adding a 4th deployment model (e.g. "edge-caching") — typical size ~500-2000 LOC: -1. `cargo new --lib crates/scenario-edge-cache` (in ASAPController workspace) OR a new crate in your own repo that depends on `asap-control-core = { git = "...", tag = "v0.1.0" }`. +1. `cargo new --lib crates/deployment-model-edge-cache` (in ASAPController workspace) OR a new crate in your own repo that depends on `asap-control-core = { git = "...", tag = "v0.1.0" }`. 2. Four files in `src/`: - - `lib.rs` — `impl Scenario for EdgeCacheScenario` - - `rules.rs` — pick rules from `core::optimizer::rules::*` + add scenario-specific rules + - `lib.rs` — `impl DeploymentModel for EdgeCacheDeploymentModel` + - `rules.rs` — pick rules from `core::optimizer::rules::*` + add deployment-model-specific rules - `topology.rs` — declare a `TopologyDescriptor` (e.g. `TwoStage { edge, backend }`) or pick a pre-baked core one - `emit/edge_agent_config.rs` — `impl PlanEmitter` -3. Optionally: `cost.rs` if the scenario needs its own cost model; otherwise use `core::optimizer::cost::*`. +3. Optionally: `cost.rs` if the deployment model needs its own cost model; otherwise use `core::optimizer::cost::*`. 4. If landing in ASAPController workspace: 1 line in `bin/asap-controller/main.rs` to register + 1 line in workspace `Cargo.toml`. If out-of-tree: publish as a binary in your own repo. -**No changes** to runtime, core, or other scenarios. That's the extension-point test passing. +**No changes** to runtime, core, or other deployment models. That's the extension-point test passing. -### Why post-migration scenarios are much smaller than pre-migration code +### Why post-migration deployment models are much smaller than pre-migration code -Pre-migration: each scenario ships its own rule engine, cost traits, allocator, topology handling, sketch catalogue. Total per scenario: 3000-8000 LOC. +Pre-migration: each deployment model ships its own rule engine, cost traits, allocator, topology handling, sketch catalogue. Total per deployment model: 3000-8000 LOC. -Post-migration: those all come from `core::optimizer::*` + `core::physical::*` as imports. Per-scenario code is only: +Post-migration: those all come from `core::optimizer::*` + `core::physical::*` as imports. Per-deployment-model code is only: - Which rules to enable (usually a dozen lines — pick from `core::optimizer::rules`) - Topology declaration (a dozen lines — often reusing a pre-baked `core::physical::topology::*`) -- Scenario-specific rules that truly can't be shared (100-500 LOC each) -- Scenario-specific cost models (if different from core's generic) +- DeploymentModel-specific rules that truly can't be shared (100-500 LOC each) +- DeploymentModel-specific cost models (if different from core's generic) - Emitter(s) for the output format (100-1000 LOC) -Typical post-migration scenario: **500-2500 LOC**. Extremely easy to understand, review, and evolve. +Typical post-migration deployment model: **500-2500 LOC**. Extremely easy to understand, review, and evolve. ## Rollback strategy From 5d73f1322b5edba9f21589a7f556162a125c3f94 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 4 May 2026 17:14:56 -0400 Subject: [PATCH 08/10] =?UTF-8?q?docs:=20split=20L3/L4=20IR=20=E2=80=94=20?= =?UTF-8?q?intent=5Falgebra=20(L3)=20vs=20sketch=5Falgebra=20(L4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Naming was inverted: L3 was called "sketch algebra" but contains no sketches (intent-only); L4 is where sketches actually get bound. Split the module accordingly so the names match the contents: - L3: `core::intent_algebra` — QueryExpr + AggIntent (intent only, no sketch type, no params). - L4: `core::sketch_algebra` — SketchExpr (sketch-bound: kind + params committed). Promoted from a #### subsection of the old L3 section to a sibling ### section. Touches the §3 layer table, §6 module sections, the directory tree, the §14 glossary L3/L4 entries, and ~10 cross-references across README.md, design.md, migration-plan.md. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 9 +++++---- docs/design.md | 37 +++++++++++++++++++------------------ docs/migration-plan.md | 6 +++--- 3 files changed, 27 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 5ca75ad7..85e85e28 100644 --- a/README.md +++ b/README.md @@ -23,8 +23,8 @@ This repo merges three previously-separate projects into a single workspace with |---|---|---| | 1 | Query-language parsing (PromQL, SQL, DataFusion, ElasticDSL) | `core::query_language` | | 2 | Per-language logical plan (relational algebra tree) | `core::logical_plan` | -| 3 | Sketch algebra — language-, deployment-, AND data-model-independent IR (`QueryExpr` + `AggIntent`, intent only). Supports both time-series and tabular data via a `Source` sum inside `Scan`. | `core::sketch_algebra` | -| 4 | Cost-aware optimizer — rule engine + shared rule library; deployment models pick rules | **framework in `core::optimizer`**; choices in each deployment model | +| 3 | Intent algebra — language-, deployment-, AND data-model-independent IR (`QueryExpr` + `AggIntent`, intent only — no sketch type, no params). Supports both time-series and tabular data via a `Source` sum inside `Scan`. | `core::intent_algebra` | +| 4 | Cost-aware optimizer — rule engine + shared rule library; deployment models pick rules. Produces the **sketch algebra** IR `SketchExpr` (sketch-bound: kind + params committed). | **framework in `core::optimizer`**; sketch-bound IR in `core::sketch_algebra`; rule choices in each deployment model | | 5 | Physical plan — stage allocation + emit to wire format | **framework in `core::physical`**; topology + emitter in each deployment model | **Core owns the shared infrastructure across all 5 layers**, not just L1-3. Deployment models are thin: they pick which rules fire (L4), declare their deployment topology (L5), and provide an emitter for their output format. Typical deployment model crate: **500-2500 LOC**. @@ -34,9 +34,10 @@ crates/ core/ # all 5 layers of shared infrastructure; no I/O query_language/ # L1 — parsers logical_plan/ # L2 — per-language algebra trees - sketch_algebra/ # L3 — QueryExpr + AggIntent (intent only, ~25 variants superset) + intent_algebra/ # L3 IR — QueryExpr + AggIntent (intent only, ~25 variants superset) + sketch_algebra/ # L4 IR — SketchExpr (sketch-bound: kind + params committed) lower/ # L1→L2→L3 passes - optimizer/ # L4 framework — rule engine + shared rule library + cost traits + optimizer/ # L4 framework — rule engine + shared rule library + cost traits; produces SketchExpr physical/ # L5 framework — PhysicalPlanner + stage allocator + topology + sketch catalogue pipeline/ # L1→…→L5 driver, parameterized on deployment model runtime/ # HTTP / OpAMP / replanner / store — service skeleton diff --git a/docs/design.md b/docs/design.md index 4b767ded..bbaefa47 100644 --- a/docs/design.md +++ b/docs/design.md @@ -34,11 +34,11 @@ DataCollector/controller already documents its query→sketch translation as a 5 |---|-------|--------------|-------------------| | 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql,mod}.rs`; asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | | 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/algebra/lower.rs`; asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | -| 3 | **Sketch Logical Plan** (sketch algebra) | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | -| 4 | **Sketch Optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 and emit sketch-bound L3+. ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | +| 3 | **Intent algebra** | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | +| 4 | **Sketch algebra + optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 (`QueryExpr`) and emit the sketch-bound IR (`SketchExpr`). ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library + the sketch-bound IR `core::sketch_algebra::SketchExpr`; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | | 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; deployment models supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | -**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every deployment model reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same sketch-algebra IR (intent only, no sketch binding). +**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every deployment model reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same intent-algebra IR (intent only, no sketch binding). **L4 and L5 also have substantial common infrastructure.** Initially we assumed deployment models owned L4/L5 wholesale; on closer inspection what's actually deployment-model-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes deployment models significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. @@ -71,7 +71,7 @@ Practically, this means: - `deployment-model-asapfusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). - A hypothetical OLAP deployment model that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. -See §6 `core::sketch_algebra` for the concrete type sketches. +See §6 `core::intent_algebra` for the concrete type sketches. ### Scope: start single-query, grow into workload-aware @@ -164,9 +164,10 @@ ASAPController/ │ ├── core/ # Shared infrastructure across all 5 layers; no I/O │ │ ├── query_language/ # L1: per-language parsers — promql/, sql/, datafusion/, elasticdsl/ │ │ ├── logical_plan/ # L2: per-language algebra tree (Aggregate/Window/Filter/…) -│ │ ├── sketch_algebra/ # L3: QueryExpr + AggIntent + Schema/HasSchema + sketch directory +│ │ ├── intent_algebra/ # L3 IR: QueryExpr + AggIntent + Schema/HasSchema (intent only) +│ │ ├── sketch_algebra/ # L4 IR: SketchExpr (sketch-bound — kind + params committed) │ │ ├── lower/ # L1→L2→L3 lowering passes (one entry per language) -│ │ ├── optimizer/ # L4 framework: +│ │ ├── optimizer/ # L4 framework — produces SketchExpr from QueryExpr: │ │ │ ├── engine/ # rule driver — fixed-point iteration, cycle detection, priority │ │ │ ├── trait/ # OptimizerRule + RuleCategory (PushDown / Fusion / Elim / Bind) │ │ │ ├── rules/ # shared rule library (e.g. sketch-binding rules; stream-vs-batch picker) @@ -254,7 +255,7 @@ Each returns a language-flavored AST type. No sketch awareness. A **per-language** algebra tree — one `enum LogicalPlan` per language. Preserves language-specific semantics (PromQL instant vs range vector, SQL window frames, Elastic buckets) that would be lossy to collapse this early. Types are symmetric: `Aggregate { AggFunc }`, `Window`, `Filter`, `Sort`, `Limit`. No sketch names yet. -### `core::sketch_algebra` — Layer 3 +### `core::intent_algebra` — Layer 3 The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `HistogramQuantile`, no `PromQLSubquery` — those are L2 PromQL nodes), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP deployment models) via a `Source` sum type inside `QueryExpr::Scan`. @@ -416,7 +417,7 @@ pub trait HasSchema { } ``` -Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/sketch_algebra/schema.rs`). Each row reads independently: every column position, type, and constraint is named explicitly rather than carried by a shorthand like `S` or `L` / `R`. +Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/intent_algebra/schema.rs`). Each row reads independently: every column position, type, and constraint is named explicitly rather than carried by a shorthand like `S` or `L` / `R`. | Node | Input schemas | Output schema | |---|---|---| @@ -500,9 +501,9 @@ impl AggIntent { **Why `Rate` and `Increase` survive that argument.** They are not "Sum / Count over a Window with a different name" — they include PromQL's counter-reset adjustment, which is a non-trivial transformation an exact `Sum` does not perform. They earn distinct intent variants because they parameterise different physical operators (delta-set aggregators bind on these intents directly). If a non-PromQL streaming language has the same notion (e.g. SQL `RATE() OVER (RANGE)`), it lowers to the same intent — the intent vocabulary names the operation, not the language. -#### L4 sketch-bound IR — `SketchExpr` +### `core::sketch_algebra` — Layer 4 IR (`SketchExpr`) -L4 binding rules consume L3 `QueryExpr` and produce `SketchExpr`. This is the IR L5 emitters consume. The two-IR split — pure-logical L3 (`QueryExpr`) and sketch-bound L4 (`SketchExpr`) — gives L4 rule application a clean type signature: `fn apply(&QueryExpr, &Constraints) -> Option`, and the boundary cannot be silently violated. +L4 binding rules consume L3 `QueryExpr` (in `core::intent_algebra`) and produce `SketchExpr` (in `core::sketch_algebra`). This is the IR L5 emitters consume. The two-IR split — intent-only L3 (`QueryExpr`) and sketch-bound L4 (`SketchExpr`), in two separate modules — gives L4 rule application a clean type signature: `fn apply(&QueryExpr, &Constraints) -> Option`, and the boundary cannot be silently violated. ```rust pub enum SketchExpr { @@ -553,7 +554,7 @@ pub enum SketchExpr { The optimizer's job is to selectively replace logical aggregates / joins with their sketch-bound variants when a binding rule fires; everything else stays inside `SketchExpr::Logical(…)`. -##### Per-node input/output spec for `SketchExpr` +#### Per-node input/output spec for `SketchExpr` L4 introduces a new field type into `Schema`: @@ -589,7 +590,7 @@ Two type-system invariants make L4 robust: ### `core::lower` — L1 → L2 → L3 passes -One pass per language, each producing the same `sketch_algebra::QueryExpr`: +One pass per language, each producing the same `intent_algebra::QueryExpr`: ```rust pub fn lower_promql(ast: PromqlAst, schema: &MetricSchema) -> Result; @@ -931,7 +932,7 @@ Each deployment model crate is a library with: - **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. - **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. - **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. -- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::sketch_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). +- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::intent_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). - **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this deployment model covers most sketch types) + a deployment-model-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. - **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. - **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `deployment-model-asaplifecycle` calls them when it needs to POST to ASAPQuery-backend. @@ -944,7 +945,7 @@ Each deployment model crate is a library with: - **Data model**: tabular. L2→L3 lowering produces `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when fusion extends to multi-table). Crucially **not time-indexed** — fusion works on arbitrary DataFusion relations; time is just another column if present. - **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode deployment model. - **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so deployment models that want to take a DataFusion plan as input have a canonical name for it. -- **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::sketch_algebra::AggIntent` subset. +- **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::intent_algebra::AggIntent` subset. - **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + deployment-model-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. - **L5 topology**: `core::physical::topology::ZeroStage` (in-process). - **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this deployment model is library-mode. @@ -1100,9 +1101,9 @@ This matters: a user who only wants `deployment-model-asapfusion` (e.g., an offl 10. **How far do shared L4/L5 rules live in core before they become deployment-model-specific?** The rule of thumb: if ≥2 current deployment models would use it, it lives in `core::optimizer::rules::*`. If only 1 deployment model uses it *and* it depends on deployment-model-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the deployment model crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `deployment-model-asaplifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the deployment model; lift to core once a second deployment model wants it. Don't pre-emptively generalize. -11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP deployment models)?** Already handled — see §3 "Data-model support" and §6 `core::sketch_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP deployment model picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. +11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP deployment models)?** Already handled — see §3 "Data-model support" and §6 `core::intent_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP deployment model picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. -### Resolved during the L3 IR cleanup (see §6 `core::sketch_algebra`) +### Resolved during the L3 IR cleanup (see §6 `core::intent_algebra`) The following questions came up during review of the previous `QueryExpr` draft and are now settled. Listed here so the trail is visible: @@ -1155,8 +1156,8 @@ Terms that are project-specific or that get conflated. Where the term has a Rust - **L1 — query language** — Raw query string + parser (PromQL / SQL / DataFusion / ElasticDSL). - **L2 — logical plan** — Per-language relational algebra tree (`PromqlLogicalPlan`, `SqlLogicalPlan`, etc.). -- **L3 — sketch algebra** (`QueryExpr`) — Symbolic, intent-only IR. No sketch type, no params. `AggIntent` carries accuracy targets only. -- **L4 — sketch-bound IR** (`SketchExpr`) — Same DAG shape, sketches now committed (kind + params). +- **L3 — intent algebra** (`QueryExpr`, `core::intent_algebra`) — Symbolic, intent-only IR. No sketch type, no params. `AggIntent` carries accuracy targets only. The "algebra" is the operator surface (`Scan`, `Filter`, `Aggregate`, `Window`, …); the algebra is *intent-only* because no sketches have been bound yet. +- **L4 — sketch algebra** (`SketchExpr`, `core::sketch_algebra`) — Same DAG shape as L3, but sketches now committed (kind + params). Produced by L4 binding rules; consumed by L5 emitters. Note the naming inversion vs. earlier drafts: "sketch algebra" is L4 (where sketches actually live), not L3. - **L5 — physical plan** — Stage-assigned, executor-targeted, ready to serialize. Produced by `PhysicalPlanner`, written out by `PlanEmitter`. ### Metadata sources (see §6 "DAG schema, DB schema, sketch catalog") diff --git a/docs/migration-plan.md b/docs/migration-plan.md index 147f203e..ba8918cb 100644 --- a/docs/migration-plan.md +++ b/docs/migration-plan.md @@ -36,14 +36,14 @@ Ordered by risk, lowest first. Each phase is a single PR unless noted. ### Phase 1 — Core extraction: all-5-layer shared infrastructure -**Scope:** Lift the DC controller's **layers 1–3** (query parsing + per-language algebra + sketch-algebra IR) into `crates/core/` verbatim, AND lift L4 and L5 **shared infrastructure**: rule engine, shared rule library, cost-model traits, `PhysicalPlanner` trait, stage allocator framework, sketch catalogue. Deployment models in later phases pick from this common base and only add deployment-model-specific pieces. +**Scope:** Lift the DC controller's **layers 1–3** (query parsing + per-language algebra + intent-algebra IR) into `crates/core/` verbatim, AND lift L4 and L5 **shared infrastructure**: rule engine, shared rule library, cost-model traits, `PhysicalPlanner` trait, stage allocator framework, sketch catalogue, sketch-bound L4 IR (`SketchExpr` in `core::sketch_algebra`). Deployment models in later phases pick from this common base and only add deployment-model-specific pieces. Per design §3, these five layers' infrastructure is query-language-independent AND deployment-model-independent; the concrete rule set, topology, and emitter are deployment-model-specific. **Work — L1-3 (straight lift from DC):** - **`core::query_language`** (L1): lift `DataCollector/controller/src/query_parser/{promql,sql,mod}.rs` → `core/src/query_language/`. One module per language. - **`core::logical_plan`** (L2): lift per-language algebra tree definitions → `core/src/logical_plan/`. One `enum LogicalPlan` per language. Add `core::logical_plan::datafusion` as a thin re-export of `datafusion::logical_expr::LogicalPlan` for fusion's L2. -- **`core::sketch_algebra`** (L3): lift `algebra/{expr,directory}.rs` → `core/src/sketch_algebra/`. Enforce **intent-only** — `AggIntent` carries accuracy target, never sketch type. **Generalize `QueryExpr::Scan` over data model**: introduce a `Source` sum (`TimeSeries` / `Table` / `Join` variants) inside `Scan`, and a `DataModel` enum (`TimeSeries` / `Tabular` / `Any`). DC's current scan logic becomes `Source::TimeSeries`; `Source::Table` is new (stub impl fine for Phase 1 — deployment-model-asapfusion fills it in Phase 3). Add `AggIntent::requires() -> DataModel` so L4 rules can skip non-applicable intents. See design §3 "Data-model support" and §6 `core::sketch_algebra`. ~300 LOC on top of the DC lift. +- **`core::intent_algebra`** (L3): lift `algebra/{expr,directory}.rs` → `core/src/intent_algebra/`. Enforce **intent-only** — `AggIntent` carries accuracy target, never sketch type. **Generalize `QueryExpr::Scan` over data model**: introduce a `Source` sum (`TimeSeries` / `Table` / `Join` variants) inside `Scan`, and a `DataModel` enum (`TimeSeries` / `Tabular` / `Any`). DC's current scan logic becomes `Source::TimeSeries`; `Source::Table` is new (stub impl fine for Phase 1 — deployment-model-asapfusion fills it in Phase 3). Add `AggIntent::requires() -> DataModel` so L4 rules can skip non-applicable intents. See design §3 "Data-model support" and §6 `core::intent_algebra`. ~300 LOC on top of the DC lift. - **`core::lower`**: lift `algebra/lower.rs` (L1→L2→L3 passes) → `core/src/lower/`. One `lower_` entry point per language. Each lowering produces the appropriate `Source` variant (PromQL / SQL → `Source::TimeSeries`; DataFusion → `Source::Table`). **Work — L4 framework (NEW):** @@ -156,7 +156,7 @@ This phase is larger than a straight lift because two structural conformance cha - **This is the conformance cost the user explicitly accepted for architectural uniformity.** Budget accordingly. **Work — L3 / L4 split (NEW):** -- In `core::lower::promql_to_sketch_algebra`: `PromqlLogicalPlan → QueryExpr` producing **intent-only** L3 (no sketch names). +- In `core::lower::promql_to_intent_algebra`: `PromqlLogicalPlan → QueryExpr` producing **intent-only** L3 (no sketch names). - `Statistic` enum → `AggIntent` subset (9 of DC's 25 variants). No sketch type, no sketch params at L3. - In `deployment-model-asapquery/src/rules.rs`: picks core's `BindKllOnQuantile`, `BindCmsOnCount`, etc. from Phase 1's shared rule library + adds a deployment-model-specific `PrecomputeEngineBindRule` (handles the precompute-engine-specific flavor: which sketches are available in ASAPQuery-backend's accumulator set, what params match the engine's config schema, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). - Update `build_agg_configs_for_statistics` to call `core::optimizer::engine::RuleEngine::run()` with the deployment model's rule set instead of the inline `map_statistic_to_precompute_operator` fusion. From 58f0af4a3ddff0700a2fb948baaf171df01bdb8a Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Mon, 4 May 2026 17:28:41 -0400 Subject: [PATCH 09/10] docs: clarify Schema.unique_keys is opt-in metadata for reuse-aware planning The previous doc comment described unique_keys as a "sketch-reuse driver" without flagging that it's optional infrastructure for one specific feature (`CostModel::workload_cost` cross-query reuse credit). Tighten the comment to call out: who populates it (per-node spec), who reads it (workload_cost only), and what doesn't read it (Bind* rules, push-down, L5 emitters, single-query plans). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/design.md b/docs/design.md index bbaefa47..11aa9d3e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -400,8 +400,22 @@ pub struct Schema { /// Index into `fields` for the time axis, if any. PromQL leaves carry one; /// SQL leaves may or may not. pub time_index: Option, - /// Functional dependencies / unique-key sets. L4 reuses these to recognise - /// when two sub-expressions produce identical streams (sketch-reuse driver). + /// Optional metadata for reuse-aware planning. Each inner `Vec` is + /// a set of column indices that together uniquely identify rows; the outer + /// `Vec` allows multiple unique-key sets (e.g. primary key + another unique + /// constraint). + /// + /// **Populated by**: the per-node input/output spec (e.g. `Aggregate { by, .. }` + /// emits `unique_keys = [by]`; `Distinct { cols }` adds `cols`; `Project` + /// carries forward the retained columns; most other nodes pass through). + /// + /// **Consumed by**: `CostModel::workload_cost` only — the reuse-aware path + /// that credits shared sub-expressions across multiple queries. Single-query + /// plans, the `Bind*` rules, push-down rules, and L5 emitters do not read + /// this field. If the reuse path is deferred (see §13 future work), this + /// field is dead weight; it lives here so the metadata is available the + /// moment workload-aware planning lands without requiring an L3-wide + /// schema change. pub unique_keys: Vec>, } From e7424984bf2d20b128c200501aa998092c854610 Mon Sep 17 00:00:00 2001 From: Zeying Zhu Date: Tue, 5 May 2026 12:55:27 -0400 Subject: [PATCH 10/10] docs: add end-to-end example tracing one query through all five layers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Worked example for `quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m])` showing the L1→L5 transformation under each of the three deployment models, so readers can see how the layered IR + topology-as-parameter design plays out concretely. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/design.md | 188 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 188 insertions(+) diff --git a/docs/design.md b/docs/design.md index 11aa9d3e..5b8b1d8e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -909,6 +909,194 @@ Registration is by-type; the runtime looks up by `DeploymentModelId` (either fro No core change. +### End-to-end example — one query through all five layers + +A worked trace of a single PromQL query as it flows L1 → L5. The query is intentionally simple (one metric, one window, one aggregate) so the IR shapes stay readable; production workloads have larger DAGs but the per-layer transformation is the same. + +**Input.** A `QuerySpec` arrives at `POST /plan` carrying: + +```text +quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m]) +``` + +with `accuracy: AccuracyTarget::Epsilon(0.01)` and `language: PromQL`. + +#### L1 — query language (parse to `PromqlAst`) + +`core::query_language::promql::parse` wraps `promql-parser`. Output (sketch — actual fields come from the upstream crate): + +```rust +PromqlAst::Call { + func: BuiltinFn::QuantileOverTime, + args: vec![ + Expr::Number(0.99), + Expr::MatrixSelector { + name: "http_request_duration_seconds", + matchers: vec![LabelMatcher::Eq("service", "api")], + range: Duration::from_secs(300), + }, + ], +} +``` + +Pure language-level parse — no schema lookup, no sketch awareness. Same call returns the same AST regardless of deployment model. + +#### L2 — language logical plan (`PromqlLogicalPlan` tree) + +`core::lower::promql::lower_to_logical` walks the AST against a `MetricSchema` resolved from Prometheus's `/api/v1/labels`. The result preserves PromQL semantics — `quantile_over_time` is a range-vector aggregate, distinct from a generic `Aggregate { Quantile }` over a `Window`: + +```rust +PromqlLogicalPlan::RangeAggregate { + func: PromqlRangeAggFunc::QuantileOverTime { q: 0.99 }, + range: Duration::from_secs(300), + matrix: Box::new(PromqlLogicalPlan::MatrixSelector { + metric: "http_request_duration_seconds", + matchers: vec![("service", LabelOp::Eq, "api")], + }), +} +``` + +Per language: the SQL form `SELECT approx_percentile(latency, 0.99) FROM events WHERE service='api' AND ts > now() - INTERVAL '5 min'` lands in `SqlLogicalPlan` with a different shape. L2 is per-language; the shapes converge at L3. + +#### L3 — intent algebra (`QueryExpr` + `AggIntent`) + +`core::lower::promql::lower_to_intent` strips PromQL-specific shapes. `quantile_over_time` does **not** survive — the canonical L3 form for a windowed quantile is `Window` over `Aggregate{Quantile}` (see §6 design rule 1, "no `WindowedAgg`"; see §6 "Why no `QuantileOverTime` intent?"). The accuracy target threads through from the `QuerySpec`: + +```rust +QueryExpr::Aggregate { + by: vec![], + aggs: vec![AggIntent::Quantile { + q: 0.99, + accuracy: AccuracyTarget::Epsilon(0.01), + }], + having: None, + child: Box::new(QueryExpr::Window { + kind: WindowKind::Sliding, + size: Duration::from_secs(300), + slide: None, + child: Box::new(QueryExpr::Scan { + source: Source::TimeSeries { + metric: MetricRef::from("http_request_duration_seconds"), + time: TimeRange::default(), // supplied by QuerySpec at evaluation + labels: LabelFilter::eq("service", "api"), + }, + predicates: vec![], + }), + }), +} +``` + +Per-edge schemas (derived per the §6 input/output spec): + +| Edge | Schema | +|---|---| +| `Scan` → `Window` | `{value: Float64, ts: Int64@time_index, service: Utf8}` | +| `Window` → `Aggregate` | the above + `{window_id: Int64, window_start: Int64, window_end: Int64}` | +| `Aggregate` → root | `{p99_value: Float64}` (one column per `AggIntent::Quantile`, typed via `output_type`) | + +This IR is identical across deployment models — same DAG, same intent, same accuracy target. Below this point each deployment model picks its own L4 rules. + +#### L4 — sketch algebra (`SketchExpr`) + +The shared rule `core::optimizer::rules::BindKllOnQuantile` matches `Aggregate { aggs: [Quantile{q, accuracy}] }`, consults the sketch catalog (KLL has `supported_intents: [Quantile]`, is mergeable, satisfies `ε=0.01` at `k=200`), and rewrites the matched sub-DAG into a `SketchAgg` wrapped in a `SketchEstimate` (the readout). Everything not rewritten passes through in `SketchExpr::Logical(...)`: + +```rust +SketchExpr::SketchEstimate { + sketch_input: Box::new(SketchExpr::SketchAgg { + child: Box::new(SketchExpr::Logical(/* the L3 Window+Scan subtree */)), + sketch: SketchKind::Kll, + params: SketchParams::Kll(KllParams { k: 200 }), + col: ColumnRef::value(), + by: vec![], + }), + query: SketchQuery::Quantile { q: 0.99 }, +} +``` + +Catalog-derived edge schemas: + +| Edge | Schema | +|---|---| +| inner `Logical(Window)` → `SketchAgg` | as L3 above (logical pass-through) | +| `SketchAgg` → `SketchEstimate` | `{kll: Sketch(Kll, KllParams{k:200})}` | +| `SketchEstimate` → root | `{p99_value: Float64}` (the `Sketch(...)` field type does not propagate past `SketchEstimate`) | + +The `Sketch(Kll, KllParams{k:200})` field type is the L4 type-system invariant — a downstream `SketchMerge` over a mismatched `Sketch(Cms, …)` input would fail at plan time, before L5 ever sees it. `BindKllOnQuantile` lives once in core and fires for all three deployment models; deployment-model-specific rules (lifecycle's `StageAwarePushDown`, fusion's `HashModeRule`) chain before or after. + +#### L5 — physical plan (stage allocation + emission, per deployment model) + +L4 chose the sketch family + params. L5 colors the DAG by `StageId` and emits per-executor configs. Same `SketchExpr` input; topology and emitter differ per deployment model. + +##### deployment-model-asaplifecycle (3-stage, OpAMP + backend POST) + +`StageAllocator` colors against `topology::ThreeStage`: + +| Node | StageId | Why | +|---|---|---| +| `Scan` + `Window` + `SketchAgg` | `"edge"` | scrape happens on the agent host; KLL build is mergeable so it's safe to run early | +| `SketchMerge` (inserted on cut edge) | `"gateway"` | catalog says `KLL.mergeable = true`; reduces N edge streams to 1 | +| `SketchEstimate` | `"backend"` | readout where users query | + +`PhysicalPlanner` then fans out via `DeploymentConstraints::executors()`. Given a deployment with 3 edge agents + 1 gateway + 1 backend: + +```rust +vec![ + (ExecutorId("edge-001"), OpAmpRemoteConfig { /* OTel YAML: scrape http_request_duration_seconds{service=api}, + sliding 5m window, KLL k=200, forward to gateway-001 */ }), + (ExecutorId("edge-002"), OpAmpRemoteConfig { /* identical OTel YAML; differs only by agent_id */ }), + (ExecutorId("edge-003"), OpAmpRemoteConfig { /* … */ }), + (ExecutorId("gateway-001"), OpAmpRemoteConfig { /* receive 3 KLL streams, SketchMerge, forward to backend */ }), + (ExecutorId("backend-001"), AsapqueryBackendConfig { /* read merged KLL, SketchEstimate q=0.99 */ }), +] +``` + +The 3 edge configs are byte-identical except for `agent_id` — that's the §3 "one stage may have N executors all materialised from the same per-stage sub-DAG" property. The `AsapqueryBackendConfig` is produced by calling `deployment-model-asapquery::yaml::StreamingConfigEmitter` (see §8 asymmetric dep). + +##### deployment-model-asapquery (1-stage, YAML) + +`StageAllocator` against `topology::SingleStage` returns everything on `"backend"`. `PhysicalPlanner` produces one config per backend executor (typically one): + +```yaml +# streaming_config.yaml — emitted by deployment-model-asapquery::yaml::StreamingConfigEmitter +operators: + - name: kll_p99_request_duration + kind: DatasketchesKLL + params: { k: 200 } + input: + metric: http_request_duration_seconds + label_filter: { service: api } + window: { kind: sliding, size: 5m } +estimates: + - sketch: kll_p99_request_duration + query: { kind: quantile, q: 0.99 } +``` + +This is the byte-for-byte format ASAPQuery-backend already consumes — `inference_config.yaml` is emitted alongside by `InferenceConfigEmitter` for the readout side. Both are wire invariants (see §10). + +##### deployment-model-asapfusion (0-stage, in-process `LogicalPlan`) + +`StageAllocator` against `topology::ZeroStage` returns the whole DAG in-process. `PhysicalPlanner` rewrites the caller's `datafusion::LogicalPlan`, replacing the original `Aggregate(quantile(0.99, ...))` node with an `Extension` wrapping the KLL sketch op: + +```rust +LogicalPlan::Extension(Extension { + node: Arc::new(SketchAggExt { + kind: SketchKind::Kll, + params: KllParams { k: 200 }, + input: /* original Window+Filter+Scan subtree, unchanged */, + estimate: SketchQuery::Quantile { q: 0.99 }, + }), +}) +``` + +Returned to the caller's `SessionContext` for execution by `asap-fusion`'s in-process operator. No wire format, no external executor; the data plane is the caller's process. Note also that `asap-fusion`'s entry point is `Source::Table` (not `Source::TimeSeries`) — this PromQL example is only illustrative for the fusion deployment model, which in practice consumes a pre-built DataFusion `LogicalPlan` and skips L1 (see §8). + +#### What this example demonstrates + +- **L1 → L3 are deployment-model-independent.** All three deployment models see the same `QueryExpr` for this query. +- **L4 binding is a shared rule.** `BindKllOnQuantile` lives once in `core::optimizer::rules` and fires for all three. +- **L5 is where deployment models diverge.** Same `SketchExpr` → three topologies → three emitter outputs. The topology is a parameter, not an axis of code (§3). +- **The sketch path is selected, not mandated.** If the `QuerySpec` had `accuracy: AccuracyTarget::Exact`, no `Bind*` rule would fire; L4 would pass `SketchExpr::Logical(QueryExpr::Aggregate{…})` through, and L5 would target an exact `HashAgg` (§1 goal 6). + ## 7. Runtime crate details Lifted from DC controller with zero semantic change: