diff --git a/crates/sketch/src/exec.rs b/crates/sketch/src/exec.rs index 16318fce..bdbf8aad 100644 --- a/crates/sketch/src/exec.rs +++ b/crates/sketch/src/exec.rs @@ -1,6 +1,6 @@ //! Serving-time execution model for the L4 IR — the counterpart to //! `asap_plan::bind`'s planning-time `QueryExpr -> L4Node`. See -//! `docs/l4node-execution-model.md` for the design (planning vs. serving +//! `docs/l4-summary-bound-ir.md` for the design (planning vs. serving //! L4, the nested-composition rules, the open questions); this module is //! the implementation of it. @@ -12,7 +12,7 @@ use crate::expr::{L4Node, SummaryExpr}; use crate::sketch::{SketchQuery, SummaryKind, SummaryParams}; /// Deployment-supplied backend for executing an [`L4Node`] tree against -/// materialized state — see `docs/l4node-execution-model.md`. +/// materialized state — see `docs/l4-summary-bound-ir.md`. pub trait SummaryExecutor { /// Reference to one materialized summary instance (e.g. a sid). type Handle: Clone; diff --git a/docs/design.md b/docs/design.md index 8c00f1c5..73d63163 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,1517 +1,209 @@ # ASAPController — design -Repo: **`github.com/ProjectASAP/ASAPController`**. - -> **Status (reconciled with the landed workspace).** This document is the -> *target* design. What exists today is the query→IR core, split into a -> layer-named crate stack — see **§5.1 Landed layout**. The optimizer / -> physical / runtime / deployment-model / bin crates described below are -> **planned**; where a section covers unbuilt pieces it is marked accordingly. -> The `core::` names used throughout §6 predate the split; **§6.0** -> maps them to the crates that ship them today. - -Merges three existing codebases into one: - -1. **`DataCollector/controller`** (Rust) — service-shaped; end-to-end data lifecycle planner (collection → transmission → storage → analytics query); OpAMP + HTTP; SLA-driven replanning loop. -2. **`ASAPQuery[-backend]/asap-planner-rs`** (Rust) — CLI-shaped; analytics-query-only planner; YAML-in → YAML-out. Two divergent copies today. -3. **`asap-fusion`** (Rust) — library-shaped; DataFusion operator-level rewrite rules with sketch awareness; multi-query batch fusion is aspirational, executor is a thin wrapper. - -## 1. Goals - -1. **One repo, one workspace.** One `Cargo.toml` at root, one place to file issues, one release cadence. -2. **Common core, pluggable deployment models.** The three existing codebases each solve a *different* planning problem. Factor out what's actually shared; keep the deployment-model-specific parts as crates so each deployment model can evolve independently. -3. **Extensible for future deployment models** (4th, 5th, …). A new deployment model should land as a new crate that implements well-defined traits — not by touching the core or the runtime. -4. **Preserve the two deployment shapes:** - - **Service**: long-running HTTP+OpAMP process that accepts live `QuerySpec`s, replans on SLA violations, pushes configs to agents/backends. (DC controller today.) - - **CLI**: one-shot "read workload YAML → emit `streaming_config.yaml` + `inference_config.yaml`". (`asap-planner-rs` today.) - Both should be thin shells over the same core. -5. **No regression in today's wire contracts.** `POST /api/v1/streaming-config` to ASAPQuery-backend and OpAMP push to agents must keep working byte-for-byte through the migration. The backend's capability-miss callback `ControllerClient.create_plan` must keep working. -6. **Sketch is a primitive, not a mandate.** The optimizer selects among physical alternatives for each logical operator — e.g. `HashJoin` / `SortMergeJoin` / `SketchJoin`, or `SortAgg` / `HashAgg` / `SketchAgg` — the same way a traditional DB optimizer picks a join algorithm. A plan may come back with zero sketch operators when an exact path Pareto-dominates for the query's accuracy target. Sketches are one primitive class the optimizer can reach for; the framework does not privilege them. - -## 2. Non-goals - -- **Not** redesigning the `Plan` IR end-to-end. DC's `algebra/` + fusion's `translator/optimizer/executor` stay where they are semantically; we merge them into a shared `Plan` crate with a clean trait seam, not a green-field rewrite. -- **Not** unifying DataFusion's `LogicalPlan` with DC's custom algebra at the type level. Those are different universes. Deployment models own their IR; core owns the *staged dispatch contract*. -- **Not** in-scope: changing the existing wire protocol between controller and backend. ASAPQuery-backend keeps consuming `streaming_config.yaml` and keeps responding on `/api/v1/plan`. - -## 3. The organizing spine: DC controller's 5-layer pipeline - -DataCollector/controller already documents its query→sketch translation as a 5-layer pipeline (`DataCollector/controller/docs/query-to-sketch-translation.md`). That's the spine of this merger: - -| # | Layer | What it does | Today's locations | -|---|-------|--------------|-------------------| -| 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql,mod}.rs`; asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | -| 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/algebra/lower.rs`; asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | -| 3 | **Intent algebra** | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `PromQLSubquery` — a PromQL L2 node that lowers to data-model-agnostic shapes here; one deliberate exception is `AggIntent::HistogramQuantile`, kept as a distinct intent because classic cumulative-`le`-bucket interpolation is *semantically* different from the sketch-able generic `Quantile` — pre-aggregated bucket counts can't be re-sketched. See #43/#79 and the histogram note in §6.2). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/algebra/{expr,directory}.rs`; asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4** | -| 4 | **Sketch algebra + optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 (`QueryExpr`) and emit the sketch-bound IR (`SketchExpr`). ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library + the sketch-bound IR `core::sketch_algebra::SketchExpr`; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/algebra/optimizer.rs` (rules); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator` | -| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; deployment models supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/algebra/{physical,allocator,plan}.rs`; asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/` | - -**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every deployment model reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same intent-algebra IR (intent only, no sketch binding). - -**L4 and L5 also have substantial common infrastructure.** Initially we assumed deployment models owned L4/L5 wholesale; on closer inspection what's actually deployment-model-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes deployment models significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. - -### Terminology: bind, implementation, match - -"Bind" gets reused informally for several *different* things near L3/L4 — this stack settled on more specific names to keep them apart. Reference source: `crates/plan/src/lib.rs`'s own module doc (`asap-plan`) — that's the version closest to the code and least likely to drift; this table is a pointer to it, not a replacement for it. - -| Term | Layer | Meaning | Lives in | -|---|---|---|---| -| **Bind #1** | L2 | *Name resolution*: `ColumnRef` (a name) → `ColumnId` (a concrete schema column) — the classic RDBMS "Parse → Bind → Optimize" pipeline sense | `asap_l2::binder::Binder` | -| **Implementation** | L3→L4, *one node* | Choosing a concrete physical realization — a sketch family, an exact accumulator, or pass-through — for one `AggIntent` | `asap_plan::boundary::implementation_for` | -| **`implement_tree`** | L3→L4, *whole tree* | Walk a whole `QueryExpr` tree, calling `implementation_for` per node, and emit the complete L4 `SummaryExpr`/`L4Node` DAG | `asap_plan::bind::implement_tree` (module is named `bind` for historical reasons; its content is "implementation", not the L2 or L4→L5 senses below) | -| **Match** | L3→L4-adjacent, but a different question | *"Does an already-**available** `Implementation` satisfy a **required** one?"* — materialized-view matching / "answering queries using views", narrowed to this crate's summary vocabulary. Distinct from `implementation_for`: that answers "how would I build this from scratch"; `Matcher` answers "does something that already exists already answer this". `asap-plan` ships the trait with **no default implementation and no shipped instance** — which `Implementation`s are actually available anywhere (an inventory) is entirely a downstream deployment's concern, the same reason `CostModel` ships no opinionated instance either | `asap_plan::boundary::Matcher` (trait only); reference downstream impl: ASAPQuery-backend's `control_plane::sketch_algebra::capability::Capability::is_satisfied_by` | -| **Bind #2** | L4→L5 | A *deployment's* own physical binder, additionally deciding **placement** (edge vs. backend, wire format, …) — a genuinely different, deployment-specific decision core doesn't model at all | e.g. ASAPQuery-backend's `control_plane::sketch_algebra::rules::bind_*` | - -Both "Implementation" rows are named after Cascades/Volcano query-optimization terminology (an *implementation rule* is logical → physical, as distinct from a *transformation rule*, logical → logical) specifically to avoid becoming a fifth colliding sense of "bind". - -### Sketch binding lives in L4, not L3 - -A key clarification after cross-checking the three source repos: **L3 is intent-only**. DC's `AggIntent` names *what* to compute (`Quantile(0.99, ε=0.01)`, `Cardinality(δ=0.001)`, …) without committing to a sketch type. Picking KLL vs DDSketch, CMS vs CMS-with-heap, parameter sizes — all of that is L4's job, driven by deployment constraints. - -More generally: **L1–L3 lower into a logical representation + `AggIntent`; L4 and L5 choose the concrete execution plan.** That choice is a standard physical-operator selection — `HashJoin` vs `SortMergeJoin` vs `SketchJoin`; `SortAgg` vs `HashAgg` vs `SketchAgg`. "Use a sketch" is one option among several; the same L4 rule framework that picks sketch parameters also picks between sketch and non-sketch operators when a rule is registered for the intent. This keeps the existing 5-layer split intact: nothing about the layering presupposes the output contains a sketch. - -Today: -- DC: correctly separated (L3 has `AggIntent`, L4 picks sketch via cost model). -- asap-fusion: correctly separated — `SketchConfigRule` at L4 fills `SketchConfig::NULL` with concrete `CountMinSketch{5,4096}` / `KLL{k=200,m=8}`. -- asap-planner-rs: **L3+L4 fused today**. `map_statistic_to_precompute_operator` jumps from `Statistic` straight to `AggregationType::DatasketchesKLL{k=200}` in one call. **Phase 4 splits this** — `Statistic → AggIntent` at L3, `AggIntent + DeploymentConstraints → AggregationType + SketchParams` at L4. - -### Intent vocabulary: DC's `AggIntent` is a superset; deployment models use subsets - -DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) language-flavored synonyms (`QuantileOverTime → Window + Quantile`) no longer earn their own intent. The core started at 9 (`Count, Sum, Min, Max, Quantile, TopK, Cardinality, Rate, Increase`) and — per the "ceiling is bounded by genuinely-distinct operations, not language-flavored synonyms" rule — has since grown to ~40 shipped variants as the PromQL function families landed (see `crates/ir/src/intent_algebra/agg_intent.rs`, the source of truth): stddev/variance/avg, the counter-derivative family (`Changes`/`Delta`/`Deriv`/`Resets`/`PredictLinear`/…, #44), native-histogram accessors + classic-bucket `HistogramQuantile` (#43; see the histogram note in §6), element-wise `Math` (#45), presence (#47), time/calendar `TimeFn` (#46), `Group`/`CountValues` (#49), and the extra `*_over_time` reducers (#51). Each is a genuinely-distinct operation — per-series vs cross-series semantics, distinct output columns, distinct sketchability — not a synonym. Planner's 9-variant `Statistic` maps directly: `Topk` keeps its own intent (heavy-hitter sketches like SpaceSaving / CMS-with-heap compute it as a single primitive, so the intent earns L3 visibility). Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that deployment models opt into. - -### Data-model support: both time-series and tabular - -ASAPQuery-backend / DC controller operate on time-series data (metrics + labels + timestamp); asap-fusion operates on tabular data (DataFusion `LogicalPlan` over relations) that may or may not be time-indexed. These two data models differ fundamentally in their leaf shape (`metric + labels + time` vs. `table + columns`), but they share everything above the leaf — filter semantics, aggregation semantics, sketches themselves. - -Core handles this with: -- **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Deployment models' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. -- **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Min`, `Max`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase` — both carry PromQL counter-reset semantics), or tabular-only (future additions for joins, correlated subqueries). -- **Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values; that stream can come from a time-series window or a table column, the sketch does not know or care. So `BindKllOnQuantile` and siblings work uniformly across both. - -Practically, this means: -- `deployment-model-asapquery` + `deployment-model-asaplifecycle` lower into `QueryExpr` with `Source::TimeSeries` leaves. -- `deployment-model-asapfusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). -- A hypothetical OLAP deployment model that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. - -See §6 `core::intent_algebra` for the concrete type sketches. - -### Scope: start single-query, grow into workload-aware - -The initial implementation can operate on **one query at a time** — L4 picks physical operators per query against per-query constraints, and `CostModel::workload_cost` degenerates to a sum of per-plan costs. This matches today's three source repos (all single-query planners) and is the minimum bar for parity during the migration. - -**Workload-awareness is an extension, not a rewrite.** When ≥2 queries are planned together, `workload_cost` credits shared sub-expressions (sketches, precomputed aggregates) so the planner can pick a plan for `q1` that lets `q2` read its output for free. Nothing in the L1–L5 spine changes — only the cost objective widens and the rule engine gains cross-plan visibility. See §6 `core::cost` and §13 future work. - -### L2 is mandatory; the tree shape is an evolvable contract - -Every deployment model must produce an L2 tree, even when the source language didn't originally come as one. asap-planner-rs's current approach (PromQL pattern catalogue → `IntermediateAggConfig`) skips L2; Phase 4 will reverse-engineer the five PromQL pattern shapes into a `PromqlLogicalPlan` tree so the L1→L2→L3 pipeline is uniform. - -A future deployment model whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. - -### System I/O contract — what the controller takes in, what it emits - -The controller is a **planner**, not an executor. It does not run queries; it decides where each piece of a query runs. The data plane (OTel collectors / ASAPQuery-backend / DataFusion `SessionContext`) runs them. - -**System input — what the controller takes in.** A `QueryWorkload` (one or more `QuerySpec`s) plus deployment context (available executors, their capabilities, current SLA targets, telemetry of recent violations). The `QuerySpec` carries the raw query string in its source language (PromQL / SQL / DataFusion / ElasticDSL) and the accuracy / latency / cost target. Workloads can arrive via four entry points (HTTP `POST /plan`, OpAMP capability-miss callback, YAML file for the CLI shell, query-log replay); they all normalise to `QueryWorkload` before L1. - -**Workload features that the controller cares about, beyond the queries themselves:** -- Execution model: **batch** (one-shot YAML, query-log replay) vs **streaming** (live pipeline that must keep producing results as data arrives). -- Data input source: time-series scrape, tabular relation, query log replay, etc. Drives `Source` variant choice in L3. -- Reuse opportunity: ≥2 queries planned together → `CostModel::workload_cost` credits shared sub-expressions. - -**System output — what the controller emits.** For each registered executor in the deployment, a sub-DAG of the optimized plan plus the configuration that lets that executor run it. Concretely: - -| Output | Consumer | Wire shape | -|---|---|---| -| Per-executor sub-DAG assignment | The executor (edge agent / gateway / backend / DataFusion session) | OpAMP `RemoteConfig` (OTel YAML) / `streaming_config.yaml` POST / rewritten `LogicalPlan` | -| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge / compute-from-raw over the precompute engine, …) | Implied by the per-executor configs; not a separate artifact | -| Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | - -**Premise behind the staged dispatch.** Two distinct concepts: - -- **Stage** (`StageId`) — a categorical *tier* in the data lifecycle (edge / gateway / backend / in-process). The topology declares which stages exist (3-stage / 1-stage / 0-stage). Stages are roles, not instances. -- **Executor** (`Executor`, defined in `core::physical::executor`) — a *concrete runtime instance* that occupies a stage. Carries `id`, `stage: StageId`, `capabilities`, and `address` (OpAMP agent / HTTP endpoint / in-process handle). One stage may have N executors — e.g. a 50-host edge fleet is 50 `Executor`s all with `stage = StageId("edge")`; a singleton backend is one `Executor` at `stage = StageId("backend")`. - -Every stage is in principle capable of running the entire query tree — all stages speak the same physical operators. The controller's job is to decide *which stage* runs *which sub-tree / sub-DAG* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by `StageId`, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring at stage granularity; the per-deployment-model `PhysicalPlanner` then materialises each stage's sub-DAG into one config per `Executor` at that stage (varying only by per-executor connection / identity details). The executor list comes from `DeploymentConstraints::executors()`. - -This is what makes the topology a *parameter* rather than an axis of code: the same `SketchExpr` plus a different `TopologyDescriptor` produces edge-only / 1-stage / 3-stage / 0-stage placements without rewriting the plan. - -**Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialize into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. - -This drives the core/deployment model split: - -- **The shared infrastructure crates** own all 5 layers. Landed today (§5.1): the front ends (L1→L2), `asap-l2` (L2 relational + the L2→L3 converter), `asap-ir` (L3 canonical IR), `asap-sketch` (L4 IR), and `asap-plan` (L4 optimizer — CSE + the L3→L4 `implement_tree` pass (see §3 terminology) + the sketch-vs-exact `boundary` decision (#98), with the rule engine + cost model as stubs). Planned: the L5 stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. (The original design put all of this in one `core/` crate; §5.1 splits it by role.) -- **Each deployment model is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any deployment-model-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. - -## 4. Principles - -### P1. Core owns shared infrastructure across all 5 layers; deployment models own choices - -See §3. Core is NOT just types + trait stubs — it ships working parsers + lowering passes (L1-3), a rule engine + rule library + cost-model traits (L4 framework), and a stage allocator + physical-plan framework + sketch catalogue (L5 framework). What deployment models plug in is **which rules fire** (picking from core's library + adding their own), **deployment topology** (how many stages; DC=3, query=1, fusion=0), and an **emitter** for the output format. A deployment model that accepts all core's default L4 rules and uses `core::physical::single_stage_topology` is maybe 200 lines of code. - -### P2. Core has no I/O - -No HTTP, no OpAMP, no YAML, no Prometheus scrape, no `tokio::spawn`. Pure algorithms over in-memory data. This is what makes deployment models unit-testable without running the runtime. - -### P3. Runtime is a thin binary, deployment models are libraries - -The `asap-controller` binary is assembled from: runtime (HTTP/OpAMP/replanner/store) + N deployment model crates registered as plugins. Swapping deployment models is a build-time feature flag or a runtime registry entry. No deployment model may reach directly into another. - -### P4. One input boundary, one output boundary - -**Input**: everything that enters the controller (HTTP `QuerySpec`, Prometheus query log replay, YAML workload, capability-miss callback) normalizes into a single `QueryWorkload` type in core — a collection of `QuerySpec`s each feeding L1→L2→L3. - -**Output**: L5 emitters (OpAMP `RemoteConfig`, backend `StreamingConfig` POST, one-shot YAML file, rewritten DataFusion `LogicalPlan`) all implement a `PlanEmitter` trait. Deployment models supply emitter implementations; core doesn't know which emitters exist. - -This is the "extension point for future deployment models" — a new deployment model adds an L4 rule set + an L5 emitter and registers them. - -### P5. No feature-flag spaghetti - -If something must be optional (e.g. OpAMP for deployments that don't run OTel collectors), it's a separate crate, not a `cfg` block in core. - -## 5. Repo layout - -### 5.1 Landed layout (today) - -The workspace is split by pipeline role into a layer-named crate stack. The -dependency arrows only ever point **up** — `asap-ir` never depends on a front -end, an optimizer, or a runtime. - -Dependency table (clearer than arrows). Every crate depends on `asap-ir`; -`asap-ir` depends on nothing above it. - -| Crate | Role | Depends on | -|---|---|---| -| `asap-ir` | **L3 canonical IR only** — `QueryExpr` + `AggIntent` + scalar expr + schema + names | *(nothing)* | -| `asap-l2` | L2 relational algebra + L2→L3 converter (`convert_root`, binder, resolution) | `asap-ir` | -| `asap-sketch` | L4 sketch-bound IR (`SummaryExpr` / `L4Node`) | `asap-ir` | -| `asap-plan` | optimizer — CSE (landed) + stubs | `asap-ir` | -| `asap-frontend-promql` | PromQL L1→L2 | `asap-ir`, `asap-l2`, promql-parser | -| `asap-frontend-sql` | SQL L1→L2 | `asap-ir`, `asap-l2`, datafusion | -| `asap-lower` | facade over both front ends + cross-language tests | the two `frontend-*` | -| `asap-e2e` | PromQL→L3 integration tests | `asap-frontend-promql` | - -``` -crates/ - ir/ # L3 canonical IR: query_expr + agg_intent + expr_ir + - # schema + names (+ types, workload). No query-language - # deps. (crate: asap-ir) - l2/ # L2 relational algebra + L2→L3 converter: relational + - # lower (convert_root) + canonicalize (shared post- - # lowering normalization, #34) + binder + - # column_resolution. (crate: asap-l2) - sketch/ # L4 sketch-bound IR (SummaryExpr / L4Node; constructed by - # asap-plan's bind pass, #98). (crate: asap-sketch) - plan/ # optimizer layer: CSE + the L3→L4 bind pass + the - # sketch-vs-exact boundary (#98); cost_model is a - # stub (#6/#33). (crate: asap-plan) - frontend-promql/ # PromQL L1→L2, promql-parser; + histogram.rs sample-type - # metadata (#79). (crate: asap-frontend-promql) - frontend-sql/ # SQL L1→L2, datafusion. (crate: asap-frontend-sql) - lower/ # facade re-exporting both front ends; owns the - # cross-language equivalence tests. (crate: asap-lower) - e2e/ # PromQL→L3 integration tests + fixtures. (crate: asap-e2e) -``` - -Two isolation wins: -- **The front ends quarantine their parsers**: a PromQL-only caller depends on - `asap-frontend-promql` and never compiles DataFusion, and vice-versa (verified - with `cargo tree`). This is why the old monolithic `LoweringError` was - partitioned into `PromqlError` / `SqlError` — a shared error embedding - `DataFusionError` would have leaked DataFusion into the PromQL crate. -- **L3-only consumers stay lean**: `asap-sketch` and `asap-plan` depend on - `asap-ir` alone and never pull the L2 relational tree / converter / binder - (`asap-l2`) — only the front ends, which actually *lower* queries, need it. - -**Not yet built** (see §5.2): the L4 optimizer engine + rule library, the L5 -physical framework, the runtime service, the `deployment-model-*` crates, the -proto crate, and the `bin/` entrypoints. `asap-plan` is the placeholder where -the optimizer work lands. - -### 5.2 Target layout (planned) - -The full target the landed stack is growing into — the optimizer/physical -frameworks, runtime, and per-deployment-model crates are the main additions. -(Note: the target originally imagined one `core/` crate with submodules; -the landed layout instead splits those submodules into the separate crates in -§5.1. §6.0 maps between the two.) - -``` -ASAPController/ -├── Cargo.toml # workspace -├── README.md -├── docs/ -│ ├── design.md # this file -│ ├── migration-plan.md # the companion file -│ ├── deployment-models/ # per-deployment-model design notes -│ └── adr/ # architecture decision records -├── proto/ -│ ├── asap_control.proto # Plan IR + QueryWorkload on the wire -│ └── opamp.proto # vendored from DataCollector -├── crates/ -│ ├── core/ # Shared infrastructure across all 5 layers; no I/O -│ │ ├── query_language/ # L1: per-language parsers — promql/, sql/, datafusion/, elasticdsl/ -│ │ ├── logical_plan/ # L2: per-language algebra tree (Aggregate/Window/Filter/…) -│ │ ├── intent_algebra/ # L3 IR: QueryExpr + AggIntent + Schema/HasSchema (intent only) -│ │ ├── sketch_algebra/ # L4 IR: SketchExpr (sketch-bound — kind + params committed) -│ │ ├── lower/ # L1→L2→L3 lowering passes (one entry per language) -│ │ ├── optimizer/ # L4 framework — produces SketchExpr from QueryExpr: -│ │ │ ├── engine/ # rule driver — fixed-point iteration, cycle detection, priority -│ │ │ ├── trait/ # OptimizerRule + RuleCategory (PushDown / Fusion / Elim / Bind) -│ │ │ ├── rules/ # shared rule library (e.g. sketch-binding rules; stream-vs-batch picker) -│ │ │ └── cost/ # CostModel trait + generic impls (memory budget, accuracy degradation) -│ │ ├── physical/ # L5 framework: -│ │ │ ├── planner_trait/ # PhysicalPlanner trait -│ │ │ ├── stage_allocator/ # generic topology-driven allocator -│ │ │ ├── topology/ # Topology descriptor types (edge/gateway/backend, single, zero) -│ │ │ ├── executor/ # Executor type — concrete runtime instance occupying a stage -│ │ │ └── sketch_catalog/ # candidate sketch types + parameter constraints -│ │ ├── pipeline/ # orchestrates L1→L2→L3→L4→L5, parameterized on deployment model -│ │ ├── workload/ # QueryWorkload wrapper around QuerySpecs -│ │ ├── emit/ # PlanEmitter trait — implemented by deployment model L5 -│ │ ├── registry/ # DeploymentModelRegistry, DeploymentModelId -│ │ └── telemetry/ # tracing macros, metric names (no exporter) -│ ├── runtime/ # service skeleton — HTTP, OpAMP, replanner -│ │ ├── http/ # axum surface — /plan, /replan, /metrics, /status -│ │ ├── opamp/ # WebSocket OpAMP server -│ │ ├── monitor/ # Scraper, Thresholds, Violation -│ │ ├── replan/ # Replanner — SLA + expiry triggers -│ │ ├── store/ # PlanStore, WorkloadStore -│ │ └── backend_client/ # HTTP client (pushes to ASAPQuery-backend, etc.) -│ ├── deployment-model-asaplifecycle/ # thin — picks rules, 3-stage topology, OTel/backend emitters -│ │ ├── rules.rs # L4 rule selection + DC-specific rules (stage-aware push-down) -│ │ ├── topology.rs # 3-stage: edge / gateway / backend -│ │ ├── cost.rs # deployment-model-specific cost impls — delta / online / pareto / tco -│ │ └── emit/ # OpAmpRemoteConfig + AsapqueryBackendConfig emitters -│ ├── deployment-model-asapquery/ # thin — rules, 1-stage topology, YAML emitters + query-log input -│ │ ├── rules.rs # L4 rule selection + sketch-binding rule (split from map_statistic_*) -│ │ ├── topology.rs # 1-stage: backend-only -│ │ ├── emit/ # StreamingConfig.yaml + InferenceConfig.yaml -│ │ ├── query_log/ # extra L1 input: Prometheus query-log replay -│ │ └── schema/ # PromQLSchema discovery from Prometheus (feeds L1) -│ ├── deployment-model-asapfusion/ # thin — DF-flavored rules, 0-stage, in-process emit -│ │ ├── rules.rs # L4 rules for DataFusion LogicalPlan (sketch-aware rewrites) -│ │ ├── topology.rs # 0-stage: in-process -│ │ ├── emit/ # rewritten DataFusion LogicalPlan -│ │ ├── executor/ # DataFusion SessionContext wrapper (library-mode execution) -│ │ └── sketch_support/ # asap_sketchlib-backed rewrites -│ ├── control-proto/ # generated from proto/ (tonic/prost) -│ └── testing/ # test fixtures + harness shared across deployment models -└── bin/ - ├── asap-controller/ # long-running service with all deployment models - │ └── main.rs # axum + OpAMP + replanner + registered deployment models - ├── asap-query/ # one-shot CLI for deployment-model-asapquery (what asap-planner-rs is today) - │ └── main.rs # clap — read workload YAML, emit two YAMLs - ├── asap-lifecycle/ # OPTIONAL: standalone service with only deployment-model-asaplifecycle - │ └── main.rs # slimmer image — no DataFusion, no query YAML emitters - └── asap-fusion-bench/ # OPTIONAL: benchmark harness over deployment-model-asapfusion - └── main.rs # criterion entry; used by researchers -``` - -**Per-deployment-model standalone binaries** are first-class. Each `bin//` is a thin shell that `use`s only the deployment model crates it needs — so `bin/asap-lifecycle/` doesn't pull `datafusion` into its dep tree, and `bin/asap-fusion-bench/` doesn't pull `axum`/`opamp`. Feature flags on the workspace root let you `cargo build -p asap-lifecycle` and get a minimal binary. - -### Why three deployment model crates today, not one monolithic `deployment-models/` - -Each deployment model has a different problem shape: - -| | **lifecycle** | **query** | **fusion** | -|---|---|---|---| -| Input | workload + live metrics | workload YAML / query log | DataFusion `LogicalPlan` | -| Decision unit | end-to-end staged pipeline | per-aggregation YAML | per-operator rewrite | -| Output | OpAMP OTel config + `StreamingConfig` YAML | `streaming_config.yaml` + `inference_config.yaml` | rewritten `LogicalPlan` | -| Trigger | QuerySpec, SLA violation, expiry | one-shot CLI invocation | a DataFusion session constructing a query | -| Cost model | accuracy × latency × $ staged | single-query accuracy/latency | operator selectivity / sketch feasibility | - -Mashing them into one crate means the union of all their dependencies (sqlparser + promql-parser + DataFusion + OpAMP proto + Prometheus client) bleeds into every downstream user. Separating them means a user who only needs `deployment-model-asapfusion` (an offline query-optimization benchmark, say) can depend on it without pulling OpAMP. - -## 6. Core crate details (layers 1–3 + driver) - -Core is not a trait-stubs library. It ships real L1/L2/L3 code lifted from DC's `controller/src/query_parser/` + `controller/src/algebra/` and exposes a small set of traits for L4/L5 plugin points. - -### 6.0 `core::` → landed crate map - -This section predates the crate split; the `core::` names below map to -the landed crates (§5.1) as follows: - -| §6 name | Landed crate / module | Status | -|---|---|---| -| `core::query_language` (L1) | `asap-frontend-promql`, `asap-frontend-sql` (parse step) | landed (PromQL + SQL; DataFusion/ElasticDSL planned) | -| `core::logical_plan` (L2) | `asap-l2::relational` (emitted by the front ends) | landed | -| `core::lower` (L1→L2→L3) | `asap-frontend-*` (L1→L2) + `asap-l2::convert_root` (L2→L3, with the binder + column resolution) | landed, split per language | -| `core::intent_algebra` (L3) | `asap-ir::intent_algebra` | landed | -| `core::sketch_algebra` (L4 IR) | `asap-sketch` | landed | -| `core::optimizer` (L4 framework) | `asap-plan` | landed, but not as a general rule engine — see §6.4 below for what's actually there vs. the general `OptimizerRule`/`RuleEngine` this section originally sketched (never built; superseded by the narrower `boundary`/`bind`/`CostModel` design, #98) | -| `core::cost` | `asap-plan::cost_model` | landed (the `CostModel` trait + `DefaultCostModel`; deliberately narrow — see §6.4) | -| `core::physical` (L5) | — | planned | -| `core::pipeline`, `core::emit`, `core::registry`, `core::workload` | `asap-ir::workload` (workload types only); rest planned | partial | - -The prose in §6.x describes the intended design; read the crate names through -this map. - -### `core::query_language` — Layer 1 - -Per-language parsers, one module each: - -- `promql/` — wraps `promql-parser`. Input: `&str` PromQL. Output: `PromqlAst`. -- `sql/` — wraps `sqlparser`. Input: `&str` SQL. Output: `SqlAst`. -- `datafusion/` — wraps DataFusion's own parser. Output: `DfAst`. -- `elasticdsl/` — wraps `elastic_dsl_utilities`. Output: `EsAst`. - -Each returns a language-flavored AST type. No sketch awareness. - -### `core::logical_plan` — Layer 2 - -A **per-language** algebra tree — one `enum LogicalPlan` per language. Preserves language-specific semantics (PromQL instant vs range vector, SQL window frames, Elastic buckets) that would be lossy to collapse this early. Types are symmetric: `Aggregate { AggFunc }`, `Window`, `Filter`, `Sort`, `Limit`. No sketch names yet. - -### `core::intent_algebra` — Layer 3 - -The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `PromQLSubquery` beyond the structural range marker), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP deployment models) via a `Source` sum type inside `QueryExpr::Scan`. - -#### Design rules for L3 - -1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). The exception is when an "intent" is its own physical primitive: heavy-hitter top-k is served by sketches (SpaceSaving, CMS-with-heap) as a single operation, so `AggIntent::TopK` is a first-class intent at L3 — distinct from the generic `Sort + Limit` operator pair, which still appears in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). -2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3 — unless the operation is genuinely semantically distinct. `HistogramQuantile` is the worked example of that line (revised by #43/#79): an early draft lowered it to bucket reads + a generic `Quantile`, but classic cumulative-`le`-bucket interpolation is **not** the same operation as a sketch-able quantile — pre-aggregated bucket counts can't be re-sketched, while raw samples / native histograms can (with an accuracy target). So `histogram_quantile` has **two lowerings**: the classic-bucket form → `AggIntent::HistogramQuantile` (exact interpolation, no accuracy target), the native/raw form → generic `AggIntent::Quantile` (sketch-able). Discrimination is by declared sample type (`HistogramCatalog`, `crates/frontend-promql/src/histogram.rs`) with a structural `by (le)`/`_bucket` heuristic as fallback. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. -3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. -4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. -5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding context. The shape is a DAG, not a tree, because **a producer node can have multiple parents that share its precomputed intermediate state** — instead of recomputing the same sub-expression once per consumer, the consumers fan in to a single node and read its output. Three sources of fan-in: - 1. **Explicit, in-query.** SQL CTEs (`WITH name AS (expr) SELECT ... FROM name JOIN name AS n2 ON ...`) and PromQL recording rules name a sub-expression and reference it N times; each reference becomes a `QueryExpr::Ref(name)` parent of the named producer. - 2. **L4 reuse rules.** When two queries planned together both need (e.g.) a p99 quantile of the same series, the optimizer can introduce a shared sketch node whose output feeds both — even though neither query author wrote a CTE. - 3. **L4 stage / shard structure.** A pre-aggregate computed once on the edge can feed multiple downstream gateway-stage operators. - - `CostModel::workload_cost` credits a shared producer's build cost once across all consumers, which is what makes reuse a Pareto win. Tree IRs lose this — they have to duplicate the producer for every consumer. - -```rust -// Sketch of the node set — crates/ir/src/intent_algebra/query_expr.rs is the -// source of truth. Shipped additions beyond this draft: scalar leaves -// `Scalar(f64)` / `EvalTime` (#35/#46), the scalar⇄vector bridges -// `VectorFromScalar` / `ScalarFromVector` (#48), `Relabel` (label rewrite, -// #50), `Sample` (limitk/limit_ratio series sampling, #86), `InfoJoin` (info- -// metric label enrichment, #84), and the structural `TimeRange` range marker. -pub enum QueryExpr { - // ── Base relations ──────────────────────────────────────────────────── - /// A metric stream / table / join. Outermost leaf. - Scan { source: Source, predicates: Vec }, - /// Reference to a CTE / let-binding by name; resolved at plan time. - Ref(String), - - // ── Filtering & projection ──────────────────────────────────────────── - /// σ — row-level filter (WHERE / PromQL label matchers). - Filter { child: Box, pred: Predicate }, - /// π — column projection (SELECT list). - Project { child: Box, cols: Vec }, - - // ── Aggregation (logical, intent-only) ──────────────────────────────── - /// γ + α — GROUP BY + aggregate intents, with optional HAVING. - /// `aggs` carry `AggIntent`; concrete sketch / non-sketch operator - /// is chosen by L4 and lives in the L4-extended IR (`SketchExpr`). - Aggregate { child: Box, reduction: Reduction, - aggs: Vec, having: Option }, - - // ── Time / streaming windows ────────────────────────────────────────── - /// ψ — tumbling / sliding / session window over the time axis. Defines - /// the lifecycle (flush / reset bounds) of any aggregate in its sub-tree / sub-DAG. - /// PromQL `[5m]` and streaming windows lower here. SQL `OVER (...)` - /// analytic frames are a different node — see `WindowFunc` below. - Window { child: Box, kind: WindowKind, - size: Duration, slide: Option }, - - // ── Distributed-execution structure ─────────────────────────────────── - // There is no `Partition` node (issue #12). Grouping has one home per - // concept: a reducing GROUP BY → `Aggregate.reduction`; per-group *ranking* - // (split without reduce) → `Sort.partition_by` (below); a parallel/sharding - // split is physical and lives in L5's stage allocator, not the symbolic IR. - // All three carry the same `GroupKeys` type — a newtype over the positional - // `Vec` — so "per-group keys" has a single spelling. - /// δ — SQL `DISTINCT` / row deduplication on `cols`. - Distinct { child: Box, cols: Vec }, - /// ⊕ — union of sub-results from independent stages or shards (the - /// exact-merge case). Sketch unions are a separate node in `SketchExpr` - /// because they carry sketch-family / params type constraints. - Merge { children: Vec }, - - // ── Joins (logical) ─────────────────────────────────────────────────── - /// Logical join. L4 picks the physical alternative — `HashJoin` / - /// `SortMergeJoin` / `SketchJoin` (e.g. KMV / theta-sketch / join-sample) - /// — based on selectivity, memory budget, and accuracy target. The - /// sketch-aware variant lives in `SketchExpr::SketchJoin`. - Join { kind: JoinKind, left: Box, right: Box, - pred: Option }, - - // ── Set operators ───────────────────────────────────────────────────── - /// UNION / INTERSECT / EXCEPT, with or without ALL. - SetOp { kind: SetOpKind, all: bool, - left: Box, right: Box }, - - // ── Ordering & limiting ─────────────────────────────────────────────── - /// Generic order-by — survives L3 for non-heavy-hitter cases - /// (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`). `partition_by` - /// makes the ordering per-group ("rank within each group"): the home for a - /// generic (non-heavy-hitter) `topk by (host)` / `bottomk` grouping and SQL - /// `… OVER (PARTITION BY …)`. Row-preserving (schema pass-through); empty = - /// a global order-by. (Issue #12: this replaces the old `Partition` node.) - Sort { child: Box, keys: Vec, partition_by: GroupKeys }, - /// `LIMIT n OFFSET k`. The heavy-hitter shape (`ORDER BY count DESC - /// LIMIT k`, PromQL `topk(k, …)`) is recognised at L1→L2→L3 lowering - /// and produces `AggIntent::TopK` rather than generic `Sort + Limit`, - /// so heavy-hitter sketches (SpaceSaving, CMS-with-heap) bind on the - /// intent. Generic `Sort + Limit` flows through unchanged. - Limit { child: Box, n: u64, offset: u64 }, - - // ── Subquery / CTE ──────────────────────────────────────────────────── - Subquery { child: Box, alias: String }, - /// SQL `WITH name AS (expr) IN body`; lowering target for PromQL - /// recording-rule bindings. - LetBinding { name: String, expr: Box, body: Box }, - - // ── Analytic (OVER) window functions ────────────────────────────────── - /// SQL `OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)`. - /// Distinct from `Window` above — that is a streaming/tumbling window - /// over the time axis; this is an analytic frame over already-grouped rows. - WindowFunc { child: Box, func: WindowFuncKind, - partition_by: GroupKeys, order_by: Vec, - frame: Option }, - - // ── Binary composition ──────────────────────────────────────────────── - /// Arithmetic / comparison / boolean composition between two relational - /// sub-expressions (PromQL binary ops including `and`/`or`/`unless`, - /// SQL boolean composition). - BinaryOp { op: BinaryOpKind, lhs: Box, rhs: Box, - vector_match: Option }, -} - -pub enum WindowKind { Tumbling, Sliding, Session } - -// As shipped (crates/ir/src/intent_algebra/query_expr.rs), Source is thinner -// than this draft: `TimeSeries { metric }` / `Table { table_ref }` only. The -// time range is a separate `TimeRange` node, label filters live on -// `Scan.predicates`/`Filter`, and columns on `Scan.schema`. `Source::Join` is -// NOT implemented — joins are the `QueryExpr::Join` node over two Scans. -pub enum Source { - /// Time-series input — deployment-model-asapquery / deployment-model-asaplifecycle shape. - TimeSeries { metric: MetricRef }, - /// Tabular input — deployment-model-asapfusion / future-OLAP shape. - Table { table_ref: TableRef }, - // Future (unimplemented draft ideas): a recursive `Join` source, - // WindowedStream, Subquery — added by deployment models that need them. -} - -pub enum DataModel { TimeSeries, Tabular, Any } - -impl Source { - pub fn data_model(&self) -> DataModel { /* … */ } - /// Output schema produced by this leaf (see "Schema flow"). - pub fn schema(&self, catalog: &SchemaCatalog) -> Schema { /* … */ } -} -``` - -#### What was removed during cleanup, and why - -| Removed | Reason | Replacement | -|---|---|---| -| `TopK { k, by }` *as a `QueryExpr` node* | A `QueryExpr`-level top-k operator collapses two distinct concepts: (a) the *intent* of "compute heavy hitters", which has its own sketch primitive, and (b) the generic operator pair `Sort + Limit`, which doesn't. Splitting them puts heavy-hitter logic at the right level | Heavy-hitter intent → `AggIntent::TopK` (L3, recognised by L1→L2→L3 lowering of `ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`); generic ordering+limit → `Sort + Limit` `QueryExpr` nodes (unchanged). Both retained — they describe different things | -| `WindowedAgg { intent, window }` | Equivalent to `Window` over `Aggregate`; redundant. Tumbling/sliding kind moves onto `Window::kind` | `Window { kind: …, … } → Aggregate { aggs: [intent] }` | -| `SketchAgg { intent, col }` | Sketch-bound; L3 must be intent-only | `Aggregate { aggs: [intent] }` at L3; L4 emits `SketchExpr::SketchAgg` | -| `JoinSketch { outer, inner, key }` | Sketch-bound physical alternative; L3 must be intent-only. Several papers describe sketch-of-join (KMV, theta, join-sample) — picking one is an L4 cost decision, not an L3 surface choice | `Join { … }` at L3; L4 emits `SketchExpr::SketchJoin` when a `Bind*OnJoin` rule fires | -| `HistogramQuantile { phi }` *as a `QueryExpr` node* | A `QueryExpr`-level operator was the wrong level. **Revised by #43/#79**: the operation itself is *not* purely a language artifact — classic cumulative-`le`-bucket interpolation is semantically distinct from a sketch-able quantile (bucket counts can't be re-sketched), so it earns an intent, not a node | `AggIntent::HistogramQuantile { q }` for the classic-bucket form (exact interpolation); the native-histogram / raw-samples form lowers to the generic sketch-able `Aggregate { aggs: [Quantile{q, …}] }`. Discriminated by declared sample type (`HistogramCatalog`, #79) with a structural heuristic fallback | -| `PromQLSubquery { range, resolution }` | Driver construct — asks the engine to evaluate the inner expression at N timestamps; not a single DAG node | Expanded by PromQL L1→L2 lowering into a set of independent `QueryExpr` instances, not preserved at L3 | -| `Dedup { col }` | Single-column-only spelling of SQL `DISTINCT` | Renamed to `Distinct { cols }`, generalised to N columns | - -#### Schema flow — every L3 edge carries a typed schema - -Every node has a derivable output schema given its input schemas and parameters. The DAG is type-checked: a `Filter` whose predicate references a column not in its child's output schema fails at plan time. - -```rust -pub struct Schema { - pub fields: Vec, - /// Index into `fields` for the time axis, if any. PromQL leaves carry one; - /// SQL leaves may or may not. - pub time_index: Option, - /// Optional metadata for reuse-aware planning. Each inner `Vec` is - /// a set of column indices that together uniquely identify rows; the outer - /// `Vec` allows multiple unique-key sets (e.g. primary key + another unique - /// constraint). - /// - /// **Populated by**: the per-node input/output spec (e.g. `Aggregate { by, .. }` - /// emits `unique_keys = [by]`; `Distinct { cols }` adds `cols`; `Project` - /// carries forward the retained columns; most other nodes pass through). - /// - /// **Consumed by**: `CostModel::workload_cost` only — the reuse-aware path - /// that credits shared sub-expressions across multiple queries. Single-query - /// plans, the `Bind*` rules, push-down rules, and L5 emitters do not read - /// this field. If the reuse path is deferred (see §13 future work), this - /// field is dead weight; it lives here so the metadata is available the - /// moment workload-aware planning lands without requiring an L3-wide - /// schema change. - pub unique_keys: Vec>, -} - -pub struct Field { - pub name: String, - pub dtype: DataType, // Int64 / Float64 / Utf8 / Map / … - pub nullable: bool, -} - -pub trait HasSchema { - fn input_schemas(&self) -> Vec<&Schema>; - fn output_schema(&self, inputs: &[&Schema], cat: &SchemaCatalog) -> Schema; -} -``` - -Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/intent_algebra/schema.rs`). Each row reads independently: every column position, type, and constraint is named explicitly rather than carried by a shorthand like `S` or `L` / `R`. - -| Node | Input schemas | Output schema | -|---|---|---| -| `Scan { source }` | none — leaf node | `source.schema(catalog)` — derived from the source's catalog metadata (TimeSeries metric labels + value + timestamp; or Table columns from `information_schema`; or recursive `Source::Join`) | -| `Ref(name)` | none — pointer node | the output schema of the `LetBinding` whose `name` matches; resolved at plan time | -| `Filter { pred }` | one input schema (the child's output) | the child's input schema unchanged — `Filter` is a row-level refinement; no columns added, removed, or re-typed | -| `Project { cols }` | one input schema | the input schema projected to `cols` — fields filtered and reordered to match `cols`; `time_index` and `unique_keys` carried over for retained columns | -| `Aggregate { by, aggs }` | one input schema | the `by` columns (carried verbatim from input) followed by one new column per entry in `aggs`, each named and typed by `AggIntent::output_type(input_field)`; `unique_keys = [by]` | -| `Window { kind, size, slide }` | one input schema, **must** contain a `time_index` field | the input schema extended with synthetic `window_id` and `window_start` / `window_end` metadata fields | -| `Distinct { cols }` | one input schema | the input schema with `unique_keys` tightened to include `cols`; field types unchanged | -| `Merge` | N input schemas, all union-compatible (same field names, same types, same nullability, same `time_index` position if any) | the first input's schema (representative; checked union-compatible with the rest) | -| `Join { kind, pred }` | two input schemas — left and right children | the concatenation of left's fields and right's fields, minus columns that USING / NATURAL deduplicates; nullability widened on the OUTER side for outer joins | -| `SetOp { kind, all }` | two input schemas, must be union-compatible (as for `Merge`) | the left input's schema | -| `Sort { keys, partition_by }` | one input schema; every field referenced in `keys` / `partition_by` must be present in it | the input schema unchanged (row-preserving) — `partition_by` makes the order per-group (`topk by (…)` / `OVER (PARTITION BY …)`); there is no separate `Partition` node (issue #12) | -| `Limit { n, offset }` | one input schema | the input schema unchanged | -| `Subquery { alias }` | one input schema | the input schema with `alias` applied as the table alias to all field names | -| `LetBinding { name, expr, body }` | `expr` produces an intermediate schema bound to `name`; `body` consumes it (and any other in-scope bindings) | the `body`'s output schema | -| `WindowFunc { func, partition_by, order_by, frame }` | one input schema; every field in `partition_by` / `order_by` must be present | the input schema extended with one new column carrying the analytic-function output (named after `func`, typed per `func`) | -| `BinaryOp { op, vector_match }` | two input schemas — left and right operands. PromQL vector-match constraints (`on`/`ignoring` + `group_left`/`group_right`) govern label-set compatibility | for arithmetic/comparison `op`: the left input's schema with the value column re-typed to the result of `op`; for boolean `op` (`and`, `or`, `unless`): the left input's schema with a boolean value column | - -#### DAG schema, DB schema, sketch catalog — three distinct metadata sources - -These three are sometimes conflated and shouldn't be. Only the first two are *schemas* (descriptions of stream / table shape); the sketch catalog is a *registry* of available primitives, not a description of a stream: - -| Source | Where it lives | What it describes | Who reads it | -|---|---|---|---| -| **DAG schema** | On every edge of the L3 / L4 / L5 DAG (`Schema` above) | Columns + types flowing between operators | L4 rules (selectivity estimation, push-down legality), L5 emitter | -| **DB / source schema** | The query target (Prometheus TSDB metric metadata, SQL `information_schema`, DataFusion catalog) | What metrics / tables / columns exist in the data plane, with their types and indexing | `core::lower::*` to resolve names during L1→L2; exposed through a `SchemaCatalog` interface | -| **Sketch catalog** | `core::physical::sketch_catalog` (built at startup; static) | What sketches the runtime can build; what intents each one serves; mergeability, accuracy / confidence guarantees, supported aggregation keys, parameter ranges | L4 binding rules to choose a sketch for an `AggIntent`; L5 to instantiate the sketch | - -L1→L2 lowering reads the **DB schema** to resolve symbols. L3 onward, every edge carries a **DAG schema** that is type-checked locally. L4 binding rules consult the **sketch catalog** to map an intent to a concrete sketch under the deployment's constraints. They are three separate inputs to three distinct decisions. - -#### `AggIntent` — what to compute, not how - -```rust -pub enum AggIntent { - // Data-model-agnostic - Count { accuracy: AccuracyTarget }, - Sum, - Min, Max, - Quantile { q: f64, accuracy: AccuracyTarget }, - /// Heavy-hitter top-k. Distinct from generic `Sort + Limit` because a - /// dedicated sketch primitive (SpaceSaving, CMS-with-heap, Misra-Gries) - /// computes it as a single operation. L1→L2→L3 lowering produces this - /// when it recognises a heavy-hitter shape (`ORDER BY count DESC LIMIT k`, - /// PromQL `topk(k, …)`); other ordering+limit cases stay as - /// `QueryExpr::Sort + QueryExpr::Limit`. - TopK { k: usize, by: Vec, accuracy: AccuracyTarget }, - Cardinality { accuracy: AccuracyTarget }, - - // Time-series streaming derivatives — specific operations, not just - // "Sum / Count over a Window". `Rate` is the per-second average derivative - // computed with PromQL's counter-reset adjustment, not a generic windowed - // mean. Kept distinct because (a) they have counter-reset semantics that - // exact `Sum` does not, and (b) sketch backends specialised for - // derivatives (e.g. delta-set aggregator) bind on these intents directly. - Rate { window: Duration }, - Increase { window: Duration }, - - // Tabular / OLAP — added as deployment models demand - // CorrelatedSubqueryCount { … }, ApproxJoinCardinality { … }, -} - -impl AggIntent { - /// Which data-model this intent semantically requires. L4 rules - /// consult this to skip non-applicable intents (e.g. `Rate` over - /// a `Source::Table` is nonsense). - pub fn requires(&self) -> DataModel { /* … */ } - /// Output column type — used by L3 schema derivation for `Aggregate`. - pub fn output_type(&self, input: &Field) -> DataType { /* … */ } - /// Which sketch families in the catalog can serve this intent. - /// Read by L4 binding rules. - pub fn candidate_sketches(&self) -> &'static [SketchKind] { /* … */ } -} -``` - -**Why `TopK` *is* an intent (and `Sort + Limit` is not collapsed into it).** Heavy-hitter top-k has a dedicated sketch primitive — SpaceSaving / CMS-with-heap / Misra-Gries compute it in a single pass with sub-linear memory. L4 binding rules want to fire on the *intent* "give me the top-k frequent items" rather than on a syntactic shape, the same argument that makes `Quantile` an intent rather than a `Sort` + "pick the φ-th element" pattern. So `AggIntent::TopK` lives at L3. Generic `QueryExpr::Sort + QueryExpr::Limit` *also* survives, because not every order-by-limit query is heavy-hitter (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`); these have no sketch alternative and stay as generic operators. The canonical-form invariant is preserved because L1→L2→L3 lowering picks one or the other deterministically based on whether it recognises a heavy-hitter pattern. - -**Why no `QuantileOverTime` intent?** It duplicated `Quantile` over a `Window`. The window — its kind, its size, its slide — is fully captured by the surrounding `Window { … }` node; the quantile *operation* is the same regardless of whether the input was a windowed time-series or a row-grouped table. PromQL's `quantile_over_time(0.99, m[5m])` lowers cleanly to `Window{size=5m} → Aggregate{aggs:[Quantile{q=0.99}]}`. One intent halves the L4 rule surface (one bind rule per operation, not per language-flavor of an operation). - -**Why `Rate` and `Increase` survive that argument.** They are not "Sum / Count over a Window with a different name" — they include PromQL's counter-reset adjustment, which is a non-trivial transformation an exact `Sum` does not perform. They earn distinct intent variants because they parameterise different physical operators (delta-set aggregators bind on these intents directly). If a non-PromQL streaming language has the same notion (e.g. SQL `RATE() OVER (RANGE)`), it lowers to the same intent — the intent vocabulary names the operation, not the language. - -### `asap-sketch` — Layer 4 IR - -The L4 IR itself (`L4Node`/`SummaryExpr`, the per-node schema rules, the -type-system invariants that make it robust) now lives in -[`docs/l4node-execution-model.md`](./l4node-execution-model.md) — see its -"Planning-time L4" section — alongside the serving-time execution model -that consumes the same IR (#155/#156), rather than duplicated here. - -L4 binding rules consume L3 `QueryExpr` (`asap-ir::intent_algebra`) and -produce the sketch-bound L4 IR (`asap-sketch`); this is the IR L5 emitters -consume. The two-IR split — intent-only L3 and sketch-bound L4, in two -separate crates — gives L4 rule application a clean type signature and -means the boundary can't be silently violated. - -**Why a `Source` sum instead of two parallel `QueryExpr` trees:** most `QueryExpr` nodes (`Filter`, `Aggregate`) are data-model-agnostic — filter semantics are the same whether the input is a time-series window or a table scan. Only the leaf `Scan` differs. Keeping one tree with a polymorphic leaf means L4 rules like `BindKllOnQuantile` work uniformly across both data models; rules that care about data-model specifics (stage-aware push-down for TS; join-selectivity for tabular) gate on `source.data_model()` + `intent.requires()`. - -**Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values. That stream can come from a time-series window (`Source::TimeSeries`) or a table column (`Source::Table`); the sketch doesn't know or care. So `BindKllOnQuantile` works identically regardless of `Source`. - -### `core::lower` — L1 → L2 → L3 passes - -One pass per language, each producing the same `intent_algebra::QueryExpr`: - -```rust -pub fn lower_promql(ast: PromqlAst, schema: &MetricSchema) -> Result; -pub fn lower_sql(ast: SqlAst, schema: &TableSchema) -> Result; -pub fn lower_datafusion(ast: DfAst, ctx: &DfContext) -> Result; -pub fn lower_elasticdsl(ast: EsAst, schema: &IndexSchema) -> Result; -``` - -Once a query hits L3 it's language-agnostic. All deployment models downstream see the same IR. - -### `core::pipeline` — orchestration - -The L1→…→L5 driver. Parameterised on a deployment model's optimizer rules (L4) + emitter (L5): - -```rust -pub struct Pipeline { - deployment_model: S, -} - -impl Pipeline { - pub fn run(&self, workload: &QueryWorkload) -> Result { - let l3: Vec = workload.queries() - .iter() - .map(|q| self.parse_and_lower(q)) // L1→L2→L3 - .collect::>()?; - let l4 = self.deployment_model.optimizer().optimize(l3)?; // L4 - let l5 = self.deployment_model.physical().lower(l4)?; // L5 - self.deployment_model.emitter().emit(&l5) // L5 → bytes/protobuf/DF plan - } -} -``` - -Core owns the driver. Deployment models own what goes into each deployment-model-specific seam. - -### `core::optimizer` — Layer 4 framework - -**This section originally sketched a general Cascades-style rule engine -(`OptimizerRule` + `RuleEngine`, fixed-point rewriting, a shared rule -library) before any of L4 was implemented.** That design was never -built — what actually shipped in `asap-plan` (crate `crates/plan`) is -narrower and more specific to the one decision L4 genuinely needs to -make (which physical summary realizes an `AggIntent`), plus the -supporting pieces that decision needs. The old sketch is kept below, -struck through in spirit, so anyone who remembers it (or finds it in -git blame) can see explicitly that it was superseded rather than just -silently vanish — see the real architecture first. - -**What's actually there** (see §3 for the "implementation" vs. "bind" -vs. "match" terminology these use): - -- [`boundary::implementation_for`] / [`implementation_for_with`] — the - per-node decision (§3's "Implementation" row): `AggIntent → - Implementation` (`Sketch { kind, params }` | `ExactAccumulator { - kind, params }` | `PassThrough`). Exhaustive over the `AggIntent` - vocabulary — a new variant is a compile error until given an explicit - realization, so there's no silent fall-through. -- [`bind::implement_tree`] / [`implement_tree_with`] — walks a whole - `QueryExpr` tree calling `implementation_for` per node, emitting the - complete L4 `SummaryExpr`/`L4Node` DAG (§3's `implement_tree` row). - Deliberately conservative about *where* it looks for a realizable - `Aggregate`: a non-`Aggregate` parent (`Filter`, `Window`, ...) - wraps the whole subtree as `Logical` rather than rewriting through - it — see the note below on what a deployment does about that. -- [`cost_model::CostModel`] — the **one** pluggable extension point, - not a general rule interface: re-ranks the candidate summary - families `boundary::summary_candidates` returns for an intent (best - choice first), rather than rewriting the tree arbitrarily. - `DefaultCostModel` preserves the built-in static preference order; - every entry point not given an explicit `&dyn CostModel` runs - against it. Deliberately narrow (issues #6, #33) — "summary" already - covers non-sketch realizations the day a new `SummaryKind` variant - lands, so this one interface is meant to stay the only extension - point rather than growing into a second `RuleEngine`. -- [`cse::dedupe_subtrees`] — workload-level Common Sub-Expression - Elimination: hoists sub-DAGs structurally identical across ≥2 query - roots into shared `LetBinding`s so the cost model credits a shared - producer once. Scoped to the basic "≥2 roots, identical - `Aggregate`-child sub-trees" case; alpha-equivalence / schema-merge / - nested CSE is explicitly a downstream optimization, not part of the - IR contract. -- [`boundary::Matcher`] — §3's "Match" row: "does an already-available - `Implementation` satisfy a required one." Ships as a trait with no - default implementation and no shipped instance, same shape and - reasoning as `CostModel` — which `Implementation`s are actually - available anywhere is an inventory only a downstream deployment has. - -**What a deployment does beyond this**: `implement_tree`'s -conservative stop at non-`Aggregate` nodes, and any additional -algebraic rewriting beyond summary-candidate ranking (push-downs, -fusion, dead-code elimination, …), are *not* modeled in `asap-plan` at -all — they're each deployment's own problem today. ASAPQuery-backend's -`control_plane` crate is the reference example: its own -`sketch_algebra::rules::bind_*` modules (§3's "Bind #2") and -`optimizer::` module implement exactly this kind of deployment-specific -logic downstream, without `asap-plan` needing to know about it. - -
-Historical sketch (never built) — general rule engine + shared rule library - -```rust -// trait (never built) -pub trait OptimizerRule: Send + Sync { - fn name(&self) -> &'static str; - fn category(&self) -> RuleCategory; // PushDown | Fusion | Elim | Bind | ... - fn priority(&self) -> u16; - fn apply(&self, expr: &QueryExpr, c: &DeploymentConstraints) -> Option; -} - -// engine (never built) — fixed-point iteration, cycle detection, priority ordering -pub struct RuleEngine { /* ... */ } -impl RuleEngine { - pub fn new(rules: Vec>) -> Self { /* ... */ } - pub fn run(&self, exprs: Vec, c: &DeploymentConstraints) - -> Result, OptError>; -} - -// shared rule library (never built) — opt-in from deployment models -pub mod rules { - pub struct BindKllOnQuantile; // AggIntent::Quantile → bind KLL(k by accuracy) - pub struct BindCmsOnCount; // AggIntent::Count → bind CMS(w, d) - pub struct BindHllOnCardinality; // AggIntent::Cardinality → bind HLL(p) - pub struct FusionPassthrough; // Aggregate over Filter → push Filter under Aggregate - pub struct ElimNoopFilter; // Filter(true) → child - // ... etc - impl OptimizerRule for BindKllOnQuantile { /* ... */ } -} -``` - -`DeploymentConstraints` and the `Executor`/`StageId` registration -sketch that used to follow this belong to the L5 discussion — see -`core::physical` below, also never built. - -
- -### `core::physical` — Layer 5 framework - -**Status: planned, not yet built.** No `asap-physical` crate (or -equivalent) exists in this workspace today — the sketch below is the -intended design for L5 (stage allocation + emission), not a -description of shipped code. §6.0's crate map lists `core::physical` -as `—` / planned for exactly this reason. Treat the trait/struct names -below as a target to design against, not an API to depend on. - -```rust -pub trait PhysicalPlanner { - type Topology: TopologyDescriptor; - type Output; - fn lower(&self, l4: Vec, t: &Self::Topology) -> Result; -} - -pub trait TopologyDescriptor { - fn stages(&self) -> &[StageDescriptor]; - fn edges(&self) -> &[StageEdge]; -} - -// pre-baked topologies in core -pub mod topology { - pub struct ThreeStage { /* edge → gateway → backend */ } - pub struct SingleStage { /* backend-only */ } - pub struct ZeroStage; /* in-process */ -} - -/// Identifier for a stage role (categorical tier in the data lifecycle). -pub struct StageId(pub String); // "edge" / "gateway" / "backend" / "in-process" - -/// A **concrete runtime instance** that executes a sub-DAG of the plan. -/// One stage may have many executors — e.g. a 50-host edge fleet is 50 -/// executors all with `stage = StageId("edge")`; a singleton backend is -/// one executor at `stage = StageId("backend")`. The stage allocator -/// works at stage granularity; the deployment-model planner / emitter -/// materialises each stage's sub-DAG into one concrete config per executor. -pub struct Executor { - pub id: ExecutorId, // stable identifier across replans - pub stage: StageId, // which stage this executor occupies - pub capabilities: ExecutorCaps, // memory budget, available sketch backends, network neighbours - pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle -} - -pub struct ExecutorId(pub String); - -pub enum ExecutorAddr { - OpAmpAgent(AgentId), // OTel collector managed via OpAMP (DC lifecycle) - HttpEndpoint(Url), // ASAPQuery-backend, etc. - InProcess, // asap-fusion library-mode SessionContext -} - -// generic stage allocator — given a QueryExpr tree + a topology, decide which -// ops land on which stage subject to constraints. Stage-level only; per-executor -// fan-out happens in the deployment model's PhysicalPlanner using the executor -// list from `DeploymentConstraints::executors()`. -pub struct StageAllocator; -impl StageAllocator { - pub fn allocate( - &self, exprs: &[QueryExpr], topology: &T, c: &DeploymentConstraints, - ) -> Result, PlanError>; -} - -// sketch catalogue — what sketches exist, what they support, what params they -// accept. Built at startup from the registered sketch backends; queried by L4 -// binding rules to map an `AggIntent` → `SketchKind` + `SketchParams`, and by -// L5 to instantiate the chosen sketch. -pub struct SketchCatalog { - pub entries: Vec, -} - -pub struct SketchEntry { - pub kind: SketchKind, // Kll, Cms, Hll, DDSketch, KMV, Theta, … - /// Which `AggIntent` variants this sketch can serve. - pub supported_intents: &'static [IntentTag], // Quantile, Count, Cardinality, JoinCardinality, … - /// Mergeability — sketches built on disjoint inputs combine without re-scan. - /// (CMS / HLL / KLL / theta = mergeable; SpaceSaving = approximately; - /// some heavy-hitter variants = no.) - pub mergeable: Mergeability, - /// Whether the sketch supports point deletion (CMS update with -1, - /// deletable Bloom filter) — gates `SketchExpr::SketchDelete`. - pub deletable: bool, - /// Whether the sketch admits a linear inverse (CMS, theta, count-based) - /// — gates `SketchExpr::SketchSubtract`. - pub subtractable: bool, - /// Accuracy / confidence model: error bounds as a function of params. - /// `(eps, delta)` for randomised sketches; absolute error for KLL; etc. - pub accuracy: AccuracyModel, - /// Aggregation keys this sketch supports natively. Some sketches are - /// keyed (CMS over (label, value)); others are unkeyed (HLL). - pub aggregated_keys: KeyShape, // Unkeyed | KeyedScalar | KeyedTuple - /// Parameter ranges + defaults. Catalog rejects out-of-range params at - /// L4 bind time so L5 never sees an unsupported configuration. - pub param_ranges: ParamRanges, - /// Memory + CPU model used by `CostModel`. Function of params. - pub cost_model: SketchCostModel, -} -``` - -Deployment models use these pieces: - -```rust -// in deployment-model-asaplifecycle -use asap_physical::{StageAllocator, topology::ThreeStage, Executor}; - -impl PhysicalPlanner for LifecyclePlanner { - type Topology = ThreeStage; - type Output = Vec<(ExecutorId, ExecutorPlan)>; // one entry per executor - fn lower(&self, l4: Vec, t: &ThreeStage) -> Result<_, _> { - // 1. Stage-level: colour the DAG by StageId. - let assignments = StageAllocator.allocate(&l4, t, &self.constraints)?; - // 2. Per-executor fan-out: for each Executor in the deployment, - // pick the sub-DAG assigned to its stage and produce its config. - let executors: &[Executor] = self.constraints.executors(); - // deployment-model-specific post-processing: DC's delta_cost logic, - // backend_client push preparation, etc. - Ok(/* ... */) - } -} -``` - -### `core::plan` — shared traits bridging layers - -**Status: planned, not yet built — and a naming collision worth -flagging.** This `core::plan` (a cross-layer `DeploymentModel` trait -composing L4 rules + L5 topology + emission + constraints) is a -*different thing* from the real, landed `asap-plan` crate -(`crates/plan`), which §6.0's crate map lists under `core::optimizer` -(L4 only — see that section above for what's actually there). Neither -`DeploymentModel` nor the `OptimizerRule`/`PhysicalPlanner`/ -`PlanEmitter`/`DeploymentConstraints` types it references exist -anywhere in this workspace today; this section predates the crate -split and was never reconciled with it. Read as a target design for -how a deployment model might eventually compose L4+L5+emission, not a -description of `asap-plan`. - -```rust -pub trait DeploymentModel { - type Topology: TopologyDescriptor; - type EmitterOutput; - fn rules(&self) -> Vec>; - fn topology(&self) -> &Self::Topology; - fn physical(&self) -> &dyn PhysicalPlanner; - fn emitter(&self) -> &dyn PlanEmitter; - fn constraints(&self) -> &dyn DeploymentConstraints; -} -``` - -### `core::workload` - -One public type for every kind of input: - -```rust -pub enum QueryWorkload { - /// A single QuerySpec from an HTTP POST. DC's current entry point. - Single(QuerySpec), - /// A hand-authored YAML workload file. asap-planner-rs's current entry point. - AuthoredSet(Vec), - /// A Prometheus query log replayed. asap-planner-rs's alt entry point. - QueryLog(QueryLogReplay), -} - -pub struct QuerySpec { /* metric_name, aggregations, time_window, SLAs, ... */ } -``` - -All three deployment models take `&QueryWorkload`. Same type, different planners. - -### `core::cost` - -Extracted from DC's `planner/*` (delta, online, pareto, tco). Deployment models pick the models they need. - -```rust -pub trait CostModel { - fn accuracy(&self, plan: &Plan) -> Accuracy; - fn latency(&self, plan: &Plan) -> Duration; - fn dollars(&self, plan: &Plan) -> Dollars; - - /// Workload-level total cost. A straight sum of per-plan costs - /// under-estimates how good a plan is when multiple queries share - /// computation — a sketch or a precomputed aggregate built for - /// one query serves the others for free. Implementations must - /// identify reusable sub-expressions across `plans` and credit - /// their build cost once, so the planner prefers plans that - /// maximise reuse when the total-cost objective allows. - fn workload_cost(&self, plans: &[Plan]) -> WorkloadCost; -} - -pub struct WorkloadCost { - pub total_latency: Duration, - pub total_dollars: Dollars, - /// Per-plan contribution, for EXPLAIN / observability. - pub per_plan: Vec, - /// Sub-expressions built once, consumed by ≥2 plans. Drives - /// decisions like "build a KLL for p99 that q1 and q2 both read" - /// vs "run exact select-n per query". - pub reused: Vec, -} -``` - -The reuse model is not sketch-specific: any primitive that is costly to build once and cheap to query (sketches, materialized aggregates, cached scan results, future wavelet summaries) plugs into the same `ReusedComponent` accounting. - -### `core::emit` - -```rust -pub trait PlanEmitter { - type PlanInput; - type Output; // YAML bytes, OpAMP RemoteConfig, rewritten LogicalPlan - fn emit(&self, plan: &Self::PlanInput) -> Result; -} -``` - -Concrete emitters live in deployment model crates: - -- `deployment-model-asapquery::yaml::StreamingConfigEmitter` → `streaming_config.yaml` bytes -- `deployment-model-asapquery::yaml::InferenceConfigEmitter` → `inference_config.yaml` bytes -- `deployment-model-asaplifecycle::emit::OpAmpRemoteConfigEmitter` → `opamp::RemoteConfig` protobuf -- `deployment-model-asapfusion::emit::DataFusionPlanEmitter` → rewritten `datafusion::LogicalPlan` - -### `core::registry` - -The extension point. The runtime binary does: - -```rust -let mut reg = DeploymentModelRegistry::new(); -reg.register::(); -reg.register::(); -reg.register::(); -// add more... -``` - -Registration is by-type; the runtime looks up by `DeploymentModelId` (either from the `QuerySpec.deployment_model` field or from an HTTP route). A 4th deployment model is added by: - -1. New crate `deployment-model-/` -2. Implement `DeploymentModel` trait (wraps a `Planner` + its `PlanEmitter`s) -3. Register in `bin/asap-controller/main.rs` - -No core change. - -### End-to-end example — one query through all five layers - -A worked trace of a single PromQL query as it flows L1 → L5. The query is intentionally simple (one metric, one window, one aggregate) so the IR shapes stay readable; production workloads have larger DAGs but the per-layer transformation is the same. - -**Input.** A `QuerySpec` arrives at `POST /plan` carrying: - -```text -quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m]) -``` - -with `accuracy: AccuracyTarget::Epsilon(0.01)` and `language: PromQL`. - -#### L1 — query language (parse to `PromqlAst`) - -`core::query_language::promql::parse` wraps `promql-parser`. Output (sketch — actual fields come from the upstream crate): - -```rust -PromqlAst::Call { - func: BuiltinFn::QuantileOverTime, - args: vec![ - Expr::Number(0.99), - Expr::MatrixSelector { - name: "http_request_duration_seconds", - matchers: vec![LabelMatcher::Eq("service", "api")], - range: Duration::from_secs(300), - }, - ], -} -``` - -Pure language-level parse — no schema lookup, no sketch awareness. Same call returns the same AST regardless of deployment model. - -#### L2 — language logical plan (`PromqlLogicalPlan` tree) - -`core::lower::promql::lower_to_logical` walks the AST against a `MetricSchema` resolved from Prometheus's `/api/v1/labels`. The result preserves PromQL semantics — `quantile_over_time` is a range-vector aggregate, distinct from a generic `Aggregate { Quantile }` over a `Window`: - -```rust -PromqlLogicalPlan::RangeAggregate { - func: PromqlRangeAggFunc::QuantileOverTime { q: 0.99 }, - range: Duration::from_secs(300), - matrix: Box::new(PromqlLogicalPlan::MatrixSelector { - metric: "http_request_duration_seconds", - matchers: vec![("service", LabelOp::Eq, "api")], - }), -} -``` - -Per language: the SQL form `SELECT approx_percentile(latency, 0.99) FROM events WHERE service='api' AND ts > now() - INTERVAL '5 min'` lands in `SqlLogicalPlan` with a different shape. L2 is per-language; the shapes converge at L3. - -#### L3 — intent algebra (`QueryExpr` + `AggIntent`) - -`core::lower::promql::lower_to_intent` strips PromQL-specific shapes. `quantile_over_time` does **not** survive — the canonical L3 form for a windowed quantile is `Window` over `Aggregate{Quantile}` (see §6 design rule 1, "no `WindowedAgg`"; see §6 "Why no `QuantileOverTime` intent?"). The accuracy target threads through from the `QuerySpec`: - -```rust -QueryExpr::Aggregate { - by: vec![], - aggs: vec![AggIntent::Quantile { - q: 0.99, - accuracy: AccuracyTarget::Epsilon(0.01), - }], - having: None, - child: Box::new(QueryExpr::Window { - kind: WindowKind::Sliding, - size: Duration::from_secs(300), - slide: None, - child: Box::new(QueryExpr::Scan { - source: Source::TimeSeries { - metric: MetricRef::from("http_request_duration_seconds"), - time: TimeRange::default(), // supplied by QuerySpec at evaluation - labels: LabelFilter::eq("service", "api"), - }, - predicates: vec![], - }), - }), -} -``` - -Per-edge schemas (derived per the §6 input/output spec): - -| Edge | Schema | -|---|---| -| `Scan` → `Window` | `{value: Float64, ts: Int64@time_index, service: Utf8}` | -| `Window` → `Aggregate` | the above + `{window_id: Int64, window_start: Int64, window_end: Int64}` | -| `Aggregate` → root | `{p99_value: Float64}` (one column per `AggIntent::Quantile`, typed via `output_type`) | - -This IR is identical across deployment models — same DAG, same intent, same accuracy target. Below this point each deployment model picks its own L4 rules. - -#### L4 — sketch algebra (`SketchExpr`) - -The shared rule `core::optimizer::rules::BindKllOnQuantile` matches `Aggregate { aggs: [Quantile{q, accuracy}] }`, consults the sketch catalog (KLL has `supported_intents: [Quantile]`, is mergeable, satisfies `ε=0.01` at `k=200`), and rewrites the matched sub-DAG into a `SketchAgg` wrapped in a `SketchEstimate` (the readout). Everything not rewritten passes through in `SketchExpr::Logical(...)`: - -```rust -SketchExpr::SketchEstimate { - sketch_input: Box::new(SketchExpr::SketchAgg { - child: Box::new(SketchExpr::Logical(/* the L3 Window+Scan subtree */)), - sketch: SketchKind::Kll, - params: SketchParams::Kll(KllParams { k: 200 }), - col: ColumnRef::value(), - by: vec![], - }), - query: SketchQuery::Quantile { q: 0.99 }, -} -``` - -Catalog-derived edge schemas: - -| Edge | Schema | -|---|---| -| inner `Logical(Window)` → `SketchAgg` | as L3 above (logical pass-through) | -| `SketchAgg` → `SketchEstimate` | `{kll: Sketch(Kll, KllParams{k:200})}` | -| `SketchEstimate` → root | `{p99_value: Float64}` (the `Sketch(...)` field type does not propagate past `SketchEstimate`) | - -The `Sketch(Kll, KllParams{k:200})` field type is the L4 type-system invariant — a downstream `SketchMerge` over a mismatched `Sketch(Cms, …)` input would fail at plan time, before L5 ever sees it. `BindKllOnQuantile` lives once in core and fires for all three deployment models; deployment-model-specific rules (lifecycle's `StageAwarePushDown`, fusion's `HashModeRule`) chain before or after. - -#### L5 — physical plan (stage allocation + emission, per deployment model) - -L4 chose the sketch family + params. L5 colors the DAG by `StageId` and emits per-executor configs. Same `SketchExpr` input; topology and emitter differ per deployment model. - -##### deployment-model-asaplifecycle (3-stage, OpAMP + backend POST) - -`StageAllocator` colors against `topology::ThreeStage`: - -| Node | StageId | Why | -|---|---|---| -| `Scan` + `Window` + `SketchAgg` | `"edge"` | scrape happens on the agent host; KLL build is mergeable so it's safe to run early | -| `SketchMerge` (inserted on cut edge) | `"gateway"` | catalog says `KLL.mergeable = true`; reduces N edge streams to 1 | -| `SketchEstimate` | `"backend"` | readout where users query | - -`PhysicalPlanner` then fans out via `DeploymentConstraints::executors()`. Given a deployment with 3 edge agents + 1 gateway + 1 backend: - -```rust -vec![ - (ExecutorId("edge-001"), OpAmpRemoteConfig { /* OTel YAML: scrape http_request_duration_seconds{service=api}, - sliding 5m window, KLL k=200, forward to gateway-001 */ }), - (ExecutorId("edge-002"), OpAmpRemoteConfig { /* identical OTel YAML; differs only by agent_id */ }), - (ExecutorId("edge-003"), OpAmpRemoteConfig { /* … */ }), - (ExecutorId("gateway-001"), OpAmpRemoteConfig { /* receive 3 KLL streams, SketchMerge, forward to backend */ }), - (ExecutorId("backend-001"), AsapqueryBackendConfig { /* read merged KLL, SketchEstimate q=0.99 */ }), -] -``` - -The 3 edge configs are byte-identical except for `agent_id` — that's the §3 "one stage may have N executors all materialised from the same per-stage sub-DAG" property. The `AsapqueryBackendConfig` is produced by calling `deployment-model-asapquery::yaml::StreamingConfigEmitter` (see §8 asymmetric dep). - -##### deployment-model-asapquery (1-stage, YAML) - -`StageAllocator` against `topology::SingleStage` returns everything on `"backend"`. `PhysicalPlanner` produces one config per backend executor (typically one): - -```yaml -# streaming_config.yaml — emitted by deployment-model-asapquery::yaml::StreamingConfigEmitter -operators: - - name: kll_p99_request_duration - kind: DatasketchesKLL - params: { k: 200 } - input: - metric: http_request_duration_seconds - label_filter: { service: api } - window: { kind: sliding, size: 5m } -estimates: - - sketch: kll_p99_request_duration - query: { kind: quantile, q: 0.99 } -``` - -This is the byte-for-byte format ASAPQuery-backend already consumes — `inference_config.yaml` is emitted alongside by `InferenceConfigEmitter` for the readout side. Both are wire invariants (see §10). - -##### deployment-model-asapfusion (0-stage, in-process `LogicalPlan`) - -`StageAllocator` against `topology::ZeroStage` returns the whole DAG in-process. `PhysicalPlanner` rewrites the caller's `datafusion::LogicalPlan`, replacing the original `Aggregate(quantile(0.99, ...))` node with an `Extension` wrapping the KLL sketch op: - -```rust -LogicalPlan::Extension(Extension { - node: Arc::new(SketchAggExt { - kind: SketchKind::Kll, - params: KllParams { k: 200 }, - input: /* original Window+Filter+Scan subtree, unchanged */, - estimate: SketchQuery::Quantile { q: 0.99 }, - }), -}) -``` - -Returned to the caller's `SessionContext` for execution by `asap-fusion`'s in-process operator. No wire format, no external executor; the data plane is the caller's process. Note also that `asap-fusion`'s entry point is `Source::Table` (not `Source::TimeSeries`) — this PromQL example is only illustrative for the fusion deployment model, which in practice consumes a pre-built DataFusion `LogicalPlan` and skips L1 (see §8). - -#### What this example demonstrates - -- **L1 → L3 are deployment-model-independent.** All three deployment models see the same `QueryExpr` for this query. -- **L4 binding is a shared rule.** `BindKllOnQuantile` lives once in `core::optimizer::rules` and fires for all three. -- **L5 is where deployment models diverge.** Same `SketchExpr` → three topologies → three emitter outputs. The topology is a parameter, not an axis of code (§3). -- **The sketch path is selected, not mandated.** If the `QuerySpec` had `accuracy: AccuracyTarget::Exact`, no `Bind*` rule would fire; L4 would pass `SketchExpr::Logical(QueryExpr::Aggregate{…})` through, and L5 would target an exact `HashAgg` (§1 goal 6). - -## 7. Runtime crate details - -Lifted from DC controller with zero semantic change: - -- `http/` — axum server on `:8080`. Routes: `POST /plan`, `POST /replan`, `GET /plans/:id`, `GET /status`, `GET /metrics` -- `opamp/` — WebSocket OpAMP server on `:4320`. Same protocol DC speaks today. -- `monitor/` — `Scraper` polls Prometheus, emits `Violation` -- `replan/` — subscribes to violations + expiry ticks, re-invokes the deployment model registry -- `store/` — in-memory `PlanStore` + `WorkloadStore`. Pluggable backend later. -- `backend_client/` — pushes `StreamingConfig` YAML to ASAPQuery-backend's `/api/v1/streaming-config` endpoint. Factored out so new deployment models can push to other backends. - -Runtime depends on `core` but NOT on any deployment model crate directly. It talks to deployment models via the registry. - -## 8. Deployment model crate details (each owns its L4 + L5) - -Each deployment model crate is a library with: - -- A `DeploymentModel` impl that registers its planner, its emitters, and its HTTP routes (if any). -- Its own config types — no shared config crate. -- Its own cost model — maybe using `core::cost` primitives, maybe not. -- Its own integration tests. - -### `deployment-model-asaplifecycle` (thin) - -- **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **L4 rules**: picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules that require stage awareness (`StageAwarePushDown`, `TransmissionCostRewrite`). Adding a stage-specific rule = a new file in `deployment-model-asaplifecycle/src/rules.rs`, impl `OptimizerRule`. Unchanged rules come from core. -- **L5 topology**: `core::physical::topology::ThreeStage` (edge → gateway → backend). No deployment-model-specific allocator logic — `StageAllocator` handles the tree walk; lifecycle only declares what the topology looks like. -- **Cost models**: `delta / online-EMA / Pareto / TCO` — these live in `deployment-model-asaplifecycle/src/cost.rs` because they're specific to the DC deployment's network/compute assumptions. Implement `core::optimizer::cost::CostModel`. -- **L5 emitters**: `OpAmpRemoteConfigEmitter` (per-role OTel YAML over OpAMP WebSocket) and `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery`'s `StreamingConfigEmitter` for the YAML bytes, then POSTs to backend). -- **HTTP route**: `POST /plan` for full-lifecycle planning, `POST /replan` for SLA-triggered. -- **Size estimate**: ~1500 LOC (was ~5000 pre-refactor). Cost models are the bulk; rule selection + topology + emitter are each a few hundred lines. - -### `deployment-model-asapquery` (thin) - -- **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. -- **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. -- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::intent_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). -- **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this deployment model covers most sketch types) + a deployment-model-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. -- **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. -- **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `deployment-model-asaplifecycle` calls them when it needs to POST to ASAPQuery-backend. -- **Extra L1 inputs**: `query_log/` for Prometheus-query-log replay (unique to this deployment model); `schema/` for `PromQLSchema` discovery from a live Prometheus URL. -- **HTTP route**: `POST /plan/query` (JSON `QuerySpec` in, YAML stream out). Also backs the `bin/asap-query` one-shot CLI. -- **Size estimate**: ~2500 LOC (was ~6000 pre-refactor — saved by picking from shared rule library; still pays the L2 tree + L3/L4 split refactor cost, which is one-time). - -### `deployment-model-asapfusion` (thin) - -- **Data model**: tabular. L2→L3 lowering produces `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when fusion extends to multi-table). Crucially **not time-indexed** — fusion works on arbitrary DataFusion relations; time is just another column if present. -- **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode deployment model. -- **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so deployment models that want to take a DataFusion plan as input have a canonical name for it. -- **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::intent_algebra::AggIntent` subset. -- **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + deployment-model-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. -- **L5 topology**: `core::physical::topology::ZeroStage` (in-process). -- **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this deployment model is library-mode. -- **Executor**: `ASAPExecutor` wraps a DataFusion `SessionContext`. Users construct `deployment-model-asapfusion` in-process. -- **HTTP route**: none by default. -- **Conformance cost**: near zero. Fusion's `SketchConfigRule` logic is replaced with picks from `core::optimizer::rules::*`; its `HashModeRule` stays deployment-model-specific. -- **Size estimate**: ~1800 LOC (was ~3000 pre-refactor; the `SketchConfigRule` code folds into core's shared rule library). - -The sketch microbenchmarks (KLL/CMS) move with the crate and keep running. The TODO items from `asap-fusion/TODO.md` (batch/multi-query execution, time semantics, distributed model) remain open but are now filed against `deployment-model-asapfusion/TODO.md` in-repo. - -### Data plane communication - -Control plane (ASAPController) and data plane (OTel agents / ASAPQuery-backend / DataFusion runtimes) always talk **over wire**, never via in-process calls. This is unchanged from today: - -| Deployment model | Data plane lives in | Wire protocol | -|---|---|---| -| lifecycle | DataCollector (OTel collectors, agent + backend roles) | OpAMP WebSocket (config push) + HTTP POST (`StreamingConfig` → ASAPQuery-backend) + Prometheus scrape (metrics in) | -| query | ASAPQuery-backend (query engine, SimpleMapStore) | HTTP POST `/api/v1/streaming-config` + `/api/v1/plan` (capability-miss callback in) + YAML file on disk (init-container mode) | -| fusion | Caller's DataFusion `SessionContext` | in-process library call (no wire) | - -**Data plane code stays in its original repo.** ASAPController only owns the control plane. The merger doesn't move OTel collectors out of DataCollector, doesn't move the query engine out of ASAPQuery-backend, and doesn't move DataFusion out of asap-fusion's users. Each data plane keeps its own release cadence. - -### Deployment model placement: in-repo or out-of-repo - -Because core owns the L4/L5 infrastructure (not just L1-3), a deployment model crate is small and largely self-contained. That means deployment models can live either: - -- **Inside ASAPController workspace** — `crates/deployment-model-/`, lockstep release with core, cross-deployment-model changes are one PR. -- **In their own downstream repo** — declares `asap-ir` as a git-tagged dep, releases independently, owns its own CI. - -Both produce functionally identical artifacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. - -Default recommendation: -- **deployment-model-asaplifecycle** and **deployment-model-asapquery** in ASAPController (they share `deployment-model-asapquery`'s YAML emitter via a workspace `path = "../deployment-model-asapquery"` dep — trivial in-workspace). -- **deployment-model-asapfusion** out-of-tree in `asap-fusion` repo (research project with independent benchmark cadence; depends on `asap-ir` + `asap-plan` as published git tags). - -Future deployment models choose whichever placement fits the team that owns them. - -### Asymmetric dependency - -`deployment-model-asaplifecycle` depends on `deployment-model-asapquery` because the `StreamingConfig` YAML emitter is authoritative there. When both live in ASAPController workspace this is a trivial `path =` dep. If one is ever moved out-of-tree, we lift the emitter into core to avoid a cross-repo Cargo dep. - -## 9. Extension point — a 4th deployment model - -With the L4/L5 framework in core, adding a new deployment model is mostly picking + a bit of glue. A hypothetical "edge caching" deployment model that decides which queries to cache at the edge vs. backend would land as: - -``` -crates/deployment-model-edge-cache/ # or your own repo -├── Cargo.toml # depends on asap-ir -├── src/ -│ ├── lib.rs # impl DeploymentModel for EdgeCacheDeploymentModel -│ ├── rules.rs # pick core rules + add EdgeCacheBindRule -│ ├── topology.rs # TwoStage { edge, backend } TopologyDescriptor -│ ├── cost.rs # impl CostModel — hit ratio × bandwidth -│ └── emit/ -│ └── edge_agent_config.rs # impl PlanEmitter — emits edge-agent YAML -└── tests/ -``` - -Total touch outside the new crate: -- 1 line in `bin/asap-controller/main.rs` to register (if in-workspace) OR a published binary in your own repo -- 1 line in workspace `Cargo.toml` (if in-workspace) -- optional: add `DeploymentModelId::EdgeCache` to `core::registry` - -Typical size: ~500-2000 LOC depending on how deployment-model-specific the rules / cost model / emitter are. If the deployment model accepts core's defaults everywhere, ~300 LOC is realistic. - -### Standalone binary for one deployment model - -If you want a slim binary that runs only one deployment model: - -``` -bin/asap-edge-cache/ -├── Cargo.toml # depends only on runtime + deployment-model-edge-cache, not other deployment models -└── src/main.rs # registers only EdgeCacheDeploymentModel; no DataFusion, no query YAML -``` - -Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql-parser` unless this deployment model needs it. Useful when a deployment is dedicated to one deployment model (e.g. an edge-caching-only service). - -## 10. Wire protocols — what changes, what doesn't - -**Unchanged** (hard contract with other systems): -- `POST /api/v1/streaming-config` on ASAPQuery-backend — consumed YAML format -- `POST /api/v1/plan` on DC controller (from backend capability-miss) — request JSON -- OpAMP `ServerToAgent.remote_config` payload — OTel collector YAML -- Prometheus scrape format - -**Internal** (controller's own surface, still HTTP+JSON for now): -- `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artifacts (URIs). `/api/v1/plan` proxies to this. -- `GET /plans/:id` — plan inspection -- `GET /status` — runtime + deployment model health - -**New** (internal-only, proto): -- `proto/asap_control.proto` defines `Plan`, `PlanNode`, `Expr` for persistence + store-internal serialization. Not on the wire between services. Optional; initial migration skips this and persists via `serde_json`. - -## 11. Dependencies and Cargo surface - -Workspace `Cargo.toml`: - -```toml -[workspace] -resolver = "2" -members = [ - # landed today (§5.1) - "crates/ir", - "crates/l2", - "crates/sketch", - "crates/plan", - "crates/frontend-promql", - "crates/frontend-sql", - "crates/lower", - "crates/e2e", - # planned - "crates/runtime", - "crates/deployment-model-asaplifecycle", - "crates/deployment-model-asapquery", - "crates/deployment-model-asapfusion", - "crates/control-proto", - "crates/testing", - "bin/asap-controller", - "bin/asap-query", -] - -[workspace.dependencies] -tokio = "1" -axum = "0.7" -reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } -prost = "0.13" -tonic = "0.12" -serde = { version = "1", features = ["derive"] } -serde_yaml = "0.9" -tracing = "0.1" -thiserror = "1" -# deployment-model-specific deps kept in deployment model crates -``` - -Core depends only on `serde`, `tracing`, `thiserror`, and small utility crates. No HTTP, no DataFusion, no OpAMP. - -Runtime depends on core + `axum` + `reqwest` + `tokio-tungstenite` + `prost` (OpAMP proto). - -Deployment models depend on core. deployment-model-asapquery also pulls in `promql-parser`, `sqlparser`. deployment-model-asapfusion pulls in `datafusion` + `arrow`. deployment-model-asaplifecycle pulls in `sqlparser` + `promql-parser` + the cost-model math crates. - -This matters: a user who only wants `deployment-model-asapfusion` (e.g., an offline benchmark) gets DataFusion but NOT axum/OpAMP. - -## 12. Open questions - -1. **Do we need a cross-deployment-model cost model?** Today DC's lifecycle planner and asap-planner-rs's query planner have overlapping but not identical cost models. Answer for now: keep them separate in deployment model crates; let them re-converge organically. If a third deployment model needs the same model, lift at that point. - -2. **Where does the backend's `ControllerClient.create_plan` call land?** Initially: HTTP `POST /api/v1/plan` on the controller, handled by `deployment-model-asaplifecycle` (same as today). Long-term: could route by `QuerySpec.deployment_model` to a different planner, but that's a follow-up. - -3. **Which deployment model owns `StreamingConfig` YAML emission?** Both `deployment-model-asaplifecycle` and `deployment-model-asapquery` emit it today (DC's `config/generate_streaming_config_yaml` and `output/generator.rs` in `asap-planner-rs`). Plan: **one emitter in `deployment-model-asapquery`**, called from both. This is why `deployment-model-asaplifecycle` depends on `deployment-model-asapquery` in the dependency graph (asymmetric — query does not depend on lifecycle). - -4. **Do we vendor DataFusion's IR into core?** No. `deployment-model-asapfusion` owns its DataFusion-flavored plan; `core::plan::Plan` stays an enum with a variant that wraps a `datafusion::LogicalPlan` behind a feature flag. Core itself never reaches into DataFusion types. - -5. **OpAMP proto: vendored, or from crates.io?** Today DC vendors. Recommend: keep vendored in `proto/opamp.proto`, generate via `prost-build` in `crates/control-proto`. Same thing DC does today, just moved. - -6. **What happens to ASAPQuery-backend's `asap-planner-rs` directory post-migration?** Deleted. ASAPQuery-backend's docker-compose drops the `asap-planner-rs` init container; the controller's `deployment-model-asapquery` now runs in-process (service mode) or as the `asap-query` CLI (one-shot mode). The ASAPQuery-backend repo shrinks by two directories. - -7. **Versioning?** Start at `0.1.0` on the workspace. Deployment models can rev independently later via per-crate versions, but initially lockstep. - -8. **What if a future deployment model can't fit a tree L2?** L2 is currently a per-language tree, mandatory for every deployment model. asap-planner-rs's Phase 4 conformance cost (reverse-engineering PromQL pattern templates into a `PromqlLogicalPlan` tree) was taken deliberately to keep the architecture uniform. If a future deployment model's source language doesn't map naturally onto a tree (e.g. a constraint-based or dataflow-graph query language), that's the moment to re-examine the L2 contract — the `core::logical_plan` module is a single Rust trait + per-language types, not a deep assumption baked across the codebase. Until then, L2 = tree. - -9. **Deployment model placement — in ASAPController workspace, or in own repo?** Both supported, same architecture either way (see §8 "Deployment model placement"). Default: `deployment-model-asaplifecycle` and `deployment-model-asapquery` in ASAPController (easy cross-deployment-model changes, shared YAML emitter); `deployment-model-asapfusion` in the `asap-fusion` repo (research cadence). A new deployment model picks based on team ownership and release cadence preferences. - -10. **How far do shared L4/L5 rules live in core before they become deployment-model-specific?** The rule of thumb: if ≥2 current deployment models would use it, it lives in `core::optimizer::rules::*`. If only 1 deployment model uses it *and* it depends on deployment-model-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the deployment model crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `deployment-model-asaplifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the deployment model; lift to core once a second deployment model wants it. Don't pre-emptively generalize. - -11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP deployment models)?** Already handled — see §3 "Data-model support" and §6 `core::intent_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP deployment model picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. - -### Resolved during the L3 IR cleanup (see §6 `core::intent_algebra`) - -The following questions came up during review of the previous `QueryExpr` draft and are now settled. Listed here so the trail is visible: - -- **Output of `SketchAgg` vs `WindowedAgg`?** Neither node exists at L3 anymore. `SketchAgg` is an L4 sketch-bound node (`SketchExpr::SketchAgg`) emitted by binding rules; `WindowedAgg` was redundant with `Window` over `Aggregate` and removed. Output of `SketchAgg` is a sketch-typed column carrying the partial state; `SketchEstimate` reads it out into a scalar / vector. -- **Is `TopK` different from `Sort + Limit`?** They overlap on heavy-hitter queries but are different concepts. `TopK` is now an `AggIntent` at L3 (heavy-hitter intent — SpaceSaving / CMS-with-heap / Misra-Gries serve it as a single primitive), while generic `Sort + Limit` survives as `QueryExpr` operators for non-heavy-hitter cases (`ORDER BY name LIMIT 10`). L1→L2→L3 lowering picks the intent form when it recognises a heavy-hitter pattern (`ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`) and the operator form otherwise. -- **What is `JoinSketch`?** A sketch-aware join (KMV / theta / join-sample). Moved to L4 (`SketchExpr::SketchJoin`) so the choice "exact join vs sketch join" is an L4 cost decision against `Join`, not a competing L3 surface. -- **`HistogramQuantile` and `PromQLSubquery` feel out of place** — as `QueryExpr` *nodes*, they were. `PromQLSubquery` expands during PromQL L1→L2 lowering, not a DAG node. `histogram_quantile` was initially lowered to bucket reads + a generic `Quantile` intent, then **revised (#43/#79)**: classic-bucket interpolation is semantically distinct from a sketch-able quantile (bucket counts can't be re-sketched), so it returned as `AggIntent::HistogramQuantile` — an intent, not a node — with the native/raw-samples form still lowering to the generic sketch-able `Quantile`. -- **Sketch subtract / delete / estimate operators?** Added: `SketchExpr::SketchSubtract`, `SketchDelete`, `SketchEstimate`. Catalog flags (`subtractable`, `deletable`) gate which sketch families admit them. -- **What's `Dedup` exactly?** SQL `DISTINCT`. Renamed to `Distinct { cols }` and generalised from one column to N. -- **Why differentiate `Quantile` and `QuantileOverTime`?** No reason — the surrounding `Window` already encodes the temporal axis. `QuantileOverTime` removed from `AggIntent`. -- **`WindowedAgg` vs `Window` + `Agg`?** Equivalent; `WindowedAgg` removed. `WindowKind::{Tumbling, Sliding, Session}` lives on the `Window` node so the streaming-window kind is explicit; SQL analytic `OVER (...)` stays on the separate `WindowFunc` node. -- **Need input/output specs more fine-grained than `QueryExpr`?** Done: every L3 edge carries a typed `Schema` (fields + time index + unique-key sets), and §6 lists per-node input/output schemas. - -## 13. Future work - -### Extending the primitive set beyond sketches - -The L4 rule engine + `OptimizerRule` trait are **primitive-agnostic** — nothing in the framework is sketch-specific above the rule library. Future primitive classes land as additional rules against the same trait + additional entries in `CostModel`, not as new translation phases or parallel IRs. Candidates we explicitly anticipate: - -1. **Wavelets** — a sibling approximation family (Haar / DWT + coefficient thresholding). Strong on smooth, low-entropy signals where thresholded coefficient sets dramatically out-compress randomized sketches. Slots in as a new physical alternative for the same `AggIntent` variants sketches serve today (range-sum, heavy-hitter, quantile). No change to L3. - -2. **Reuse / precomputation as first-class primitives.** Materialized views, cached scan results, shared sub-expressions. The cost-model hook already exists (`CostModel::workload_cost` + `ReusedComponent`); the missing piece is rules that *introduce* a reuse node — e.g. "build this aggregate once for `q1`, rewrite `q2` to read from it". Orthogonal to approximation: you can reuse an exact aggregate or an approximate one. - -3. **Other approximation algorithms** — sampling, coresets, online PCA / linear-regression normal equations, naive-Bayes-with-conjugate-priors. Anything with a monoid-shaped build + bounded error fits the same contract a sketch does. - -The guiding principle: **same framework, different rule.** A new primitive class is never a translation-layer change — only a new rule, a new cost-model entry, and (if it introduces a new runtime op) a new physical operator in L5. - -## 14. Glossary - -Terms that are project-specific or that get conflated. Where the term has a Rust counterpart, the type is shown in backticks. - -### Architecture roles - -- **Deployment model** — A concrete bundle of (L4 rule choices) + (L5 topology) + (emitter), packaged as a `deployment-model-*` crate. Three exist today: `asaplifecycle`, `asapquery`, `asapfusion`. -- **Stage** (`StageId`) — A *categorical tier* in the data lifecycle: edge / gateway / backend / in-process. Roles, not instances. Declared by the deployment model's `TopologyDescriptor`. -- **Executor** (`Executor`) — A *concrete runtime instance* occupying a stage: a specific OTel agent, the ASAPQuery-backend process, a DataFusion `SessionContext`. One stage may have N executors (e.g. 50 edge agents). Carries `id`, `stage: StageId`, `capabilities`, `address`. -- **Topology** (`TopologyDescriptor`) — Declares which stages exist + how they connect (3-stage / 1-stage / 0-stage). Categorical, not instance-level. -- **DeploymentConstraints** — Trait object owned by each deployment model. Carries memory budgets, network topology, available sketch backends, and the registered `Executor` list. Threaded through L4 rules and the stage allocator. - -### Layer drivers - -- **RuleEngine** — Generic L4 driver in core (fixed-point iteration + cycle detection + priority ordering). One implementation; same code for every deployment model. -- **OptimizerRule** — Trait for an L4 rewrite. Categories: `PushDown | Fusion | Elim | Bind | StageRouting`. Deployment models pick which rules from `core::optimizer::rules` + add their own. -- **CostModel** — Trait scoring a plan's accuracy / latency / dollars. Generic implementations in core; deployment-model-specific cost models (DC's delta / online / pareto / TCO) live in their crates. -- **StageAllocator** — Generic L5 algorithm in core. Colors the L4-bound DAG by `StageId`. **Stage-granularity only — no executor knowledge.** One implementation; same code for every deployment model. -- **PhysicalPlanner** — Per-deployment-model L5 *driver* (one impl per deployment model). Calls `StageAllocator`, then fans the per-stage sub-DAGs out to executors via `DeploymentConstraints::executors()`, then produces the deployment-specific output type. Compare to allocator: planner = whole L5 driver, allocator = the stage-coloring step it uses. -- **PlanEmitter** — Per-deployment-model trait that serialises the planner's output to its wire format (OpAMP `RemoteConfig` / `streaming_config.yaml` / rewritten DataFusion `LogicalPlan`). - -### IR by layer - -- **L1 — query language** — Raw query string + parser (PromQL / SQL / DataFusion / ElasticDSL). -- **L2 — logical plan** — Per-language relational algebra tree (`PromqlLogicalPlan`, `SqlLogicalPlan`, etc.). -- **L3 — intent algebra** (`QueryExpr`, `asap-ir::intent_algebra`) — Symbolic, intent-only IR. No sketch type, no params. `AggIntent` carries accuracy targets only. The "algebra" is the operator surface (`Scan`, `Filter`, `Aggregate`, `Window`, …); the algebra is *intent-only* because no sketches have been bound yet. -- **L4 — sketch algebra** (`SummaryExpr` / `L4Node` in `asap-sketch`; this doc's prose elsewhere still says `SketchExpr` in places — the IR itself moved to [`l4node-execution-model.md`](./l4node-execution-model.md), which uses the shipped names throughout) — Same DAG shape as L3, but sketches now committed (kind + params). Produced by the L3→L4 binding pass (`asap_plan::bind`, #98); consumed by L5 emitters. Note the naming inversion vs. earlier drafts: "sketch algebra" is L4 (where sketches actually live), not L3. -- **L5 — physical plan** — Stage-assigned, executor-targeted, ready to serialize. Produced by `PhysicalPlanner`, written out by `PlanEmitter`. - -### Metadata sources (see §6 "DAG schema, DB schema, sketch catalog") - -- **DAG schema** (`Schema`) — Columns + types + `unique_keys` carried on every L3/L4/L5 edge. Type-checked locally at each node. -- **DB schema** / **source schema** (`SchemaCatalog`) — Data-plane metadata (Prometheus TSDB / SQL `information_schema` / DataFusion catalog). Read by `core::lower::*` for L1→L2 symbol resolution. -- **Sketch catalog** (`SketchCatalog`) — Runtime registry of available primitives: what sketches exist, mergeability, deletability, parameter ranges, accuracy model. Consulted by L4 binding rules. **Not a schema** — a catalog of available primitives, not a description of a stream. - -### Workload + identity - -- **QueryWorkload** — Top-level controller input: one or more `QuerySpec`s plus workload features (batch vs streaming, reuse opportunity, data source). Arrives via HTTP POST, OpAMP capability-miss callback, YAML file, or query-log replay. -- **QuerySpec** — A single query: source-language string + accuracy / latency / cost target. - -## 15. Success criteria - -The migration is done when: - -1. `asap-controller` binary runs and passes DC controller's existing integration tests (OpAMP push, backend config POST, SLA replan). -2. `asap-query` binary takes the same YAML input asap-planner-rs does today and produces byte-identical `streaming_config.yaml` + `inference_config.yaml` (fuzz-test against a corpus of fixtures). -3. ASAPQuery-backend's docker-compose no longer starts `asap-planner-rs`; the controller handles both shapes. -4. `asap-fusion`'s microbenchmarks still run under `deployment-model-asapfusion` with identical numbers. -5. The `DataCollector/controller/`, `ASAPQuery/asap-planner-rs/`, `ASAPQuery-backend/asap-planner-rs/`, and `asap-fusion/` directories are deletable (or already deleted) without breaking any currently-running deployment. -6. A new hypothetical deployment model can be added with zero changes outside its crate + one line in `bin/asap-controller/main.rs`. +ASAPController **plans** queries through five layers — turning a query +string into a plan, not running it: +- **L1** — parse each query language into its own native shape +- **L2** — unify every language's parsed representation into one shared + logical plan, and resolve column names to schema positions +- **L3** — unify semantically-equivalent queries into one canonical, + language-independent form expressed as *intent* — what to compute, + without committing to how +- **L4** — decide how each intent is answered: which summary (if any) + realizes it, still without committing to where it runs +- **L5** — decide where and how each piece of the plan executes + physically (placement, parallelism, lifecycle stage) + +**Answering** a query at request time — using whatever L1-L5 already +decided — is a separate concern from planning, covered in its own +"Serving-time execution" section after L5, below. + +## L1 — query language (front end) + +**Job: elaborate a raw query string into that language's own native +representation, independently per language — remaining agnostic to +every other query language, whose reconciliation is deferred to L2.** +- Each supported query language owns its own parser and produces its + own native shape. This layer has no shared vocabulary between + languages, and no summary awareness. + +```mermaid +flowchart LR + P["Query string (language A)"] --> PA["language A's parser"] --> PR["language A's native representation"] + S["Query string (language B)"] --> SA["language B's parser"] --> SR["language B's native representation"] +``` + +- Doc: [`l1-query-language.md`](./l1-query-language.md) + +## L2 — logical plan + +**Job: unify every language's L1 representation into one shared +relational tree (or DAG), and resolve symbolic column references to +schema positions.** +- **This is more than syntax translation.** Each language's own + operators must be *interpreted* into the same shared relational + vocabulary (filter, project, aggregate, window, sort, limit, join, + set operations, …). Languages differ in which shapes they can + already recognize directly at this layer and which ones only reach a + more generic shape here. **No cross-language normalization of + equivalent shapes happens yet** — two queries that mean the same + thing may still look structurally different after L2; that + convergence is L3's job. +- Columns are still named references at this layer, not positions — + resolving a name to a position happens here, but still per language, + since different languages carry different schema guarantees (a fully + known schema vs. one only known by whatever a query happens to + reference). + +```mermaid +flowchart LR + PR["language A representation"] --> PW["interpret into shared\nrelational vocabulary"] --> L2T["shared logical plan\n(one type, every language)"] + SR["language B representation"] --> SW["interpret into shared\nrelational vocabulary"] --> L2T + L2T --> B["resolve names to schema positions"] +``` + +- Doc: [`l2-logical-plan.md`](./l2-logical-plan.md) + +## L3 — intent algebra + +**Job: specify intent — the computation and its accuracy requirement — +while remaining agnostic to implementation strategy, which L4 alone +decides.** +- **The cross-language normalization step lives here.** Semantically + equivalent queries, however each language happened to express them, + converge on one canonical shape at this layer. A shared logical-plan + type alone (L2) isn't enough for this: different languages can reach + the same L2 shape by different paths, and some equivalent + computations only become recognizable as equivalent once normalized. +- The result is a language- and deployment-independent canonical + intent tree: what to compute, expressed declaratively (e.g. "a + quantile to this accuracy," "the top-k by this ranking"), not + committed to any particular implementation strategy. Deployment here + refers to a physical execution context — e.g. parallelism and the + lifecycle stage a computation runs at. +- No summary type or parameters are committed here; that's explicitly + deferred to L4, below. +- Row identity (which positions uniquely identify a row) and + cross-query sharing of common sub-computations are properties of + this canonical form, not inherited from L2. + +```mermaid +flowchart LR + L2T["shared logical plan"] --> C["structural translation"] --> R["intent tree\n(still language-shaped)"] + R --> CN["canonicalize"] --> L3["canonical intent tree\n(same shape for equivalent\nqueries, any source language)"] +``` + +- Doc: [`l3-intent-algebra.md`](./l3-intent-algebra.md) + +## L4 — summary-bound IR + +**Job (planning-time only): decide, for each intent, whether and how it +is answered by a summary rather than by scanning raw data** — +symbolically picking a summary family and its parameters, with no +reference to what's actually stored anywhere. **This is planning-time +only** — the serving-time half, which actually answers a query using +what got decided here, is covered in its own section below (after L5). +- An "implementation" is the concrete realization chosen for one piece + of intent — e.g. a summary family and its parameters, an exact + accumulator, or passing through to compute directly from raw data. +- Agnostic to physical execution (where something runs, how parallel it + is) — that's L5's decision alone. + +```mermaid +flowchart LR + L3["canonical intent tree"] --> IT["choose a summary\n(or none) per intent"] --> L4N["summary-bound plan"] +``` + +- Doc: [`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md) + +## L5 — physical plan + +**Job: commit to physical execution — assign each piece of an +already-summary-bound plan to a physical execution mode (e.g. lifecycle +stage, parallelism) — taking L4's summary choices as fixed.** +- Inputs to L5: which physical stages exist and how they connect + (topology), and the budgets/capabilities a given deployment offers + (deployment constraints). +- This layer is inherently deployment-specific. ASAPController defines + the interface a deployment implements, not a concrete physical + planner. + +```mermaid +flowchart LR + L4N["summary-bound plan"] --> PP + subgraph PP["deployment's own physical planner\n(implements a shared interface)"] + SA["assign stage + parallelism\nper piece of the plan"] + end + PP --> ART["deployment-specific artifact"] +``` + +- Doc: [`l5-physical-plan.md`](./l5-physical-plan.md) + +## Serving-time execution + +Everything above (L1-L5) is the **planning** pipeline: turning a query +string into a plan. This section is different in kind — it's what +actually **answers** a query at request time, using whatever plan L1-L5 +already decided. It's kept as its own interface on purpose: a +downstream deployment consumes it independently of how the plan was +produced. + +**Job: walk an already-decided summary-bound plan against whatever is +actually materialized right now, and produce an answer.** Reality can +diverge from the plan in ways planning time never sees — missing data, +multiple instances needing a merge, instances that disagree on +parameters — which is why serving needs its own error cases, distinct +from anything planning-time binding raises. +- The structural rules — which nestings of a summary-bound plan are + valid, what must agree for a merge to be legal — are shared and + deployment-independent. Storage, summary math, and readout are + entirely deployment-supplied. +- Planning and serving are two separate interfaces a downstream + deployment implements against. + +```mermaid +flowchart LR + L4N["summary-bound plan\n(from L4 planning)"] --> EX["deployment-supplied executor"] --> V["answer"] +``` + +- Doc: [`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md) — the + "Serving-time" section covers this in full. + +## Glossary + +Terms that are project-specific, or that get conflated across layers. +Full detail lives in the layer doc linked from each entry; this is a +quick reference, not a replacement for it. + +### Architecture terms + +- **Front end** — The per-language component that elaborates a raw + query string into that language's own native representation. L1's + job, one per language. See [`l1-query-language.md`](./l1-query-language.md). +- **Bind** — Name resolution: mapping a symbolic column reference to a + schema position. Introduced at L2. See + [`l2-logical-plan.md`](./l2-logical-plan.md). +- **Summary** — Umbrella term for whatever answers a piece of intent + without re-scanning raw samples: an approximate sketch or an exact + accumulator. See [`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md). +- **Cost model** — The extension point that ranks candidate summary + choices for a given intent. See + [`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md). +- **Deployment model** — A concrete bundle of (L4 summary choices) + + (L5 topology) + configuration emission, packaged for one downstream + deployment. See [`l5-physical-plan.md`](./l5-physical-plan.md). +- **Stage** / **Executor** / **Topology** / **Deployment constraints** — + L5 concepts: a categorical placement tier, a concrete runtime + instance, which stages exist and how they connect, and the + per-deployment budgets/capabilities. See + [`l5-physical-plan.md`](./l5-physical-plan.md). + +### Workload + +The normalized input meant to sit in front of every query entry point. + +- **Query workload** — a batch of one-shot and/or repeating queries, + all in the same query language, each with an optional accuracy + and/or latency requirement. +- **Data characteristics** — workload-level facts (series/row count, + sample rate, wire size, key distribution) used to size summaries + without running the query. diff --git a/docs/intent-algebra-reconciliation.md b/docs/intent-algebra-reconciliation.md deleted file mode 100644 index 34f93391..00000000 --- a/docs/intent-algebra-reconciliation.md +++ /dev/null @@ -1,120 +0,0 @@ -# `intent_algebra` reconciliation plan (ASAPController ⇄ ASAPQuery-backend) - -> **Status (prerequisite done):** the *intra-repo* reconciliation — PR #4's -> name-based SQL L3 ⇄ PR #5's positional L3 — is complete on `feat/promql-l1-l3`. -> Both front ends (PromQL + SQL) now lower onto **one positional IR** via the -> shared `convert_root`: `expr_ir` is the SQL∪PromQL scalar superset, `AggIntent` -> carries `col: Option`, the converter is bottom-up schema-aware, and -> the DataFusion front end emits relational L2. The plan below (ASAP ⇄ -> control_plane) is the *next* step, now unblocked. - -> **Reconciliation note (post-reorg; snapshot below is historical).** This doc -> predates the crate split: `crates/core/src/intent_algebra` is now -> `crates/ir/src/intent_algebra` (L3), with `relational`/`lower`/`binder`/ -> `column_resolution` in `crates/l2` and `cse` in `crates/plan` (see -> `design.md` §5.1/§6.0). The line counts and the "CP has 11 intents ASAP -> lacks" gap are a point-in-time snapshot: the PromQL function-family work -> (#43–#51) has since grown ASAP's `AggIntent` to ~40 variants, absorbing the -> CP extras (counter derivatives, presence, `Absent*`, …) and adding families -> CP never had. Read the tables below as the *evidence that motivated the -> plan*, not the current state; `crates/ir/src/intent_algebra/agg_intent.rs` -> is the source of truth. - -ASAPController's `crates/core/src/intent_algebra` is a slimmed, refactored fork -of the canonical L3 IR in `ASAPQuery-backend/control_plane/src/intent_algebra`. -This is the first concrete step of the L4/L5 consolidation: produce **one** -canonical `intent_algebra` so that (a) both repos stop drifting, and (b) -control_plane's L4 (optimizer + sketch_algebra) and L5 (physical + emit) can be -ported onto the shared IR the PromQL/SQL front-ends already lower into. - -**Base = control_plane's L3** (the richer original); ASAPController's -correctness fixes and multi-language front-end are layered on top. - -## Drift summary (evidence) - -Line counts, `intent_algebra/*.rs`, control_plane (CP) vs ASAPController (ASAP): - -| file | CP | ASAP | finding | -|---|---:|---:|---| -| `schema.rs` | 332 | 332 | **byte-identical** (already shared) | -| `query_expr.rs` | 1193 | 445 | ASAP relational nodes ⊆ CP; CP's "extra" variants are the **inlined scalar IR** ASAP split into `expr_ir.rs` | -| `agg_intent.rs` | 673 | 245 | **additive both ways**: CP has 11 ASAP lacks; ASAP has `StdDev`/`Variance` CP lacks | -| `relational.rs` | 920 | 212 | CP superset, **incl. `Project`** (ASAP lacks; SQL needs it) | -| `lower.rs` | 845 | 294 | CP richer; ASAP carries the per-branch-binding fix + accuracy threading | -| `binder.rs` | 300 | 189 | CP richer; same `SchemaCatalog`/`UsageDerivedCatalog` API | -| `cse.rs` | 324 | 228 | ASAP carries the structural-`PartialEq` key fix | -| `column_resolution.rs` | 465 | 177 | CP richer | -| `mod.rs` | 125 | 41 | **same export shape**, CP exports more | -| `expr_ir.rs`, `names.rs` | inlined | separate | file-organization difference | - -Verdict: a **contained merge**, not a rewrite. `schema.rs` is already shared, the -node set is a clean superset, `AggIntent` is a union, and the public module API -matches. Cost concentrates in `lower.rs` and `expr_ir.rs`. - -Behavioral specifics found in CP: -- `convert_root(legacy)` takes **no accuracy** — it hardcodes `Exact` / - `Epsilon(0.05)` in the converter. ASAP threads a per-query `AccuracyTarget`. -- CP's converter threads **one root schema to all branches** — i.e. it has the - same per-branch-binding bug ASAP already fixed. -- CP `AggIntent` carries `accuracy` and the 11 extra intents - (`Changes/Resets/Deriv/Delta/PredictLinear/Absent/Present/Idelta/Irate/HoltWinters/Frequency`), - but **lacks `StdDev`/`Variance`** — it fans those out into a `Merge` of - quantile aggregates in `lower.rs`. -- CP sources `AccuracyTarget` / `BindingName` from a `types_v2` module. - -Two payoffs from basing on CP's L3: -- Several functions ASAPController currently **rejects** (`changes`, `resets`, - `deriv`, `delta`, `predict_linear`, `absent`, `present`) gain intents → become - lowerable. -- `irate` becomes distinguishable from `rate` (CP has a separate `Irate` intent); - our `rate≡irate` equivalence was a consequence of the slimmer vocabulary. - -## Decisions to settle (sign-off needed) - -| # | Fork | Options | Recommendation | -|---|---|---|---| -| **D1** | `StdDev` / `Variance` | CP's `Merge`-of-quantiles fan-out **vs** ASAP's first-class `AggIntent` | **CP fan-out** (less L4 work); add first-class intents only if L4 can bind them | -| **D2** | scalar IR location | CP inlines in `query_expr`/`relational` **vs** ASAP's separate `expr_ir.rs` (`L3Expr`) | **ASAP's `expr_ir.rs`**, extended to CP's scalar superset | -| **D3** | shared scalar types home | CP `types_v2` **vs** ASAP `names.rs` + `types.rs` (`AccuracyTarget`, `BindingName`, `QueryId`) | **one `core::types` module**; both already expose the same names | - -## File-by-file tasks - -Base = control_plane's file unless noted; "port" = bring ASAP's delta onto the CP base. - -| file | base | tasks | effort | risk | -|---|---|---|---|---| -| `schema.rs` | identical | **no-op** — adopt as-is | none | none | -| `query_expr.rs` | CP | adopt CP node set (ASAP ⊆ CP); per **D2** reference `expr_ir::L3Expr` instead of inline scalars; confirm CP covers ASAP methods (`output_schema_in`, …) | low | low | -| `agg_intent.rs` | CP | adopt CP's full vocabulary; resolve **D1**; reconcile `output_column` / `agg_accuracy` | low–med | low | -| `relational.rs` | CP | adopt CP (incl. `Project`); per **D2** point scalar refs at `expr_ir` | low | low | -| `lower.rs` | CP | **main work**: (1) port per-branch binding to `BinaryOp`/`Join`/`SetOp` arms; (2) add `acc: &AccuracyTarget` to `convert`/`convert_root`, replace hardcoded defaults; (3) verify nested-aggregate convert supports the two-level `sum(rate)` shape; (4) reflect **D1** | **med** | med | -| `binder.rs` | CP | adopt CP; verify `SchemaCatalog`/`UsageDerivedCatalog` parity | low | low | -| `cse.rs` | CP | **port** the structural-`PartialEq` key fix if CP keys by `Debug` | low | low | -| `column_resolution.rs` | CP | adopt CP | low | low | -| `expr_ir.rs` | ASAP (keep, **D2**) | **med work**: extend `L3Expr` to CP's scalar superset (`FunctionCall`, `InList`, `Between`, `IsNull`, `ScalarSubquery`, `Cast`) + our `Regex`/`NotRegex` | med | low | -| `names.rs` | merge → `types` | fold `BindingName`/`QueryId` into the shared types module (**D3**); update CP's `types_v2` imports | low | low | -| `mod.rs` | CP + ASAP | union the exports; keep `expr_ir` + shared `types` | low | low | - -## Cross-cutting (part of the merge, outside `intent_algebra`) - -- **`types_v2` → `core::types`** (D3): one home for `AccuracyTarget`, `BindingName`, `QueryId`. -- **Front-ends**: keep ASAPController's **PromQL + SQL** lowering (CP has PromQL only) and the correctness fixes (`sum(rate)`, `histogram_quantile`, matcher canonicalization) — retarget onto the unified `QueryExpr`. Lives in `crates/frontend-promql` / `crates/frontend-sql` (with `crates/lower` as the re-export facade). -- **Tests**: bring the `conformance` / `equivalence` / `corpus` suites onto the unified crate and merge with CP's `lower.rs` / `cse.rs` unit tests. Use the full suite as the acceptance gate. - -## Recommended order - -1. `schema.rs` (free) + `types`/`names.rs` (D3) — shared primitives. -2. `expr_ir.rs` to CP's scalar superset (D2) — unblocks the type layer. -3. `query_expr.rs` + `relational.rs` + `agg_intent.rs` (D1) — the type layer. -4. `binder.rs` + `column_resolution.rs` + `cse.rs` (+ port the CSE key fix). -5. `lower.rs` — per-branch binding + accuracy threading (the one med-risk file). -6. Retarget front-ends + bring tests; run the full ASAPController suite against the unified crate. - -## Effort summary - -- **adopt-CP (low):** schema, query_expr, relational, binder, column_resolution, mod. -- **port-our-fix (low):** cse, names/types. -- **real work (med):** `lower.rs` (per-branch binding + accuracy), `expr_ir.rs` (scalar superset). -- **decision with semantic weight:** D1 only. - -Open prerequisite (tracked separately): host the unified crate as **(A)** a cross-repo shared crate or **(B)** absorb control_plane into the ASAPController monorepo (design.md §5/§8). Recommendation: **B** for the core IR — a cross-repo git dependency on the central IR forces lock-step two-repo changes (cf. the `promql-parser` private-mirror friction). diff --git a/docs/l1-query-language.md b/docs/l1-query-language.md new file mode 100644 index 00000000..50757d9a --- /dev/null +++ b/docs/l1-query-language.md @@ -0,0 +1,69 @@ +# L1 — query language + +Per-language front ends, one per supported query language. Each turns a +raw query string into that language's own native shape — whatever +representation is most natural for that language (e.g. a parsed AST, +or an already-planned relational tree) — with no summary awareness and +no shared vocabulary yet; that only starts once every front end reaches +L2. + +Front ends never depend on each other: each language's parsing path is +fully independent, so adding or removing a supported language never +touches any other front end. + +A single query language may support more than one concrete surface +dialect at this layer — different syntax that still elaborates to the +same native representation. Dialect differences are purely a parsing +concern; they never affect what a later layer can plan. + +## Parse → lower: converging on one shape + +Different front ends can look nothing alike at L1 — one may start from +a bare AST, another from an already-planned tree — but every front end +must converge on the *same* L2 shape, through the *same* L2 entry +point. That convergence, not the L1 parsing itself, is the part that +matters at the design level: + +``` +Query string (language A) Query string (language B) + │ language A's own parser │ language B's own parser + ▼ ▼ +language A's native representation ← L1 language B's native representation ← L1 + │ interpret into shared vocabulary │ interpret into shared vocabulary + ▼ ▼ + shared logical plan ← L2 — same type for every language + columns are still named references + │ + │ one shared L2 → L3 step, for every language: + │ 1. bind — name → schema-position resolution + │ 2. convert — purely structural L2 → L3 translation + │ 3. canonicalize — shared cross-language normalization + ▼ + canonical intent tree ← L3 — columns are positions + + a self-contained schema on every scan +``` + +Binding is not itself a layer — it's the pass that sits on the L2 → L3 +edge. *Why* the canonical form uses positional column identity, how +binding differs between a schema-less source and a catalog-backed one, +and what a unique key is for are all L2/L3 design questions, covered in +[`l2-logical-plan.md`](./l2-logical-plan.md) and +[`l3-intent-algebra.md`](./l3-intent-algebra.md) — this section only +needs to establish that every front end feeds the identical mechanism. + +### The nesting contract + +Nesting — an operator tree inside a source clause or a function +argument, an aggregate over an aggregate, a binary operation over two +subtrees, a range function over a sub-query, and so on — is required to +lower **structurally**: the canonical intent tree is a recursive, +arbitrarily-nestable structure, so "an operation over a sub-query" +needs no special-cased IR shape. Any language whose grammar allows +nesting must be able to express it this way, without a parallel +representation for the nested case. + +Not every syntactically-expressible nesting shape has to be supported. +A front end is expected to cleanly reject a shape it can't yet +represent — rather than silently mis-lowering it — since a +resolvable-later gap is one design choice away from becoming a +correctness bug if it's lowered wrong instead of rejected outright. diff --git a/docs/l2-logical-plan.md b/docs/l2-logical-plan.md new file mode 100644 index 00000000..ba80657c --- /dev/null +++ b/docs/l2-logical-plan.md @@ -0,0 +1,88 @@ +# L2 — logical plan + +Every front end's output converges on one shared relational tree (or +DAG) type at this layer — not one type per language. Still no summary +names, no positional column identity — those are L2 → L3 concerns +(below). The tree covers the standard relational vocabulary — source +scan, filter, projection, aggregation (grouping + aggregate functions), +windowing, deduplication, heavy-hitter ranking, set/join operators, +ordering, and named/referenced sub-plans (for expressing shared +sub-computations) — plus a small number of extension points for +per-language surface features that don't yet have a general relational +equivalent. Column references are still symbolic names at this layer, +resolved to positions by binding, below. + +## Name resolution: binding + +Binding walks the L2 tree and produces one self-contained schema that +every column reference in the resulting L3 tree indexes into: it seeds +columns from whatever catalog is available (or a minimal always-present +floor if the catalog knows nothing), and accounts for any name the +query references that the catalog didn't already know about. A schema +records whether it's a *complete* enumeration of a row's columns or an +*open* superset a runtime row may exceed — different query languages +sit at different points on that spectrum (a language with no fixed +schema, only label-like references, versus one with a declared +catalog) and binding accommodates both without forcing them into the +same shape. + +Why isolate name resolution as its own pass, rather than resolving +inline while translating: + +- **Everything downstream becomes purely structural and total.** Once + a schema exists, later passes work over positions, not names — a + column reference can't fail to resolve once binding has already run, + so every "column not found"-shaped failure is concentrated in one + place. +- **Schema/catalog policy is swappable independently of translation.** + Binding is generic over where schema information comes from; a + language with no catalog and a language with a real one both flow + through the same binding step, differing only in what the catalog + supplies. +- **It's the natural home for resolution-policy questions.** A + reference to "every column except these" can only be resolved once a + schema is known — and for an open schema, the full complement can't + be enumerated at all, so that has to be represented and deferred + rather than resolved eagerly. + +Each independent sub-tree (e.g. either side of a binary operation) +binds against its own schema, since the two sides may reference +entirely different sources — but a side must still see names +referenced by an *enclosing* operation (a grouping key mentioned above, +but not inside, either branch), so an enclosing scope's referenced +names are threaded down into each side's own binding pass. + +## The L2 → L3 step + +One shared step — used by every front end, regardless of language — +does three things in order: bind names to schema positions (above), +translate the tree structurally into L3's own shape, then run a shared +cross-language normalization pass so that semantically equivalent +queries, from any supported language, arrive at the same canonical +shape (covered in [`l3-intent-algebra.md`](./l3-intent-algebra.md)). +This ordering matters: normalization operates on already-resolved, +already-translated trees, so it never has to reason about per-language +surface syntax. + +## Column identity: two sources of truth + +A source may arrive at L2 in one of two shapes: schema-less (known only +by whatever columns a query happens to reference) or already +schema-carrying (arriving with a declared, complete column set from an +external catalog). Binding accommodates both: + +- A **schema-less** source falls back to binding's own usage-derived + schema — open, since a runtime row may carry more than what the + query referenced. +- An **already schema-carrying** source keeps its own declared schema + as-is — closed, since the catalog is a complete enumeration — and + binding's own fallback computation for that source goes unused. + +Either way, every column reference downstream resolves to a position +into *some* schema, uniformly; the difference is only where that +schema came from. This asymmetry also carries into whether a source +declares a **unique key** (a set of columns that together identify a +row): an already-cataloged source can carry a real one through +unchanged, while a usage-derived schema has no way to prove one, and so +never gets one. Why a unique key matters is covered in +[`l3-intent-algebra.md`](./l3-intent-algebra.md#unique-keys-and-cross-query-sharing). diff --git a/docs/l3-intent-algebra.md b/docs/l3-intent-algebra.md new file mode 100644 index 00000000..5b69e5a4 --- /dev/null +++ b/docs/l3-intent-algebra.md @@ -0,0 +1,153 @@ +# L3 — intent algebra + +The language- and deployment-independent canonical form. Pure intent at +this layer: no summary types, no summary parameters, no physical +operator choice — which concrete strategy answers a given piece of +intent is entirely an L4 decision (see +[`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md)). Data-model +agnostic by design: the same tree shape is meant to cover both +time-series-style sources and tabular sources, so a source's data model +is a property of its scan node, not a fork in the tree type. + +## Design rules + +1. **One canonical form per plan.** No two variants whose semantics + decompose into each other — e.g. a window over an aggregate is the + only windowed-aggregate shape; there is no separate combined node + for it. This keeps later rule-matching unambiguous. The one + deliberate exception: an operation that has a genuinely different + computational strategy available (e.g. heavy-hitter top-k, servable + by a dedicated streaming primitive) gets its own first-class intent + rather than being expressed purely as a composition of more generic + operators — but the generic composition (sort + limit) still exists + for cases that don't have that special strategy available. +2. **Language-orthogonal, except where an operation is genuinely + semantically distinct.** Two operations that look similar in some + source language's surface syntax but aren't actually interchangeable + (e.g. a cumulative-bucket interpolation vs. a genuinely + re-summarizable quantile) must be represented as separate intents, + discriminated by what they actually require to compute — never by + which language's syntax happened to produce them. +3. **Intent at L3, summary at L4.** An intent like "a quantile to this + accuracy" says what to compute; it never says which summary computes + it. +4. **A DAG, not a tree.** A producer can have more than one consumer + sharing its computed result — from explicit naming in the source + query (e.g. a named sub-query), from a later layer's decision to + reuse a shared computation across two different queries, or from + physical structure (a pre-aggregate feeding several downstream + consumers). The canonical form supports this by construction, via a + binding/reference pair of node kinds — a tree-only representation + would have to duplicate the producer per consumer and lose the + sharing. + +## The intent vocabulary + +Base relations, filtering/projection, aggregation (grouping keys + +aggregate intents), deduplication, windowing (including time-range and +time-shift), set/join operators, arithmetic, ordering, named/referenced +sub-plans, analytic (partitioned) window functions, scalar/vector +bridges for languages that distinguish the two, and a small number of +language-specific extension points with no general equivalent yet. +Grouping keys are one shared shape wherever "operate per group" is +needed (aggregation, ordering-with-partition, analytic windows) — a +grouping can either name its keys directly or specify them by exclusion +(everything *except* a given set), with the exclusion form only +resolvable once a schema is known. + +There is deliberately no separate node for "top-k" or "windowed +aggregate" as syntactic shapes — see design rule 1; both fold into +composition of the generic operators instead, except for the one +strategy-driven exception noted there. + +**Why a scan's source is a variant, not a separate tree type per data +model.** Most operators (filter, aggregate, …) have identical semantics +regardless of whether the input is a time-series window or a table scan +— only the leaf differs. One tree with a polymorphic leaf lets +data-model-agnostic rules apply uniformly across every data model; +rules that do care about data-model specifics gate on that leaf's +declared kind. Summaries themselves are meant to be data-model-agnostic +by construction — a streaming summary ingests a stream of values +regardless of what produced that stream. + +## The intent catalog + +What to compute, not how. Roughly: data-model-agnostic reducers (count, +sum, min, max, average, standard deviation, variance, quantile, top-k, +cardinality — each optionally carrying an accuracy target where +approximation is possible), counter-aware streaming derivatives (rate, +increase — genuinely distinct from a plain windowed sum, since they +must account for counter resets, which is why they're their own +intents), further counter-derivative functions, native-histogram +accessors, per-sample transforms, and a generic extension point for +deployment-specific intents the core vocabulary doesn't know the shape +of. + +**Why a streaming derivative is its own intent, not "sum with a +different name."** Because it isn't computable the same way — it +requires awareness of counter resets that a plain windowed reduction +doesn't need. Folding it into a generic reducer would either lose that +awareness or force every reducer to carry it. + +**Why there's no separate "quantile over a window" intent.** It would +duplicate "quantile" composed with "window" — the window's shape (kind, +size, slide) is already fully captured by the surrounding window +operator, and the quantile computation itself doesn't care whether its +input came from a windowed time series or a grouped table. One intent, +composed with the generic window operator, covers every language's +version of this idiom. + +## Why column identity is positional + +Everywhere the canonical form names a column — a grouping key, a +unique key, a designated time column — it is a position into a schema, +not a string. A layer-2 name means nothing without knowing which +schema it resolves against and at what offset; a position is settled +once, at bind time, and by construction can't fail to resolve for any +later pass. This also makes the canonical tree self-describing: every +scan carries its own schema, so any sub-tree's output schema is +computable purely from its inputs, without external context. A named +alias for the column-identity type (rather than a bare integer) is +still kept, so code that touches it can express "this is a column +position" as a distinct kind of value, not just any number. + +## Schema flow + +Every edge in the canonical tree carries a schema: its columns, which +column (if any) is the designated time index, its unique keys, and +whether it's closed (a complete enumeration) or open (a runtime row may +carry more). The tree is locally type-checked: a node's output schema +is a pure function of its inputs' schemas and its own parameters, +verifiable without consulting the rest of the tree. + +Three distinct kinds of schema-shaped information are easy to conflate +and shouldn't be: the schema flowing along the canonical tree's own +edges; the schema of the underlying data source itself (what tables or +metrics actually exist, consulted only during L1 → L2 name resolution); +and a catalog of what summaries exist and what they can serve +(consulted only from L4 onward). Each is read by a different layer, for +a different purpose. + +## Unique keys and cross-query sharing + +A unique key is a set of column positions that, together, identify a +row uniquely; a schema can carry more than one such key. Each operator +derives its output's unique keys from its inputs' — most operators +either pass keys through unchanged, or (for a grouping aggregation) +manufacture a fresh key from the grouping columns, since grouping by a +column set makes that set unique in the result. An operator whose +output can no longer guarantee row-level uniqueness (a projection, a +label rewrite, a union, a join) resets its unique keys to empty rather +than guess. + +**What this is for.** When several queries are planned together, a +shared sub-computation can be computed once and reused by every +consumer that needs it — but only if the producer is guaranteed to emit +the *same* rows for every consumer. A provable unique key is what +licenses that guarantee; structural identity between two consumers' +requested sub-computations tells you they *want* the same producer, but +not that its output is *stable* across reads. Without a provable unique +key, sharing is unsound and each consumer must recompute independently +— which is why a source with no way to prove a key (an open, +usage-derived schema) can never participate in this kind of sharing, +while a source with a declared key from an external catalog can. diff --git a/docs/l4-summary-bound-ir.md b/docs/l4-summary-bound-ir.md new file mode 100644 index 00000000..9935dde5 --- /dev/null +++ b/docs/l4-summary-bound-ir.md @@ -0,0 +1,148 @@ +# L4 — summary-bound IR + +The point where a plan commits to *what* answers each intent that L3's +canonical form deliberately leaves open. "Summary" is the umbrella term +for whatever answers an intent without re-scanning raw samples — +concretely, either an approximate sketch (e.g. a quantile sketch, a +cardinality sketch, a frequency/heavy-hitter sketch) or an exact +accumulator (e.g. sum, count, min/max, rate, increase). New primitive +classes are meant to slot in as additional summary kinds, not as a new +IR or a new translation layer. + +This layer has two distinct halves, deliberately kept separate: +**planning-time** binding (deciding, symbolically, which summary +answers which intent) and **serving-time** execution (actually +resolving that decision against whatever is materialized right now). +They're covered in turn below. + +## Planning-time: binding + +A decision, made once per query shape, symbolically: given an intent +and its accuracy requirement, which summary family and parameters +answer it — with no reference to what's actually stored anywhere yet. + +### The shape of a summary-bound plan + +A summary-bound plan mirrors the canonical intent tree but replaces +each summarizable node with a decision: + +- **Unbound / logical** — nothing rewrote this node (e.g. a filter or a + projection); it stays as ordinary logical computation over its + child's output. +- **Summary aggregation** — a summary family and its parameters, chosen + for the intent and its accuracy target, over a designated column and + grouping keys. +- **Summary-aware join** — a join realized via a summary technique + (e.g. a cardinality-aware join sketch), rather than a full + materialized join. +- **Summary subtraction / deletion** — algebraic operations on an + already-built summary, only valid for summary kinds that support + them. +- **Summary readout** — reading a value out of an already-built + summary; the inverse of building one. +- **Summary merge** — combining multiple built summaries into one; only + valid when every input agrees on family and parameters. + +A summary-bound plan is a DAG, not just a tree — a shared +sub-computation can appear as more than one reference to the same bound +node, mirroring the sharing already present in the canonical form (see +[`l3-intent-algebra.md`](./l3-intent-algebra.md)). Binding only fires +where an intent is recognizable in the tree; anything underneath an +unrewritten (logical) parent stays logical too — extending binding to +rewrite through a logical parent is a known open design question, not +attempted by the base design. + +Each bound node's output schema extends the canonical schema with one +new possibility: a field whose type *is* a built summary (identified by +its family and parameters). Two summary-typed fields are only +compatible if their family and parameters match exactly — which is what +lets subtraction, deletion, and merge reject a mismatched combination +as a planning-time error rather than a runtime one. + +## Serving-time: execution + +A lookup, done on every query: resolve each leaf of an already-bound +plan against whatever summaries are actually materialized right now. +Reality can diverge from what planning assumed — data might be missing, +several instances of the same summary might need merging, instances +might disagree on parameters — so serving needs its own error cases, +distinct from anything planning-time binding can raise. + +### Split of responsibility + +The *structural* rules are shared and deployment-independent: which +nestings of a summary-bound plan are valid, and what must agree for a +merge to be legal. Storage, summary math, and readout are entirely +deployment-supplied — this layer defines the interface a deployment +implements, not an implementation of summaries themselves. Concretely, +a deployment supplies: how to find candidate materialized instances for +a bound node, how to fetch one instance's state, how to merge several +instances' states, how to read a value out of a state, and how to +evaluate an unrewritten logical node directly. A shared execution +routine walks the bound tree and calls into these deployment-supplied +operations, so every deployment gets the same walk and merge logic for +free. + +Finding candidates for a summary-aggregation node requires seeing that +node's entire input sub-tree, not just a bare name — a summary +aggregation node carries no source identity of its own; that identity +lives further down, inside a scan. Walking down to find it is +deployment-specific knowledge (how a real store names and indexes +summarized data); the shared execution routine doesn't need to +interpret it, only pass the sub-tree along. + +### Nested composition + +A bound plan nests to arbitrary depth by construction, so execution is +plain recursion with no special-casing for depth. Two composition rules +matter: + +- **Only a node that produces summary state (a summary aggregation, or + another merge) can be a merge's child.** A readout or a logical node + has already collapsed to a plain value; merging values isn't the + operation a merge models. +- **A merge of merges preserves the family/parameter agreement + transitively** — the agreed family and parameters propagate upward + through every nesting level, so a nested merge is checked the same + way a flat one is. + +Summary-aggregation-over-summary-aggregation is itself valid and +expected — e.g. a two-stage precompute pipeline where one tier streams +a per-group reduction and another tier builds a richer summary over +that stream is a real, intended shape. + +**Deliberately left open:** whether a summary aggregation can validly +sit above a *readout* — "build a new summary from another summary's +already-computed value, at query time." Every design considered so far +only builds summaries incrementally from raw samples, at ingest time; +there's no established answer for building one from a +query-time-derived scalar. A deployment that hits this shape should +treat it as unsupported rather than assume either answer. + +### What's out of scope here + +Placement — *which machine* runs a piece of the plan — is L5's concern +entirely (see [`l5-physical-plan.md`](./l5-physical-plan.md)). This +layer is only about what "answer a query against already-materialized +summaries" means, on whichever machine ends up doing it. Some +operations (summary-aware join, subtraction, deletion) have no defined +execution behavior yet, because no binding rule produces them yet in +the base design — a deployment should add that behavior only once it +has a real need for the shape, rather than speculatively guessing the +right one now. + +## Choosing a summary for an intent + +Which summary family (if any) answers a given intent is itself a +pluggable decision: a default choice exists per intent, but a +deployment can supply its own ranking to prefer an alternative family +where one exists (e.g. an alternative quantile sketch, an alternative +cardinality sketch). This ranking is the one general-purpose extension +point in this layer — a deployment customizes *which* summary is +chosen, without touching how binding or execution works structurally. + +Not every intent has a summary counterpart. An intent whose accurate +computation requires richer partial state than a single summary value +can capture, or whose result is fundamentally non-mergeable across +partitions, has no summary realization by design — it always stays as +ordinary logical computation over raw data. diff --git a/docs/l4node-execution-model.md b/docs/l4node-execution-model.md deleted file mode 100644 index 22c3f979..00000000 --- a/docs/l4node-execution-model.md +++ /dev/null @@ -1,263 +0,0 @@ -# L4 - -L4 is the sketch-bound IR — `L4Node`/`SummaryExpr` in `asap-sketch`, -committing *what* answers each intent (a sketch family + params, or an -exact accumulator) that L3 (`QueryExpr`) deliberately leaves open. It -supports two different operations, previously only one of which had a -home in `ASAPController`. - -## Planning-time L4 - -`asap_plan::bind`: `QueryExpr -> L4Node`. A *decision*, made once per query -shape, symbolically — which sketch family and params should answer this -intent, with no reference to what's actually stored anywhere. -`docs/design.md`'s "Sketch binding is already committed by L4" describes -this half at the architecture level; this section has the IR itself. - -### The IR - -```rust -pub struct L4Node { - pub expr: SummaryExpr, - pub schema: L4Schema, -} - -pub enum SummaryExpr { - /// No L4 rule rewrote this L3 node (e.g. `Filter`, `Project`, `Sort`). - Logical(Box), - - /// Sketch aggregation — `sketch`/`params` chosen from the catalog for - /// the `AggIntent` under an `AccuracyTarget`. - SummaryAgg { - child: Rc, - sketch: SummaryKind, - params: SummaryParams, - col: ColumnRef, // the column being summarised - reduction: Reduction, // GROUP BY keys, or "no grouping concept" (#163) - }, - - /// Sketch-aware join (KMV/theta for cardinality, join-sample for - /// sampling). Emitted only when a `Bind*OnJoin` rule fires. - SummaryJoin { outer: Rc, inner: Rc, key: ColumnRef, sketch: SummaryKind, params: SummaryParams }, - - /// Subtract one sketch from another — requires catalog `subtractable`. - SummarySubtract { left: Rc, right: Rc }, - - /// Delete a key from a sketch — requires catalog `deletable`. - SummaryDelete { sketch_input: Rc, key: ColumnRef }, - - /// Read a value out of a built sketch — inverse of `SummaryAgg`. The - /// `Sketch(...)` schema type does not propagate past an estimate. - SummaryEstimate { sketch_input: Rc, query: SketchQuery }, - - /// Union of sketches — requires catalog `mergeable`; all children must - /// agree on `(sketch, params)` (see "Serving-time L4" below for how - /// this same requirement re-appears, and why, at query time). - SummaryMerge { children: Vec> }, -} -``` - -`L4Schema` extends L3's schema with one new field type, -`L4DataType::Sketch(SummaryKind, SummaryParams)` — a schema whose -value-bearing field carries this dtype is a "sketch-state schema." The -`(kind, params)` pair is the type identity: two sketch-state fields are -compatible only if both match exactly, which is what lets `SummarySubtract`/ -`SummaryDelete`/`SummaryMerge` reject a family/param mismatch as a -plan-time type error rather than a runtime one. Per-node schema rule -summary: `SummaryAgg` outputs its grouping columns verbatim plus one -`Sketch(sketch, params)` field; `SummaryEstimate` outputs a plain -row-shaped schema (the sketch field doesn't survive it); -`SummarySubtract`/`SummaryMerge` require -every input's `Sketch(s, p)` to already agree, and the catalog capability -flag (`subtractable`/`deletable`/`mergeable`) gates whether the node can -fire at all. - -Traversing from the root yields a DAG, not just a tree — shared -sub-expressions appear as multiple `Rc` references to the same `L4Node` -(fan-in is otherwise expressed via L3's own `QueryExpr::LetBinding`/`Ref`). -`asap_plan::bind` only fires on the `Aggregate` spine; anything under an -unrewritten logical parent stays `Logical(...)` unbound — "rewriting -through logical parents" is tracked separately (#6/#33), not attempted -here. - -## Serving-time L4 - -`L4Node -> Value`. A *lookup*, done on every query: resolve each leaf of -an already-decided tree against whatever is actually materialized right -now. Reality can diverge from the plan in ways planning time never sees — -missing data, multiple instances needing a merge, instances that disagree -on params — which is why this half needs its own error cases -(`NoCandidates`, `MergeKindParamsMismatch`), distinct from anything -`asap-plan`'s binder raises. Until now nothing in `ASAPController` defined -this half; every deployment answering queries would otherwise reinvent -the same recursive walk and merge rules independently. -`crates/sketch/src/exec.rs` (`SummaryExecutor` trait + `execute` -function) is the shared answer — the rest of this doc is its design; see -the module docs for the authoritative detail. - -### Split of responsibility - -`asap-sketch` owns the *structural* rules — which nestings are valid, that -a `SummaryMerge`'s children must agree on `(SummaryKind, SummaryParams)`. -The deployment owns storage, sketch math, and readout — `asap-sketch` has -no KLL/CMS/HLL/DDSketch implementation of its own, so `SummaryExecutor`'s -`State`/`merge_states`/`readout` are entirely deployment-supplied. - -```rust -pub trait SummaryExecutor { - type Handle: Clone; // e.g. a data plane's sid - type State; // decoded sketch/accumulator bytes - type Value; // a query answer - type Error; - type GroupKey: Clone + Ord + Default; // e.g. a label-value map - - fn find_candidates(&self, sketch: &SummaryKind, params: &SummaryParams, - col: &ColumnRef, reduction: &Reduction, child: &L4Node) - -> Result, Self::Error>; - fn fetch_state(&self, handle: &Self::Handle) -> Result; - fn merge_states(&self, states: Vec) -> Result; - fn readout(&self, state: &Self::State, query: &SketchQuery) -> Result; - fn logical(&self, expr: &QueryExpr) -> Result; -} - -pub fn execute(node: &L4Node, exec: &E) -> Result, ExecError>; -``` - -`find_candidates` receives the `SummaryAgg`'s whole child subtree (not just -a metric name) because that's the only way to name what to look up — -`SummaryAgg` itself carries no metric/source field; that identity lives -further down, typically inside a `Logical(Window{Scan{..}})` leaf. Walking -down to find it is deployment-specific `QueryExpr`-shape knowledge -(mirrors how a real store's edge-fact extraction already works); `execute` -doesn't interpret it. - -**Grouping (`reduction`).** `SummaryAgg` carries `reduction: Reduction` — -the same L3 type (`asap_ir::intent_algebra::Reduction`) the `Aggregate` -node it was bound from carried (issue #165), reused verbatim rather than -flattened to a bare `Vec`. `execute` has no schema-level -knowledge of what a "group value" looks like for a given deployment — -that's `find_candidates`'s job: it tags every handle it returns with the -`GroupKey` (an opaque, deployment-chosen type) that handle belongs to. -`execute` groups those tagged handles itself before folding -(`fold_states`/`merge_states` only ever combine same-group states), and -every `ExecOutcome` — `State` or `Value` — is a per-group list, never a -single bare value. A `Logical` leaf (which has no grouping concept -either, but for a different reason — it's deferred entirely to the -deployment) is simply a list of one entry under `GroupKey::default()`, -not a different shape — callers don't need to special-case it. - -`Reduction` is two variants, not a bare `Vec` (issue #163): -`Reduce(by)` is a genuine cross-series reduction — an *empty* `by` still -means exactly one group, so `find_candidates` must tag every handle it -returns with one shared `GroupKey` there. `PerEntity` means there is no -grouping concept for this shape at all (a bare per-series range function -like `quantile_over_time(...)`, with no aggregation operator and so no -`by(...)` to begin with) — `find_candidates` must tag every handle with -its own distinct `GroupKey` there, one per entity, and `execute` must -never merge across them. Both used to collapse to the same `by: []` on -`SummaryAgg`, with nothing in the trait able to tell them apart; -`asap-plan::bind` decides which `Reduction` a node gets once, at L3→L4 -binding, and carries it onto the L4 node unchanged — the same value that -already decided the node's per-entity-vs-cross-series *schema* — so the -two can't drift out of sync. - -`SummaryMerge`'s `(SummaryKind, SummaryParams)` agreement check stays -*global* across every group from every child (it's a planning-time -property of the node, fixed before any group value is known); the actual -state-folding happens *within* each group independently, never across -groups. First landed narrower (a single ungrouped value per tree) in #155; -broadened to a per-`GroupKey` shape in response to #159 once a real -consumer (`data_plane`'s grouped PromQL queries — `sum by (zone)`, -`quantile by (zone)(...)`, the common case, not an edge case) showed the -original shape would have silently merged every group's state together; -`by` was then replaced with `Reduction` in response to #163, once -implementing `find_candidates` for real showed an empty `by` alone -couldn't tell a per-entity shape from an explicit full reduction. - -### Nested composition - -`L4Node`'s children are `Rc`, so the tree nests to arbitrary depth -by construction — `execute` is plain recursion, no depth special-casing. -Two things aren't obvious from the type alone: - -- **Only state-producing nodes can be a `SummaryMerge` child** — - `SummaryAgg` or another `SummaryMerge`. A `SummaryEstimate` or `Logical` - child has already collapsed to a value; merging values isn't the - operation this models. `execute` rejects it (`MergeChildNotState`). -- **Merge-of-merges preserves the `(kind, params)` agreement transitively** - — `execute` propagates the agreed `(SummaryKind, SummaryParams)` up - through every level via `ExecOutcome::State`, so nested merges are - checked the same way a flat one is, without re-deriving anything from - schema. - -`SummaryAgg` nesting itself (`SummaryAgg{child: SummaryAgg{..}}`, e.g. -`quantile(0.9, sum by (job) (m))`) is valid and already binds correctly -today (`asap-plan`'s own `nested_aggregates_bind_per_node` test) — the -child is "the input rows this aggregate is built from," and a two-stage -precompute pipeline (one tier streams per-job sums, another tier builds a -KLL over that stream) is a real shape. - -**Deliberately left open**: whether a `SummaryAgg` can validly sit *above* -a `SummaryEstimate` — "build a new summary from another summary's -already-computed readout, at query time." Every materialized-summary store -checked against this design only serves summaries built incrementally at -*ingest* time from raw samples; there's no precedent for building one from -a query-time-derived scalar. A `SummaryExecutor` that hits this shape -should treat it as unsupported rather than assume either answer. - -### What's out of scope here - -Placement/stage-allocation (L5, `core::physical::executor` in -`docs/design.md`) is unrelated — that's about *which machine* runs a -sub-DAG, decided at plan time. This module is about what "run a sub-DAG -against already-materialized state" means at *query* time, on whichever -machine ends up doing it. `SummaryJoin`/`SummarySubtract`/`SummaryDelete` -have no trait methods yet — no `Bind*` path produces them, so `execute` -just reports them unsupported; add methods when a real consumer needs one -rather than guessing the right shape now. - -### Consumer - -ASAPQuery-backend's `data_plane` is the first (planned) consumer — see -its own `data_plane/docs/l4node-plan-executor-design.md` for how -`ASAPQueryEngine` implements `SummaryExecutor` (`Handle = sid`, -`find_candidates` via its sid catalog, `merge_states`/`readout` via -`asap_sketchlib`) and the deployment-specific concerns this module doesn't -cover (raw-AST PromQL fallbacks, the `(kind, params)` catalog-drift -question at scale, rollout). - -## Supported operators - -### `SummaryExpr` operators - -| Variant | Planning-time (`asap_plan::bind`) | Serving-time (`exec.rs`) | -|---|---|---| -| `Logical` | produced (the fallback for anything not bound) | supported — delegates to `SummaryExecutor::logical` | -| `SummaryAgg` | produced | supported | -| `SummaryEstimate` | produced | supported | -| `SummaryMerge` | **not produced by anything today** — meant to be inserted by an L5 stage allocator on cut edges (design.md), and no allocator does that yet, in this crate or ASAPQuery-backend's `control_plane` | supported (`execute` fully implements the merge-precondition check and recursion — it just has nothing to run against yet) | -| `SummaryJoin` / `SummarySubtract` / `SummaryDelete` | not produced by any `Bind*` rule | not supported — `execute` returns `ExecError::NotYetSupported` | - -### `SummaryKind` (via `boundary::summary_candidates` / `implementation_for`) - -All 14 variants are wired into the boundary decision; `DefaultCostModel` -picks the first column below, a custom `CostModel::rank_candidates` can -promote the second: - -| `AggIntent` | Default | Also a candidate | -|---|---|---| -| `Quantile` | `Kll` | `DDSketch` | -| `Cardinality` | `Hll` | `Theta`, `Kmv` | -| `Count` (approximate) | `Cms` | `CountSketch` | -| `TopK` | `CmsWithHeap` | `CountSketchWithHeap` | -| `Sum` | `Sum` (exact accumulator) | — | -| `Min` / `Max` | `MinMax` (exact accumulator) | — | -| `Rate` | `Rate` (exact accumulator) | — | -| `Increase` | `Increase` (exact accumulator) | — | -| `Count` (exact) | `Count` (exact accumulator) | — | - -`Avg`/`StdDev`/`Variance` and the per-series/range-reducer intents -(`Changes`, `Delta`, `Deriv`, `HistogramQuantile`, …) have no `SummaryKind` -at all — `implementation_for` returns `PassThrough` for them by design -(richer partial state than a single accumulator value, or genuinely -non-mergeable), so they stay `SummaryExpr::Logical`. diff --git a/docs/l5-physical-plan.md b/docs/l5-physical-plan.md new file mode 100644 index 00000000..fd3cc1bd --- /dev/null +++ b/docs/l5-physical-plan.md @@ -0,0 +1,74 @@ +# L5 — physical plan + +**Status: an interface design, not a concrete implementation.** +Everything below describes the intended shape of this layer — what a +deployment must implement — rather than a shipped component. This +layer's concrete realization is inherently deployment-specific: a +downstream deployment supplies its own instance of it. + +## What L5 is for + +L3 and L4 are symbolic — they describe computation and summary choices +without committing to *where* anything runs. L5 produces the concrete, +placed plan: which physical stage each piece of computation runs at, +how much parallelism it gets, and — ultimately — a form a specific +deployment can act on (e.g. serialize into that deployment's own +configuration format). The same summary-bound plan, combined with a +different topology, is meant to produce a different placement without +re-running any earlier layer — topology is a parameter to this layer, +not a fork in its logic. + +## Core concepts + +- **Physical planner** — the interface a deployment implements: given a + summary-bound plan and a topology, produce that deployment's own + physical output. +- **Topology** — which physical stages exist (e.g. edge, gateway, + backend, or a single in-process stage) and how they connect. A + deployment supplies its own topology; the physical planner interface + is generic over it. +- **Stage** — a categorical placement tier (e.g. "edge," "backend"). +- **Executor** — a concrete runtime instance that executes a piece of + the plan, situated at one stage. A single stage can back many + executors (e.g. a large edge fleet all at the "edge" stage); a + singleton stage backs exactly one. +- **Deployment constraints** — the per-deployment budgets and + capabilities (memory, available summary backends, network topology, + the concrete list of executors) that placement must respect. + +## Division of responsibility + +Placement is meant to split into two decisions: + +1. **Stage-level allocation** — given the summary-bound plan, the + topology, and deployment constraints, decide which *stage* each + piece of the plan runs at. This decision is stage-level only — + generic across deployments, since it only needs to know which + stages exist and how they connect, not which concrete executors a + deployment happens to have. +2. **Per-executor fan-out** — given a stage-level assignment, decide + which concrete executor(s) at that stage actually run each piece. + This is deployment-specific, since it needs the deployment's own + executor list and its own policy for spreading work across them. + +Stage-level allocation is also where a summary merge is meant to get +inserted — wherever a summary-bound plan is cut across a stage +boundary that produces more than one instance of the same summary +state, those instances need to be merged before serving can use them +as a single value (see +[`l4-summary-bound-ir.md`](./l4-summary-bound-ir.md)). + +## Summary catalog + +A registry of what summaries exist, keyed by summary family: which +intents each can serve, whether it supports merging/deletion/subtraction, +its accuracy model, what grouping shapes it supports, its valid +parameter ranges, and its cost characteristics. Built once, ahead of +time; read by L4's binding decision (to choose a family) and by L5 (to +instantiate it). Rejecting an out-of-range parameter belongs here, at +catalog level, so that L5 never has to handle an unsupported +configuration reaching it. This catalog is meant to be shared +infrastructure that every deployment reads from, rather than something +each deployment reinvents independently — though which summary +families and parameter ranges are actually registered is itself +deployment policy. diff --git a/docs/migration-plan.md b/docs/migration-plan.md deleted file mode 100644 index a3642507..00000000 --- a/docs/migration-plan.md +++ /dev/null @@ -1,374 +0,0 @@ -# ASAPController — migration plan - -Companion to `design.md`. Concrete phase-by-phase plan to get from today's three-repo mess to the target layout. - -> **Status (reconciled with the landed workspace).** Phases 0–1 (skeleton + -> the L1–L3 query→IR core) have substantially landed, but **not** as the single -> `core` crate this plan describes: the core was built and then split by -> pipeline role into the layer-named stack — `asap-ir`, `asap-l2`, `asap-sketch`, -> `asap-plan`, `asap-frontend-promql`, `asap-frontend-sql`, `asap-lower`, -> `asap-e2e` (see `design.md` §5.1). The L4/L5 framework, runtime, and -> `deployment-model-*` crates (Phases 2–7) are **not yet built**. The -> `core::` names below are the original plan; read them through the map -> in `design.md` §6.0. Per-phase "**As landed**" notes flag where reality -> diverged. - -## Premise - -- **Target repo**: `github.com/ProjectASAP/ASAPController` — has the landed core (`design.md` §5.1); runtime + deployment-model crates pending. -- **Sources**: `DataCollector/controller/`, `ASAPQuery-backend/asap-planner-rs/` (the *newer* of the two copies), `asap-fusion/`. -- **Hard constraint**: ASAPQuery-backend must keep running throughout. `POST /api/v1/streaming-config` contract must not break. OpAMP push to agents must not break. -- **Soft constraint**: minimise the "big-bang" window. Each phase lands a working state. - -## Phase-by-phase plan - -Ordered by risk, lowest first. Each phase is a single PR unless noted. - ---- - -### Phase 0 — Skeleton (ASAPController) - -**Scope:** Create the empty workspace + crate skeleton; no logic yet. - -**Work:** -- In `ASAPController/`: workspace `Cargo.toml` listing all 7 crates as empty. -- Create each crate with a `lib.rs` containing just `// placeholder` and a minimal `Cargo.toml`. -- Check in `README.md`, `docs/design.md` (this design), `docs/migration-plan.md` (this file). -- Copy `DataCollector/opamp.proto` → `ASAPController/proto/opamp.proto`. Wire `control-proto` to generate from it via `prost-build`. Verify `cargo build` succeeds. -- CI: basic `cargo build && cargo test && cargo clippy -- -D warnings && cargo fmt --check` on every PR. - -**Risk:** ~zero. No semantics migrated yet. - -**Exit criteria:** `cargo build` green on empty workspace; CI running. - -**PR size:** ~200 LOC (boilerplate). - -> **As landed.** The workspace exists but the crate set differs from the "7 -> crates" this phase imagined (which were `core` + `runtime` + -> deployment-models + …). What ships today is the layer-named core split: -> `ir` / `sketch` / `plan` / `frontend-promql` / `frontend-sql` / `lower` / -> `e2e`. The `runtime` / `deployment-model-*` / `control-proto` / `bin` -> members are still to come. - ---- - -### Phase 1 — Core extraction: all-5-layer shared infrastructure - -**Scope:** Lift the DC controller's **layers 1–3** (query parsing + per-language algebra + intent-algebra IR) into `crates/core/` verbatim, AND lift L4 and L5 **shared infrastructure**: rule engine, shared rule library, cost-model traits, `PhysicalPlanner` trait, stage allocator framework, sketch catalogue, sketch-bound L4 IR (`SketchExpr` in `core::sketch_algebra`). Deployment models in later phases pick from this common base and only add deployment-model-specific pieces. - -Per design §3, these five layers' infrastructure is query-language-independent AND deployment-model-independent; the concrete rule set, topology, and emitter are deployment-model-specific. - -**Work — L1-3 (straight lift from DC):** -- **`core::query_language`** (L1): lift `DataCollector/controller/src/query_parser/{promql,sql,mod}.rs` → `core/src/query_language/`. One module per language. -- **`core::logical_plan`** (L2): lift per-language algebra tree definitions → `core/src/logical_plan/`. One `enum LogicalPlan` per language. Add `core::logical_plan::datafusion` as a thin re-export of `datafusion::logical_expr::LogicalPlan` for fusion's L2. -- **`core::intent_algebra`** (L3): lift `algebra/{expr,directory}.rs` → `core/src/intent_algebra/`. Enforce **intent-only** — `AggIntent` carries accuracy target, never sketch type. **Generalize `QueryExpr::Scan` over data model**: introduce a `Source` sum (`TimeSeries` / `Table` / `Join` variants) inside `Scan`, and a `DataModel` enum (`TimeSeries` / `Tabular` / `Any`). DC's current scan logic becomes `Source::TimeSeries`; `Source::Table` is new (stub impl fine for Phase 1 — deployment-model-asapfusion fills it in Phase 3). Add `AggIntent::requires() -> DataModel` so L4 rules can skip non-applicable intents. See design §3 "Data-model support" and §6 `core::intent_algebra`. ~300 LOC on top of the DC lift. -- **`core::lower`**: lift `algebra/lower.rs` (L1→L2→L3 passes) → `core/src/lower/`. One `lower_` entry point per language. Each lowering produces the appropriate `Source` variant (PromQL / SQL → `Source::TimeSeries`; DataFusion → `Source::Table`). - -**Work — L4 framework (NEW):** -- **`core::optimizer::engine`**: rule driver — fixed-point iteration, cycle detection, priority ordering. Generic over `OptimizerRule`. -- **`core::optimizer::rule` trait**: `OptimizerRule` + `RuleCategory` enum (`PushDown | Fusion | Elim | Bind | StageRouting`). Priority u16. `apply(&expr, &constraints) -> Option`. -- **`core::optimizer::rules`**: shared rule library. Lift the **deployment-model-agnostic rules** from DC's `algebra/optimizer.rs`: - - `BindKllOnQuantile` (intent → KLL with params from accuracy) - - `BindCmsOnCount` (+ `BindCmsOnSum`) - - `BindHllOnCardinality` - - `BindDdsketchOnQuantile` (alternative to KLL) - - `FusionPassthrough` (Aggregate over Filter → push Filter under) - - `ElimNoopFilter`, `ElimNoopSort`, `ElimNoopLimit` - - DeploymentModel-specific rules (stage-aware push-down, TCO-aware deferral) STAY in `deployment-model-asaplifecycle` — not lifted. -- **`core::optimizer::cost`**: `CostModel` trait + `Accuracy` / `Latency` / `Dollars` value types. Lift DC's **generic** cost functions (memory budget, accuracy degradation curve). DeploymentModel-specific cost models (delta, online, pareto, tco) STAY in `deployment-model-asaplifecycle`. -- **`core::plan`**: `DeploymentConstraints` trait — memory budgets, network topology descriptor, available sketch backends. - -**Work — L5 framework (NEW):** -- **`core::physical::planner_trait`**: `PhysicalPlanner` trait parameterized on `Topology` + `Output`. -- **`core::physical::topology`**: `TopologyDescriptor` trait + pre-baked impls — `ThreeStage` (edge/gateway/backend), `SingleStage` (backend-only), `ZeroStage` (in-process). -- **`core::physical::stage_allocator`**: generic topology-driven allocator. Lift DC's `SketchAllocator` from `algebra/allocator.rs`, parameterize the hardcoded 3-stage assumption over `TopologyDescriptor`. -- **`core::physical::sketch_catalog`**: candidate sketch types + parameter constraints. Lift from DC's `algebra/directory.rs`. - -**Work — other core pieces:** -- **`core::pipeline`**: the L1→…→L5 driver. ~300 LOC. -- **`core::workload`**: lift `QuerySpec` + add `QueryWorkload` sum type. -- **`core::emit`**: `PlanEmitter` trait + `EmitError`. -- **`core::registry`**: `DeploymentModelRegistry`, `DeploymentModelId` enum. - -**Testing:** -- Port DC's existing `query_parser/` + `algebra/` unit tests verbatim. All pass after the move. -- Golden-file round-trip test for L1→L2→L3 (parse → lower → JSON of `QueryExpr`). -- Golden-file test for L4 shared rule library: each rule has a "before → after" `QueryExpr` fixture demonstrating its behavior. -- Unit tests for `StageAllocator` under each pre-baked topology. - -**Risk:** medium-high. This phase does more than a straight lift — it generalizes DC's hardcoded `SketchAllocator` over topology + lifts a subset of optimizer rules into a shared library. Keep changes mechanical; any semantic revision is deferred to Phase 5/7. - -**Decision points:** -1. `asap_types` — publish to a private git tag in Phase 0. -2. `promql-parser` / `sqlparser` versions — adopt DC's newer versions. -3. **Which DC rules go into `core::optimizer::rules`?** Rule of thumb: if it depends only on `QueryExpr` + `DeploymentConstraints` (no DC-specific types), lift. If it reaches into DC's stage graph, backend_client, or OTel YAML shape, it stays in `deployment-model-asaplifecycle`. Default: lift `Bind*`, `Elim*`, `FusionPassthrough`; keep `StageAwarePushDown`, `TransmissionCostRewrite` in deployment-model-asaplifecycle. - -**Exit criteria:** `cargo build` green; DC's parser + algebra + subset of optimizer unit tests pass from `core/`; `StageAllocator` unit-tested against all pre-baked topologies; golden-file corpus locked in. - -**PR size:** ~6300 LOC (L1-3 lift ~4000 + L4/L5 framework ~2000 + `Source` sum / `DataModel` generalization ~300). Could split into (a) L1-3 lift + traits + `Source` sum, (b) L4/L5 framework — prefer this if the PR review would otherwise be unwieldy. - -> **As landed.** The **L1–L3** half shipped, but as separate crates rather than -> `core::*` submodules, and as a *slimmed, refactored fork* rather than a -> verbatim DC lift (see `docs/intent-algebra-reconciliation.md`): -> `core::query_language`/`logical_plan`/`lower` → the L1→L2 half of -> `asap-frontend-promql` / `asap-frontend-sql` + the shared L2→L3 `convert_root` -> in `asap-ir::intent_algebra::lower`; `core::intent_algebra` → `asap-ir` -> (with the `Source` / `DataModel` generalization); `core::sketch_algebra` → -> `asap-sketch`. The **L4/L5 framework** (optimizer engine, rule library, cost -> model, `StageAllocator`, `PhysicalPlanner`, sketch catalogue) is **not built**: -> `asap-plan` is a placeholder holding workload-level CSE, with `cost_model` / -> `boundary` / `canonicalize` as stubs. - ---- - -### Phase 2 — Runtime extraction: HTTP + store - -**Scope:** Lift DC controller's HTTP server + in-memory stores into `crates/runtime/`. OpAMP + replanner + monitor wait till later phases (they can pull behind a `feature` flag for now). - -**Work:** -- `runtime/http/` — copy `DataCollector/controller/src/main.rs`'s axum routes verbatim, split into modules by route. Keep all behavior. -- `runtime/store/` — lift `store/PlanStore` and `store/WorkloadStore` as-is. -- `runtime/backend_client/` — lift `backend_client.rs` as-is. -- The HTTP handlers still reference DC's planner concretely at this point. That's OK; we'll abstract in Phase 4. - -**Risk:** low — all code is a 1:1 copy. Tests that shipped with DC should pass as-is. - -**Exit criteria:** `cargo build` green; runtime compiles against core trait stubs; DC controller's existing HTTP integration tests (if any) pass when wired to runtime. - -**PR size:** ~1500 LOC (mostly moves, little rewriting). - ---- - -### Phase 3 — `deployment-model-asapfusion`: first thin deployment model - -**Scope:** Land `deployment-model-asapfusion` as a thin deployment model that picks rules from core's shared library. Lowest-risk because fusion's existing `SketchConfigRule` maps directly onto core's `BindKllOnQuantile` + `BindCmsOnCount` + `BindHllOnCardinality` that Phase 1 already lifted. - -**Decision — where does `deployment-model-asapfusion` live?** Default: **in the `asap-fusion` repo**, as a crate that depends on `asap-ir` + `asap-plan`. asap-fusion is a research project with independent release cadence; keeping it in-tree in its own repo preserves that. If the team prefers lockstep, the alternative is `crates/deployment-model-asapfusion/` in ASAPController workspace — architecture is identical either way (see design §8). - -This plan assumes the in-repo-at-asap-fusion placement. If in-workspace is chosen, swap paths accordingly. - -**Work:** -- In `asap-fusion/`: add new crate `deployment-model-asapfusion/` that depends on `asap-ir`. Move the translator + optimizer + executor here. -- **Fill in `Source::Table` lowering**: Phase 1 stubbed this; now implement the L2 (DataFusion `LogicalPlan`) → L3 (`QueryExpr` with `Source::Table`) lowering pass in `core::lower::datafusion` — or, if fusion prefers, keep the lowering inside the deployment model crate and use the `Source::Table` type directly. Either way, Phase 1's stub becomes concrete. -- `impl DeploymentModel for ASAPFusionDeploymentModel`: - - `rules()` picks `BindKllOnQuantile`, `BindCmsOnCount`, `BindHllOnCardinality` from `core::optimizer::rules::*` (drops asap-fusion's `SketchConfigRule` in favor of core's rules). - - Adds deployment-model-specific `HashModeRule` (stays local — depends on DataFusion-specific types). - - `topology()` returns `core::physical::topology::ZeroStage`. - - `emitter()` returns the DataFusion `LogicalPlan` rewriter. -- Port the microbenchmarks to `deployment-model-asapfusion/benches/`. Verify numbers within 5% of pre-migration baseline. -- Legacy `asap-fusion` crate becomes a thin shim that re-exports the new crate, so in-flight benchmark harnesses keep compiling. - -**Risk:** low — no service consumes asap-fusion today; only downstream is its own benches. The rule-lift cleanup (`SketchConfigRule` → core rules) is the main change; golden-file test over the 3 existing rewrites catches any divergence. - -**Exit criteria:** `cargo bench -p deployment-model-asapfusion` produces numbers within 5% of pre-migration; 3 rewrites documented in `asap-fusion/TESTS.md` produce identical output. - -**PR size:** ~1800 LOC (asap-fusion's ~3k lines minus what folds into core's shared rule library). - ---- - -### Phase 4 — `deployment-model-asapquery`: migrate asap-planner-rs's newer copy - -**Scope:** Move `ASAPQuery-backend/asap-planner-rs/` (the newer copy) into `crates/deployment-model-asapquery/`, **and refactor it to fit the 5-layer model**: reverse-engineer PromQL pattern templates into an L2 tree, and split L3 intent from L4 sketch binding. Delete the older copy in `ASAPQuery/`. - -This phase is larger than a straight lift because two structural conformance changes are bundled in. - -**Work — the lift:** -- Diff the two `asap-planner-rs` copies. Newer (backend) copy wins. **Port only the newer; discard the older.** -- Copy `ASAPQuery-backend/asap-planner-rs/src/{lib,main}.rs` + `src/{planner,output,query_log,prometheus_client}/` → `crates/deployment-model-asapquery/src/`. -- `impl DeploymentModel for ASAPQueryDeploymentModel` — registers YAML emitters (`StreamingConfigEmitter` + `InferenceConfigEmitter`) and HTTP route `POST /plan/query`. -- Port the CLI: `bin/asap-query/main.rs` with clap flags mirroring `asap-planner-rs`'s current CLI. - -**Work — L2 tree conformance (NEW):** -- Define `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes planner currently template-matches (`OnlyTemporal`×2, `OnlySpatial`, `OneTemporalOneSpatial`×2) as first-class L2 tree nodes (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`). -- Write L1→L2 lowering in `core::lower::promql` that takes a `promql_parser::parser::Expr` and produces a `PromqlLogicalPlan`. The five existing pattern shapes become five recognized L2 tree shapes plus a generic fall-through. -- Retire planner's `PromQLPattern` / `PromQLPatternBuilder` / `QueryPatternType` — replaced by tree-shape inspection at L3 lowering. -- Do the same for SQL (retire `SQLPatternMatcher`, produce `SqlLogicalPlan`). -- **This is the conformance cost the user explicitly accepted for architectural uniformity.** Budget accordingly. - -**Work — L3 / L4 split (NEW):** -- In `core::lower::promql_to_intent_algebra`: `PromqlLogicalPlan → QueryExpr` producing **intent-only** L3 (no sketch names). -- `Statistic` enum → `AggIntent` subset (9 of DC's 25 variants). No sketch type, no sketch params at L3. -- In `deployment-model-asapquery/src/rules.rs`: picks core's `BindKllOnQuantile`, `BindCmsOnCount`, etc. from Phase 1's shared rule library + adds a deployment-model-specific `PrecomputeEngineBindRule` (handles the precompute-engine-specific flavor: which sketches are available in ASAPQuery-backend's accumulator set, what params match the engine's config schema, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). -- Update `build_agg_configs_for_statistics` to call `core::optimizer::engine::RuleEngine::run()` with the deployment model's rule set instead of the inline `map_statistic_to_precompute_operator` fusion. -- `IntermediateAggConfig` loses its inline sketch binding; it's now an L4 output type produced by `PrecomputeEngineBindRule`. - -**Work — topology + emitter:** -- `deployment-model-asapquery/src/topology.rs`: declare `core::physical::topology::SingleStage` (backend-only). -- `deployment-model-asapquery/src/emit/`: `StreamingConfigEmitter` + `InferenceConfigEmitter`. Lift from asap-planner-rs's `output/generator.rs`. These are the authoritative YAML emitters (deployment-model-asaplifecycle calls them). - -**Testing:** -- **Golden-file test**: capture a corpus of today's `asap-planner-rs` inputs → outputs. The new CLI must produce byte-identical output. Put under `bin/asap-query/tests/golden/`. This is the non-negotiable safety net for the refactor. -- **L2 round-trip test**: `parse → build PromqlLogicalPlan → pretty-print back → parse` stays stable across tree rewrites. -- **L3 intent-only test**: after L3 lowering, assert `AggIntent` carries no sketch-type information (type-system-enforced, not runtime-enforced — `AggIntent::Quantile` carries `AccuracyTarget`, not `SketchType`). - -**Risk:** medium-high. Two refactors on top of a lift. Golden-file corpus is the gate. - -**Parallel work** (separate PR, not dependent): -- Update ASAPQuery-backend's `docker-compose-precompute.yml` to swap the `asap-planner-rs` init container for `asap-controller plan --workload /config/controller-config.yaml --output-dir /asap-planner-output`. Ship **after** ASAPController publishes its first binary release. - -**Exit criteria:** -- `asap-query` CLI produces byte-identical YAML to `asap-planner-rs` on the golden-file corpus. -- HTTP `POST /plan/query` returns the same YAML over HTTP. -- `AggIntent` post-L3 contains zero `AggregationType` / sketch params (type-enforced). -- Planner's `PromQLPattern*` / `SQLPatternMatcher` files are deleted. - -**PR size:** ~4500 LOC (straight lift ~2500 after L4 rules move to core's shared library; L2-tree work + L3/L4 split adds ~2000). - ---- - -### Phase 5 — `deployment-model-asaplifecycle`: DC's remaining deployment-model-specific pieces - -**Scope:** Land `deployment-model-asaplifecycle` as a thin deployment model. L1-3 + L4 shared rules + L5 framework already moved in Phase 1; only DC's **deployment-model-specific** L4/L5 pieces remain: stage-aware rewrite rules, cost models, the 3-stage `ThreeStage` topology's DC-specific constraints, OpAMP + backend emitters, SLA replanner wiring. - -**Work — deployment model crate:** -- **Rule selection**: `deployment-model-asaplifecycle/src/rules.rs` picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules: - - `StageAwarePushDown` — pushes ops toward edge stage when data size + edge memory budget permit - - `TransmissionCostRewrite` — uses the TCO model to defer aggregation when it saves bandwidth - - `DeltaRewrite` — replaces raw sends with delta-encoded sends when the online cost model says it pays off - Each impls `core::optimizer::rule::OptimizerRule`. -- **Cost models**: `deployment-model-asaplifecycle/src/cost.rs` — lift `DataCollector/controller/src/planner/{delta_cost_model,online_cost_model,pareto,tco,rules,baseline,cost_model}.rs`. Each impls `core::optimizer::cost::CostModel`. These stay deployment-model-specific because they encode DC's deployment assumptions (edge memory, WAN bandwidth, multi-tenant). -- **Topology**: `deployment-model-asaplifecycle/src/topology.rs` — uses `core::physical::topology::ThreeStage`; only deployment-model-specific bit is declaring which stage roles map to which OpAMP `AgentRole` enum values. -- **Emitters**: `deployment-model-asaplifecycle/src/emit/` — `OpAmpRemoteConfigEmitter` (per-role OTel YAML, lifted from `DataCollector/controller/src/config/generate_{agent,backend}_config*.rs`) + `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery::StreamingConfigEmitter` for the YAML bytes, then POSTs via `runtime::backend_client`). -- `impl DeploymentModel for ASAPLifecycleDeploymentModel` — registers HTTP routes `POST /plan` and `POST /replan`. - -**Work — runtime wiring:** -- **OpAMP server**: move `DataCollector/controller/src/opamp/` → `runtime/opamp/`. OpAMP is a transport, not a plan-type, so it lives in runtime. -- **Replanner + monitor**: move `DataCollector/controller/src/{replan,monitor}.rs` → `runtime/{replan,monitor}/`. -- **Dispatch**: `runtime/http/` routes `POST /plan` through `DeploymentModelRegistry` instead of calling DC's planner directly. - -**Integration test:** -- Send a `QuerySpec` over HTTP → core runs L1-3 → `ASAPLifecycleDeploymentModel`'s rule set fires (core rules + DC-specific rules) → `StageAllocator` with `ThreeStage` topology produces staged plan → emitters produce OTel YAML + `StreamingConfig` → fake backend asserts POST body matches golden file. Port DC's existing test of this shape. - -**Risk:** medium. Cost-model code (`delta/online/pareto/tco`) is ~40KB of subtle logic; port mechanically — do not touch the logic. The generic pieces already in core (stage allocator, bind rules) have been unit-tested in Phase 1, reducing this phase's surface area meaningfully. - -**Exit criteria:** DC controller's existing integration tests pass against `asap-controller` binary. SLA-driven replan fires on violation. OpAMP push to agents + backend POST both byte-identical to DC today. - -**PR size:** ~3500 LOC (down from the previous 5000 estimate, because Phase 1 now lifts L4's shared rule library + L5's `StageAllocator` framework). Cost models are ~1800 LOC of that 3500; rules + topology + emitters + integration tests make up the rest. - ---- - -### Phase 6 — Cutover: ASAPQuery-backend deletes its planner copy, DC deletes its controller dir - -**Scope:** Remove duplication. This phase is small code-wise but high blast radius. - -**Work:** -- **ASAPQuery-backend PR**: - - Delete `asap-planner-rs/` directory. - - Update workspace `Cargo.toml` to drop the member. - - Update `asap-quickstart/docker-compose-precompute.yml` to replace the `asap-planner-rs` init container with an `asap-controller plan` invocation. - - Update `asap-quickstart/Dockerfile.queryengine-local` if it ever references asap-planner-rs (likely it doesn't). - - Capability-miss callback in `SimpleEngine` continues to `POST /api/v1/plan` on the controller — no code change, same endpoint. -- **ASAPQuery PR**: delete `asap-planner-rs/` (the older copy). Any downstream refs in ASAPQuery update to point at ASAPController's `asap-query` binary. -- **DataCollector PR**: delete `controller/` directory. Anyone who ran `cargo build -p controller` now runs `cargo build -p asap-controller` against the new repo. Update DC's README to redirect. -- **asap-fusion PR**: convert the repo to a thin shim that re-exports `deployment-model-asapfusion` OR archive the repo outright if no external user depends on it. Recommendation: **archive**. - -**Risk:** medium. Deletion is final but reversible via `git revert`; actual blast radius is low because Phase 5 has already proven the new binary works. - -**Exit criteria:** -- `ASAPQuery-backend` workspace builds without `asap-planner-rs/`. -- Docker Compose starts successfully with the new init container. -- CI on all four repos passes. - -**PR size:** ~500 LOC across 3–4 PRs (all in source repos, not ASAPController). - ---- - -### Phase 7 — Core refactor: tighten traits now that 3 deployment models exist - -**Scope:** Look at the shape of the `Planner` / `Optimizer` / `PlanRewriteRule` traits with 3 real implementations, tighten anything awkward. - -**Work:** -- Look at any `todo!()` or `#[allow(unused)]` left over from Phase 1's trait stubs. -- Tighten associated types. -- Document the stable contract in `docs/design.md`. -- Publish `asap-ir` 0.1.0 to a private registry if external users want to depend on it. - -**Risk:** low — internal cleanup. - -**PR size:** ~500 LOC. - ---- - -## Timeline - -| Phase | PRs | Cumulative LOC | Est. wall time (1 engineer) | -|---|---|---|---| -| 0. Skeleton | 1 | ~200 | 1 day | -| 1. Core — L1-3 + L4/L5 framework + shared rules + `Source`/`DataModel` | 1–2 | ~6500 | 1.5 weeks | -| 2. Runtime HTTP+store | 1 | ~7700 | 3 days | -| 3. deployment-model-asapfusion (thin; rules fold into core library) | 1 | ~9500 | 1 week | -| 4. deployment-model-asapquery (L2 tree + L3/L4 split + YAML + CLI) | 1 | ~14000 | 2 weeks | -| 5. deployment-model-asaplifecycle (cost models + deployment-model-specific rules + OpAMP wiring) | 2 | ~17500 | 1 week | -| 6. Cutover (delete in source repos) | 3–4 | ~18000 | 1 week | -| 7. Core refactor | 1 | ~18500 | 3 days | - -**Total**: ~6.5 weeks for one engineer. Phase 1 grew (+0.5w) because it now also lifts L4/L5 shared infrastructure. Phases 3 and 5 shrank correspondingly (-1w combined) because fusion's `SketchConfigRule`-style bindings and lifecycle's `StageAllocator` now come from core's shared library. Net: slightly faster total, much cleaner per-deployment-model crates (each ~1500-2500 LOC of deployment-model-specific code rather than 5000+). - -Parallelisable if two people: lifecycle (Phase 5) can start as soon as Phase 1's L4/L5 framework lands (before Phase 4's L2 tree work). - -## Risk register - -| Risk | Likelihood | Impact | Mitigation | -|---|---|---|---| -| YAML emission byte-differences between old and new planner | Medium | Breaks ASAPQuery-backend's config consumer | Golden-file test corpus in Phase 4; fuzz harness over the YAML schema | -| OpAMP protocol detail drifts during port | Low | Breaks agent config push | Port opamp/ verbatim in Phase 2's runtime crate; don't touch until it's proven to work | -| DataFusion version skew in deployment-model-asapfusion | Low | Benchmarks drift | Pin DataFusion version in workspace `Cargo.toml`; measure benches before + after | -| Two asap-planner-rs copies have subtle diff we miss | Medium | Behaviour regression | Explicit diff + reconcile step in Phase 4; one authoritative copy (the backend's newer one) | -| ASAPQuery-backend deploys break during cutover | Low | Production outage | Phase 6 only after Phase 5 fully validates; staged rollout of docker-compose change | -| Scope creep — "while we're at it, redesign the cost model" | High | Doubles timeline | Explicit non-goal list in design doc §2; defer all rewrites to post-migration phases | -| Core traits are wrong; need to rev after deployment models land | Medium | Phase 7 PR larger than expected | Accept; Phase 7 is planned for exactly this. Don't over-design Phase 1. | -| PromQL L2 tree reverse-engineering loses a pattern-match case planner relied on | Medium | Generated YAML diverges from legacy | Golden-file corpus covers all 5 pattern shapes + generic fallthrough; tree-shape inspection at L3 lowering is a direct 1:1 translation of today's pattern-match logic, not a reinterpretation | -| L3/L4 split misses a case in `map_statistic_to_precompute_operator` | Medium | Wrong sketch selected for some `(Statistic, Treatment)` combos | Every branch of the current function must round-trip via the new L3→L4 path on the golden-file corpus; add table-driven unit tests enumerating every `Statistic × Treatment` combination | -| `AggIntent` as a 25-variant superset grows unbounded as deployment models add needs | Low | Core churn | Variants added only for intent shapes used by ≥1 shipped deployment model; never speculative. | - -## Decision log (during migration) - -These are the questions that will come up mid-migration. Pre-decide as many as possible: - -1. **Which version of `asap_types`?** ASAPQuery-backend's current one, pinned to a git SHA. -2. **Which `promql-parser` + `sqlparser` versions?** DC controller's (newer). ASAPQuery-backend's planner uses older versions; port it to the newer ones during Phase 4. -3. **How is `deployment-model-asaplifecycle`'s `StreamingConfig` YAML emitter related to `deployment-model-asapquery`'s?** One emitter in `deployment-model-asapquery`, called from `deployment-model-asaplifecycle`. Asymmetric dependency. -4. **Does `asap-controller` have feature flags to disable deployment models at build time?** Yes, one `--features lifecycle,query,fusion` with all three enabled by default. A minimal CLI binary (`asap-query`) disables lifecycle and fusion. -5. **Who owns the Cargo.lock?** ASAPController. Downstream repos (ASAPQuery-backend, DataCollector) do NOT depend on ASAPController as a Cargo path dep — they pull the published binary via Docker or pin a git SHA. -6. **Does `ControllerClient` on the ASAPQuery-backend side need changes?** No. It keeps POSTing to `/api/v1/plan`; the controller's routing layer dispatches to `deployment-model-asaplifecycle` (same as DC controller does today). -7. **L2 tree for asap-planner-rs — mandatory or optional?** Mandatory. planner's current template-catalogue approach is replaced with a proper `PromqlLogicalPlan` tree in Phase 4. The extra conformance work is taken to keep the 5-layer model uniform. See design doc §12 Q8 for the escape hatch if a future deployment model genuinely can't fit a tree. -8. **Sketch binding — L3 or L4?** L4. DC and fusion already do this correctly; planner's `map_statistic_to_precompute_operator` conflates L3+L4 and is split during Phase 4. `AggIntent` at L3 carries intent + accuracy only; concrete `AggregationType` / `SketchParams` are produced by an L4 rule. -9. **Intent vocabulary when deployment models disagree in width?** Core's `AggIntent` is the superset (DC's ~25 variants). Each deployment model only uses / produces / accepts the subset it needs (planner 9, fusion 3). Adding a new intent variant is a core change that deployment models opt into. -10. **Tabular vs time-series data models?** Handled in core from Phase 1 via the `Source` sum (`TimeSeries` / `Table` / `Join`) inside `QueryExpr::Scan` and the `DataModel` tag on `AggIntent`. ASAPQuery produces `Source::TimeSeries`, asap-fusion produces `Source::Table`. `Source::Table` is stubbed in Phase 1 and filled in by Phase 3 (fusion). Sketches themselves are data-model-agnostic; the generalization is purely at the IR leaf. - -## What a new deployment model looks like post-migration - -This is the test of the architecture. Adding a 4th deployment model (e.g. "edge-caching") — typical size ~500-2000 LOC: - -1. `cargo new --lib crates/deployment-model-edge-cache` (in ASAPController workspace) OR a new crate in your own repo that depends on `asap-ir = { git = "...", tag = "v0.1.0" }`. -2. Four files in `src/`: - - `lib.rs` — `impl DeploymentModel for EdgeCacheDeploymentModel` - - `rules.rs` — pick rules from `core::optimizer::rules::*` + add deployment-model-specific rules - - `topology.rs` — declare a `TopologyDescriptor` (e.g. `TwoStage { edge, backend }`) or pick a pre-baked core one - - `emit/edge_agent_config.rs` — `impl PlanEmitter` -3. Optionally: `cost.rs` if the deployment model needs its own cost model; otherwise use `core::optimizer::cost::*`. -4. If landing in ASAPController workspace: 1 line in `bin/asap-controller/main.rs` to register + 1 line in workspace `Cargo.toml`. If out-of-tree: publish as a binary in your own repo. - -**No changes** to runtime, core, or other deployment models. That's the extension-point test passing. - -### Why post-migration deployment models are much smaller than pre-migration code - -Pre-migration: each deployment model ships its own rule engine, cost traits, allocator, topology handling, sketch catalogue. Total per deployment model: 3000-8000 LOC. - -Post-migration: those all come from `core::optimizer::*` + `core::physical::*` as imports. Per-deployment-model code is only: -- Which rules to enable (usually a dozen lines — pick from `core::optimizer::rules`) -- Topology declaration (a dozen lines — often reusing a pre-baked `core::physical::topology::*`) -- DeploymentModel-specific rules that truly can't be shared (100-500 LOC each) -- DeploymentModel-specific cost models (if different from core's generic) -- Emitter(s) for the output format (100-1000 LOC) - -Typical post-migration deployment model: **500-2500 LOC**. Extremely easy to understand, review, and evolve. - -## Rollback strategy - -Each phase is a `git revert`-able PR. The three source repos (`DataCollector`, `ASAPQuery`, `ASAPQuery-backend`, `asap-fusion`) are NOT modified until Phase 6. If Phase 5 reveals the design is wrong, we can revert 0–4 and start over without disrupting any live service. - -Phase 6 is the only one-way door. Before landing it: -- At least 1 week of the new `asap-controller` binary running in staging. -- Golden-file corpus at 100% match rate for 1 week. -- SLA replan observed firing correctly at least once in staging. diff --git a/docs/promql-lowering.md b/docs/promql-lowering.md deleted file mode 100644 index 1f1f87fe..00000000 --- a/docs/promql-lowering.md +++ /dev/null @@ -1,439 +0,0 @@ -# PromQL lowering — positional `ColumnId`, the Binder, and CSE - -How `asap-frontend-promql` turns a PromQL string into the canonical Layer-3 -intent algebra, and *why* the IR uses positional column identity, an explicit -name-resolution pass, and unique-key metadata. - -This is the PromQL companion to [`design.md` §6](design.md) ("Core crate -details — Layer 3"). It mirrors the `asapquery-backend` control-plane IR: -**two IRs joined by a Binder.** - -## The pipeline at a glance - -``` -PromQL string - │ parse promql-parser (ProjectASAP fork, branch asap) - ▼ -Expr AST ← L1 - │ front-end lowering crates/frontend-promql/src/promql.rs - ▼ -relational::QueryExpr ← L2 — columns are NAMES (ColumnRef::Named, Aggregate.keys: Vec) - │ Binder pass: build Schema + crates/l2/src/binder.rs - │ resolve names → ColumnId + crates/l2/src/column_resolution.rs - ▼ -query_expr::QueryExpr ← L3 — columns are POSITIONS (Aggregate.by: Vec) - + a self-contained Schema rides on each Scan -``` - -The three layers are **L1 (`Expr AST`) → L2 (`relational::QueryExpr`, names) → -L3 (`query_expr::QueryExpr`, positions)**. The **Binder is not a layer** — it is -the pass that sits on the L2→L3 edge, turning names into positional `ColumnId`s. -`convert_root` (`crates/l2/src/lower.rs`) runs it first, converts the L2 tree -structurally, then runs the shared `canonicalize` pass (`crates/l2/src/canonicalize.rs`) -so both front ends emit one canonical form: - -```rust -pub fn convert_root(legacy: &LQueryExpr, accuracy: &AccuracyTarget) - -> Result -{ - let fallback = Binder::new().bind(legacy); // ← L2→L3 name resolution, once - let l3 = convert(legacy, &fallback, accuracy)?; // ← purely structural - Ok(canonicalize(l3)) // ← shared normalization (#34) -} -``` - -The worked example at the end traces a query through all three layers (with the -Binder pass shown explicitly between L2 and L3). - ---- - -## Why positional `ColumnId` - -`ColumnId = usize` (`schema.rs`), an index into `Schema::columns`. Everywhere -the *canonical* IR names a column — `Aggregate.by`, `Schema::unique_keys`, -`Schema::time_index` — it is a position, not a string. - -The IR is deliberately split in two: - -| | Layer-2 `relational::QueryExpr` | Canonical `query_expr::QueryExpr` | -|---|---|---| -| Column identity | `ColumnRef::Named(String)`, `Aggregate.keys: Vec` | `ColumnId = usize`, `Aggregate.by: Vec` | -| Source of names | whatever the PromQL parser emits | resolved against a `Schema` | - -Why convert names → positions at all: - -1. **Identity is settled once.** A string `"service"` means nothing until you - know which schema it lives in and at what offset. If every downstream pass - (schema flow, push-down, CSE, cost model, L5 emitters) carried names, each - would re-resolve and each would own a "column not found" failure path. A - `ColumnId` is an array index that **cannot dangle** — resolution already - happened. -2. **The canonical tree is self-describing.** The `Scan` node carries the - `Schema`, so any sub-tree's output schema is computable without surrounding - context (`QueryExpr::output_schema_in`). Positions index straight into it. -3. **It matches the backend wire format.** `ColumnId` is aliased to `usize` - specifically to line up with `design.md`'s `unique_keys: Vec>`. - The point of the restructure was convergence with the backend IR, not a - parallel L3. - -The named alias is kept (rather than a bare `usize`) so code can still pattern -on intent — *"this is a column position, not just any number."* - ---- - -## Why the Binder is its own pass - -`Binder::bind` (`binder.rs`) walks the L2 tree and returns **one** -self-contained `Schema { columns, time_index, unique_keys }` that every -`ColumnId` in the converted tree indexes into. It: - -1. Seeds columns from the catalog (`SchemaCatalog::columns_for`), or the - `(ts, value)` floor if the catalog knows nothing. -2. Guarantees that `(ts, value)` floor is present. -3. Appends one `Utf8` column per referenced-but-unknown name — collected from - `Aggregate.keys`, `TopK.by`, `Partition.keys` (`collect_referenced_columns`). - -Why isolate this instead of resolving inline during lowering: - -- **The converter becomes purely structural and total.** Once the schema - exists, positional resolution downstream can't fail to *find* a column. Every - failure mode (`ResolveError::NotFound`, `NoSampleValue`, `WildcardNotPositional`) - is concentrated in this one pass. -- **Schema/catalog policy is swappable without touching lowering.** The default - `UsageDerivedCatalog` knows nothing — honest for observability, where metric - label sets are open-ended. A registry-backed `SchemaCatalog` is future work, - and the Binder pass does not change when it lands — only the catalog impl - swaps. -- **It is the natural home for resolution-policy errors.** A usage-derived - schema can't enumerate a metric's full label set, so any operator needing the - *complement* of a label set defers to it. `without(labels)` (issue #39) is the - worked example: it can't list "all labels *except* these" at lowering time, so - the excluded labels are seeded + resolved positionally and the grouping is - carried as [`GroupKeys::without`](../crates/ir/src/intent_algebra/query_expr.rs) - — the kept (complement) set stays open and is resolved by the runtime, and the - aggregate's output schema stays open rather than freezing to closed. - -### Binary-op sides bind independently — with inherited keys - -Each side of a `BinaryOp` re-runs the Binder against its **own** sub-tree -(`convert_root` per side), because the two branches may scan different metrics -with different label sets and each side's columns must bind to positions in its -own leaf — not the other's. But a side must still see label names referenced by -an **enclosing** node: `sum by (__name__)(a or b)` (or `sum by (job)(a or b)`) -groups by a key that appears in *neither* side's own matchers, and every series -carries `__name__` (the metric name) regardless (issue #52). So the converter -seeds those **inherited** names — the enclosing scope's referenced columns minus -the ones referenced inside the binary op itself, i.e. exactly the ancestor keys — -into each side via `Binder::bind_with_inherited`. Subtracting the binary op's own -references is what keeps one side's labels from leaking into the other. - ---- - -## Why `unique_keys` / CSE - -`unique_keys: Vec>` (`schema.rs`). Each inner vec is a set of -column positions that *together* uniquely identify a row; the outer vec allows -several such sets. It is populated by the per-node output-schema rule — -`Aggregate { by, .. }` emits `unique_keys = [by-positions]` when `by` is -non-empty (`query_expr.rs`), most other nodes pass through. - -**What it is for: workload-level CSE.** When several queries are planned -together, `cse::dedupe_subtrees` hoists a shared sub-DAG into a `LetBinding` so -the cost model credits the producer once, with each root referencing it via -`Ref`. But sharing is only sound if the producer emits *the same rows* for every -consumer — and a unique key is exactly what proves that. - -`cse_reuse_is_legal` is the gatekeeper (`schema.rs`): - -```rust -pub fn cse_reuse_is_legal(producer_schema: &Schema, consumer_count: usize) - -> Result<(), CseError> -{ - if consumer_count < 2 { return Err(CseError::InsufficientConsumers(consumer_count)); } - if !producer_schema.has_unique_key() { return Err(CseError::NoUniqueKeys); } - Ok(()) -} -``` - -Why `unique_keys` rather than just deduping structurally-identical subtrees: -structural identity (`format!("{child:?}")`) tells you two consumers *want* the -same producer — it does **not** tell you the producer's output is *stable across -reads*. Without a provable unique key, two `Ref`s could observe different row -sets, and crediting the sharing would be unsound. Structural identity is the -candidate-finder; `unique_keys` is the correctness predicate. Expressing keys as -`ColumnId` sets is what lets the deduper assert this cheaply — another reason -positions exist. - -**Status:** this PR lands the scaffolding (`dedupe_subtrees`, -`cse_reuse_is_legal`, `Schema::unique_keys`, `LetBinding`/`Ref`). The cost-model -integration that makes it influence planning is tracked in **#6**. Single-query -plans never read `unique_keys`. - ---- - -## The nesting contract (issue #27) - -Nesting is **structural** at L3: `QueryExpr` is a recursive, box-owned tree and -every operator accepts an arbitrary child, so "function over subquery" needs no -special IR. The contract below says which *composite arguments* the front ends -actually lower today, and which are **cleanly rejected** (never silently -mislowered) — each row is pinned by a named test. - -### Lowers - -| Shape | Example | Pinned by | -|---|---|---| -| Aggregate over aggregate (any depth) | `max(sum by (job) (rate(m[5m])))` | `outer_aggregate_over_nested_aggregate_nests`; e2e `q27_max_over_sum_by_job_over_rate` | -| Aggregate over a binary op | `sum(rate(a[5m]) + rate(b[5m]))` | `aggregate_over_binary_op_nests` | -| Binary op over two arbitrary subtrees | `sum by (j)(rate(a[5m])) / sum by (j)(rate(b[5m]))` | e2e `q25_div_over_complex_subtrees` | -| Ranking over a nested aggregate (generic top-k) | `topk(3, sum by (i)(rate(m[5m])))` → `Sort → Limit` | `topk_over_nested_aggregate_is_generic_sort_limit` | -| Heavy-hitter top-k (frequency shape) | `topk(10, count_over_time(m[1m]))` → `AggIntent::TopK` | `topk_over_count_is_heavy_hitter` | -| Range function over a sub-query (#42) | `max_over_time(rate(m[5m])[1h:])` | `over_time_of_subquery_reduces_per_series` | -| Nested sub-queries (range fn over range fn over range fn; default resolution) | `max_over_time(deriv(rate(distance_covered_total[5s])[30s:5s])[10m:])` (Prometheus docs example) | `nested_subquery_from_prometheus_docs`; e2e `q27_nested_subquery_prometheus_docs_example` | -| Outer group key absent from a nested aggregate's (closed) output (#53) | `sum(sum by (k)(m)) by (j)` — `j` provably absent → dropped per PromQL's absent-label grouping semantics | `outer_group_key_absent_from_nested_aggregate_is_dropped`; e2e `q53_outer_group_key_absent_from_nested_aggregate` | -| Unary negation, anywhere in a nest (#36) | `sum(-m)` → `expr * -1` (`Mul` against `Scalar(-1)`); `-(const)` folds to a negated `Scalar` | `unary_negation_lowers_as_multiply_by_minus_one`; e2e `q36_sum_of_negation_nests` | -| `without(...)` grouping on a reducing aggregation (#39) | `sum without (instance) (…)` → `GroupKeys::without`; the complement stays open, deferred to runtime | `sum_without_groups_by_the_complement`; e2e `q39_sum_without_instance_over_rate` | -| SQL derived tables / inline views (#29) | `SELECT … FROM (SELECT … GROUP BY …) t`, incl. aggregate-over-aggregate | `derived_table_aggregate_over_aggregate_nests` (`sql_lowering.rs`) | - -### Rejected cleanly - -| Shape | Why | Pinned by | -|---|---|---| -| `without(...)` on `topk`/`bottomk`/`limitk` (#39) | only reducing aggregations model `without`; a ranking/sampling would need without-partitioning, so reject rather than silently treat as `by` | `without_on_topk_is_rejected` | -| SQL subquery-valued **predicate** expressions — scalar `x > (SELECT …)`, `IN (SELECT …)`, `EXISTS` / `NOT EXISTS` / `NOT IN`, correlated or not | the v1 decision on #27's open question: these need a subquery node in the L2 expression IR + a correlated-vs-uncorrelated representation choice; rejected until that lands (derived tables in `FROM` are the supported nesting shape) | `scalar_subquery_in_predicate_is_rejected`, `semi_join_is_rejected_not_mislowered`, `exists_subquery_in_predicate_is_rejected` (`sql_lowering.rs`) | - -The dividing line: **relational** nesting (an operator tree in `FROM` / a -function argument) lowers structurally; **expression-level** subqueries (a -query embedded inside a scalar predicate) are the one shape with no L2/L3 -representation yet, and they are the only nesting rejections that are not -tracked by a more specific issue. - ---- - -## Worked example — one query through L1 → L2 → L3 - -Four steps: the three layers, plus the Binder pass shown explicitly on the -L2→L3 edge. - -```promql -topk by (service) (10, count_over_time(requests{env="prod"}[1m])) -``` - -This is the heavy-hitter case (`topk` over `count` → one-pass sketch), exercised -by `topk_over_count_is_heavy_hitter_topk` in `crates/frontend-promql/tests/promql_lowering.rs`. - -### Stage 1 — L1 parse (`promql-parser`) - -``` -Expr::Aggregate { - op: topk, - param: NumberLiteral(10), - modifier: by (service), - expr: Expr::Call { - func: count_over_time, - args: [ Expr::MatrixSelector { vs: requests{env="prod"}, range: 1m } ], - }, -} -``` - -### Stage 2 — L2 relational IR (names) · `promql.rs` - -`walk_aggregate` resolves the group modifier to `keys = ["service"]`, lowers the -inner `count_over_time(...[1m])` to `Inner { metric: "requests", -matchers: [env=="prod"], window: 1m, func: Count }`, and — because the op is -`topk` *and* the inner func is `Count` — picks the heavy-hitter branch -(`Outer::TopK { k: 10, descending: true }` → `heavy_hitter == true`): - -``` -TopK { k: 10, by: ["service"], ← columns are still NAMES - input: Window { duration: 1m, slide: None, - input: Filter { pred: Compare { left: Column("env"), op: Eq, right: "prod" }, - input: Source(SourceSpec { name: "requests" }) } } } -``` - -No `Aggregate` wraps the scan — the heavy-hitter sketch counts directly off the -windowed scan (`window_scan`). Grouping rides as a *name list* on `TopK.by`, -awaiting resolution. - -### Stage 3 — Binder pass (L2→L3 edge): build the Schema, resolve names → `ColumnId` - -`Binder::bind` walks the L2 tree: - -- `source_name() == "requests"`; `UsageDerivedCatalog` returns `None` → start - from the `(ts, value)` floor. -- `collect_referenced_columns` finds `TopK.by = ["service"]` → append `service` - as a `Utf8` column. - -Result — the single self-contained schema: - -``` -Schema { - columns: [ ts:Timestamp(0), value:Float64(1), service:Utf8(2) ], - time_index: Some(0), - unique_keys: [], ← UsageDerivedCatalog proves no unique key -} -``` - -`resolve_named_keys(["service"], schema)` → `"service"` is at position 2 → -`by = [2]`. - -### Stage 4 — L3 canonical IR (positions) · `lower.rs convert` - -The `TopK` arm rewrites to the canonical `Aggregate{TopK}`, threading the -accuracy target and the resolved positional `by`; the `Filter`-over-`Source` -folds into `Scan.predicates`; the bound schema rides on the `Scan`: - -``` -Aggregate { - by: [2], ← service, POSITIONAL now - aggs: [ TopK { k: 10, accuracy: } ], - having: None, - child: Window { kind: Tumbling, size: 1m, slide: None, - child: Scan { - source: TimeSeries { metric: "requests" }, - predicates: [ Compare { left: Column("env"), op: Eq, right: "prod" } ], - schema: Schema { [ts, value, service], time_index: Some(0), unique_keys: [] }, - } } } -``` - -### Schema flow & the CSE gate on this tree - -`output_schema_in` for the top `Aggregate`: `by = [2]` is non-empty, so its -output schema is - -``` -Schema { - columns: [ service:Utf8, topk_10:Utf8 ], ← group key + TopK output - time_index: None, - unique_keys: [[0]], ← the group key is now a unique key -} -``` - -Now suppose a second query shared the same `Window → Scan` producer. The deduper -would propose hoisting it and call -`cse_reuse_is_legal(window.output_schema(), 2)`. The window passes the Scan's -schema through unchanged — and that schema's `unique_keys` is **empty** (the -`UsageDerivedCatalog` couldn't prove one). So the gate returns -`Err(NoUniqueKeys)` and the producer is **not** shared — each consumer -recomputes it. - -That refusal is the design working as intended: under the default catalog we -cannot assert that a raw windowed scan yields identical rows across reads, so we -decline to share rather than risk an unsound plan. A registry-backed -`SchemaCatalog` that declared, say, `(ts, service)` unique on `requests` would -populate `Scan.schema.unique_keys`, flip the gate green, and let the windowed -scan be hoisted into a `LetBinding` — without any change to the Binder or -converter (cf. the `dedupe_subtrees_basic` test, which constructs exactly such a -Scan). The next section traces exactly that. - ---- - -## `unique_keys` propagation - -`unique_keys` is **not** something the Binder computes — `Binder::bind` always -emits `unique_keys: Vec::new()`. It enters at the leaf (from the catalog) and is -then derived edge-by-edge by each operator's output-schema rule -(`QueryExpr::output_schema_in`). The rules: - -| Operator | `unique_keys` of its output | -|---|---| -| `Scan` | **verbatim** from the schema the Binder/catalog built | -| `Window`, `Filter`, `Partition`, `Sort`, `Limit`, `Subquery`, `Project` | **pass through** the child's unchanged | -| `Aggregate { by }` | **replaced** with `[[0..by.len()]]` — the group keys, *re-based to output positions*; empty when `by` is empty | -| `Distinct { cols }` | child's keys **plus** `cols` added as a new key (`add_unique_key`) | -| `Merge` | first child's | -| `SetOp`, `Join`, `BinaryOp` | left / `lhs` child's | - -Two rules carry the weight: leaf-bearing operators **pass keys through** untouched, -while `Aggregate` **manufactures** a key — grouping by a column makes that column -unique in the result, so it becomes the new key (and the input's keys are -dropped, because the grouped output no longer has those rows). - -### Same tree, under a registry catalog - -Take the worked example's canonical tree, but bind it with a `SchemaCatalog` that -declares `(ts, service)` unique on `requests`. Now the leaf schema arrives with a -key, and we can watch it flow up (bottom → top): - -``` - ── unique_keys on this edge ── -Scan { requests, [[0, 2]] ← from catalog - schema: [ts(0), value(1), service(2)], (ts, service) - unique_keys = [[0, 2]] } - ▲ -Window { 1m } [[0, 2]] ← pass-through - ▲ (time_index present) -Aggregate { by:[2]=service, aggs:[TopK{10}] } [[0]] ← REPLACED - output cols: [service(0), topk_10(1)] by re-based to - output position 0 -``` - -Three things to read off this: - -1. **Scan** hands up the catalog's `[[0, 2]]` verbatim. -2. **Window** (and any `Filter`/`Sort`/`Limit` between) passes `[[0, 2]]` straight - through — these operators don't change which rows are distinct. -3. **Aggregate** does *not* forward `[[0, 2]]`. After `GROUP BY service`, the old - per-sample identity is gone; what's unique now is `service` itself — and in the - output schema `service` sits at **position 0**, so the derived key is `[[0]]`, - not `[[2]]`. This re-basing is why keys are positional `ColumnId`s, not names: - the same column is id `2` below the aggregate and id `0` above it. - -### `ColumnId` is relative to a schema — the `2` vs `0` - -A `ColumnId` is a position *within one schema*. The input and output edges of -`Aggregate` are **different schemas**, so the *same* logical `service` column -gets a different id on each. Watch it with sample rows. - -**Input edge** (below the aggregate) — `service` is column **2**: - -| `ts` · id 0 | `value` · id 1 | `service` · id 2 | -|---|---|---| -| 100 | 0.5 | api | -| 100 | 0.3 | web | -| 200 | 0.7 | api | -| 200 | 0.4 | web | - -→ "group by `service`" is written `by = [2]`. And every `(ts, service)` combo -occurs once, so the edge carries `unique_keys = [[0, 2]]`. - -**Output edge** (above the aggregate) — a *brand-new* table; `service` is now -column **0**: - -| `service` · id 0 | `topk_10` · id 1 | -|---|---| -| api | … | -| web | … | - -→ after grouping, each `service` appears exactly once, so `service` *alone* is -the key — at its **new** position: `unique_keys = [[0]]`. - -So `2` and `0` both name `service`; they differ only because input and output are -different schemas. This is also the cleanest way to see how the two concepts -divide up: - -- **`ColumnId`** answers *"which column"* — a pointer (`by = [2]`: group by the - column at position 2). Used everywhere a column must be named. -- **`unique_keys`** answers *"which set(s) of columns are jointly non-duplicating"* - — a fact about the data, *written using* `ColumnId`s (`[[0, 2]]`: columns 0 and - 2 together identify a row). Read only by the CSE gate. - -The outer `Vec` allows several such sets: `[[0, 2]]` = one key (the pair); -`[[0], [1, 2]]` = two independent keys. - -### Why this makes CSE legal here - -The shared producer the deduper would hoist is the `Window → Scan` sub-tree. Its -output edge now carries `unique_keys = [[0, 2]]`, so: - -``` -cse_reuse_is_legal( window.output_schema() // unique_keys = [[0, 2]] - , 2 /* consumers */ ) ==> Ok(()) -``` - -The gate fires green, the `Window → Scan` is hoisted into a `LetBinding`, and both -queries `Ref` it — scan + window computed once. Under the default -`UsageDerivedCatalog` the very same tree carries `unique_keys = []` at every edge -(nothing manufactures a key below the top `Aggregate`), so the gate returns -`Err(NoUniqueKeys)` and each consumer recomputes. **The only thing that changed -was the leaf key the catalog supplied; propagation and the gate did the rest.**