refactor(intent_algebra): retire SketchAgg/WindowedAgg — legacy_expr is a pure L2 IR

control_plane/docs/design.md

@@ -44,8 +44,8 @@ DataCollector/controller already documents its query→sketch translation as a 5
 | # | Layer | What it does | Today's locations |
 |---|-------|--------------|-------------------|
 | 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql}.rs` + the per-language façade `controller/src/query_parser/language/{promql,sql,elastic_dsl}/`; asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream). *Refactor 2026-05 absorbed `controller/src/query_language/` into `query_parser::language/`.* |
-| 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/language_logical_plan/{lower,plan}.rs` + `controller/src/intent_algebra/legacy_lower.rs` (legacy L2→L3 lowering pending unification with `intent_algebra/lower.rs`); asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** |
-| 3 | **Intent algebra** | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/intent_algebra/{agg_intent,query_expr,schema,lower,cse}.rs` (canonical L3) + `controller/src/intent_algebra/legacy_expr.rs` (legacy heavy-IR pending dedup with the canonical types); asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4**. *Refactor 2026-05 absorbed `controller/src/algebra/expr.rs` into `intent_algebra/legacy_expr.rs`.* |
+| 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/language_logical_plan/{lower,plan}.rs` + `controller/src/intent_algebra/legacy_expr.rs` (the legacy L2 relational IR the `query_parser` front ends emit); asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** |
+| 3 | **Intent algebra** | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/intent_algebra/{agg_intent,query_expr,schema,lower,cse}.rs` (canonical L3) + `controller/src/intent_algebra/legacy_to_canonical.rs` (the L2→L3 converter: folds the legacy `legacy_expr` relational tree straight to the canonical IR, single-statistic sketchable `Aggregate` fusion included — no intermediate fused L3 IR); asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4**. *Refactor 2026-05 absorbed `controller/src/algebra/expr.rs` into `intent_algebra/legacy_expr.rs`.* |
 | 4 | **Sketch algebra + optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 (`QueryExpr`) and emit the sketch-bound IR (`SketchExpr`). ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library + the sketch-bound IR `core::sketch_algebra::SketchExpr`; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/sketch_algebra/` (IR + binding rules) + `controller/src/optimizer/{engine,trait_def,baseline,rules/,cost/}.rs` (rule engine + cost models); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator`. *Refactor 2026-05 absorbed `controller/src/algebra/optimizer.rs` (→ `optimizer/engine.rs`) and `controller/src/planner/{cost_model,delta_cost_model,online_cost_model,pareto,tco,wire_cost,rules,baseline_planner}.rs` (→ `optimizer/{cost/,rules/,baseline.rs}`).* |
 | 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; deployment models supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/physical/{allocator,planner,plan,sketch_catalog,stage_split,topology,colored_dag/}.rs` (allocator framework + sketch catalogue + typed three-stage colouring) + `controller/src/emit/{stage_config,otap,telegraf,agent,backend,asapquery_backend,precompute,trait_def}.rs` (per-deployment-model emitters + `PlanEmitter` trait) + `controller/src/pipeline.rs` (L1→…→L5 driver, formerly `analyzer.rs`); asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/`. *Refactor 2026-05 absorbed `controller/src/algebra/{physical,allocator,plan,directory}.rs` and `controller/src/planner/stage_split.rs` and the legacy `controller/src/stage_split/` framework into `physical/`; absorbed `controller/src/config/` into `emit/` + `workload.rs`; renamed `analyzer.rs` to `pipeline.rs`.* |
 
@@ -169,7 +169,7 @@ If something must be optional (e.g. OpAMP for deployments that don't run OTel co
 > |---|---|
 > | `crates/core/query_language/` | `controller/src/query_parser/language/` |
 > | `crates/core/logical_plan/` | `controller/src/language_logical_plan/` |
-> | `crates/core/intent_algebra/` | `controller/src/intent_algebra/` (with `legacy_expr` / `legacy_lower` carrying the older `algebra::expr`-flavored IR pending unification) |
+> | `crates/core/intent_algebra/` | `controller/src/intent_algebra/` (with `legacy_expr` carrying the legacy L2 relational IR and `legacy_to_canonical` converting it to the canonical L3 types) |
 > | `crates/core/sketch_algebra/` | `controller/src/sketch_algebra/` |
 > | `crates/core/optimizer/{engine,trait,rules,cost}/` | `controller/src/optimizer/{engine.rs,trait_def.rs,rules/,cost/,baseline.rs}` |
 > | `crates/core/physical/{planner_trait,stage_allocator,topology,executor,sketch_catalog}/` | `controller/src/physical/{planner,allocator,plan,stage_split,sketch_catalog,topology,colored_dag/}.rs` |
@@ -289,7 +289,7 @@ Core is not a trait-stubs library. It ships real L1/L2/L3 code lifted from DC's
 > | `core::logical_plan` | `controller/src/language_logical_plan/` |
 > | `core::intent_algebra` | `controller/src/intent_algebra/` (canonical) + `controller/src/intent_algebra/legacy_{expr,lower}.rs` (legacy IR pending unification) |
 > | `core::sketch_algebra` | `controller/src/sketch_algebra/` |
-> | `core::lower` | per-layer: `controller/src/language_logical_plan/lower.rs` + `controller/src/intent_algebra/{lower,legacy_lower}.rs` + `controller/src/sketch_algebra/lower.rs` |
+> | `core::lower` | per-layer: `controller/src/language_logical_plan/lower.rs` + `controller/src/intent_algebra/{lower,legacy_to_canonical}.rs` + `controller/src/sketch_algebra/lower.rs` |
 > | `core::optimizer::engine` | `controller/src/optimizer/engine.rs` |
 > | `core::optimizer::trait` | `controller/src/optimizer/trait_def.rs` (placeholder) |
 > | `core::optimizer::rules` | `controller/src/optimizer/rules/` |
@@ -1858,7 +1858,7 @@ The pre-refactor layout grew organically as the controller absorbed three legacy
 | Retired path | Replacement |
 |---|---|
 | `controller/src/algebra/expr.rs` | `controller/src/intent_algebra/legacy_expr.rs` |
-| `controller/src/algebra/lower.rs` | `controller/src/intent_algebra/legacy_lower.rs` |
+| `controller/src/algebra/lower.rs` | `controller/src/intent_algebra/legacy_to_canonical.rs` (the L2→L3 lowering, folded into the legacy→canonical converter) |
 | `controller/src/algebra/directory.rs` | `controller/src/physical/sketch_catalog.rs` |
 | `controller/src/algebra/physical.rs` | `controller/src/physical/planner.rs` |
 | `controller/src/algebra/allocator.rs` | `controller/src/physical/allocator.rs` |
@@ -1885,6 +1885,6 @@ The pre-refactor layout grew organically as the controller absorbed three legacy
 **TODOs left from the refactor:**
 
 - `controller/src/emit/stage_config.rs` (3,020 lines, formerly `config/stage_config.rs`) was moved whole rather than split into `emit/opamp.rs` + `emit/streaming_config.rs` + `emit/inference_config.rs` per design.md §5. The monolith mixes OTel-collector YAML emit, ASAPQuery-backend JSON emit, storage-routing JSON emit, and shared internals; a clean split needs ownership reorganisation, not file renames. Tracked for a follow-up.
-- `controller/src/intent_algebra/{legacy_expr,legacy_lower}.rs` carry the older `algebra::expr`-flavored `QueryExpr` / `AggIntent` / `WindowSpec` IR alongside the canonical `intent_algebra::{query_expr,agg_intent,lower}` types. Most of the controller still consumes the legacy types (query_parser, language_logical_plan, physical/, optimizer/, emit/); migrating each call site onto the canonical types is a follow-up.
+- `controller/src/intent_algebra/legacy_expr.rs` carries the legacy **L2 relational** `QueryExpr` the `query_parser` front ends emit; `legacy_to_canonical.rs` is the single pass that converts it (sketch-fusion folded in) to the canonical `intent_algebra::{query_expr,agg_intent}` L3 types. The sketch-fused `SketchAgg` / `WindowedAgg` legacy variants and the standalone `legacy_lower` pass have been retired. Several controller modules (query_parser, language_logical_plan, physical/, optimizer/, emit/) still reference `legacy_expr` for shared leaf types; fully deleting the legacy tree is a follow-up gated on the canonical `Predicate` covering the remaining `ScalarExpr` variants.
 - `controller/src/optimizer/trait_def.rs` (`OptimizerRule`), `controller/src/emit/trait_def.rs` (`PlanEmitter`), `controller/src/deployment_model.rs` (`DeploymentModelRegistry`) ship as placeholders — the existing free-function emitters and concrete rule loops still drive behaviour. Migrating them onto the trait surfaces lands when the per-deployment-model crate split lands.
 

control_plane/src/intent_algebra/binder.rs

@@ -50,7 +50,7 @@
 //! legacy IR is retired it moves into the `core::lower` L1→L2→L3 passes
 //! proper (the `lower_*(ast, schema)` signatures in design.md §6).
 
-use crate::intent_algebra::legacy_expr::{ColumnRef, QueryExpr as LQueryExpr};
+use crate::intent_algebra::legacy_expr::QueryExpr as LQueryExpr;
 use crate::intent_algebra::schema::{Column, DataType, Schema};
 
 /// The DB / source-schema metadata source from design.md §6 "three
@@ -166,19 +166,17 @@ fn default_leaf_columns() -> Vec<Column> {
     ]
 }
 
-/// Walk the legacy tree and collect every distinct column / group-key
-/// name the legacy → canonical converter resolves positionally:
-/// `SketchAgg.col` / `WindowedAgg.col` (when `Named`), `Aggregate.keys`,
+/// Walk the legacy tree and collect every distinct group-key name the
+/// legacy → canonical converter resolves positionally: `Aggregate.keys`,
 /// `TopK.by`, and `Partition.keys`. Sorted + de-duplicated for a stable,
 /// deterministic column order.
+///
+/// `AggItem.col` (the statistic's *input* column) is deliberately not
+/// collected — the converter never resolves it positionally; it only
+/// ever resolves group-by keys.
 fn collect_referenced_columns(tree: &LQueryExpr) -> Vec<String> {
     let mut out: Vec<String> = Vec::new();
     tree.walk(&mut |node| match node {
-        LQueryExpr::SketchAgg { col, .. } | LQueryExpr::WindowedAgg { col, .. } => {
-            if let ColumnRef::Named(name) = col {
-                out.push(name.clone());
-            }
-        }
         LQueryExpr::Aggregate { keys, .. } => out.extend(keys.iter().cloned()),
         LQueryExpr::TopK { by, .. } => out.extend(by.iter().cloned()),
         LQueryExpr::Partition { keys, .. } => out.extend(keys.keys().iter().cloned()),
@@ -194,7 +192,6 @@ fn collect_referenced_columns(tree: &LQueryExpr) -> Vec<String> {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::intent_algebra::agg_intent::AggIntent;
     use crate::intent_algebra::legacy_expr::{
         AggFunc, AggItem, ColumnRef as LColumnRef, PartitionKeys, QueryExpr as LQueryExpr,
         SourceSpec,
@@ -213,22 +210,6 @@ mod tests {
         assert_eq!(schema.time_index, Some(0));
     }
 
-    #[test]
-    fn sketch_agg_named_col_lands_in_schema() {
-        // SketchAgg { col: Named("price") } over Source — "price" must be
-        // resolvable, i.e. present in the bound schema.
-        let tree = LQueryExpr::SketchAgg {
-            op: AggIntent::Sum,
-            col: LColumnRef::Named("price".into()),
-            input: Box::new(src("trades")),
-        };
-        let schema = Binder::new().bind(&tree);
-        assert!(schema.column_id("price").is_some(), "price should be bound: {schema:?}");
-        // floor still there.
-        assert!(schema.column_id("ts").is_some());
-        assert!(schema.column_id("value").is_some());
-    }
-
     #[test]
     fn aggregate_keys_and_partition_keys_land_in_schema() {
         // Aggregate { keys: ["region"] } and Partition { By(["host"]) }.
@@ -309,11 +290,7 @@ mod tests {
                 }
             }
         }
-        let tree = LQueryExpr::SketchAgg {
-            op: AggIntent::Sum,
-            col: LColumnRef::Named("datacenter".into()),
-            input: Box::new(src("known_metric")),
-        };
+        let tree = src("known_metric");
         let schema = Binder::with_catalog(FixedCatalog).bind(&tree);
         // `datacenter` came from the catalog, not usage-synthesis — and it
         // is non-nullable, unlike a usage-derived column.