Skip to content

refactor(legacy_expr): step β — thread Schema through legacy planning stack - #139

Merged
zzylol merged 1 commit into
mainfrom
refactor/legacy-expr-step-beta-schema-threading
May 12, 2026
Merged

zzylol merged 1 commit into
mainfrom
refactor/legacy-expr-step-beta-schema-threading

Conversation

@zzylol

@zzylol zzylol commented May 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Plumbing-only refactor. After Step α (PR #138) gave us canonical AggIntent inside legacy QueryExpr variants, the next blocker for variant-by-variant consumer migration (Step γ) is that consumers can't resolve ColumnRef::Named(\"host\")ColumnId(2) without a Schema.

Step β threads Schema through every consumer-side walker via traversal-environment parameter (approach c — lightest touch; mirrors how canonical QueryExpr already passes schema downward).

What landed

New: controller/src/intent_algebra/column_resolution.rs (256 lines, 8 tests)

pub fn resolve_column_ref(col: &ColumnRef, schema: &Schema)
    -> Result<ColumnId, ResolveError>;
pub fn resolve_column_refs(cols: &[ColumnRef], schema: &Schema)
    -> Result<Vec<ColumnId>, ResolveError>;

pub fn infer_schema_for_root(expr: &QueryExpr) -> Schema;
pub fn infer_source_schema(metric_name: &str) -> Schema;

infer_source_schema synthesises the default time-series shape (ts: Timestamp, value: Float64) per the deferred SchemaCatalog decision. infer_schema_for_root walks to the outermost Source leaf via existing QueryExpr::source_name() and applies the default; falls back to Schema::default() for bare Ref(_) trees.

Threading applied to

  • optimizer::engine::RewriteRule::try_rewrite — trait + 12 impls
  • optimizer::engine::QueryOptimizer::{apply_all, recurse_children, apply_rules_at}
  • physical::allocator::SketchAllocator::{alloc_node, alloc_sketch_agg}
  • physical::planner::plan_node
  • physical::stage_split::{walk, promql_from_qe}
  • query_parser::mod::QeCollector::visit
  • intent_algebra::legacy_lower::lower_to_sketch_algebra_with_schema (new sibling; old fn delegates after deriving schema)

Public entry points (SketchAllocator::allocate, plan, split_expr_by_stage, expr_to_promql, lower_to_sketch_algebra, qe_to_parsed_query, QueryOptimizer::optimize) keep existing signatures — each derives the root schema and threads it inward. Backwards compatibility intact.

Build + test

  • cargo build --release -p controller clean
  • cargo build --release -p query_engine_rust clean
  • cargo test --release -p controller --lib674 passed (was 666 after Step α; +8 new column_resolution tests)
  • cargo test --release -p controller --bin controller — 27 passed

7 TODOs flagged for Step γ

  1. Aggregate { keys: Vec<String> }Vec<ColumnId> migration — needs schema transformation as we descend into Aggregate's child
  2. Distinct { cols } consumer in physical::stage_split currently logs ColumnIds without using them — BackendSubPlan needs a distinct_cols: Vec<ColumnId> field
  3. Project { cols } reshapes columns canonically; legacy passes parent schema through verbatim
  4. LetBinding / Ref need a BindingScope analogue plumbed alongside Schema
  5. Label sets (*labels) aren't representable in canonical Schema today — needs a DataType::LabelSet or parallel side-channel before full Source → Scan migration
  6. CostModel::estimate(&self, expr: &QueryExpr) doesn't take Schema (inspects only outermost node). Future cost estimators may need the signature extended
  7. optimizer::engine::HydraConversion (disabled in Step α) could now consult the schema to emit canonical Aggregate { by, aggs: [inner] } instead of inlining into SketchAgg.op

Diff: 8 files, +579 / -151

🤖 Generated with Claude Code

… stack

Plumbing-only refactor. After Step α (PR #138) gave us canonical
AggIntent inside legacy QueryExpr variants, the next blocker for
variant-by-variant consumer migration (Step γ) is that consumers can't
resolve `ColumnRef::Named("host")` → `ColumnId(2)` without a Schema.

Step β threads Schema through every consumer-side walker via approach
(c) — traversal-environment parameter (lightest touch; mirrors how
canonical QueryExpr already passes schema downward).

## What landed

### New: `controller/src/intent_algebra/column_resolution.rs` (256 lines, 8 tests)

```rust
pub fn resolve_column_ref(col: &ColumnRef, schema: &Schema)
    -> Result<ColumnId, ResolveError>;
pub fn resolve_column_refs(cols: &[ColumnRef], schema: &Schema)
    -> Result<Vec<ColumnId>, ResolveError>;

pub fn infer_schema_for_root(expr: &QueryExpr) -> Schema;
pub fn infer_source_schema(metric_name: &str) -> Schema;
```

`ResolveError` enumerates `NotFound { name, available }`,
`NoSampleValue { available }`, `WildcardNotPositional`.

`infer_source_schema` synthesises the default time-series shape:
- `Column { name: "ts", dtype: Timestamp, nullable: false }` at index 0
- `Column { name: "value", dtype: Float64, nullable: false }` at index 1
- Conservative `unique_keys: vec![]` (no catalog to assert primary key)

`infer_schema_for_root` walks to the outermost `Source` leaf via the
existing `QueryExpr::source_name()` helper and applies
`infer_source_schema`; falls back to `Schema::default()` for trees
rooted at a bare `Ref(_)`.

### Functions/methods that gained a `parent_schema: &Schema` parameter

- `optimizer::engine::RewriteRule::try_rewrite` — trait + 12 impls
- `optimizer::engine::QueryOptimizer::{apply_all, recurse_children, apply_rules_at}`
- `physical::allocator::SketchAllocator::{alloc_node, alloc_sketch_agg}`
- `physical::planner::plan_node` (free fn)
- `physical::stage_split::{walk, promql_from_qe}` (free fns)
- `query_parser::mod::QeCollector::visit`
- `intent_algebra::legacy_lower::lower_to_sketch_algebra_with_schema`
  (new sibling — old `lower_to_sketch_algebra` now delegates after
  deriving schema via `infer_schema_for_root`)

Public entry points (`SketchAllocator::allocate`, `plan`,
`split_expr_by_stage`, `expr_to_promql`, `lower_to_sketch_algebra`,
`qe_to_parsed_query`, `QueryOptimizer::optimize`) keep their existing
signatures; each derives the root schema and threads it inward.
Backwards compatibility intact.

### Usage proof (physical/stage_split.rs:Distinct arm)

```rust
QueryExpr::Distinct { cols, input } => {
    plan.backend.has_dedup = true;
    for c in cols {
        if let ColumnRef::Named(_) = c {
            match resolve_column_ref(c, parent_schema) {
                Ok(_id) => { /* Step γ TODO: distinct_cols: Vec<ColumnId> */ }
                Err(e) => plan.deferral_log.push(format!(
                    "Distinct col resolution deferred to Step γ: {e}")),
            }
        }
    }
    walk(input, plan, budgets, parent_schema);
}
```

## Build + test

- `cargo build --release -p controller` — clean
- `cargo build --release -p query_engine_rust` — clean (pre-existing
  `e2e_modified_otlp_sketch_path` test compile error in
  `query_engine_rust --lib` was already on main; out of scope)
- `cargo test --release -p controller --lib` — **674 passed**
  (was 666 after Step α; +8 new column_resolution tests)
- `cargo test --release -p controller --bin controller` — 27 passed

## 7 TODOs flagged for Step γ

1. `Aggregate { keys: Vec<String> }` → `Vec<ColumnId>` — needs schema
   transformation as we descend into Aggregate's child
2. `Distinct { cols }` consumer in `physical::stage_split` logs
   `ColumnId`s instead of using them — `BackendSubPlan` needs a
   `distinct_cols: Vec<ColumnId>` field
3. `Project { cols }` reshapes columns canonically; legacy passes
   parent schema through verbatim
4. `LetBinding` / `Ref` don't carry a `BindingScope` analogue in
   legacy; Step γ needs to plumb a scope map alongside Schema
5. Label sets (`*labels` open-set) aren't representable in canonical
   `Schema` today — needs a `DataType::LabelSet` or parallel
   side-channel before full Source→Scan migration
6. `CostModel::estimate(&self, expr: &QueryExpr)` doesn't take Schema
   (inspects only outermost node). Future cost estimators may need
   the signature extended
7. `optimizer::engine::HydraConversion` (disabled in Step α) could
   now consult the schema to emit canonical
   `Aggregate { by, aggs: [inner] }` instead of inlining into
   `SketchAgg.op`

## Diff: 8 files, +579 / -151 (256-line new file + 7 modified)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant