feat: general query algebra — full SQL/PromQL AST, cost optimizer, sketch allocator

[zzylol]

Summary

Closes #95.

This PR introduces controller/src/algebra/ — a fully general query algebra that can express all SQL and PromQL queries as an AST, with cost-based rewrite rules and per-stage sketch allocation.

Design answers (open questions from issue #95)

Question	Answer
ExactAgg(Sum/Count/Min/Max)	Mergeable → Backend stage
ExactAgg(Avg)	NOT mergeable → Db stage (avg-of-avgs is wrong)
Stage budgets	StageResourceBudgets struct (new in types.rs), sourced from WorkloadCharacteristics.memory_budget_bytes; all fields Option<u64> so None = unlimited
Query format	expr_to_promql() outputs valid PromQL; sketch insertion from OTel → Precompute is a separate concern

---

New module: controller/src/algebra/

expr.rs — Full QueryExpr + ScalarExpr algebra

QueryExpr (relational): 22 variants covering the entire SQL + PromQL surface:

• Base: Source, Ref
• Filtering/projection: Filter, Project
• Aggregation: Aggregate (multi-function), SketchAgg (single sketch op)
• Streaming: Window (tumbling/sliding), Partition, Dedup, TopK, Merge
• Joins: Join (7 kinds), JoinSketch (sketch push-down)
• Set ops: SetOp (Union/Intersect/Except with ALL)
• Ordering: Sort, Limit
• Scoping: Subquery, LetBinding (CTEs / WITH)
• Analytic: WindowFunc (RANK/LAG/LEAD/…) with ROWS/RANGE/GROUPS frames
• PromQL-specific: HistogramQuantile, PromQLSubquery ([range:resolution]), BinaryOp with VectorMatch (on/ignoring/group_left/group_right)

ScalarExpr (row-level): 13 variants — Column, Literal, BinaryOp, UnaryOp, FunctionCall, ScalarSubquery, InList, InSubquery, Between, IsNull, Case, Cast, VectorBinaryOp

AggFunc: Count/Sum/Avg/Min/Max/StdDev/Variance/Quantile(φ)/CountDistinct/HeavyHitters{k}/Rate/Increase/Delta/Custom

Bridge: QueryExpr::from_sketch_expr() converts the existing SketchExpr tree, preserving full backward compatibility with all existing SQL/PromQL parsers.

plan.rs — Annotated plan nodes

• PipelineStage: Agent < Backend < Precompute < Db (total order)
• ExecutionMode: Sketch | Exact | Passthrough
• CostEstimate: bytes/sec, memory_bytes, cpu_micros/sample, compression_ratio
• NodeAnnotation: sketch_type, sketch_params, delta_enabled, delta_threshold, rationale, budget_demotion flag
• PlanNode: wraps a QueryExpr with stage/mode/cost/annotation + children
• PlanSummary: serialisable JSON summary (bandwidth saved, agent/backend memory, per-node annotations) — suitable for inclusion in /api/v1/plan response

optimizer.rs — Cost-based fixed-point optimizer

12 rewrite rules applied until fixed point (max 32 iterations):

Rule	Transformation
R1 PredicatePushDown	Filter(Window(e)) → Window(Filter(e))
R2 MergeLifting	SketchAgg(Merge([a,b])) → Merge([SketchAgg(a), SketchAgg(b)]) for mergeable ops
R3 HLLDedupElim	HLL(Dedup(e)) → HLL(e) (HLL is intrinsically distinct)
R4 FilterWindowSwap	redundant with R1, attributed separately for logging
R5 TopKFusion	Limit(n, Sort([c DESC], e)) → TopK(n, [c], e)
R6 HistogramQuantileFusion	ensures φ is in DDSketch quantile list
R7 SubqueryDecorrelation	hoists correlated ScalarSubquery into LetBinding
R8 CommonSubexprElim	extracts repeated Source nodes into LetBindings
R9 HydraConversion	Partition(≥2 keys, SketchAgg(op)) → SketchAgg(Hydra{op, keys}) when cost improves
R10 WindowMerge	Window(d, Window(d, e)) → Window(d, e)
R11 PartitionElim	Partition(By([]), e) → e
R12 SetOpFusion	Union(Merge(a,b), Merge(c)) → Merge(a,b,c)

allocator.rs — SketchAllocator

Converts QueryExpr → PlanNode with budget-driven stage demotion:

Node type	Default stage	Demotion rule
Source, Filter, Window, Partition, Dedup	Agent	—
SketchAgg (sketchable, mergeable)	Agent	Agent budget exceeded → Backend → Precompute
SketchAgg(Exact(Avg))	Db	never (non-mergeable)
SketchAgg(Exact(Sum/Count/Min/Max))	Backend	—
TopK	Precompute	— (CountSketch)
Merge, JoinSketch	Backend	—
Aggregate, Project, Sort, Limit, Join, SetOp, WindowFunc	Db	—
HistogramQuantile, PromQLSubquery, BinaryOp	Precompute (if child is sketch)	else Db
Subquery, LetBinding	body's stage	propagated

---

Known gaps

Gap	Detail	Severity
Actual parsers don't emit QueryExpr yet	The existing sql.rs and promql.rs parsers still emit SketchExpr. QueryExpr::from_sketch_expr() bridges that, but a direct QueryExpr parser isn't written — so HistogramQuantile, PromQLSubquery, and WindowFunc nodes are unreachable from a real query string today.	Medium — follow-up task
PromQL offset / @ modifiers	No offset or timestamp-pin field on Window / Source	Minor
*_over_time PromQL functions	No dedicated AggFunc variants; falls back to Custom(String)	Minor
LATERAL joins / GROUPING SETS	Not in JoinKind / Aggregate	Minor

See issue #109 for the full coverage matrix.

---

Test coverage

82 new passing tests across all four modules (expr / plan / optimizer / allocator), covering:

• Full SketchExpr → QueryExpr bridge round-trips
• All 12 optimizer rules (positive + negative cases)
• Budget-driven demotion: Agent → Backend → Precompute
• Non-mergeable Exact(Avg) always lands at Db
• PlanSummary bandwidth savings and budget-demotion detection
• Memory estimate formulas for all sketch types

Test plan

• cargo build — clean
• cargo test algebra — 82/82 pass
• Integration: wire SketchAllocator into handle_plan endpoint (follow-up)
• Integration: expose PlanSummary in /api/v1/plan JSON response (follow-up)
• Update promql.rs and sql.rs to emit QueryExpr directly (tracked in algebra: QueryExpr coverage — what SQL/PromQL is supported and what is not #109)

🤖 Generated with Claude Code