Skip to content

feat: sketch capability model + per-stage budgets + optimization problem formulation - #119

Merged
zzylol merged 2 commits into
mainfrom
pr/window-optimizer-gaps
Apr 4, 2026
Merged

zzylol merged 2 commits into
mainfrom
pr/window-optimizer-gaps

Conversation

@zzylol

@zzylol zzylol commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Wires the existing DeploymentConstraints into the actual optimizer pipeline.
Previously DeploymentConstraints and with_constraints() were defined but
never called — the optimizer ran unconstrained.

Changes

controller/src/main.rs

  • Both handle_plan() optimizer calls now use QueryOptimizer::with_constraints()
    instead of QueryOptimizer::new()
  • DeploymentConstraints::from_budgets(&StageResourceBudgets) converts
    workload-derived budgets into optimizer constraints

controller/src/algebra/optimizer.rs

New: DeploymentConstraints::from_budgets()

  • Converts StageResourceBudgets (agent/backend memory) into DeploymentConstraints

Enhanced: DefaultCostModel::estimate()

  • WindowedAgg costed with actual sketch memory from directory::estimated_sketch_memory_bytes
  • Memory penalty: sketches exceeding agent_memory_bytes get 10× memory cost
  • Bandwidth penalty: output exceeding agent_backend_bandwidth get 10× bandwidth cost
  • Sliding window penalty: if agent_supports_sliding is false, sliding WindowedAgg gets 10× CPU cost
  • Partition and Dedup have non-unity cost factors (0.8, 0.9)

How it works

main.rs:
  budgets = StageResourceBudgets::from_workload_chars(&wc)
  constraints = DeploymentConstraints::from_budgets(&budgets)
  optimizer = QueryOptimizer::with_constraints(raw_bps, constraints)
  (opt_qe, _) = optimizer.optimize(qe)

The optimizer's cost model now returns inflated costs when constraints are
violated, causing cost-sensitive rewrite rules to prefer alternatives
(e.g., deferring a large sketch from Agent to Backend).

Tests

3 new tests:

  • constraints_from_budgets — verifies conversion from StageResourceBudgets
  • constrained_optimizer_penalises_large_sketch — tiny budget → 10× memory penalty
  • unconstrained_optimizer_normal_cost — no constraints → normal cost

318 total tests pass.

🤖 Generated with Claude Code

The optimizer now receives deployment constraints from the workload's
StageResourceBudgets and uses them in cost estimation:

1. DeploymentConstraints::from_budgets(StageResourceBudgets) — converts
   agent/backend memory budgets into optimizer constraints

2. main.rs handle_plan() now calls QueryOptimizer::with_constraints()
   instead of QueryOptimizer::new() — both optimizer call sites updated

3. DefaultCostModel enhanced:
   - Memory penalty: sketches exceeding agent_memory_bytes get 10× cost
   - Bandwidth penalty: output exceeding agent_backend_bandwidth get 10× cost
   - Sliding window penalty: if agent doesn't support sliding, 10× CPU cost
   - WindowedAgg costed with actual sketch memory from directory estimates
   - Partition and Dedup have non-unity cost factors

3 new tests: constraints_from_budgets, constrained_optimizer_penalises,
unconstrained_optimizer_normal_cost. 318 total pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the pr/window-optimizer-gaps branch from e799eb6 to 6ddb929 Compare April 4, 2026 17:47
@zzylol zzylol changed the title feat: SQL TUMBLE/HOP, WindowedAgg fusion, optimizer deployment constraints, PhysicalPlan tree feat: wire DeploymentConstraints into optimizer — cost-aware sketch placement Apr 4, 2026
@zzylol
zzylol changed the base branch from pr/shared-lowering to main April 4, 2026 17:47
@zzylol
zzylol force-pushed the pr/window-optimizer-gaps branch 6 times, most recently from ccec4ea to 670cac9 Compare April 4, 2026 19:26
Both optimizer calls now use with_constraints() AND physical_plan_to_staged():
  constraints = DeploymentConstraints::from_budgets(&budgets)
  (opt_qe, _) = QueryOptimizer::with_constraints(raw_bps, constraints).optimize(qe)
  (staged, _) = physical_plan_to_staged(&opt_qe, &budgets)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the pr/window-optimizer-gaps branch from 670cac9 to e029122 Compare April 4, 2026 19:35
@zzylol zzylol changed the title feat: wire DeploymentConstraints into optimizer — cost-aware sketch placement feat: sketch capability model + per-stage budgets + optimization problem formulation Apr 4, 2026
@zzylol
zzylol merged commit 0a3ba71 into main Apr 4, 2026
@zzylol
zzylol deleted the pr/window-optimizer-gaps branch April 4, 2026 19:39
zzylol added a commit that referenced this pull request Apr 4, 2026
Rebased on latest main. Dropped stale controller changes (already on
main via PRs #96, #116, #117, #119).

Only benchmark tooling remains:
- replay.py: replay DEBS dataset through OTel collector
- run.py: orchestrate benchmark runs
- scrape.py: collect Prometheus metrics during run
- analyze.py: parse benchmark results
- compare.py: compare sketch results to ground truth
- summarize.py: generate summary tables
- ground_truth/: ground truth computation for accuracy checking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 4, 2026
Rebased on latest main. Dropped stale controller changes (already on
main via PRs #96, #116, #117, #119).

Only benchmark tooling remains:
- replay.py: replay DEBS dataset through OTel collector
- run.py: orchestrate benchmark runs
- scrape.py: collect Prometheus metrics during run
- analyze.py: parse benchmark results
- compare.py: compare sketch results to ground truth
- summarize.py: generate summary tables
- ground_truth/: ground truth computation for accuracy checking

Co-authored-by: zz_y <zeyingz@umd.edu>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SieDeta pushed a commit that referenced this pull request Apr 17, 2026
…ets (#119)

The optimizer now receives deployment constraints from the workload's
StageResourceBudgets and uses them in cost estimation:

1. DeploymentConstraints::from_budgets(StageResourceBudgets) — converts
   agent/backend memory budgets into optimizer constraints

2. main.rs handle_plan() now calls QueryOptimizer::with_constraints()
   instead of QueryOptimizer::new() — both optimizer call sites updated

3. DefaultCostModel enhanced:
   - Memory penalty: sketches exceeding agent_memory_bytes get 10× cost
   - Bandwidth penalty: output exceeding agent_backend_bandwidth get 10× cost
   - Sliding window penalty: if agent doesn't support sliding, 10× CPU cost
   - WindowedAgg costed with actual sketch memory from directory estimates
   - Partition and Dedup have non-unity cost factors

3 new tests: constraints_from_budgets, constrained_optimizer_penalises,
unconstrained_optimizer_normal_cost. 318 total pass.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SieDeta pushed a commit that referenced this pull request Apr 17, 2026
Rebased on latest main. Dropped stale controller changes (already on
main via PRs #96, #116, #117, #119).

Only benchmark tooling remains:
- replay.py: replay DEBS dataset through OTel collector
- run.py: orchestrate benchmark runs
- scrape.py: collect Prometheus metrics during run
- analyze.py: parse benchmark results
- compare.py: compare sketch results to ground truth
- summarize.py: generate summary tables
- ground_truth/: ground truth computation for accuracy checking

Co-authored-by: zz_y <zeyingz@umd.edu>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant