Skip to content

feat: 5-layer architecture — AggIntent, WindowSpec, WindowedAgg (#115) - #116

Merged
zzylol merged 9 commits into
mainfrom
115-layering-of-control-plane
Apr 4, 2026
Merged

zzylol merged 9 commits into
mainfrom
115-layering-of-control-plane

Conversation

@zzylol

@zzylol zzylol commented Apr 2, 2026

Copy link
Copy Markdown
Contributor

Closes #115.

Summary

Refactors the controller into a clean 5-layer architecture, separating the sketch logical plan (what to compute) from physical implementation (how to compute it).

The 5 Layers

Query workloads
  │  Layer 1 — Query Language (PromQL, SQL, DataFusion, ElasticDSL, ...)
  ▼
Language-specific AST
  │  Layer 2 — Language Logical Plan
  ▼
Language Logical Plan
  │  Layer 3 — Sketch Logical Plan (Sketch Algebra)
  ▼
Sketch Logical Plan (QueryExpr with AggIntent)
  │  Layer 4 — Sketch Optimizer (rewrite rules)
  ▼
Optimised Sketch Logical Plan
  │  Layer 5 — Physical Execution Plan
  ▼
Physical Plan (OTel processors, PromSketch, DB queries, on-device)

Key changes

SketchAggOpAggIntent (implementation-independent)

Before (physical names) After (logical intent)
DDSketch { quantiles, epsilon } Quantile { quantiles, accuracy }
HLL { registers } Cardinality { accuracy }
CountMin { width, depth } / CountSketch { width, depth } Frequency { accuracy }
ExactMinMax { min, max } Extrema { min, max }
Hydra { inner, partition_keys } PerPartition { inner, keys }

The logical plan now says "I need a quantile at φ=0.99 with ≤1% error" — not "use DDSketch". The physical planner (Layer 5) decides the implementation.

WindowSpec + WindowKind — unified window model

WindowKind Semantics Origins
Tumbling { size } Fixed-size, non-overlapping SQL TUMBLE, Elastic fixed_interval
Sliding { size, slide } Fixed-size, overlapping PromQL [5m] range vector, SQL HOP
Unbounded All samples SQL GROUP BY without time
Landmark Epoch to now Running aggregates
Session { gap } Gap-based Elastic session windows

WindowedAgg — bundled window + sketch aggregation

In sketch systems the window defines the sketch lifecycle (when to flush/reset). WindowedAgg { agg, window, col, input } bundles them so the physical planner can select the best implementation:

  • OTel tumbling flush (time.NewTicker)
  • PromSketch ExponentialHistogram (supports tumbling + sliding natively)
  • DB GROUP BY time_bucket()

Updated doc

controller/docs/query-to-sketch-translation.md rewritten with the 5-layer architecture, AggIntent semantics, and WindowSpec design.

Files changed (9 files, +454 −300)

  • algebra/expr.rs — AggIntent, WindowSpec, WindowKind, WindowedAgg, backward-compat alias
  • algebra/directory.rs — updated to work with AggIntent; memory estimates from accuracy
  • algebra/allocator.rs — WindowedAgg arm, AggIntent dispatch
  • algebra/optimizer.rs — rules match on logical intents (Cardinality, Quantile, etc.)
  • planner/stage_split.rs — WindowedAgg handling, AggIntent dispatch
  • query_parser/promql.rs — emits AggIntent (Quantile, Cardinality, Frequency)
  • query_parser/mod.rs — QeCollector handles WindowedAgg + AggIntent
  • algebra/mod.rs — re-exports AggIntent, WindowSpec, WindowKind
  • docs/query-to-sketch-translation.md — 5-layer architecture doc

Test plan

  • All 279 tests pass
  • cargo build clean
  • pub type SketchAggOp = AggIntent provides backward compatibility

🤖 Generated with Claude Code

Refactors the sketch algebra to cleanly separate the logical plan
(Layer 3) from physical implementation details (Layer 5).

Core changes:
- Rename SketchAggOp → AggIntent with implementation-independent variants:
  DDSketch{quantiles,epsilon} → Quantile{quantiles,accuracy}
  HLL{registers}              → Cardinality{accuracy}
  CountMin/CountSketch{w,d}   → Frequency{accuracy}
  ExactMinMax{min,max}        → Extrema{min,max}
  Hydra{inner,partition_keys} → PerPartition{inner,keys}
- Add WindowSpec{kind,time_col} and WindowKind enum
  (Tumbling, Sliding, Unbounded, Landmark, Session)
- Add QueryExpr::WindowedAgg{agg,window,col,input} — bundles
  window and sketch agg because the window defines sketch lifecycle
- Keep SketchAggOp as backward-compat type alias for AggIntent
- Update directory, allocator, optimizer, stage_split, parsers
- Update docs/query-to-sketch-translation.md with 5-layer design

The logical plan now expresses WHAT to compute (quantile at φ=0.99
with ≤1% error) not HOW (DDSketch vs KLL vs PromSketch EHKLL).
Physical implementation selection moves to the physical planner.

279 tests pass, no business logic changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol zzylol linked an issue Apr 2, 2026 that may be closed by this pull request
…cal.rs

Phase 1 — Layer 3 cleanup:
- Remove SketchAggOp type alias and all backward-compat helpers
  (default_ddsketch, default_hll, default_count_min, default_count_sketch)
- All callers now use implementation-independent names:
  default_quantile, default_cardinality, default_frequency
- Zero physical sketch names (DDSketch, HLL, CountMin) remain in Layer 3

Phase 2 — Layer 5 formalization:
- New algebra/physical.rs with PhysicalAggOp struct and resolve() function
- PhysicalAggOp bundles: intent + sketch_type + sketch_params + memory estimate
- resolve() is the single Layer 3→5 boundary: AggIntent → physical implementation
- stage_split.rs and allocator.rs now call physical::resolve() instead of
  inline directory::sketch_type_for_op/sketch_params_for_op

Phase 3 — Layer documentation:
- promql.rs header documents Layers 1→3 lowering
- algebra/mod.rs documents the full 5-layer module layout
- docs/query-to-sketch-translation.md updated with 5-layer architecture

283 tests pass (279 + 4 new physical.rs tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the 115-layering-of-control-plane branch from 6b52a69 to 340f27a Compare April 3, 2026 02:52
zzylol and others added 4 commits April 3, 2026 20:49
Both parsers now emit Layer 2 relational operators (Aggregate + AggFunc),
and a shared lowering pass (algebra/lower.rs) converts to Layer 3
sketch algebra (SketchAgg + AggIntent).

Layer compliance:
- Layer 1-2 (parsers): zero AggIntent in code — only Aggregate + AggFunc
- Layer 2→3 (lower.rs): shared lower_to_sketch_algebra() for both languages
- Layer 3 (expr.rs): AggIntent, zero physical sketch names in code
- Layer 4 (optimizer.rs): matches AggIntent, zero SketchType references
- Layer 5 (physical.rs): resolve() maps AggIntent → SketchType + SketchParams

Changes:
- promql.rs: walk_call_to_op returns AggFunc (not AggIntent);
  build_qe_sketched → build_qe_aggregate emitting Aggregate + Window nodes
- lower.rs: new shared lowering pass (18 tests) — walks all 25 QueryExpr
  variants, converts single-agg Aggregate → SketchAgg where sketchable
- mod.rs: parse_query_expr calls lower_to_sketch_algebra after parsing

301 tests pass (283 + 18 new lowering tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…, L4 is deployment-aware

- Layers 1-3: query-language-independent and workload-independent
  (same AggIntent whether from PromQL, SQL, DataFusion, or ElasticDSL)
- Layer 4: deployment-constraint-aware optimizer (memory budgets,
  network topology, available backends)
- Layer 5: deployment-specific physical plan

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. WindowedAgg emission: lower.rs fuses Window + Aggregate → WindowedAgg
   when the inner Aggregate is sketchable (2 new tests)

2. SQL TUMBLE/HOP/time_bucket: SQL parser detects TUMBLE(ts, INTERVAL),
   HOP(ts, slide, size), and time_bucket('5 minutes', ts) in GROUP BY
   and emits Window nodes (4 new tests)

3. PhysicalPlan tree: physical.rs expanded with PhysicalOp enum
   (OtelSketchBuild, PromSketchBuild, SketchMerge, SketchEval, Exchange),
   Placement enum, PhysicalNode tree, PhysicalCost, resolve_window()
   (4 new tests)

4. Optimizer deployment constraints: DeploymentConstraints struct with
   agent/backend memory budgets, promsketch_available, agent_supports_sliding,
   bandwidth. QueryOptimizer::with_constraints() builder. DefaultCostModel
   penalises sketches exceeding agent budget.

5. Doc examples: all 4 concrete examples rewritten to show all 5 layers
   (Layer 1 AST → Layer 2 Aggregate+AggFunc → Layer 3 AggIntent →
   Layer 4 optimizer → Layer 5 physical resolve + config emit)

311 tests pass (301 + 10 new).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Fix label_matchers YAML format: controller now emits structured
   [{key: "env", value: "prod"}] matching the Go LabelMatcher struct,
   not flat ["env=prod"] strings.

2. Add bootstrap config endpoints for collectors:
   - GET /api/v1/collector-config/agent → default agent YAML
   - GET /api/v1/collector-config/backend → default backend YAML
   Collectors can start with:
     ./sketchcol --config "http://controller:8080/api/v1/collector-config/agent"

3. Add sketchcol unified collector builder config with ALL sketch
   processors (DDSketch, HLL, KLL, CountSketch, CountMinSketch +
   merge variants) in a single binary.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the 115-layering-of-control-plane branch from e3f8de5 to 9351f2e Compare April 4, 2026 16:01
…ectStore

Replace OTel/PromSketch-specific names with deployment-agnostic terms:
- edge processors (sketch build at data source)
- backend sketchDB (merge + approximate query)
- backend original DB (exact computation)
- object store (raw data backup)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the 115-layering-of-control-plane branch from 9351f2e to a4714ec Compare April 4, 2026 16:06
zzylol and others added 2 commits April 4, 2026 11:08
…owering

Section 3 now clearly separates:
- 3.1 Layer 1→2: language AST → language logical plan (Aggregate + AggFunc)
  Both PromQL and SQL produce the same relational operators, no sketch names.
- 3.2 Layer 2→3: shared lowering algorithm converts AggFunc → AggIntent
  Includes the full mapping table and Window→WindowedAgg fusion rule.

Removes all physical sketch names (DDSketch, HLL, CountMin) from the
parsing section — those belong only to Layer 5.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The PromQL parser now puts partition keys from `topk by (dims)` into
the inner Aggregate's `keys` field instead of wrapping with a separate
Partition node. This fixes two bugs:

1. Count inside topk was not lowered to Frequency — the lowering pass
   saw Count-without-GROUP-BY and skipped it. Now it sees
   Count-with-GROUP-BY → Frequency.

2. Double Partition wrapping — topk added Partition, and
   apply_qe_partition added another. Now the lowering pass creates
   the single Partition from the Aggregate's keys.

Example 5 now produces:
  TopK { 10, by: ["service"],
    Partition { ["service"],
      WindowedAgg { Frequency, Tumbling(1m), Filter(Source) } } }

Doc updated: fixed PromQL syntax (topk by (service) (10, ...)),
added Layer 1 + Layer 2 sections, corrected Layer 3 output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the 115-layering-of-control-plane branch from f3bc44d to f169ef0 Compare April 4, 2026 16:33
@zzylol
zzylol merged commit 70caaab into main Apr 4, 2026
@zzylol
zzylol deleted the 115-layering-of-control-plane branch April 4, 2026 16:34
zzylol added a commit that referenced this pull request Apr 4, 2026
Main had the placeholder physical.rs (from PR #116 merge). This branch
has the real physical planner with plan(), Placement, Exchange insertion.
Keep ours.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 4, 2026
Rebased on latest main. Dropped stale controller changes (already on
main via PRs #96, #116, #117, #119).

Only benchmark tooling remains:
- replay.py: replay DEBS dataset through OTel collector
- run.py: orchestrate benchmark runs
- scrape.py: collect Prometheus metrics during run
- analyze.py: parse benchmark results
- compare.py: compare sketch results to ground truth
- summarize.py: generate summary tables
- ground_truth/: ground truth computation for accuracy checking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Apr 4, 2026
Rebased on latest main. Dropped stale controller changes (already on
main via PRs #96, #116, #117, #119).

Only benchmark tooling remains:
- replay.py: replay DEBS dataset through OTel collector
- run.py: orchestrate benchmark runs
- scrape.py: collect Prometheus metrics during run
- analyze.py: parse benchmark results
- compare.py: compare sketch results to ground truth
- summarize.py: generate summary tables
- ground_truth/: ground truth computation for accuracy checking

Co-authored-by: zz_y <zeyingz@umd.edu>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SieDeta pushed a commit that referenced this pull request Apr 17, 2026
#116)

* feat: 5-layer architecture — AggIntent, WindowSpec, WindowedAgg

Refactors the sketch algebra to cleanly separate the logical plan
(Layer 3) from physical implementation details (Layer 5).

Core changes:
- Rename SketchAggOp → AggIntent with implementation-independent variants:
  DDSketch{quantiles,epsilon} → Quantile{quantiles,accuracy}
  HLL{registers}              → Cardinality{accuracy}
  CountMin/CountSketch{w,d}   → Frequency{accuracy}
  ExactMinMax{min,max}        → Extrema{min,max}
  Hydra{inner,partition_keys} → PerPartition{inner,keys}
- Add WindowSpec{kind,time_col} and WindowKind enum
  (Tumbling, Sliding, Unbounded, Landmark, Session)
- Add QueryExpr::WindowedAgg{agg,window,col,input} — bundles
  window and sketch agg because the window defines sketch lifecycle
- Keep SketchAggOp as backward-compat type alias for AggIntent
- Update directory, allocator, optimizer, stage_split, parsers
- Update docs/query-to-sketch-translation.md with 5-layer design

The logical plan now expresses WHAT to compute (quantile at φ=0.99
with ≤1% error) not HOW (DDSketch vs KLL vs PromSketch EHKLL).
Physical implementation selection moves to the physical planner.

279 tests pass, no business logic changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: complete 5-layer implementation — remove SketchAggOp, add physical.rs

Phase 1 — Layer 3 cleanup:
- Remove SketchAggOp type alias and all backward-compat helpers
  (default_ddsketch, default_hll, default_count_min, default_count_sketch)
- All callers now use implementation-independent names:
  default_quantile, default_cardinality, default_frequency
- Zero physical sketch names (DDSketch, HLL, CountMin) remain in Layer 3

Phase 2 — Layer 5 formalization:
- New algebra/physical.rs with PhysicalAggOp struct and resolve() function
- PhysicalAggOp bundles: intent + sketch_type + sketch_params + memory estimate
- resolve() is the single Layer 3→5 boundary: AggIntent → physical implementation
- stage_split.rs and allocator.rs now call physical::resolve() instead of
  inline directory::sketch_type_for_op/sketch_params_for_op

Phase 3 — Layer documentation:
- promql.rs header documents Layers 1→3 lowering
- algebra/mod.rs documents the full 5-layer module layout
- docs/query-to-sketch-translation.md updated with 5-layer architecture

283 tests pass (279 + 4 new physical.rs tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: both PromQL and SQL now follow 5-layer architecture

Both parsers now emit Layer 2 relational operators (Aggregate + AggFunc),
and a shared lowering pass (algebra/lower.rs) converts to Layer 3
sketch algebra (SketchAgg + AggIntent).

Layer compliance:
- Layer 1-2 (parsers): zero AggIntent in code — only Aggregate + AggFunc
- Layer 2→3 (lower.rs): shared lower_to_sketch_algebra() for both languages
- Layer 3 (expr.rs): AggIntent, zero physical sketch names in code
- Layer 4 (optimizer.rs): matches AggIntent, zero SketchType references
- Layer 5 (physical.rs): resolve() maps AggIntent → SketchType + SketchParams

Changes:
- promql.rs: walk_call_to_op returns AggFunc (not AggIntent);
  build_qe_sketched → build_qe_aggregate emitting Aggregate + Window nodes
- lower.rs: new shared lowering pass (18 tests) — walks all 25 QueryExpr
  variants, converts single-agg Aggregate → SketchAgg where sketchable
- mod.rs: parse_query_expr calls lower_to_sketch_algebra after parsing

301 tests pass (283 + 18 new lowering tests).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: fix layer descriptions — L1-3 are language/workload-independent, L4 is deployment-aware

- Layers 1-3: query-language-independent and workload-independent
  (same AggIntent whether from PromQL, SQL, DataFusion, or ElasticDSL)
- Layer 4: deployment-constraint-aware optimizer (memory budgets,
  network topology, available backends)
- Layer 5: deployment-specific physical plan

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: complete 5-layer implementation — all gaps fixed

1. WindowedAgg emission: lower.rs fuses Window + Aggregate → WindowedAgg
   when the inner Aggregate is sketchable (2 new tests)

2. SQL TUMBLE/HOP/time_bucket: SQL parser detects TUMBLE(ts, INTERVAL),
   HOP(ts, slide, size), and time_bucket('5 minutes', ts) in GROUP BY
   and emits Window nodes (4 new tests)

3. PhysicalPlan tree: physical.rs expanded with PhysicalOp enum
   (OtelSketchBuild, PromSketchBuild, SketchMerge, SketchEval, Exchange),
   Placement enum, PhysicalNode tree, PhysicalCost, resolve_window()
   (4 new tests)

4. Optimizer deployment constraints: DeploymentConstraints struct with
   agent/backend memory budgets, promsketch_available, agent_supports_sliding,
   bandwidth. QueryOptimizer::with_constraints() builder. DefaultCostModel
   penalises sketches exceeding agent budget.

5. Doc examples: all 4 concrete examples rewritten to show all 5 layers
   (Layer 1 AST → Layer 2 Aggregate+AggFunc → Layer 3 AggIntent →
   Layer 4 optimizer → Layer 5 physical resolve + config emit)

311 tests pass (301 + 10 new).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: fix end-to-end OTel integration gaps

1. Fix label_matchers YAML format: controller now emits structured
   [{key: "env", value: "prod"}] matching the Go LabelMatcher struct,
   not flat ["env=prod"] strings.

2. Add bootstrap config endpoints for collectors:
   - GET /api/v1/collector-config/agent → default agent YAML
   - GET /api/v1/collector-config/backend → default backend YAML
   Collectors can start with:
     ./sketchcol --config "http://controller:8080/api/v1/collector-config/agent"

3. Add sketchcol unified collector builder config with ALL sketch
   processors (DDSketch, HLL, KLL, CountSketch, CountMinSketch +
   merge variants) in a single binary.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update physical plan terminology — edge/sketchDB/originalDB/objectStore

Replace OTel/PromSketch-specific names with deployment-agnostic terms:
- edge processors (sketch build at data source)
- backend sketchDB (merge + approximate query)
- backend original DB (exact computation)
- object store (raw data backup)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: rewrite §3 as two clear steps — Layer 1→2 parsing + Layer 2→3 lowering

Section 3 now clearly separates:
- 3.1 Layer 1→2: language AST → language logical plan (Aggregate + AggFunc)
  Both PromQL and SQL produce the same relational operators, no sketch names.
- 3.2 Layer 2→3: shared lowering algorithm converts AggFunc → AggIntent
  Includes the full mapping table and Window→WindowedAgg fusion rule.

Removes all physical sketch names (DDSketch, HLL, CountMin) from the
parsing section — those belong only to Layer 5.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: propagate topk partition keys into inner Aggregate GROUP BY

The PromQL parser now puts partition keys from `topk by (dims)` into
the inner Aggregate's `keys` field instead of wrapping with a separate
Partition node. This fixes two bugs:

1. Count inside topk was not lowered to Frequency — the lowering pass
   saw Count-without-GROUP-BY and skipped it. Now it sees
   Count-with-GROUP-BY → Frequency.

2. Double Partition wrapping — topk added Partition, and
   apply_qe_partition added another. Now the lowering pass creates
   the single Partition from the Aggregate's keys.

Example 5 now produces:
  TopK { 10, by: ["service"],
    Partition { ["service"],
      WindowedAgg { Frequency, Tumbling(1m), Filter(Source) } } }

Doc updated: fixed PromQL syntax (topk by (service) (10, ...)),
added Layer 1 + Layer 2 sections, corrected Layer 3 output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SieDeta pushed a commit that referenced this pull request Apr 17, 2026
Rebased on latest main. Dropped stale controller changes (already on
main via PRs #96, #116, #117, #119).

Only benchmark tooling remains:
- replay.py: replay DEBS dataset through OTel collector
- run.py: orchestrate benchmark runs
- scrape.py: collect Prometheus metrics during run
- analyze.py: parse benchmark results
- compare.py: compare sketch results to ground truth
- summarize.py: generate summary tables
- ground_truth/: ground truth computation for accuracy checking

Co-authored-by: zz_y <zeyingz@umd.edu>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

layering of control plane

1 participant