feat: 5-layer architecture — AggIntent, WindowSpec, WindowedAgg (#115) - #116
Merged
Merged
Conversation
Refactors the sketch algebra to cleanly separate the logical plan
(Layer 3) from physical implementation details (Layer 5).
Core changes:
- Rename SketchAggOp → AggIntent with implementation-independent variants:
DDSketch{quantiles,epsilon} → Quantile{quantiles,accuracy}
HLL{registers} → Cardinality{accuracy}
CountMin/CountSketch{w,d} → Frequency{accuracy}
ExactMinMax{min,max} → Extrema{min,max}
Hydra{inner,partition_keys} → PerPartition{inner,keys}
- Add WindowSpec{kind,time_col} and WindowKind enum
(Tumbling, Sliding, Unbounded, Landmark, Session)
- Add QueryExpr::WindowedAgg{agg,window,col,input} — bundles
window and sketch agg because the window defines sketch lifecycle
- Keep SketchAggOp as backward-compat type alias for AggIntent
- Update directory, allocator, optimizer, stage_split, parsers
- Update docs/query-to-sketch-translation.md with 5-layer design
The logical plan now expresses WHAT to compute (quantile at φ=0.99
with ≤1% error) not HOW (DDSketch vs KLL vs PromSketch EHKLL).
Physical implementation selection moves to the physical planner.
279 tests pass, no business logic changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…cal.rs Phase 1 — Layer 3 cleanup: - Remove SketchAggOp type alias and all backward-compat helpers (default_ddsketch, default_hll, default_count_min, default_count_sketch) - All callers now use implementation-independent names: default_quantile, default_cardinality, default_frequency - Zero physical sketch names (DDSketch, HLL, CountMin) remain in Layer 3 Phase 2 — Layer 5 formalization: - New algebra/physical.rs with PhysicalAggOp struct and resolve() function - PhysicalAggOp bundles: intent + sketch_type + sketch_params + memory estimate - resolve() is the single Layer 3→5 boundary: AggIntent → physical implementation - stage_split.rs and allocator.rs now call physical::resolve() instead of inline directory::sketch_type_for_op/sketch_params_for_op Phase 3 — Layer documentation: - promql.rs header documents Layers 1→3 lowering - algebra/mod.rs documents the full 5-layer module layout - docs/query-to-sketch-translation.md updated with 5-layer architecture 283 tests pass (279 + 4 new physical.rs tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
115-layering-of-control-plane
branch
from
April 3, 2026 02:52
6b52a69 to
340f27a
Compare
Both parsers now emit Layer 2 relational operators (Aggregate + AggFunc), and a shared lowering pass (algebra/lower.rs) converts to Layer 3 sketch algebra (SketchAgg + AggIntent). Layer compliance: - Layer 1-2 (parsers): zero AggIntent in code — only Aggregate + AggFunc - Layer 2→3 (lower.rs): shared lower_to_sketch_algebra() for both languages - Layer 3 (expr.rs): AggIntent, zero physical sketch names in code - Layer 4 (optimizer.rs): matches AggIntent, zero SketchType references - Layer 5 (physical.rs): resolve() maps AggIntent → SketchType + SketchParams Changes: - promql.rs: walk_call_to_op returns AggFunc (not AggIntent); build_qe_sketched → build_qe_aggregate emitting Aggregate + Window nodes - lower.rs: new shared lowering pass (18 tests) — walks all 25 QueryExpr variants, converts single-agg Aggregate → SketchAgg where sketchable - mod.rs: parse_query_expr calls lower_to_sketch_algebra after parsing 301 tests pass (283 + 18 new lowering tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…, L4 is deployment-aware - Layers 1-3: query-language-independent and workload-independent (same AggIntent whether from PromQL, SQL, DataFusion, or ElasticDSL) - Layer 4: deployment-constraint-aware optimizer (memory budgets, network topology, available backends) - Layer 5: deployment-specific physical plan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. WindowedAgg emission: lower.rs fuses Window + Aggregate → WindowedAgg
when the inner Aggregate is sketchable (2 new tests)
2. SQL TUMBLE/HOP/time_bucket: SQL parser detects TUMBLE(ts, INTERVAL),
HOP(ts, slide, size), and time_bucket('5 minutes', ts) in GROUP BY
and emits Window nodes (4 new tests)
3. PhysicalPlan tree: physical.rs expanded with PhysicalOp enum
(OtelSketchBuild, PromSketchBuild, SketchMerge, SketchEval, Exchange),
Placement enum, PhysicalNode tree, PhysicalCost, resolve_window()
(4 new tests)
4. Optimizer deployment constraints: DeploymentConstraints struct with
agent/backend memory budgets, promsketch_available, agent_supports_sliding,
bandwidth. QueryOptimizer::with_constraints() builder. DefaultCostModel
penalises sketches exceeding agent budget.
5. Doc examples: all 4 concrete examples rewritten to show all 5 layers
(Layer 1 AST → Layer 2 Aggregate+AggFunc → Layer 3 AggIntent →
Layer 4 optimizer → Layer 5 physical resolve + config emit)
311 tests pass (301 + 10 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Fix label_matchers YAML format: controller now emits structured
[{key: "env", value: "prod"}] matching the Go LabelMatcher struct,
not flat ["env=prod"] strings.
2. Add bootstrap config endpoints for collectors:
- GET /api/v1/collector-config/agent → default agent YAML
- GET /api/v1/collector-config/backend → default backend YAML
Collectors can start with:
./sketchcol --config "http://controller:8080/api/v1/collector-config/agent"
3. Add sketchcol unified collector builder config with ALL sketch
processors (DDSketch, HLL, KLL, CountSketch, CountMinSketch +
merge variants) in a single binary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
115-layering-of-control-plane
branch
from
April 4, 2026 16:01
e3f8de5 to
9351f2e
Compare
…ectStore Replace OTel/PromSketch-specific names with deployment-agnostic terms: - edge processors (sketch build at data source) - backend sketchDB (merge + approximate query) - backend original DB (exact computation) - object store (raw data backup) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
115-layering-of-control-plane
branch
from
April 4, 2026 16:06
9351f2e to
a4714ec
Compare
…owering Section 3 now clearly separates: - 3.1 Layer 1→2: language AST → language logical plan (Aggregate + AggFunc) Both PromQL and SQL produce the same relational operators, no sketch names. - 3.2 Layer 2→3: shared lowering algorithm converts AggFunc → AggIntent Includes the full mapping table and Window→WindowedAgg fusion rule. Removes all physical sketch names (DDSketch, HLL, CountMin) from the parsing section — those belong only to Layer 5. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The PromQL parser now puts partition keys from `topk by (dims)` into
the inner Aggregate's `keys` field instead of wrapping with a separate
Partition node. This fixes two bugs:
1. Count inside topk was not lowered to Frequency — the lowering pass
saw Count-without-GROUP-BY and skipped it. Now it sees
Count-with-GROUP-BY → Frequency.
2. Double Partition wrapping — topk added Partition, and
apply_qe_partition added another. Now the lowering pass creates
the single Partition from the Aggregate's keys.
Example 5 now produces:
TopK { 10, by: ["service"],
Partition { ["service"],
WindowedAgg { Frequency, Tumbling(1m), Filter(Source) } } }
Doc updated: fixed PromQL syntax (topk by (service) (10, ...)),
added Layer 1 + Layer 2 sections, corrected Layer 3 output.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
115-layering-of-control-plane
branch
from
April 4, 2026 16:33
f3bc44d to
f169ef0
Compare
zzylol
added a commit
that referenced
this pull request
Apr 4, 2026
Main had the placeholder physical.rs (from PR #116 merge). This branch has the real physical planner with plan(), Placement, Exchange insertion. Keep ours. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 4, 2026
Rebased on latest main. Dropped stale controller changes (already on main via PRs #96, #116, #117, #119). Only benchmark tooling remains: - replay.py: replay DEBS dataset through OTel collector - run.py: orchestrate benchmark runs - scrape.py: collect Prometheus metrics during run - analyze.py: parse benchmark results - compare.py: compare sketch results to ground truth - summarize.py: generate summary tables - ground_truth/: ground truth computation for accuracy checking Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 4, 2026
Rebased on latest main. Dropped stale controller changes (already on main via PRs #96, #116, #117, #119). Only benchmark tooling remains: - replay.py: replay DEBS dataset through OTel collector - run.py: orchestrate benchmark runs - scrape.py: collect Prometheus metrics during run - analyze.py: parse benchmark results - compare.py: compare sketch results to ground truth - summarize.py: generate summary tables - ground_truth/: ground truth computation for accuracy checking Co-authored-by: zz_y <zeyingz@umd.edu> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SieDeta
pushed a commit
that referenced
this pull request
Apr 17, 2026
#116) * feat: 5-layer architecture — AggIntent, WindowSpec, WindowedAgg Refactors the sketch algebra to cleanly separate the logical plan (Layer 3) from physical implementation details (Layer 5). Core changes: - Rename SketchAggOp → AggIntent with implementation-independent variants: DDSketch{quantiles,epsilon} → Quantile{quantiles,accuracy} HLL{registers} → Cardinality{accuracy} CountMin/CountSketch{w,d} → Frequency{accuracy} ExactMinMax{min,max} → Extrema{min,max} Hydra{inner,partition_keys} → PerPartition{inner,keys} - Add WindowSpec{kind,time_col} and WindowKind enum (Tumbling, Sliding, Unbounded, Landmark, Session) - Add QueryExpr::WindowedAgg{agg,window,col,input} — bundles window and sketch agg because the window defines sketch lifecycle - Keep SketchAggOp as backward-compat type alias for AggIntent - Update directory, allocator, optimizer, stage_split, parsers - Update docs/query-to-sketch-translation.md with 5-layer design The logical plan now expresses WHAT to compute (quantile at φ=0.99 with ≤1% error) not HOW (DDSketch vs KLL vs PromSketch EHKLL). Physical implementation selection moves to the physical planner. 279 tests pass, no business logic changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: complete 5-layer implementation — remove SketchAggOp, add physical.rs Phase 1 — Layer 3 cleanup: - Remove SketchAggOp type alias and all backward-compat helpers (default_ddsketch, default_hll, default_count_min, default_count_sketch) - All callers now use implementation-independent names: default_quantile, default_cardinality, default_frequency - Zero physical sketch names (DDSketch, HLL, CountMin) remain in Layer 3 Phase 2 — Layer 5 formalization: - New algebra/physical.rs with PhysicalAggOp struct and resolve() function - PhysicalAggOp bundles: intent + sketch_type + sketch_params + memory estimate - resolve() is the single Layer 3→5 boundary: AggIntent → physical implementation - stage_split.rs and allocator.rs now call physical::resolve() instead of inline directory::sketch_type_for_op/sketch_params_for_op Phase 3 — Layer documentation: - promql.rs header documents Layers 1→3 lowering - algebra/mod.rs documents the full 5-layer module layout - docs/query-to-sketch-translation.md updated with 5-layer architecture 283 tests pass (279 + 4 new physical.rs tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: both PromQL and SQL now follow 5-layer architecture Both parsers now emit Layer 2 relational operators (Aggregate + AggFunc), and a shared lowering pass (algebra/lower.rs) converts to Layer 3 sketch algebra (SketchAgg + AggIntent). Layer compliance: - Layer 1-2 (parsers): zero AggIntent in code — only Aggregate + AggFunc - Layer 2→3 (lower.rs): shared lower_to_sketch_algebra() for both languages - Layer 3 (expr.rs): AggIntent, zero physical sketch names in code - Layer 4 (optimizer.rs): matches AggIntent, zero SketchType references - Layer 5 (physical.rs): resolve() maps AggIntent → SketchType + SketchParams Changes: - promql.rs: walk_call_to_op returns AggFunc (not AggIntent); build_qe_sketched → build_qe_aggregate emitting Aggregate + Window nodes - lower.rs: new shared lowering pass (18 tests) — walks all 25 QueryExpr variants, converts single-agg Aggregate → SketchAgg where sketchable - mod.rs: parse_query_expr calls lower_to_sketch_algebra after parsing 301 tests pass (283 + 18 new lowering tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: fix layer descriptions — L1-3 are language/workload-independent, L4 is deployment-aware - Layers 1-3: query-language-independent and workload-independent (same AggIntent whether from PromQL, SQL, DataFusion, or ElasticDSL) - Layer 4: deployment-constraint-aware optimizer (memory budgets, network topology, available backends) - Layer 5: deployment-specific physical plan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: complete 5-layer implementation — all gaps fixed 1. WindowedAgg emission: lower.rs fuses Window + Aggregate → WindowedAgg when the inner Aggregate is sketchable (2 new tests) 2. SQL TUMBLE/HOP/time_bucket: SQL parser detects TUMBLE(ts, INTERVAL), HOP(ts, slide, size), and time_bucket('5 minutes', ts) in GROUP BY and emits Window nodes (4 new tests) 3. PhysicalPlan tree: physical.rs expanded with PhysicalOp enum (OtelSketchBuild, PromSketchBuild, SketchMerge, SketchEval, Exchange), Placement enum, PhysicalNode tree, PhysicalCost, resolve_window() (4 new tests) 4. Optimizer deployment constraints: DeploymentConstraints struct with agent/backend memory budgets, promsketch_available, agent_supports_sliding, bandwidth. QueryOptimizer::with_constraints() builder. DefaultCostModel penalises sketches exceeding agent budget. 5. Doc examples: all 4 concrete examples rewritten to show all 5 layers (Layer 1 AST → Layer 2 Aggregate+AggFunc → Layer 3 AggIntent → Layer 4 optimizer → Layer 5 physical resolve + config emit) 311 tests pass (301 + 10 new). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: fix end-to-end OTel integration gaps 1. Fix label_matchers YAML format: controller now emits structured [{key: "env", value: "prod"}] matching the Go LabelMatcher struct, not flat ["env=prod"] strings. 2. Add bootstrap config endpoints for collectors: - GET /api/v1/collector-config/agent → default agent YAML - GET /api/v1/collector-config/backend → default backend YAML Collectors can start with: ./sketchcol --config "http://controller:8080/api/v1/collector-config/agent" 3. Add sketchcol unified collector builder config with ALL sketch processors (DDSketch, HLL, KLL, CountSketch, CountMinSketch + merge variants) in a single binary. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update physical plan terminology — edge/sketchDB/originalDB/objectStore Replace OTel/PromSketch-specific names with deployment-agnostic terms: - edge processors (sketch build at data source) - backend sketchDB (merge + approximate query) - backend original DB (exact computation) - object store (raw data backup) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite §3 as two clear steps — Layer 1→2 parsing + Layer 2→3 lowering Section 3 now clearly separates: - 3.1 Layer 1→2: language AST → language logical plan (Aggregate + AggFunc) Both PromQL and SQL produce the same relational operators, no sketch names. - 3.2 Layer 2→3: shared lowering algorithm converts AggFunc → AggIntent Includes the full mapping table and Window→WindowedAgg fusion rule. Removes all physical sketch names (DDSketch, HLL, CountMin) from the parsing section — those belong only to Layer 5. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: propagate topk partition keys into inner Aggregate GROUP BY The PromQL parser now puts partition keys from `topk by (dims)` into the inner Aggregate's `keys` field instead of wrapping with a separate Partition node. This fixes two bugs: 1. Count inside topk was not lowered to Frequency — the lowering pass saw Count-without-GROUP-BY and skipped it. Now it sees Count-with-GROUP-BY → Frequency. 2. Double Partition wrapping — topk added Partition, and apply_qe_partition added another. Now the lowering pass creates the single Partition from the Aggregate's keys. Example 5 now produces: TopK { 10, by: ["service"], Partition { ["service"], WindowedAgg { Frequency, Tumbling(1m), Filter(Source) } } } Doc updated: fixed PromQL syntax (topk by (service) (10, ...)), added Layer 1 + Layer 2 sections, corrected Layer 3 output. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SieDeta
pushed a commit
that referenced
this pull request
Apr 17, 2026
Rebased on latest main. Dropped stale controller changes (already on main via PRs #96, #116, #117, #119). Only benchmark tooling remains: - replay.py: replay DEBS dataset through OTel collector - run.py: orchestrate benchmark runs - scrape.py: collect Prometheus metrics during run - analyze.py: parse benchmark results - compare.py: compare sketch results to ground truth - summarize.py: generate summary tables - ground_truth/: ground truth computation for accuracy checking Co-authored-by: zz_y <zeyingz@umd.edu> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #115.
Summary
Refactors the controller into a clean 5-layer architecture, separating the sketch logical plan (what to compute) from physical implementation (how to compute it).
The 5 Layers
Key changes
SketchAggOp→AggIntent(implementation-independent)DDSketch { quantiles, epsilon }Quantile { quantiles, accuracy }HLL { registers }Cardinality { accuracy }CountMin { width, depth }/CountSketch { width, depth }Frequency { accuracy }ExactMinMax { min, max }Extrema { min, max }Hydra { inner, partition_keys }PerPartition { inner, keys }The logical plan now says "I need a quantile at φ=0.99 with ≤1% error" — not "use DDSketch". The physical planner (Layer 5) decides the implementation.
WindowSpec+WindowKind— unified window modelTumbling { size }Sliding { size, slide }UnboundedLandmarkSession { gap }WindowedAgg— bundled window + sketch aggregationIn sketch systems the window defines the sketch lifecycle (when to flush/reset).
WindowedAgg { agg, window, col, input }bundles them so the physical planner can select the best implementation:time.NewTicker)GROUP BY time_bucket()Updated doc
controller/docs/query-to-sketch-translation.mdrewritten with the 5-layer architecture, AggIntent semantics, and WindowSpec design.Files changed (9 files, +454 −300)
algebra/expr.rs— AggIntent, WindowSpec, WindowKind, WindowedAgg, backward-compat aliasalgebra/directory.rs— updated to work with AggIntent; memory estimates from accuracyalgebra/allocator.rs— WindowedAgg arm, AggIntent dispatchalgebra/optimizer.rs— rules match on logical intents (Cardinality, Quantile, etc.)planner/stage_split.rs— WindowedAgg handling, AggIntent dispatchquery_parser/promql.rs— emits AggIntent (Quantile, Cardinality, Frequency)query_parser/mod.rs— QeCollector handles WindowedAgg + AggIntentalgebra/mod.rs— re-exports AggIntent, WindowSpec, WindowKinddocs/query-to-sketch-translation.md— 5-layer architecture docTest plan
cargo buildcleanpub type SketchAggOp = AggIntentprovides backward compatibility🤖 Generated with Claude Code