From 364f7997535b635b41ed46894e9997d384fa0808 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 26 Aug 2026 11:15:14 -0600 Subject: [PATCH 01/12] docs: plan ASAPPlanner workload migration --- ...-asapplanner-workload-planner-migration.md | 465 ++++++++++++++++++ 1 file changed, 465 insertions(+) create mode 100644 control_plane/docs/design-asapplanner-workload-planner-migration.md diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md new file mode 100644 index 00000000..edc65e78 --- /dev/null +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -0,0 +1,465 @@ +# Replace the legacy query planner with ASAPPlanner workload planning + +> Status: proposed, 2026-08-26 +> +> Scope: migrate ASAPQuery-backend from its overlapping, per-query +> query/replacement planning paths to ASAPPlanner's workload-wide replacement +> search. This document does **not** propose removing ASAPQuery's +> deployment-specific placement, materialization, routing, or serving logic. + +## 1. Decision + +Retire the legacy ASAPQuery query/replacement planner after the new path has +passed shadow and end-to-end validation. + +ASAPPlanner becomes the single owner of: + +```text +query text + -> pre-ASAP IR + -> workload common-subexpression sharing + -> replacement candidate search + -> cost-based strategy selection + -> selected post-ASAP DAG +``` + +ASAPQuery-backend remains the owner of: + +```text +selected post-ASAP DAG + -> deployment placement + -> collector/backend stage allocation + -> BackendPlan + -> materialization and routing + -> data-plane execution and archive fallback +``` + +The boundary is intentional: ASAPPlanner decides *what a query sub-DAG may be +replaced with*; ASAPQuery decides *where the selected replacement runs, how it +is represented on the wire, and how it is served*. + +## 2. Why the legacy query planner should be removed + +ASAPQuery currently has several overlapping planning paths: + +- `query_planning.rs` reduces a set of query strings into per-metric + capabilities and then unions the required sketch families. +- `asap_tier_implement.rs` finds independently realizable aggregate roots and + implements each root separately. +- `sketch_algebra/lower.rs` and parts of `optimizer/rules` make additional + summary-family or replacement decisions. +- serving-time lowering can reconstruct or infer decisions after the control + plane has already planned them. + +Flattening a query into `(metric, capability, sketch family)` loses the IR +relationships needed for workload optimizations. In particular, it cannot +faithfully represent: + +- common subexpressions shared by multiple queries; +- one shared scan or shared summary sub-DAG with several consumers; +- `AvgToSumOverCountStrategy`; +- group-by rollup reuse; +- shared versus independent grouped summaries, including Hydra layouts; +- deriving a smaller compatible top-k result from a larger one; +- post-ASAP node provenance and strategy explanations. + +Keeping two implementations for these decisions would also let the control +plane, data plane, and ASAPPlanner silently select different physical summary +families or parameters for the same query. + +## 3. Target architecture + +```text +WorkloadPlanRequest + queries: [{id, language, text, accuracy, schema_refs}] + table_schemas: [...] + | + v + parse and schema-bind every query + | + v + Vec<(QueryId, Rc)> + | + v + ASAPPlanner search_workload_with + | + v + PlanSpace::global_selection + | + v + ASAPQuery SelectedWorkloadPlan + / \ + v v + ExplainPlan deployment placement + | + v + BackendPlan + | + v + data_plane +``` + +The selected workload and its shared `Rc`/`Rc` +identities must remain intact until placement and materialization are complete. +It must not be flattened into independent per-query or per-metric plans before +that point. + +## 4. What is removed, retained, and changed + +### 4.1 Remove after cutover + +- The capability-union allocation algorithm in `query_planning.rs` as a + production planner. +- Per-aggregate-root `implement_tree` behavior in + `asap_tier_implement.rs`. +- Replacement/sketch-family choice duplicated in `optimizer/rules` and + `sketch_algebra/lower.rs`. +- Serving-time use of a cost model to guess a decision already made by the + control plane. +- `BackendStageConfig -> BackendPlan` as the canonical source of planning + decisions. It can remain temporarily as a legacy compatibility adapter. + +Delete these paths only after the new workload path is the production source +of `BackendPlan`. During migration they remain available for shadow comparison +and rollback. + +### 4.2 Retain and adapt + +- The HTTP API and workload registry. +- Replanning triggers and runtime telemetry. +- ASAPQuery's cost model, implemented through ASAPPlanner's `CostModel` trait. +- Deployment constraints and resource budgets. +- Stage splitting and placement across collector, backend, and archive. +- Collector configuration generation. +- `PolicyFingerprint` and persistent materialization identity. +- `BackendPlan`, `RoutingIndex`, push, hot reload, and plan versioning. +- Data-plane summary execution and cold/archive fallback. +- Monitoring and epsilon-allocation behavior that is independent of logical + replacement search. + +### 4.3 Add + +- A typed batch workload request. +- A deployment-owned `SelectedWorkloadPlan` representation. +- A materializer from ASAPPlanner's global selection into the selected + post-ASAP DAG and then into `BackendPlan`. +- An explain/debug representation with explicit pre-node, decision, and + post-node mappings. +- Planning phase timings, search-size metrics, deadlines, and cancellation. + +## 5. Migration stack + +Each phase should land as a separate, buildable PR. Later PRs may be stacked +while earlier ones are under review. + +### PR 1: ASAPPlanner pin and API compatibility + +Move every ASAPPlanner dependency to the same immutable revision. The first +target containing the workload planner and DAG-viewer changes is +`747c66a8958409afd727b4d8046e16c653d228f6` (ASAPPlanner PR #283). + +Update together: + +- `planner-types` in `control_plane`, `data_plane`, and the local shared-types + crate; +- `asap-aware-mapping`; +- `asap-frontend-promql`; +- `Cargo.lock`. + +The new mapping API removes the former `bind` and `boundary` modules. Migrate +call sites to `asap_aware_mapping::replacement` and its public re-exports: + +- `Implementation` and candidate enumeration; +- `ReplacementStrategy` and `ReplacementSubDAG`; +- `search_workload_with` and `default_strategies_with`; +- `PlanSpace::global_selection`; +- `replacement::default_size_params`. + +Adapt the ASAPQuery cost model: + +- rank `SketchAlgorithm` values and return an exact permutation of the input; +- size `SketchParams` for the selected algorithm; +- preserve deployment-specific extension realization; +- supply CSE recompute and shared-maintenance costs; +- supply subpopulation estimates and grouping-state costs when statistics are + available; +- expose numeric `estimate_cost` values for observability. + +Acceptance criteria: + +- `cargo build --workspace` succeeds; +- existing control-plane and data-plane tests pass; +- no ASAPPlanner crate is pinned to a different revision; +- the compatibility path does not change production output yet. + +### PR 2: Canonical batch workload input + +Add a new API such as `POST /api/v1/workloads/plan`: + +```json +{ + "queries": [ + { + "id": "q1", + "language": "promql", + "text": "sum by (service) (rate(requests_total[5m]))", + "accuracy": { "epsilon": 0.01 }, + "schema_refs": [] + } + ], + "table_schemas": [] +} +``` + +Requirements: + +- stable, caller-visible query IDs; +- one typed `AccuracyTarget` per query, with an optional workload default; +- PromQL lowering first, retaining an explicit language field; +- table schemas and schema binding before enabling SQL; +- request limits and validation; +- old single-query endpoints adapt into a one-query workload rather than + maintaining a second planner. + +The result of this phase is a vector of named `Rc` roots, not a +`QuerySetPlan` of flattened capabilities. + +Acceptance criteria: + +- one and multiple queries enter the same code path; +- query IDs survive lowering and error reporting; +- equivalent subtrees remain shareable across roots; +- malformed queries and schemas return per-query diagnostics. + +### PR 3: Workload-wide search and selection + +Run one planning operation for the complete workload: + +```rust +let strategies = default_strategies_with(&backend_cost_model); +let space = search_workload_with(roots, &strategies); +let selection = space.global_selection(&backend_cost_model); +``` + +Build `SelectedWorkloadPlan` from the result while preserving shared node +identity. It must support selected `Replacement::Summary` and +`Replacement::Rewrite` candidates and recursively compose choices for +descendant targets. + +This phase should activate and test: + +- `SketchAlgorithmStrategy`; +- `SharedSubtreeStrategy`; +- `HydraGroupingStrategy`; +- `AvgToSumOverCountStrategy`; +- `RollupStrategy`; +- `TopKLimitReuseStrategy`. + +ASAPPlanner deliberately owns candidate search but not ASAPQuery deployment +placement. Do not copy `dag_export`'s JSON into the runtime contract and do +not infer mappings from labels, hashes, strategy rationale, or viewer node +signatures. + +Acceptance criteria: + +- compatible scans and sub-DAGs have one shared node identity; +- incompatible filters, windows, groupings, orderings, and schemas do not + share; +- every chosen replacement records its real strategy name and target; +- selection is deterministic for identical workload, statistics, and cost + model inputs. + +### PR 4: Selected workload to BackendPlan + +Introduce a direct conversion: + +```text +SelectedWorkloadPlan + -> deployment placement + -> materializations and readouts + -> BackendPlan +``` + +Continue using `PolicyFingerprint` for persistent runtime identity. Exporter +IDs such as DAG node IDs or `workload_node_id` are scoped to an explain result +and must not become materialization keys. + +Review whether the wire needs additive fields for: + +- exact versus sketch summary family; +- sketch algorithm and parameters; +- independent versus Hydra grouping layout; +- shared materialization dependencies; +- multiple query/readout consumers of one materialization; +- derived readouts such as `avg = sum / count`, rollups, and top-k prefix + reuse. + +Use additive protobuf fields and retain backward decoding during rollout. + +Acceptance criteria: + +- two queries sharing one summary produce one materialization and multiple + legal routes/readouts; +- materialization fingerprints are stable across replans; +- protobuf encode/decode and hot reload preserve the chosen plan; +- the data plane never has to run a cost model to reconstruct the choice. + +### PR 5: Data-plane execution coverage + +Make the data plane execute or explicitly reject every post-ASAP shape the +control plane may select. Cover at least: + +- exact accumulators; +- sketch aggregation and estimate; +- sum/count composition for rewritten Avg; +- reading top-k `k_small` from a compatible `k_large` summary; +- rollup derivation; +- Hydra grouping layouts; +- shared summary dependencies and merges. + +Unsupported candidates must be removed before selection or fail planning with +a clear capability diagnostic. They must never be accepted by the control +plane and fail later during query serving. + +Acceptance criteria: + +- control plane selects once and the data plane consumes that exact choice; +- family, parameters, grouping, and accuracy are not re-derived at serving + time; +- archive fallback remains available for unsupported queries; +- end-to-end tests prove plan push, hot reload, and serving. + +### PR 6: Explain and planner observability + +Add an optional explain response or endpoint containing: + +- the original pre-ASAP DAG; +- the selected post-ASAP DAG; +- explicit pre-target -> decision -> post-node mappings; +- strategy, concise rationale, rank, and estimated cost; +- query ownership of shared nodes; +- edge schemas; +- phase timings. + +The shape may reuse ASAPPlanner's `dag_export` domain types where they are +appropriate, but production planning must call Rust APIs directly. It must +not start the Python viewer server or invoke the `dag_export` binary as a +subprocess. + +Record at least: + +- parse and schema-binding time; +- pre-ASAP IR and CSE time; +- replacement search time; +- global-selection time; +- post-ASAP materialization time; +- `BackendPlan` construction and push time; +- target, candidate, shared-node, and selected-materialization counts. + +### PR 7: Shadow rollout and legacy deletion + +Introduce a feature flag such as `ASAP_WORKLOAD_PLANNER_V2` and use three +rollout modes: + +1. `legacy`: legacy planner emits; new planner is disabled. +2. `shadow`: legacy planner emits; new planner runs and differences are + recorded. +3. `selected`: new planner emits; legacy planner is available only for + rollback. + +Compare: + +- selected summary family and parameters; +- materialization count and fingerprints; +- routing coverage; +- estimated resource cost; +- warm versus archive placement; +- planning latency and errors. + +After one release cycle without unexplained mismatches: + +- delete the capability-union production planner; +- delete per-root implementation code; +- delete duplicate replacement decisions in optimizer/lowering modules; +- delete serving-time cost-based reconstruction; +- remove the legacy `BackendStageConfig -> BackendPlan` path once no other + caller requires it; +- keep only the single-query API adapters, not a single-query planner. + +## 6. Cross-repository golden workload + +Use the ASAPPlanner DAG-viewer demo queries as shared regression fixtures: + +- `q1`: grouped count; +- `q2`: grouped Avg rewritten to sum/count while sharing input with `q1`; +- `q3`: top-5 over a rate/frequency summary; +- `q4`: compatible top-10, allowing `q3` to derive from the larger result; +- `q6`: join query sharing a compatible input scan with other queries. + +Add negative fixtures for: + +- different filter predicates; +- incompatible group-by reductions; +- different windows; +- different sort keys; +- incompatible schemas or table bindings; +- top-k inputs that differ below the limit. + +Tests are required at four boundaries: + +1. query workload -> selected strategies; +2. selected workload -> `BackendPlan`; +3. protobuf -> data-plane hot reload and `RoutingIndex`; +4. ingest -> plan push -> warm query response, including archive fallback. + +## 7. Complexity and safety limits + +ASAPPlanner stores alternatives in memo groups rather than enumerating the +Cartesian product of whole plans. This avoids exponential copying of complete +workload DAGs, but it does not make every strategy linear: + +- common-subexpression discovery is approximately linear in reachable IR + nodes, subject to hashing/equality checks; +- ordinary candidate generation is proportional to discovered targets, + registered strategies, and candidates per target; +- rollup sibling discovery may compare aggregate pairs; +- top-k reuse may compare compatible limit pairs; +- candidate sorting adds per-group sorting cost; +- global selection is a topological dynamic-programming pass over the + discovered reference graph. + +Protect the control plane with: + +- maximum query, schema, IR-node, and candidate counts; +- a planning deadline and cancellation token; +- ASAPPlanner's search-iteration cap plus a deployment-level deadline; +- normalized-workload caching keyed by query, schema, accuracy, planner + revision, statistics epoch, and cost-model version; +- benchmarks for 1, 10, 50, and 100-query workloads; +- alerts for planning latency, candidate growth, timeout, and fallback rate. + +## 8. Non-goals + +- Moving deployment placement into ASAPPlanner. +- Making the DAG-viewer JSON the control-plane/data-plane protocol. +- Using viewer/export node IDs as persistent materialization identity. +- Removing archive fallback. +- Enabling SQL without a real schema catalog and binder. +- Deleting the legacy planner before shadow validation and rollback are in + place. + +## 9. Completion criteria + +The migration is complete when: + +- all production queries enter one workload-aware ASAPPlanner path; +- the selected post-ASAP plan is the sole source of `BackendPlan` decisions; +- shared sub-DAGs remain shared through materialization and serving; +- the data plane does not independently select summary families or params; +- q1/q2/q3/q4/q6 pass cross-repository end-to-end tests; +- explain output maps every selected post-ASAP replacement explicitly to its + pre-ASAP target; +- the legacy capability-union and per-root implementation planners have been + removed; +- deployment placement, collector configuration, routing, execution, and + archive fallback remain owned by ASAPQuery-backend. From e9b3f25330197ba9a5f3751a2011aab635e2e885 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 26 Aug 2026 11:27:55 -0600 Subject: [PATCH 02/12] docs: design recurring rule workload planning --- ...-asapplanner-workload-planner-migration.md | 378 +++++++++++++++++- 1 file changed, 373 insertions(+), 5 deletions(-) diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index edc65e78..e7ae5bcf 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -71,7 +71,7 @@ families or parameters for the same query. ```text WorkloadPlanRequest - queries: [{id, language, text, accuracy, schema_refs}] + queries: [{id, language, text, accuracy, schema_refs, recurrence?}] table_schemas: [...] | v @@ -215,6 +215,7 @@ Requirements: - stable, caller-visible query IDs; - one typed `AccuracyTarget` per query, with an optional workload default; +- an optional explicit recurrence and workload source for scheduled workloads; - PromQL lowering first, retaining an explicit language field; - table schemas and schema binding before enabling SQL; - request limits and validation; @@ -386,7 +387,361 @@ After one release cycle without unexplained mismatches: caller requires it; - keep only the single-query API adapters, not a single-query planner. -## 6. Cross-repository golden workload +## 6. Recurring workloads and Prometheus rules + +Repeated execution is a workload property, not an execution loop inside the +query planner. ASAPPlanner should compile a recurring query into a reusable +temporal plan. A scheduler, ruler, or materializer triggers that plan at each +evaluation timestamp. + +The recommended first deployment is a **materializer**: ASAP accelerates and +materializes the expensive numeric expression while Prometheus remains the +authority for rule scheduling, pending/firing state, `for`, +`keep_firing_for`, and Alertmanager integration. + +```text +Prometheus rule files + | + v +Rule importer ---> normalized recurring workload + | interval, offset, phase, expression, correctness + v +ASAPPlanner -----> shared temporal aggregations + | window, retention, labels, summary/error policy + v +ASAP QueryEngine + | + +----> optional ASAP-aware ruler + | + +----> materializer writes derived series to Prometheus + | + v + Prometheus evaluates alert state + (`for`, `keep_firing_for`, Alertmanager) +``` + +Prometheus rule groups have semantics that a simple cron loop does not +provide: rules in a group use the same evaluation timestamp, execute +sequentially, and skip scheduled evaluations while the previous group +evaluation is still running. The rule importer and trigger service must +preserve these semantics. See the +[Prometheus recording and alerting rule documentation](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/). + +Historical ASAPQuery planner code is useful migration input here: it infers +`repetition_delay_ms` from the median query-log inter-arrival time +([frequency.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/query_log/frequency.rs)) +and exposes per-query-group repetition configuration +([input.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/config/input.rs)). +That inference remains useful for ad hoc dashboards. A declared rule schedule, +however, is authoritative and must not be replaced by frequency inference. + +### 6.1 Recurrence and correctness in the workload model + +Add an optional recurrence to a query intent: + +```rust +struct Recurrence { + interval_ms: u64, + phase_ms: u64, + query_offset_ms: u64, + missed_tick_policy: MissedTickPolicy, + concurrency_key: String, +} + +enum MissedTickPolicy { + Skip, +} + +struct QueryIntent { + expression: String, + normalized_expression: String, + evaluation: EvaluationKind, + recurrence: Option, + correctness: CorrectnessPolicy, + source: WorkloadSource, +} +``` + +`concurrency_key` identifies the Prometheus rule group. `Skip` is the first +and initially only missed-tick behavior because it matches Prometheus rule +groups. + +Example normalized input: + +```yaml +query_groups: + - id: high-error-rate + schedule: + interval_ms: 30000 + phase_ms: 0 + query_offset_ms: 15000 + missed_tick_policy: skip + + rules: + - alert: HighErrorRate + expr: | + sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) + / + sum by (service) (rate(http_requests_total[5m])) + > 0.05 + for_ms: 300000 + keep_firing_for_ms: 60000 + correctness: exact_or_validate +``` + +Keep these time concepts separate: + +- `interval_ms` controls how often the instant expression is evaluated. +- `[5m]` in PromQL is the data lookback. +- `query_offset_ms` changes the logical evaluation timestamp, not the + interval. +- `for_ms` and `keep_firing_for_ms` belong to alert-state management, not + aggregation planning. +- A rule evaluation is an instant query even when its expression contains + range vectors such as `[5m]`. + +### 6.2 Hybrid precompute and residual planning + +Alert expressions commonly end in a comparison: + +```promql +sum(rate(errors_total[5m])) / sum(rate(requests_total[5m])) > 0.05 +``` + +The historical PromQL planner intentionally excludes comparison and set +operators from its binary-arm decomposition +([promql.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/planner/promql.rs)). +The new workload planner should not reject the complete alert because the +outer comparison is not streamable. It should separate supported expensive +leaves from an exact residual: + +```text +Precomputed leaf A: sum(rate(errors_total[5m])) +Precomputed leaf B: sum(rate(requests_total[5m])) + +Residual: A / B > 0.05 +``` + +Represent this boundary explicitly: + +```rust +struct HybridQueryPlan { + precomputed_leaves: Vec, + residual_expression: ResidualExpr, + output_labels: LabelContract, +} +``` + +The residual is a typed expression tree, not a string substitution. Its label +contract must preserve PromQL vector matching and label propagation. The +residual comparison is evaluated exactly over exact values or validated +estimate intervals. This also permits ASAP to accelerate supported inner +sub-DAGs beneath otherwise unsupported outer functions. + +### 6.3 Temporal alignment, watermark, and retention + +Extend the selected temporal aggregation with explicit alignment metadata: + +```rust +struct WindowPlan { + size_ms: u64, + anchor_epoch_ms: i64, + allowed_lateness_ms: u64, + retention_buckets: u64, +} +``` + +For a scheduler trigger at timestamp `T`: + +```text +logical evaluation time = T - query_offset +read only buckets whose watermark covers the logical evaluation time +``` + +The common anchor is required for correctness. Equal-duration buckets with +different boundaries cannot be merged as though they represented the same +logical interval. + +The historical window planner chooses tumbling windows from the repeat +interval and range-query step, and explicitly disables sliding windows because +they crash Arroyo +([window.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/planner/window.rs)). +Keep the first recurring-rule implementation conservative: + +- use anchored tumbling panes; +- require pane size to divide the evaluation interval; +- require pane size to divide the range-query step, when present; +- require pane size to be at least the scrape interval; +- require pane size not to exceed the expression lookback; +- retain enough closed panes for the maximum lookback plus allowed lateness. + +When several rules need the same aggregation at different compatible +intervals, materialize the smallest compatible pane and merge panes for the +slower rule. Do not allocate one streaming aggregation per alert interval. + +### 6.4 Alert correctness policy + +Alert planning requires a stronger correctness contract than ordinary +dashboard planning: + +```rust +enum CorrectnessPolicy { + Exact, + Approximate, + ExactOrValidate, +} +``` + +For `ExactOrValidate`, propagate an error interval through the residual +expression and compare the complete interval with the alert threshold: + +```text +interval entirely above threshold -> true +interval entirely below threshold -> false +interval overlaps threshold -> exact fallback +``` + +For example, an estimate of `5.4% +/- 0.2%` is safely above a `5.0%` +threshold because its lower bound is `5.2%`. If the interval crosses `5.0%`, +evaluate the exact expression through Prometheus/archive instead. + +Only deterministic or otherwise policy-approved bounds may make an alert +decision directly. A sketch without an applicable bound must use exact ASAP +state, validation fallback, or Prometheus evaluation. The fallback result and +its logical evaluation timestamp must be recorded so one uncertain estimate +does not accidentally reset or advance a Prometheus `for` interval. + +### 6.5 Keep scheduling outside ASAPPlanner + +The planner remains deterministic and free of timers: + +```text +plan(workload, schemas, existing_plan, statistics) -> PlanDiff +``` + +A scheduler, ruler, or materializer owns time: + +1. Determine the group evaluation timestamp. +2. Apply `query_offset_ms` to obtain logical evaluation time. +3. Prevent overlapping evaluations for the same `concurrency_key`. +4. Invoke QueryEngine with the selected plan and timestamp. +5. Evaluate the exact/validated residual. +6. Materialize a result or update alert state. + +#### Materialize into Prometheus first + +At each scheduled timestamp, evaluate the accelerated numeric expression and +remote-write a derived series such as: + +```promql +asap:high_error_rate:ratio{service="checkout"} 0.073 +``` + +Generate or configure the Prometheus alert as: + +```yaml +- alert: HighErrorRate + expr: asap:high_error_rate:ratio > 0.05 + for: 5m + keep_firing_for: 1m +``` + +This preserves Prometheus rule reload, group ordering, labels, annotations, +pending/firing state, limits, and Alertmanager integration. The materialized +sample must be committed before the corresponding Prometheus evaluation. A +group `query_offset` can provide a data-availability margin; Prometheus +documents this as a use case for +[rule query offset](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule-query-offset). + +#### ASAP-aware ruler later + +An ASAP-aware ruler can call an instant-query endpoint directly and avoid +intermediate series. It must first implement or reuse Prometheus-compatible +group scheduling, missed-iteration handling, label/annotation templates, +state persistence, `for`, `keep_firing_for`, limits, reload behavior, and +Alertmanager delivery. This is intentionally not the first milestone. + +### 6.6 Incremental replanning and rule reload + +The historical input model already contains `existing_streaming_config` and +`existing_inference_config`, although they are reserved rather than acted on +([input.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/config/input.rs)). +Use the equivalent current-plan input to produce a versioned diff: + +```rust +struct PlanDiff { + reuse: Vec, + add: Vec, + resize: Vec, + retire: Vec, +} +``` + +Use two identities rather than conflating logical reuse with one physical +revision: + +```text +LogicalAggregationId = hash( + normalized leaf expression, + metric and filters, + grouping labels, + statistic, + window size and anchor +) + +MaterializationFingerprint = hash( + LogicalAggregationId, + summary family and parameters, + grouping layout, + physical format version +) +``` + +This lets a parameter change be represented as `resize` instead of appearing +as an unrelated logical aggregation, while each physical state still has a +stable content-addressed fingerprint. + +Safe rule reload is: + +1. Parse and validate the complete new rule set. +2. Generate `PlanDiff` against the active version. +3. Reuse unchanged aggregations and start additions/resizes. +4. Warm new aggregations for their maximum lookback. +5. Atomically switch query/rule mappings at a group evaluation boundary. +6. Retire unreferenced physical states after their retention horizon. + +### 6.7 Recurring-workload delivery track + +Build this after the base workload planner in Section 5 can produce and serve +a selected `BackendPlan`: + +1. Add a Prometheus rule-file importer and explicit `Recurrence`. +2. Canonicalize PromQL ASTs and deduplicate equivalent precompute leaves. +3. Add comparison-aware hybrid planning and typed residual expressions. +4. Add window anchoring, watermark, lateness, and retention metadata. +5. Implement exact-only materialization back into Prometheus. +6. Add `ExactOrValidate` for summaries with usable bounds. +7. Implement current-plan diffs, warm cutovers, and evaluation-boundary swaps. +8. Only then consider an ASAP-native ruler. + +The conceptual change is: + +```text +Current: +query string + inferred repetition delay + -> static aggregation config + +Target: +scheduled query intent + correctness contract + -> shared temporal aggregation plan + + exact/validated residual expression + + incremental deployment plan +``` + +The same model generalizes to recurring dashboards, scheduled SQL reports, +SLO evaluation, and periodic anomaly detection. + +## 7. Cross-repository golden workload Use the ASAPPlanner DAG-viewer demo queries as shared regression fixtures: @@ -412,7 +767,7 @@ Tests are required at four boundaries: 3. protobuf -> data-plane hot reload and `RoutingIndex`; 4. ingest -> plan push -> warm query response, including archive fallback. -## 7. Complexity and safety limits +## 8. Complexity and safety limits ASAPPlanner stores alternatives in memo groups rather than enumerating the Cartesian product of whole plans. This avoids exponential copying of complete @@ -438,17 +793,18 @@ Protect the control plane with: - benchmarks for 1, 10, 50, and 100-query workloads; - alerts for planning latency, candidate growth, timeout, and fallback rate. -## 8. Non-goals +## 9. Non-goals - Moving deployment placement into ASAPPlanner. - Making the DAG-viewer JSON the control-plane/data-plane protocol. - Using viewer/export node IDs as persistent materialization identity. - Removing archive fallback. - Enabling SQL without a real schema catalog and binder. +- Reimplementing Prometheus scheduling or alert state in ASAPPlanner. - Deleting the legacy planner before shadow validation and rollback are in place. -## 9. Completion criteria +## 10. Completion criteria The migration is complete when: @@ -463,3 +819,15 @@ The migration is complete when: removed; - deployment placement, collector configuration, routing, execution, and archive fallback remain owned by ASAPQuery-backend. + +Recurring-rule support is complete only when: + +- rule schedules override frequency inference; +- evaluation interval, query offset, lookback, and alert-state durations stay + distinct throughout the plan; +- temporal panes have an explicit common anchor and watermark contract; +- hybrid plans execute precomputed leaves plus an exact or validated residual; +- uncertain approximate results fall back without corrupting alert state; +- rule reload uses a warm, versioned `PlanDiff` cutover; +- the initial production path materializes into Prometheus while Prometheus + retains alert-state authority. From 4731f0d583edbd319c2f7ad92738cae398f3d42e Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 26 Aug 2026 11:51:16 -0600 Subject: [PATCH 03/12] docs: reuse existing ASAPPlanner workload types --- ...-asapplanner-workload-planner-migration.md | 270 +++++++++--------- 1 file changed, 135 insertions(+), 135 deletions(-) diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index e7ae5bcf..f91f1911 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -70,9 +70,9 @@ families or parameters for the same query. ## 3. Target architecture ```text -WorkloadPlanRequest - queries: [{id, language, text, accuracy, schema_refs, recurrence?}] - table_schemas: [...] +HTTP/YAML adapter + -> ASAPPlanner QueryWorkload + BatchEntry / RepeatingEntry / QueryRequirements | v parse and schema-bind every query @@ -118,6 +118,18 @@ that point. control plane. - `BackendStageConfig -> BackendPlan` as the canonical source of planning decisions. It can remain temporarily as a legacy compatibility adapter. +- The placeholder `types_v2::WorkloadPlan`, `BindingName`, and + `QueryExprPlaceholder`; ASAPPlanner already represents the workload as + canonical roots and performs real `Rc`-identity CSE without `Ref` or + `LetBinding` placeholders. +- The backend-local `types_v2::QueryLanguage`; use ASAPPlanner's workload + language type. +- Backend-local one-shot/periodic query-shape modeling where it duplicates + `BatchEntry` versus `RepeatingEntry`. +- The duplicate `DataDistribution` and overlapping statistical fields in + `WorkloadCharacteristics`; populate ASAPPlanner `DataCharacteristics` and + keep only deployment-only constraints such as collector memory budget in + the placement layer. Delete these paths only after the new workload path is the production source of `BackendPlan`. During migration they remain available for shadow comparison @@ -139,12 +151,12 @@ and rollback. ### 4.3 Add -- A typed batch workload request. +- Thin HTTP/YAML adapters into ASAPPlanner's existing `QueryWorkload` model. - A deployment-owned `SelectedWorkloadPlan` representation. - A materializer from ASAPPlanner's global selection into the selected post-ASAP DAG and then into `BackendPlan`. -- An explain/debug representation with explicit pre-node, decision, and - post-node mappings. +- An explain/debug endpoint over ASAPPlanner's existing DAG export and + replacement-explanation types. - Planning phase timings, search-size metrics, deadlines, and cancellation. ## 5. Migration stack @@ -185,6 +197,20 @@ Adapt the ASAPQuery cost model: available; - expose numeric `estimate_cost` values for observability. +Use ASAPPlanner's `AccuracyTarget` as the only correctness/accuracy input +model. During compatibility migration, `control_plane::types_v2` may re-export +that upstream type, but ASAPQuery must not define a second semantic equivalent. +Delete after API migration: + +- the legacy `accuracy_sla: f64` fields and their `1.0 - accuracy_sla` + conversions; +- the data-plane routing `AccuracyTarget::{Exact, Approximate}` enum; +- any backend-local `CorrectnessPolicy` proposal. + +Routing, placement, and execution should consume the selected plan and the +original upstream `AccuracyTarget`, not collapse it to an exact/approximate +boolean. + Acceptance criteria: - `cargo build --workspace` succeeds; @@ -192,9 +218,10 @@ Acceptance criteria: - no ASAPPlanner crate is pinned to a different revision; - the compatibility path does not change production output yet. -### PR 2: Canonical batch workload input +### PR 2: Adopt ASAPPlanner's canonical workload input -Add a new API such as `POST /api/v1/workloads/plan`: +Add a new API such as `POST /api/v1/workloads/plan`. Its JSON is an API DTO, +not another planner-domain model: ```json { @@ -214,16 +241,24 @@ Add a new API such as `POST /api/v1/workloads/plan`: Requirements: - stable, caller-visible query IDs; -- one typed `AccuracyTarget` per query, with an optional workload default; -- an optional explicit recurrence and workload source for scheduled workloads; -- PromQL lowering first, retaining an explicit language field; -- table schemas and schema binding before enabling SQL; +- map every accuracy field directly into + `QueryRequirements.accuracy: Option`; +- map one-shot queries into `BatchEntry` and scheduled queries into + `RepeatingEntry`; +- use ASAPPlanner's `QueryLanguage` rather than the backend's local language + enum; +- group mixed-language API requests into one ASAPPlanner `QueryWorkload` per + language until upstream supports mixed languages in one workload; +- pass table schemas through ASAPPlanner's existing `SchemaCatalog` binder + before enabling SQL; - request limits and validation; - old single-query endpoints adapt into a one-query workload rather than maintaining a second planner. -The result of this phase is a vector of named `Rc` roots, not a -`QuerySetPlan` of flattened capabilities. +ASAPPlanner already owns canonicalization and workload CSE. The backend should +invoke those existing paths and preserve the resulting named +`Rc` roots; it must not implement its own AST normalization, +semantic equality, or `QuerySetPlan` capability flattening. Acceptance criteria: @@ -342,10 +377,11 @@ Add an optional explain response or endpoint containing: - edge schemas; - phase timings. -The shape may reuse ASAPPlanner's `dag_export` domain types where they are -appropriate, but production planning must call Rust APIs directly. It must -not start the Python viewer server or invoke the `dag_export` binary as a -subprocess. +Use ASAPPlanner's existing `DagGraph`, `DagDecision`, `TargetReplacement`, +`WorkloadGraph`, `export_post_asap`, and replacement-explanation APIs. Do not +define a second backend graph/decision schema. Production planning calls these +Rust APIs directly; it must not start the Python viewer server or invoke the +`dag_export` binary as a subprocess. Record at least: @@ -400,14 +436,20 @@ authority for rule scheduling, pending/firing state, `for`, `keep_firing_for`, and Alertmanager integration. ```text -Prometheus rule files +Prometheus rule files (ASAPQuery-backend boundary) | v -Rule importer ---> normalized recurring workload - | interval, offset, phase, expression, correctness - v -ASAPPlanner -----> shared temporal aggregations - | window, retention, labels, summary/error policy +ASAPQuery rule importer + |----> generic RepeatingEntry + QueryRequirements ----+ + | | + +----> backend-only scheduler/rule metadata | + offset, group ordering, alert state | + v + ASAPPlanner + | + v +ASAPQuery deployment -----> shared temporal aggregations + | window, retention, labels, summary policy v ASAP QueryEngine | @@ -435,36 +477,35 @@ and exposes per-query-group repetition configuration That inference remains useful for ad hoc dashboards. A declared rule schedule, however, is authoritative and must not be replaced by frequency inference. -### 6.1 Recurrence and correctness in the workload model +### 6.1 Use ASAPPlanner's existing repeating-workload model unchanged -Add an optional recurrence to a query intent: +ASAPPlanner already defines `QueryRequirements.accuracy: Option`, +`RepeatingEntry`, and `RepetitionInterval`. They already contain everything +replacement planning needs: the expression, repetition interval, accuracy, +and optional latency requirement. Do not add a parallel ASAPQuery workload +type and do not add `phase`, `query_offset`, missed-tick behavior, or rule-group +identity to ASAPPlanner's model. ```rust -struct Recurrence { - interval_ms: u64, - phase_ms: u64, - query_offset_ms: u64, - missed_tick_policy: MissedTickPolicy, - concurrency_key: String, +struct RepeatingEntry { + query: Query, + interval: RepetitionInterval, + requirements: Option, } +``` -enum MissedTickPolicy { - Skip, -} +The Prometheus importer returns two linked outputs: -struct QueryIntent { - expression: String, - normalized_expression: String, - evaluation: EvaluationKind, - recurrence: Option, - correctness: CorrectnessPolicy, - source: WorkloadSource, -} -``` +1. existing ASAPPlanner `RepeatingEntry` values for planning; +2. scheduler-only rule metadata containing group identity/order, + `query_offset`, missed-tick behavior, `for`, and `keep_firing_for`. -`concurrency_key` identifies the Prometheus rule group. `Skip` is the first -and initially only missed-tick behavior because it matches Prometheus rule -groups. +The importer and the second output belong to ASAPQuery-backend, not +ASAPPlanner. The second output is never part of `QueryWorkload`, lowering, +replacement search, post-ASAP IR, or `PlanSpace`. ASAPPlanner receives no +Prometheus rule-group concept at any layer. Normalized expression text is +derived from ASAPPlanner's canonical pre-ASAP IR and must not be stored as a +second caller-controlled truth. Example normalized input: @@ -486,7 +527,9 @@ query_groups: > 0.05 for_ms: 300000 keep_firing_for_ms: 60000 - correctness: exact_or_validate + accuracy: + epsilon: 0.002 + delta: 0.01 ``` Keep these time concepts separate: @@ -500,47 +543,10 @@ Keep these time concepts separate: - A rule evaluation is an instant query even when its expression contains range vectors such as `[5m]`. -### 6.2 Hybrid precompute and residual planning +### 6.2 Temporal alignment, watermark, and retention -Alert expressions commonly end in a comparison: - -```promql -sum(rate(errors_total[5m])) / sum(rate(requests_total[5m])) > 0.05 -``` - -The historical PromQL planner intentionally excludes comparison and set -operators from its binary-arm decomposition -([promql.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/planner/promql.rs)). -The new workload planner should not reject the complete alert because the -outer comparison is not streamable. It should separate supported expensive -leaves from an exact residual: - -```text -Precomputed leaf A: sum(rate(errors_total[5m])) -Precomputed leaf B: sum(rate(requests_total[5m])) - -Residual: A / B > 0.05 -``` - -Represent this boundary explicitly: - -```rust -struct HybridQueryPlan { - precomputed_leaves: Vec, - residual_expression: ResidualExpr, - output_labels: LabelContract, -} -``` - -The residual is a typed expression tree, not a string substitution. Its label -contract must preserve PromQL vector matching and label propagation. The -residual comparison is evaluated exactly over exact values or validated -estimate intervals. This also permits ASAP to accelerate supported inner -sub-DAGs beneath otherwise unsupported outer functions. - -### 6.3 Temporal alignment, watermark, and retention - -Extend the selected temporal aggregation with explicit alignment metadata: +The deployment plan, rather than ASAPPlanner's workload model, records runtime +alignment metadata: ```rust struct WindowPlan { @@ -551,11 +557,13 @@ struct WindowPlan { } ``` -For a scheduler trigger at timestamp `T`: +For a scheduler trigger at timestamp `T`, the scheduler invokes QueryEngine at +the already-offset logical timestamp: ```text logical evaluation time = T - query_offset -read only buckets whose watermark covers the logical evaluation time +QueryEngine receives logical evaluation time +runtime reads only buckets whose watermark covers that time ``` The common anchor is required for correctness. Equal-duration buckets with @@ -579,39 +587,28 @@ When several rules need the same aggregation at different compatible intervals, materialize the smallest compatible pane and merge panes for the slower rule. Do not allocate one streaming aggregation per alert interval. -### 6.4 Alert correctness policy +### 6.3 AccuracyTarget and KeepPreAsap are sufficient -Alert planning requires a stronger correctness contract than ordinary -dashboard planning: +Do not add `CorrectnessPolicy::{Exact, Approximate, ExactOrValidate}`. +ASAPPlanner's existing `AccuracyTarget` is the single source of truth: ```rust -enum CorrectnessPolicy { - Exact, - Approximate, - ExactOrValidate, -} -``` - -For `ExactOrValidate`, propagate an error interval through the residual -expression and compare the complete interval with the alert threshold: - -```text -interval entirely above threshold -> true -interval entirely below threshold -> false -interval overlaps threshold -> exact fallback +AccuracyTarget::Exact +AccuracyTarget::Epsilon(epsilon) +AccuracyTarget::EpsilonDelta { epsilon, delta } ``` -For example, an estimate of `5.4% +/- 0.2%` is safely above a `5.0%` -threshold because its lower bound is `5.2%`. If the interval crosses `5.0%`, -evaluate the exact expression through Prometheus/archive instead. +`Exact` excludes approximate candidates. A node for which no valid exact ASAP +replacement exists remains `KeepPreAsap`, which means ASAPQuery executes the +original pre-ASAP subtree from raw/archive data. `Epsilon` and +`EpsilonDelta` allow ASAPPlanner to choose a summary sized to that target. -Only deterministic or otherwise policy-approved bounds may make an alert -decision directly. A sketch without an applicable bound must use exact ASAP -state, validation fallback, or Prometheus evaluation. The fallback result and -its logical evaluation timestamp must be recorded so one uncertain estimate -does not accidentally reset or advance a Prometheus `for` interval. +No `ResidualExpr`, `GuardedResult`, interval-propagation layer, or conditional +exact-fallback policy is needed for this migration. ASAPQuery consumes +`SummaryNode` replacements and executes `KeepPreAsap` exactly; it must not +reinterpret the accuracy requirement or define a competing correctness enum. -### 6.5 Keep scheduling outside ASAPPlanner +### 6.4 Keep scheduling outside ASAPPlanner The planner remains deterministic and free of timers: @@ -623,10 +620,10 @@ A scheduler, ruler, or materializer owns time: 1. Determine the group evaluation timestamp. 2. Apply `query_offset_ms` to obtain logical evaluation time. -3. Prevent overlapping evaluations for the same `concurrency_key`. +3. Prevent overlapping evaluations for the same Prometheus rule group. 4. Invoke QueryEngine with the selected plan and timestamp. -5. Evaluate the exact/validated residual. -6. Materialize a result or update alert state. +5. Execute selected summary nodes and any `KeepPreAsap` subtree. +6. Materialize the result or update alert state. #### Materialize into Prometheus first @@ -661,7 +658,7 @@ group scheduling, missed-iteration handling, label/annotation templates, state persistence, `for`, `keep_firing_for`, limits, reload behavior, and Alertmanager delivery. This is intentionally not the first milestone. -### 6.6 Incremental replanning and rule reload +### 6.5 Incremental replanning and rule reload The historical input model already contains `existing_streaming_config` and `existing_inference_config`, although they are reserved rather than acted on @@ -710,19 +707,21 @@ Safe rule reload is: 5. Atomically switch query/rule mappings at a group evaluation boundary. 6. Retire unreferenced physical states after their retention horizon. -### 6.7 Recurring-workload delivery track +### 6.6 Recurring-workload delivery track Build this after the base workload planner in Section 5 can produce and serve a selected `BackendPlan`: -1. Add a Prometheus rule-file importer and explicit `Recurrence`. -2. Canonicalize PromQL ASTs and deduplicate equivalent precompute leaves. -3. Add comparison-aware hybrid planning and typed residual expressions. -4. Add window anchoring, watermark, lateness, and retention metadata. -5. Implement exact-only materialization back into Prometheus. -6. Add `ExactOrValidate` for summaries with usable bounds. -7. Implement current-plan diffs, warm cutovers, and evaluation-boundary swaps. -8. Only then consider an ASAP-native ruler. +1. In ASAPQuery-backend, add a Prometheus rule-file importer that emits + existing generic `RepeatingEntry` values plus separate backend scheduler + metadata; make no ASAPPlanner rule-group model change. +2. Feed imported expressions through ASAPPlanner's existing canonicalization + and workload CSE. +3. Add deployment window anchoring, watermark, lateness, and retention + metadata. +4. Materialize exact or selected approximate results back into Prometheus. +5. Implement current-plan diffs, warm cutovers, and evaluation-boundary swaps. +6. Only then consider an ASAP-native ruler. The conceptual change is: @@ -732,9 +731,8 @@ query string + inferred repetition delay -> static aggregation config Target: -scheduled query intent + correctness contract +ASAPPlanner RepeatingEntry + AccuracyTarget -> shared temporal aggregation plan - + exact/validated residual expression + incremental deployment plan ``` @@ -812,6 +810,8 @@ The migration is complete when: - the selected post-ASAP plan is the sole source of `BackendPlan` decisions; - shared sub-DAGs remain shared through materialization and serving; - the data plane does not independently select summary families or params; +- ASAPPlanner's `AccuracyTarget` is the only accuracy model and legacy + `accuracy_sla`/local exact-vs-approximate enums have been removed; - q1/q2/q3/q4/q6 pass cross-repository end-to-end tests; - explain output maps every selected post-ASAP replacement explicitly to its pre-ASAP target; @@ -823,11 +823,11 @@ The migration is complete when: Recurring-rule support is complete only when: - rule schedules override frequency inference; -- evaluation interval, query offset, lookback, and alert-state durations stay - distinct throughout the plan; +- only the generic evaluation interval enters ASAPPlanner; query offset and + alert-state durations remain scheduler metadata; - temporal panes have an explicit common anchor and watermark contract; -- hybrid plans execute precomputed leaves plus an exact or validated residual; -- uncertain approximate results fall back without corrupting alert state; +- `AccuracyTarget::Exact` plans either select exact ASAP summaries or execute + `KeepPreAsap` from raw/archive data; - rule reload uses a warm, versioned `PlanDiff` cutover; - the initial production path materializes into Prometheus while Prometheus retains alert-state authority. From 39d8083564002000140677c157690c8e7f56bf54 Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 26 Aug 2026 11:53:12 -0600 Subject: [PATCH 04/12] docs: isolate Prometheus query and rule adapters --- ...-asapplanner-workload-planner-migration.md | 126 +++++++++++++----- 1 file changed, 92 insertions(+), 34 deletions(-) diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index f91f1911..575fa370 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -70,7 +70,8 @@ families or parameters for the same query. ## 3. Target architecture ```text -HTTP/YAML adapter +Protocol/source adapters + -> ASAPQuery O11yMetricsQuery -> ASAPPlanner QueryWorkload BatchEntry / RepeatingEntry / QueryRequirements | @@ -78,7 +79,7 @@ HTTP/YAML adapter parse and schema-bind every query | v - Vec<(QueryId, Rc)> + Vec<(CallerId, Rc)> | v ASAPPlanner search_workload_with @@ -151,7 +152,10 @@ and rollback. ### 4.3 Add -- Thin HTTP/YAML adapters into ASAPPlanner's existing `QueryWorkload` model. +- An adapter-neutral ASAPQuery `O11yMetricsQuery` ingestion model. +- Independent source/protocol adapters, beginning with Prometheus query and + rule adapters, that produce `O11yMetricsQuery` values. +- A thin `O11yMetricsQuery -> ASAPPlanner QueryWorkload` conversion. - A deployment-owned `SelectedWorkloadPlan` representation. - A materializer from ASAPPlanner's global selection into the selected post-ASAP DAG and then into `BackendPlan`. @@ -220,8 +224,10 @@ Acceptance criteria: ### PR 2: Adopt ASAPPlanner's canonical workload input -Add a new API such as `POST /api/v1/workloads/plan`. Its JSON is an API DTO, -not another planner-domain model: +Add a new API such as `POST /api/v1/workloads/plan`. Its JSON is decoded by a +source adapter into ASAPQuery's adapter-neutral `O11yMetricsQuery`, then +converted into ASAPPlanner's existing workload types. Neither layer is a +second planner-domain model: ```json { @@ -255,6 +261,51 @@ Requirements: - old single-query endpoints adapt into a one-query workload rather than maintaining a second planner. +Use a module boundary such as: + +```text +control_plane/src/o11y_query/ + mod.rs # O11yMetricsQuery and generic conversion + adapters/ + mod.rs # adapter trait/error contract + prometheus/ + query.rs # Prometheus instant/range request adapter + rules.rs # Prometheus rule-file adapter + runtime.rs # Prometheus-only scheduling/alert metadata +``` + +Keep this as an independent dependency boundary even if it initially lives in +the `control_plane` crate: + +```text +adapters/prometheus -> O11yMetricsQuery -> ASAPPlanner workload types +ASAPPlanner -X-> adapters/prometheus +planner/search -X-> Prometheus rule/runtime types +``` + +It can move into a workspace adapter crate later without changing planner or +deployment APIs. + +`O11yMetricsQuery` contains only source-independent query information and +reuses ASAPPlanner vocabulary for language, requirements, accuracy, and repeat +interval. It must not copy `QueryExpr`, `AccuracyTarget`, or `QueryLanguage` +into new local enums: + +```rust +struct O11yMetricsQuery { + id: Id, + query: Query, + language: QueryLanguage, + requirements: Option, + repetition: Option, +} +``` + +The generic converter maps `repetition: None` to `BatchEntry` and +`Some(interval)` to `RepeatingEntry`. Caller IDs remain beside the workload +entries and become the generic IDs passed to CSE/search; they do not require a +second planner `QueryId` type. + ASAPPlanner already owns canonicalization and workload CSE. The backend should invoke those existing paths and preserve the resulting named `Rc` roots; it must not implement its own AST normalization, @@ -436,19 +487,21 @@ authority for rule scheduling, pending/firing state, `for`, `keep_firing_for`, and Alertmanager integration. ```text -Prometheus rule files (ASAPQuery-backend boundary) - | - v -ASAPQuery rule importer - |----> generic RepeatingEntry + QueryRequirements ----+ - | | - +----> backend-only scheduler/rule metadata | - offset, group ordering, alert state | - v - ASAPPlanner - | - v -ASAPQuery deployment -----> shared temporal aggregations +Prometheus instant/range queries ----+ + | +Prometheus rule files ---------------+--> adapters/prometheus + | | + | +--> Prometheus-only + | runtime metadata + | (adapter boundary) + v + O11yMetricsQuery + | + v + ASAPPlanner QueryWorkload + | + v +ASAPQuery deployment ----------> shared temporal aggregations | window, retention, labels, summary policy v ASAP QueryEngine @@ -494,18 +547,21 @@ struct RepeatingEntry { } ``` -The Prometheus importer returns two linked outputs: +The Prometheus rules adapter returns two linked outputs: -1. existing ASAPPlanner `RepeatingEntry` values for planning; +1. generic `O11yMetricsQuery` values, subsequently converted into existing + ASAPPlanner `RepeatingEntry` values; 2. scheduler-only rule metadata containing group identity/order, `query_offset`, missed-tick behavior, `for`, and `keep_firing_for`. -The importer and the second output belong to ASAPQuery-backend, not -ASAPPlanner. The second output is never part of `QueryWorkload`, lowering, -replacement search, post-ASAP IR, or `PlanSpace`. ASAPPlanner receives no -Prometheus rule-group concept at any layer. Normalized expression text is -derived from ASAPPlanner's canonical pre-ASAP IR and must not be stored as a -second caller-controlled truth. +Both the Prometheus query adapter and rules adapter live under the same +independent `adapters/prometheus` boundary. Adapter output (1) crosses into +ASAPQuery core only as `O11yMetricsQuery`; output (2) remains in the +Prometheus adapter/runtime integration. It never enters `O11yMetricsQuery`, +`QueryWorkload`, lowering, replacement search, post-ASAP IR, or `PlanSpace`. +ASAPPlanner receives no Prometheus query-protocol or rule-group concept at any +layer. Normalized expression text is derived from ASAPPlanner's canonical +pre-ASAP IR and must not be stored as a second caller-controlled truth. Example normalized input: @@ -712,16 +768,18 @@ Safe rule reload is: Build this after the base workload planner in Section 5 can produce and serve a selected `BackendPlan`: -1. In ASAPQuery-backend, add a Prometheus rule-file importer that emits - existing generic `RepeatingEntry` values plus separate backend scheduler - metadata; make no ASAPPlanner rule-group model change. -2. Feed imported expressions through ASAPPlanner's existing canonicalization +1. Add the adapter-neutral `O11yMetricsQuery` model and its direct conversion + into ASAPPlanner `BatchEntry`/`RepeatingEntry` workloads. +2. Add an independent `adapters/prometheus` module with query and rule-file + adapters. Emit `O11yMetricsQuery` plus separately contained Prometheus + runtime metadata; make no ASAPPlanner Prometheus model change. +3. Feed adapted expressions through ASAPPlanner's existing canonicalization and workload CSE. -3. Add deployment window anchoring, watermark, lateness, and retention +4. Add deployment window anchoring, watermark, lateness, and retention metadata. -4. Materialize exact or selected approximate results back into Prometheus. -5. Implement current-plan diffs, warm cutovers, and evaluation-boundary swaps. -6. Only then consider an ASAP-native ruler. +5. Materialize exact or selected approximate results back into Prometheus. +6. Implement current-plan diffs, warm cutovers, and evaluation-boundary swaps. +7. Only then consider an ASAP-native ruler. The conceptual change is: From fcbce314786ef64919a692c6ce0268ad9bfb9e2c Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 26 Aug 2026 12:00:05 -0600 Subject: [PATCH 05/12] docs: clarify planner backend and adapter ownership --- ...-asapplanner-workload-planner-migration.md | 177 ++++++++++++++---- 1 file changed, 141 insertions(+), 36 deletions(-) diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index 575fa370..ebc379ce 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -38,6 +38,60 @@ The boundary is intentional: ASAPPlanner decides *what a query sub-DAG may be replaced with*; ASAPQuery decides *where the selected replacement runs, how it is represented on the wire, and how it is served*. +### 1.1 Ownership after reviewing the current code + +Add or complete in **ASAPPlanner**: + +- Lower `RepeatingEntry` workloads in the PromQL and SQL frontends. The + workload type exists today, but the batch frontend helpers currently consume + only `query_batch`. +- Provide one generic workload-lowering facade that preserves caller IDs and + returns named canonical roots, without introducing protocol concepts. +- Keep canonicalization, schema binding, workload CSE, replacement strategies, + candidate search, cost ranking, and global selection upstream. +- Add a public, deployment-neutral operation that applies a + `GlobalSelection` to all workload roots and returns the complete selected + post-ASAP workload DAG with explicit target/decision provenance. Do not make + each downstream consumer reproduce the substitution code used by + `dag_export`. +- Keep `AccuracyTarget`, `QueryRequirements`, `BatchEntry`, `RepeatingEntry`, + `QueryLanguage`, `DataCharacteristics`, pre/post-ASAP schemas, and DAG + explain/export types canonical upstream. +- If recurring-query interval ever changes candidate costing or enables + generic cross-interval reuse, implement that as a protocol-neutral workload + strategy over `RepeatingEntry`. Do not add Prometheus schedules or rule + groups. +- If logical plan identity/diffing is needed by more than one deployment, add + a canonical semantic ID and logical selected-plan diff upstream. Physical + rollout remains downstream. + +Add or retain in **ASAPQuery control plane**: + +- `O11yMetricsQuery` ingestion and source adapters. +- A deployment cost-model implementation using runtime statistics and budgets. +- Executor-capability validation for selected post-ASAP shapes. +- Placement across collector/backend/archive, physical fingerprints, + `BackendPlan`, collector configuration, and routing entries. +- Physical `DeploymentPlanDiff`, warm-up, atomic route switch, retirement, and + replanning triggers. +- Explain HTTP endpoints and planning telemetry, using upstream explain types. + +Retain in **ASAPQuery data plane/runtime**: + +- Execution of selected `SummaryNode` and `KeepPreAsap` trees. +- `BackendPlan` hot reload, `RoutingIndex`, storage lookup, watermarks, + lateness/readiness checks, and archive fallback. +- Protocol response formatting and the existing Prometheus HTTP serving + adapter. + +Keep in **source/protocol adapters**, outside planner and deployment core: + +- Prometheus HTTP query request parsing. +- Prometheus rule-file parsing and rule-group scheduling metadata. +- `query_offset`, missed iterations, ordering, `for`, `keep_firing_for`, label + and annotation templates, and Alertmanager integration. +- Translation into protocol-neutral `O11yMetricsQuery` values. + ## 2. Why the legacy query planner should be removed ASAPQuery currently has several overlapping planning paths: @@ -88,10 +142,10 @@ Protocol/source adapters PlanSpace::global_selection | v - ASAPQuery SelectedWorkloadPlan + ASAPPlanner SelectedPostAsapWorkload / \ v v - ExplainPlan deployment placement + Explain view ASAPQuery DeploymentPlan | v BackendPlan @@ -156,9 +210,9 @@ and rollback. - Independent source/protocol adapters, beginning with Prometheus query and rule adapters, that produce `O11yMetricsQuery` values. - A thin `O11yMetricsQuery -> ASAPPlanner QueryWorkload` conversion. -- A deployment-owned `SelectedWorkloadPlan` representation. -- A materializer from ASAPPlanner's global selection into the selected - post-ASAP DAG and then into `BackendPlan`. +- An ASAPQuery deployment-plan representation built from ASAPPlanner's + upstream-materialized selected post-ASAP workload DAG. +- A materializer from that selected DAG into placement and `BackendPlan`. - An explain/debug endpoint over ASAPPlanner's existing DAG export and replacement-explanation types. - Planning phase timings, search-size metrics, deadlines, and cancellation. @@ -286,25 +340,58 @@ planner/search -X-> Prometheus rule/runtime types It can move into a workspace adapter crate later without changing planner or deployment APIs. -`O11yMetricsQuery` contains only source-independent query information and -reuses ASAPPlanner vocabulary for language, requirements, accuracy, and repeat -interval. It must not copy `QueryExpr`, `AccuracyTarget`, or `QueryLanguage` -into new local enums: +`O11yMetricsQuery` contains only source-independent identity plus one existing +ASAPPlanner workload entry. It must not flatten and copy the fields of +`BatchEntry`/`RepeatingEntry`, or copy `QueryExpr`, `AccuracyTarget`, or +`QueryLanguage` into new local enums: ```rust struct O11yMetricsQuery { id: Id, - query: Query, language: QueryLanguage, - requirements: Option, - repetition: Option, + entry: O11yWorkloadEntry, +} + +enum O11yWorkloadEntry { + OneShot(BatchEntry), + Repeating(RepeatingEntry), +} +``` + +The generic converter groups entries by `QueryLanguage`, moves their existing +entry values into `QueryWorkload`, and keeps caller IDs beside them for +lowering/CSE/search. It performs no query parsing, canonicalization, accuracy +conversion, or schedule interpretation. Caller IDs do not require a second +planner `QueryId` type. + +Define the adapter contract around an input and two deliberately separated +outputs: + +```rust +trait O11yMetricsAdapter { + type PrivateMetadata; + + fn adapt( + &self, + input: Input, + ) -> Result, AdapterError>; +} + +struct AdaptedMetricsInput { + queries: Vec>, + private_metadata: M, } ``` -The generic converter maps `repetition: None` to `BatchEntry` and -`Some(interval)` to `RepeatingEntry`. Caller IDs remain beside the workload -entries and become the generic IDs passed to CSE/search; they do not require a -second planner `QueryId` type. +Only `queries` crosses into planning. `private_metadata` remains owned by the +adapter/runtime integration and is indexed by caller ID. For an ordinary +Prometheus query it can be empty; for a rule file it contains rule groups and +alerting semantics. + +Do not extend the existing data-plane `QueryRequestAdapter` for this purpose. +That trait is an Axum/HTTP serving adapter coupled to execution timestamps and +`QueryResult` response formatting. Planning ingestion adapters belong in the +control-plane boundary and share only protocol parsing utilities where useful. ASAPPlanner already owns canonicalization and workload CSE. The backend should invoke those existing paths and preserve the resulting named @@ -318,7 +405,18 @@ Acceptance criteria: - equivalent subtrees remain shareable across roots; - malformed queries and schemas return per-query diagnostics. -### PR 3: Workload-wide search and selection +### PR 3: Complete ASAPPlanner workload planning APIs + +Land the generic upstream gaps first: + +- PromQL/SQL lowering for `repeating_queries` as well as `query_batch`; +- named-root lowering that preserves caller IDs; +- materialization of `GlobalSelection` into a complete selected post-ASAP + workload DAG with explicit decision provenance. + +No ASAPQuery, Prometheus, placement, wire, or runtime types enter these APIs. + +### PR 4: Workload-wide search and deployment selection Run one planning operation for the complete workload: @@ -328,10 +426,9 @@ let space = search_workload_with(roots, &strategies); let selection = space.global_selection(&backend_cost_model); ``` -Build `SelectedWorkloadPlan` from the result while preserving shared node -identity. It must support selected `Replacement::Summary` and -`Replacement::Rewrite` candidates and recursively compose choices for -descendant targets. +Consume ASAPPlanner's materialized selected workload DAG while preserving +shared node identity. ASAPQuery must not implement its own recursive +`Replacement::Summary`/`Replacement::Rewrite` substitution engine. This phase should activate and test: @@ -356,13 +453,14 @@ Acceptance criteria: - selection is deterministic for identical workload, statistics, and cost model inputs. -### PR 4: Selected workload to BackendPlan +### PR 5: Selected workload to BackendPlan Introduce a direct conversion: ```text -SelectedWorkloadPlan +SelectedPostAsapWorkload -> deployment placement + -> DeploymentPlan -> materializations and readouts -> BackendPlan ``` @@ -391,7 +489,7 @@ Acceptance criteria: - protobuf encode/decode and hot reload preserve the chosen plan; - the data plane never has to run a cost model to reconstruct the choice. -### PR 5: Data-plane execution coverage +### PR 6: Data-plane execution coverage Make the data plane execute or explicitly reject every post-ASAP shape the control plane may select. Cover at least: @@ -416,7 +514,7 @@ Acceptance criteria: - archive fallback remains available for unsupported queries; - end-to-end tests prove plan push, hot reload, and serving. -### PR 6: Explain and planner observability +### PR 7: Explain and planner observability Add an optional explain response or endpoint containing: @@ -444,7 +542,7 @@ Record at least: - `BackendPlan` construction and push time; - target, candidate, shared-node, and selected-materialization counts. -### PR 7: Shadow rollout and legacy deletion +### PR 8: Shadow rollout and legacy deletion Introduce a feature flag such as `ASAP_WORKLOAD_PLANNER_V2` and use three rollout modes: @@ -669,7 +767,11 @@ reinterpret the accuracy requirement or define a competing correctness enum. The planner remains deterministic and free of timers: ```text -plan(workload, schemas, existing_plan, statistics) -> PlanDiff +ASAPPlanner: +select(workload, schemas, statistics) -> SelectedPostAsapWorkload + +ASAPQuery control plane: +diff(active_deployment, selected_workload) -> DeploymentPlanDiff ``` A scheduler, ruler, or materializer owns time: @@ -719,19 +821,22 @@ Alertmanager delivery. This is intentionally not the first milestone. The historical input model already contains `existing_streaming_config` and `existing_inference_config`, although they are reserved rather than acted on ([input.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/config/input.rs)). -Use the equivalent current-plan input to produce a versioned diff: +ASAPQuery uses the active deployment plus the new upstream-selected workload +to produce a versioned physical diff: ```rust -struct PlanDiff { - reuse: Vec, - add: Vec, - resize: Vec, +struct DeploymentPlanDiff { + reuse: Vec, + add: Vec, + resize: Vec, retire: Vec, } ``` Use two identities rather than conflating logical reuse with one physical -revision: +revision. The logical semantic ID should come from ASAPPlanner if/when it +provides a public canonical identity; the physical fingerprint remains an +ASAPQuery deployment identity: ```text LogicalAggregationId = hash( @@ -757,7 +862,7 @@ stable content-addressed fingerprint. Safe rule reload is: 1. Parse and validate the complete new rule set. -2. Generate `PlanDiff` against the active version. +2. Generate `DeploymentPlanDiff` against the active version. 3. Reuse unchanged aggregations and start additions/resizes. 4. Warm new aggregations for their maximum lookback. 5. Atomically switch query/rule mappings at a group evaluation boundary. @@ -819,7 +924,7 @@ Add negative fixtures for: Tests are required at four boundaries: 1. query workload -> selected strategies; -2. selected workload -> `BackendPlan`; +2. `SelectedPostAsapWorkload` -> `DeploymentPlan` -> `BackendPlan`; 3. protobuf -> data-plane hot reload and `RoutingIndex`; 4. ingest -> plan push -> warm query response, including archive fallback. @@ -886,6 +991,6 @@ Recurring-rule support is complete only when: - temporal panes have an explicit common anchor and watermark contract; - `AccuracyTarget::Exact` plans either select exact ASAP summaries or execute `KeepPreAsap` from raw/archive data; -- rule reload uses a warm, versioned `PlanDiff` cutover; +- rule reload uses a warm, versioned `DeploymentPlanDiff` cutover; - the initial production path materializes into Prometheus while Prometheus retains alert-state authority. From 00d0264af7f014c7f7b411c4c4ac392e377a7b8c Mon Sep 17 00:00:00 2001 From: zz_y Date: Wed, 26 Aug 2026 12:15:17 -0600 Subject: [PATCH 06/12] docs: keep IDs and plan commitment downstream --- ...-asapplanner-workload-planner-migration.md | 67 +++++++++---------- 1 file changed, 33 insertions(+), 34 deletions(-) diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index ebc379ce..d2d59ec3 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -45,15 +45,8 @@ Add or complete in **ASAPPlanner**: - Lower `RepeatingEntry` workloads in the PromQL and SQL frontends. The workload type exists today, but the batch frontend helpers currently consume only `query_batch`. -- Provide one generic workload-lowering facade that preserves caller IDs and - returns named canonical roots, without introducing protocol concepts. - Keep canonicalization, schema binding, workload CSE, replacement strategies, candidate search, cost ranking, and global selection upstream. -- Add a public, deployment-neutral operation that applies a - `GlobalSelection` to all workload roots and returns the complete selected - post-ASAP workload DAG with explicit target/decision provenance. Do not make - each downstream consumer reproduce the substitution code used by - `dag_export`. - Keep `AccuracyTarget`, `QueryRequirements`, `BatchEntry`, `RepeatingEntry`, `QueryLanguage`, `DataCharacteristics`, pre/post-ASAP schemas, and DAG explain/export types canonical upstream. @@ -68,6 +61,9 @@ Add or complete in **ASAPPlanner**: Add or retain in **ASAPQuery control plane**: - `O11yMetricsQuery` ingestion and source adapters. +- Caller-ID correlation outside the planner workload: zip ordered lowering + results with adapter-owned IDs before passing `(Id, Rc)` roots to + search. - A deployment cost-model implementation using runtime statistics and budgets. - Executor-capability validation for selected post-ASAP shapes. - Placement across collector/backend/archive, physical fingerprints, @@ -140,12 +136,10 @@ Protocol/source adapters | v PlanSpace::global_selection - | - v - ASAPPlanner SelectedPostAsapWorkload / \ v v - Explain view ASAPQuery DeploymentPlan + upstream explain ASAPQuery DeploymentPlan + views (apply choices + placement) | v BackendPlan @@ -210,9 +204,9 @@ and rollback. - Independent source/protocol adapters, beginning with Prometheus query and rule adapters, that produce `O11yMetricsQuery` values. - A thin `O11yMetricsQuery -> ASAPPlanner QueryWorkload` conversion. -- An ASAPQuery deployment-plan representation built from ASAPPlanner's - upstream-materialized selected post-ASAP workload DAG. -- A materializer from that selected DAG into placement and `BackendPlan`. +- An ASAPQuery `DeploymentPlan` builder that consumes canonical roots plus + ASAPPlanner `GlobalSelection`, applies the chosen replacements in a + deployment-aware way, assigns placement, and emits `BackendPlan`. - An explain/debug endpoint over ASAPPlanner's existing DAG export and replacement-explanation types. - Planning phase timings, search-size metrics, deadlines, and cancellation. @@ -358,11 +352,13 @@ enum O11yWorkloadEntry { } ``` -The generic converter groups entries by `QueryLanguage`, moves their existing -entry values into `QueryWorkload`, and keeps caller IDs beside them for -lowering/CSE/search. It performs no query parsing, canonicalization, accuracy -conversion, or schedule interpretation. Caller IDs do not require a second -planner `QueryId` type. +The generic converter groups entries by `QueryLanguage` and moves their +existing entry values into `QueryWorkload`. Caller IDs remain in a parallel +adapter-owned vector/map. ASAPPlanner frontend lowering returns results in +entry order; ASAPQuery zips those results back to caller IDs before passing +`(Id, Rc)` roots to CSE/search. It performs no query parsing, +canonicalization, accuracy conversion, or schedule interpretation. Caller IDs +do not require a planner `QueryId` type or a new named-lowering API. Define the adapter contract around an input and two deliberately separated outputs: @@ -405,16 +401,15 @@ Acceptance criteria: - equivalent subtrees remain shareable across roots; - malformed queries and schemas return per-query diagnostics. -### PR 3: Complete ASAPPlanner workload planning APIs +### PR 3: Complete repeating workload lowering in ASAPPlanner -Land the generic upstream gaps first: +Land the one generic upstream gap: -- PromQL/SQL lowering for `repeating_queries` as well as `query_batch`; -- named-root lowering that preserves caller IDs; -- materialization of `GlobalSelection` into a complete selected post-ASAP - workload DAG with explicit decision provenance. +- PromQL/SQL lowering for `repeating_queries` as well as `query_batch`, with + one result per entry in input order. -No ASAPQuery, Prometheus, placement, wire, or runtime types enter these APIs. +No caller ID, ASAPQuery, Prometheus, placement, wire, or runtime types enter +these APIs. ### PR 4: Workload-wide search and deployment selection @@ -426,9 +421,13 @@ let space = search_workload_with(roots, &strategies); let selection = space.global_selection(&backend_cost_model); ``` -Consume ASAPPlanner's materialized selected workload DAG while preserving -shared node identity. ASAPQuery must not implement its own recursive -`Replacement::Summary`/`Replacement::Rewrite` substitution engine. +Build ASAPQuery's `DeploymentPlan` from the canonical roots plus +`GlobalSelection`. This is the downstream commitment/materialization boundary +ASAPPlanner's crate documentation assigns to a deployment: apply chosen +`Replacement::Summary`/`Replacement::Rewrite` alternatives, preserve shared +node identity, validate executor support, and decide placement. Keep this +logic in one control-plane module so explain, stage allocation, and +`BackendPlan` generation cannot implement competing substitutions. This phase should activate and test: @@ -458,9 +457,9 @@ Acceptance criteria: Introduce a direct conversion: ```text -SelectedPostAsapWorkload - -> deployment placement - -> DeploymentPlan +canonical roots + GlobalSelection + -> ASAPQuery DeploymentPlan + -> apply choices + deployment placement -> materializations and readouts -> BackendPlan ``` @@ -768,7 +767,7 @@ The planner remains deterministic and free of timers: ```text ASAPPlanner: -select(workload, schemas, statistics) -> SelectedPostAsapWorkload +search(workload roots, cost model) -> PlanSpace + GlobalSelection ASAPQuery control plane: diff(active_deployment, selected_workload) -> DeploymentPlanDiff @@ -924,7 +923,7 @@ Add negative fixtures for: Tests are required at four boundaries: 1. query workload -> selected strategies; -2. `SelectedPostAsapWorkload` -> `DeploymentPlan` -> `BackendPlan`; +2. canonical roots + `GlobalSelection` -> `DeploymentPlan` -> `BackendPlan`; 3. protobuf -> data-plane hot reload and `RoutingIndex`; 4. ingest -> plan push -> warm query response, including archive fallback. From 42a16d398881bb89e227962222812fcae640b920 Mon Sep 17 00:00:00 2001 From: zz_y Date: Thu, 27 Aug 2026 14:24:04 -0600 Subject: [PATCH 07/12] docs: compile selected DAG into two physical subplans with shared identity ASAPPlanner's own scope statement (README "Scope"; asap-aware-mapping/README.md "Non-Goals") is explicit that it does not choose collector/backend placement, transport mode, or physical resources - confirmed against its current crates/types/src/post_asap module (SummaryAgg/SummaryFamilyType/GroupingStrategy/Reduction/SummaryEstimate), not older docs. Add design-compiled-plan-collector-backend-split.md: the ASAPQuery control plane compiles a selected post-ASAP DAG into a CompiledPlan carrying a CollectorSubplan (asap_edge YAML via OpAMP) and a BackendSubplan (BackendPlan), sharing one plan_id/plan_version/ activation/expiry/backend_compat identity - closing the exact gap ASAPCollector PR #558 documents (no plan_id/version/activation/expiry/ backend-compat on the OpAMP wire today) - rather than serializing the selected DAG directly into collector YAML. Update the migration doc (PR #444) to reference the compile step wherever it previously said "collector/backend stage allocation" or "collector configuration generation", fix the target-architecture diagram to show both subplans instead of only BackendPlan, add the legacy physical::plan::PlanNode/PipelineStage allocator to the post-cutover removal list, broaden PR 5 from a BackendPlan-only conversion to the full two-subplan compile, and note open ASAPPlanner PRs #300 (explicit update/readout phase boundary - the same boundary this split already uses structurally) and #299 (accuracy propagation) as tracked, non-blocking upstream changes for PR 6's coverage list. Correct design-backend-plan-wire-format.md's BackendPlan::plan_id comment ("observability only, not identity") to reflect its new role as the cross-subplan join key. Co-Authored-By: Claude Sonnet 5 --- ...-asapplanner-workload-planner-migration.md | 142 +++++- .../docs/design-backend-plan-wire-format.md | 8 +- ...n-compiled-plan-collector-backend-split.md | 430 ++++++++++++++++++ 3 files changed, 559 insertions(+), 21 deletions(-) create mode 100644 control_plane/docs/design-compiled-plan-collector-backend-split.md diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index d2d59ec3..23498799 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -28,9 +28,9 @@ ASAPQuery-backend remains the owner of: ```text selected post-ASAP DAG -> deployment placement - -> collector/backend stage allocation - -> BackendPlan - -> materialization and routing + -> compile into two physical subplans sharing one plan identity: + collector subplan (OpAMP / asap_edge YAML) + backend subplan (BackendPlan / materialization and routing) -> data-plane execution and archive fallback ``` @@ -38,6 +38,29 @@ The boundary is intentional: ASAPPlanner decides *what a query sub-DAG may be replaced with*; ASAPQuery decides *where the selected replacement runs, how it is represented on the wire, and how it is served*. +**The selected post-ASAP DAG is not the collector config, and must not be +compiled as if it were.** ASAPPlanner's own scope statement is explicit that +it does not choose collector/backend placement, transport mode, or physical +resources (`README.md` "Scope"; `asap-aware-mapping/README.md` +"Non-Goals") — a `SummaryAgg` node names a logical aggregation, not a +collector process, shard count, or window/transport parameter. Treating the +selected DAG as directly serializable into `asap_edge` YAML skips the one +step that actually assigns placement, and lets the collector-side and +backend-side views of the same decision be derived independently, with +nothing pinning them to having come from the same selection (see §2's +"silently select different physical summary families" failure mode, which +this restates for the collector/backend split specifically). The compile +step described in +[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) +is where that placement decision is actually made: one pass over one +selected DAG emits a `CompiledPlan` carrying a `CollectorSubplan` and a +`BackendSubplan` that share one `plan_id`/`plan_version` — the identity that +document closes ASAPCollector's own documented contract gap with (no +`plan_id`, version, activation/expiry, or backend-compatibility identifier +on the OpAMP wire today). Every "collector/backend stage allocation" and +"collector configuration generation" reference below (§1.1, §4.2) means that +document's compile step, not a direct IR-to-YAML dump. + ### 1.1 Ownership after reviewing the current code Add or complete in **ASAPPlanner**: @@ -142,12 +165,25 @@ Protocol/source adapters views (apply choices + placement) | v - BackendPlan - | - v - data_plane + compile -> CompiledPlan + (one plan_id/plan_version, + shared by both subplans below) + / \ + v v + CollectorSubplan BackendSubplan + (asap_edge YAML (BackendPlan: + via OpAMP) materializations + routing) + | | + v v + ASAPCollector data_plane ``` +The two subplans are never emitted independently of each other — see +[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) +for the compile step that produces both from one pass over one +`GlobalSelection`, and for why a direct DAG-to-YAML dump on the collector +side (skipping this step) is not an equivalent shortcut. + The selected workload and its shared `Rc`/`Rc` identities must remain intact until placement and materialization are complete. It must not be flattened into independent per-query or per-metric plans before @@ -179,6 +215,15 @@ that point. `WorkloadCharacteristics`; populate ASAPPlanner `DataCharacteristics` and keep only deployment-only constraints such as collector memory budget in the placement layer. +- `physical::plan::PlanNode`/`PipelineStage` and + `physical::allocator::SketchAllocator`, once the compile step in + [`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) + is the production source of `CollectorSubplan`/`BackendSubplan`. They + annotate a locally-typed `QueryExpr` tree with a per-node `PipelineStage` + tag rather than compiling ASAPPlanner's own selected `SummaryNode` DAG, + and their `PipelineStage::Agent`/`Backend` split predates a shared + cross-subplan `plan_id`. Keep them live until that compile step replaces + their callers — not before. Delete these paths only after the new workload path is the production source of `BackendPlan`. During migration they remain available for shadow comparison @@ -190,8 +235,12 @@ and rollback. - Replanning triggers and runtime telemetry. - ASAPQuery's cost model, implemented through ASAPPlanner's `CostModel` trait. - Deployment constraints and resource budgets. -- Stage splitting and placement across collector, backend, and archive. -- Collector configuration generation. +- Stage splitting and placement across collector, backend, and archive — + retargeted to compile from ASAPPlanner's selected `SummaryNode` DAG + instead of the legacy local `QueryExpr`/`PipelineStage` tree; see + [`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). +- Collector configuration generation — as the `CollectorSubplan` half of + that same compile step, not a separate code path. - `PolicyFingerprint` and persistent materialization identity. - `BackendPlan`, `RoutingIndex`, push, hot reload, and plan versioning. - Data-plane summary execution and cold/archive fallback. @@ -206,7 +255,9 @@ and rollback. - A thin `O11yMetricsQuery -> ASAPPlanner QueryWorkload` conversion. - An ASAPQuery `DeploymentPlan` builder that consumes canonical roots plus ASAPPlanner `GlobalSelection`, applies the chosen replacements in a - deployment-aware way, assigns placement, and emits `BackendPlan`. + deployment-aware way, assigns placement, and compiles a `CompiledPlan` + (`CollectorSubplan` + `BackendSubplan`, sharing one `plan_id`) — see + [`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). - An explain/debug endpoint over ASAPPlanner's existing DAG export and replacement-explanation types. - Planning phase timings, search-size metrics, deadlines, and cancellation. @@ -452,28 +503,55 @@ Acceptance criteria: - selection is deterministic for identical workload, statistics, and cost model inputs. -### PR 5: Selected workload to BackendPlan +### PR 5: Selected workload to `CompiledPlan` (collector + backend subplans) -Introduce a direct conversion: +Introduce a direct compile step, not a `BackendPlan`-only conversion — see +[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) +for the full design this stack step implements: ```text canonical roots + GlobalSelection -> ASAPQuery DeploymentPlan -> apply choices + deployment placement - -> materializations and readouts - -> BackendPlan + -> compile into CompiledPlan { plan_id, plan_version, activation, expiry, + backend_compat, collector, backend } + collector: CollectorSubplan (asap_edge YAML per edge, config_hash) + backend: BackendSubplan (BackendPlan: materializations + routing) ``` -Continue using `PolicyFingerprint` for persistent runtime identity. Exporter -IDs such as DAG node IDs or `workload_node_id` are scoped to an explain result -and must not become materialization keys. +`SummaryAgg` nodes compile into the collector subplan (update side); +`SummaryEstimate`/`SummaryMerge`/`SummarySubtract`/`SummaryDelete`/ +`SummaryJoin` subtrees compile into the backend subplan (readout side) — +this split is structural, derived from the selected DAG's own node kinds, +not a second per-metric classification pass. A selected DAG must never be +serialized into `asap_edge` YAML directly: neither `edge_id`, `shard_count`, +transport mode, nor any other physical parameter exists in ASAPPlanner's +output (its own non-goals), so something has to assign them, and that +assignment is what turns one selection into two *agreeing* subplans instead +of two independently-guessed ones. + +Continue using `PolicyFingerprint` for persistent runtime identity *within* +one subplan (materialization reuse/diff/resize across replans). `plan_id` +answers a different question — whether the collector subplan and the +backend subplan now active were compiled together — and is carried +identically on both (see the compile doc §3/§7 for why `BackendPlan`'s +existing `plan_id` field, currently documented as "observability only," is +redefined to be this identity, not a second one). Exporter IDs such as DAG +node IDs or `workload_node_id` are scoped to an explain result and must not +become materialization keys. Review whether the wire needs additive fields for: -- exact versus sketch summary family; +- exact versus sketch summary family, using `SummaryFamilyType` directly + (see the compile doc §7 — not a re-flattened `SummaryKind`/`SummaryParams` + pair, which does not match ASAPPlanner's current post-ASAP IR shape); - sketch algorithm and parameters; - independent versus Hydra grouping layout; -- shared materialization dependencies; +- shared materialization dependencies, including which `EdgeAssignment`(s) + supply a materialization's input summary state (`Materialization.sources` + in the compile doc §7) — the field that lets the backend reject a plan + whose collector subplan doesn't actually produce what this materialization + expects; - multiple query/readout consumers of one materialization; - derived readouts such as `avg = sum / count`, rollups, and top-k prefix reuse. @@ -486,7 +564,12 @@ Acceptance criteria: legal routes/readouts; - materialization fingerprints are stable across replans; - protobuf encode/decode and hot reload preserve the chosen plan; -- the data plane never has to run a cost model to reconstruct the choice. +- the data plane never has to run a cost model to reconstruct the choice; +- a `CollectorSubplan` and `BackendSubplan` produced by the same compile + call carry the same `plan_id`/`plan_version`, and a deliberately + mismatched pair (e.g. an old collector config against a new + `BackendPlan`) is detectable from those fields alone, without needing to + diff YAML against protobuf by hand. ### PR 6: Data-plane execution coverage @@ -505,6 +588,25 @@ Unsupported candidates must be removed before selection or fail planning with a clear capability diagnostic. They must never be accepted by the control plane and fail later during query serving. +**Track, don't block on, two open upstream ASAPPlanner PRs that change this +list's shape.** +[#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) (open) adds +`SummaryExpr::ExactTransform`/`ExactPostProcess` — composed exact-over- +summary and summary-over-exact plans across an explicit update/readout +boundary — which is exactly the boundary +[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) +§1 uses to split a selected DAG between the two subplans; if it merges, that +document's structural partition rule should be re-expressed in terms of its +`ExecutionAvailability`/`PhaseAssignment` types, per that section's own +forward note, without changing which nodes land in which subplan. +[#299](https://github.com/ProjectASAP/ASAPPlanner/pull/299) (open) propagates +end-to-end accuracy guarantees for approximate-over-approximate plans, which +bears on this PR's "accuracy... not re-derived at serving time" criterion +below once summary-over-summary composition is selectable. Neither PR is a +migration-stack dependency — this stack should not stall waiting for them — +but PR 6's "cover at least" list should be revisited against whichever of +the two has merged by the time it lands. + Acceptance criteria: - control plane selects once and the data plane consumes that exact choice; diff --git a/control_plane/docs/design-backend-plan-wire-format.md b/control_plane/docs/design-backend-plan-wire-format.md index 2fb81439..3a7ad6cf 100644 --- a/control_plane/docs/design-backend-plan-wire-format.md +++ b/control_plane/docs/design-backend-plan-wire-format.md @@ -77,7 +77,13 @@ know or care. ```rust pub struct BackendPlan { - pub plan_id: PlanId, // observability only, not identity + // Cross-subplan identity as of design-compiled-plan-collector-backend- + // split.md: shared verbatim with this plan's CollectorSubplan, so the + // two can be checked for agreement. `PolicyFingerprint` below remains + // the identity of one materialization; `plan_id` answers "were the + // collector and backend subplans compiled together," which no + // per-materialization fingerprint can answer on its own. + pub plan_id: PlanId, pub generated_at: DateTime, /// Every materialization this backend should build/maintain, diff --git a/control_plane/docs/design-compiled-plan-collector-backend-split.md b/control_plane/docs/design-compiled-plan-collector-backend-split.md new file mode 100644 index 00000000..c7c8938f --- /dev/null +++ b/control_plane/docs/design-compiled-plan-collector-backend-split.md @@ -0,0 +1,430 @@ +# Compiling a selected post-ASAP DAG into two physical subplans + +> Status: proposed, 2026-08-27 +> +> Scope: the interface between ASAPQuery-backend's control plane and its two +> executors — ASAPCollector (via OpAMP) and the ASAPQuery-backend data plane +> (via `BackendPlan`). This document replaces the implicit assumption, in +> earlier design notes, that a selected post-ASAP node can be serialized +> more or less directly into collector YAML. It complements +> [`design-asapplanner-workload-planner-migration.md`](design-asapplanner-workload-planner-migration.md) +> (the planner-migration boundary) and +> [`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) +> (the backend half of the wire contract this document extends). + +## 0. The boundary this document sits on + +ASAPPlanner's own scope statement (`README.md`, "Scope") is explicit: + +> not caring about CTSA stages i.e. whether a part of a plan is executed at +> the collector or at the analytics stage +> not caring about assignment of physical resources, like CPU threads and +> memory, to nodes in the ASAP plan + +and `docs/design_docs/asap-aware-mapping/README.md`'s non-goals repeat this +for the mapping layer specifically: no CPU/memory assignment, no machine +placement, no scheduling, no admission control, no low-level execution +tuning. ASAPPlanner's own README names the resulting open question directly +("Open questions", #1): its output "has semantics of batch query execution +over data at rest" and "needs to be converted into two plans: (1) streaming +dataflow graph that computes summaries on raw data, and (2) batch query +execution plan that uses summaries to answer queries" — collector and +backend, in this deployment's vocabulary. + +So: ASAPPlanner selects **what** replaces a query sub-DAG (which summary +family, algorithm, parameters, grouping layout, and how per-query readouts +compose over shared summary state). ASAPQuery-backend's control plane +decides **where** each piece of that selected DAG runs, **how** it is +represented on each wire, and **that** both sides agree they are running the +same decision. This document is the second half — the "where/how/that" — +concretely. + +## 1. What ASAPPlanner hands us today + +Grounded in `crates/types/src/post_asap/{mod,expr,schema,sketch}.rs` on +ASAPPlanner `main`, not carried over from older docs. The selected output of +`asap_aware_mapping::replacement::search_workload_with(...).global_selection(...)` +is a DAG of `Rc` (shared `Rc` = shared physical state — see +[migration doc](design-asapplanner-workload-planner-migration.md) §3), each +node one of: + +- **`SummaryExpr::SummaryAgg { child, family, col, reduction, grouping }`** + — the *update* side: consumes raw/plain input and produces summary state. + `family: SummaryFamilyType` is one of `ExactAggregate(ExactKind, + ExactParams)`, `Sketch(SketchKind, GroupingStrategy)` (`SketchKind` itself + nests `category`/`algorithm`/`params` — `Kll`/`DDSketch`/`Hll`/`Cms`/ + `CmsWithHeap`/`Kmv`/`Theta`/`CountSketch`/`CountSketchWithHeap`, each with + its own concrete `SketchParams`), `Sample(SamplingKind, SamplingParams)`, + `Wavelet(WaveletKind, WaveletParams)`, or `StatModel(StatModelKind, + StatModelParams)`. `reduction: Reduction` is `Reduce(GroupKeys)` or + `PerEntity` (`crates/types/src/pre_asap/query_expr.rs`). `grouping: + GroupingStrategy` is `PerSubpopulationInstance` (default) or + `SharedMultiSubpopulation { kind: HydraKind, params: HydraParams }`. +- **`SummaryExpr::SummaryEstimate { summary_input, query }`** — the + *readout* side: reads a `SketchQuery` (`Quantile`/`PointCount`/ + `Cardinality`/`TopK`) out of already-built summary state, producing a + plain value. Summary-state typing does not propagate past this node. +- **`SummaryExpr::SummaryMerge { children }`** / **`SummarySubtract`** / + **`SummaryDelete`** / **`SummaryJoin`** — combine or transform summary + state; still summary-typed in/out, still on the readout side of any + `SummaryEstimate` that eventually consumes them. +- **`SummaryExpr::KeepPreAsap(Rc)`** — no replacement chosen; + executed against raw/archive data, never against collector-maintained + state. + +**Forward note on an open upstream PR.** ASAPPlanner PR +[#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) (open, not yet +merged) proposes to make exactly this update/readout distinction an +explicit, validated field: `post_asap::phase::ExecutionAvailability { +UpdateValue, SummaryState, ReadoutValue }`, with `SummaryAgg.child` typed to +accept only `UpdateValue` (or nested exact-accumulator state), and +`SummaryEstimate` typed `SummaryState -> ReadoutValue`. The split this +document defines (§2) is derived from the same structural fact — +`SummaryAgg` is the only node that *consumes* plain/update values and +*produces* summary state — so it does not depend on #300 landing, but it is +literally the same boundary #300 gives a name to. If/when #300 merges, §2's +partition rule should be re-expressed as "everything upstream of and +including a `PhaseAssignment` boundary at `SummaryState`" rather than +re-derived structurally; no other part of this design changes. + +## 2. Why not compile 1:1, node-by-node, straight to collector YAML + +A naive compiler would walk the selected DAG and, for each `SummaryAgg`, +emit one `asap_edge.metrics[]` entry with the same `family`/`col`/ +`reduction` fields, then hand the whole thing to whichever process asks for +it. This does not work, for reasons that are all direct consequences of §0's +non-goals: + +1. **No stage/edge/shard is chosen.** ASAPPlanner has no concept of "which + collector process" or "how many shards" — `SummaryAgg` names a logical + aggregation, not a physical instance of one. Something has to decide + fan-out: one `SummaryAgg` shared by two queries might still be one + physical summary; one `SummaryAgg` under high cardinality might be + sharded across `shard_count` collector processes and merged with + `SummaryMerge` before it ever reaches a `SummaryEstimate`. That decision + is deployment placement, owned here, not upstream. +2. **No transport/physical parameters exist upstream.** `edge_id`, + `window_duration`, `warm_allowed_lateness`, `drop_original`, + `delta_transmission`, `delta_threshold` (see + [ASAPCollector's OpAMP interface doc](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md)) + are bandwidth/latency/resource trade-offs a deployment makes; ASAPPlanner + has no field for any of them and should not grow one (they are the + physical-resource assignment its own non-goals name explicitly). +3. **Sharing and sharding both break 1-selected-node = 1-wire-fragment.** A + shared `Rc` reached by two query roots must still be *one* + collector-side summary and *one* backend-side `Materialization` with two + `RoutingEntry` rows (see + [`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) + §3 on why routing is a separate table). A single logical node sharded for + cardinality must become *several* collector-side instances merged back + into *one* backend-side materialization. Neither direction is a + serialization concern; both require an explicit compile/allocate pass. +4. **Two independent readings of the same DAG can silently disagree.** + [`design-asapplanner-workload-planner-migration.md`](design-asapplanner-workload-planner-migration.md) + §2 already names this failure mode for the legacy planner ("the control + plane, data plane, and ASAPPlanner silently select different physical + summary families or parameters for the same query"). If the collector + subplan and the backend subplan are derived independently — even from + the same selected DAG, by two different code paths, at two different + times — nothing stops them drifting. A single compile step that emits + both subplans from one pass over one selected DAG, stamped with one + shared identity (§4), is what removes that possibility structurally + instead of by convention. +5. **The two wires evolve independently and are consumed by different + processes at different times.** OpAMP YAML is read by ASAPCollector; + `BackendPlan` protobuf is read by `data_plane`. Neither should decode + the other's format, and neither should decode ASAPPlanner's internal + Rust IR — that IR is not a stable cross-process wire contract and was + never meant to be one (`design-asapplanner-workload-planner-migration.md` + §5/PR4: *"Do not copy `dag_export`'s JSON into the runtime contract and + do not infer mappings from labels, hashes, strategy rationale, or viewer + node signatures."*). + +## 3. `CompiledPlan`: one compile step, two subplans, one identity + +```rust +/// The output of compiling one `GlobalSelection` for one deployment +/// topology. This is control_plane's L5 (see +/// design-target-architecture.md §2, "L5 — physical plan": the one layer +/// this deployment owns in full because no upstream `asap-physical` crate +/// exists) — and it is the *only* thing that leaves the control plane's +/// planning boundary. Neither subplan is ever emitted independently of the +/// other; they are two views produced by the same compile call. +pub struct CompiledPlan { + /// Shared identity across BOTH subplans — the field this document adds + /// to close ASAPCollector's own documented gap (§6 below). Content + /// addressed: a hash of the selected DAG's structure plus the + /// deployment topology/constraints the compiler ran against, so two + /// compiles of the same selection against the same topology produce + /// the same `plan_id` and two different selections never collide. + pub plan_id: PlanId, + /// Monotonic per-`plan_id` counter — bumped on re-compile against an + /// unchanged selection (e.g. a resize), not on every replan. + pub plan_version: u64, + /// Not-before: neither subplan should be treated as authoritative + /// before this time. Lets a warm cutover (see migration doc §6.5, + /// `DeploymentPlanDiff`) land both subplans ahead of the switch. + pub activation: DateTime, + /// Not-after / supersede horizon. `None` for "until superseded." + pub expiry: Option>, + /// Identifies the *wire schema version* the backend subplan requires, + /// so a collector/backend pair that somehow ends up on mismatched + /// deploys fails a compatibility check instead of silently serving + /// under the wrong contract. Distinct from `plan_id`: this changes on + /// a schema/deploy version bump, not on every replan. + pub backend_compat: BackendCompatId, + + pub collector: CollectorSubplan, + pub backend: BackendSubplan, +} + +pub struct CollectorSubplan { + pub plan_id: PlanId, // == CompiledPlan::plan_id + pub plan_version: u64, // == CompiledPlan::plan_version + pub backend_compat: BackendCompatId, + /// One entry per collector fleet member this plan touches. + pub edges: Vec, +} + +pub struct EdgeAssignment { + pub edge_id: String, + /// Today's `asap_edge` processor fields (§5) — produced by compiling + /// the `SummaryAgg` nodes assigned to this edge, not authored ad hoc. + pub config: AsapEdgeConfig, + /// Opaque identity of *this edge's* exact YAML body — unchanged + /// semantics from ASAPCollector's existing `config_hash` (it still + /// identifies collector-config bytes, nothing more); `plan_id` is the + /// new, separate field that identifies the plan those bytes were + /// compiled from. + pub config_hash: ConfigHash, +} + +pub struct BackendSubplan { + pub plan_id: PlanId, // == CompiledPlan::plan_id + pub plan_version: u64, // == CompiledPlan::plan_version + /// Today's `BackendPlan` (design-backend-plan-wire-format.md §3), with + /// one change: `BackendPlan::plan_id` stops being "observability only" + /// (its current doc comment) and becomes literally + /// `CompiledPlan::plan_id` — see §6. + pub backend_plan: BackendPlan, +} +``` + +## 4. Compile algorithm + +Input: the materialized selection (`GlobalSelection::materialize()`'s +`Rc` roots, per +[migration doc](design-asapplanner-workload-planner-migration.md) §5/PR4 — +shared node identity intact) plus this deployment's topology and +constraints (collector fleet membership, per-edge shard/memory budgets, +transport cost model — the same inputs `physical::colored_dag` already +takes today, see §8). + +1. **Partition by node kind**, not by heuristic: every `SummaryAgg` + reached anywhere in the selection is an *update-side* node; every + `SummaryEstimate`/`SummaryMerge`/`SummarySubtract`/`SummaryDelete`/ + `SummaryJoin` is a *readout-side* node (it consumes summary state and + either produces more summary state for further readout-side composition, + or a plain value). `KeepPreAsap` subtrees are neither — they stay the + backend/archive fallback path already described in + [`design-target-architecture.md`](design-target-architecture.md) §3. +2. **Allocate each distinct `SummaryAgg` (by `Rc` identity) to one or more + collector edges.** A single logical `SummaryAgg` may become several + `EdgeAssignment` entries (sharding by cardinality/volume budget) or share + one existing edge with another `SummaryAgg` from a different query root + (the shared-`Rc` case). This is where `shard_count`, `edge_id` selection, + and per-edge resource budgeting happen — genuinely new information, not + copied from the selection. +3. **Insert `SummaryMerge` at the shard boundary** when step 2 sharded a + node: the collector side ships `shard_count` partial states: the backend + side's `Materialization` reflects one logical summary, reconciled via + merge before any `SummaryEstimate` reads it (`SummaryMerge`'s own + catalog-`mergeable` requirement, already enforced upstream, is what makes + this legal at all). +4. **Compile every readout-side subtree into `Materialization` + + `RoutingEntry` rows** in the backend subplan (§7), each one recording + which `EdgeAssignment`(s) supply its input summary state. +5. **Decide transport parameters** (`delta_transmission`, `delta_threshold`, + `drop_original`, `warm_allowed_lateness`) per `EdgeAssignment` from the + deployment cost model — never from the selection, which has no opinion + on transport (§2.2). +6. **Compute `plan_id`** from the compiled structure (§3), stamp it plus + `plan_version`/`activation`/`expiry`/`backend_compat` identically onto + both subplans, and return the `CompiledPlan`. + +Steps 2–3 are exactly the job `physical::allocator::SketchAllocator` and +`physical::stage_split` already do today against the *legacy* locally-typed +`QueryExpr`/`PipelineStage` tree (`physical/plan.rs`); this document asks +for the same allocation job, retargeted to consume ASAPPlanner's own +`SummaryNode` selection instead of a parallel local IR — see §9. + +## 5. `SummaryAgg` → `asap_edge.metrics[]`, field by field + +| Selected-DAG source | `asap_edge.metrics[]` field | Notes | +|---|---|---| +| `col` | `metric` | Direct. | +| `family: ExactAggregate(Sum\|Count\|MinMax\|Increase\|Rate, _)` | `family: sum` (+ new `exact_kind`) | Today's `asap_edge` schema only lists sketch families (`sum, ddsketch, kll, hll, countsketch, countminsketch`, per ASAPCollector's doc) — needs an `exact_kind` discriminator to carry `Count`/`MinMax`/`Increase`/`Rate`, not just `Sum`. **Gap to close in ASAPCollector's schema**, flagged in §6. | +| `family: Sketch(SketchKind{algorithm, params, ..}, grouping)` | `family`, `relative_accuracy`/`k`/`rows`,`cols` | `algorithm` selects the enum value; `params` fills the matching size field(s) directly — `SketchParams::DDSketch{alpha}` → `relative_accuracy`, `SketchParams::Kll{k}` → `k`, `SketchParams::Cms{width,depth}`/`CountSketch{width,depth}` → `cols`/`rows`. `CmsWithHeap`/`CountSketchWithHeap`/`Kmv`/`Theta` have no `asap_edge` field yet — **gap**, see §6. | +| `grouping: SharedMultiSubpopulation{kind, params}` | *(none today)* | Hydra layouts have no `asap_edge` representation at all yet — **gap**, see §6. `PerSubpopulationInstance` (the default) is today's only implicit behavior and needs no new field. | +| `reduction: Reduce(by)` | `mode: whole_stream`, `aggregate_by: by` | Combine matching series, retain `by` as the output grouping key — this is the compile step's translation, not a 1:1 field rename: `Reduction` and `asap_edge`'s `mode`/`aggregate_by` are different vocabularies (§6). | +| `reduction: PerEntity` | `mode: per_series`, `aggregate_by: []` | No cross-series merge, matching `PerEntity`'s "never merges across entities." | +| *(not in selection — deployment decision)* | `edge_id`, `shard_count`, `window_duration`, `warm_allowed_lateness`, `drop_original`, `delta_transmission`, `delta_threshold`, `sample_p`, `max_series` | Filled by compile steps 2/5, from topology/cost-model inputs, never from the selected DAG. | + +`item_label` (set/frequency item column) comes from the deployment's +`Frequency` extension realization +(`design-target-architecture.md`'s `CostModel::realize_extension`), the +same place it's produced today. + +## 6. Gaps this closes vs. what it still leaves open + +**Closes**, on the ASAPCollector side (its own documented gap, verbatim from +[`opamp-config-push.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md)'s +"Current contract gap" section): *"the implemented OpAMP YAML schema +currently has no explicit `plan_id`, `plan_version`, activation time, expiry +time, or backend compatibility identifier. `config_hash` identifies the +remote collector configuration; it is not a complete versioned end-to-end +plan contract."* `CompiledPlan`'s envelope (§3) is exactly those five +fields, carried on both `CollectorSubplan` and `BackendSubplan`. The MVP +harness (per that same doc) can now compare `plan_id`/`plan_version` +reported by a collector's `AgentToServer` health/status against the +`plan_id`/`plan_version` the backend reports as active, instead of only +having `config_hash` (which proves the collector loaded *some* YAML, not +that it's the YAML compiled alongside the currently-active `BackendPlan`). +This is additive to — not a replacement for — ASAPCollector's own +`AgentRemoteConfig`/`config_hash` mechanics; see that document for exactly +where in the OpAMP message envelope these fields should be encoded (an +ASAPCollector-side decision this document does not make unilaterally). + +**Opens**, in `asap_edge`'s own schema (ASAPCollector-owned, needs a +follow-up there, not answered here): an `exact_kind` discriminator for +`ExactAggregate` families beyond `Sum`; fields for `CmsWithHeap`/ +`CountSketchWithHeap`/`Kmv`/`Theta`; and a grouping-strategy block for +`SharedMultiSubpopulation`/Hydra (`hydra_kind`, `shared_rows`, +`shared_columns` or `shared_buckets`, mirroring +`crates/types/src/post_asap/sketch.rs`'s `HydraParams` shape). None of +these are exercised by this deployment's current MVP metric set, so they +are out of scope for the first `CompiledPlan` implementation, but the field +table in §5 should not be read as "these are all the families `asap_edge` +will ever need" — only as what compiles today. + +**Stays open**, and is explicitly out of scope here: the rollup algebra +question already on record in +[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) +§7 ("which `SummaryKind`s roll up safely"), and the composed exact/summary +execution gaps tracked against ASAPPlanner PR #300 / issue #171. +`CompiledPlan` treats a `RollupStrategy` selection the same as any other +readout-side subtree (§4 step 4) — it does not independently re-derive +rollup legality, which remains ASAPPlanner's decision to have made during +selection. + +## 7. Backend subplan: one correction to the existing `Materialization` shape + +[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) +§3 defines `Materialization.kind: SummaryKind` / `params: SummaryParams` as +a flat pair, citing `asap_sketch::SummaryKind`/`SummaryParams`. Those exact +type names do not exist in ASAPPlanner's current `crates/types::post_asap` +(§1) — the flat-pair shape predates the current IR, which nests kind+params +*per family* inside `SummaryFamilyType` (and nests a further +algorithm+params level specifically for `Sketch`, plus the orthogonal +`GroupingStrategy` axis). `Materialization` should be updated to carry the +current type directly, the same "reuse the canonical vocabulary, don't +re-flatten it" principle §3 of that document already states as its own +goal: + +```rust +pub struct Materialization { + pub fingerprint: PolicyFingerprint, + pub source: Source, + pub window: WindowSpec, + pub group_by: Vec, + pub rollup: Vec, + + /// Was `kind: SummaryKind, params: SummaryParams`. Now the current + /// upstream type directly — carries grouping layout for `Sketch` too, + /// which the old flat pair had no field for at all. + pub family: SummaryFamilyType, + pub col: ColumnRef, + + /// New: which `EdgeAssignment`(s) this materialization's input summary + /// state comes from. Lets the backend validate, at plan-apply time, + /// that the collector subplan sharing this `CompiledPlan::plan_id` + /// actually produces a `family`-compatible input — the concrete + /// mechanism behind the migration doc's completion criterion + /// ("Backend applies an incompatible plan -> reject the emitted state + /// or fail the run"). + pub sources: Vec, + + pub retention: Option, +} + +pub struct EdgeSourceRef { + pub edge_id: String, + pub metric: String, // matches an EdgeAssignment's asap_edge.metrics[].metric +} +``` + +`BackendPlan.plan_id`'s doc comment ("observability only, not identity") no +longer holds under this design — see §3: it becomes the field two subplans +are joined on. Content-addressed `PolicyFingerprint` remains correct as the +identity of one `Materialization` (reuse/diff/resize within a single +backend subplan, per the migration doc's `DeploymentPlanDiff`); `plan_id` +now answers a different question — "were these two subplans compiled +together" — that `PolicyFingerprint` was never meant to answer. + +## 8. What does not change + +- `PolicyFingerprint`, `RoutingIndex`, hot reload, `DeploymentPlanDiff`, + warm cutover, and archive fallback — all as designed in + `design-backend-plan-wire-format.md` and + `design-asapplanner-workload-planner-migration.md` §5/§6. +- ASAPCollector's `AgentRemoteConfig`/`AgentConfigMap`/`config_hash` + mechanics and apply/restart semantics — unchanged; this document adds + envelope fields alongside them, per §6. +- The `asap_edge` processor's already-documented fields (§5's left two + columns) — extended, not replaced. + +## 9. Migration notes + +- `physical::plan::PlanNode` / `PipelineStage` / `physical::allocator:: + SketchAllocator` / `physical::stage_split` operate on this repo's own + locally-typed `QueryExpr` (`crate::intent_algebra`), annotated with a + per-node `PipelineStage` tag on one combined tree. `CompiledPlan` replaces + that shape with two explicit typed subplans compiled from ASAPPlanner's + own selected `SummaryNode` DAG. This module belongs on the + [migration doc](design-asapplanner-workload-planner-migration.md) §4.1 + removal list (it is not currently listed there) — add it once PR4/PR5 of + that stack lands, not before, since it is still the live path until then. +- `emit::agent::generate_agent_collector_config` currently builds one + processor keyed by `cfg.sketch_type` per collector — a shape that + predates the unified `asap_edge` processor with a `metrics[]` list that + ASAPCollector's own OpAMP doc now documents as canonical. It should become + the `EdgeAssignment -> asap_edge YAML` serializer described here, which is + a strictly larger rewrite than a field-mapping change — flagging it here + so it isn't mistaken for a small follow-up. +- Both subplans should land behind the same `ASAP_WORKLOAD_PLANNER_V2` + shadow-rollout flag the migration doc already proposes (§5/PR8): in + `shadow` mode, compile `CompiledPlan` and record `plan_id` agreement and + field-level diffs against the legacy allocator's output without pushing + either subplan, exactly mirroring that section's existing comparison + list. + +## 10. Open questions + +- **Where in the OpAMP envelope do `plan_id`/`plan_version`/`activation`/ + `expiry`/`backend_compat` live?** A sibling top-level YAML key next to + `processors.asap_edge`, a field inside `asap_edge` itself, or a separate + `AgentConfigFile` entry — this is ASAPCollector's schema to own; this + document only establishes that the fields must exist and must be + identical to the backend subplan's copy. +- **`backend_compat` granularity.** One id per `BackendPlan` proto schema + version, or one per `(schema version, family vocabulary version)` so an + `asap_edge` schema gap closing (§6) doesn't force every unrelated plan to + recompute compatibility — needs a decision before the field ships, not + after. +- **Cross-shard `SummaryMerge` placement.** Step 3 (§4) inserts + `SummaryMerge` "at the shard boundary" without saying which physical + stage performs it — collector-side gateway merge vs. backend-side merge + on ingest are both live options already implied by `PipelineStage`'s + existing `Backend`/`Precompute` distinction, and the choice affects + `EdgeAssignment` fan-in bandwidth materially. Needs its own short design + pass, not resolved here. From 76da9d42c9ce3e89689f1ac84854d4df52adc9d4 Mon Sep 17 00:00:00 2001 From: zz_y Date: Thu, 27 Aug 2026 15:17:43 -0600 Subject: [PATCH 08/12] docs: redesign asap_edge.metrics[] to name post_asap directly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Checked against the real processor (opentelemetry-collector-contrib-patch/processor/asapedgeprocessor/ config.go), not the doc summary of it: today's MetricFamily.Mode is a bare two-value string (per_series/whole_stream, ParseAggMode-validated) and AggregateBy is a plain []string with no way to express GroupKeys.without - neither can distinguish Reduction::PerEntity from a genuine zero-key Reduce, the exact ambiguity Reduction was introduced to remove. Family is a flat string with no exact_kind discriminator and no GroupingStrategy field at all. Redesign MetricFamily to name every field and enum value directly from post_asap - Source/Family/ExactKind/ReduceBy/ReduceWithout/PerEntity/ Grouping/HydraKind/SharedRows/SharedColumns - while keeping Go's flat mapstructure-struct idiom (matching this same file's own FamilyKind/ Tier/ColdFormat pattern) rather than grafting a serde-style nested tagged union onto a decoder that was never built for one. The exact_kind and grouping gaps §6 previously listed as open follow-ups close as a direct consequence of the realignment, not as separate work. Explicit about scope: this is a schema proposal against real code, not a claim ASAPCollector has implemented it, and lists which MetricFamily fields have no DAG counterpart and correctly stay untouched (tier, spatial_filter, gos_*, emit_heap/weight_mode, threshold/CDM, cold archive, control_channel). Co-Authored-By: Claude Sonnet 5 --- ...n-compiled-plan-collector-backend-split.md | 161 +++++++++++++++--- 1 file changed, 134 insertions(+), 27 deletions(-) diff --git a/control_plane/docs/design-compiled-plan-collector-backend-split.md b/control_plane/docs/design-compiled-plan-collector-backend-split.md index c7c8938f..eadca646 100644 --- a/control_plane/docs/design-compiled-plan-collector-backend-split.md +++ b/control_plane/docs/design-compiled-plan-collector-backend-split.md @@ -258,22 +258,128 @@ Steps 2–3 are exactly the job `physical::allocator::SketchAllocator` and for the same allocation job, retargeted to consume ASAPPlanner's own `SummaryNode` selection instead of a parallel local IR — see §9. -## 5. `SummaryAgg` → `asap_edge.metrics[]`, field by field - -| Selected-DAG source | `asap_edge.metrics[]` field | Notes | -|---|---|---| -| `col` | `metric` | Direct. | -| `family: ExactAggregate(Sum\|Count\|MinMax\|Increase\|Rate, _)` | `family: sum` (+ new `exact_kind`) | Today's `asap_edge` schema only lists sketch families (`sum, ddsketch, kll, hll, countsketch, countminsketch`, per ASAPCollector's doc) — needs an `exact_kind` discriminator to carry `Count`/`MinMax`/`Increase`/`Rate`, not just `Sum`. **Gap to close in ASAPCollector's schema**, flagged in §6. | -| `family: Sketch(SketchKind{algorithm, params, ..}, grouping)` | `family`, `relative_accuracy`/`k`/`rows`,`cols` | `algorithm` selects the enum value; `params` fills the matching size field(s) directly — `SketchParams::DDSketch{alpha}` → `relative_accuracy`, `SketchParams::Kll{k}` → `k`, `SketchParams::Cms{width,depth}`/`CountSketch{width,depth}` → `cols`/`rows`. `CmsWithHeap`/`CountSketchWithHeap`/`Kmv`/`Theta` have no `asap_edge` field yet — **gap**, see §6. | -| `grouping: SharedMultiSubpopulation{kind, params}` | *(none today)* | Hydra layouts have no `asap_edge` representation at all yet — **gap**, see §6. `PerSubpopulationInstance` (the default) is today's only implicit behavior and needs no new field. | -| `reduction: Reduce(by)` | `mode: whole_stream`, `aggregate_by: by` | Combine matching series, retain `by` as the output grouping key — this is the compile step's translation, not a 1:1 field rename: `Reduction` and `asap_edge`'s `mode`/`aggregate_by` are different vocabularies (§6). | -| `reduction: PerEntity` | `mode: per_series`, `aggregate_by: []` | No cross-series merge, matching `PerEntity`'s "never merges across entities." | -| *(not in selection — deployment decision)* | `edge_id`, `shard_count`, `window_duration`, `warm_allowed_lateness`, `drop_original`, `delta_transmission`, `delta_threshold`, `sample_p`, `max_series` | Filled by compile steps 2/5, from topology/cost-model inputs, never from the selected DAG. | - -`item_label` (set/frequency item column) comes from the deployment's -`Frequency` extension realization -(`design-target-architecture.md`'s `CostModel::realize_extension`), the -same place it's produced today. +## 5. Redesigning `asap_edge.metrics[]` to match the DAG's own shape + +The compile step's output should not reinvent a second vocabulary for what +the selected DAG already names. Checked against the real processor, not a +doc summary of it — +[`opentelemetry-collector-contrib-patch/processor/asapedgeprocessor/config.go`](https://github.com/ProjectASAP/ASAPCollector/blob/main/opentelemetry-collector-contrib-patch/processor/asapedgeprocessor/config.go) — +today's `MetricFamily` struct does exactly that in two places: `Mode` is a +bare string validated to exactly two values (`precompute.ParseAggMode` +accepts only `per_series`/`whole_stream`), and `AggregateBy []string` has no +way to express `GroupKeys.without`. Neither can distinguish +`Reduction::PerEntity` from a genuine zero-key `Reduce` — the exact +ambiguity ASAPPlanner's own `Reduction` type was introduced to remove +(issue #163, per `crates/types/src/post_asap/expr.rs`'s own doc comment on +`SummaryAgg.reduction`). `Family` is a flat string covering only `sum`/ +`ddsketch`/`kll`/`hll`/`countsketch`/`countminsketch`, with no discriminator +for the other four `ExactKind` variants and no field at all for +`GroupingStrategy`. + +Go's `mapstructure` decoding has no polymorphic nested-union support the way +`serde` does, and this file's own existing pattern (`FamilyKind`/`Tier`/ +`ColdFormat`) is already flat discriminator-plus-sibling-fields, not +nesting. So "aligned with the DAG" here means *the same flat shape*, with +every field name and enum spelling drawn directly from `post_asap` — not a +Rust-style nested tagged union grafted onto a struct that was never built to +decode one: + +```go +// today // redesigned +type MetricFamily struct { type MetricFamily struct { + Metric string Source string // SummaryAgg.col + Family FamilyKind Family FamilyKind // exact|ddsketch|kll|hll| + // sum|ddsketch|kll| // cms|count_sketch| + // hll|countsketch| // cms_with_heap| + // countminsketch // count_sketch_with_heap| + // kmv|theta + AggregateBy []string ExactKind ExactKind // sum|count|min_max| + // Mode: "" | "per_series" | // increase|rate — read only + // "whole_stream" — free string, // when family=exact + // no per_entity / without concept + Mode string // Reduction, named directly. + // PerEntity excludes ReduceBy/ + // ReduceWithout — Validate() enforces it. + ReduceBy []string + ReduceWithout bool + PerEntity bool + + // GroupingStrategy — no field + // existed before. + Grouping GroupingKind // per_subpopulation_instance + // (default) | + // shared_multi_subpopulation + HydraKind string + SharedRows uint32 + SharedColumns uint32 + + RelativeAccuracy, K, Rows, Cols RelativeAccuracy, K, Rows, Cols + // ...ItemLabel, SampleP, // ...unchanged, see below + // MaxSeries, Tier, SpatialFilter, + // GosDeltaEpsilon, GosSites, + // EmitHeap, HeapSize, WeightMode, + // Threshold, HLLSparse +} } +``` + +Worked example — p99 latency by `(service, region)`, an exact per-zone +sum, and a Hydra-CMS unique-IP count by zone: + +```yaml +metrics: + - source: request_duration_seconds + family: ddsketch + reduce_by: [service, region] + reduce_without: false + relative_accuracy: 0.01 + delta_transmission: true + + - source: page_views + family: sum + exact_kind: sum + reduce_by: [zone] + + - source: unique_ips + family: cms + grouping: shared_multi_subpopulation + hydra_kind: cms + shared_rows: 4 + shared_columns: 2048 + reduce_by: [zone] + rows: 4 + cols: 2048 +``` + +Two consequences fall out of the realignment itself, not as separate +follow-ups: an `exact_kind` slot and a `grouping`/`hydra_kind` slot exist +because every `post_asap` variant now has somewhere to go — they were +"gaps" in the old flat vocabulary specifically because that vocabulary was +invented independently of `SummaryFamilyType`/`GroupingStrategy` rather than +read off them. `cms_with_heap`/`count_sketch_with_heap`/`kmv`/`theta` are +named for completeness; the processor doesn't implement them yet (today's +`EmitHeap: true` on `count_sketch` approximates `CountSketchWithHeap` as a +special case) — naming the slot doesn't imply the runtime behind it exists. + +`edge_id`/`shard_count`/`window_duration`/`warm_allowed_lateness`/ +`drop_original` stay top-level `Config` fields, and `tier`/`spatial_filter`/ +`sample_p`/`max_series`/`item_label`/`delta_transmission`/`delta_threshold`/ +`gos_delta_epsilon`/`gos_sites`/`emit_heap`/`heap_size`/`weight_mode`/ +`hll_sparse`/`threshold{…}`/`cold{…}`/`control_channel{…}` all stay exactly +as they are on `MetricFamily` — collector-implementation and deployment +knobs with no `post_asap` counterpart to align to. Realigning them would +mean inventing DAG concepts that don't exist, the same mistake in reverse. +`item_label` in particular still comes from the deployment's `Frequency` +extension realization (`design-target-architecture.md`'s +`CostModel::realize_extension`), the same place it's produced today. + +This is a schema proposal, not a claim that ASAPCollector has implemented +it. `Source`/`ReduceBy`/`ReduceWithout`/`PerEntity`/`Grouping`/`HydraKind`/ +`SharedRows`/`SharedColumns`/`ExactKind` do not exist on `MetricFamily` +today; `Metric`/`AggregateBy`/`Mode` remain the only way to express this +today. A migration should decode both old and new field names for one +release (`Metric`→`Source`, `AggregateBy`→`ReduceBy`, `Mode` derived from +`PerEntity`/`ReduceWithout`) rather than break existing deployed configs on +cutover. ## 6. Gaps this closes vs. what it still leaves open @@ -295,17 +401,18 @@ This is additive to — not a replacement for — ASAPCollector's own where in the OpAMP message envelope these fields should be encoded (an ASAPCollector-side decision this document does not make unilaterally). -**Opens**, in `asap_edge`'s own schema (ASAPCollector-owned, needs a -follow-up there, not answered here): an `exact_kind` discriminator for -`ExactAggregate` families beyond `Sum`; fields for `CmsWithHeap`/ -`CountSketchWithHeap`/`Kmv`/`Theta`; and a grouping-strategy block for -`SharedMultiSubpopulation`/Hydra (`hydra_kind`, `shared_rows`, -`shared_columns` or `shared_buckets`, mirroring -`crates/types/src/post_asap/sketch.rs`'s `HydraParams` shape). None of -these are exercised by this deployment's current MVP metric set, so they -are out of scope for the first `CompiledPlan` implementation, but the field -table in §5 should not be read as "these are all the families `asap_edge` -will ever need" — only as what compiles today. +**Opens**, in `asap_edge`'s own schema (ASAPCollector-owned — §5's redesign +proposes the shape, but implementing it there is a separate, tracked change, +not something this document does unilaterally): the `exact_kind` and +`grouping`/`hydra_kind` slots §5 proposes cover every `post_asap` variant +that exists today, but `cms_with_heap`/`count_sketch_with_heap`/`kmv`/ +`theta` name families the processor doesn't build yet — exercising them +still needs real runtime support, not just a schema slot. None of that is +exercised by this deployment's current MVP metric set, so implementing the +unbuilt families is out of scope for the first `CompiledPlan` +implementation; §5's redesign should not be read as "these are all the +families `asap_edge` will ever need" — only as "every family that exists +today has somewhere to go." **Stays open**, and is explicitly out of scope here: the rollup algebra question already on record in From 8c226733ea924cd604ea5fc86231a64aca8afca4 Mon Sep 17 00:00:00 2001 From: zz_y Date: Thu, 27 Aug 2026 15:59:09 -0600 Subject: [PATCH 09/12] docs: align Planner integration and compiled plan contracts --- ...-asapplanner-workload-planner-migration.md | 235 ++++++++++---- .../docs/design-backend-plan-wire-format.md | 96 +++--- ...n-compiled-plan-collector-backend-split.md | 304 ++++++------------ 3 files changed, 332 insertions(+), 303 deletions(-) diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index 23498799..630bd35d 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -29,7 +29,7 @@ ASAPQuery-backend remains the owner of: selected post-ASAP DAG -> deployment placement -> compile into two physical subplans sharing one plan identity: - collector subplan (OpAMP / asap_edge YAML) + collector subplan (OpAMP / versioned CollectorPlan YAML) backend subplan (BackendPlan / materialization and routing) -> data-plane execution and archive fallback ``` @@ -171,7 +171,7 @@ Protocol/source adapters / \ v v CollectorSubplan BackendSubplan - (asap_edge YAML (BackendPlan: + (CollectorPlan YAML (BackendPlan: via OpAMP) materializations + routing) | | v v @@ -262,6 +262,50 @@ and rollback. replacement-explanation types. - Planning phase timings, search-size metrics, deadlines, and cancellation. +### 4.4 Integration contract across all open ASAPPlanner PRs + +The integration must track ASAPPlanner's public planning types and APIs, not +copy proposed branch types into ASAPQuery. As of 2026-08-27, every open +ASAPPlanner PR falls into one of the following classes: + +| PR | Change | ASAPQuery integration consequence | Runtime-wire consequence | +| --- | --- | --- | --- | +| [#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) | Adds `ExactTransform`, `ExactPostProcess`, `ExecutionAvailability`, `PhaseAssignment`, and phase validation for exact/summary composition | When merged, partition the selected DAG by validated execution availability rather than re-deriving update/readout stages from node names. Advertise mixed-execution capabilities through the deployment cost model. | Collector plan accepts update-side exact transforms only when the collector advertises them. Backend plan carries readout-side exact post-processing. Invalid readout-under-maintenance plans are rejected before physical compilation. | +| [#299](https://github.com/ProjectASAP/ASAPPlanner/pull/299) | Adds typed end-to-end `ResultGuarantee`, error metrics, bound/probability expressions, provenance, budget allocation, and rejected candidates | Use `search_workload_with_targets`; pass root `QueryRequirements.accuracy`; preserve the selected guarantee and rejection reason. Never rank a candidate that upstream rejected for accuracy. | Backend materializations/routes retain the selected guarantee. Collector plan retains the original accuracy constraint and concrete summary parameters; query reports compare the backend result against the selected guarantee. Unknown guarantees fail closed. | +| [#295](https://github.com/ProjectASAP/ASAPPlanner/pull/295) | Adds recurrence-aware CSE cost profiles and `global_selection_with_recurrence` | Convert `RepeatingEntry` intervals plus ingest/update statistics into upstream recurrence inputs and use recurrence-aware global selection. One-shot/repeating mixtures supply the required horizon explicitly. | None directly. Scheduling, watermark, lateness, and retention remain downstream runtime policy. | +| [#293](https://github.com/ProjectASAP/ASAPPlanner/pull/293) | Lets `CostModel` explicitly opt `TopK { accuracy: Exact }` into approximate heap-sketch candidates with a real sizing target | Implement `topk_exact_accuracy_target` only if product policy permits approximation for an exact-requested TopK. The opt-in returns an explicit non-Exact budget; it must never use a hidden default or clamp-sized pseudo-budget. | The selected plan records that the realization is approximate, its effective target, algorithm, parameters, and guarantee. Pass-through remains a candidate/fallback. | +| [#291](https://github.com/ProjectASAP/ASAPPlanner/pull/291) | Adds optional caller-proven `Concat` discriminator unique-key metadata | Update exhaustive `QueryExpr` visitors and preserve the metadata through canonical roots. Do not invent discriminator keys downstream. | No new collector primitive. A retained `Concat` stays backend/archive-side unless a later selected strategy gives it an executable summary realization. | +| [#296](https://github.com/ProjectASAP/ASAPPlanner/pull/296) | Adds workload cost/benefit annotations to DAG export and viewer | Consume the upstream explain/export fields in the optional explain endpoint. Do not recompute or scrape viewer output. | None; explain metadata is not a runtime plan identity or wire contract. | +| [#292](https://github.com/ProjectASAP/ASAPPlanner/pull/292) | Corrects DAG-viewer node categories and adds drift checks | No planning dependency. Accept new upstream explain kind/category output when the pinned revision includes it. | None; viewer categories must never drive placement or execution. | + +This table is an integration audit, not a requirement to wait for every PR or +to combine unmerged branches. ASAPQuery pins one immutable, tested +ASAPPlanner revision. Code is written against the public API at that revision; +when an upstream PR merges, the pin-update PR adds the corresponding adapter, +compiler, capability, and golden-test changes in the same commit. + +The open PRs must not be imitated locally in advance. In particular: + +- do not define backend copies of `ExecutionAvailability`, `ResultGuarantee`, + recurrence profiles, or `ConcatDiscriminatorKey`; +- do not use DAG-export JSON, viewer categories, decision rationale strings, + or debug node IDs as runtime input; +- do not enable a newly nameable summary/phase until the collector and data + plane advertise compatible execution capabilities; and +- do not silently discard a new field when exhaustive upstream types change. + Compilation must fail with a typed unsupported-shape diagnostic until the + new variant is deliberately mapped. + +The integration test matrix has three lanes: + +1. **Pinned baseline:** build and run against the single revision in + `Cargo.lock`. +2. **Pin update:** for each newly merged contract-affecting Planner PR, update + all Planner crates together and run cross-repository golden workloads. +3. **Capability mismatch:** deliberately select or decode a plan shape that + one executor does not support and prove planning fails before either + subplan is activated. + ## 5. Migration stack Each phase should land as a separate, buildable PR. Later PRs may be stacked @@ -269,9 +313,10 @@ while earlier ones are under review. ### PR 1: ASAPPlanner pin and API compatibility -Move every ASAPPlanner dependency to the same immutable revision. The first -target containing the workload planner and DAG-viewer changes is -`747c66a8958409afd727b4d8046e16c653d228f6` (ASAPPlanner PR #283). +Move every ASAPPlanner dependency to the same immutable revision of `main`. +Do not pin to an open PR branch and do not combine commits from several open +branches. Record the Planner commit in build metadata and emitted plan +artifacts. Update together: @@ -290,7 +335,7 @@ call sites to `asap_aware_mapping::replacement` and its public re-exports: - `PlanSpace::global_selection`; - `replacement::default_size_params`. -Adapt the ASAPQuery cost model: +Adapt the ASAPQuery cost model to the public API present at that pin: - rank `SketchAlgorithm` values and return an exact permutation of the input; - size `SketchParams` for the selected algorithm; @@ -300,6 +345,22 @@ Adapt the ASAPQuery cost model: available; - expose numeric `estimate_cost` values for observability. +When the pin contains the corresponding open-PR work described in §4.4: + +- #295: supply update/evaluation rates, one-shot counts, horizon, maintenance, + read, build, and raw-recompute costs, then call recurrence-aware global + selection for mixed workloads; +- #293: implement the TopK-Exact opt-in hook only with an explicit approved + approximation budget; +- #299: supply accuracy propagation statistics and call the target-aware + workload search API; and +- #300: advertise only the exact-transform/post-process phase capabilities + the deployed collector and backend actually execute. + +Missing runtime statistics remain unknown, never numeric zero. An unavailable +hook or type at the pinned revision is simply absent from the adapter; it is +not recreated as a backend-local compatibility type. + Use ASAPPlanner's `AccuracyTarget` as the only correctness/accuracy input model. During compatibility migration, `control_plane::types_v2` may re-export that upstream type, but ASAPQuery must not define a second semantic equivalent. @@ -319,6 +380,7 @@ Acceptance criteria: - `cargo build --workspace` succeeds; - existing control-plane and data-plane tests pass; - no ASAPPlanner crate is pinned to a different revision; +- the recorded Planner revision matches every linked Planner crate; - the compatibility path does not change production output yet. ### PR 2: Adopt ASAPPlanner's canonical workload input @@ -454,7 +516,7 @@ Acceptance criteria: ### PR 3: Complete repeating workload lowering in ASAPPlanner -Land the one generic upstream gap: +Complete the generic upstream lowering gap: - PromQL/SQL lowering for `repeating_queries` as well as `query_batch`, with one result per entry in input order. @@ -462,6 +524,23 @@ Land the one generic upstream gap: No caller ID, ASAPQuery, Prometheus, placement, wire, or runtime types enter these APIs. +If ASAPPlanner #295 is present at the pinned revision, lowering and selection +remain separate operations: the frontend lowers every `RepeatingEntry`, then +ASAPQuery supplies the entry intervals and workload ingest rate to +`recurrence_profiles` and calls `global_selection_with_recurrence`. The +Prometheus adapter owns rule-group scheduling; the Planner receives only +protocol-neutral recurrence and cost inputs. + +Acceptance criteria: + +- batch and repeating entries return one canonical root per input entry in + stable order; +- two repeating consumers with different intervals contribute the sum of + their evaluation rates to a shared sub-DAG; +- missing or invalid recurrence statistics produce a typed failure or the + documented structural fallback, never a fabricated zero cost; and +- one-shot/repeating mixtures require an explicit costing horizon. + ### PR 4: Workload-wide search and deployment selection Run one planning operation for the complete workload: @@ -472,6 +551,12 @@ let space = search_workload_with(roots, &strategies); let selection = space.global_selection(&backend_cost_model); ``` +The snippet is the baseline API. When the pinned revision contains #299, +construct the space with root accuracy targets and the deployment accuracy +model. When it contains #295, select with the recurrence-aware API. When both +are present, accuracy rejection happens before recurrence-aware cost ranking; +cost must never resurrect an accuracy-invalid candidate. + Build ASAPQuery's `DeploymentPlan` from the canonical roots plus `GlobalSelection`. This is the downstream commitment/materialization boundary ASAPPlanner's crate documentation assigns to a deployment: apply chosen @@ -502,6 +587,10 @@ Acceptance criteria: - every chosen replacement records its real strategy name and target; - selection is deterministic for identical workload, statistics, and cost model inputs. +- an upstream accuracy rejection remains rejected under every downstream cost + or placement decision; +- a TopK-Exact sketch candidate exists only when the #293 hook explicitly + supplies its effective approximation target. ### PR 5: Selected workload to `CompiledPlan` (collector + backend subplans) @@ -515,7 +604,8 @@ canonical roots + GlobalSelection -> apply choices + deployment placement -> compile into CompiledPlan { plan_id, plan_version, activation, expiry, backend_compat, collector, backend } - collector: CollectorSubplan (asap_edge YAML per edge, config_hash) + collector: CollectorSubplan (versioned CollectorPlan YAML per edge, + carried by OpAMP with config_hash) backend: BackendSubplan (BackendPlan: materializations + routing) ``` @@ -524,29 +614,32 @@ canonical roots + GlobalSelection `SummaryJoin` subtrees compile into the backend subplan (readout side) — this split is structural, derived from the selected DAG's own node kinds, not a second per-metric classification pass. A selected DAG must never be -serialized into `asap_edge` YAML directly: neither `edge_id`, `shard_count`, -transport mode, nor any other physical parameter exists in ASAPPlanner's -output (its own non-goals), so something has to assign them, and that -assignment is what turns one selection into two *agreeing* subplans instead -of two independently-guessed ones. +serialized directly as collector processor configuration: neither `edge_id`, +shard count, streaming window, transport mode, nor any other physical +parameter exists in ASAPPlanner's output. The physical compiler adds those +decisions and emits the versioned `CollectorPlan` contract defined by +ASAPCollector. That assignment is what turns one selection into two +*agreeing* subplans instead of two independently guessed ones. Continue using `PolicyFingerprint` for persistent runtime identity *within* one subplan (materialization reuse/diff/resize across replans). `plan_id` answers a different question — whether the collector subplan and the backend subplan now active were compiled together — and is carried identically on both (see the compile doc §3/§7 for why `BackendPlan`'s -existing `plan_id` field, currently documented as "observability only," is -redefined to be this identity, not a second one). Exporter IDs such as DAG -node IDs or `workload_node_id` are scoped to an explain result and must not -become materialization keys. +`plan_id` is this identity, not a second one). Exporter IDs such as DAG node +IDs or `workload_node_id` are scoped to an explain result and must not become +materialization keys. -Review whether the wire needs additive fields for: +The two wire contracts contain typed fields for: - exact versus sketch summary family, using `SummaryFamilyType` directly - (see the compile doc §7 — not a re-flattened `SummaryKind`/`SummaryParams` - pair, which does not match ASAPPlanner's current post-ASAP IR shape); + (see the compile doc §7 and backend wire-format doc §3); - sketch algorithm and parameters; - independent versus Hydra grouping layout; +- selected result guarantee and its machine-readable provenance when #299 is + present at the Planner pin; +- execution phase/availability and exact transform/post-process operators when + #300 is present at the Planner pin; - shared materialization dependencies, including which `EdgeAssignment`(s) supply a materialization's input summary state (`Materialization.sources` in the compile doc §7) — the field that lets the backend reject a plan @@ -564,12 +657,18 @@ Acceptance criteria: legal routes/readouts; - materialization fingerprints are stable across replans; - protobuf encode/decode and hot reload preserve the chosen plan; +- the OpAMP entry is exactly `asap-collector-plan.yaml` with + `application/yaml`, and bootstrap OTel configuration is not overwritten by + a workload replan; - the data plane never has to run a cost model to reconstruct the choice; - a `CollectorSubplan` and `BackendSubplan` produced by the same compile call carry the same `plan_id`/`plan_version`, and a deliberately mismatched pair (e.g. an old collector config against a new `BackendPlan`) is detectable from those fields alone, without needing to - diff YAML against protobuf by hand. + diff YAML against protobuf by hand; +- `RemoteConfigStatus.APPLIED` alone cannot pass activation: the collector's + semantic application report and emitted materialization identities must + agree with the backend plan. ### PR 6: Data-plane execution coverage @@ -588,30 +687,22 @@ Unsupported candidates must be removed before selection or fail planning with a clear capability diagnostic. They must never be accepted by the control plane and fail later during query serving. -**Track, don't block on, two open upstream ASAPPlanner PRs that change this -list's shape.** -[#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) (open) adds -`SummaryExpr::ExactTransform`/`ExactPostProcess` — composed exact-over- -summary and summary-over-exact plans across an explicit update/readout -boundary — which is exactly the boundary -[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) -§1 uses to split a selected DAG between the two subplans; if it merges, that -document's structural partition rule should be re-expressed in terms of its -`ExecutionAvailability`/`PhaseAssignment` types, per that section's own -forward note, without changing which nodes land in which subplan. -[#299](https://github.com/ProjectASAP/ASAPPlanner/pull/299) (open) propagates -end-to-end accuracy guarantees for approximate-over-approximate plans, which -bears on this PR's "accuracy... not re-derived at serving time" criterion -below once summary-over-summary composition is selectable. Neither PR is a -migration-stack dependency — this stack should not stall waiting for them — -but PR 6's "cover at least" list should be revisited against whichever of -the two has merged by the time it lands. +The exact coverage list is derived from the pinned Planner revision and the +complete open-PR audit in §4.4. In particular, a pin containing #300 adds +exact-transform/post-process and phase-validation cases; a pin containing +#299 adds guarantee propagation, budget, rejection, and unknown-guarantee +cases; a pin containing #291 adds exhaustive retained-`Concat` handling. +Viewer-only changes in #296/#292 add no execution cases. The migration does +not wait for open PRs, but every pin update expands this matrix in the same PR +that changes the upstream types. Acceptance criteria: - control plane selects once and the data plane consumes that exact choice; - family, parameters, grouping, and accuracy are not re-derived at serving time; +- result guarantees and phase assignments present in the selected Planner DAG + survive physical compilation without being weakened or guessed; - archive fallback remains available for unsupported queries; - end-to-end tests prove plan push, hot reload, and serving. @@ -842,7 +933,7 @@ When several rules need the same aggregation at different compatible intervals, materialize the smallest compatible pane and merge panes for the slower rule. Do not allocate one streaming aggregation per alert interval. -### 6.3 AccuracyTarget and KeepPreAsap are sufficient +### 6.3 Accuracy requirements, guarantees, and exact fallback Do not add `CorrectnessPolicy::{Exact, Approximate, ExactOrValidate}`. ASAPPlanner's existing `AccuracyTarget` is the single source of truth: @@ -853,15 +944,39 @@ AccuracyTarget::Epsilon(epsilon) AccuracyTarget::EpsilonDelta { epsilon, delta } ``` -`Exact` excludes approximate candidates. A node for which no valid exact ASAP -replacement exists remains `KeepPreAsap`, which means ASAPQuery executes the -original pre-ASAP subtree from raw/archive data. `Epsilon` and +By default, `Exact` excludes approximate candidates. A node for which no valid +exact ASAP replacement exists remains `KeepPreAsap`, which means ASAPQuery +executes the original pre-ASAP subtree from raw/archive data. `Epsilon` and `EpsilonDelta` allow ASAPPlanner to choose a summary sized to that target. -No `ResidualExpr`, `GuardedResult`, interval-propagation layer, or conditional -exact-fallback policy is needed for this migration. ASAPQuery consumes -`SummaryNode` replacements and executes `KeepPreAsap` exactly; it must not -reinterpret the accuracy requirement or define a competing correctness enum. +If the pinned revision contains #293, an ASAPQuery deployment may explicitly +offer approximate heap-sketch candidates for `TopK { accuracy: Exact }`, but +only by returning a concrete non-Exact sizing target from the upstream cost +model hook. This is a visible product-policy exception: pass-through remains +available, and the compiled plan records the effective approximation target. +It must not be generalized to other Exact intents. + +If the pinned revision contains #299, `AccuracyTarget` is the requested +constraint and `ResultGuarantee` is the Planner-computed guarantee of a +selected result. They are not interchangeable. ASAPQuery: + +1. supplies root targets to target-aware workload search; +2. lets Planner reject unknown or insufficient composed guarantees before + cost ranking; +3. preserves the selected guarantee, metric, probability bound, and + provenance in `BackendPlan` and explain output; and +4. treats an unknown guarantee as unavailable, never exact and never zero. + +The CollectorPlan carries the original constraint and concrete summary +parameters needed to build state. The backend plan carries the selected +result guarantee because readout and composition occur there. The MVP report +compares observed error using that same metric and bound. + +No backend-local `ResidualExpr`, `GuardedResult`, guarantee algebra, or +conditional exact-fallback policy is needed for this migration. ASAPQuery +consumes Planner replacements/guarantees and executes `KeepPreAsap` exactly; +it must not reinterpret the requirement or define a competing correctness +enum. ### 6.4 Keep scheduling outside ASAPPlanner @@ -1025,9 +1140,12 @@ Add negative fixtures for: Tests are required at four boundaries: 1. query workload -> selected strategies; -2. canonical roots + `GlobalSelection` -> `DeploymentPlan` -> `BackendPlan`; -3. protobuf -> data-plane hot reload and `RoutingIndex`; -4. ingest -> plan push -> warm query response, including archive fallback. +2. canonical roots + `GlobalSelection` -> `DeploymentPlan` -> matching + `CollectorPlan` and `BackendPlan`; +3. OpAMP/YAML and backend protobuf -> semantic activation, data-plane hot + reload, and `RoutingIndex`; +4. ingest -> both plan pushes -> identity-matched warm query response, + including archive fallback. ## 8. Complexity and safety limits @@ -1071,11 +1189,17 @@ Protect the control plane with: The migration is complete when: - all production queries enter one workload-aware ASAPPlanner path; -- the selected post-ASAP plan is the sole source of `BackendPlan` decisions; +- the selected post-ASAP plan plus one downstream physical compile is the sole + source of matching `CollectorPlan` and `BackendPlan` decisions; - shared sub-DAGs remain shared through materialization and serving; - the data plane does not independently select summary families or params; -- ASAPPlanner's `AccuracyTarget` is the only accuracy model and legacy - `accuracy_sla`/local exact-vs-approximate enums have been removed; +- ASAPPlanner's `AccuracyTarget` and, when available at the pin, + `ResultGuarantee` are the only accuracy contract; legacy + `accuracy_sla`, local exact-vs-approximate enums, and backend guarantee + algebra have been removed; +- every open ASAPPlanner PR in §4.4 is either absent from the immutable pin or + has its documented adapter/compiler/capability tests in the same pin-update + change; - q1/q2/q3/q4/q6 pass cross-repository end-to-end tests; - explain output maps every selected post-ASAP replacement explicitly to its pre-ASAP target; @@ -1090,8 +1214,9 @@ Recurring-rule support is complete only when: - only the generic evaluation interval enters ASAPPlanner; query offset and alert-state durations remain scheduler metadata; - temporal panes have an explicit common anchor and watermark contract; -- `AccuracyTarget::Exact` plans either select exact ASAP summaries or execute - `KeepPreAsap` from raw/archive data; +- `AccuracyTarget::Exact` plans either select exact summaries, execute + `KeepPreAsap` from raw/archive data, or use only the explicit, recorded + TopK exception enabled through ASAPPlanner #293's cost-model hook; - rule reload uses a warm, versioned `DeploymentPlanDiff` cutover; - the initial production path materializes into Prometheus while Prometheus retains alert-state authority. diff --git a/control_plane/docs/design-backend-plan-wire-format.md b/control_plane/docs/design-backend-plan-wire-format.md index 3a7ad6cf..c02ec147 100644 --- a/control_plane/docs/design-backend-plan-wire-format.md +++ b/control_plane/docs/design-backend-plan-wire-format.md @@ -20,9 +20,9 @@ Planning (`control_plane`) and serving (`data_plane`) are different processes with different jobs — planning picks a summary family and sizes its parameters from a query shape and an accuracy target, symbolically; serving walks already-materialized state and answers queries against it. -Those two steps must agree on **exactly** which `(SummaryKind, -SummaryParams)` a given metric's materialization uses, because -`SummaryExecutor::find_candidates` matches on that pair *by strict +Those two steps must agree on **exactly** which `SummaryFamilyType` a given +metric's materialization uses, including algorithm, parameters, and grouping +layout, because `SummaryExecutor::find_candidates` matches on that state type *by strict equality*, on purpose: this deployment chose exact agreement over silently serving an answer under a looser accuracy guarantee than what was actually planned (see `summary_executor.rs::summary_params_match`'s @@ -40,7 +40,7 @@ correct shape is: `control_plane` decides once, writes the decision into decision back — never re-derives it. This has one direct implication for `data_plane`'s serving-time L4 -lowering: it should resolve a query's `(SummaryKind, SummaryParams)` by +lowering: it should resolve a query's selected `SummaryFamilyType` by looking it up in `RoutingIndex` (built from the `BackendPlan` `control_plane` already pushed), not by invoking a `CostModel` a second time at query time. `CostModel::rank_candidates`/`size_params` are a **planning-time-only** @@ -57,23 +57,19 @@ query. actually planned, not by independently re-classifying the query and hoping the classification matches reality. - Reuse this deployment's existing canonical vocabulary - (`asap_ir`/`asap_sketch`'s `QueryExpr`/`AggIntent`/`SummaryKind`/ - `SummaryParams`, and `control_plane`'s own `Capability`/`PolicyFingerprint`) + (ASAPPlanner's `QueryExpr`/`AggIntent`/`SummaryFamilyType`, and + `control_plane`'s own `Capability`/`PolicyFingerprint`) directly. No parallel, wire-specific re-encoding of concepts that already have a canonical type. ## 3. `Materialization`: exact and approximate are already the same shape -`asap_sketch::SummaryKind`/`SummaryParams` already unify "approximate -sketch" and "exact accumulator" into one vocabulary — `Sum`/`Count`/ -`MinMax`/`Increase`/`Rate` are `SummaryKind` variants exactly like `Kll`/ -`DDSketch`/`Hll`/`Cms`, distinguished only by `SummaryKind::is_exact()`. -There is no reason for the wire format to reintroduce a -`Sketch`-vs-`ExactAggregate` split on top of a vocabulary that has already -closed that split. A `Materialization`'s payload is a `(SummaryKind, -SummaryParams)` pair, full stop — whichever family it names decides -whether readout is exact or approximate; the wire schema doesn't need to -know or care. +ASAPPlanner's current `SummaryFamilyType` is the canonical union of plain, +exact-aggregate, sketch, sample, wavelet, and statistical-model state. A +sketch-valued family carries its concrete `SketchKind` and +`GroupingStrategy`; `SketchKind` carries category, algorithm, and parameters. +`BackendPlan` preserves that selected type instead of flattening it into the +obsolete local `(SummaryKind, SummaryParams)` pair. ```rust pub struct BackendPlan { @@ -84,6 +80,10 @@ pub struct BackendPlan { // collector and backend subplans compiled together," which no // per-materialization fingerprint can answer on its own. pub plan_id: PlanId, + pub plan_version: u64, + pub activation: DateTime, + pub expiry: Option>, + pub backend_compat: BackendCompatId, pub generated_at: DateTime, /// Every materialization this backend should build/maintain, @@ -107,13 +107,19 @@ pub struct Materialization { pub group_by: Vec, pub rollup: Vec, - /// What this materialization actually is. `kind.is_exact()` tells - /// readers whether this is an exact accumulator or an approximate - /// sketch — no separate enum arm needed for that distinction. - pub kind: SummaryKind, - pub params: SummaryParams, + /// The exact selected Planner state type, including concrete sketch + /// algorithm/parameters and grouping layout where applicable. + pub family: SummaryFamilyType, pub col: ColumnRef, + /// Collector assignments that produce this state under the matching + /// CollectorPlan. + pub sources: Vec, + + /// Present when the pinned Planner revision computes a guarantee for + /// this selected result. Unknown is not represented as exact. + pub guarantee: Option, + pub retention: Option, } @@ -136,11 +142,11 @@ materialization is one new `RoutingEntry`, not a change to the materialization itself. **Why `Capability` still exists as its own type, distinct from -`(SummaryKind, SummaryParams)`:** `Capability` is the *family-level* +`SummaryFamilyType`:** `Capability` is the *family-level* question ("does anything at all answer a `QuantileApprox` shape for this metric") used for coarse routing decisions — miss detection, archive -fallback, "should I even try the sketch tier." `(SummaryKind, -SummaryParams)` is the *exact* question `SummaryExecutor::find_candidates` +fallback, "should I even try the sketch tier." `SummaryFamilyType` is the +*exact* question `SummaryExecutor::find_candidates` needs — because two candidates must agree exactly to be legally mergeable via `merge_states`, family-level compatibility alone isn't enough to decide that. These are genuinely different match precisions for genuinely @@ -175,8 +181,8 @@ struct MetricBucket { filter_ids: Vec, windows: Vec, capabilities: Vec, - kinds: Vec, - params: Vec, + families: Vec, + guarantees: Vec>, fingerprints: Vec, storage_backends: Vec, } @@ -201,7 +207,7 @@ Match algorithm on Tier-1 miss: family-level. - **`SummaryExecutor::find_candidates`** (per-`L4Node`-leaf, called during `asap_sketch::exec::execute()`'s walk): match exact - `(SummaryKind, SummaryParams)` — required for anything that can feed + exact `SummaryFamilyType` equality — required for anything that can feed a `SummaryMerge`, and the reason `l4_lowering.rs` no longer needs to independently observe or guess this (see §5). 6. **Whole-query resolution only:** if more than one candidate survives, @@ -231,7 +237,7 @@ and should be built that way rather than reinvented as nested hash maps. Today, `data_plane`'s serving-time L4 lowering (`l4_lowering.rs`) parses the raw query string down to a canonical `QueryExpr`, then has to -*independently reconstruct* which `(SummaryKind, SummaryParams)` a +*independently reconstruct* which `SummaryFamilyType` a metric's registered sid actually uses by inspecting the `SketchStore`'s own metadata (`ObservedFamilyCostModel`) before it can bind an `L4Node` that `find_candidates` will actually match. That's a real, working @@ -242,7 +248,7 @@ directly. Once `RoutingIndex` exists, serving-time lowering simplifies to: parse to `QueryExpr` (L1-L3, still genuinely needed — a query's *shape* has to be recovered from its text regardless of any wire format), then resolve the -query's `(SummaryKind, SummaryParams)` via `RoutingIndex`'s +query's selected `SummaryFamilyType` via `RoutingIndex`'s `find_candidates`-mode lookup directly, and construct the `L4Node` from that pair — no `CostModel::rank_candidates`/`size_params` call at serving time at all. `CostModel` becomes exactly what its name says: a @@ -253,13 +259,15 @@ replacement for it. ## 6. Transport -**Proto, not YAML/JSON.** `control_plane` and `data_plane` deploy -atomically in this deployment (no independent rollout, no coordinated -upgrade window), so there's no wire-compatibility constraint across -versions to protect. `SummaryParams`/`SummaryKind` become proper `oneof`s, -checked at compile time on both ends — no untyped `parameters: -HashMap` bag to fall back into. The project already has -proto precedent (`asap_otel_proto` for OTLP ingest). +**Proto, not YAML/JSON.** `BackendPlan` is a typed protobuf contract. +`SummaryFamilyType` and its family-specific values become proper protobuf +`oneof`s, with invalid family/parameter combinations rejected during decode. +`backend_compat` explicitly protects coordinated use with the independently +delivered CollectorPlan and emitted summary-state schema; deployment timing +must not be treated as an implicit compatibility guarantee. Additive fields +and backward decoding support controlled rollout, with unknown required +variants rejected rather than placed in an untyped +`HashMap` bag. ## 7. Open questions @@ -272,16 +280,16 @@ proto precedent (`asap_otel_proto` for OTLP ingest). - **Rollup algebra.** Step 2 above assumes "a materialization grouped by `(zone, region)` can answer a query grouped by `(zone)` alone" is a known-safe operation gated by the `rollup` field. The precise algebra — - which `SummaryKind`s roll up safely (`Sum`/`Count`-family: yes; + which summary families roll up safely (`Sum`/`Count`-family: yes; `Quantile`: generally no without re-estimation error) — needs its own short design pass before this step can be implemented as described. -- **Exact top-k has no `SummaryKind` to name.** `asap_sketch::SummaryKind` - has no exact (non-approximate) top-k variant — `TopK` only exists as an - approximate family (`CmsWithHeap`/`CountSketchWithHeap`). A - `Materialization` for `topk(k, metric)` at `accuracy: Exact` can't be - expressed in this schema until that vocabulary gap closes upstream (see - the tracked issue for this — an ASAPController-side vocabulary - extension, not a `BackendPlan` schema question). +- **Exact-requested top-k policy.** ASAPPlanner PR #293 permits a deployment + cost model to offer `CmsWithHeap`/`CountSketchWithHeap` for + `TopK { accuracy: Exact }` only with an explicit effective approximation + target, while retaining pass-through. If the pinned revision contains that + hook and ASAPQuery opts in, `BackendPlan` records the approximate family, + effective target, and selected guarantee; it must not describe the result as + exact. Without the opt-in, the query routes to exact raw/archive execution. - **`RoutingIndex` performance.** New query-time hot path in a latency-sensitive service — needs a benchmark pass against the current lookup, not just a correctness pass, before it can replace anything. diff --git a/control_plane/docs/design-compiled-plan-collector-backend-split.md b/control_plane/docs/design-compiled-plan-collector-backend-split.md index eadca646..dd4f757c 100644 --- a/control_plane/docs/design-compiled-plan-collector-backend-split.md +++ b/control_plane/docs/design-compiled-plan-collector-backend-split.md @@ -151,12 +151,10 @@ non-goals: /// planning boundary. Neither subplan is ever emitted independently of the /// other; they are two views produced by the same compile call. pub struct CompiledPlan { - /// Shared identity across BOTH subplans — the field this document adds - /// to close ASAPCollector's own documented gap (§6 below). Content - /// addressed: a hash of the selected DAG's structure plus the - /// deployment topology/constraints the compiler ran against, so two - /// compiles of the same selection against the same topology produce - /// the same `plan_id` and two different selections never collide. + /// Shared identity across BOTH subplans. Content addressed from the + /// selected DAG, topology identity, and semantic constraints. Mutable + /// sizing and lifecycle values are deliberately excluded and ordered by + /// `plan_version` instead. pub plan_id: PlanId, /// Monotonic per-`plan_id` counter — bumped on re-compile against an /// unchanged selection (e.g. a resize), not on every replan. @@ -181,6 +179,8 @@ pub struct CompiledPlan { pub struct CollectorSubplan { pub plan_id: PlanId, // == CompiledPlan::plan_id pub plan_version: u64, // == CompiledPlan::plan_version + pub activation: DateTime, + pub expiry: Option>, pub backend_compat: BackendCompatId, /// One entry per collector fleet member this plan touches. pub edges: Vec, @@ -202,10 +202,12 @@ pub struct EdgeAssignment { pub struct BackendSubplan { pub plan_id: PlanId, // == CompiledPlan::plan_id pub plan_version: u64, // == CompiledPlan::plan_version - /// Today's `BackendPlan` (design-backend-plan-wire-format.md §3), with - /// one change: `BackendPlan::plan_id` stops being "observability only" - /// (its current doc comment) and becomes literally - /// `CompiledPlan::plan_id` — see §6. + pub activation: DateTime, + pub expiry: Option>, + pub backend_compat: BackendCompatId, + /// `BackendPlan` from design-backend-plan-wire-format.md §3. Its + /// envelope fields equal this `BackendSubplan` and the matching + /// `CollectorSubplan`. pub backend_plan: BackendPlan, } ``` @@ -258,185 +260,92 @@ Steps 2–3 are exactly the job `physical::allocator::SketchAllocator` and for the same allocation job, retargeted to consume ASAPPlanner's own `SummaryNode` selection instead of a parallel local IR — see §9. -## 5. Redesigning `asap_edge.metrics[]` to match the DAG's own shape - -The compile step's output should not reinvent a second vocabulary for what -the selected DAG already names. Checked against the real processor, not a -doc summary of it — -[`opentelemetry-collector-contrib-patch/processor/asapedgeprocessor/config.go`](https://github.com/ProjectASAP/ASAPCollector/blob/main/opentelemetry-collector-contrib-patch/processor/asapedgeprocessor/config.go) — -today's `MetricFamily` struct does exactly that in two places: `Mode` is a -bare string validated to exactly two values (`precompute.ParseAggMode` -accepts only `per_series`/`whole_stream`), and `AggregateBy []string` has no -way to express `GroupKeys.without`. Neither can distinguish -`Reduction::PerEntity` from a genuine zero-key `Reduce` — the exact -ambiguity ASAPPlanner's own `Reduction` type was introduced to remove -(issue #163, per `crates/types/src/post_asap/expr.rs`'s own doc comment on -`SummaryAgg.reduction`). `Family` is a flat string covering only `sum`/ -`ddsketch`/`kll`/`hll`/`countsketch`/`countminsketch`, with no discriminator -for the other four `ExactKind` variants and no field at all for -`GroupingStrategy`. - -Go's `mapstructure` decoding has no polymorphic nested-union support the way -`serde` does, and this file's own existing pattern (`FamilyKind`/`Tier`/ -`ColdFormat`) is already flat discriminator-plus-sibling-fields, not -nesting. So "aligned with the DAG" here means *the same flat shape*, with -every field name and enum spelling drawn directly from `post_asap` — not a -Rust-style nested tagged union grafted onto a struct that was never built to -decode one: - -```go -// today // redesigned -type MetricFamily struct { type MetricFamily struct { - Metric string Source string // SummaryAgg.col - Family FamilyKind Family FamilyKind // exact|ddsketch|kll|hll| - // sum|ddsketch|kll| // cms|count_sketch| - // hll|countsketch| // cms_with_heap| - // countminsketch // count_sketch_with_heap| - // kmv|theta - AggregateBy []string ExactKind ExactKind // sum|count|min_max| - // Mode: "" | "per_series" | // increase|rate — read only - // "whole_stream" — free string, // when family=exact - // no per_entity / without concept - Mode string // Reduction, named directly. - // PerEntity excludes ReduceBy/ - // ReduceWithout — Validate() enforces it. - ReduceBy []string - ReduceWithout bool - PerEntity bool - - // GroupingStrategy — no field - // existed before. - Grouping GroupingKind // per_subpopulation_instance - // (default) | - // shared_multi_subpopulation - HydraKind string - SharedRows uint32 - SharedColumns uint32 - - RelativeAccuracy, K, Rows, Cols RelativeAccuracy, K, Rows, Cols - // ...ItemLabel, SampleP, // ...unchanged, see below - // MaxSeries, Tier, SpatialFilter, - // GosDeltaEpsilon, GosSites, - // EmitHeap, HeapSize, WeightMode, - // Threshold, HLLSparse -} } -``` - -Worked example — p99 latency by `(service, region)`, an exact per-zone -sum, and a Hydra-CMS unique-IP count by zone: - -```yaml -metrics: - - source: request_duration_seconds - family: ddsketch - reduce_by: [service, region] - reduce_without: false - relative_accuracy: 0.01 - delta_transmission: true - - - source: page_views - family: sum - exact_kind: sum - reduce_by: [zone] - - - source: unique_ips - family: cms - grouping: shared_multi_subpopulation - hydra_kind: cms - shared_rows: 4 - shared_columns: 2048 - reduce_by: [zone] - rows: 4 - cols: 2048 -``` - -Two consequences fall out of the realignment itself, not as separate -follow-ups: an `exact_kind` slot and a `grouping`/`hydra_kind` slot exist -because every `post_asap` variant now has somewhere to go — they were -"gaps" in the old flat vocabulary specifically because that vocabulary was -invented independently of `SummaryFamilyType`/`GroupingStrategy` rather than -read off them. `cms_with_heap`/`count_sketch_with_heap`/`kmv`/`theta` are -named for completeness; the processor doesn't implement them yet (today's -`EmitHeap: true` on `count_sketch` approximates `CountSketchWithHeap` as a -special case) — naming the slot doesn't imply the runtime behind it exists. - -`edge_id`/`shard_count`/`window_duration`/`warm_allowed_lateness`/ -`drop_original` stay top-level `Config` fields, and `tier`/`spatial_filter`/ -`sample_p`/`max_series`/`item_label`/`delta_transmission`/`delta_threshold`/ -`gos_delta_epsilon`/`gos_sites`/`emit_heap`/`heap_size`/`weight_mode`/ -`hll_sparse`/`threshold{…}`/`cold{…}`/`control_channel{…}` all stay exactly -as they are on `MetricFamily` — collector-implementation and deployment -knobs with no `post_asap` counterpart to align to. Realigning them would -mean inventing DAG concepts that don't exist, the same mistake in reverse. -`item_label` in particular still comes from the deployment's `Frequency` -extension realization (`design-target-architecture.md`'s -`CostModel::realize_extension`), the same place it's produced today. - -This is a schema proposal, not a claim that ASAPCollector has implemented -it. `Source`/`ReduceBy`/`ReduceWithout`/`PerEntity`/`Grouping`/`HydraKind`/ -`SharedRows`/`SharedColumns`/`ExactKind` do not exist on `MetricFamily` -today; `Metric`/`AggregateBy`/`Mode` remain the only way to express this -today. A migration should decode both old and new field names for one -release (`Metric`→`Source`, `AggregateBy`→`ReduceBy`, `Mode` derived from -`PerEntity`/`ReduceWithout`) rather than break existing deployed configs on -cutover. +## 5. Collector subplan wire contract + +The authoritative collector-side schema is ASAPCollector's +[`ASAPQuery-to-ASAPCollector collection-plan interface`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). +This document does not define a second flat `asap_edge.metrics[]` schema. + +For each `EdgeAssignment`, the compiler emits one versioned +`CollectorPlan` YAML document in the exact OpAMP `AgentConfigMap` entry +`asap-collector-plan.yaml`, with content type `application/yaml`. The OpAMP +protobuf is the transport envelope; the YAML document is the typed physical +execution contract. It is not a serialized ASAPPlanner Rust DAG and it is +not a complete OTel Collector configuration. + +The compiler maps the selected DAG into that schema as follows: + +| Selected post-ASAP field | `CollectorPlan` field | +| --- | --- | +| `SummaryAgg` identity | `materializations[].logical_node_ref` plus a content-addressed `materializations[].id` | +| bound `Source` and predicates | `materializations[].input.metric` and canonical `input.matchers` | +| `SummaryAgg.col` | `materializations[].input.value` | +| `SummaryFamilyType` | `materializations[].summary.family` | +| sketch algorithm and parameters | `summary.algorithm` and typed `summary.parameters` | +| Planner accuracy constraint | `summary.accuracy` | +| `Reduction::PerEntity` | `reduction.kind: per_entity` | +| `Reduction::Reduce(GroupKeys)` | `reduction.kind: reduce`, explicit `by`, and `without` | +| `GroupingStrategy` | `grouping.kind`, plus Hydra kind/parameters for shared grouping | + +The physical compiler adds fields ASAPPlanner intentionally does not own: +target agent/edge, capability snapshot, concrete streaming windows, local +shards, exporter reference, and raw/full/delta transmission policy. These +fields must never be inferred by ASAPCollector from missing values. + +The complete plan envelope carries `plan_id`, `plan_version`, `activation`, +`expiry`, and `backend_compat` verbatim from `CompiledPlan`. Each emitted +summary or delta also carries those compatibility identities plus its +materialization, window, producer, and sequence/checkpoint identity. + +Unsupported Planner alternatives remain visible in the logical candidate +space but cannot be emitted unless the targeted collector capability snapshot +and backend compatibility ID both support them. The compiler chooses another +valid candidate or exact fallback; it never renames an unsupported algorithm +to a similar supported one. In particular, shared Hydra grouping must not be +silently flattened to independent per-group state. ## 6. Gaps this closes vs. what it still leaves open -**Closes**, on the ASAPCollector side (its own documented gap, verbatim from -[`opamp-config-push.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md)'s -"Current contract gap" section): *"the implemented OpAMP YAML schema -currently has no explicit `plan_id`, `plan_version`, activation time, expiry -time, or backend compatibility identifier. `config_hash` identifies the -remote collector configuration; it is not a complete versioned end-to-end -plan contract."* `CompiledPlan`'s envelope (§3) is exactly those five -fields, carried on both `CollectorSubplan` and `BackendSubplan`. The MVP -harness (per that same doc) can now compare `plan_id`/`plan_version` -reported by a collector's `AgentToServer` health/status against the -`plan_id`/`plan_version` the backend reports as active, instead of only -having `config_hash` (which proves the collector loaded *some* YAML, not -that it's the YAML compiled alongside the currently-active `BackendPlan`). -This is additive to — not a replacement for — ASAPCollector's own -`AgentRemoteConfig`/`config_hash` mechanics; see that document for exactly -where in the OpAMP message envelope these fields should be encoded (an -ASAPCollector-side decision this document does not make unilaterally). - -**Opens**, in `asap_edge`'s own schema (ASAPCollector-owned — §5's redesign -proposes the shape, but implementing it there is a separate, tracked change, -not something this document does unilaterally): the `exact_kind` and -`grouping`/`hydra_kind` slots §5 proposes cover every `post_asap` variant -that exists today, but `cms_with_heap`/`count_sketch_with_heap`/`kmv`/ -`theta` name families the processor doesn't build yet — exercising them -still needs real runtime support, not just a schema slot. None of that is -exercised by this deployment's current MVP metric set, so implementing the -unbuilt families is out of scope for the first `CompiledPlan` -implementation; §5's redesign should not be read as "these are all the -families `asap_edge` will ever need" — only as "every family that exists -today has somewhere to go." +**Closes in the target design**, on the ASAPCollector side: the +`CollectorPlan` envelope now has explicit `plan_id`, `plan_version`, +`activation`, `expiry`, and `backend_compat` fields, carried identically on +the matching backend plan. It is delivered as the +`asap-collector-plan.yaml` OpAMP config-map entry. The collector returns the +semantic result through the `io.asap.collector.plan.v1` / +`application_report` custom message. The MVP harness compares the active +plan and materialization identities on both sides instead of treating +OpAMP's `config_hash` or `RemoteConfigStatus.APPLIED` as proof of semantic +activation. + +This remains a target contract rather than a claim about current runtime +behavior. ASAPCollector currently writes a complete OTel YAML file, restarts, +and reports only the OpAMP config hash after a syntax check. Implementing the +new parser, atomic activation, and application report is a separate code +change. + +**Opens**, in ASAPCollector's execution layer: the target schema can name all +Planner families and grouping layouts, but `cms_with_heap`, +`count_sketch_with_heap`, KMV, Theta, sampling, wavelets, statistical models, +and shared Hydra grouping still require actual collector and backend support. +Naming an algorithm in the schema does not advertise that runtime support. **Stays open**, and is explicitly out of scope here: the rollup algebra question already on record in [`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) -§7 ("which `SummaryKind`s roll up safely"), and the composed exact/summary +§7 ("which summary families roll up safely"), and the composed exact/summary execution gaps tracked against ASAPPlanner PR #300 / issue #171. `CompiledPlan` treats a `RollupStrategy` selection the same as any other readout-side subtree (§4 step 4) — it does not independently re-derive rollup legality, which remains ASAPPlanner's decision to have made during selection. -## 7. Backend subplan: one correction to the existing `Materialization` shape +## 7. Backend subplan materialization shape [`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) -§3 defines `Materialization.kind: SummaryKind` / `params: SummaryParams` as -a flat pair, citing `asap_sketch::SummaryKind`/`SummaryParams`. Those exact -type names do not exist in ASAPPlanner's current `crates/types::post_asap` -(§1) — the flat-pair shape predates the current IR, which nests kind+params -*per family* inside `SummaryFamilyType` (and nests a further -algorithm+params level specifically for `Sketch`, plus the orthogonal -`GroupingStrategy` axis). `Materialization` should be updated to carry the -current type directly, the same "reuse the canonical vocabulary, don't -re-flatten it" principle §3 of that document already states as its own -goal: +§3 carries ASAPPlanner's current `SummaryFamilyType` directly. That type +nests algorithm and parameters for sketches and retains the orthogonal +`GroupingStrategy` axis; the backend wire must not re-flatten it into a local +`SummaryKind`/`SummaryParams` vocabulary: ```rust pub struct Materialization { @@ -446,9 +355,7 @@ pub struct Materialization { pub group_by: Vec, pub rollup: Vec, - /// Was `kind: SummaryKind, params: SummaryParams`. Now the current - /// upstream type directly — carries grouping layout for `Sketch` too, - /// which the old flat pair had no field for at all. + /// Current upstream type directly, including sketch grouping layout. pub family: SummaryFamilyType, pub col: ColumnRef, @@ -466,17 +373,16 @@ pub struct Materialization { pub struct EdgeSourceRef { pub edge_id: String, - pub metric: String, // matches an EdgeAssignment's asap_edge.metrics[].metric + pub materialization_id: PolicyFingerprint, } ``` -`BackendPlan.plan_id`'s doc comment ("observability only, not identity") no -longer holds under this design — see §3: it becomes the field two subplans -are joined on. Content-addressed `PolicyFingerprint` remains correct as the -identity of one `Materialization` (reuse/diff/resize within a single -backend subplan, per the migration doc's `DeploymentPlanDiff`); `plan_id` -now answers a different question — "were these two subplans compiled -together" — that `PolicyFingerprint` was never meant to answer. +`BackendPlan.plan_id` is the field joining the two subplans. Content-addressed +`PolicyFingerprint` remains the identity of one `Materialization` +(reuse/diff/resize within a single backend subplan, per the migration doc's +`DeploymentPlanDiff`); `plan_id` answers a different question — "were these +two subplans compiled together" — that a per-materialization fingerprint +cannot answer. ## 8. What does not change @@ -484,11 +390,11 @@ together" — that `PolicyFingerprint` was never meant to answer. warm cutover, and archive fallback — all as designed in `design-backend-plan-wire-format.md` and `design-asapplanner-workload-planner-migration.md` §5/§6. -- ASAPCollector's `AgentRemoteConfig`/`AgentConfigMap`/`config_hash` - mechanics and apply/restart semantics — unchanged; this document adds - envelope fields alongside them, per §6. -- The `asap_edge` processor's already-documented fields (§5's left two - columns) — extended, not replaced. +- OpAMP's `AgentRemoteConfig`/`AgentConfigMap`/`config_hash` delivery + mechanics. `config_hash` still identifies exact remote-config bytes; it + does not replace `plan_id` or the semantic application report. +- Existing `asap_edge` runtime behavior until the versioned `CollectorPlan` + parser and apply path are implemented. ## 9. Migration notes @@ -501,13 +407,10 @@ together" — that `PolicyFingerprint` was never meant to answer. [migration doc](design-asapplanner-workload-planner-migration.md) §4.1 removal list (it is not currently listed there) — add it once PR4/PR5 of that stack lands, not before, since it is still the live path until then. -- `emit::agent::generate_agent_collector_config` currently builds one - processor keyed by `cfg.sketch_type` per collector — a shape that - predates the unified `asap_edge` processor with a `metrics[]` list that - ASAPCollector's own OpAMP doc now documents as canonical. It should become - the `EdgeAssignment -> asap_edge YAML` serializer described here, which is - a strictly larger rewrite than a field-mapping change — flagging it here - so it isn't mistaken for a small follow-up. +- `emit::agent::generate_agent_collector_config` currently builds a complete + collector YAML. It should become the `EdgeAssignment -> CollectorPlan` + serializer defined in §5. Bootstrap OTel receivers/exporters and credentials + remain deployment configuration; a workload replan must not replace them. - Both subplans should land behind the same `ASAP_WORKLOAD_PLANNER_V2` shadow-rollout flag the migration doc already proposes (§5/PR8): in `shadow` mode, compile `CompiledPlan` and record `plan_id` agreement and @@ -516,13 +419,6 @@ together" — that `PolicyFingerprint` was never meant to answer. list. ## 10. Open questions - -- **Where in the OpAMP envelope do `plan_id`/`plan_version`/`activation`/ - `expiry`/`backend_compat` live?** A sibling top-level YAML key next to - `processors.asap_edge`, a field inside `asap_edge` itself, or a separate - `AgentConfigFile` entry — this is ASAPCollector's schema to own; this - document only establishes that the fields must exist and must be - identical to the backend subplan's copy. - **`backend_compat` granularity.** One id per `BackendPlan` proto schema version, or one per `(schema version, family vocabulary version)` so an `asap_edge` schema gap closing (§6) doesn't force every unrelated plan to From 50488288fa1b190bcef93daabb6da23ef85d9fef Mon Sep 17 00:00:00 2001 From: zz_y Date: Thu, 27 Aug 2026 17:49:11 -0600 Subject: [PATCH 10/12] docs: rewrite ASAPPlanner integration migration plan --- ...-asapplanner-workload-planner-migration.md | 1557 +++++------------ ...n-compiled-plan-collector-backend-split.md | 31 +- 2 files changed, 446 insertions(+), 1142 deletions(-) diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md index 630bd35d..1b769ae5 100644 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ b/control_plane/docs/design-asapplanner-workload-planner-migration.md @@ -1,1222 +1,533 @@ -# Replace the legacy query planner with ASAPPlanner workload planning +# ASAPQuery-backend integration with ASAPPlanner -> Status: proposed, 2026-08-26 +> Status: proposed > -> Scope: migrate ASAPQuery-backend from its overlapping, per-query -> query/replacement planning paths to ASAPPlanner's workload-wide replacement -> search. This document does **not** propose removing ASAPQuery's -> deployment-specific placement, materialization, routing, or serving logic. +> Scope: migrate ASAPQuery-backend to use ASAPPlanner as its only logical +> query and summary planner, while keeping deployment planning and execution +> inside ASAPQuery-backend. -## 1. Decision +## TL;DR -Retire the legacy ASAPQuery query/replacement planner after the new path has -passed shadow and end-to-end validation. +ASAPPlanner answers: -ASAPPlanner becomes the single owner of: +> What exact or summary-based logical plans can answer this query workload, +> and which candidate should be selected under the supplied correctness and +> cost constraints? -```text -query text - -> pre-ASAP IR - -> workload common-subexpression sharing - -> replacement candidate search - -> cost-based strategy selection - -> selected post-ASAP DAG -``` +ASAPQuery-backend answers: -ASAPQuery-backend remains the owner of: +> Where should the selected logical operators run, how are their states +> transmitted and stored, and how are queries routed to the active plan? -```text -selected post-ASAP DAG - -> deployment placement - -> compile into two physical subplans sharing one plan identity: - collector subplan (OpAMP / versioned CollectorPlan YAML) - backend subplan (BackendPlan / materialization and routing) - -> data-plane execution and archive fallback -``` +The integration boundary is the selected post-ASAP workload DAG. ASAPQuery +must not duplicate Planner parsing, canonicalization, summary selection, +accuracy algebra, common-subexpression search, or candidate ranking. +ASAPPlanner must not own collector placement, runtime windows, OpAMP, +transmission mode, storage routing, deployment rollout, or query serving. -The boundary is intentional: ASAPPlanner decides *what a query sub-DAG may be -replaced with*; ASAPQuery decides *where the selected replacement runs, how it -is represented on the wire, and how it is served*. - -**The selected post-ASAP DAG is not the collector config, and must not be -compiled as if it were.** ASAPPlanner's own scope statement is explicit that -it does not choose collector/backend placement, transport mode, or physical -resources (`README.md` "Scope"; `asap-aware-mapping/README.md` -"Non-Goals") — a `SummaryAgg` node names a logical aggregation, not a -collector process, shard count, or window/transport parameter. Treating the -selected DAG as directly serializable into `asap_edge` YAML skips the one -step that actually assigns placement, and lets the collector-side and -backend-side views of the same decision be derived independently, with -nothing pinning them to having come from the same selection (see §2's -"silently select different physical summary families" failure mode, which -this restates for the collector/backend split specifically). The compile -step described in -[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) -is where that placement decision is actually made: one pass over one -selected DAG emits a `CompiledPlan` carrying a `CollectorSubplan` and a -`BackendSubplan` that share one `plan_id`/`plan_version` — the identity that -document closes ASAPCollector's own documented contract gap with (no -`plan_id`, version, activation/expiry, or backend-compatibility identifier -on the OpAMP wire today). Every "collector/backend stage allocation" and -"collector configuration generation" reference below (§1.1, §4.2) means that -document's compile step, not a direct IR-to-YAML dump. - -### 1.1 Ownership after reviewing the current code - -Add or complete in **ASAPPlanner**: - -- Lower `RepeatingEntry` workloads in the PromQL and SQL frontends. The - workload type exists today, but the batch frontend helpers currently consume - only `query_batch`. -- Keep canonicalization, schema binding, workload CSE, replacement strategies, - candidate search, cost ranking, and global selection upstream. -- Keep `AccuracyTarget`, `QueryRequirements`, `BatchEntry`, `RepeatingEntry`, - `QueryLanguage`, `DataCharacteristics`, pre/post-ASAP schemas, and DAG - explain/export types canonical upstream. -- If recurring-query interval ever changes candidate costing or enables - generic cross-interval reuse, implement that as a protocol-neutral workload - strategy over `RepeatingEntry`. Do not add Prometheus schedules or rule - groups. -- If logical plan identity/diffing is needed by more than one deployment, add - a canonical semantic ID and logical selected-plan diff upstream. Physical - rollout remains downstream. - -Add or retain in **ASAPQuery control plane**: - -- `O11yMetricsQuery` ingestion and source adapters. -- Caller-ID correlation outside the planner workload: zip ordered lowering - results with adapter-owned IDs before passing `(Id, Rc)` roots to - search. -- A deployment cost-model implementation using runtime statistics and budgets. -- Executor-capability validation for selected post-ASAP shapes. -- Placement across collector/backend/archive, physical fingerprints, - `BackendPlan`, collector configuration, and routing entries. -- Physical `DeploymentPlanDiff`, warm-up, atomic route switch, retirement, and - replanning triggers. -- Explain HTTP endpoints and planning telemetry, using upstream explain types. - -Retain in **ASAPQuery data plane/runtime**: - -- Execution of selected `SummaryNode` and `KeepPreAsap` trees. -- `BackendPlan` hot reload, `RoutingIndex`, storage lookup, watermarks, - lateness/readiness checks, and archive fallback. -- Protocol response formatting and the existing Prometheus HTTP serving - adapter. - -Keep in **source/protocol adapters**, outside planner and deployment core: - -- Prometheus HTTP query request parsing. -- Prometheus rule-file parsing and rule-group scheduling metadata. -- `query_offset`, missed iterations, ordering, `for`, `keep_firing_for`, label - and annotation templates, and Alertmanager integration. -- Translation into protocol-neutral `O11yMetricsQuery` values. - -## 2. Why the legacy query planner should be removed - -ASAPQuery currently has several overlapping planning paths: - -- `query_planning.rs` reduces a set of query strings into per-metric - capabilities and then unions the required sketch families. -- `asap_tier_implement.rs` finds independently realizable aggregate roots and - implements each root separately. -- `sketch_algebra/lower.rs` and parts of `optimizer/rules` make additional - summary-family or replacement decisions. -- serving-time lowering can reconstruct or infer decisions after the control - plane has already planned them. - -Flattening a query into `(metric, capability, sketch family)` loses the IR -relationships needed for workload optimizations. In particular, it cannot -faithfully represent: - -- common subexpressions shared by multiple queries; -- one shared scan or shared summary sub-DAG with several consumers; -- `AvgToSumOverCountStrategy`; -- group-by rollup reuse; -- shared versus independent grouped summaries, including Hydra layouts; -- deriving a smaller compatible top-k result from a larger one; -- post-ASAP node provenance and strategy explanations. - -Keeping two implementations for these decisions would also let the control -plane, data plane, and ASAPPlanner silently select different physical summary -families or parameters for the same query. - -## 3. Target architecture +One downstream physical compile converts the selected DAG into two matching +runtime plans: ```text -Protocol/source adapters - -> ASAPQuery O11yMetricsQuery - -> ASAPPlanner QueryWorkload - BatchEntry / RepeatingEntry / QueryRequirements - | - v - parse and schema-bind every query - | - v - Vec<(CallerId, Rc)> - | - v - ASAPPlanner search_workload_with - | - v - PlanSpace::global_selection - / \ - v v - upstream explain ASAPQuery DeploymentPlan - views (apply choices + placement) - | - v - compile -> CompiledPlan - (one plan_id/plan_version, - shared by both subplans below) - / \ - v v - CollectorSubplan BackendSubplan - (CollectorPlan YAML (BackendPlan: - via OpAMP) materializations + routing) - | | - v v - ASAPCollector data_plane +Query workload + | + v +ASAPPlanner + selected post-ASAP workload DAG + | + v +ASAPQuery-backend physical compiler + | + +-----------------------------+ + | | + v v +CollectorPlan BackendPlan +ASAPCollector ASAPQuery data plane +build/transmit state ingest/store/read state ``` -The two subplans are never emitted independently of each other — see -[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) -for the compile step that produces both from one pass over one -`GlobalSelection`, and for why a direct DAG-to-YAML dump on the collector -side (skipping this step) is not an equivalent shortcut. - -The selected workload and its shared `Rc`/`Rc` -identities must remain intact until placement and materialization are complete. -It must not be flattened into independent per-query or per-metric plans before -that point. - -## 4. What is removed, retained, and changed - -### 4.1 Remove after cutover - -- The capability-union allocation algorithm in `query_planning.rs` as a - production planner. -- Per-aggregate-root `implement_tree` behavior in - `asap_tier_implement.rs`. -- Replacement/sketch-family choice duplicated in `optimizer/rules` and - `sketch_algebra/lower.rs`. -- Serving-time use of a cost model to guess a decision already made by the - control plane. -- `BackendStageConfig -> BackendPlan` as the canonical source of planning - decisions. It can remain temporarily as a legacy compatibility adapter. -- The placeholder `types_v2::WorkloadPlan`, `BindingName`, and - `QueryExprPlaceholder`; ASAPPlanner already represents the workload as - canonical roots and performs real `Rc`-identity CSE without `Ref` or - `LetBinding` placeholders. -- The backend-local `types_v2::QueryLanguage`; use ASAPPlanner's workload - language type. -- Backend-local one-shot/periodic query-shape modeling where it duplicates - `BatchEntry` versus `RepeatingEntry`. -- The duplicate `DataDistribution` and overlapping statistical fields in - `WorkloadCharacteristics`; populate ASAPPlanner `DataCharacteristics` and - keep only deployment-only constraints such as collector memory budget in - the placement layer. -- `physical::plan::PlanNode`/`PipelineStage` and - `physical::allocator::SketchAllocator`, once the compile step in - [`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) - is the production source of `CollectorSubplan`/`BackendSubplan`. They - annotate a locally-typed `QueryExpr` tree with a per-node `PipelineStage` - tag rather than compiling ASAPPlanner's own selected `SummaryNode` DAG, - and their `PipelineStage::Agent`/`Backend` split predates a shared - cross-subplan `plan_id`. Keep them live until that compile step replaces - their callers — not before. - -Delete these paths only after the new workload path is the production source -of `BackendPlan`. During migration they remain available for shadow comparison -and rollback. - -### 4.2 Retain and adapt - -- The HTTP API and workload registry. -- Replanning triggers and runtime telemetry. -- ASAPQuery's cost model, implemented through ASAPPlanner's `CostModel` trait. -- Deployment constraints and resource budgets. -- Stage splitting and placement across collector, backend, and archive — - retargeted to compile from ASAPPlanner's selected `SummaryNode` DAG - instead of the legacy local `QueryExpr`/`PipelineStage` tree; see - [`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). -- Collector configuration generation — as the `CollectorSubplan` half of - that same compile step, not a separate code path. -- `PolicyFingerprint` and persistent materialization identity. -- `BackendPlan`, `RoutingIndex`, push, hot reload, and plan versioning. -- Data-plane summary execution and cold/archive fallback. -- Monitoring and epsilon-allocation behavior that is independent of logical - replacement search. - -### 4.3 Add - -- An adapter-neutral ASAPQuery `O11yMetricsQuery` ingestion model. -- Independent source/protocol adapters, beginning with Prometheus query and - rule adapters, that produce `O11yMetricsQuery` values. -- A thin `O11yMetricsQuery -> ASAPPlanner QueryWorkload` conversion. -- An ASAPQuery `DeploymentPlan` builder that consumes canonical roots plus - ASAPPlanner `GlobalSelection`, applies the chosen replacements in a - deployment-aware way, assigns placement, and compiles a `CompiledPlan` - (`CollectorSubplan` + `BackendSubplan`, sharing one `plan_id`) — see - [`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). -- An explain/debug endpoint over ASAPPlanner's existing DAG export and - replacement-explanation types. -- Planning phase timings, search-size metrics, deadlines, and cancellation. - -### 4.4 Integration contract across all open ASAPPlanner PRs - -The integration must track ASAPPlanner's public planning types and APIs, not -copy proposed branch types into ASAPQuery. As of 2026-08-27, every open -ASAPPlanner PR falls into one of the following classes: - -| PR | Change | ASAPQuery integration consequence | Runtime-wire consequence | -| --- | --- | --- | --- | -| [#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) | Adds `ExactTransform`, `ExactPostProcess`, `ExecutionAvailability`, `PhaseAssignment`, and phase validation for exact/summary composition | When merged, partition the selected DAG by validated execution availability rather than re-deriving update/readout stages from node names. Advertise mixed-execution capabilities through the deployment cost model. | Collector plan accepts update-side exact transforms only when the collector advertises them. Backend plan carries readout-side exact post-processing. Invalid readout-under-maintenance plans are rejected before physical compilation. | -| [#299](https://github.com/ProjectASAP/ASAPPlanner/pull/299) | Adds typed end-to-end `ResultGuarantee`, error metrics, bound/probability expressions, provenance, budget allocation, and rejected candidates | Use `search_workload_with_targets`; pass root `QueryRequirements.accuracy`; preserve the selected guarantee and rejection reason. Never rank a candidate that upstream rejected for accuracy. | Backend materializations/routes retain the selected guarantee. Collector plan retains the original accuracy constraint and concrete summary parameters; query reports compare the backend result against the selected guarantee. Unknown guarantees fail closed. | -| [#295](https://github.com/ProjectASAP/ASAPPlanner/pull/295) | Adds recurrence-aware CSE cost profiles and `global_selection_with_recurrence` | Convert `RepeatingEntry` intervals plus ingest/update statistics into upstream recurrence inputs and use recurrence-aware global selection. One-shot/repeating mixtures supply the required horizon explicitly. | None directly. Scheduling, watermark, lateness, and retention remain downstream runtime policy. | -| [#293](https://github.com/ProjectASAP/ASAPPlanner/pull/293) | Lets `CostModel` explicitly opt `TopK { accuracy: Exact }` into approximate heap-sketch candidates with a real sizing target | Implement `topk_exact_accuracy_target` only if product policy permits approximation for an exact-requested TopK. The opt-in returns an explicit non-Exact budget; it must never use a hidden default or clamp-sized pseudo-budget. | The selected plan records that the realization is approximate, its effective target, algorithm, parameters, and guarantee. Pass-through remains a candidate/fallback. | -| [#291](https://github.com/ProjectASAP/ASAPPlanner/pull/291) | Adds optional caller-proven `Concat` discriminator unique-key metadata | Update exhaustive `QueryExpr` visitors and preserve the metadata through canonical roots. Do not invent discriminator keys downstream. | No new collector primitive. A retained `Concat` stays backend/archive-side unless a later selected strategy gives it an executable summary realization. | -| [#296](https://github.com/ProjectASAP/ASAPPlanner/pull/296) | Adds workload cost/benefit annotations to DAG export and viewer | Consume the upstream explain/export fields in the optional explain endpoint. Do not recompute or scrape viewer output. | None; explain metadata is not a runtime plan identity or wire contract. | -| [#292](https://github.com/ProjectASAP/ASAPPlanner/pull/292) | Corrects DAG-viewer node categories and adds drift checks | No planning dependency. Accept new upstream explain kind/category output when the pinned revision includes it. | None; viewer categories must never drive placement or execution. | - -This table is an integration audit, not a requirement to wait for every PR or -to combine unmerged branches. ASAPQuery pins one immutable, tested -ASAPPlanner revision. Code is written against the public API at that revision; -when an upstream PR merges, the pin-update PR adds the corresponding adapter, -compiler, capability, and golden-test changes in the same commit. - -The open PRs must not be imitated locally in advance. In particular: - -- do not define backend copies of `ExecutionAvailability`, `ResultGuarantee`, - recurrence profiles, or `ConcatDiscriminatorKey`; -- do not use DAG-export JSON, viewer categories, decision rationale strings, - or debug node IDs as runtime input; -- do not enable a newly nameable summary/phase until the collector and data - plane advertise compatible execution capabilities; and -- do not silently discard a new field when exhaustive upstream types change. - Compilation must fail with a typed unsupported-shape diagnostic until the - new variant is deliberately mapped. - -The integration test matrix has three lanes: - -1. **Pinned baseline:** build and run against the single revision in - `Cargo.lock`. -2. **Pin update:** for each newly merged contract-affecting Planner PR, update - all Planner crates together and run cross-repository golden workloads. -3. **Capability mismatch:** deliberately select or decode a plan shape that - one executor does not support and prove planning fails before either - subplan is activated. - -## 5. Migration stack - -Each phase should land as a separate, buildable PR. Later PRs may be stacked -while earlier ones are under review. - -### PR 1: ASAPPlanner pin and API compatibility - -Move every ASAPPlanner dependency to the same immutable revision of `main`. -Do not pin to an open PR branch and do not combine commits from several open -branches. Record the Planner commit in build metadata and emitted plan -artifacts. - -Update together: - -- `planner-types` in `control_plane`, `data_plane`, and the local shared-types - crate; -- `asap-aware-mapping`; -- `asap-frontend-promql`; -- `Cargo.lock`. - -The new mapping API removes the former `bind` and `boundary` modules. Migrate -call sites to `asap_aware_mapping::replacement` and its public re-exports: - -- `Implementation` and candidate enumeration; -- `ReplacementStrategy` and `ReplacementSubDAG`; -- `search_workload_with` and `default_strategies_with`; -- `PlanSpace::global_selection`; -- `replacement::default_size_params`. - -Adapt the ASAPQuery cost model to the public API present at that pin: - -- rank `SketchAlgorithm` values and return an exact permutation of the input; -- size `SketchParams` for the selected algorithm; -- preserve deployment-specific extension realization; -- supply CSE recompute and shared-maintenance costs; -- supply subpopulation estimates and grouping-state costs when statistics are - available; -- expose numeric `estimate_cost` values for observability. - -When the pin contains the corresponding open-PR work described in §4.4: - -- #295: supply update/evaluation rates, one-shot counts, horizon, maintenance, - read, build, and raw-recompute costs, then call recurrence-aware global - selection for mixed workloads; -- #293: implement the TopK-Exact opt-in hook only with an explicit approved - approximation budget; -- #299: supply accuracy propagation statistics and call the target-aware - workload search API; and -- #300: advertise only the exact-transform/post-process phase capabilities - the deployed collector and backend actually execute. - -Missing runtime statistics remain unknown, never numeric zero. An unavailable -hook or type at the pinned revision is simply absent from the adapter; it is -not recreated as a backend-local compatibility type. - -Use ASAPPlanner's `AccuracyTarget` as the only correctness/accuracy input -model. During compatibility migration, `control_plane::types_v2` may re-export -that upstream type, but ASAPQuery must not define a second semantic equivalent. -Delete after API migration: - -- the legacy `accuracy_sla: f64` fields and their `1.0 - accuracy_sla` - conversions; -- the data-plane routing `AccuracyTarget::{Exact, Approximate}` enum; -- any backend-local `CorrectnessPolicy` proposal. - -Routing, placement, and execution should consume the selected plan and the -original upstream `AccuracyTarget`, not collapse it to an exact/approximate -boolean. - -Acceptance criteria: - -- `cargo build --workspace` succeeds; -- existing control-plane and data-plane tests pass; -- no ASAPPlanner crate is pinned to a different revision; -- the recorded Planner revision matches every linked Planner crate; -- the compatibility path does not change production output yet. - -### PR 2: Adopt ASAPPlanner's canonical workload input - -Add a new API such as `POST /api/v1/workloads/plan`. Its JSON is decoded by a -source adapter into ASAPQuery's adapter-neutral `O11yMetricsQuery`, then -converted into ASAPPlanner's existing workload types. Neither layer is a -second planner-domain model: - -```json -{ - "queries": [ - { - "id": "q1", - "language": "promql", - "text": "sum by (service) (rate(requests_total[5m]))", - "accuracy": { "epsilon": 0.01 }, - "schema_refs": [] - } - ], - "table_schemas": [] -} -``` +Both plans carry the same plan and materialization identities. Neither plan +is activated unless both sides validate the same compiled decision. + +## 1. Goals -Requirements: - -- stable, caller-visible query IDs; -- map every accuracy field directly into - `QueryRequirements.accuracy: Option`; -- map one-shot queries into `BatchEntry` and scheduled queries into - `RepeatingEntry`; -- use ASAPPlanner's `QueryLanguage` rather than the backend's local language - enum; -- group mixed-language API requests into one ASAPPlanner `QueryWorkload` per - language until upstream supports mixed languages in one workload; -- pass table schemas through ASAPPlanner's existing `SchemaCatalog` binder - before enabling SQL; -- request limits and validation; -- old single-query endpoints adapt into a one-query workload rather than - maintaining a second planner. - -Use a module boundary such as: +The migration must produce one planning path that: + +- accepts one-shot and repeating PromQL workloads, with SQL enabled only when + a real schema catalog is available; +- preserves workload-wide sharing instead of flattening queries into + independent metric/sketch requests; +- uses ASAPPlanner's selected summary family, algorithm, parameters, + reduction, grouping strategy, readout, and accuracy contract exactly; +- compiles that selection into compatible collector and backend plans; +- lets the data plane execute the selected plan without planning again; +- rejects unsupported or incompatible plans before activation; +- supports shadow comparison and rollback during migration; and +- removes the legacy planner only after the new path passes end-to-end gates. + +## 2. Non-goals + +This migration does not: + +- move physical placement or scheduling into ASAPPlanner; +- serialize ASAPPlanner's internal Rust DAG as a runtime wire format; +- use DAG-viewer JSON, node colors, explain IDs, or rationale strings as + execution input; +- require every summary family or every open Planner proposal to be supported + by the MVP runtime; +- remove exact raw/archive fallback; +- reimplement Prometheus rule scheduling or alert state in ASAPPlanner; or +- delete the legacy path before shadow validation and rollback exist. -```text -control_plane/src/o11y_query/ - mod.rs # O11yMetricsQuery and generic conversion - adapters/ - mod.rs # adapter trait/error contract - prometheus/ - query.rs # Prometheus instant/range request adapter - rules.rs # Prometheus rule-file adapter - runtime.rs # Prometheus-only scheduling/alert metadata -``` +## 3. Ownership + +### ASAPPlanner owns logical planning + +ASAPPlanner is the canonical owner of: + +- query parsing and semantic lowering; +- pre-ASAP IR and canonicalization; +- schema binding; +- workload-level common-subexpression discovery; +- exact and summary-based candidate generation; +- summary family, algorithm, parameters, and grouping alternatives; +- logical rewrites and reuse opportunities; +- accuracy requirements and, when supported by the pinned revision, result + guarantees; +- cost-aware candidate search and global selection; and +- logical explain and rejection information. -Keep this as an independent dependency boundary even if it initially lives in -the `control_plane` crate: +### ASAPQuery-backend owns physical planning and rollout -```text -adapters/prometheus -> O11yMetricsQuery -> ASAPPlanner workload types -ASAPPlanner -X-> adapters/prometheus -planner/search -X-> Prometheus rule/runtime types -``` +The ASAPQuery control plane owns: + +- source/protocol adapters and caller identity; +- runtime statistics, budgets, and its ASAPPlanner cost-model implementation; +- executor capability discovery; +- collector/backend/archive placement; +- physical streaming windows and lateness policy; +- raw, full-summary, and delta-summary transmission decisions; +- materialization identity and storage routing; +- creation of matching `CollectorPlan` and `BackendPlan` artifacts; +- deployment diff, warm-up, activation, retirement, and rollback; and +- planning telemetry and operator-facing explain endpoints. -It can move into a workspace adapter crate later without changing planner or -deployment APIs. - -`O11yMetricsQuery` contains only source-independent identity plus one existing -ASAPPlanner workload entry. It must not flatten and copy the fields of -`BatchEntry`/`RepeatingEntry`, or copy `QueryExpr`, `AccuracyTarget`, or -`QueryLanguage` into new local enums: - -```rust -struct O11yMetricsQuery { - id: Id, - language: QueryLanguage, - entry: O11yWorkloadEntry, -} - -enum O11yWorkloadEntry { - OneShot(BatchEntry), - Repeating(RepeatingEntry), -} -``` +### ASAPQuery data plane owns execution + +The data plane owns: + +- backend-plan installation and atomic hot reload; +- summary-state ingestion, validation, storage, and merge; +- query-time readout and remaining backend-side logical operators; +- readiness, freshness, and watermark checks; +- exact archive fallback; and +- Prometheus-compatible request and response behavior. + +### ASAPCollector owns collector-plan execution + +ASAPCollector owns: -The generic converter groups entries by `QueryLanguage` and moves their -existing entry values into `QueryWorkload`. Caller IDs remain in a parallel -adapter-owned vector/map. ASAPPlanner frontend lowering returns results in -entry order; ASAPQuery zips those results back to caller IDs before passing -`(Id, Rc)` roots to CSE/search. It performs no query parsing, -canonicalization, accuracy conversion, or schedule interpretation. Caller IDs -do not require a planner `QueryId` type or a new named-lowering API. - -Define the adapter contract around an input and two deliberately separated -outputs: - -```rust -trait O11yMetricsAdapter { - type PrivateMetadata; - - fn adapt( - &self, - input: Input, - ) -> Result, AdapterError>; -} - -struct AdaptedMetricsInput { - queries: Vec>, - private_metadata: M, -} -``` +- validating and atomically applying its `CollectorPlan`; +- computing the specified materializations; +- emitting raw, full, or delta payloads as directed; +- rejecting unsupported families, parameters, grouping layouts, or + transmission modes; and +- reporting semantic plan activation and emitted-state evidence. -Only `queries` crosses into planning. `private_metadata` remains owned by the -adapter/runtime integration and is indexed by caller ID. For an ordinary -Prometheus query it can be empty; for a rule file it contains rule groups and -alerting semantics. +## 4. Stable integration contracts -Do not extend the existing data-plane `QueryRequestAdapter` for this purpose. -That trait is an Axum/HTTP serving adapter coupled to execution timestamps and -`QueryResult` response formatting. Planning ingestion adapters belong in the -control-plane boundary and share only protocol parsing utilities where useful. +### 4.1 Workload input + +Protocol adapters convert external requests into ASAPPlanner's canonical +workload model. They may attach caller IDs outside the Planner value, but they +must not copy Planner domain types into a second backend schema. + +For each query, the planning input contains at least: + +- query language and expression; +- one-shot or repeating evaluation shape; +- requested accuracy; +- optional latency requirement; +- schema/catalog reference when required; and +- protocol-neutral recurrence information. + +Prometheus-only behavior remains adapter/runtime metadata, including rule +group ordering, query offset, missed iterations, `for`, `keep_firing_for`, +labels, annotations, and Alertmanager state. + +### 4.2 Planner output + +The output consumed by ASAPQuery is one selected post-ASAP DAG for the whole +workload, with shared node identity intact. It may contain: + +- `SummaryAgg` producers; +- exact accumulators and approximate summaries; +- `SummaryEstimate` readouts; +- reductions and grouping strategies; +- summary merge/subtract/delete/join operations; +- logical rewrites and shared sub-DAGs; and +- `KeepPreAsap` exact fallback subtrees. + +ASAPQuery does not flatten this DAG into `(metric, query type, sketch)` rows +before placement. Doing so would lose sharing, composition, grouping, rollup, +and provenance. + +### 4.3 Physical compile -ASAPPlanner already owns canonicalization and workload CSE. The backend should -invoke those existing paths and preserve the resulting named -`Rc` roots; it must not implement its own AST normalization, -semantic equality, or `QuerySetPlan` capability flattening. +The physical compiler consumes exactly one selected DAG plus deployment +topology, capabilities, statistics, and resource constraints. It produces one +`CompiledPlan` with: -Acceptance criteria: +- a `CollectorSubplan` containing one `CollectorPlan` per targeted collector; +- a `BackendSubplan` containing the matching `BackendPlan`; +- shared `plan_id`, `plan_version`, `activation`, `expiry`, and + `backend_compat` values; and +- shared content-addressed materialization identities. + +The collector and backend portions are emitted by the same compile operation. +Two independent compilers must not reinterpret the selected DAG separately. + +The detailed split is defined in +[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). -- one and multiple queries enter the same code path; -- query IDs survive lowering and error reporting; -- equivalent subtrees remain shareable across roots; -- malformed queries and schemas return per-query diagnostics. +### 4.4 Collector interface -### PR 3: Complete repeating workload lowering in ASAPPlanner +The collector half follows ASAPCollector's +[collection-plan interface](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md): -Complete the generic upstream lowering gap: +- OpAMP protobuf is the delivery envelope; +- `asap-collector-plan.yaml` is the exact config-map entry; +- the entry contains a versioned `CollectorPlan`, not a complete OTel + bootstrap configuration; +- summary family, algorithm, parameters, accuracy, reduction, and grouping + retain Planner semantics; +- source binding, windows, placement, and transmission are added by the + physical compiler; and +- semantic activation requires the collector application report, not only + `RemoteConfigStatus.APPLIED`. -- PromQL/SQL lowering for `repeating_queries` as well as `query_batch`, with - one result per entry in input order. +### 4.5 Backend interface -No caller ID, ASAPQuery, Prometheus, placement, wire, or runtime types enter -these APIs. +`BackendPlan` carries: -If ASAPPlanner #295 is present at the pinned revision, lowering and selection -remain separate operations: the frontend lowers every `RepeatingEntry`, then -ASAPQuery supplies the entry intervals and workload ingest rate to -`recurrence_profiles` and calls `global_selection_with_recurrence`. The -Prometheus adapter owns rule-group scheduling; the Planner receives only -protocol-neutral recurrence and cost inputs. +- the shared compiled-plan envelope; +- every planned materialization and its exact `SummaryFamilyType`; +- source, filter, window, reduction, grouping, and storage route; +- collector source references; +- query capabilities and readouts satisfied by each materialization; and +- the selected result guarantee when the pinned Planner revision supplies + one. -Acceptance criteria: +The data plane builds its routing index from this plan. It must not run a cost +model or infer summary parameters from stored state at query time. -- batch and repeating entries return one canonical root per input entry in - stable order; -- two repeating consumers with different intervals contribute the sum of - their evaluation rates to a shared sub-DAG; -- missing or invalid recurrence statistics produce a typed failure or the - documented structural fallback, never a fabricated zero cost; and -- one-shot/repeating mixtures require an explicit costing horizon. +The detailed backend contract is defined in +[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md). -### PR 4: Workload-wide search and deployment selection +## 5. Planner version and open-PR policy -Run one planning operation for the complete workload: +ASAPQuery pins every ASAPPlanner crate to one immutable revision of Planner +`main`. It never combines several open PR branches in production. A pin update +changes all Planner crates and the lockfile together. -```rust -let strategies = default_strategies_with(&backend_cost_model); -let space = search_workload_with(roots, &strategies); -let selection = space.global_selection(&backend_cost_model); -``` +Open Planner work is handled according to its integration effect: -The snippet is the baseline API. When the pinned revision contains #299, -construct the space with root accuracy targets and the deployment accuracy -model. When it contains #295, select with the recurrence-aware API. When both -are present, accuracy rejection happens before recurrence-aware cost ranking; -cost must never resurrect an accuracy-invalid candidate. - -Build ASAPQuery's `DeploymentPlan` from the canonical roots plus -`GlobalSelection`. This is the downstream commitment/materialization boundary -ASAPPlanner's crate documentation assigns to a deployment: apply chosen -`Replacement::Summary`/`Replacement::Rewrite` alternatives, preserve shared -node identity, validate executor support, and decide placement. Keep this -logic in one control-plane module so explain, stage allocation, and -`BackendPlan` generation cannot implement competing substitutions. - -This phase should activate and test: - -- `SketchAlgorithmStrategy`; -- `SharedSubtreeStrategy`; -- `HydraGroupingStrategy`; -- `AvgToSumOverCountStrategy`; -- `RollupStrategy`; -- `TopKLimitReuseStrategy`. - -ASAPPlanner deliberately owns candidate search but not ASAPQuery deployment -placement. Do not copy `dag_export`'s JSON into the runtime contract and do -not infer mappings from labels, hashes, strategy rationale, or viewer node -signatures. - -Acceptance criteria: - -- compatible scans and sub-DAGs have one shared node identity; -- incompatible filters, windows, groupings, orderings, and schemas do not - share; -- every chosen replacement records its real strategy name and target; -- selection is deterministic for identical workload, statistics, and cost - model inputs. -- an upstream accuracy rejection remains rejected under every downstream cost - or placement decision; -- a TopK-Exact sketch candidate exists only when the #293 hook explicitly - supplies its effective approximation target. - -### PR 5: Selected workload to `CompiledPlan` (collector + backend subplans) - -Introduce a direct compile step, not a `BackendPlan`-only conversion — see -[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md) -for the full design this stack step implements: +| Planner PR | Integration effect | Adoption rule | +| --- | --- | --- | +| [#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) | Adds explicit update/readout phases and exact-summary composition | After merge, use validated execution availability to partition collector and backend operators. Enable only phases supported by both executors. | +| [#299](https://github.com/ProjectASAP/ASAPPlanner/pull/299) | Adds typed result guarantees, error propagation, budgets, and accuracy rejections | After merge, supply root accuracy targets, reject invalid guarantees before cost ranking, and preserve the selected guarantee in `BackendPlan`. | +| [#295](https://github.com/ProjectASAP/ASAPPlanner/pull/295) | Adds recurrence-aware CSE and global selection | After merge, provide evaluation/update rates and an explicit horizon for mixed one-shot/repeating workloads. Scheduling remains outside Planner. | +| [#293](https://github.com/ProjectASAP/ASAPPlanner/pull/293) | Allows an explicit approximate TopK candidate for an Exact-requested TopK | Opt in only through the upstream hook with an approved non-Exact sizing target. Record the effective approximation; retain exact pass-through. | +| [#291](https://github.com/ProjectASAP/ASAPPlanner/pull/291) | Adds optional `Concat` discriminator unique-key metadata | Preserve it in exhaustive IR visitors. Do not invent keys downstream. It creates no collector primitive by itself. | +| [#296](https://github.com/ProjectASAP/ASAPPlanner/pull/296) | Adds explain/viewer cost annotations | Consume only in explain output. Never use it for execution, identity, or placement. | +| [#292](https://github.com/ProjectASAP/ASAPPlanner/pull/292) | Corrects viewer node categories | No runtime integration effect. | -```text -canonical roots + GlobalSelection - -> ASAPQuery DeploymentPlan - -> apply choices + deployment placement - -> compile into CompiledPlan { plan_id, plan_version, activation, expiry, - backend_compat, collector, backend } - collector: CollectorSubplan (versioned CollectorPlan YAML per edge, - carried by OpAMP with config_hash) - backend: BackendSubplan (BackendPlan: materializations + routing) -``` +This table is an audit, not a merge dependency list. The migration proceeds +against the pinned baseline. When a contract-affecting PR merges, the pin +update must include its adapter/compiler/capability changes and golden tests +in the same backend PR. + +ASAPQuery must not pre-copy proposed Planner types such as phase assignments, +result guarantees, recurrence profiles, or discriminator keys. Until a type +exists at the pin, it is absent. When a new upstream variant appears, physical +compilation fails with an unsupported-shape diagnostic until it is mapped +deliberately. + +## 6. Accuracy and cost + +### Accuracy + +`AccuracyTarget` is the requested correctness constraint. If the pinned +Planner revision supplies `ResultGuarantee`, that is the computed guarantee +of the selected result. They are different values and both remain owned by +Planner. -`SummaryAgg` nodes compile into the collector subplan (update side); -`SummaryEstimate`/`SummaryMerge`/`SummarySubtract`/`SummaryDelete`/ -`SummaryJoin` subtrees compile into the backend subplan (readout side) — -this split is structural, derived from the selected DAG's own node kinds, -not a second per-metric classification pass. A selected DAG must never be -serialized directly as collector processor configuration: neither `edge_id`, -shard count, streaming window, transport mode, nor any other physical -parameter exists in ASAPPlanner's output. The physical compiler adds those -decisions and emits the versioned `CollectorPlan` contract defined by -ASAPCollector. That assignment is what turns one selection into two -*agreeing* subplans instead of two independently guessed ones. - -Continue using `PolicyFingerprint` for persistent runtime identity *within* -one subplan (materialization reuse/diff/resize across replans). `plan_id` -answers a different question — whether the collector subplan and the -backend subplan now active were compiled together — and is carried -identically on both (see the compile doc §3/§7 for why `BackendPlan`'s -`plan_id` is this identity, not a second one). Exporter IDs such as DAG node -IDs or `workload_node_id` are scoped to an explain result and must not become -materialization keys. - -The two wire contracts contain typed fields for: - -- exact versus sketch summary family, using `SummaryFamilyType` directly - (see the compile doc §7 and backend wire-format doc §3); -- sketch algorithm and parameters; -- independent versus Hydra grouping layout; -- selected result guarantee and its machine-readable provenance when #299 is - present at the Planner pin; -- execution phase/availability and exact transform/post-process operators when - #300 is present at the Planner pin; -- shared materialization dependencies, including which `EdgeAssignment`(s) - supply a materialization's input summary state (`Materialization.sources` - in the compile doc §7) — the field that lets the backend reject a plan - whose collector subplan doesn't actually produce what this materialization - expects; -- multiple query/readout consumers of one materialization; -- derived readouts such as `avg = sum / count`, rollups, and top-k prefix - reuse. - -Use additive protobuf fields and retain backward decoding during rollout. - -Acceptance criteria: - -- two queries sharing one summary produce one materialization and multiple - legal routes/readouts; -- materialization fingerprints are stable across replans; -- protobuf encode/decode and hot reload preserve the chosen plan; -- the OpAMP entry is exactly `asap-collector-plan.yaml` with - `application/yaml`, and bootstrap OTel configuration is not overwritten by - a workload replan; -- the data plane never has to run a cost model to reconstruct the choice; -- a `CollectorSubplan` and `BackendSubplan` produced by the same compile - call carry the same `plan_id`/`plan_version`, and a deliberately - mismatched pair (e.g. an old collector config against a new - `BackendPlan`) is detectable from those fields alone, without needing to - diff YAML against protobuf by hand; -- `RemoteConfigStatus.APPLIED` alone cannot pass activation: the collector's - semantic application report and emitted materialization identities must - agree with the backend plan. - -### PR 6: Data-plane execution coverage - -Make the data plane execute or explicitly reject every post-ASAP shape the -control plane may select. Cover at least: - -- exact accumulators; -- sketch aggregation and estimate; -- sum/count composition for rewritten Avg; -- reading top-k `k_small` from a compatible `k_large` summary; -- rollup derivation; -- Hydra grouping layouts; -- shared summary dependencies and merges. - -Unsupported candidates must be removed before selection or fail planning with -a clear capability diagnostic. They must never be accepted by the control -plane and fail later during query serving. - -The exact coverage list is derived from the pinned Planner revision and the -complete open-PR audit in §4.4. In particular, a pin containing #300 adds -exact-transform/post-process and phase-validation cases; a pin containing -#299 adds guarantee propagation, budget, rejection, and unknown-guarantee -cases; a pin containing #291 adds exhaustive retained-`Concat` handling. -Viewer-only changes in #296/#292 add no execution cases. The migration does -not wait for open PRs, but every pin update expands this matrix in the same PR -that changes the upstream types. - -Acceptance criteria: - -- control plane selects once and the data plane consumes that exact choice; -- family, parameters, grouping, and accuracy are not re-derived at serving - time; -- result guarantees and phase assignments present in the selected Planner DAG - survive physical compilation without being weakened or guessed; -- archive fallback remains available for unsupported queries; -- end-to-end tests prove plan push, hot reload, and serving. - -### PR 7: Explain and planner observability - -Add an optional explain response or endpoint containing: - -- the original pre-ASAP DAG; -- the selected post-ASAP DAG; -- explicit pre-target -> decision -> post-node mappings; -- strategy, concise rationale, rank, and estimated cost; -- query ownership of shared nodes; -- edge schemas; -- phase timings. - -Use ASAPPlanner's existing `DagGraph`, `DagDecision`, `TargetReplacement`, -`WorkloadGraph`, `export_post_asap`, and replacement-explanation APIs. Do not -define a second backend graph/decision schema. Production planning calls these -Rust APIs directly; it must not start the Python viewer server or invoke the -`dag_export` binary as a subprocess. - -Record at least: - -- parse and schema-binding time; -- pre-ASAP IR and CSE time; -- replacement search time; -- global-selection time; -- post-ASAP materialization time; -- `BackendPlan` construction and push time; -- target, candidate, shared-node, and selected-materialization counts. - -### PR 8: Shadow rollout and legacy deletion - -Introduce a feature flag such as `ASAP_WORKLOAD_PLANNER_V2` and use three -rollout modes: - -1. `legacy`: legacy planner emits; new planner is disabled. -2. `shadow`: legacy planner emits; new planner runs and differences are - recorded. -3. `selected`: new planner emits; legacy planner is available only for - rollback. - -Compare: - -- selected summary family and parameters; -- materialization count and fingerprints; -- routing coverage; -- estimated resource cost; -- warm versus archive placement; -- planning latency and errors. - -After one release cycle without unexplained mismatches: - -- delete the capability-union production planner; -- delete per-root implementation code; -- delete duplicate replacement decisions in optimizer/lowering modules; -- delete serving-time cost-based reconstruction; -- remove the legacy `BackendStageConfig -> BackendPlan` path once no other - caller requires it; -- keep only the single-query API adapters, not a single-query planner. - -## 6. Recurring workloads and Prometheus rules - -Repeated execution is a workload property, not an execution loop inside the -query planner. ASAPPlanner should compile a recurring query into a reusable -temporal plan. A scheduler, ruler, or materializer triggers that plan at each -evaluation timestamp. - -The recommended first deployment is a **materializer**: ASAP accelerates and -materializes the expensive numeric expression while Prometheus remains the -authority for rule scheduling, pending/firing state, `for`, -`keep_firing_for`, and Alertmanager integration. +The required order is: ```text -Prometheus instant/range queries ----+ - | -Prometheus rule files ---------------+--> adapters/prometheus - | | - | +--> Prometheus-only - | runtime metadata - | (adapter boundary) - v - O11yMetricsQuery - | - v - ASAPPlanner QueryWorkload - | - v -ASAPQuery deployment ----------> shared temporal aggregations - | window, retention, labels, summary policy - v -ASAP QueryEngine - | - +----> optional ASAP-aware ruler - | - +----> materializer writes derived series to Prometheus - | - v - Prometheus evaluates alert state - (`for`, `keep_firing_for`, Alertmanager) +candidate generation + -> guarantee propagation + -> reject candidates outside the requested target + -> cost ranking among valid candidates + -> global selection + -> physical placement ``` -Prometheus rule groups have semantics that a simple cron loop does not -provide: rules in a group use the same evaluation timestamp, execute -sequentially, and skip scheduled evaluations while the previous group -evaluation is still running. The rule importer and trigger service must -preserve these semantics. See the -[Prometheus recording and alerting rule documentation](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/). - -Historical ASAPQuery planner code is useful migration input here: it infers -`repetition_delay_ms` from the median query-log inter-arrival time -([frequency.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/query_log/frequency.rs)) -and exposes per-query-group repetition configuration -([input.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/config/input.rs)). -That inference remains useful for ad hoc dashboards. A declared rule schedule, -however, is authoritative and must not be replaced by frequency inference. - -### 6.1 Use ASAPPlanner's existing repeating-workload model unchanged - -ASAPPlanner already defines `QueryRequirements.accuracy: Option`, -`RepeatingEntry`, and `RepetitionInterval`. They already contain everything -replacement planning needs: the expression, repetition interval, accuracy, -and optional latency requirement. Do not add a parallel ASAPQuery workload -type and do not add `phase`, `query_offset`, missed-tick behavior, or rule-group -identity to ASAPPlanner's model. - -```rust -struct RepeatingEntry { - query: Query, - interval: RepetitionInterval, - requirements: Option, -} -``` +Cost and placement cannot resurrect an accuracy-invalid candidate. Unknown +error bounds, probabilities, or required statistics remain unknown; they are +never converted to zero or exact. -The Prometheus rules adapter returns two linked outputs: - -1. generic `O11yMetricsQuery` values, subsequently converted into existing - ASAPPlanner `RepeatingEntry` values; -2. scheduler-only rule metadata containing group identity/order, - `query_offset`, missed-tick behavior, `for`, and `keep_firing_for`. - -Both the Prometheus query adapter and rules adapter live under the same -independent `adapters/prometheus` boundary. Adapter output (1) crosses into -ASAPQuery core only as `O11yMetricsQuery`; output (2) remains in the -Prometheus adapter/runtime integration. It never enters `O11yMetricsQuery`, -`QueryWorkload`, lowering, replacement search, post-ASAP IR, or `PlanSpace`. -ASAPPlanner receives no Prometheus query-protocol or rule-group concept at any -layer. Normalized expression text is derived from ASAPPlanner's canonical -pre-ASAP IR and must not be stored as a second caller-controlled truth. - -Example normalized input: - -```yaml -query_groups: - - id: high-error-rate - schedule: - interval_ms: 30000 - phase_ms: 0 - query_offset_ms: 15000 - missed_tick_policy: skip - - rules: - - alert: HighErrorRate - expr: | - sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) - / - sum by (service) (rate(http_requests_total[5m])) - > 0.05 - for_ms: 300000 - keep_firing_for_ms: 60000 - accuracy: - epsilon: 0.002 - delta: 0.01 -``` +`AccuracyTarget::Exact` normally selects an exact summary or `KeepPreAsap`. +The only planned exception is the explicit TopK policy exposed by Planner +#293. If enabled, the compiled plan states that the chosen realization is +approximate and records its effective target and guarantee. -Keep these time concepts separate: - -- `interval_ms` controls how often the instant expression is evaluated. -- `[5m]` in PromQL is the data lookback. -- `query_offset_ms` changes the logical evaluation timestamp, not the - interval. -- `for_ms` and `keep_firing_for_ms` belong to alert-state management, not - aggregation planning. -- A rule evaluation is an instant query even when its expression contains - range vectors such as `[5m]`. - -### 6.2 Temporal alignment, watermark, and retention - -The deployment plan, rather than ASAPPlanner's workload model, records runtime -alignment metadata: - -```rust -struct WindowPlan { - size_ms: u64, - anchor_epoch_ms: i64, - allowed_lateness_ms: u64, - retention_buckets: u64, -} -``` +### Cost -For a scheduler trigger at timestamp `T`, the scheduler invokes QueryEngine at -the already-offset logical timestamp: +ASAPQuery supplies deployment-specific cost information through Planner's +cost-model interface. Inputs may include: -```text -logical evaluation time = T - query_offset -QueryEngine receives logical evaluation time -runtime reads only buckets whose watermark covers that time -``` +- ingest/update rate; +- query evaluation rate; +- summary maintenance and read cost; +- exact recomputation cost; +- initial materialization cost; +- expected cardinality and subpopulation count; +- memory, network, and storage budgets; and +- the comparison horizon for mixed recurring and one-shot work. -The common anchor is required for correctness. Equal-duration buckets with -different boundaries cannot be merged as though they represented the same -logical interval. +Missing statistics remain unknown. Structural fallback is allowed only when +Planner defines it explicitly. Backend code must not treat missing data as a +free operation. -The historical window planner chooses tumbling windows from the repeat -interval and range-query step, and explicitly disables sliding windows because -they crash Arroyo -([window.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/planner/window.rs)). -Keep the first recurring-rule implementation conservative: +Physical placement applies deployment constraints after logical selection. It +does not choose a different logical summary because a selected placement is +inconvenient; it either finds a valid placement, asks Planner to select among +capability-constrained candidates, or fails planning. -- use anchored tumbling panes; -- require pane size to divide the evaluation interval; -- require pane size to divide the range-query step, when present; -- require pane size to be at least the scrape interval; -- require pane size not to exceed the expression lookback; -- retain enough closed panes for the maximum lookback plus allowed lateness. +## 7. Repeating workloads -When several rules need the same aggregation at different compatible -intervals, materialize the smallest compatible pane and merge panes for the -slower rule. Do not allocate one streaming aggregation per alert interval. +ASAPPlanner receives protocol-neutral repeating-query intervals. These +intervals may affect workload CSE and cost selection, but they do not make +Planner a scheduler. -### 6.3 Accuracy requirements, guarantees, and exact fallback +Keep these concepts separate: -Do not add `CorrectnessPolicy::{Exact, Approximate, ExactOrValidate}`. -ASAPPlanner's existing `AccuracyTarget` is the single source of truth: +- evaluation interval: how often the query runs; +- query lookback: the data range in the query, such as `[5m]`; +- physical pane size: chosen by the ASAPQuery physical compiler; +- query offset: changes the logical evaluation timestamp; +- allowed lateness and watermark: runtime readiness policy; and +- alert `for`/`keep_firing_for`: Prometheus alert-state behavior. -```rust -AccuracyTarget::Exact -AccuracyTarget::Epsilon(epsilon) -AccuracyTarget::EpsilonDelta { epsilon, delta } -``` +For the MVP, physical windows use anchored tumbling panes. A chosen pane must +compose into every claimed query range, and the backend must query only panes +whose watermark covers the logical evaluation time. Incompatible window +alignment is a planning failure, not an approximate answer. -By default, `Exact` excludes approximate candidates. A node for which no valid -exact ASAP replacement exists remains `KeepPreAsap`, which means ASAPQuery -executes the original pre-ASAP subtree from raw/archive data. `Epsilon` and -`EpsilonDelta` allow ASAPPlanner to choose a summary sized to that target. +When multiple recurring queries can reuse one materialization, recurrence- +aware selection uses their combined evaluation rate. Prometheus schedules +remain authoritative; inferred dashboard frequency must not override a +declared rule interval. -If the pinned revision contains #293, an ASAPQuery deployment may explicitly -offer approximate heap-sketch candidates for `TopK { accuracy: Exact }`, but -only by returning a concrete non-Exact sizing target from the upstream cost -model hook. This is a visible product-policy exception: pass-through remains -available, and the compiled plan records the effective approximation target. -It must not be generalized to other Exact intents. +## 8. Capability and failure contract -If the pinned revision contains #299, `AccuracyTarget` is the requested -constraint and `ResultGuarantee` is the Planner-computed guarantee of a -selected result. They are not interchangeable. ASAPQuery: +Before selection and again before activation, ASAPQuery validates the chosen +plan against collector and backend capability snapshots. -1. supplies root targets to target-aware workload search; -2. lets Planner reject unknown or insufficient composed guarantees before - cost ranking; -3. preserves the selected guarantee, metric, probability bound, and - provenance in `BackendPlan` and explain output; and -4. treats an unknown guarantee as unavailable, never exact and never zero. +The following conditions fail closed: -The CollectorPlan carries the original constraint and concrete summary -parameters needed to build state. The backend plan carries the selected -result guarantee because readout and composition occur there. The MVP report -compares observed error using that same metric and bound. +- Planner selects a family, algorithm, parameter set, grouping strategy, + execution phase, or readout unsupported by an assigned executor; +- the collector and backend disagree on plan or materialization identity; +- the backend expects a different summary family, parameters, grouping, + window, or state encoding from the collector; +- an accuracy guarantee is missing or insufficient where one is required; +- a delta mode lacks compatible sequencing/checkpoint semantics; +- a plan is stale, expired, or not yet active; +- application evidence is missing; or +- a query result would use state from an earlier plan/run. -No backend-local `ResidualExpr`, `GuardedResult`, guarantee algebra, or -conditional exact-fallback policy is needed for this migration. ASAPQuery -consumes Planner replacements/guarantees and executes `KeepPreAsap` exactly; -it must not reinterpret the requirement or define a competing correctness -enum. +No failure may silently substitute another summary, loosen accuracy, erase a +grouping strategy, treat a missing series as zero, or return a plausible +result from incompatible state. -### 6.4 Keep scheduling outside ASAPPlanner +## 9. Migration phases -The planner remains deterministic and free of timers: +### Phase 0: pin and baseline -```text -ASAPPlanner: -search(workload roots, cost model) -> PlanSpace + GlobalSelection +- Pin all Planner crates to one immutable revision. +- Record that revision in build and planning artifacts. +- Capture legacy plans and query results for the golden workload. +- Add deadlines, size limits, and planning telemetry before enabling shadow + workloads. -ASAPQuery control plane: -diff(active_deployment, selected_workload) -> DeploymentPlanDiff -``` +Exit gate: the workspace builds and existing behavior is unchanged. -A scheduler, ruler, or materializer owns time: +### Phase 1: canonical workload ingestion -1. Determine the group evaluation timestamp. -2. Apply `query_offset_ms` to obtain logical evaluation time. -3. Prevent overlapping evaluations for the same Prometheus rule group. -4. Invoke QueryEngine with the selected plan and timestamp. -5. Execute selected summary nodes and any `KeepPreAsap` subtree. -6. Materialize the result or update alert state. +- Route single and multi-query inputs through one workload adapter. +- Add repeating-query input without moving scheduler metadata into Planner. +- Preserve caller IDs and per-query diagnostics outside canonical Planner + values. -#### Materialize into Prometheus first +Exit gate: one-shot and repeating entries lower deterministically, and shared +canonical sub-DAGs remain shared. -At each scheduled timestamp, evaluate the accelerated numeric expression and -remote-write a derived series such as: +### Phase 2: workload selection in shadow mode -```promql -asap:high_error_rate:ratio{service="checkout"} 0.073 -``` +- Run Planner candidate search and global selection for the full workload. +- Supply the ASAPQuery cost model, accuracy inputs, recurrence data, and + capability constraints supported at the pin. +- Export logical explain and typed rejection information. +- Compare with legacy decisions without changing production plans. -Generate or configure the Prometheus alert as: +Exit gate: every difference is explained by a documented semantic, accuracy, +cost, or sharing improvement; unexplained differences block rollout. -```yaml -- alert: HighErrorRate - expr: asap:high_error_rate:ratio > 0.05 - for: 5m - keep_firing_for: 1m -``` +### Phase 3: compile both physical plans -This preserves Prometheus rule reload, group ordering, labels, annotations, -pending/firing state, limits, and Alertmanager integration. The materialized -sample must be committed before the corresponding Prometheus evaluation. A -group `query_offset` can provide a data-availability margin; Prometheus -documents this as a use case for -[rule query offset](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#rule-query-offset). - -#### ASAP-aware ruler later - -An ASAP-aware ruler can call an instant-query endpoint directly and avoid -intermediate series. It must first implement or reuse Prometheus-compatible -group scheduling, missed-iteration handling, label/annotation templates, -state persistence, `for`, `keep_firing_for`, limits, reload behavior, and -Alertmanager delivery. This is intentionally not the first milestone. - -### 6.5 Incremental replanning and rule reload - -The historical input model already contains `existing_streaming_config` and -`existing_inference_config`, although they are reserved rather than acted on -([input.rs](https://github.com/ProjectASAP/ASAPQuery/blob/1fc18be81ec2e0f44fa0ded85151b513ee5312b7/asap-planner-rs/src/config/input.rs)). -ASAPQuery uses the active deployment plus the new upstream-selected workload -to produce a versioned physical diff: - -```rust -struct DeploymentPlanDiff { - reuse: Vec, - add: Vec, - resize: Vec, - retire: Vec, -} -``` +- Convert one selected workload DAG into matching collector and backend + subplans. +- Preserve shared materializations and readout consumers. +- Produce stable materialization fingerprints and the shared plan envelope. +- Validate both subplans before either is pushed. -Use two identities rather than conflating logical reuse with one physical -revision. The logical semantic ID should come from ASAPPlanner if/when it -provides a public canonical identity; the physical fingerprint remains an -ASAPQuery deployment identity: +Exit gate: golden tests prove that a deliberately mismatched collector/backend +pair is rejected and that matching plans round-trip through both wire formats. -```text -LogicalAggregationId = hash( - normalized leaf expression, - metric and filters, - grouping labels, - statistic, - window size and anchor -) - -MaterializationFingerprint = hash( - LogicalAggregationId, - summary family and parameters, - grouping layout, - physical format version -) -``` +### Phase 4: executor coverage and end-to-end shadowing -This lets a parameter change be represented as `resize` instead of appearing -as an unrelated logical aggregation, while each physical state still has a -stable content-addressed fingerprint. +- Install BackendPlan without query-time replanning. +- Push CollectorPlan and require semantic application reports. +- Exercise exact accumulators, supported summaries, reductions, grouping, + readouts, merge, raw/full/delta modes, and archive fallback. +- Compare aligned ASAP and exact results, freshness, and plan identities. -Safe rule reload is: +Exit gate: every selectable runtime shape is executed correctly or rejected +before selection; missing evidence fails the run. -1. Parse and validate the complete new rule set. -2. Generate `DeploymentPlanDiff` against the active version. -3. Reuse unchanged aggregations and start additions/resizes. -4. Warm new aggregations for their maximum lookback. -5. Atomically switch query/rule mappings at a group evaluation boundary. -6. Retire unreferenced physical states after their retention horizon. +### Phase 5: selected rollout -### 6.6 Recurring-workload delivery track +Use three rollout modes: -Build this after the base workload planner in Section 5 can produce and serve -a selected `BackendPlan`: +| Mode | Planner executed | Production plans emitted | +| --- | --- | --- | +| `legacy` | legacy only | legacy | +| `shadow` | legacy and Planner | legacy | +| `selected` | Planner | compiled Planner-derived plans | -1. Add the adapter-neutral `O11yMetricsQuery` model and its direct conversion - into ASAPPlanner `BatchEntry`/`RepeatingEntry` workloads. -2. Add an independent `adapters/prometheus` module with query and rule-file - adapters. Emit `O11yMetricsQuery` plus separately contained Prometheus - runtime metadata; make no ASAPPlanner Prometheus model change. -3. Feed adapted expressions through ASAPPlanner's existing canonicalization - and workload CSE. -4. Add deployment window anchoring, watermark, lateness, and retention - metadata. -5. Materialize exact or selected approximate results back into Prometheus. -6. Implement current-plan diffs, warm cutovers, and evaluation-boundary swaps. -7. Only then consider an ASAP-native ruler. +Roll out `selected` by workload/tenant. Keep the previous valid compiled plan +available for immediate rollback. A failed new plan never replaces it. -The conceptual change is: +Exit gate: the agreed observation period has no unexplained correctness, +freshness, capability, or plan-identity failures. -```text -Current: -query string + inferred repetition delay - -> static aggregation config - -Target: -ASAPPlanner RepeatingEntry + AccuracyTarget - -> shared temporal aggregation plan - + incremental deployment plan -``` +### Phase 6: delete legacy planning + +Remove legacy paths that: + +- union per-query capabilities into sketch choices; +- implement aggregate roots independently; +- choose summary families outside Planner; +- infer planned parameters from stored state at serving time; or +- generate collector and backend plans from separate logical decisions. + +Retain protocol adapters, physical placement, runtime cost inputs, +CollectorPlan/BackendPlan compilation, routing, execution, and archive +fallback. -The same model generalizes to recurring dashboards, scheduled SQL reports, -SLO evaluation, and periodic anomaly detection. +Exit gate: production has one logical planning path and one physical compile +path, with rollback based on previous compiled plans rather than legacy +planning. -## 7. Cross-repository golden workload +## 10. Validation -Use the ASAPPlanner DAG-viewer demo queries as shared regression fixtures: +### Cross-repository golden workload -- `q1`: grouped count; -- `q2`: grouped Avg rewritten to sum/count while sharing input with `q1`; -- `q3`: top-5 over a rate/frequency summary; -- `q4`: compatible top-10, allowing `q3` to derive from the larger result; -- `q6`: join query sharing a compatible input scan with other queries. +Maintain one versioned workload covering: -Add negative fixtures for: +- a repeated subexpression shared by multiple queries; +- compatible and incompatible filters; +- per-entity and grouped reductions, including empty global reduction; +- independent grouping and a capability-rejected shared grouping case; +- exact aggregation and each claimed MVP summary family; +- one-shot and repeating queries with different intervals; +- raw, full, and delta transmission; +- an unsupported query that takes exact fallback; +- an accuracy-invalid candidate; +- a stale or mismatched plan; and +- cold/archive fallback. -- different filter predicates; -- incompatible group-by reductions; -- different windows; -- different sort keys; -- incompatible schemas or table bindings; -- top-k inputs that differ below the limit. +For each pinned Planner revision, retain: -Tests are required at four boundaries: +- input workload and schemas; +- Planner revision and configuration; +- canonical and selected logical explain artifacts; +- rejected candidates and reasons; +- compiled collector and backend plans; +- capability snapshots; +- application reports; +- emitted materialization identities; +- query results and aligned exact results; and +- derived correctness, accuracy, freshness, latency, and cost measurements. -1. query workload -> selected strategies; -2. canonical roots + `GlobalSelection` -> `DeploymentPlan` -> matching - `CollectorPlan` and `BackendPlan`; -3. OpAMP/YAML and backend protobuf -> semantic activation, data-plane hot - reload, and `RoutingIndex`; -4. ingest -> both plan pushes -> identity-matched warm query response, - including archive fallback. +### Required test boundaries -## 8. Complexity and safety limits +1. External workload input to canonical Planner workload. +2. Canonical workload to selected post-ASAP DAG. +3. Selected DAG to matching CollectorPlan and BackendPlan. +4. Both wire formats to semantic activation. +5. Observations to stored summary state. +6. Query to summary readout or explicit exact fallback. +7. Replan to warm cutover, retirement, and rollback. -ASAPPlanner stores alternatives in memo groups rather than enumerating the -Cartesian product of whole plans. This avoids exponential copying of complete -workload DAGs, but it does not make every strategy linear: +### Pin-update tests -- common-subexpression discovery is approximately linear in reachable IR - nodes, subject to hashing/equality checks; -- ordinary candidate generation is proportional to discovered targets, - registered strategies, and candidates per target; -- rollup sibling discovery may compare aggregate pairs; -- top-k reuse may compare compatible limit pairs; -- candidate sorting adds per-group sorting cost; -- global selection is a topological dynamic-programming pass over the - discovered reference graph. +Every Planner pin update must: -Protect the control plane with: +- build all Planner-dependent crates at one revision; +- rerun the golden workload; +- update exhaustive mappings for new IR variants; +- prove unsupported new shapes fail before activation; +- compare selected materializations and guarantees against the previous pin; + and +- explain every intentional difference. -- maximum query, schema, IR-node, and candidate counts; -- a planning deadline and cancellation token; -- ASAPPlanner's search-iteration cap plus a deployment-level deadline; -- normalized-workload caching keyed by query, schema, accuracy, planner - revision, statistics epoch, and cost-model version; -- benchmarks for 1, 10, 50, and 100-query workloads; -- alerts for planning latency, candidate growth, timeout, and fallback rate. +## 11. Operational limits -## 9. Non-goals +The control plane enforces checked-in limits for: -- Moving deployment placement into ASAPPlanner. -- Making the DAG-viewer JSON the control-plane/data-plane protocol. -- Using viewer/export node IDs as persistent materialization identity. -- Removing archive fallback. -- Enabling SQL without a real schema catalog and binder. -- Reimplementing Prometheus scheduling or alert state in ASAPPlanner. -- Deleting the legacy planner before shadow validation and rollback are in - place. +- queries and schemas per workload; +- reachable IR nodes; +- candidate groups and candidates per group; +- planning iterations and wall time; +- explain artifact size; and +- materializations per compiled plan. -## 10. Completion criteria +Planning supports cancellation and reports phase timings, candidate counts, +rejections, selected cost, fallback, and timeout. A timeout or exceeded limit +returns a typed planning failure or configured exact fallback; it never emits +a partial compiled plan. -The migration is complete when: +## 12. Definition of done + +The migration is complete when all of the following are true in the same +supported release: - all production queries enter one workload-aware ASAPPlanner path; -- the selected post-ASAP plan plus one downstream physical compile is the sole - source of matching `CollectorPlan` and `BackendPlan` decisions; -- shared sub-DAGs remain shared through materialization and serving; -- the data plane does not independently select summary families or params; -- ASAPPlanner's `AccuracyTarget` and, when available at the pin, - `ResultGuarantee` are the only accuracy contract; legacy - `accuracy_sla`, local exact-vs-approximate enums, and backend guarantee - algebra have been removed; -- every open ASAPPlanner PR in §4.4 is either absent from the immutable pin or - has its documented adapter/compiler/capability tests in the same pin-update - change; -- q1/q2/q3/q4/q6 pass cross-repository end-to-end tests; -- explain output maps every selected post-ASAP replacement explicitly to its - pre-ASAP target; -- the legacy capability-union and per-root implementation planners have been - removed; -- deployment placement, collector configuration, routing, execution, and - archive fallback remain owned by ASAPQuery-backend. - -Recurring-rule support is complete only when: - -- rule schedules override frequency inference; -- only the generic evaluation interval enters ASAPPlanner; query offset and - alert-state durations remain scheduler metadata; -- temporal panes have an explicit common anchor and watermark contract; -- `AccuracyTarget::Exact` plans either select exact summaries, execute - `KeepPreAsap` from raw/archive data, or use only the explicit, recorded - TopK exception enabled through ASAPPlanner #293's cost-model hook; -- rule reload uses a warm, versioned `DeploymentPlanDiff` cutover; -- the initial production path materializes into Prometheus while Prometheus - retains alert-state authority. +- every Planner crate is pinned to the same recorded revision; +- the selected post-ASAP DAG is the only logical source for runtime plans; +- one compiler emits matching CollectorPlan and BackendPlan artifacts; +- shared workload sub-DAGs remain shared through materialization and serving; +- the data plane never selects or sizes summaries independently; +- accuracy targets and guarantees are preserved without backend-local + reinterpretation; +- every selected runtime shape is supported by both executors or rejected + before activation; +- semantic collector activation and backend activation agree on plan and + materialization identities; +- exact fallback remains available for unsupported queries; +- cross-repository golden and end-to-end tests pass; +- shadow rollout and rollback have been exercised; and +- legacy logical planning and serving-time reconstruction have been removed. diff --git a/control_plane/docs/design-compiled-plan-collector-backend-split.md b/control_plane/docs/design-compiled-plan-collector-backend-split.md index dd4f757c..29e70431 100644 --- a/control_plane/docs/design-compiled-plan-collector-backend-split.md +++ b/control_plane/docs/design-compiled-plan-collector-backend-split.md @@ -120,10 +120,8 @@ non-goals: into *one* backend-side materialization. Neither direction is a serialization concern; both require an explicit compile/allocate pass. 4. **Two independent readings of the same DAG can silently disagree.** - [`design-asapplanner-workload-planner-migration.md`](design-asapplanner-workload-planner-migration.md) - §2 already names this failure mode for the legacy planner ("the control - plane, data plane, and ASAPPlanner silently select different physical - summary families or parameters for the same query"). If the collector + The [migration plan](design-asapplanner-workload-planner-migration.md) + requires one physical compile path for exactly this reason. If the collector subplan and the backend subplan are derived independently — even from the same selected DAG, by two different code paths, at two different times — nothing stops them drifting. A single compile step that emits @@ -135,10 +133,7 @@ non-goals: `BackendPlan` protobuf is read by `data_plane`. Neither should decode the other's format, and neither should decode ASAPPlanner's internal Rust IR — that IR is not a stable cross-process wire contract and was - never meant to be one (`design-asapplanner-workload-planner-migration.md` - §5/PR4: *"Do not copy `dag_export`'s JSON into the runtime contract and - do not infer mappings from labels, hashes, strategy rationale, or viewer - node signatures."*). + never meant to be one (migration plan §§2 and 4.2). ## 3. `CompiledPlan`: one compile step, two subplans, one identity @@ -160,7 +155,7 @@ pub struct CompiledPlan { /// unchanged selection (e.g. a resize), not on every replan. pub plan_version: u64, /// Not-before: neither subplan should be treated as authoritative - /// before this time. Lets a warm cutover (see migration doc §6.5, + /// before this time. Lets a warm cutover (see migration plan §9, /// `DeploymentPlanDiff`) land both subplans ahead of the switch. pub activation: DateTime, /// Not-after / supersede horizon. `None` for "until superseded." @@ -188,8 +183,8 @@ pub struct CollectorSubplan { pub struct EdgeAssignment { pub edge_id: String, - /// Today's `asap_edge` processor fields (§5) — produced by compiling - /// the `SummaryAgg` nodes assigned to this edge, not authored ad hoc. + /// The versioned CollectorPlan fields (§5), produced by compiling the + /// `SummaryAgg` nodes assigned to this edge, not authored ad hoc. pub config: AsapEdgeConfig, /// Opaque identity of *this edge's* exact YAML body — unchanged /// semantics from ASAPCollector's existing `config_hash` (it still @@ -215,9 +210,8 @@ pub struct BackendSubplan { ## 4. Compile algorithm Input: the materialized selection (`GlobalSelection::materialize()`'s -`Rc` roots, per -[migration doc](design-asapplanner-workload-planner-migration.md) §5/PR4 — -shared node identity intact) plus this deployment's topology and +`Rc` roots, with the shared identity required by migration plan +§4.2 intact) plus this deployment's topology and constraints (collector fleet membership, per-edge shard/memory budgets, transport cost model — the same inputs `physical::colored_dag` already takes today, see §8). @@ -404,15 +398,14 @@ cannot answer. per-node `PipelineStage` tag on one combined tree. `CompiledPlan` replaces that shape with two explicit typed subplans compiled from ASAPPlanner's own selected `SummaryNode` DAG. This module belongs on the - [migration doc](design-asapplanner-workload-planner-migration.md) §4.1 - removal list (it is not currently listed there) — add it once PR4/PR5 of - that stack lands, not before, since it is still the live path until then. + migration plan §9 Phase 6 removal list — remove it only after the compiled + plan path is selected and rollback no longer depends on legacy planning. - `emit::agent::generate_agent_collector_config` currently builds a complete collector YAML. It should become the `EdgeAssignment -> CollectorPlan` serializer defined in §5. Bootstrap OTel receivers/exporters and credentials remain deployment configuration; a workload replan must not replace them. -- Both subplans should land behind the same `ASAP_WORKLOAD_PLANNER_V2` - shadow-rollout flag the migration doc already proposes (§5/PR8): in +- Both subplans should land behind the same workload-planner rollout mode + the migration plan defines (§9 Phase 5): in `shadow` mode, compile `CompiledPlan` and record `plan_id` agreement and field-level diffs against the legacy allocator's output without pushing either subplan, exactly mirroring that section's existing comparison From 49ca873e6a874b07cb7ed7089ae78b6f8fc1ecad Mon Sep 17 00:00:00 2001 From: zz_y Date: Thu, 27 Aug 2026 17:51:20 -0600 Subject: [PATCH 11/12] docs: rewrite compiled and backend plan designs --- .../docs/design-backend-plan-wire-format.md | 597 +++++++------- ...n-compiled-plan-collector-backend-split.md | 726 ++++++++---------- 2 files changed, 613 insertions(+), 710 deletions(-) diff --git a/control_plane/docs/design-backend-plan-wire-format.md b/control_plane/docs/design-backend-plan-wire-format.md index c02ec147..b73fcb1b 100644 --- a/control_plane/docs/design-backend-plan-wire-format.md +++ b/control_plane/docs/design-backend-plan-wire-format.md @@ -1,295 +1,310 @@ -# `BackendPlan`: the control-plane → data-plane wire contract - -> Scope: the wire format and query-time structure connecting `control_plane` -> (planning) to `data_plane` (serving) in this deployment. This document -> designs the target shape directly — it does not narrate what currently -> exists or what a migration path looks like; see PR history for that. - -## 0. One-sentence version - -`BackendPlan` is the typed message `control_plane` pushes to `data_plane` -describing every materialization it has decided on and what query -capabilities each one answers; `RoutingIndex` is the structure `data_plane` -builds from it and consults at query time — both planning time and serving -time end up reading the *same* decision, instead of serving time -re-deriving one independently. - -## 1. Why this exists: planning decides once, serving must reuse it exactly - -Planning (`control_plane`) and serving (`data_plane`) are different -processes with different jobs — planning picks a summary family and sizes -its parameters from a query shape and an accuracy target, symbolically; -serving walks already-materialized state and answers queries against it. -Those two steps must agree on **exactly** which `SummaryFamilyType` a given -metric's materialization uses, including algorithm, parameters, and grouping -layout, because `SummaryExecutor::find_candidates` matches on that state type *by strict -equality*, on purpose: this deployment chose exact agreement over silently -serving an answer under a looser accuracy guarantee than what was -actually planned (see `summary_executor.rs::summary_params_match`'s -rationale). - -The wire format is the mechanism that keeps those two sides honest. If -serving time ever has to *guess* what was planned — e.g. re-deriving a -family/params choice from a hardcoded default accuracy, independent of -what a specific metric's workload actually requested — that guess will -occasionally be wrong, and the two sides drift apart silently (a query -that should be servable warm capability-misses to archive for no -query-shape reason, only because the guess didn't match reality). The -correct shape is: `control_plane` decides once, writes the decision into -`BackendPlan`, and `data_plane` (via `RoutingIndex`) reads that same -decision back — never re-derives it. - -This has one direct implication for `data_plane`'s serving-time L4 -lowering: it should resolve a query's selected `SummaryFamilyType` by -looking it up in `RoutingIndex` (built from the `BackendPlan` `control_plane` -already pushed), not by invoking a `CostModel` a second time at query time. -`CostModel::rank_candidates`/`size_params` are a **planning-time-only** -concern — they run once, when `BackendPlan` is built, never again per -query. - -## 2. Goals - -- One wire message, typed end to end, that both sketch-based and exact - materializations populate identically — no "exact means unplanned" - loophole. -- A query-time structure in `data_plane` that resolves "which - materialization answers this query" by direct lookup against what was - actually planned, not by independently re-classifying the query and - hoping the classification matches reality. -- Reuse this deployment's existing canonical vocabulary - (ASAPPlanner's `QueryExpr`/`AggIntent`/`SummaryFamilyType`, and - `control_plane`'s own `Capability`/`PolicyFingerprint`) - directly. No parallel, wire-specific re-encoding of concepts that already - have a canonical type. - -## 3. `Materialization`: exact and approximate are already the same shape - -ASAPPlanner's current `SummaryFamilyType` is the canonical union of plain, -exact-aggregate, sketch, sample, wavelet, and statistical-model state. A -sketch-valued family carries its concrete `SketchKind` and -`GroupingStrategy`; `SketchKind` carries category, algorithm, and parameters. -`BackendPlan` preserves that selected type instead of flattening it into the -obsolete local `(SummaryKind, SummaryParams)` pair. - -```rust -pub struct BackendPlan { - // Cross-subplan identity as of design-compiled-plan-collector-backend- - // split.md: shared verbatim with this plan's CollectorSubplan, so the - // two can be checked for agreement. `PolicyFingerprint` below remains - // the identity of one materialization; `plan_id` answers "were the - // collector and backend subplans compiled together," which no - // per-materialization fingerprint can answer on its own. - pub plan_id: PlanId, - pub plan_version: u64, - pub activation: DateTime, - pub expiry: Option>, - pub backend_compat: BackendCompatId, - pub generated_at: DateTime, - - /// Every materialization this backend should build/maintain, - /// keyed by content-addressed identity. - pub materializations: HashMap, - - /// Query-time capability routing — see §4 for why this is a - /// separate table rather than 1:1 with `materializations`. - pub routing: Vec, - - pub monitors: Vec, // CDM, unchanged concept -} - -pub struct Materialization { - pub fingerprint: PolicyFingerprint, - - // Reuses asap_ir/asap_sketch types directly — no re-flattening into - // ad-hoc string/HashMap fields. - pub source: Source, // metric + label filter - pub window: WindowSpec, // kind / size / slide - pub group_by: Vec, - pub rollup: Vec, - - /// The exact selected Planner state type, including concrete sketch - /// algorithm/parameters and grouping layout where applicable. - pub family: SummaryFamilyType, - pub col: ColumnRef, - - /// Collector assignments that produce this state under the matching - /// CollectorPlan. - pub sources: Vec, - - /// Present when the pinned Planner revision computes a guarantee for - /// this selected result. Unknown is not represented as exact. - pub guarantee: Option, - - pub retention: Option, -} - -pub struct RoutingEntry { - pub satisfies: Capability, // control_plane's own coarse, - // family-level routing vocabulary - pub materialization: PolicyFingerprint, - pub storage_backend: StorageBackend, -} +# BackendPlan: control-plane to data-plane contract + +> Status: proposed +> +> Scope: the typed runtime contract by which ASAPQuery-backend's control +> plane tells its data plane what summary materializations exist, how they +> are ingested, and which query capabilities they serve. + +## TL;DR + +Planning chooses once; serving reuses that exact decision. + +`BackendPlan` is the backend half of a `CompiledPlan`. It describes the state +that matching CollectorPlans produce and the readout/routing decisions the +data plane must apply. It prevents the data plane from independently guessing +summary family, parameters, grouping, windows, accuracy, or storage. + +```text +selected Planner DAG + | + v +physical compile + / \ +CollectorPlan BackendPlan + | | +summary state ingest + route + readout ``` -**Why `routing` is a separate table from `materializations`, not a 1:1 -map:** one materialization can satisfy several query capabilities — a -single KLL(k=200) sketch built for p99 can also answer p50/p95, and -`Capability::is_satisfied_by` already does family-wildcard matching -(`QuantileApprox(Any)` vs `QuantileApprox(DDSketch)`). Binding "what's -built" to "what it can answer" 1:1 can't express that reuse; a separate -routing table can — adding a new answerable shape for an existing -materialization is one new `RoutingEntry`, not a change to the -materialization itself. - -**Why `Capability` still exists as its own type, distinct from -`SummaryFamilyType`:** `Capability` is the *family-level* -question ("does anything at all answer a `QuantileApprox` shape for this -metric") used for coarse routing decisions — miss detection, archive -fallback, "should I even try the sketch tier." `SummaryFamilyType` is the -*exact* question `SummaryExecutor::find_candidates` -needs — because two candidates must agree exactly to be legally mergeable -via `merge_states`, family-level compatibility alone isn't enough to -decide that. These are genuinely different match precisions for genuinely -different callers (see §4) — collapsing them into one would either make -routing too strict (rejecting a real family match on an incidental params -difference) or make serving too loose (merging things that don't actually -agree). - -## 4. `RoutingIndex`: query-time structure in `data_plane` - -Built fresh from each `BackendPlan` push, atomically swapped -(`arc-swap`) so a config update never serves from a half-updated index. - -**Tier 1 — exact fingerprint, O(1).** Compute the incoming query's own -shape fingerprint using the same hash function as `PolicyFingerprint` -(metric + agg shape + params + grouping/rollup labels + window + -normalized filter) and look it up directly in `materializations`. Covers -the common case — a query structurally identical to something -`control_plane` already planned (the majority of fixed-dashboard traffic). - -**Tier 2 — structural/capability match, on Tier-1 miss.** - -```rust -pub struct RoutingIndex { - exact: HashMap, // Tier 1 - by_metric: HashMap, // Tier 2 -} - -/// Columnar, not `Vec` nested in a `HashMap` — see §4.1. -struct MetricBucket { - group_shape_ids: Vec, - filter_ids: Vec, - windows: Vec, - capabilities: Vec, - families: Vec, - guarantees: Vec>, - fingerprints: Vec, - storage_backends: Vec, -} +## 1. Why BackendPlan exists + +Control-plane planning and data-plane serving happen in different processes +and at different times. They must nevertheless agree on: + +- which materializations should exist; +- their exact summary state types; +- where their payloads come from; +- which windows and groups they represent; +- which queries/readouts they can answer; +- which guarantees apply; and +- which plan version is active. + +Without a typed plan, serving tends to reconstruct decisions from query text, +hard-coded defaults, or observed storage metadata. That can silently select a +different algorithm or parameters, miss valid state, merge incompatible +state, or serve under the wrong accuracy contract. + +BackendPlan makes the control-plane decision authoritative. + +## 2. Relationship to other plans + +ASAPPlanner supplies the selected logical post-ASAP DAG. ASAPQuery's physical +compiler creates: + +- CollectorPlan, which constructs and transmits materializations; and +- BackendPlan, which validates, stores, routes, merges, and reads them. + +The two are defined together in +[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). + +BackendPlan is not: + +- a copy of ASAPPlanner's internal DAG; +- a query-language AST; +- a collector configuration; +- a storage inventory discovered after planning; or +- an explain/viewer artifact. + +## 3. Plan envelope + +Every BackendPlan contains: + +| Field | Meaning | +| --- | --- | +| `plan_id` | Identity shared with every CollectorPlan compiled from the same decision. | +| `plan_version` | Ordered version within that plan identity. | +| `activation` | Earliest time this plan may serve queries. | +| `expiry` | Optional time after which this plan is invalid. | +| `backend_compat` | Backend-plan and summary-state schema compatibility identity. | +| `generated_at` | Time the control plane emitted the plan. | +| `planner_revision` | Immutable Planner revision that produced the logical selection. | + +The data plane rejects stale, conflicting, premature, expired, or +incompatible plans. Reapplying the same plan/version/content is idempotent. + +## 4. Materializations + +A materialization describes one logical summary state expected by the +backend. It contains: + +- content-addressed materialization identity; +- bound metric/source and canonical filters; +- summarized value or item label; +- physical/logical window contract; +- reduction and grouping layout; +- exact `SummaryFamilyType`, including concrete algorithm and parameters; +- collector producer references; +- state encoding/schema compatibility; +- storage destination and retention; and +- selected result guarantee when supplied by Planner. + +Exact accumulators and approximate summaries use the same materialization +concept. The family discriminator determines which state and parameters are +valid. An algorithm/parameter mismatch is a decoding or validation failure, +not an untyped configuration bag. + +The materialization identity includes every semantic property required for +safe reuse and merge. Two states with the same metric name but different +filters, reductions, grouping, windows, families, parameters, or encoding are +different materializations. + +## 5. Routing and readout + +Materializations describe what state exists. Routing entries describe what +queries that state can answer. These are separate because one materialization +may serve several readouts or queries. + +For example, one sufficiently accurate quantile materialization may support +p50, p95, and p99 readouts. It remains one maintained state with several +routing entries. + +A routing entry identifies: + +- the query capability/readout it satisfies; +- the materialization used; +- any remaining backend-side operation; +- required grouping/window compatibility; +- the storage tier; and +- exact fallback behavior on a miss. + +Routing never changes the materialization's summary semantics. A capability +match can choose among already valid planned routes; it cannot reinterpret +stored state as another family or parameterization. + +## 6. Two levels of matching + +Serving needs two different matching strengths. + +### Capability matching + +Capability matching answers: + +> Is there a planned materialization that can answer this logical query shape? + +It may allow a compatible family-level or readout-level relationship, such as +using one quantile summary for several ranks or rolling up a mergeable finer +grouping when the plan explicitly permits it. + +### State compatibility matching + +State compatibility answers: + +> Can these exact payloads be decoded, merged, and read under this +> materialization contract? + +This comparison is strict. Family, algorithm, parameters, grouping layout, +window, encoding, plan identity, and materialization identity must agree. + +Capability compatibility never implies state compatibility. The backend may +route a query to a compatible planned materialization, but it may merge only +strictly compatible states. + +## 7. Accuracy guarantees + +`AccuracyTarget` is the requested constraint used during planning. When the +pinned Planner revision provides `ResultGuarantee`, BackendPlan preserves the +selected result's: + +- error metric; +- bound expression; +- failure probability; +- provenance and budget allocation; and +- any unavailable statistic that kept the guarantee unknown. + +The data plane does not recompute or weaken that guarantee. Unknown is not +zero and not exact. A route that requires a guarantee fails when the stored +plan lacks a sufficient one. + +For an explicitly permitted approximate realization of an Exact-requested +TopK query, BackendPlan records the effective approximation target and +guarantee. It must never label that realization exact. + +## 8. Source and payload validation + +Each ingested summary payload identifies: + +- plan ID and version; +- backend compatibility identity; +- materialization ID; +- collector/producer ID; +- window identity; +- state schema version; and +- full/delta sequencing and checkpoint identity when applicable. + +The backend rejects payloads that are unknown, stale, incompatible, out of +sequence, or addressed to another active plan. It does not place them into a +best-effort metric-name bucket. + +A plan may reference several collector producers for one sharded +materialization. The backend merges them only if the materialization contract +and family algebra allow it. + +## 9. Installation and lifecycle + +BackendPlan installation is staged and atomic: + +1. Decode and validate the complete plan. +2. Validate every family, readout, route, source, and storage capability. +3. Build a new routing/index view without mutating the active view. +4. Confirm compatibility with the matching collector subplan. +5. Mark the plan staged until its activation time and collector application + evidence are available. +6. Atomically switch query routing to the new view. +7. Retain the previous view until in-flight readers and the configured drain + horizon finish. + +An invalid update never partially changes routing. The previous unexpired +plan remains available for rollback. + +## 10. Query-time behavior + +At query time, the data plane: + +1. parses/canonicalizes the query only as needed to identify its planned + logical shape; +2. looks up routes installed from BackendPlan; +3. verifies readiness, freshness, grouping, window, and guarantee; +4. fetches strictly compatible states; +5. merges and reads them according to the planned operation; and +6. returns the result or takes the explicit exact fallback. + +It does not invoke Planner candidate search, run a planning cost model, resize +a sketch, or infer a missing parameter. + +## 11. Example + +Suppose the workload contains: + +```promql +quantile_over_time(0.95, request_duration_seconds[5m]) ``` -Match algorithm on Tier-1 miss: - -1. `by_metric.get(metric_id)`. -2. Filter `group_shape_ids` for rows equal to, or a rollup-derivable parent - of, the query's requested group-by. -3. Filter remaining rows by window compatibility — exact match, or the - existing window-merge/closest-pane logic - (`storage_engines/sketch_db/query/window_merger.rs`). -4. Filter remaining rows for filter *subsumption* (materialization's - baked-in filter ⊆ query's filter — not equality). -5. **Match precision depends on the caller** — this is the one place - `RoutingIndex` has two genuinely different read modes, not a single - shared one: - - **Whole-query resolution** (deciding whether this query can be - served warm at all, and which storage backend to route to): match - `Capability::is_satisfied_by(query_agg_intent, accuracy)` — - family-level. - - **`SummaryExecutor::find_candidates`** (per-`L4Node`-leaf, called - during `asap_sketch::exec::execute()`'s walk): match exact - exact `SummaryFamilyType` equality — required for anything that can feed - a `SummaryMerge`, and the reason `l4_lowering.rs` no longer needs to - independently observe or guess this (see §5). -6. **Whole-query resolution only:** if more than one candidate survives, - rank by error bound / storage-tier cost and return the single winner. - **`find_candidates` only:** skip ranking — return every row that - survived step 5's exact match, so `execute()` can fold them all via - `merge_states`. -7. No survivors → archive/Thanos fallback routing (whole-query mode), or - `ExecError::NoCandidates` (`find_candidates` mode) — two callers of the - same "empty" outcome. - -### 4.1 Why columnar + interned ids, not nested `HashMap, Vec<_>>` - -Consistency with an existing, already production-validated pattern one -layer down: `storage_engines/sketch_db/index/epoch_columnar.rs` already -does exactly this for the physical sketch index — `LabelValuesId = u32` -interns each group-by label-values vector so the hot loop compares a -4-byte int instead of walking a `BTreeMap`, and the epoch -store itself is parallel arrays ("range scan touches only `windows_col`"). -`RoutingIndex` should use `GroupShapeId`/`FilterId` interning and -parallel-array storage per `MetricBucket` for the same reason: at the -scale of thousands of planned materializations per metric, Tier-2 lookup -is effectively a small in-memory OLAP scan, not a handful of hash lookups, -and should be built that way rather than reinvented as nested hash maps. - -## 5. What this replaces at serving time - -Today, `data_plane`'s serving-time L4 lowering (`l4_lowering.rs`) parses -the raw query string down to a canonical `QueryExpr`, then has to -*independently reconstruct* which `SummaryFamilyType` a -metric's registered sid actually uses by inspecting the `SketchStore`'s -own metadata (`ObservedFamilyCostModel`) before it can bind an `L4Node` -that `find_candidates` will actually match. That's a real, working -mechanism, but it's inherently a *reconstruction* — it infers the plan -from its side effect (what got registered), rather than reading the plan -directly. - -Once `RoutingIndex` exists, serving-time lowering simplifies to: parse to -`QueryExpr` (L1-L3, still genuinely needed — a query's *shape* has to be -recovered from its text regardless of any wire format), then resolve the -query's selected `SummaryFamilyType` via `RoutingIndex`'s -`find_candidates`-mode lookup directly, and construct the `L4Node` from -that pair — no `CostModel::rank_candidates`/`size_params` call at serving -time at all. `CostModel` becomes exactly what its name says: a -planning-time cost model, invoked once when `BackendPlan` is built, never -re-invoked per query. `ObservedFamilyCostModel`'s SketchStore-introspection -approach was always a stopgap for the absence of this lookup, not a -replacement for it. - -## 6. Transport - -**Proto, not YAML/JSON.** `BackendPlan` is a typed protobuf contract. -`SummaryFamilyType` and its family-specific values become proper protobuf -`oneof`s, with invalid family/parameter combinations rejected during decode. -`backend_compat` explicitly protects coordinated use with the independently -delivered CollectorPlan and emitted summary-state schema; deployment timing -must not be treated as an implicit compatibility guarantee. Additive fields -and backward decoding support controlled rollout, with unknown required -variants rejected rather than placed in an untyped -`HashMap` bag. - -## 7. Open questions - -- **Query-side fingerprinting.** Does Tier-1 matching reuse - `PolicyFingerprint::from_config` verbatim, or does an incoming query - need a distinct fingerprint function (it carries no `original_yaml` or - `num_aggregates_to_retain` — both already excluded from the hash per - `policy_fingerprint.rs`'s own doc comment, so likely fine as-is, but - worth verifying rather than assuming). -- **Rollup algebra.** Step 2 above assumes "a materialization grouped by - `(zone, region)` can answer a query grouped by `(zone)` alone" is a - known-safe operation gated by the `rollup` field. The precise algebra — - which summary families roll up safely (`Sum`/`Count`-family: yes; - `Quantile`: generally no without re-estimation error) — needs its own - short design pass before this step can be implemented as described. -- **Exact-requested top-k policy.** ASAPPlanner PR #293 permits a deployment - cost model to offer `CmsWithHeap`/`CountSketchWithHeap` for - `TopK { accuracy: Exact }` only with an explicit effective approximation - target, while retaining pass-through. If the pinned revision contains that - hook and ASAPQuery opts in, `BackendPlan` records the approximate family, - effective target, and selected guarantee; it must not describe the result as - exact. Without the opt-in, the query routes to exact raw/archive execution. -- **`RoutingIndex` performance.** New query-time hot path in a - latency-sensitive service — needs a benchmark pass against the current - lookup, not just a correctness pass, before it can replace anything. +The selected and compiled plan may contain one DDSketch materialization with +one-minute panes and a p95 routing/readout entry. BackendPlan records the +DDSketch parameter, per-entity reduction, independent grouping layout, +producer collectors, storage route, window composition, and selected +guarantee. + +When the query arrives, the data plane finds that route, fetches five +compatible panes, merges DDSketch state, and reads p95. It does not run a cost +model to reconsider KLL or choose a new DDSketch parameter. + +## 12. Wire-format principles + +BackendPlan is a typed protobuf contract. + +The schema follows these principles: + +- family-specific values use typed discriminated variants; +- invalid family/parameter combinations are unrepresentable or rejected; +- additive optional fields support controlled rollout; +- unknown required variants fail closed; +- compatibility is explicit through `backend_compat`; +- materialization identity is stable and content-addressed; and +- debug/explain fields are not runtime identity. + +Exact protobuf field numbers and generated-language types belong in the +protocol definition and implementation review, not this design document. + +## 13. Fail-closed behavior + +The plan or query fails when: + +- the plan envelope is stale or incompatible; +- a materialization family/parameter/grouping/window is unsupported; +- a collector source does not match the expected contract; +- state payload identities or sequences are invalid; +- a requested route has no valid materialization; +- a required guarantee is absent or insufficient; +- required state is empty, stale, or incomplete; or +- exact fallback is required but unavailable. + +The backend must not return a plausible summary result from mismatched state, +silently drop missing series, treat missing data as zero, or hide a failure as +a routing miss. + +## 14. Non-goals + +This document does not define: + +- Planner candidate generation or ranking; +- CollectorPlan; +- summary-state byte encoding; +- storage-engine implementation; +- query parser implementation; +- routing-index data structures or performance optimizations; or +- protobuf field numbering. + +## 15. Definition of done + +The BackendPlan design is satisfied when: + +- control plane and data plane share one typed contract; +- every expected collector materialization has an exact backend declaration; +- one materialization can serve several explicit routes/readouts; +- state merge uses strict compatibility; +- guarantees remain Planner-derived and fail closed; +- plan installation and routing switch atomically; +- query serving performs no independent summary planning; +- stale or mismatched payloads are rejected; and +- end-to-end tests prove matching plans serve and mismatched plans fail. diff --git a/control_plane/docs/design-compiled-plan-collector-backend-split.md b/control_plane/docs/design-compiled-plan-collector-backend-split.md index 29e70431..eb67a9c5 100644 --- a/control_plane/docs/design-compiled-plan-collector-backend-split.md +++ b/control_plane/docs/design-compiled-plan-collector-backend-split.md @@ -1,426 +1,314 @@ -# Compiling a selected post-ASAP DAG into two physical subplans +# Compiling one Planner decision into collector and backend plans -> Status: proposed, 2026-08-27 +> Status: proposed > -> Scope: the interface between ASAPQuery-backend's control plane and its two -> executors — ASAPCollector (via OpAMP) and the ASAPQuery-backend data plane -> (via `BackendPlan`). This document replaces the implicit assumption, in -> earlier design notes, that a selected post-ASAP node can be serialized -> more or less directly into collector YAML. It complements -> [`design-asapplanner-workload-planner-migration.md`](design-asapplanner-workload-planner-migration.md) -> (the planner-migration boundary) and -> [`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) -> (the backend half of the wire contract this document extends). - -## 0. The boundary this document sits on - -ASAPPlanner's own scope statement (`README.md`, "Scope") is explicit: - -> not caring about CTSA stages i.e. whether a part of a plan is executed at -> the collector or at the analytics stage -> not caring about assignment of physical resources, like CPU threads and -> memory, to nodes in the ASAP plan - -and `docs/design_docs/asap-aware-mapping/README.md`'s non-goals repeat this -for the mapping layer specifically: no CPU/memory assignment, no machine -placement, no scheduling, no admission control, no low-level execution -tuning. ASAPPlanner's own README names the resulting open question directly -("Open questions", #1): its output "has semantics of batch query execution -over data at rest" and "needs to be converted into two plans: (1) streaming -dataflow graph that computes summaries on raw data, and (2) batch query -execution plan that uses summaries to answer queries" — collector and -backend, in this deployment's vocabulary. - -So: ASAPPlanner selects **what** replaces a query sub-DAG (which summary -family, algorithm, parameters, grouping layout, and how per-query readouts -compose over shared summary state). ASAPQuery-backend's control plane -decides **where** each piece of that selected DAG runs, **how** it is -represented on each wire, and **that** both sides agree they are running the -same decision. This document is the second half — the "where/how/that" — -concretely. - -## 1. What ASAPPlanner hands us today - -Grounded in `crates/types/src/post_asap/{mod,expr,schema,sketch}.rs` on -ASAPPlanner `main`, not carried over from older docs. The selected output of -`asap_aware_mapping::replacement::search_workload_with(...).global_selection(...)` -is a DAG of `Rc` (shared `Rc` = shared physical state — see -[migration doc](design-asapplanner-workload-planner-migration.md) §3), each -node one of: - -- **`SummaryExpr::SummaryAgg { child, family, col, reduction, grouping }`** - — the *update* side: consumes raw/plain input and produces summary state. - `family: SummaryFamilyType` is one of `ExactAggregate(ExactKind, - ExactParams)`, `Sketch(SketchKind, GroupingStrategy)` (`SketchKind` itself - nests `category`/`algorithm`/`params` — `Kll`/`DDSketch`/`Hll`/`Cms`/ - `CmsWithHeap`/`Kmv`/`Theta`/`CountSketch`/`CountSketchWithHeap`, each with - its own concrete `SketchParams`), `Sample(SamplingKind, SamplingParams)`, - `Wavelet(WaveletKind, WaveletParams)`, or `StatModel(StatModelKind, - StatModelParams)`. `reduction: Reduction` is `Reduce(GroupKeys)` or - `PerEntity` (`crates/types/src/pre_asap/query_expr.rs`). `grouping: - GroupingStrategy` is `PerSubpopulationInstance` (default) or - `SharedMultiSubpopulation { kind: HydraKind, params: HydraParams }`. -- **`SummaryExpr::SummaryEstimate { summary_input, query }`** — the - *readout* side: reads a `SketchQuery` (`Quantile`/`PointCount`/ - `Cardinality`/`TopK`) out of already-built summary state, producing a - plain value. Summary-state typing does not propagate past this node. -- **`SummaryExpr::SummaryMerge { children }`** / **`SummarySubtract`** / - **`SummaryDelete`** / **`SummaryJoin`** — combine or transform summary - state; still summary-typed in/out, still on the readout side of any - `SummaryEstimate` that eventually consumes them. -- **`SummaryExpr::KeepPreAsap(Rc)`** — no replacement chosen; - executed against raw/archive data, never against collector-maintained - state. - -**Forward note on an open upstream PR.** ASAPPlanner PR -[#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) (open, not yet -merged) proposes to make exactly this update/readout distinction an -explicit, validated field: `post_asap::phase::ExecutionAvailability { -UpdateValue, SummaryState, ReadoutValue }`, with `SummaryAgg.child` typed to -accept only `UpdateValue` (or nested exact-accumulator state), and -`SummaryEstimate` typed `SummaryState -> ReadoutValue`. The split this -document defines (§2) is derived from the same structural fact — -`SummaryAgg` is the only node that *consumes* plain/update values and -*produces* summary state — so it does not depend on #300 landing, but it is -literally the same boundary #300 gives a name to. If/when #300 merges, §2's -partition rule should be re-expressed as "everything upstream of and -including a `PhaseAssignment` boundary at `SummaryState`" rather than -re-derived structurally; no other part of this design changes. - -## 2. Why not compile 1:1, node-by-node, straight to collector YAML - -A naive compiler would walk the selected DAG and, for each `SummaryAgg`, -emit one `asap_edge.metrics[]` entry with the same `family`/`col`/ -`reduction` fields, then hand the whole thing to whichever process asks for -it. This does not work, for reasons that are all direct consequences of §0's -non-goals: - -1. **No stage/edge/shard is chosen.** ASAPPlanner has no concept of "which - collector process" or "how many shards" — `SummaryAgg` names a logical - aggregation, not a physical instance of one. Something has to decide - fan-out: one `SummaryAgg` shared by two queries might still be one - physical summary; one `SummaryAgg` under high cardinality might be - sharded across `shard_count` collector processes and merged with - `SummaryMerge` before it ever reaches a `SummaryEstimate`. That decision - is deployment placement, owned here, not upstream. -2. **No transport/physical parameters exist upstream.** `edge_id`, - `window_duration`, `warm_allowed_lateness`, `drop_original`, - `delta_transmission`, `delta_threshold` (see - [ASAPCollector's OpAMP interface doc](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md)) - are bandwidth/latency/resource trade-offs a deployment makes; ASAPPlanner - has no field for any of them and should not grow one (they are the - physical-resource assignment its own non-goals name explicitly). -3. **Sharing and sharding both break 1-selected-node = 1-wire-fragment.** A - shared `Rc` reached by two query roots must still be *one* - collector-side summary and *one* backend-side `Materialization` with two - `RoutingEntry` rows (see - [`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) - §3 on why routing is a separate table). A single logical node sharded for - cardinality must become *several* collector-side instances merged back - into *one* backend-side materialization. Neither direction is a - serialization concern; both require an explicit compile/allocate pass. -4. **Two independent readings of the same DAG can silently disagree.** - The [migration plan](design-asapplanner-workload-planner-migration.md) - requires one physical compile path for exactly this reason. If the collector - subplan and the backend subplan are derived independently — even from - the same selected DAG, by two different code paths, at two different - times — nothing stops them drifting. A single compile step that emits - both subplans from one pass over one selected DAG, stamped with one - shared identity (§4), is what removes that possibility structurally - instead of by convention. -5. **The two wires evolve independently and are consumed by different - processes at different times.** OpAMP YAML is read by ASAPCollector; - `BackendPlan` protobuf is read by `data_plane`. Neither should decode - the other's format, and neither should decode ASAPPlanner's internal - Rust IR — that IR is not a stable cross-process wire contract and was - never meant to be one (migration plan §§2 and 4.2). - -## 3. `CompiledPlan`: one compile step, two subplans, one identity - -```rust -/// The output of compiling one `GlobalSelection` for one deployment -/// topology. This is control_plane's L5 (see -/// design-target-architecture.md §2, "L5 — physical plan": the one layer -/// this deployment owns in full because no upstream `asap-physical` crate -/// exists) — and it is the *only* thing that leaves the control plane's -/// planning boundary. Neither subplan is ever emitted independently of the -/// other; they are two views produced by the same compile call. -pub struct CompiledPlan { - /// Shared identity across BOTH subplans. Content addressed from the - /// selected DAG, topology identity, and semantic constraints. Mutable - /// sizing and lifecycle values are deliberately excluded and ordered by - /// `plan_version` instead. - pub plan_id: PlanId, - /// Monotonic per-`plan_id` counter — bumped on re-compile against an - /// unchanged selection (e.g. a resize), not on every replan. - pub plan_version: u64, - /// Not-before: neither subplan should be treated as authoritative - /// before this time. Lets a warm cutover (see migration plan §9, - /// `DeploymentPlanDiff`) land both subplans ahead of the switch. - pub activation: DateTime, - /// Not-after / supersede horizon. `None` for "until superseded." - pub expiry: Option>, - /// Identifies the *wire schema version* the backend subplan requires, - /// so a collector/backend pair that somehow ends up on mismatched - /// deploys fails a compatibility check instead of silently serving - /// under the wrong contract. Distinct from `plan_id`: this changes on - /// a schema/deploy version bump, not on every replan. - pub backend_compat: BackendCompatId, - - pub collector: CollectorSubplan, - pub backend: BackendSubplan, -} - -pub struct CollectorSubplan { - pub plan_id: PlanId, // == CompiledPlan::plan_id - pub plan_version: u64, // == CompiledPlan::plan_version - pub activation: DateTime, - pub expiry: Option>, - pub backend_compat: BackendCompatId, - /// One entry per collector fleet member this plan touches. - pub edges: Vec, -} - -pub struct EdgeAssignment { - pub edge_id: String, - /// The versioned CollectorPlan fields (§5), produced by compiling the - /// `SummaryAgg` nodes assigned to this edge, not authored ad hoc. - pub config: AsapEdgeConfig, - /// Opaque identity of *this edge's* exact YAML body — unchanged - /// semantics from ASAPCollector's existing `config_hash` (it still - /// identifies collector-config bytes, nothing more); `plan_id` is the - /// new, separate field that identifies the plan those bytes were - /// compiled from. - pub config_hash: ConfigHash, -} - -pub struct BackendSubplan { - pub plan_id: PlanId, // == CompiledPlan::plan_id - pub plan_version: u64, // == CompiledPlan::plan_version - pub activation: DateTime, - pub expiry: Option>, - pub backend_compat: BackendCompatId, - /// `BackendPlan` from design-backend-plan-wire-format.md §3. Its - /// envelope fields equal this `BackendSubplan` and the matching - /// `CollectorSubplan`. - pub backend_plan: BackendPlan, -} +> Scope: the ASAPQuery-backend physical-planning step between ASAPPlanner's +> selected post-ASAP workload DAG and the two runtime executors: +> ASAPCollector and the ASAPQuery data plane. + +## TL;DR + +ASAPPlanner selects a logical plan. That plan says which summaries and exact +operations answer a workload, but it intentionally does not choose machines, +shards, runtime windows, transport modes, or storage routes. + +ASAPQuery-backend performs one physical compile that produces both runtime +views of the decision: + +```text +selected post-ASAP workload DAG + | + v + physical compilation + / \ + v v +CollectorSubplan BackendSubplan +CollectorPlan(s) BackendPlan ``` -## 4. Compile algorithm - -Input: the materialized selection (`GlobalSelection::materialize()`'s -`Rc` roots, with the shared identity required by migration plan -§4.2 intact) plus this deployment's topology and -constraints (collector fleet membership, per-edge shard/memory budgets, -transport cost model — the same inputs `physical::colored_dag` already -takes today, see §8). - -1. **Partition by node kind**, not by heuristic: every `SummaryAgg` - reached anywhere in the selection is an *update-side* node; every - `SummaryEstimate`/`SummaryMerge`/`SummarySubtract`/`SummaryDelete`/ - `SummaryJoin` is a *readout-side* node (it consumes summary state and - either produces more summary state for further readout-side composition, - or a plain value). `KeepPreAsap` subtrees are neither — they stay the - backend/archive fallback path already described in - [`design-target-architecture.md`](design-target-architecture.md) §3. -2. **Allocate each distinct `SummaryAgg` (by `Rc` identity) to one or more - collector edges.** A single logical `SummaryAgg` may become several - `EdgeAssignment` entries (sharding by cardinality/volume budget) or share - one existing edge with another `SummaryAgg` from a different query root - (the shared-`Rc` case). This is where `shard_count`, `edge_id` selection, - and per-edge resource budgeting happen — genuinely new information, not - copied from the selection. -3. **Insert `SummaryMerge` at the shard boundary** when step 2 sharded a - node: the collector side ships `shard_count` partial states: the backend - side's `Materialization` reflects one logical summary, reconciled via - merge before any `SummaryEstimate` reads it (`SummaryMerge`'s own - catalog-`mergeable` requirement, already enforced upstream, is what makes - this legal at all). -4. **Compile every readout-side subtree into `Materialization` + - `RoutingEntry` rows** in the backend subplan (§7), each one recording - which `EdgeAssignment`(s) supply its input summary state. -5. **Decide transport parameters** (`delta_transmission`, `delta_threshold`, - `drop_original`, `warm_allowed_lateness`) per `EdgeAssignment` from the - deployment cost model — never from the selection, which has no opinion - on transport (§2.2). -6. **Compute `plan_id`** from the compiled structure (§3), stamp it plus - `plan_version`/`activation`/`expiry`/`backend_compat` identically onto - both subplans, and return the `CompiledPlan`. - -Steps 2–3 are exactly the job `physical::allocator::SketchAllocator` and -`physical::stage_split` already do today against the *legacy* locally-typed -`QueryExpr`/`PipelineStage` tree (`physical/plan.rs`); this document asks -for the same allocation job, retargeted to consume ASAPPlanner's own -`SummaryNode` selection instead of a parallel local IR — see §9. - -## 5. Collector subplan wire contract - -The authoritative collector-side schema is ASAPCollector's -[`ASAPQuery-to-ASAPCollector collection-plan interface`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). -This document does not define a second flat `asap_edge.metrics[]` schema. - -For each `EdgeAssignment`, the compiler emits one versioned -`CollectorPlan` YAML document in the exact OpAMP `AgentConfigMap` entry -`asap-collector-plan.yaml`, with content type `application/yaml`. The OpAMP -protobuf is the transport envelope; the YAML document is the typed physical -execution contract. It is not a serialized ASAPPlanner Rust DAG and it is -not a complete OTel Collector configuration. - -The compiler maps the selected DAG into that schema as follows: - -| Selected post-ASAP field | `CollectorPlan` field | +The two subplans share the same plan and materialization identities. They are +never derived independently and are never considered active unless both sides +confirm a compatible decision. + +## 1. Why this layer exists + +ASAPPlanner owns logical choices such as: + +- exact accumulator versus approximate summary; +- summary family, algorithm, and parameters; +- reduction and grouping strategy; +- logical sharing and composition; +- summary readout; and +- exact fallback. + +The runtime still needs deployment decisions that do not belong in Planner: + +- which collector or backend stage runs each operation; +- how logical state is sharded or shared; +- which streaming panes materialize a query time range; +- whether state is sent as raw observations, full summaries, or deltas; +- where state is stored and queried; +- when a new plan becomes active; and +- how incompatible or failed updates are rolled back. + +Skipping this layer causes two common failures: + +1. treating a logical `SummaryAgg` as though it already names a collector + process and runtime configuration; and +2. deriving collector and backend plans separately, allowing family, + parameters, grouping, windows, or identities to drift. + +The physical compiler closes both gaps in one operation. + +## 2. Input contract + +The compiler receives: + +- the selected post-ASAP DAG for the whole workload; +- stable workload/query correlation information; +- collector and backend capability snapshots; +- deployment topology and stage boundaries; +- workload statistics and resource constraints; +- runtime window, freshness, and retention policy; and +- transmission and storage policy. + +Shared logical nodes remain shared at this boundary. The compiler must not +first flatten the workload into independent per-query or per-metric rows. + +The selected DAG may contain summary producers, estimates, merges, +subtractions, deletes, joins, exact operations, shared sub-DAGs, and +`KeepPreAsap` fallback. A pinned Planner revision may also carry explicit +execution phases and result guarantees. The compiler consumes those canonical +values rather than defining local equivalents. + +## 3. Output contract + +One compile returns one `CompiledPlan` with: + +- a collector subplan containing one `CollectorPlan` for every targeted + collector; +- a backend subplan containing one `BackendPlan` for the ASAPQuery data + plane; +- a shared plan envelope; +- content-addressed materialization identities; and +- a validation record showing that both subplans were produced from the same + selected DAG and capability snapshots. + +### Shared plan envelope + +| Field | Meaning | | --- | --- | -| `SummaryAgg` identity | `materializations[].logical_node_ref` plus a content-addressed `materializations[].id` | -| bound `Source` and predicates | `materializations[].input.metric` and canonical `input.matchers` | -| `SummaryAgg.col` | `materializations[].input.value` | -| `SummaryFamilyType` | `materializations[].summary.family` | -| sketch algorithm and parameters | `summary.algorithm` and typed `summary.parameters` | -| Planner accuracy constraint | `summary.accuracy` | -| `Reduction::PerEntity` | `reduction.kind: per_entity` | -| `Reduction::Reduce(GroupKeys)` | `reduction.kind: reduce`, explicit `by`, and `without` | -| `GroupingStrategy` | `grouping.kind`, plus Hydra kind/parameters for shared grouping | - -The physical compiler adds fields ASAPPlanner intentionally does not own: -target agent/edge, capability snapshot, concrete streaming windows, local -shards, exporter reference, and raw/full/delta transmission policy. These -fields must never be inferred by ASAPCollector from missing values. - -The complete plan envelope carries `plan_id`, `plan_version`, `activation`, -`expiry`, and `backend_compat` verbatim from `CompiledPlan`. Each emitted -summary or delta also carries those compatibility identities plus its -materialization, window, producer, and sequence/checkpoint identity. - -Unsupported Planner alternatives remain visible in the logical candidate -space but cannot be emitted unless the targeted collector capability snapshot -and backend compatibility ID both support them. The compiler chooses another -valid candidate or exact fallback; it never renames an unsupported algorithm -to a similar supported one. In particular, shared Hydra grouping must not be -silently flattened to independent per-group state. - -## 6. Gaps this closes vs. what it still leaves open - -**Closes in the target design**, on the ASAPCollector side: the -`CollectorPlan` envelope now has explicit `plan_id`, `plan_version`, -`activation`, `expiry`, and `backend_compat` fields, carried identically on -the matching backend plan. It is delivered as the -`asap-collector-plan.yaml` OpAMP config-map entry. The collector returns the -semantic result through the `io.asap.collector.plan.v1` / -`application_report` custom message. The MVP harness compares the active -plan and materialization identities on both sides instead of treating -OpAMP's `config_hash` or `RemoteConfigStatus.APPLIED` as proof of semantic -activation. - -This remains a target contract rather than a claim about current runtime -behavior. ASAPCollector currently writes a complete OTel YAML file, restarts, -and reports only the OpAMP config hash after a syntax check. Implementing the -new parser, atomic activation, and application report is a separate code -change. - -**Opens**, in ASAPCollector's execution layer: the target schema can name all -Planner families and grouping layouts, but `cms_with_heap`, -`count_sketch_with_heap`, KMV, Theta, sampling, wavelets, statistical models, -and shared Hydra grouping still require actual collector and backend support. -Naming an algorithm in the schema does not advertise that runtime support. - -**Stays open**, and is explicitly out of scope here: the rollup algebra -question already on record in -[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) -§7 ("which summary families roll up safely"), and the composed exact/summary -execution gaps tracked against ASAPPlanner PR #300 / issue #171. -`CompiledPlan` treats a `RollupStrategy` selection the same as any other -readout-side subtree (§4 step 4) — it does not independently re-derive -rollup legality, which remains ASAPPlanner's decision to have made during -selection. - -## 7. Backend subplan materialization shape - -[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md) -§3 carries ASAPPlanner's current `SummaryFamilyType` directly. That type -nests algorithm and parameters for sketches and retains the orthogonal -`GroupingStrategy` axis; the backend wire must not re-flatten it into a local -`SummaryKind`/`SummaryParams` vocabulary: - -```rust -pub struct Materialization { - pub fingerprint: PolicyFingerprint, - pub source: Source, - pub window: WindowSpec, - pub group_by: Vec, - pub rollup: Vec, - - /// Current upstream type directly, including sketch grouping layout. - pub family: SummaryFamilyType, - pub col: ColumnRef, - - /// New: which `EdgeAssignment`(s) this materialization's input summary - /// state comes from. Lets the backend validate, at plan-apply time, - /// that the collector subplan sharing this `CompiledPlan::plan_id` - /// actually produces a `family`-compatible input — the concrete - /// mechanism behind the migration doc's completion criterion - /// ("Backend applies an incompatible plan -> reject the emitted state - /// or fail the run"). - pub sources: Vec, - - pub retention: Option, -} - -pub struct EdgeSourceRef { - pub edge_id: String, - pub materialization_id: PolicyFingerprint, -} +| `plan_id` | Content-addressed identity of the logical selection, topology identity, and semantic constraints. | +| `plan_version` | Ordered update within the same plan identity, such as changed sizing or lifecycle policy. | +| `activation` | Earliest time at which both subplans may become authoritative. | +| `expiry` | Optional time after which the plan may no longer produce or serve state. | +| `backend_compat` | Compatibility identity for BackendPlan and emitted summary-state schemas. | +| `planner_revision` | Immutable ASAPPlanner revision used for the selection. | + +Mutable lifecycle or sizing fields do not silently change the identity of an +existing version. Reusing the same `(plan_id, plan_version)` for different +content is invalid. + +### Materialization identity + +A materialization is the physical state built for one selected logical +summary producer. Its identity includes every property required for safe +reuse and merge, including: + +- bound source and filters; +- summarized value/item; +- summary family, algorithm, and parameters; +- reduction and grouping layout; +- logical/physical window contract; and +- state schema compatibility. + +Placement, transport cadence, or storage location may change without +pretending a semantically different summary is the same materialization. + +Explain or viewer node IDs are traceability metadata, not materialization +identity. + +## 4. Partitioning the selected DAG + +The baseline partition follows data availability: + +- operations that consume new observations and construct maintained state + belong on the update side; +- operations that read, merge, or transform maintained state into query + answers belong on the readout side; and +- `KeepPreAsap` remains exact backend/archive execution. + +In the current Planner vocabulary, `SummaryAgg` is the principal update-side +boundary and `SummaryEstimate` is a readout. Summary merge and other state +composition are placed according to topology, capabilities, and transport +cost without changing their logical semantics. + +If the pinned Planner revision provides explicit execution availability or +phase assignments, those validated phases are authoritative. The compiler +must not infer a conflicting phase from node names. + +An operation may be assigned only to an executor that advertises its full +semantics. A nameable Planner alternative is not automatically deployable. + +## 5. Collector subplan + +The collector subplan follows ASAPCollector's +[collection-plan interface](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). + +For each assigned collector, it specifies: + +- target collector and capability snapshot; +- the shared plan envelope; +- source metrics and matchers; +- materialization identities; +- summary family, algorithm, parameters, and accuracy requirement; +- reduction and grouping semantics; +- concrete streaming windows and lateness; +- local sharding; +- raw, full, or delta transmission; and +- backend endpoint/schema compatibility. + +OpAMP is the delivery transport. The payload is the versioned +`asap-collector-plan.yaml` document, not ASAPPlanner's Rust IR and not the +collector's complete bootstrap configuration. + +## 6. Backend subplan + +The backend subplan follows +[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md). + +It specifies: + +- the same plan envelope; +- every expected materialization; +- the collector assignments producing its state; +- exact summary family, algorithm, parameters, grouping, and windows; +- ingestion/storage destination; +- query capabilities and readouts satisfied by the materialization; +- remaining backend-side operators; and +- selected result guarantees when provided by Planner. + +The backend does not reconstruct the chosen summary from query text or stored +state. It installs and executes the control plane's exact decision. + +## 7. Sharing, sharding, and merge + +One shared logical producer remains one logical materialization even when it +serves several queries. The backend may associate several routing/readout +entries with that one materialization. + +A materialization may have several physical producers when sharded across +collectors. The compiler records all producers and inserts a compatible merge +at the selected boundary. A merge is legal only when every input agrees on +the materialization contract and the selected family supports merge. + +Sharding does not create several unrelated logical summaries, and sharing +does not allow consumers with incompatible filters, reductions, grouping, +windows, parameters, or guarantees to reuse state. + +## 8. Window and transmission decisions + +Planner time ranges express query semantics. The physical compiler chooses +streaming panes capable of answering those ranges. + +For the MVP: + +- panes are anchored and tumbling; +- pane composition must exactly cover each claimed query range; +- allowed lateness and watermark behavior are explicit; +- incompatible alignment is rejected; and +- freshness policy is shared with the backend readiness check. + +Transmission is independent of logical summary choice: + +- `raw` forwards selected observations for exact/backend execution; +- `full` sends complete summary state; and +- `delta` sends ordered state changes plus periodic full checkpoints. + +Delta is legal only when collector and backend advertise the same state, +sequence, and checkpoint semantics. Every payload identifies its plan, +materialization, producer, window, sequence, and base/checkpoint. + +## 9. Compile and activation sequence + +1. Validate the selected DAG against both capability snapshots. +2. Allocate update/readout operators and physical producers. +3. Choose compatible windows, transmission, and storage routes. +4. Construct materialization identities. +5. Emit both subplans from the same in-memory decision. +6. Validate cross-subplan equality for all shared contracts. +7. Stage both subplans before `activation`. +8. Require backend installation and collector semantic application reports. +9. Route queries to the new plan only after both sides report compatible + active identities. +10. Retire old state after its readers and lateness horizon drain. + +If any step fails, the previous unexpired plan remains authoritative. A +partial push, OpAMP delivery acknowledgement, file write, or process restart +does not constitute plan activation. + +## 10. Fail-closed rules + +The compiler or runtime rejects the plan when: + +- an assigned executor lacks a required family, algorithm, grouping, phase, + readout, window, or transmission capability; +- collector and backend materialization contracts differ; +- a required accuracy guarantee is unknown or insufficient; +- a merge combines incompatible state; +- delta sequencing/checkpoint semantics do not match; +- plan versions conflict or lifecycle conditions disallow activation; or +- semantic application evidence is missing. + +It must never substitute another family, parameter, grouping layout, +accuracy target, or transmission semantics to make an invalid plan appear +deployable. + +## 11. Example + +For: + +```promql +quantile_over_time(0.95, request_duration_seconds{region="us-east"}[5m]) ``` -`BackendPlan.plan_id` is the field joining the two subplans. Content-addressed -`PolicyFingerprint` remains the identity of one `Materialization` -(reuse/diff/resize within a single backend subplan, per the migration doc's -`DeploymentPlanDiff`); `plan_id` answers a different question — "were these -two subplans compiled together" — that a per-materialization fingerprint -cannot answer. - -## 8. What does not change - -- `PolicyFingerprint`, `RoutingIndex`, hot reload, `DeploymentPlanDiff`, - warm cutover, and archive fallback — all as designed in - `design-backend-plan-wire-format.md` and - `design-asapplanner-workload-planner-migration.md` §5/§6. -- OpAMP's `AgentRemoteConfig`/`AgentConfigMap`/`config_hash` delivery - mechanics. `config_hash` still identifies exact remote-config bytes; it - does not replace `plan_id` or the semantic application report. -- Existing `asap_edge` runtime behavior until the versioned `CollectorPlan` - parser and apply path are implemented. - -## 9. Migration notes - -- `physical::plan::PlanNode` / `PipelineStage` / `physical::allocator:: - SketchAllocator` / `physical::stage_split` operate on this repo's own - locally-typed `QueryExpr` (`crate::intent_algebra`), annotated with a - per-node `PipelineStage` tag on one combined tree. `CompiledPlan` replaces - that shape with two explicit typed subplans compiled from ASAPPlanner's - own selected `SummaryNode` DAG. This module belongs on the - migration plan §9 Phase 6 removal list — remove it only after the compiled - plan path is selected and rollback no longer depends on legacy planning. -- `emit::agent::generate_agent_collector_config` currently builds a complete - collector YAML. It should become the `EdgeAssignment -> CollectorPlan` - serializer defined in §5. Bootstrap OTel receivers/exporters and credentials - remain deployment configuration; a workload replan must not replace them. -- Both subplans should land behind the same workload-planner rollout mode - the migration plan defines (§9 Phase 5): in - `shadow` mode, compile `CompiledPlan` and record `plan_id` agreement and - field-level diffs against the legacy allocator's output without pushing - either subplan, exactly mirroring that section's existing comparison - list. - -## 10. Open questions -- **`backend_compat` granularity.** One id per `BackendPlan` proto schema - version, or one per `(schema version, family vocabulary version)` so an - `asap_edge` schema gap closing (§6) doesn't force every unrelated plan to - recompute compatibility — needs a decision before the field ships, not - after. -- **Cross-shard `SummaryMerge` placement.** Step 3 (§4) inserts - `SummaryMerge` "at the shard boundary" without saying which physical - stage performs it — collector-side gateway merge vs. backend-side merge - on ingest are both live options already implied by `PipelineStage`'s - existing `Backend`/`Precompute` distinction, and the choice affects - `EdgeAssignment` fan-in bandwidth materially. Needs its own short design - pass, not resolved here. +Planner may select a per-entity DDSketch summary and a p95 readout. The +physical compiler may then: + +- assign DDSketch construction to selected collectors; +- choose one-minute panes that compose into the five-minute query range; +- transmit deltas every ten seconds with periodic full checkpoints; +- declare one shared DDSketch materialization in BackendPlan; and +- route the p95 query readout to that materialization. + +The compiler does not change DDSketch to KLL, change the accuracy parameter, +or aggregate series together merely because another physical layout would be +cheaper. Such a change requires selection of a different valid Planner +candidate. + +## 12. Non-goals + +This document does not define: + +- query parsing or summary selection; +- internal Planner serialization; +- exact protobuf field numbers; +- summary-state byte encoding; +- collector bootstrap configuration; +- storage-engine implementation; or +- query-engine implementation details. + +## 13. Definition of done + +The compiled-plan design is satisfied when: + +- one compile produces both subplans; +- all shared identities and semantic fields match; +- shared Planner nodes remain shared materializations; +- sharded producers merge only under a valid contract; +- unsupported shapes fail before activation; +- both plans stage and activate atomically from the user's perspective; +- emitted state carries the active identities; +- the data plane serves without replanning; and +- a deliberately mismatched or partially applied plan is rejected in an + end-to-end test. From 9b74237ca02eddfff1a40fad2bcf12dec4a07078 Mon Sep 17 00:00:00 2001 From: zz_y Date: Thu, 27 Aug 2026 17:59:29 -0600 Subject: [PATCH 12/12] docs: reorganize control-plane design by ownership --- control_plane/docs/README.md | 32 + control_plane/docs/asapplanner-integration.md | 177 ++ ...nd-plan-wire-format.md => backend-plan.md} | 5 +- ...-asapplanner-workload-planner-migration.md | 533 ----- .../docs/design-target-architecture.md | 303 --- control_plane/docs/design.md | 1888 ----------------- ...-backend-split.md => physical-planning.md} | 29 +- .../docs/query-to-edge-scope-and-sparse.md | 163 -- .../docs/query-to-sketch-translation.md | 754 ------- docs/adding-a-new-sketch.md | 2 +- 10 files changed, 241 insertions(+), 3645 deletions(-) create mode 100644 control_plane/docs/README.md create mode 100644 control_plane/docs/asapplanner-integration.md rename control_plane/docs/{design-backend-plan-wire-format.md => backend-plan.md} (98%) delete mode 100644 control_plane/docs/design-asapplanner-workload-planner-migration.md delete mode 100644 control_plane/docs/design-target-architecture.md delete mode 100644 control_plane/docs/design.md rename control_plane/docs/{design-compiled-plan-collector-backend-split.md => physical-planning.md} (90%) delete mode 100644 control_plane/docs/query-to-edge-scope-and-sparse.md delete mode 100644 control_plane/docs/query-to-sketch-translation.md diff --git a/control_plane/docs/README.md b/control_plane/docs/README.md new file mode 100644 index 00000000..0c46724c --- /dev/null +++ b/control_plane/docs/README.md @@ -0,0 +1,32 @@ +# ASAPQuery-backend control-plane design + +> Status: active + +## TL;DR + +These documents describe only the design owned by ASAPQuery-backend: its +integration boundary with ASAPPlanner, physical compilation for the collector +and backend, and the plan consumed by the ASAPQuery data plane. + +ASAPPlanner owns PromQL parsing, pre-ASAP and post-ASAP IR, query-to-summary +mapping, accuracy reasoning, workload sharing, candidate generation, and +logical plan selection. Refer to the +[ASAPPlanner repository](https://github.com/ProjectASAP/ASAPPlanner) for those +designs; they are intentionally not repeated here. + +## Documents + +| Document | ASAPQuery-backend-owned scope | +| --- | --- | +| [ASAPPlanner integration](asapplanner-integration.md) | Ownership boundary and integration contract for consuming a selected logical workload plan. | +| [Physical planning](physical-planning.md) | Placement, windows, representation, transmission, and compilation into matching collector and backend plans. | +| [BackendPlan](backend-plan.md) | Versioned contract installed and executed by the ASAPQuery data plane. | + +The corresponding collector-facing plan is documented by +[ASAPCollector](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). + +## Scope rule + +A design belongs here only when ASAPQuery-backend owns the decision or must +enforce the contract at runtime. If ASAPPlanner owns the decision, this +directory links to Planner instead of maintaining a second description. diff --git a/control_plane/docs/asapplanner-integration.md b/control_plane/docs/asapplanner-integration.md new file mode 100644 index 00000000..95a658d4 --- /dev/null +++ b/control_plane/docs/asapplanner-integration.md @@ -0,0 +1,177 @@ +# ASAPQuery-backend integration with ASAPPlanner + +> Status: proposed +> +> MVP relation: required for query planning and control-plane decisions. + +## TL;DR + +ASAPPlanner chooses a logical plan for a query workload. ASAPQuery-backend +turns that selected plan into deployable collector and backend plans, installs +them, and routes queries to the resulting state. + +ASAPQuery-backend does not maintain a second design for parsing PromQL, +building Planner IR, mapping queries to summaries, reasoning about accuracy, +sharing work across queries, or ranking logical candidates. Those designs are +owned by [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner). + +```text +PromQL workload + | + v +ASAPPlanner +selected logical workload plan + | + v +ASAPQuery-backend control plane +physical compilation and activation + | + +-------------------------+ + | | + v v +CollectorPlan BackendPlan +ASAPCollector ASAPQuery data plane +``` + +## Ownership boundary + +### ASAPPlanner + +Planner is the source of truth for: + +- query parsing and semantic IR; +- logical exact and summary-based alternatives; +- summary family, parameters, grouping, and readout semantics; +- accuracy constraints and logical result guarantees; +- workload-wide reuse and common subexpressions; +- logical cost comparison and candidate selection; and +- logical rejection and explanation. + +See ASAPPlanner's own design documents for those semantics. Their types and +rules must be consumed from the pinned Planner revision, not copied into this +repository. + +### ASAPQuery-backend control plane + +The control plane owns: + +- supplying workload context, runtime statistics, and executor capabilities; +- invoking the pinned Planner revision and consuming one selected workload + plan; +- choosing collector/backend placement, physical panes, transmission mode, + and storage routes; +- compiling matching CollectorPlan and BackendPlan artifacts; +- staging, activating, retiring, and rolling back plan versions; and +- exposing planning and activation status to operators. + +These decisions are described in [physical planning](physical-planning.md). + +### ASAPQuery data plane + +The data plane owns: + +- installing [BackendPlan](backend-plan.md); +- validating, ingesting, storing, and merging summary state; +- applying the selected readout and remaining backend-side operators; +- enforcing readiness, freshness, and plan identity; and +- executing an explicit exact fallback when the selected plan requires one. + +### ASAPCollector + +ASAPCollector owns validation and execution of CollectorPlan: observing the +selected metrics, maintaining the requested state, transmitting raw data or +full/delta summaries, and reporting which plan is actually active. Its +interface is documented in the +[ASAPCollector repository](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). + +## Integration contracts + +### Workload request + +ASAPQuery-backend passes the complete workload to Planner so that sharing and +global selection remain possible. A request includes the query expressions, +evaluation shape, requested accuracy, and relevant schema or cost context. +Protocol-specific scheduling and alert state remain outside Planner. + +For example, two dashboard queries that read the same five-minute latency +distribution are planned together. ASAPQuery-backend must not reduce them to +two independent `(metric, sketch)` requests before Planner can identify shared +state. + +### Selected logical plan + +The returned boundary is one selected post-ASAP workload plan from the pinned +Planner API. Shared logical nodes remain shared. Exact fallback is an explicit +part of that plan; absence of a supported summary is not permission for the +backend to invent one. + +For example, Planner may select one DDSketch producer shared by: + +```promql +quantile_over_time(0.50, request_duration_seconds[5m]) +quantile_over_time(0.95, request_duration_seconds[5m]) +``` + +The backend consumes the shared producer and two readouts. It must not select +DDSketch again from the query strings. + +### Physical plans + +One physical compile produces CollectorPlan and BackendPlan from the same +selected logical plan. Both artifacts carry the same plan version and +materialization identities. Independent compilation is invalid because it can +make the collector and data plane disagree about family, parameters, grouping, +windows, or state representation. + +### Runtime evidence + +Delivery acknowledgement is not activation evidence. Query routing changes +only after the backend has installed BackendPlan and each required collector +has reported that it applied the matching CollectorPlan. Emitted state must +also carry identities that the backend can validate. + +## Planner version policy + +All Planner crates used by one backend build are pinned to one immutable +revision. The revision is recorded in compiled plans and diagnostic output. + +An upstream Planner change is adopted only after the backend has deliberately +handled any new IR variant, guarantee, capability, or cost-model requirement. +Unknown variants fail as unsupported; the backend must not pre-copy proposed +Planner types or infer semantics from explain/viewer output. + +Open Planner pull requests are compatibility inputs, not backend design. Their +details stay in Planner and in the implementation change that updates the pin, +instead of becoming a second long-lived specification here. + +## Failure behavior + +Planning or activation fails closed when: + +- the selected logical operation is unsupported by its assigned executor; +- collector and backend plans disagree on any materialization contract; +- a required accuracy guarantee is missing or insufficient; +- full/delta state semantics are incompatible; +- a plan is stale, expired, partially applied, or from another run; or +- exact fallback is required but unavailable. + +The system must not silently choose another summary, loosen an accuracy +requirement, remove grouping, or serve state from a different plan. + +## MVP requirements + +The integration is MVP-ready when one end-to-end run demonstrates that: + +- the submitted PromQL workload reaches Planner as one workload; +- the selected logical plan is preserved through physical compilation; +- matching collector and backend plans are captured as artifacts; +- ASAPCollector reports semantic application of the collector plan; +- the data plane serves supported summary-backed queries from BackendPlan; +- unsupported queries are rejected or use the selected exact fallback; and +- missing, stale, or incompatible plan evidence makes the run fail. + +## Non-goals + +This document does not define Planner IR, query-to-summary mappings, accuracy +algebra, summary algorithms, collector configuration fields, state byte +encoding, or query-engine implementation details. diff --git a/control_plane/docs/design-backend-plan-wire-format.md b/control_plane/docs/backend-plan.md similarity index 98% rename from control_plane/docs/design-backend-plan-wire-format.md rename to control_plane/docs/backend-plan.md index b73fcb1b..303c90b7 100644 --- a/control_plane/docs/design-backend-plan-wire-format.md +++ b/control_plane/docs/backend-plan.md @@ -2,6 +2,9 @@ > Status: proposed > +> MVP relation: required for the ASAPQuery data plane to ingest and query the +> state selected by the control plane. +> > Scope: the typed runtime contract by which ASAPQuery-backend's control > plane tells its data plane what summary materializations exist, how they > are ingested, and which query capabilities they serve. @@ -55,7 +58,7 @@ compiler creates: - BackendPlan, which validates, stores, routes, merges, and reads them. The two are defined together in -[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). +[`physical-planning.md`](physical-planning.md). BackendPlan is not: diff --git a/control_plane/docs/design-asapplanner-workload-planner-migration.md b/control_plane/docs/design-asapplanner-workload-planner-migration.md deleted file mode 100644 index 1b769ae5..00000000 --- a/control_plane/docs/design-asapplanner-workload-planner-migration.md +++ /dev/null @@ -1,533 +0,0 @@ -# ASAPQuery-backend integration with ASAPPlanner - -> Status: proposed -> -> Scope: migrate ASAPQuery-backend to use ASAPPlanner as its only logical -> query and summary planner, while keeping deployment planning and execution -> inside ASAPQuery-backend. - -## TL;DR - -ASAPPlanner answers: - -> What exact or summary-based logical plans can answer this query workload, -> and which candidate should be selected under the supplied correctness and -> cost constraints? - -ASAPQuery-backend answers: - -> Where should the selected logical operators run, how are their states -> transmitted and stored, and how are queries routed to the active plan? - -The integration boundary is the selected post-ASAP workload DAG. ASAPQuery -must not duplicate Planner parsing, canonicalization, summary selection, -accuracy algebra, common-subexpression search, or candidate ranking. -ASAPPlanner must not own collector placement, runtime windows, OpAMP, -transmission mode, storage routing, deployment rollout, or query serving. - -One downstream physical compile converts the selected DAG into two matching -runtime plans: - -```text -Query workload - | - v -ASAPPlanner - selected post-ASAP workload DAG - | - v -ASAPQuery-backend physical compiler - | - +-----------------------------+ - | | - v v -CollectorPlan BackendPlan -ASAPCollector ASAPQuery data plane -build/transmit state ingest/store/read state -``` - -Both plans carry the same plan and materialization identities. Neither plan -is activated unless both sides validate the same compiled decision. - -## 1. Goals - -The migration must produce one planning path that: - -- accepts one-shot and repeating PromQL workloads, with SQL enabled only when - a real schema catalog is available; -- preserves workload-wide sharing instead of flattening queries into - independent metric/sketch requests; -- uses ASAPPlanner's selected summary family, algorithm, parameters, - reduction, grouping strategy, readout, and accuracy contract exactly; -- compiles that selection into compatible collector and backend plans; -- lets the data plane execute the selected plan without planning again; -- rejects unsupported or incompatible plans before activation; -- supports shadow comparison and rollback during migration; and -- removes the legacy planner only after the new path passes end-to-end gates. - -## 2. Non-goals - -This migration does not: - -- move physical placement or scheduling into ASAPPlanner; -- serialize ASAPPlanner's internal Rust DAG as a runtime wire format; -- use DAG-viewer JSON, node colors, explain IDs, or rationale strings as - execution input; -- require every summary family or every open Planner proposal to be supported - by the MVP runtime; -- remove exact raw/archive fallback; -- reimplement Prometheus rule scheduling or alert state in ASAPPlanner; or -- delete the legacy path before shadow validation and rollback exist. - -## 3. Ownership - -### ASAPPlanner owns logical planning - -ASAPPlanner is the canonical owner of: - -- query parsing and semantic lowering; -- pre-ASAP IR and canonicalization; -- schema binding; -- workload-level common-subexpression discovery; -- exact and summary-based candidate generation; -- summary family, algorithm, parameters, and grouping alternatives; -- logical rewrites and reuse opportunities; -- accuracy requirements and, when supported by the pinned revision, result - guarantees; -- cost-aware candidate search and global selection; and -- logical explain and rejection information. - -### ASAPQuery-backend owns physical planning and rollout - -The ASAPQuery control plane owns: - -- source/protocol adapters and caller identity; -- runtime statistics, budgets, and its ASAPPlanner cost-model implementation; -- executor capability discovery; -- collector/backend/archive placement; -- physical streaming windows and lateness policy; -- raw, full-summary, and delta-summary transmission decisions; -- materialization identity and storage routing; -- creation of matching `CollectorPlan` and `BackendPlan` artifacts; -- deployment diff, warm-up, activation, retirement, and rollback; and -- planning telemetry and operator-facing explain endpoints. - -### ASAPQuery data plane owns execution - -The data plane owns: - -- backend-plan installation and atomic hot reload; -- summary-state ingestion, validation, storage, and merge; -- query-time readout and remaining backend-side logical operators; -- readiness, freshness, and watermark checks; -- exact archive fallback; and -- Prometheus-compatible request and response behavior. - -### ASAPCollector owns collector-plan execution - -ASAPCollector owns: - -- validating and atomically applying its `CollectorPlan`; -- computing the specified materializations; -- emitting raw, full, or delta payloads as directed; -- rejecting unsupported families, parameters, grouping layouts, or - transmission modes; and -- reporting semantic plan activation and emitted-state evidence. - -## 4. Stable integration contracts - -### 4.1 Workload input - -Protocol adapters convert external requests into ASAPPlanner's canonical -workload model. They may attach caller IDs outside the Planner value, but they -must not copy Planner domain types into a second backend schema. - -For each query, the planning input contains at least: - -- query language and expression; -- one-shot or repeating evaluation shape; -- requested accuracy; -- optional latency requirement; -- schema/catalog reference when required; and -- protocol-neutral recurrence information. - -Prometheus-only behavior remains adapter/runtime metadata, including rule -group ordering, query offset, missed iterations, `for`, `keep_firing_for`, -labels, annotations, and Alertmanager state. - -### 4.2 Planner output - -The output consumed by ASAPQuery is one selected post-ASAP DAG for the whole -workload, with shared node identity intact. It may contain: - -- `SummaryAgg` producers; -- exact accumulators and approximate summaries; -- `SummaryEstimate` readouts; -- reductions and grouping strategies; -- summary merge/subtract/delete/join operations; -- logical rewrites and shared sub-DAGs; and -- `KeepPreAsap` exact fallback subtrees. - -ASAPQuery does not flatten this DAG into `(metric, query type, sketch)` rows -before placement. Doing so would lose sharing, composition, grouping, rollup, -and provenance. - -### 4.3 Physical compile - -The physical compiler consumes exactly one selected DAG plus deployment -topology, capabilities, statistics, and resource constraints. It produces one -`CompiledPlan` with: - -- a `CollectorSubplan` containing one `CollectorPlan` per targeted collector; -- a `BackendSubplan` containing the matching `BackendPlan`; -- shared `plan_id`, `plan_version`, `activation`, `expiry`, and - `backend_compat` values; and -- shared content-addressed materialization identities. - -The collector and backend portions are emitted by the same compile operation. -Two independent compilers must not reinterpret the selected DAG separately. - -The detailed split is defined in -[`design-compiled-plan-collector-backend-split.md`](design-compiled-plan-collector-backend-split.md). - -### 4.4 Collector interface - -The collector half follows ASAPCollector's -[collection-plan interface](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md): - -- OpAMP protobuf is the delivery envelope; -- `asap-collector-plan.yaml` is the exact config-map entry; -- the entry contains a versioned `CollectorPlan`, not a complete OTel - bootstrap configuration; -- summary family, algorithm, parameters, accuracy, reduction, and grouping - retain Planner semantics; -- source binding, windows, placement, and transmission are added by the - physical compiler; and -- semantic activation requires the collector application report, not only - `RemoteConfigStatus.APPLIED`. - -### 4.5 Backend interface - -`BackendPlan` carries: - -- the shared compiled-plan envelope; -- every planned materialization and its exact `SummaryFamilyType`; -- source, filter, window, reduction, grouping, and storage route; -- collector source references; -- query capabilities and readouts satisfied by each materialization; and -- the selected result guarantee when the pinned Planner revision supplies - one. - -The data plane builds its routing index from this plan. It must not run a cost -model or infer summary parameters from stored state at query time. - -The detailed backend contract is defined in -[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md). - -## 5. Planner version and open-PR policy - -ASAPQuery pins every ASAPPlanner crate to one immutable revision of Planner -`main`. It never combines several open PR branches in production. A pin update -changes all Planner crates and the lockfile together. - -Open Planner work is handled according to its integration effect: - -| Planner PR | Integration effect | Adoption rule | -| --- | --- | --- | -| [#300](https://github.com/ProjectASAP/ASAPPlanner/pull/300) | Adds explicit update/readout phases and exact-summary composition | After merge, use validated execution availability to partition collector and backend operators. Enable only phases supported by both executors. | -| [#299](https://github.com/ProjectASAP/ASAPPlanner/pull/299) | Adds typed result guarantees, error propagation, budgets, and accuracy rejections | After merge, supply root accuracy targets, reject invalid guarantees before cost ranking, and preserve the selected guarantee in `BackendPlan`. | -| [#295](https://github.com/ProjectASAP/ASAPPlanner/pull/295) | Adds recurrence-aware CSE and global selection | After merge, provide evaluation/update rates and an explicit horizon for mixed one-shot/repeating workloads. Scheduling remains outside Planner. | -| [#293](https://github.com/ProjectASAP/ASAPPlanner/pull/293) | Allows an explicit approximate TopK candidate for an Exact-requested TopK | Opt in only through the upstream hook with an approved non-Exact sizing target. Record the effective approximation; retain exact pass-through. | -| [#291](https://github.com/ProjectASAP/ASAPPlanner/pull/291) | Adds optional `Concat` discriminator unique-key metadata | Preserve it in exhaustive IR visitors. Do not invent keys downstream. It creates no collector primitive by itself. | -| [#296](https://github.com/ProjectASAP/ASAPPlanner/pull/296) | Adds explain/viewer cost annotations | Consume only in explain output. Never use it for execution, identity, or placement. | -| [#292](https://github.com/ProjectASAP/ASAPPlanner/pull/292) | Corrects viewer node categories | No runtime integration effect. | - -This table is an audit, not a merge dependency list. The migration proceeds -against the pinned baseline. When a contract-affecting PR merges, the pin -update must include its adapter/compiler/capability changes and golden tests -in the same backend PR. - -ASAPQuery must not pre-copy proposed Planner types such as phase assignments, -result guarantees, recurrence profiles, or discriminator keys. Until a type -exists at the pin, it is absent. When a new upstream variant appears, physical -compilation fails with an unsupported-shape diagnostic until it is mapped -deliberately. - -## 6. Accuracy and cost - -### Accuracy - -`AccuracyTarget` is the requested correctness constraint. If the pinned -Planner revision supplies `ResultGuarantee`, that is the computed guarantee -of the selected result. They are different values and both remain owned by -Planner. - -The required order is: - -```text -candidate generation - -> guarantee propagation - -> reject candidates outside the requested target - -> cost ranking among valid candidates - -> global selection - -> physical placement -``` - -Cost and placement cannot resurrect an accuracy-invalid candidate. Unknown -error bounds, probabilities, or required statistics remain unknown; they are -never converted to zero or exact. - -`AccuracyTarget::Exact` normally selects an exact summary or `KeepPreAsap`. -The only planned exception is the explicit TopK policy exposed by Planner -#293. If enabled, the compiled plan states that the chosen realization is -approximate and records its effective target and guarantee. - -### Cost - -ASAPQuery supplies deployment-specific cost information through Planner's -cost-model interface. Inputs may include: - -- ingest/update rate; -- query evaluation rate; -- summary maintenance and read cost; -- exact recomputation cost; -- initial materialization cost; -- expected cardinality and subpopulation count; -- memory, network, and storage budgets; and -- the comparison horizon for mixed recurring and one-shot work. - -Missing statistics remain unknown. Structural fallback is allowed only when -Planner defines it explicitly. Backend code must not treat missing data as a -free operation. - -Physical placement applies deployment constraints after logical selection. It -does not choose a different logical summary because a selected placement is -inconvenient; it either finds a valid placement, asks Planner to select among -capability-constrained candidates, or fails planning. - -## 7. Repeating workloads - -ASAPPlanner receives protocol-neutral repeating-query intervals. These -intervals may affect workload CSE and cost selection, but they do not make -Planner a scheduler. - -Keep these concepts separate: - -- evaluation interval: how often the query runs; -- query lookback: the data range in the query, such as `[5m]`; -- physical pane size: chosen by the ASAPQuery physical compiler; -- query offset: changes the logical evaluation timestamp; -- allowed lateness and watermark: runtime readiness policy; and -- alert `for`/`keep_firing_for`: Prometheus alert-state behavior. - -For the MVP, physical windows use anchored tumbling panes. A chosen pane must -compose into every claimed query range, and the backend must query only panes -whose watermark covers the logical evaluation time. Incompatible window -alignment is a planning failure, not an approximate answer. - -When multiple recurring queries can reuse one materialization, recurrence- -aware selection uses their combined evaluation rate. Prometheus schedules -remain authoritative; inferred dashboard frequency must not override a -declared rule interval. - -## 8. Capability and failure contract - -Before selection and again before activation, ASAPQuery validates the chosen -plan against collector and backend capability snapshots. - -The following conditions fail closed: - -- Planner selects a family, algorithm, parameter set, grouping strategy, - execution phase, or readout unsupported by an assigned executor; -- the collector and backend disagree on plan or materialization identity; -- the backend expects a different summary family, parameters, grouping, - window, or state encoding from the collector; -- an accuracy guarantee is missing or insufficient where one is required; -- a delta mode lacks compatible sequencing/checkpoint semantics; -- a plan is stale, expired, or not yet active; -- application evidence is missing; or -- a query result would use state from an earlier plan/run. - -No failure may silently substitute another summary, loosen accuracy, erase a -grouping strategy, treat a missing series as zero, or return a plausible -result from incompatible state. - -## 9. Migration phases - -### Phase 0: pin and baseline - -- Pin all Planner crates to one immutable revision. -- Record that revision in build and planning artifacts. -- Capture legacy plans and query results for the golden workload. -- Add deadlines, size limits, and planning telemetry before enabling shadow - workloads. - -Exit gate: the workspace builds and existing behavior is unchanged. - -### Phase 1: canonical workload ingestion - -- Route single and multi-query inputs through one workload adapter. -- Add repeating-query input without moving scheduler metadata into Planner. -- Preserve caller IDs and per-query diagnostics outside canonical Planner - values. - -Exit gate: one-shot and repeating entries lower deterministically, and shared -canonical sub-DAGs remain shared. - -### Phase 2: workload selection in shadow mode - -- Run Planner candidate search and global selection for the full workload. -- Supply the ASAPQuery cost model, accuracy inputs, recurrence data, and - capability constraints supported at the pin. -- Export logical explain and typed rejection information. -- Compare with legacy decisions without changing production plans. - -Exit gate: every difference is explained by a documented semantic, accuracy, -cost, or sharing improvement; unexplained differences block rollout. - -### Phase 3: compile both physical plans - -- Convert one selected workload DAG into matching collector and backend - subplans. -- Preserve shared materializations and readout consumers. -- Produce stable materialization fingerprints and the shared plan envelope. -- Validate both subplans before either is pushed. - -Exit gate: golden tests prove that a deliberately mismatched collector/backend -pair is rejected and that matching plans round-trip through both wire formats. - -### Phase 4: executor coverage and end-to-end shadowing - -- Install BackendPlan without query-time replanning. -- Push CollectorPlan and require semantic application reports. -- Exercise exact accumulators, supported summaries, reductions, grouping, - readouts, merge, raw/full/delta modes, and archive fallback. -- Compare aligned ASAP and exact results, freshness, and plan identities. - -Exit gate: every selectable runtime shape is executed correctly or rejected -before selection; missing evidence fails the run. - -### Phase 5: selected rollout - -Use three rollout modes: - -| Mode | Planner executed | Production plans emitted | -| --- | --- | --- | -| `legacy` | legacy only | legacy | -| `shadow` | legacy and Planner | legacy | -| `selected` | Planner | compiled Planner-derived plans | - -Roll out `selected` by workload/tenant. Keep the previous valid compiled plan -available for immediate rollback. A failed new plan never replaces it. - -Exit gate: the agreed observation period has no unexplained correctness, -freshness, capability, or plan-identity failures. - -### Phase 6: delete legacy planning - -Remove legacy paths that: - -- union per-query capabilities into sketch choices; -- implement aggregate roots independently; -- choose summary families outside Planner; -- infer planned parameters from stored state at serving time; or -- generate collector and backend plans from separate logical decisions. - -Retain protocol adapters, physical placement, runtime cost inputs, -CollectorPlan/BackendPlan compilation, routing, execution, and archive -fallback. - -Exit gate: production has one logical planning path and one physical compile -path, with rollback based on previous compiled plans rather than legacy -planning. - -## 10. Validation - -### Cross-repository golden workload - -Maintain one versioned workload covering: - -- a repeated subexpression shared by multiple queries; -- compatible and incompatible filters; -- per-entity and grouped reductions, including empty global reduction; -- independent grouping and a capability-rejected shared grouping case; -- exact aggregation and each claimed MVP summary family; -- one-shot and repeating queries with different intervals; -- raw, full, and delta transmission; -- an unsupported query that takes exact fallback; -- an accuracy-invalid candidate; -- a stale or mismatched plan; and -- cold/archive fallback. - -For each pinned Planner revision, retain: - -- input workload and schemas; -- Planner revision and configuration; -- canonical and selected logical explain artifacts; -- rejected candidates and reasons; -- compiled collector and backend plans; -- capability snapshots; -- application reports; -- emitted materialization identities; -- query results and aligned exact results; and -- derived correctness, accuracy, freshness, latency, and cost measurements. - -### Required test boundaries - -1. External workload input to canonical Planner workload. -2. Canonical workload to selected post-ASAP DAG. -3. Selected DAG to matching CollectorPlan and BackendPlan. -4. Both wire formats to semantic activation. -5. Observations to stored summary state. -6. Query to summary readout or explicit exact fallback. -7. Replan to warm cutover, retirement, and rollback. - -### Pin-update tests - -Every Planner pin update must: - -- build all Planner-dependent crates at one revision; -- rerun the golden workload; -- update exhaustive mappings for new IR variants; -- prove unsupported new shapes fail before activation; -- compare selected materializations and guarantees against the previous pin; - and -- explain every intentional difference. - -## 11. Operational limits - -The control plane enforces checked-in limits for: - -- queries and schemas per workload; -- reachable IR nodes; -- candidate groups and candidates per group; -- planning iterations and wall time; -- explain artifact size; and -- materializations per compiled plan. - -Planning supports cancellation and reports phase timings, candidate counts, -rejections, selected cost, fallback, and timeout. A timeout or exceeded limit -returns a typed planning failure or configured exact fallback; it never emits -a partial compiled plan. - -## 12. Definition of done - -The migration is complete when all of the following are true in the same -supported release: - -- all production queries enter one workload-aware ASAPPlanner path; -- every Planner crate is pinned to the same recorded revision; -- the selected post-ASAP DAG is the only logical source for runtime plans; -- one compiler emits matching CollectorPlan and BackendPlan artifacts; -- shared workload sub-DAGs remain shared through materialization and serving; -- the data plane never selects or sizes summaries independently; -- accuracy targets and guarantees are preserved without backend-local - reinterpretation; -- every selected runtime shape is supported by both executors or rejected - before activation; -- semantic collector activation and backend activation agree on plan and - materialization identities; -- exact fallback remains available for unsupported queries; -- cross-repository golden and end-to-end tests pass; -- shadow rollout and rollback have been exercised; and -- legacy logical planning and serving-time reconstruction have been removed. diff --git a/control_plane/docs/design-target-architecture.md b/control_plane/docs/design-target-architecture.md deleted file mode 100644 index 3c3c5b09..00000000 --- a/control_plane/docs/design-target-architecture.md +++ /dev/null @@ -1,303 +0,0 @@ -# `control_plane` target architecture — planning-time and serving-time - -> **Status.** Target-state design, written independent of what's currently -> implemented in `control_plane`/`data_plane`. Grounded entirely in -> ASAPController's own current interfaces — specifically -> [`docs/l1-query-language.md`](https://github.com/ProjectASAP/ASAPController/blob/main/docs/l1-query-language.md) -> through -> [`l5-physical-plan.md`](https://github.com/ProjectASAP/ASAPController/blob/main/docs/l5-physical-plan.md)'s -> `## Interface` sections, added in -> [ASAPController#169](https://github.com/ProjectASAP/ASAPController/pull/169) -> (every signature there verified against ASAPController `main` at -> `cc18c98`, 2026-07-28) — not carried over from this repo's own history or -> assumed from memory. Where this doc says "should," it is describing a -> target, not certifying that the current code already matches it; see §4 -> for the gap. - -## 0. The one-sentence version - -`control_plane` should be a **thin planning-time shell** around -ASAPController's `asap-ir`/`asap-l2`/`asap-plan`/`asap-sketch` crates, -contributing exactly two things ASAPController doesn't ship: an L5 -physical-planner implementation (ASAPController has no `asap-physical` -crate — L5 is speculative there, real here) and deployment-specific L4 -extension points (`CostModel`, `Matcher`). `data_plane` should be a thin -**serving-time shell**, contributing exactly one thing: a `SummaryExecutor` -implementation. Planning and serving are two separate interfaces per -ASAPController's own `design.md` §"Serving-time execution" — this doc -treats them as two separate sections for the same reason. - -## 1. Planning vs. serving — the split ASAPController's own design draws - -```mermaid -flowchart LR - Q["query string"] --> P["PLANNING\nL1 → L2 → L3 → L4"] - P --> L4N["L4Node\n(symbolic plan)"] - L4N --> E["SERVING\nSummaryExecutor::execute"] - E --> V["answer"] -``` - -- **Planning** turns a query string into a plan. It has, by design, no - reference to what's actually materialized anywhere — it symbolically - picks a summary family and parameters from the query shape and an - accuracy target alone. -- **Serving** walks an already-decided plan against whatever is actually - materialized *right now* and produces an answer. Reality can diverge - from the plan in ways planning never sees — missing data, multiple - instances needing a merge, instances that disagree on parameters — - which is why serving needs its own error vocabulary, distinct from - planning's. -- In this deployment, `control_plane` runs **in-process** with - `data_plane` (same binary), so calling `control_plane`'s planning entry - point from `data_plane` at request time is a same-binary library call, - not a network hop or a second planning implementation living in - `data_plane`. That's a deployment-specific convenience this topology - affords, not something ASAPController assumes of every deployment. - -## 2. Planning-time — `control_plane`'s job - -### L1 — query language - -ASAPController's L1 interface (no shared trait; converging free functions): - -```rust -pub fn lower_promql(query: &str, accuracy: AccuracyTarget) -> Result; -pub async fn lower_sql(query: &str, catalog: &SqlCatalog, accuracy: AccuracyTarget) -> Result; -``` - -**Target: `control_plane` calls `asap_frontend_promql::lower_promql` directly.** -No local PromQL parser. SQL support (currently 0% adopted in this repo) is -`asap_frontend_sql::lower_sql` the same way, gated behind whatever schema -catalog this deployment can supply (`SqlCatalog` — see L2 below; PromQL has -none, SQL needs a real one). - -### L2 — logical plan / schema binding - -```rust -pub trait SchemaCatalog { fn columns_for(&self, source: &str) -> Option>; } -pub struct Binder; -pub fn convert_root(legacy: &QueryExpr, accuracy: &AccuracyTarget) -> Result; -``` - -**Target: no `control_plane`-local `SchemaCatalog` for PromQL** — this -deployment's metrics have no real catalog (a metric's label set is only -knowable from what a query references), so the default `Binder::default()` -(`UsageDerivedCatalog`) is correct as-is, same as ASAPController's own -PromQL front end uses. `convert_root` is called *inside* -`lower_promql`/`lower_sql` already — `control_plane` never calls it -directly. - -### L3 — intent algebra - -The canonical `QueryExpr`/`AggIntent`/`Schema` vocabulary (see PR #169's -`l3-intent-algebra.md` diff for the full ~30-variant `AggIntent` list this -deployment should treat as canonical, not its own subset). - -**Target: `control_plane` has zero local L3 type definitions.** Every -`QueryExpr`/`AggIntent`/`Schema` reference is `asap_ir::intent_algebra::*`, -used directly — no wrapper types, no local re-derivation of accuracy-math -helpers (`hll_accuracy`, `default_cardinality`, etc. — those already live -in `asap_ir` and should be called there, not duplicated locally). - -**One acknowledged, deliberate exception**, matching ASAPController's own -design rule ("core treats `Extension` opaquely; the owning deployment -defines and interprets `payload` itself"): this deployment's `Frequency` -capability (`AggIntent::Extension{ext_kind: "frequency", ..}` — per-series -sample-count estimation via CMS/CountSketch, with no upstream equivalent) -is genuinely deployment-specific vocabulary layered *on top of* the -canonical `AggIntent`, not a competing definition of it. It's implemented -as free functions (`frequency()`/`as_frequency()`) wrapping the foreign -`AggIntent` type (Rust's orphan rules require this), not a fork of the -type itself. - -### L4 — cost-aware binding + summary-bound IR - -```rust -pub fn implement_tree_with(expr: &QueryExpr, cost_model: &dyn CostModel) -> Result, ImplementError>; -pub trait CostModel { - fn rank_candidates(&self, intent: &AggIntent, candidates: &[SummaryKind]) -> Vec; - fn size_params(&self, kind: SummaryKind, intent: &AggIntent, eps: f64, delta: f64) -> SummaryParams { .. } - fn realize_extension(&self, ext_kind: &str, payload: &serde_json::Value) -> Implementation { .. } - fn readout_extension(&self, ext_kind: &str, payload: &serde_json::Value, col: &ColumnRef) -> SketchQuery { .. } -} -``` - -**Target: `control_plane` owns exactly one `CostModel` impl and nothing -else at this layer.** `implement_tree_with` + the `L4Node`/`SummaryExpr` -IR are pure ASAPController types, used directly, not wrapped. The -`CostModel` is the single deployment-specific extension point this layer -grants: - -- `rank_candidates` — this deployment's accuracy-bound-driven family - preference (KLL vs. DDSketch for quantiles, etc.). -- `size_params` — real parameter sizing from an accuracy target, not the - crate's built-in default formulas (this deployment has its own - calibrated cost table). -- `realize_extension`/`readout_extension` — the two hooks that make the - `Frequency` extension (§ above) actually work: turning - `Extension{"frequency", ..}` into a real `Implementation::Summary{Cms/CountSketch,..}` - and answering a readout against it. This is *the* mechanism, not a - workaround — ASAPController's `CostModel` trait exists specifically so a - deployment can plug in exactly this kind of opaque-extension realization - without touching `asap-plan`'s binding pass. - -No local `PhysicalExpr`/`L4Plan` wrapper type is needed for the IR itself -— `Rc` is the L4 output, used directly by L5 below. (A thin L5 -*placement* wrapper is still legitimate — see next section — but it should -wrap `Rc`, not reinvent the L4 algebra inside it.) - -### L5 — physical plan (control_plane's real, load-bearing contribution) - -ASAPController has **no `asap-physical` crate** — confirmed: `crates/` on -`main` is `{ir, l2, sketch, plan, frontend-promql, frontend-sql, lower, -e2e}`, no fifth entry. L5's interface section in `l5-physical-plan.md` is -explicitly marked speculative — "a concrete target to design against, not -an API to depend on." **This is the one layer `control_plane` should own -in full**, implemented *against* that speculative shape so a future real -`asap-physical` crate absorbs it with minimal churn, not built to some -other, incompatible pattern: - -```rust -pub trait PhysicalPlanner { - type Topology: TopologyDescriptor; - type Output; - fn lower(&self, l4: Rc, t: &Self::Topology) -> Result; -} -pub trait TopologyDescriptor { - fn stages(&self) -> &[StageDescriptor]; - fn edges(&self) -> &[StageEdge]; -} -pub struct StageAllocator; -impl StageAllocator { - pub fn allocate(&self, l4: Rc, topology: &T, c: &DeploymentConstraints) - -> Result, PlanError>; -} -``` - -Concretely, for this deployment: - -- **`Topology`**: a 3-stage topology (`edge` / `gateway` / `backend`) — - this repo's existing `physical::colored_dag` machinery is structurally - the right shape (`StageId`, per-stage allocator, per-stage emitter); - it should be read as *this deployment's* `TopologyDescriptor` + - `StageAllocator` + `PhysicalPlanner` impl, not a bespoke thing outside - ASAPController's pattern. -- **`Executor`**: an OpAMP-addressed agent/gateway process or an HTTP-addressed - backend — `address: ExecutorAddr` maps onto this repo's existing - OpAMP-connection / backend-HTTP-endpoint machinery. -- **Output** (`PhysicalPlanner::Output`): per-executor configuration — - OpAMP `RemoteConfig` (OTel-collector YAML) for edge/gateway executors, - `StreamingConfig` JSON POST for the backend executor. This is exactly - what `emit::{emit_edge_yaml, emit_gateway_yaml, emit_backend_streaming_config_json}` - already produce — L5's job is *stage allocation + emission format*, and - that's genuinely this repo's own concern, not something to strip out or - route through ASAPController. - -**This is the load-bearing implication of "no `asap-physical` crate -exists": `control_plane`'s physical-planning code is not legacy debt to -retire — it's the actual product.** Everything upstream of it (L1-L4) -should shrink toward zero local code; L5 should not. - -## 3. Serving-time — `data_plane`'s job - -```rust -pub trait SummaryExecutor { - type Handle: Clone; type State; type Value; type Error; - type GroupKey: Clone + Ord + Default; - fn find_candidates(&self, summary: &SummaryKind, params: &SummaryParams, col: &ColumnRef, - reduction: &Reduction, child: &L4Node) -> Result, Self::Error>; - fn fetch_state(&self, handle: &Self::Handle) -> Result; - fn merge_states(&self, states: Vec) -> Result; - fn readout(&self, state: &Self::State, query: &SketchQuery) -> Result; - fn logical(&self, expr: &QueryExpr) -> Result; -} -pub fn execute(node: &L4Node, exec: &E) -> Result, ExecError>; -``` - -**Target: `data_plane` implements `SummaryExecutor` once, and that -implementation is the sole live serving path.** `execute()` (upstream, -not deployment code) owns every *structural* rule generically — which -nestings are valid, what must agree for a merge to be legal, how deep -recursion composes. The deployment supplies only storage, summary math, -and readout, through the five trait methods: - -- `find_candidates` — resolve `(SummaryKind, SummaryParams, group_by)` - against this deployment's `SketchStore` index, using `reduction` - (`Reduce(GroupKeys)` vs. `PerEntity`) as the actual, structural signal - for grouping — **not** inferred from an empty key list (this is where - the "empty `by` is ambiguous" problem this repo's own - `data_plane/docs/l4node-plan-executor-design.md` documents gets solved - for real: `reduction` carries the distinction `L4Node` alone doesn't). - Family-compatibility inside this method's own matching logic is exactly - what `sketch_algebra::matcher::SummaryFamilyMatcher` (the `asap_plan::Matcher` - impl) exists for. -- `fetch_state`/`merge_states` — decode + merge sketch/exact-accumulator - state from storage; this repo's existing per-family merge logic - (`AggregateCore::merge_with`, the sketch-family merge helpers) is the - right home, generalized to run through this one trait method instead of - as one-off special cases. -- `readout` — evaluate a `SketchQuery` against merged state; this repo's - existing per-sketch-family readout code is the right home. -- `logical` — evaluate a raw, unrewritten subtree directly (the - `SummaryExpr::Logical` escape hatch) — this deployment's fallback path - for anything that reaches serving time without a summary decision. - -**`storage_engines/sketch_db/query/sketch_reducer.rs` and -`asap_query_engine/shadow_compare.rs` are retired.** Neither was any more -"ground truth" than `SummaryExecutor` itself — `shadow_compare.rs`'s job -(validate `SummaryExecutor` against the legacy reducer before cutover) -was done once the cutover landed (Part A, #427), and keeping the legacy -reducer around after that only meant two independently-planned answering -mechanisms could silently disagree with each other, not that either was -more trustworthy. Shapes `SummaryExecutor` self-excludes before binding -(`rate()`/`irate()`, `topk(K, sum by(...)(rate(...)))`, keyed-CMS -point-estimate, and the outer-exact/summary composition gaps below) now -fail over to archive directly — there is no legacy fallback left, by -design, not because a rollout step is still pending. - -## 4. What this means for current code — gap against this target - -| Layer | Target | Current gap | -|---|---|---| -| L1 | `asap_frontend_promql::lower_promql` called directly; no local parser | `query_parser::parse_query_expr_canonical`/`parse_query` call `lower_promql` directly; `query_parser/promql.rs` (the local parser, ~1187 lines) is deleted. No reconciliation pass — classification (e.g. bare selectors no longer implying `Aggregate{Sum}`) follows `asap-l2`'s lowering as-is. **Closed** (#428). | -| L2 | `Binder::default()` / `convert_root` used via L1, no local schema logic | Already true in substance — `intent_algebra/{binder,column_resolution}.rs` are thin re-export shims. **Effectively closed.** | -| L3 | Zero local `QueryExpr`/`AggIntent`/`Schema` definitions | Already true — `intent_algebra/{agg_intent,query_expr,relational,schema,expr_ir}.rs` are thin re-export shims with only genuinely-local residues (`Frequency` extension helpers, `PerPartitionWrap`, PromQL-ergonomic `LabelFilter`). `intent_algebra/lower.rs` (~1000 lines) remains real local code — deliberately, for two documented reasons with no ASAPController equivalent (multi-agg fusion, the windowed-Count-as-Frequency heuristic). **Effectively closed modulo `lower.rs`'s two documented exceptions.** | -| L4 | One `CostModel` impl; `Rc` used directly | `sketch_algebra::cost_model::ControlPlaneCostModel` + `sketch_algebra::lower::bind_query_expr` (delegating to `implement_tree_in_with`) already match this shape. `sketch_algebra::matcher::SummaryFamilyMatcher` is the `Matcher` impl this section's serving-time §3 depends on. **Effectively closed** — `PhysicalExpr`/`L4Plan` is a thin, acceptable L5-placement wrapper around `Rc`, not a competing L4 algebra. | -| L5 | Full local `PhysicalPlanner`/`TopologyDescriptor`/`StageAllocator` impl | `physical/colored_dag/*` + `emit/*` already implement this shape structurally, just not against the trait names above (no literal `PhysicalPlanner` trait exists in this repo — the free functions/structs are the de facto impl). Low-priority gap: naming/trait-alignment, not missing functionality. | -| Serving | Single `SummaryExecutor` impl is the live path | **Closed.** `data_plane`'s `summary_executor.rs` implements the trait fully and is the default-on, *sole* live path (`ASAP_SUMMARY_EXECUTOR_LIVE` default flipped from off to on, #427; the legacy `sketch_reducer.rs` and diagnostic `shadow_compare.rs` are both retired). Shapes it self-excludes before binding (`rate()`/`irate()`, `topk(K, sum by(...)(rate(...)))`, keyed-CMS point-estimate) and shapes it structurally can't realize yet (composed exact/summary aggregation in either nesting order — [ASAPController#171](https://github.com/ProjectASAP/ASAPController/issues/171), e.g. `max/avg by (zone) (quantile_over_time(...))`) fail over to archive directly, with no local workaround. | - -**Net reading**: L1–L4 and the serving-time cutover are all now at -target. The earlier instinct that "`intent_algebra`/`sketch_algebra` -should be unnecessary once connected to ASAPController" is correct and -largely *already true* for L2–L4; L1 has since closed the same way -(#428), and the serving-time cutover is fully done (#427) — `sketch_reducer.rs` -and `shadow_compare.rs` are both retired, not just superseded. The -remaining serving-time gaps (rate/topk-over-rate/keyed-CMS, -ASAPController#171's composed exact/summary shapes) are genuine upstream -L4 limitations tracked in ASAPController, not something this repo routes -around locally — same category as the already-tracked `TopK { accuracy: -Exact }` gap (ASAPController#151). L5 should **not** shrink — it's this -deployment's own, permanent responsibility per ASAPController's own "no -`asap-physical` crate" status; its only remaining gap is the low-priority -naming/trait-alignment noted above. - -## 5. Open questions (carried from `data_plane/docs/l4node-plan-executor-design.md`, mostly resolved) - -Listed here because both docs describe the same target and shouldn't -drift into two different pictures of what's still open: - -1. **Grouping ambiguity for empty, sketch-family `by`. RESOLVED.** PR - #169's `l3-intent-algebra.md` interface section resolves this at the - type level — `Reduction::{Reduce(GroupKeys), PerEntity}` is exactly - the upstream IR signal this repo's design doc flagged as missing. - `data_plane`'s `find_candidates` implementation (`summary_executor.rs`'s - `resolve_group_key`) already branches on the real `Reduction` value, - not an empty-key heuristic — confirmed by reading the current code, - not assumed. This was the stated blocker for the serving-time cutover - in §3/Part A; it's why that cutover was safe to default-on already. -2. **Outer-fold family of gaps** (`topk(K, sum by (...) (rate(m[r])))`, - stacking an outer exact statistic on a sketch/exact-agg readout) — - still open, still a cross-repo IR design question per the original doc. -3. **Sizing drift** — how often a freshly-planned `SummaryParams` fails to - match what's actually registered — still only measurable empirically - once serving-time re-planning runs against real traffic. diff --git a/control_plane/docs/design.md b/control_plane/docs/design.md deleted file mode 100644 index 5dd9d097..00000000 --- a/control_plane/docs/design.md +++ /dev/null @@ -1,1888 +0,0 @@ -# ASAPController — design - -> **Mirror.** This file mirrors `docs/design.md` from -> [`ProjectASAP/ASAPController#docs/workload-plan-and-queryspec`](https://github.com/ProjectASAP/ASAPController/tree/docs/workload-plan-and-queryspec/docs) -> as of 2026-05-05. -> -> ASAPController upstream is currently doc-only. The actively-running -> controller implementation lives next to this doc in -> [`controller/src/`](../src/) — the file you're reading describes the -> **target architecture** the implementation is converging toward. -> Where this doc and the current `analyzer::QuerySpec` shape diverge, -> the doc is the design intent. - -Target repo: **`github.com/ProjectASAP/ASAPController`** (currently doc-only). - -Merges three existing codebases into one: - -1. **`DataCollector/controller`** (Rust) — service-shaped; end-to-end data lifecycle planner (collection → transmission → storage → analytics query); OpAMP + HTTP; SLA-driven replanning loop. -2. **`ASAPQuery[-backend]/asap-planner-rs`** (Rust) — CLI-shaped; analytics-query-only planner; YAML-in → YAML-out. Two divergent copies today. -3. **`asap-fusion`** (Rust) — library-shaped; DataFusion operator-level rewrite rules with sketch awareness; multi-query batch fusion is aspirational, executor is a thin wrapper. - -## 1. Goals - -1. **One repo, one workspace.** One `Cargo.toml` at root, one place to file issues, one release cadence. -2. **Common core, pluggable deployment models.** The three existing codebases each solve a *different* planning problem. Factor out what's actually shared; keep the deployment-model-specific parts as crates so each deployment model can evolve independently. -3. **Extensible for future deployment models** (4th, 5th, …). A new deployment model should land as a new crate that implements well-defined traits — not by touching the core or the runtime. -4. **Preserve the two deployment shapes:** - - **Service**: long-running HTTP+OpAMP process that accepts live `QuerySpec`s, replans on SLA violations, pushes configs to agents/backends. (DC controller today.) - - **CLI**: one-shot "read workload YAML → emit `streaming_config.yaml` + `inference_config.yaml`". (`asap-planner-rs` today.) - Both should be thin shells over the same core. -5. **No regression in today's wire contracts.** `POST /api/v1/streaming-config` to ASAPQuery-backend and OpAMP push to agents must keep working byte-for-byte through the migration. The backend's capability-miss callback `ControllerClient.create_plan` must keep working. -6. **Sketch is a primitive, not a mandate.** The optimizer selects among physical alternatives for each logical operator — e.g. `HashJoin` / `SortMergeJoin` / `SketchJoin`, or `SortAgg` / `HashAgg` / `SketchAgg` — the same way a traditional DB optimizer picks a join algorithm. A plan may come back with zero sketch operators when an exact path Pareto-dominates for the query's accuracy target. Sketches are one primitive class the optimizer can reach for; the framework does not privilege them. - -## 2. Non-goals - -- **Not** redesigning the `Plan` IR end-to-end. DC's `algebra/` + fusion's `translator/optimizer/executor` stay where they are semantically; we merge them into a shared `Plan` crate with a clean trait seam, not a green-field rewrite. -- **Not** unifying DataFusion's `LogicalPlan` with DC's custom algebra at the type level. Those are different universes. Deployment models own their IR; core owns the *staged dispatch contract*. -- **Not** in-scope: changing the existing wire protocol between controller and backend. ASAPQuery-backend keeps consuming `streaming_config.yaml` and keeps responding on `/api/v1/plan`. - -## 3. The organizing spine: DC controller's 5-layer pipeline - -DataCollector/controller already documents its query→sketch translation as a 5-layer pipeline (`DataCollector/controller/docs/query-to-sketch-translation.md`). That's the spine of this merger: - -| # | Layer | What it does | Today's locations | -|---|-------|--------------|-------------------| -| 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql}.rs` (each parses straight to the L2 relational tree below); asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | -| 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/intent_algebra/relational.rs` — the L2 relational `QueryExpr` tree the `query_parser` front ends emit (one shared tree for PromQL + SQL today; a per-language split is future work); asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | -| 3 | **Intent algebra** | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/intent_algebra/{agg_intent,query_expr,schema,lower,cse}.rs` (canonical L3) + `controller/src/intent_algebra/lower.rs` (the L2→L3 converter: folds the `relational` L2 tree straight to the canonical IR, single-statistic sketchable `Aggregate` fusion included — no intermediate fused L3 IR); asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4**. *Refactor 2026-05 absorbed `controller/src/algebra/expr.rs` into `intent_algebra/relational.rs`.* | -| 4 | **Sketch algebra + optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 (`QueryExpr`) and emit the sketch-bound IR (`SketchExpr`). ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library + the sketch-bound IR `core::sketch_algebra::SketchExpr`; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/sketch_algebra/` (IR + binding rules) + `controller/src/optimizer/{engine,trait_def,baseline,rules/,cost/}.rs` (rule engine + cost models); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator`. *Refactor 2026-05 absorbed `controller/src/algebra/optimizer.rs` (→ `optimizer/engine.rs`) and `controller/src/planner/{cost_model,delta_cost_model,online_cost_model,pareto,tco,wire_cost,rules,baseline_planner}.rs` (→ `optimizer/{cost/,rules/,baseline.rs}`).* | -| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; deployment models supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/physical/{allocator,planner,plan,sketch_catalog,stage_split,topology,colored_dag/}.rs` (allocator framework + sketch catalogue + typed three-stage colouring) + `controller/src/emit/{stage_config,otap,telegraf,agent,backend,asapquery_backend,precompute,trait_def}.rs` (per-deployment-model emitters + `PlanEmitter` trait) + `controller/src/pipeline.rs` (L1→…→L5 driver, formerly `analyzer.rs`); asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/`. *Refactor 2026-05 absorbed `controller/src/algebra/{physical,allocator,plan,directory}.rs` and `controller/src/planner/stage_split.rs` and the legacy `controller/src/stage_split/` framework into `physical/`; absorbed `controller/src/config/` into `emit/` + `workload.rs`; renamed `analyzer.rs` to `pipeline.rs`.* | - -**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every deployment model reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same intent-algebra IR (intent only, no sketch binding). - -**L4 and L5 also have substantial common infrastructure.** Initially we assumed deployment models owned L4/L5 wholesale; on closer inspection what's actually deployment-model-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes deployment models significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. - -### Sketch binding lives in L4, not L3 - -A key clarification after cross-checking the three source repos: **L3 is intent-only**. DC's `AggIntent` names *what* to compute (`Quantile(0.99, ε=0.01)`, `Cardinality(δ=0.001)`, …) without committing to a sketch type. Picking KLL vs DDSketch, CMS vs CMS-with-heap, parameter sizes — all of that is L4's job, driven by deployment constraints. - -More generally: **L1–L3 lower into a logical representation + `AggIntent`; L4 and L5 choose the concrete execution plan.** That choice is a standard physical-operator selection — `HashJoin` vs `SortMergeJoin` vs `SketchJoin`; `SortAgg` vs `HashAgg` vs `SketchAgg`. "Use a sketch" is one option among several; the same L4 rule framework that picks sketch parameters also picks between sketch and non-sketch operators when a rule is registered for the intent. This keeps the existing 5-layer split intact: nothing about the layering presupposes the output contains a sketch. - -Today: -- DC: correctly separated (L3 has `AggIntent`, L4 picks sketch via cost model). -- asap-fusion: correctly separated — `SketchConfigRule` at L4 fills `SketchConfig::NULL` with concrete `CountMinSketch{5,4096}` / `KLL{k=200,m=8}`. -- asap-planner-rs: **L3+L4 fused today**. `map_statistic_to_precompute_operator` jumps from `Statistic` straight to `AggregationType::DatasketchesKLL{k=200}` in one call. **Phase 4 splits this** — `Statistic → AggIntent` at L3, `AggIntent + DeploymentConstraints → AggregationType + SketchParams` at L4. - -### Intent vocabulary: DC's `AggIntent` is a superset; deployment models use subsets - -DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because language-flavored synonyms (`QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core is 9 (`Count, Sum, Min, Max, Quantile, TopK, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps directly: `Topk` keeps its own intent (heavy-hitter sketches like SpaceSaving / CMS-with-heap compute it as a single primitive, so the intent earns L3 visibility). Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that deployment models opt into. - -### Data-model support: both time-series and tabular - -ASAPQuery-backend / DC controller operate on time-series data (metrics + labels + timestamp); asap-fusion operates on tabular data (DataFusion `LogicalPlan` over relations) that may or may not be time-indexed. These two data models differ fundamentally in their leaf shape (`metric + labels + time` vs. `table + columns`), but they share everything above the leaf — filter semantics, aggregation semantics, sketches themselves. - -Core handles this with: -- **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Deployment models' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. -- **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Min`, `Max`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase` — both carry PromQL counter-reset semantics), or tabular-only (future additions for joins, correlated subqueries). -- **Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values; that stream can come from a time-series window or a table column, the sketch does not know or care. So `BindKllOnQuantile` and siblings work uniformly across both. - -Practically, this means: -- `deployment-model-asapquery` + `deployment-model-asaplifecycle` lower into `QueryExpr` with `Source::TimeSeries` leaves. -- `deployment-model-asapfusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). -- A hypothetical OLAP deployment model that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. - -See §6 `core::intent_algebra` for the concrete type sketches. - -### Scope: start single-query, grow into workload-aware - -The initial implementation can operate on **one query at a time** — L4 picks physical operators per query against per-query constraints, and `CostModel::workload_cost` degenerates to a sum of per-plan costs. This matches today's three source repos (all single-query planners) and is the minimum bar for parity during the migration. - -**Workload-awareness is an extension, not a rewrite.** When ≥2 queries are planned together, `workload_cost` credits shared sub-expressions (sketches, precomputed aggregates) so the planner can pick a plan for `q1` that lets `q2` read its output for free. Nothing in the L1–L5 spine changes — only the cost objective widens and the rule engine gains cross-plan visibility. See §6 `core::cost` and §13 future work. - -### L2 is mandatory; the tree shape is an evolvable contract - -Every deployment model must produce an L2 tree, even when the source language didn't originally come as one. asap-planner-rs's current approach (PromQL pattern catalogue → `IntermediateAggConfig`) skips L2; Phase 4 will reverse-engineer the five PromQL pattern shapes into a `PromqlLogicalPlan` tree so the L1→L2→L3 pipeline is uniform. - -A future deployment model whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. - -### System I/O contract — what the controller takes in, what it emits - -The controller is a **planner**, not an executor. It does not run queries; it decides where each piece of a query runs. The data plane (OTel collectors / ASAPQuery-backend / DataFusion `SessionContext`) runs them. - -**System input — what the controller takes in.** A `QueryWorkload` (one or more `QuerySpec`s) plus deployment context (available executors, their capabilities, current SLA targets, telemetry of recent violations). The `QuerySpec` carries the raw query string in its source language (PromQL / SQL / DataFusion / ElasticDSL) and the accuracy / latency / cost target. Workloads can arrive via four entry points (HTTP `POST /plan`, OpAMP capability-miss callback, YAML file for the CLI shell, query-log replay); they all normalise to `QueryWorkload` before L1. - -**Workload features that the controller cares about, beyond the queries themselves:** -- Execution model: **batch** (one-shot YAML, query-log replay) vs **streaming** (live pipeline that must keep producing results as data arrives). -- Data input source: time-series scrape, tabular relation, query log replay, etc. Drives `Source` variant choice in L3. -- Reuse opportunity: ≥2 queries planned together → `CostModel::workload_cost` credits shared sub-expressions. - -**System output — what the controller emits.** For each registered executor in the deployment, a sub-DAG of the optimized plan plus the configuration that lets that executor run it. Concretely: - -| Output | Consumer | Wire shape | -|---|---|---| -| Per-executor sub-DAG assignment | The executor (edge agent / gateway / backend / DataFusion session) | OpAMP `RemoteConfig` (OTel YAML) / `streaming_config.yaml` POST / rewritten `LogicalPlan` | -| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge / compute-from-raw over the precompute engine, …) | Implied by the per-executor configs; not a separate artifact | -| Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | - -**Premise behind the staged dispatch.** Two distinct concepts: - -- **Stage** (`StageId`) — a categorical *tier* in the data lifecycle (edge / gateway / backend / in-process). The topology declares which stages exist (3-stage / 1-stage / 0-stage). Stages are roles, not instances. -- **Executor** (`Executor`, defined in `core::physical::executor`) — a *concrete runtime instance* that occupies a stage. Carries `id`, `stage: StageId`, `capabilities`, and `address` (OpAMP agent / HTTP endpoint / in-process handle). One stage may have N executors — e.g. a 50-host edge fleet is 50 `Executor`s all with `stage = StageId("edge")`; a singleton backend is one `Executor` at `stage = StageId("backend")`. - -Every stage is in principle capable of running the entire query tree — all stages speak the same physical operators. The controller's job is to decide *which stage* runs *which sub-tree / sub-DAG* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by `StageId`, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring at stage granularity; the per-deployment-model `PhysicalPlanner` then materialises each stage's sub-DAG into one config per `Executor` at that stage (varying only by per-executor connection / identity details). The executor list comes from `DeploymentConstraints::executors()`. - -This is what makes the topology a *parameter* rather than an axis of code: the same `SketchExpr` plus a different `TopologyDescriptor` produces edge-only / 1-stage / 3-stage / 0-stage placements without rewriting the plan. - -**Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialize into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. - -This drives the core/deployment model split: - -- **`crates/core/`** owns all 5 layers of **shared infrastructure**: L1-3 end-to-end (parsers + lowering + intent-only IR), plus L4's rule engine driver + rule library + cost-model traits, plus L5's stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. -- **Each deployment model is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any deployment-model-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. - -## 4. Principles - -### P1. Core owns shared infrastructure across all 5 layers; deployment models own choices - -See §3. Core is NOT just types + trait stubs — it ships working parsers + lowering passes (L1-3), a rule engine + rule library + cost-model traits (L4 framework), and a stage allocator + physical-plan framework + sketch catalogue (L5 framework). What deployment models plug in is **which rules fire** (picking from core's library + adding their own), **deployment topology** (how many stages; DC=3, query=1, fusion=0), and an **emitter** for the output format. A deployment model that accepts all core's default L4 rules and uses `core::physical::single_stage_topology` is maybe 200 lines of code. - -### P2. Core has no I/O - -No HTTP, no OpAMP, no YAML, no Prometheus scrape, no `tokio::spawn`. Pure algorithms over in-memory data. This is what makes deployment models unit-testable without running the runtime. - -### P3. Runtime is a thin binary, deployment models are libraries - -The `asap-controller` binary is assembled from: runtime (HTTP/OpAMP/replanner/store) + N deployment model crates registered as plugins. Swapping deployment models is a build-time feature flag or a runtime registry entry. No deployment model may reach directly into another. - -### P4. One input boundary, one output boundary - -**Input**: everything that enters the controller (HTTP `QuerySpec`, Prometheus query log replay, YAML workload, capability-miss callback) normalizes into a single `QueryWorkload` type in core — a collection of `QuerySpec`s each feeding L1→L2→L3. - -**Output**: L5 emitters (OpAMP `RemoteConfig`, backend `StreamingConfig` POST, one-shot YAML file, rewritten DataFusion `LogicalPlan`) all implement a `PlanEmitter` trait. Deployment models supply emitter implementations; core doesn't know which emitters exist. - -This is the "extension point for future deployment models" — a new deployment model adds an L4 rule set + an L5 emitter and registers them. - -### P5. No feature-flag spaghetti - -If something must be optional (e.g. OpAMP for deployments that don't run OTel collectors), it's a separate crate, not a `cfg` block in core. - -## 5. Target repo layout - -> **Refactor 2026-05 — current state vs target.** The per-deployment-model crate -> split below (`crates/deployment-model-{asaplifecycle,asapquery,asapfusion}/`) -> is **deferred** until ≥2 deployment models actually ship. Today's -> implementation is a single `controller/` crate whose internal module -> structure mirrors design.md §5's modular split: -> -> | Target §5 path | Current single-crate path | -> |---|---| -> | `crates/core/query_language/` | `controller/src/query_parser/{promql,sql}.rs` (L1 parsers — emit the L2 tree directly) | -> | `crates/core/logical_plan/` | `controller/src/intent_algebra/relational.rs` (the L2 relational `QueryExpr` tree) | -> | `crates/core/intent_algebra/` | `controller/src/intent_algebra/` (with `relational` carrying the L2 relational IR and `lower` lowering it to the canonical L3 types) | -> | `crates/core/sketch_algebra/` | `controller/src/sketch_algebra/` | -> | `crates/core/optimizer/{engine,trait,rules,cost}/` | `controller/src/optimizer/{engine.rs,trait_def.rs,rules/,cost/,baseline.rs}` | -> | `crates/core/physical/{planner_trait,stage_allocator,topology,executor,sketch_catalog}/` | `controller/src/physical/{planner,allocator,plan,stage_split,sketch_catalog,topology,colored_dag/}.rs` | -> | `crates/core/pipeline/` | `controller/src/pipeline.rs` (single-file driver — was `analyzer.rs`) | -> | `crates/core/workload/` | `controller/src/workload.rs` | -> | `crates/core/emit/` | `controller/src/emit/` | -> | `crates/core/registry/` | `controller/src/deployment_model.rs` | -> | `crates/runtime/{http,opamp,monitor,replan,store,backend_client}/` | `controller/src/{main.rs,opamp/,monitor/,replan.rs,store/,backend_client.rs,metrics_exposer.rs}` | -> | `crates/deployment-model-asaplifecycle/` | folded into `controller/src/emit/{agent,backend,asapquery_backend,otap,telegraf,stage_config,precompute}.rs` + the three-stage topology in `controller/src/physical/colored_dag/` | -> -> The target multi-crate layout below remains the long-term goal but is **not a -> current migration**. The single-crate is structurally aligned to the §5 -> module boundaries so future extraction is largely `git mv` + `Cargo.toml` -> edits. - -``` -ASAPController/ -├── Cargo.toml # workspace -├── README.md -├── docs/ -│ ├── design.md # this file -│ ├── migration-plan.md # the companion file -│ ├── deployment-models/ # per-deployment-model design notes -│ └── adr/ # architecture decision records -├── proto/ -│ ├── asap_control.proto # Plan IR + QueryWorkload on the wire -│ └── opamp.proto # vendored from DataCollector -├── crates/ -│ ├── core/ # Shared infrastructure across all 5 layers; no I/O -│ │ ├── query_language/ # L1: per-language parsers — promql/, sql/, datafusion/, elasticdsl/ -│ │ ├── logical_plan/ # L2: per-language algebra tree (Aggregate/Window/Filter/…) -│ │ ├── intent_algebra/ # L3 IR: QueryExpr + AggIntent + Schema/HasSchema (intent only) -│ │ ├── sketch_algebra/ # L4 IR: SketchExpr (sketch-bound — kind + params committed) -│ │ ├── lower/ # L1→L2→L3 lowering passes (one entry per language) -│ │ ├── optimizer/ # L4 framework — produces SketchExpr from QueryExpr: -│ │ │ ├── engine/ # rule driver — fixed-point iteration, cycle detection, priority -│ │ │ ├── trait/ # OptimizerRule + RuleCategory (PushDown / Fusion / Elim / Bind) -│ │ │ ├── rules/ # shared rule library (e.g. sketch-binding rules; stream-vs-batch picker) -│ │ │ └── cost/ # CostModel trait + generic impls (memory budget, accuracy degradation) -│ │ ├── physical/ # L5 framework: -│ │ │ ├── planner_trait/ # PhysicalPlanner trait -│ │ │ ├── stage_allocator/ # generic topology-driven allocator -│ │ │ ├── topology/ # Topology descriptor types (edge/gateway/backend, single, zero) -│ │ │ ├── executor/ # Executor type — concrete runtime instance occupying a stage -│ │ │ └── sketch_catalog/ # candidate sketch types + parameter constraints -│ │ ├── pipeline/ # orchestrates L1→L2→L3→L4→L5, parameterized on deployment model -│ │ ├── workload/ # QueryWorkload wrapper around QuerySpecs -│ │ ├── emit/ # PlanEmitter trait — implemented by deployment model L5 -│ │ ├── registry/ # DeploymentModelRegistry, DeploymentModelId -│ │ └── telemetry/ # tracing macros, metric names (no exporter) -│ ├── runtime/ # service skeleton — HTTP, OpAMP, replanner -│ │ ├── http/ # axum surface — /plan, /replan, /metrics, /status -│ │ ├── opamp/ # WebSocket OpAMP server -│ │ ├── monitor/ # Scraper, Thresholds, Violation -│ │ ├── replan/ # Replanner — SLA + expiry triggers -│ │ ├── store/ # PlanStore, WorkloadStore -│ │ └── backend_client/ # HTTP client (pushes to ASAPQuery-backend, etc.) -│ ├── deployment-model-asaplifecycle/ # thin — picks rules, 3-stage topology, OTel/backend emitters -│ │ ├── rules.rs # L4 rule selection + DC-specific rules (stage-aware push-down) -│ │ ├── topology.rs # 3-stage: edge / gateway / backend -│ │ ├── cost.rs # deployment-model-specific cost impls — delta / online / pareto / tco -│ │ └── emit/ # OpAmpRemoteConfig + AsapqueryBackendConfig emitters -│ ├── deployment-model-asapquery/ # thin — rules, 1-stage topology, YAML emitters + query-log input -│ │ ├── rules.rs # L4 rule selection + sketch-binding rule (split from map_statistic_*) -│ │ ├── topology.rs # 1-stage: backend-only -│ │ ├── emit/ # StreamingConfig.yaml + InferenceConfig.yaml -│ │ ├── query_log/ # extra L1 input: Prometheus query-log replay -│ │ └── schema/ # PromQLSchema discovery from Prometheus (feeds L1) -│ ├── deployment-model-asapfusion/ # thin — DF-flavored rules, 0-stage, in-process emit -│ │ ├── rules.rs # L4 rules for DataFusion LogicalPlan (sketch-aware rewrites) -│ │ ├── topology.rs # 0-stage: in-process -│ │ ├── emit/ # rewritten DataFusion LogicalPlan -│ │ ├── executor/ # DataFusion SessionContext wrapper (library-mode execution) -│ │ └── sketch_support/ # asap_sketchlib-backed rewrites -│ ├── control-proto/ # generated from proto/ (tonic/prost) -│ └── testing/ # test fixtures + harness shared across deployment models -└── bin/ - ├── asap-controller/ # long-running service with all deployment models - │ └── main.rs # axum + OpAMP + replanner + registered deployment models - ├── asap-query/ # one-shot CLI for deployment-model-asapquery (what asap-planner-rs is today) - │ └── main.rs # clap — read workload YAML, emit two YAMLs - ├── asap-lifecycle/ # OPTIONAL: standalone service with only deployment-model-asaplifecycle - │ └── main.rs # slimmer image — no DataFusion, no query YAML emitters - └── asap-fusion-bench/ # OPTIONAL: benchmark harness over deployment-model-asapfusion - └── main.rs # criterion entry; used by researchers -``` - -**Per-deployment-model standalone binaries** are first-class. Each `bin//` is a thin shell that `use`s only the deployment model crates it needs — so `bin/asap-lifecycle/` doesn't pull `datafusion` into its dep tree, and `bin/asap-fusion-bench/` doesn't pull `axum`/`opamp`. Feature flags on the workspace root let you `cargo build -p asap-lifecycle` and get a minimal binary. - -### Why three deployment model crates today, not one monolithic `deployment-models/` - -Each deployment model has a different problem shape: - -| | **lifecycle** | **query** | **fusion** | -|---|---|---|---| -| Input | workload + live metrics | workload YAML / query log | DataFusion `LogicalPlan` | -| Decision unit | end-to-end staged pipeline | per-aggregation YAML | per-operator rewrite | -| Output | OpAMP OTel config + `StreamingConfig` YAML | `streaming_config.yaml` + `inference_config.yaml` | rewritten `LogicalPlan` | -| Trigger | QuerySpec, SLA violation, expiry | one-shot CLI invocation | a DataFusion session constructing a query | -| Cost model | accuracy × latency × $ staged | single-query accuracy/latency | operator selectivity / sketch feasibility | - -Mashing them into one crate means the union of all their dependencies (sqlparser + promql-parser + DataFusion + OpAMP proto + Prometheus client) bleeds into every downstream user. Separating them means a user who only needs `deployment-model-asapfusion` (an offline query-optimization benchmark, say) can depend on it without pulling OpAMP. - -## 6. Core crate details (layers 1–3 + driver) - -Core is not a trait-stubs library. It ships real L1/L2/L3 code lifted from DC's `controller/src/query_parser/` + `controller/src/algebra/` and exposes a small set of traits for L4/L5 plugin points. - -> **§6 / `core::*` paths vs current single-crate paths.** §6 below names -> targets like `core::query_language`, `core::sketch_algebra`, -> `core::optimizer` from the design.md §5 target layout. Today's -> single-crate (refactor 2026-05) is structurally aligned to those names -> via a 1:1 mapping. The lookup table: -> -> | `core::*` reference in §6 | Current single-crate path | -> |---|---| -> | `core::query_language` | `controller/src/query_parser/{promql,sql}.rs` | -> | `core::logical_plan` | `controller/src/intent_algebra/relational.rs` (the L2 relational `QueryExpr` tree) | -> | `core::intent_algebra` | `controller/src/intent_algebra/` (canonical `query_expr` / `agg_intent`) + `controller/src/intent_algebra/relational.rs` (the L2 relational IR) | -> | `core::sketch_algebra` | `controller/src/sketch_algebra/` | -> | `core::lower` | per-layer: `controller/src/intent_algebra/{lower,lower}.rs` + `controller/src/sketch_algebra/lower.rs` | -> | `core::optimizer::engine` | `controller/src/optimizer/engine.rs` | -> | `core::optimizer::trait` | `controller/src/optimizer/trait_def.rs` (placeholder) | -> | `core::optimizer::rules` | `controller/src/optimizer/rules/` | -> | `core::optimizer::cost` | `controller/src/optimizer/cost/` | -> | `core::physical::planner_trait` | `controller/src/physical/planner.rs` (legacy `physical_plan_to_staged` impl pending trait extraction) | -> | `core::physical::stage_allocator` | `controller/src/physical/{allocator,stage_split,colored_dag/allocator}.rs` | -> | `core::physical::topology` | `controller/src/physical/topology.rs` | -> | `core::physical::executor` | not yet materialised (placeholder absent) | -> | `core::physical::sketch_catalog` | `controller/src/physical/sketch_catalog.rs` | -> | `core::pipeline` | `controller/src/pipeline.rs` (single file — was `analyzer.rs`) | -> | `core::workload` | `controller/src/workload.rs` | -> | `core::emit` | `controller/src/emit/` | -> | `core::registry` | `controller/src/deployment_model.rs` | -> | `core::telemetry` | `controller/src/metrics_exposer.rs` (out-of-core today — drives Prometheus metrics) | -> -> Every §6 reference below should be read with this table as the -> concrete pointer. Where §6 says "lifted from DC's -> `controller/src/algebra/`", read "structurally aligned to -> `intent_algebra/relational.rs` + `physical/` + `optimizer/engine.rs` -> per the §16 ADR." - -### `core::query_language` — Layer 1 - -Per-language parsers, one module each: - -- `promql/` — wraps `promql-parser`. Input: `&str` PromQL. Output: `PromqlAst`. -- `sql/` — wraps `sqlparser`. Input: `&str` SQL. Output: `SqlAst`. -- `datafusion/` — wraps DataFusion's own parser. Output: `DfAst`. -- `elasticdsl/` — wraps `elastic_dsl_utilities`. Output: `EsAst`. - -Each returns a language-flavored AST type. No sketch awareness. - -> **Implementation status.** L1 + L2 are realised concretely, without -> the `Language`-trait / `LanguageAst` indirection the target design -> sketches. L1 lives in `controller/src/query_parser/{promql,sql}.rs` — -> each parser walks its language's AST and emits the **L2 relational -> tree directly** (`intent_algebra::relational::QueryExpr`: `Aggregate` -> / `Window` / `Filter` / `Join` / `Sort` / `Limit` / …). That tree -> *is* L2. `query_parser::parse_query_expr_canonical` then lowers it to -> the canonical L3 IR via `intent_algebra::lower` -> (sketch-fusion + the Binder folded in); `parse_query` projects the -> flat `ParsedQuery` summary the legacy analyzer / pipeline consume. -> -> An earlier `Language`-trait + `LanguageAst` + `LanguageLogicalPlan` -> scaffold (a per-language enum mirroring the target design) was built -> ahead of its call site, never wired into the pipeline, and drifted -> out of sync — its "L2" actually held the L3 tree. It was removed; the -> per-language `Language`-trait extension point gets rebuilt when a -> second real language backend (SQL beyond the current direct parser, -> DataFusion, ElasticDSL) actually needs it. - -### `core::logical_plan` — Layer 2 - -A **per-language** algebra tree — one `enum LogicalPlan` per language. Preserves language-specific semantics (PromQL instant vs range vector, SQL window frames, Elastic buckets) that would be lossy to collapse this early. Types are symmetric: `Aggregate { AggFunc }`, `Window`, `Filter`, `Sort`, `Limit`. No sketch names yet. - -### `core::intent_algebra` — Layer 3 - -The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `HistogramQuantile`, no `PromQLSubquery` — those are L2 PromQL nodes), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP deployment models) via a `Source` sum type inside `QueryExpr::Scan`. - -#### Design rules for L3 - -1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). The exception is when an "intent" is its own physical primitive: heavy-hitter top-k is served by sketches (SpaceSaving, CMS-with-heap) as a single operation, so `AggIntent::TopK` is a first-class intent at L3 — distinct from the generic `Sort + Limit` operator pair, which still appears in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). -2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artifact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. -3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. -4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. -5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding context. The shape is a DAG, not a tree, because **a producer node can have multiple parents that share its precomputed intermediate state** — instead of recomputing the same sub-expression once per consumer, the consumers fan in to a single node and read its output. Three sources of fan-in: - 1. **Explicit, in-query.** SQL CTEs (`WITH name AS (expr) SELECT ... FROM name JOIN name AS n2 ON ...`) and PromQL recording rules name a sub-expression and reference it N times; each reference becomes a `QueryExpr::Ref(name)` parent of the named producer. - 2. **L4 reuse rules.** When two queries planned together both need (e.g.) a p99 quantile of the same series, the optimizer can introduce a shared sketch node whose output feeds both — even though neither query author wrote a CTE. - 3. **L4 stage / shard structure.** A pre-aggregate computed once on the edge can feed multiple downstream gateway-stage operators. - - `CostModel::workload_cost` credits a shared producer's build cost once across all consumers, which is what makes reuse a Pareto win. Tree IRs lose this — they have to duplicate the producer for every consumer. - -```rust -pub enum QueryExpr { - // ── Base relations ──────────────────────────────────────────────────── - /// A metric stream / table / join. Outermost leaf. - Scan { source: Source, predicates: Vec }, - /// Reference to a CTE / let-binding by name; resolved at plan time. - Ref(String), - - // ── Filtering & projection ──────────────────────────────────────────── - /// σ — row-level filter (WHERE / PromQL label matchers). - Filter { child: Box, pred: Predicate }, - /// π — column projection (SELECT list). - Project { child: Box, cols: Vec }, - - // ── Aggregation (logical, intent-only) ──────────────────────────────── - /// γ + α — GROUP BY + aggregate intents, with optional HAVING. - /// `aggs` carry `AggIntent`; concrete sketch / non-sketch operator - /// is chosen by L4 and lives in the L4-extended IR (`SketchExpr`). - Aggregate { child: Box, by: Vec, - aggs: Vec, having: Option }, - - // ── Time / streaming windows ────────────────────────────────────────── - /// ψ — tumbling / sliding / session window over the time axis. Defines - /// the lifecycle (flush / reset bounds) of any aggregate in its sub-tree / sub-DAG. - /// PromQL `[5m]` and streaming windows lower here. SQL `OVER (...)` - /// analytic frames are a different node — see `WindowFunc` below. - Window { child: Box, kind: WindowKind, - size: Duration, slide: Option }, - - // ── Distributed-execution structure ─────────────────────────────────── - /// Partition the stream by key tuple (`GROUP BY` / PromQL `by (dims)`). - Partition { child: Box, keys: PartitionKeys }, - /// δ — SQL `DISTINCT` / row deduplication on `cols`. - Distinct { child: Box, cols: Vec }, - /// ⊕ — union of sub-results from independent stages or shards (the - /// exact-merge case). Sketch unions are a separate node in `SketchExpr` - /// because they carry sketch-family / params type constraints. - Merge { children: Vec }, - - // ── Joins (logical) ─────────────────────────────────────────────────── - /// Logical join. L4 picks the physical alternative — `HashJoin` / - /// `SortMergeJoin` / `SketchJoin` (e.g. KMV / theta-sketch / join-sample) - /// — based on selectivity, memory budget, and accuracy target. The - /// sketch-aware variant lives in `SketchExpr::SketchJoin`. - Join { kind: JoinKind, left: Box, right: Box, - pred: Option }, - - // ── Set operators ───────────────────────────────────────────────────── - /// UNION / INTERSECT / EXCEPT, with or without ALL. - SetOp { kind: SetOpKind, all: bool, - left: Box, right: Box }, - - // ── Ordering & limiting ─────────────────────────────────────────────── - /// Generic order-by — survives L3 for non-heavy-hitter cases - /// (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`). - Sort { child: Box, keys: Vec }, - /// `LIMIT n OFFSET k`. The heavy-hitter shape (`ORDER BY count DESC - /// LIMIT k`, PromQL `topk(k, …)`) is recognised at L1→L2→L3 lowering - /// and produces `AggIntent::TopK` rather than generic `Sort + Limit`, - /// so heavy-hitter sketches (SpaceSaving, CMS-with-heap) bind on the - /// intent. Generic `Sort + Limit` flows through unchanged. - Limit { child: Box, n: u64, offset: u64 }, - - // ── Subquery / CTE ──────────────────────────────────────────────────── - Subquery { child: Box, alias: String }, - /// SQL `WITH name AS (expr) IN body`; lowering target for PromQL - /// recording-rule bindings. - LetBinding { name: String, expr: Box, body: Box }, - - // ── Analytic (OVER) window functions ────────────────────────────────── - /// SQL `OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)`. - /// Distinct from `Window` above — that is a streaming/tumbling window - /// over the time axis; this is an analytic frame over already-grouped rows. - WindowFunc { child: Box, func: WindowFuncKind, - partition_by: Vec, order_by: Vec, - frame: Option }, - - // ── Binary composition ──────────────────────────────────────────────── - /// Arithmetic / comparison / boolean composition between two relational - /// sub-expressions (PromQL binary ops including `and`/`or`/`unless`, - /// SQL boolean composition). - BinaryOp { op: BinaryOpKind, lhs: Box, rhs: Box, - vector_match: Option }, -} - -pub enum WindowKind { Tumbling, Sliding, Session } - -pub enum Source { - /// Time-series input — deployment-model-asapquery / deployment-model-asaplifecycle shape. - TimeSeries { metric: MetricRef, time: TimeRange, labels: LabelFilter }, - /// Tabular input — deployment-model-asapfusion / future-OLAP shape. - Table { table_ref: TableRef, columns: Vec }, - /// Join over Sources composes leaf shapes recursively. - Join { left: Box, right: Box, on: JoinKey }, - // Future: WindowedStream, Subquery — added by deployment models that need them. -} - -pub enum DataModel { TimeSeries, Tabular, Any } - -impl Source { - pub fn data_model(&self) -> DataModel { /* … */ } - /// Output schema produced by this leaf (see "Schema flow"). - pub fn schema(&self, catalog: &SchemaCatalog) -> Schema { /* … */ } -} -``` - -#### What was removed during cleanup, and why - -| Removed | Reason | Replacement | -|---|---|---| -| `TopK { k, by }` *as a `QueryExpr` node* | A `QueryExpr`-level top-k operator collapses two distinct concepts: (a) the *intent* of "compute heavy hitters", which has its own sketch primitive, and (b) the generic operator pair `Sort + Limit`, which doesn't. Splitting them puts heavy-hitter logic at the right level | Heavy-hitter intent → `AggIntent::TopK` (L3, recognised by L1→L2→L3 lowering of `ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`); generic ordering+limit → `Sort + Limit` `QueryExpr` nodes (unchanged). Both retained — they describe different things | -| `WindowedAgg { intent, window }` | Equivalent to `Window` over `Aggregate`; redundant. Tumbling/sliding kind moves onto `Window::kind` | `Window { kind: …, … } → Aggregate { aggs: [intent] }` | -| `SketchAgg { intent, col }` | Sketch-bound; L3 must be intent-only | `Aggregate { aggs: [intent] }` at L3; L4 emits `SketchExpr::SketchAgg` | -| `JoinSketch { outer, inner, key }` | Sketch-bound physical alternative; L3 must be intent-only. Several papers describe sketch-of-join (KMV, theta, join-sample) — picking one is an L4 cost decision, not an L3 surface choice | `Join { … }` at L3; L4 emits `SketchExpr::SketchJoin` when a `Bind*OnJoin` rule fires | -| `HistogramQuantile { phi }` | PromQL artifact — consumes Prometheus's specific bucketed-histogram format. Language-specific | Lowers in PromQL L1→L2 to bucket reads + `Aggregate { aggs: [Quantile{q: phi, …}] }` over the bucket counts | -| `PromQLSubquery { range, resolution }` | Driver construct — asks the engine to evaluate the inner expression at N timestamps; not a single DAG node | Expanded by PromQL L1→L2 lowering into a set of independent `QueryExpr` instances, not preserved at L3 | -| `Dedup { col }` | Single-column-only spelling of SQL `DISTINCT` | Renamed to `Distinct { cols }`, generalised to N columns | - -#### Schema flow — every L3 edge carries a typed schema - -Every node has a derivable output schema given its input schemas and parameters. The DAG is type-checked: a `Filter` whose predicate references a column not in its child's output schema fails at plan time. - -```rust -pub struct Schema { - pub fields: Vec, - /// Index into `fields` for the time axis, if any. PromQL leaves carry one; - /// SQL leaves may or may not. - pub time_index: Option, - /// Optional metadata for reuse-aware planning. Each inner `Vec` is - /// a set of column indices that together uniquely identify rows; the outer - /// `Vec` allows multiple unique-key sets (e.g. primary key + another unique - /// constraint). - /// - /// **Populated by**: the per-node input/output spec (e.g. `Aggregate { by, .. }` - /// emits `unique_keys = [by]`; `Distinct { cols }` adds `cols`; `Project` - /// carries forward the retained columns; most other nodes pass through). - /// - /// **Consumed by**: `CostModel::workload_cost` only — the reuse-aware path - /// that credits shared sub-expressions across multiple queries. Single-query - /// plans, the `Bind*` rules, push-down rules, and L5 emitters do not read - /// this field. If the reuse path is deferred (see §13 future work), this - /// field is dead weight; it lives here so the metadata is available the - /// moment workload-aware planning lands without requiring an L3-wide - /// schema change. - pub unique_keys: Vec>, -} - -pub struct Field { - pub name: String, - pub dtype: DataType, // Int64 / Float64 / Utf8 / Map / … - pub nullable: bool, -} - -pub trait HasSchema { - fn input_schemas(&self) -> Vec<&Schema>; - fn output_schema(&self, inputs: &[&Schema], cat: &SchemaCatalog) -> Schema; -} -``` - -Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/intent_algebra/schema.rs`). Each row reads independently: every column position, type, and constraint is named explicitly rather than carried by a shorthand like `S` or `L` / `R`. - -| Node | Input schemas | Output schema | -|---|---|---| -| `Scan { source }` | none — leaf node | `source.schema(catalog)` — derived from the source's catalog metadata (TimeSeries metric labels + value + timestamp; or Table columns from `information_schema`; or recursive `Source::Join`) | -| `Ref(name)` | none — pointer node | the output schema of the `LetBinding` whose `name` matches; resolved at plan time | -| `Filter { pred }` | one input schema (the child's output) | the child's input schema unchanged — `Filter` is a row-level refinement; no columns added, removed, or re-typed | -| `Project { cols }` | one input schema | the input schema projected to `cols` — fields filtered and reordered to match `cols`; `time_index` and `unique_keys` carried over for retained columns | -| `Aggregate { by, aggs }` | one input schema | the `by` columns (carried verbatim from input) followed by one new column per entry in `aggs`, each named and typed by `AggIntent::output_type(input_field)`; `unique_keys = [by]` | -| `Window { kind, size, slide }` | one input schema, **must** contain a `time_index` field | the input schema extended with synthetic `window_id` and `window_start` / `window_end` metadata fields | -| `Partition { keys }` | one input schema | the input schema unchanged — logical-only marker; carries a sharding hint for L5's stage allocator | -| `Distinct { cols }` | one input schema | the input schema with `unique_keys` tightened to include `cols`; field types unchanged | -| `Merge` | N input schemas, all union-compatible (same field names, same types, same nullability, same `time_index` position if any) | the first input's schema (representative; checked union-compatible with the rest) | -| `Join { kind, pred }` | two input schemas — left and right children | the concatenation of left's fields and right's fields, minus columns that USING / NATURAL deduplicates; nullability widened on the OUTER side for outer joins | -| `SetOp { kind, all }` | two input schemas, must be union-compatible (as for `Merge`) | the left input's schema | -| `Sort { keys }` | one input schema; every field referenced in `keys` must be present in it | the input schema unchanged | -| `Limit { n, offset }` | one input schema | the input schema unchanged | -| `Subquery { alias }` | one input schema | the input schema with `alias` applied as the table alias to all field names | -| `LetBinding { name, expr, body }` | `expr` produces an intermediate schema bound to `name`; `body` consumes it (and any other in-scope bindings) | the `body`'s output schema | -| `WindowFunc { func, partition_by, order_by, frame }` | one input schema; every field in `partition_by` / `order_by` must be present | the input schema extended with one new column carrying the analytic-function output (named after `func`, typed per `func`) | -| `BinaryOp { op, vector_match }` | two input schemas — left and right operands. PromQL vector-match constraints (`on`/`ignoring` + `group_left`/`group_right`) govern label-set compatibility | for arithmetic/comparison `op`: the left input's schema with the value column re-typed to the result of `op`; for boolean `op` (`and`, `or`, `unless`): the left input's schema with a boolean value column | - -##### Implementation status - -Phase F (in `controller/src/intent_algebra/schema.rs` + `controller/src/intent_algebra/cse.rs`) lands the load-bearing consumer of `Schema::unique_keys`: - -- `cse_reuse_is_legal(producer_schema, consumer_count) -> Result<(), CseError>` — the gatekeeper. Two `QueryExpr::Ref` consumers may share a producer only when the producer's output schema has a non-empty `unique_keys` set and the consumer count is ≥ 2. This is the proof point that `unique_keys` is load-bearing — without it, the deduper conservatively refuses to share and reuse "drops on the floor" (this section, line ~1356). -- `dedupe_subtrees(roots) -> CseWorkloadPlan` — basic implementation of the workload-level CSE pass for the literal "≥2 root queries with identical sub-expressions" case (the batched-queries example, line ~1256). Detects shared `Aggregate` children, gates on `cse_reuse_is_legal`, hoists into a `LetBinding`. Richer detection (alpha-equivalence, schema-merge across compatible-but-not-identical shapes, recursive nested CSE) is downstream — Phase F lands the gate + the basic case so the cost-model side has something to credit. - -The full general CSE algorithm (the optimisation half) remains future work. - -#### DAG schema, DB schema, sketch catalog — three distinct metadata sources - -These three are sometimes conflated and shouldn't be. Only the first two are *schemas* (descriptions of stream / table shape); the sketch catalog is a *registry* of available primitives, not a description of a stream: - -| Source | Where it lives | What it describes | Who reads it | -|---|---|---|---| -| **DAG schema** | On every edge of the L3 / L4 / L5 DAG (`Schema` above) | Columns + types flowing between operators | L4 rules (selectivity estimation, push-down legality), L5 emitter | -| **DB / source schema** | The query target (Prometheus TSDB metric metadata, SQL `information_schema`, DataFusion catalog) | What metrics / tables / columns exist in the data plane, with their types and indexing | The **Binder** (below), via the `SchemaCatalog` interface (`core::intent_algebra::binder`) | -| **Sketch catalog** | `core::physical::sketch_catalog` (built at startup; static) | What sketches the runtime can build; what intents each one serves; mergeability, accuracy / confidence guarantees, supported aggregation keys, parameter ranges | L4 binding rules to choose a sketch for an `AggIntent`; L5 to instantiate the sketch | - -The **Binder** reads the **DB schema** to resolve symbols. L3 onward, every edge carries a **DAG schema** that is type-checked locally. L4 binding rules consult the **sketch catalog** to map an intent to a concrete sketch under the deployment's constraints. They are three separate inputs to three distinct decisions. - -#### The Binder — name resolution as an explicit pass - -Every mature query engine has exactly one explicit boundary where -symbolic column / table *names* are resolved against a schema source, -and everything downstream of that boundary is fully resolved: - -| Engine | Resolution pass | Resolved column identity | -|---|---|---| -| ClickHouse | `QueryAnalyzer` rewrites `IdentifierNode` → `ColumnNode` against `StorageSnapshot` | name + type + source pointer | -| Trino | `Analyzer` → `Analysis` side-table, against catalog `Metadata` | opaque, plan-local `Symbol` | -| RisingWave | `Binder` resolves against its `Catalog` | positional `InputRef { index, data_type }` | -| **ASAP control plane** | **`core::intent_algebra::binder::Binder`** | positional `ColumnId` (index into `Schema`) | - -Our canonical L3 IR already commits to positional column identity — -`Aggregate.by: Vec`, exactly RisingWave's `InputRef`. The -[`Binder`] is the pass that *produces* it: given a query tree, it walks -it, collects every referenced column / group-key name, and builds the -complete, **self-contained** [`Schema`] every `ColumnId` in the lowered -tree indexes into — the IR's own "RelationType" (Trino) / bind scope -(RisingWave). Resolution downstream is then **total** — it cannot fail -on a well-formed tree. - -The schema source is the **`SchemaCatalog`** seam: - -- The default `UsageDerivedCatalog` knows nothing — every schema is - derived purely from what the query references. This is the honest - state for the observability domain: metric label sets are open-ended - and data-dependent, there is no closed catalog to resolve against - (unlike a SQL `information_schema`). -- A registry-backed `SchemaCatalog` (a metric-schema registry) is future - work. Crucially, **the `Binder` pass does not change when it lands** — - only the catalog impl swaps. - -Placement: today the Binder runs at the L2→L3 (relational → canonical) -conversion boundary (`lower::convert_root` calls it). -Once the `relational` L2 IR is retired it moves into the `core::lower` L1→L2→L3 -passes proper — the `lower_*(ast, schema)` signatures below already -anticipate a schema parameter at that point. - -#### `AggIntent` — what to compute, not how - -```rust -pub enum AggIntent { - // Data-model-agnostic - Count { accuracy: AccuracyTarget }, - Sum, - Min, Max, - Quantile { q: f64, accuracy: AccuracyTarget }, - /// Heavy-hitter top-k. Distinct from generic `Sort + Limit` because a - /// dedicated sketch primitive (SpaceSaving, CMS-with-heap, Misra-Gries) - /// computes it as a single operation. L1→L2→L3 lowering produces this - /// when it recognises a heavy-hitter shape (`ORDER BY count DESC LIMIT k`, - /// PromQL `topk(k, …)`); other ordering+limit cases stay as - /// `QueryExpr::Sort + QueryExpr::Limit`. - TopK { k: usize, by: Vec, accuracy: AccuracyTarget }, - Cardinality { accuracy: AccuracyTarget }, - - // Time-series streaming derivatives — specific operations, not just - // "Sum / Count over a Window". `Rate` is the per-second average derivative - // computed with PromQL's counter-reset adjustment, not a generic windowed - // mean. Kept distinct because (a) they have counter-reset semantics that - // exact `Sum` does not, and (b) sketch backends specialised for - // derivatives (e.g. delta-set aggregator) bind on these intents directly. - Rate { window: Duration }, - Increase { window: Duration }, - - // Tabular / OLAP — added as deployment models demand - // CorrelatedSubqueryCount { … }, ApproxJoinCardinality { … }, -} - -impl AggIntent { - /// Which data-model this intent semantically requires. L4 rules - /// consult this to skip non-applicable intents (e.g. `Rate` over - /// a `Source::Table` is nonsense). - pub fn requires(&self) -> DataModel { /* … */ } - /// Output column type — used by L3 schema derivation for `Aggregate`. - pub fn output_type(&self, input: &Field) -> DataType { /* … */ } - /// Which sketch families in the catalog can serve this intent. - /// Read by L4 binding rules. - pub fn candidate_sketches(&self) -> &'static [SketchKind] { /* … */ } -} -``` - -**Why `TopK` *is* an intent (and `Sort + Limit` is not collapsed into it).** Heavy-hitter top-k has a dedicated sketch primitive — SpaceSaving / CMS-with-heap / Misra-Gries compute it in a single pass with sub-linear memory. L4 binding rules want to fire on the *intent* "give me the top-k frequent items" rather than on a syntactic shape, the same argument that makes `Quantile` an intent rather than a `Sort` + "pick the φ-th element" pattern. So `AggIntent::TopK` lives at L3. Generic `QueryExpr::Sort + QueryExpr::Limit` *also* survives, because not every order-by-limit query is heavy-hitter (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`); these have no sketch alternative and stay as generic operators. The canonical-form invariant is preserved because L1→L2→L3 lowering picks one or the other deterministically based on whether it recognises a heavy-hitter pattern. - -**Why no `QuantileOverTime` intent?** It duplicated `Quantile` over a `Window`. The window — its kind, its size, its slide — is fully captured by the surrounding `Window { … }` node; the quantile *operation* is the same regardless of whether the input was a windowed time-series or a row-grouped table. PromQL's `quantile_over_time(0.99, m[5m])` lowers cleanly to `Window{size=5m} → Aggregate{aggs:[Quantile{q=0.99}]}`. One intent halves the L4 rule surface (one bind rule per operation, not per language-flavor of an operation). - -**Why `Rate` and `Increase` survive that argument.** They are not "Sum / Count over a Window with a different name" — they include PromQL's counter-reset adjustment, which is a non-trivial transformation an exact `Sum` does not perform. They earn distinct intent variants because they parameterise different physical operators (delta-set aggregators bind on these intents directly). If a non-PromQL streaming language has the same notion (e.g. SQL `RATE() OVER (RANGE)`), it lowers to the same intent — the intent vocabulary names the operation, not the language. - -#### Implementation status - -Phase B (this PR) ships the L3 IR in `controller/src/intent_algebra/`: - -- `intent_algebra::AggIntent` — vocabulary above (`Count`, `Sum`, `Min`, `Max`, `Avg`, `Quantile`, `TopK`, `Cardinality`, `Frequency`, `Rate`, `Increase`). -- `intent_algebra::QueryExpr` — variant subset for the DC + PromQL deployment: `Scan`, `Window`, `Aggregate`, `LetBinding`, `Ref`. The remaining variants (`Filter`, `Project`, `Partition`, `Distinct`, `Merge`, `Join`, `SetOp`, `Sort`, `Limit`, `Subquery`, `WindowFunc`, `BinaryOp`) are deferred so each lands with a planner consumer rather than as dead code; adding them is purely additive. -- `intent_algebra::Schema` — typed schema flow with `unique_keys` (the load-bearing CSE-legality field, `cse_substitution_legal_only_with_unique_keys` test pins the invariant). -- `intent_algebra::lower_parsed_query` — `query_parser::ParsedQuery` → `QueryExpr` single-query lowering, exposed as a standalone function. The analyzer is **not** yet wired to emit `QueryExpr`; that wiring is the follow-up phase's job, kept separate so the IR rev and the consumer rev land independently. - -Follow-up phases: - -- **Phase C** — wire `Analyzer::analyze` to also produce a `QueryExpr` alongside `QueryWorkload`, populate `WorkloadPlan::roots` from the lowered roots. -- **Phase D** — workload-level CSE pass (`core::lower::workload::dedupe_subtrees`) that hoists shared sub-DAGs into `WorkloadPlan::bindings`, leaning on `Schema::unique_keys` for legality (the batched-queries example above). -- **Phase E** — L4 `SketchExpr` IR (`core::sketch_algebra`) + L4 binding rules. -- **Phase F** *(this PR)* — `CostModel::workload_cost` + `Schema::unique_keys`-based CSE legality. See the per-section "Implementation status" notes under `core::cost` and the "Schema flow" table below for what shipped / what's deferred. - -### `core::sketch_algebra` — Layer 4 IR (`SketchExpr`) - -L4 binding rules consume L3 `QueryExpr` (in `core::intent_algebra`) and produce `SketchExpr` (in `core::sketch_algebra`). This is the IR L5 emitters consume. The two-IR split — intent-only L3 (`QueryExpr`) and sketch-bound L4 (`SketchExpr`), in two separate modules — gives L4 rule application a clean type signature: `fn apply(&QueryExpr, &Constraints) -> Option`, and the boundary cannot be silently violated. - -```rust -pub enum SketchExpr { - /// Any logical L3 node passes through unchanged when no L4 rule rewrote - /// it — a `Filter` doesn't need a sketch counterpart. - Logical(QueryExpr), - - /// Sketch aggregation. L4 picked the sketch type and parameters from the - /// catalog given the `AggIntent` and `DeploymentConstraints`. - SketchAgg { - child: Box, - sketch: SketchKind, // Kll, Cms, Hll, DDSketch, CmsWithHeap, … - params: SketchParams, // catalog-validated - col: ColumnRef, - by: Vec, - }, - - /// Sketch-aware join (KMV / theta-sketch for join cardinality; - /// join-sample for join sampling). Emitted only when a `Bind*OnJoin` - /// rule fires — L3 always presents the logical `Join` for L4 to choose. - SketchJoin { - outer: Box, - inner: Box, - key: ColumnRef, - sketch: SketchKind, - params: SketchParams, - }, - - /// Subtract one sketch from another. Valid only for sketches with a - /// linear-inverse property (CMS, theta, count-based). Lets the planner - /// compute "all-A minus all-B" cardinality / count without re-scanning. - SketchSubtract { left: Box, right: Box }, - - /// Delete a key from a sketch (CMS update with -1, deletable Bloom - /// filter, …). Valid only for deletion-supporting sketches. - SketchDelete { sketch_input: Box, key: ColumnRef }, - - /// Read out a query result from a built sketch. Inverse of `SketchAgg`. - /// `query` says what to extract — quantile φ, count for key k, cardinality. - SketchEstimate { sketch_input: Box, query: SketchQuery }, - - /// ⊕ — union of sketches across stages / shards. Distinct from L3 - /// `Merge` because sketch union has type constraints (same family, - /// same params). L5 stage allocator emits this when distributing. - SketchMerge { children: Vec }, -} -``` - -The optimizer's job is to selectively replace logical aggregates / joins with their sketch-bound variants when a binding rule fires; everything else stays inside `SketchExpr::Logical(…)`. - -#### Per-node input/output spec for `SketchExpr` - -L4 introduces a new field type into `Schema`: - -```rust -pub enum DataType { - // … the L3 types (Int64, Float64, Utf8, Map<…>, …) … - /// Sketch state. Carries the sketch family + params so the type system - /// rejects merges of incompatible sketches at plan time. - Sketch(SketchKind, SketchParams), -} -``` - -A "sketch-state schema" is a regular `Schema` whose value-bearing field has a `DataType::Sketch(...)` dtype. Reading rules: - -| Node | Input schemas | Output schema | -|---|---|---| -| `Logical(qe)` | whatever the inner L3 node `qe` consumes (per the L3 table above) | whatever `qe` produces — straight pass-through | -| `SketchAgg { child, sketch, params, col, by }` | one input schema; must contain `col` (the column being summarised) and every field referenced in `by` (group keys) | the `by` columns carried over verbatim, followed by one synthetic field of dtype `Sketch(sketch, params)` carrying the partial sketch state per group; `unique_keys = [by]` | -| `SketchJoin { outer, inner, key, sketch, params }` | two input schemas (outer + inner); both must contain `key` with compatible types | one field of dtype `Sketch(sketch, params)` carrying the join-cardinality / join-sample state — read out by a downstream `SketchEstimate` | -| `SketchSubtract { left, right }` | two input schemas, each with exactly one `Sketch(s, p)` field; **`s` and `p` must match** between the two inputs (catalog rejects mismatches at plan time); the `s` family must have `subtractable = true` in the catalog | one `Sketch(s, p)` field carrying the subtracted state (same family + params as inputs) | -| `SketchDelete { sketch_input, key }` | one input schema with a `Sketch(s, p)` field whose catalog entry has `deletable = true`; plus the `key` column to delete | the input schema unchanged in type — sketch state is mutated logically (the key's contribution removed) but the field's `(sketch, params)` signature is preserved | -| `SketchEstimate { sketch_input, query }` | one input schema with a `Sketch(s, p)` field; `query` (e.g. `Quantile(φ)`, `PointCount(k)`, `Cardinality`) must appear in the catalog entry's `supported_intents` for `s` | a regular row-shaped schema carrying the answer — `Float64` for quantile, `Int64` for count / cardinality, an array of `(key, count)` for top-k. The `Sketch(...)` field type does *not* propagate downstream of an Estimate | -| `SketchMerge { children }` | N input schemas, each with a `Sketch(s, p)` field; **all N must agree on `(s, p)`**; the `s` family must have `mergeable = true` in the catalog | one `Sketch(s, p)` field carrying the unioned state (same family + params as inputs) | - -Two type-system invariants make L4 robust: - -1. **Sketch-family mismatch is a plan-time error.** `SketchSubtract` over `Sketch(KLL, …)` and `Sketch(CMS, …)` fails type-checking before L5 ever sees it. -2. **Catalog capability flags gate which nodes can fire.** `SketchSubtract` requires `subtractable`, `SketchDelete` requires `deletable`, `SketchMerge` requires `mergeable`. The catalog (see §6 `core::physical::sketch_catalog`) is the single source of truth for these flags; binding rules consult it before producing the node. - -**Why a `Source` sum instead of two parallel `QueryExpr` trees:** most `QueryExpr` nodes (`Filter`, `Aggregate`) are data-model-agnostic — filter semantics are the same whether the input is a time-series window or a table scan. Only the leaf `Scan` differs. Keeping one tree with a polymorphic leaf means L4 rules like `BindKllOnQuantile` work uniformly across both data models; rules that care about data-model specifics (stage-aware push-down for TS; join-selectivity for tabular) gate on `source.data_model()` + `intent.requires()`. - -**Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values. That stream can come from a time-series window (`Source::TimeSeries`) or a table column (`Source::Table`); the sketch doesn't know or care. So `BindKllOnQuantile` works identically regardless of `Source`. - -#### Implementation status - -**Phase C** (`feat(controller): Phase C — sketch_algebra L4 IR (SketchExpr + Bind* rules)`) lands the typed `SketchExpr` IR + `Bind*` rules in `controller/src/sketch_algebra/`. Module layout: - -- `sketch_expr.rs` — `SketchExpr` enum (`Logical` / `SketchAgg` / `SketchEstimate` / `SketchMerge` / `LetBinding` / `Ref`). -- `params.rs` — `SketchKind` + per-family `SketchParams` (`KllParams{k}`, `DDSketchParams{alpha}`, `HllParams{precision}`, `CmsParams{w,d}`, `CountSketchParams{w,d,with_heap}`). -- `schema.rs` — `SketchStateSchema` with the `(SketchKind, SketchParams)` field-type and the catalog-capability flags (`mergeable` / `subtractable` / `deletable`). -- `rules/{bind_kll_quantile, bind_ddsketch_quantile, bind_cms_count, bind_cms_topk, bind_hll_cardinality}.rs` — five `Bind*` rules implementing the `Rule` trait. -- `lower.rs` — `bind_query_expr(&QueryExpr, AccuracyTarget) -> Result`. - -**Scope reduction.** The variant set ships the subset DC + PromQL needs. `SketchJoin`, `SketchSubtract`, `SketchDelete` from the spec above are intentionally *not* surfaced yet — they're gated on rules that haven't landed. Adding them is purely additive. - -**Planner-side migration is opt-in.** `controller/src/planner/rules.rs` exposes `bind_workload_typed(&QueryWorkload) -> Option` and an env-var gate `USE_TYPED_SKETCH_ALGEBRA=1`. Existing call sites continue to use the legacy untyped binding path (`algebra::directory::sketch_type_for_agg`); the typed path runs in parallel for callers that opt in. **Phase E** (stage_split refactor) is the natural migration point at which the typed path becomes the only path. - -### `core::lower` — L1 → L2 → L3 passes - -One pass per language, each producing the same `intent_algebra::QueryExpr`: - -```rust -pub fn lower_promql(ast: PromqlAst, schema: &MetricSchema) -> Result; -pub fn lower_sql(ast: SqlAst, schema: &TableSchema) -> Result; -pub fn lower_datafusion(ast: DfAst, ctx: &DfContext) -> Result; -pub fn lower_elasticdsl(ast: EsAst, schema: &IndexSchema) -> Result; -``` - -Once a query hits L3 it's language-agnostic. All deployment models downstream see the same IR. - -The `schema` parameter on each pass is supplied by the **Binder** — see -§6 "The Binder — name resolution as an explicit pass". The Binder is the -single place symbolic names become positional `ColumnId`s; the `lower_*` -passes consume its output and never resolve names ad-hoc. Today the -Binder runs at the L2→L3 boundary inside `lower`; it folds -into these `lower_*` signatures once the `relational` L2 IR is retired. - -### `core::pipeline` — orchestration - -The L1→…→L5 driver. Parameterised on a deployment model's optimizer rules (L4) + emitter (L5): - -```rust -pub struct Pipeline { - deployment_model: S, -} - -impl Pipeline { - pub fn run(&self, workload: &QueryWorkload) -> Result { - let l3: Vec = workload.queries() - .iter() - .map(|q| self.parse_and_lower(q)) // L1→L2→L3 - .collect::>()?; - let l4 = self.deployment_model.optimizer().optimize(l3)?; // L4 - let l5 = self.deployment_model.physical().lower(l4)?; // L5 - self.deployment_model.emitter().emit(&l5) // L5 → bytes/protobuf/DF plan - } -} -``` - -Core owns the driver. Deployment models own what goes into each deployment-model-specific seam. - -### `core::optimizer` — Layer 4 framework - -Core ships the **rule engine + trait surface + a shared rule library**. Deployment models pick which rules to enable. - -```rust -// trait (in core) -pub trait OptimizerRule: Send + Sync { - fn name(&self) -> &'static str; - fn category(&self) -> RuleCategory; // PushDown | Fusion | Elim | Bind | ... - fn priority(&self) -> u16; - fn apply(&self, expr: &QueryExpr, c: &DeploymentConstraints) -> Option; -} - -// engine (in core) — fixed-point iteration, cycle detection, priority ordering -pub struct RuleEngine { /* ... */ } -impl RuleEngine { - pub fn new(rules: Vec>) -> Self { /* ... */ } - pub fn run(&self, exprs: Vec, c: &DeploymentConstraints) - -> Result, OptError>; -} - -// shared rule library (in core::optimizer::rules) — opt-in from deployment models -pub mod rules { - pub struct BindKllOnQuantile; // AggIntent::Quantile → bind KLL(k by accuracy) - pub struct BindCmsOnCount; // AggIntent::Count → bind CMS(w, d) - pub struct BindHllOnCardinality; // AggIntent::Cardinality → bind HLL(p) - pub struct FusionPassthrough; // Aggregate over Filter → push Filter under Aggregate - pub struct ElimNoopFilter; // Filter(true) → child - // ... etc - impl OptimizerRule for BindKllOnQuantile { /* ... */ } -} -``` - -Deployment models compose rule sets by picking from the shared library + adding their own: - -```rust -// in deployment-model-asaplifecycle -use asap_control_core::optimizer::{RuleEngine, rules::*}; -fn rule_set() -> Vec> { - vec![ - Box::new(BindKllOnQuantile), - Box::new(BindCmsOnCount), - Box::new(FusionPassthrough), - // DC-specific additions: - Box::new(StageAwarePushDown), // pushes ops to edge when possible - Box::new(TransmissionCostRewrite), // uses TCO model to defer aggregation - ] -} -``` - -Core also ships `DeploymentConstraints` as a trait object; each deployment model supplies a concrete impl with its deployment's memory budgets, network topology, available sketch backends, and the registered `Executor` list (one entry per concrete runtime instance, each tagged with its `StageId`). The executor list is populated from the deployment model's discovery channel — OpAMP for DC lifecycle, static config for single-backend query, the in-process `SessionContext` itself for fusion. - -### `core::physical` — Layer 5 framework - -```rust -pub trait PhysicalPlanner { - type Topology: TopologyDescriptor; - type Output; - fn lower(&self, l4: Vec, t: &Self::Topology) -> Result; -} - -pub trait TopologyDescriptor { - fn stages(&self) -> &[StageDescriptor]; - fn edges(&self) -> &[StageEdge]; -} - -// pre-baked topologies in core -pub mod topology { - pub struct ThreeStage { /* edge → gateway → backend */ } - pub struct SingleStage { /* backend-only */ } - pub struct ZeroStage; /* in-process */ -} - -/// Identifier for a stage role (categorical tier in the data lifecycle). -pub struct StageId(pub String); // "edge" / "gateway" / "backend" / "in-process" - -/// A **concrete runtime instance** that executes a sub-DAG of the plan. -/// One stage may have many executors — e.g. a 50-host edge fleet is 50 -/// executors all with `stage = StageId("edge")`; a singleton backend is -/// one executor at `stage = StageId("backend")`. The stage allocator -/// works at stage granularity; the deployment-model planner / emitter -/// materialises each stage's sub-DAG into one concrete config per executor. -pub struct Executor { - pub id: ExecutorId, // stable identifier across replans - pub stage: StageId, // which stage this executor occupies - pub capabilities: ExecutorCaps, // memory budget, available sketch backends, network neighbours - pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle -} - -pub struct ExecutorId(pub String); - -pub enum ExecutorAddr { - OpAmpAgent(AgentId), // OTel collector managed via OpAMP (DC lifecycle) - HttpEndpoint(Url), // ASAPQuery-backend, etc. - InProcess, // asap-fusion library-mode SessionContext -} - -// generic stage allocator — given a QueryExpr tree + a topology, decide which -// ops land on which stage subject to constraints. Stage-level only; per-executor -// fan-out happens in the deployment model's PhysicalPlanner using the executor -// list from `DeploymentConstraints::executors()`. -pub struct StageAllocator; -impl StageAllocator { - pub fn allocate( - &self, exprs: &[QueryExpr], topology: &T, c: &DeploymentConstraints, - ) -> Result, PlanError>; -} - -// sketch catalogue — what sketches exist, what they support, what params they -// accept. Built at startup from the registered sketch backends; queried by L4 -// binding rules to map an `AggIntent` → `SketchKind` + `SketchParams`, and by -// L5 to instantiate the chosen sketch. -pub struct SketchCatalog { - pub entries: Vec, -} - -pub struct SketchEntry { - pub kind: SketchKind, // Kll, Cms, Hll, DDSketch, KMV, Theta, … - /// Which `AggIntent` variants this sketch can serve. - pub supported_intents: &'static [IntentTag], // Quantile, Count, Cardinality, JoinCardinality, … - /// Mergeability — sketches built on disjoint inputs combine without re-scan. - /// (CMS / HLL / KLL / theta = mergeable; SpaceSaving = approximately; - /// some heavy-hitter variants = no.) - pub mergeable: Mergeability, - /// Whether the sketch supports point deletion (CMS update with -1, - /// deletable Bloom filter) — gates `SketchExpr::SketchDelete`. - pub deletable: bool, - /// Whether the sketch admits a linear inverse (CMS, theta, count-based) - /// — gates `SketchExpr::SketchSubtract`. - pub subtractable: bool, - /// Accuracy / confidence model: error bounds as a function of params. - /// `(eps, delta)` for randomised sketches; absolute error for KLL; etc. - pub accuracy: AccuracyModel, - /// Aggregation keys this sketch supports natively. Some sketches are - /// keyed (CMS over (label, value)); others are unkeyed (HLL). - pub aggregated_keys: KeyShape, // Unkeyed | KeyedScalar | KeyedTuple - /// Parameter ranges + defaults. Catalog rejects out-of-range params at - /// L4 bind time so L5 never sees an unsupported configuration. - pub param_ranges: ParamRanges, - /// Memory + CPU model used by `CostModel`. Function of params. - pub cost_model: SketchCostModel, -} -``` - -Deployment models use these pieces: - -```rust -// in deployment-model-asaplifecycle -use asap_control_core::physical::{StageAllocator, topology::ThreeStage, Executor}; - -impl PhysicalPlanner for LifecyclePlanner { - type Topology = ThreeStage; - type Output = Vec<(ExecutorId, ExecutorPlan)>; // one entry per executor - fn lower(&self, l4: Vec, t: &ThreeStage) -> Result<_, _> { - // 1. Stage-level: colour the DAG by StageId. - let assignments = StageAllocator.allocate(&l4, t, &self.constraints)?; - // 2. Per-executor fan-out: for each Executor in the deployment, - // pick the sub-DAG assigned to its stage and produce its config. - let executors: &[Executor] = self.constraints.executors(); - // deployment-model-specific post-processing: DC's delta_cost logic, - // backend_client push preparation, etc. - Ok(/* ... */) - } -} -``` - -> **Implementation status (Phase E).** The typed L5 framework lives in -> `controller/src/stage_split/` (this PR): -> - `stage_id.rs` — `StageId` enum (`Edge`/`Gateway`/`Backend`) + -> `Topology` enum. Phase E ships only `Topology::ThreeStage`; the -> `SingleStage` / `ZeroStage` variants surface as future-proofing -> stubs that error cleanly via `AllocateError::UnsupportedTopology`. -> - `colored_dag.rs` — `ColoredDag { topology, nodes, edges }` IR, the -> allocator's output. `cut_edges()` exposes cross-stage edges for -> future wire-format insertion (Phase G+). -> - `allocator.rs` — `StageAllocator::allocate(expr, topology) -> -> Result`. Implements the §6 -> batched-queries colouring rules: `Logical(Scan/Window/Aggregate)` -> + `SketchAgg` → Edge; `SketchMerge` → Gateway; `SketchEstimate` → -> Backend; `LetBinding`/`Ref` colour by their bound expression's -> stage. The L3 `Aggregate{exact}` (e.g. `Max`) reachable through -> `Logical` colours Edge — the design.md "root of q3" backend -> placement is exercised when the same exact aggregation appears -> *above* a SketchMerge sibling structure (a Phase G enhancement -> that adds an explicit `Logical(Merge)` SketchExpr variant for the -> gateway hop). -> - `emitter.rs` — `Emitter` trait + `ThreeStageEmitter` that lowers -> `ColoredDag` → `HashMap`. `StageConfig` -> carries the structural facts each downstream consumer needs: -> - `Edge` → `EdgeStageConfig { source_metric, label_filters, -> window_secs, sketch_processors, exporter_target }`. Sketch -> processor names follow the catalog (`KLL`, -> `ddsketch`, `HLL`, `countmin`, -> `countsketch`). -> - `Gateway` → `GatewayStageConfig { otlp_receiver_port, -> merge_processors, exporter_target }`. -> - `Backend` → `BackendStageConfig { aggregations, readouts }` — -> the aggregation_id ↔ (sketch_kind, params) mapping the backend's -> `OtlpReceiver` + readout catalog need. -> -> The typed path is opt-in via the `USE_TYPED_STAGE_SPLIT` env var -> consulted by `controller/src/planner/stage_split.rs::split_typed_three_stage`; -> existing untyped callers (`split_expr_by_stage`) continue to run -> unchanged. OpAMP push (`crate::opamp::OpampServer::push_to_role`) -> and backend `StreamingConfig` POST (`crate::backend_client`) wiring -> against the typed `StageConfig` is downstream (Phase G+) — Phase E -> ships only the structured per-stage output, not the wire push. - -### `core::plan` — shared traits bridging layers - -```rust -pub trait DeploymentModel { - type Topology: TopologyDescriptor; - type EmitterOutput; - fn rules(&self) -> Vec>; - fn topology(&self) -> &Self::Topology; - fn physical(&self) -> &dyn PhysicalPlanner; - fn emitter(&self) -> &dyn PlanEmitter; - fn constraints(&self) -> &dyn DeploymentConstraints; -} -``` - -### `core::workload` - -> **Implementation status (this PR).** The new types `QueryLanguage`, -> `AccuracyTarget`, `QueryShape`, `DataShape`, `QueryId`, `BindingName`, -> and a `WorkloadPlan` container live in `controller/src/types_v2.rs`. -> `analyzer::QuerySpec` carries the new fields (`id`, `language`, -> `accuracy`, `dollars`, `deployment_model`, `shape`, `data`) as -> `#[serde(default)]` additions; defaults preserve legacy behaviour for -> existing `POST /api/v1/plan` callers and the `workloads.yaml` -> pre-population path. `Analyzer::analyze` enforces the L1 cross-product -> rejections (`Streaming × Batch`, `Streaming × Mutable`) from the table -> below and resolves typed `accuracy` over legacy `accuracy_sla` with -> the typed form taking precedence. The fields are not yet load-bearing -> in `replan.rs` / `planner/` cost or binding decisions — that's a -> separate downstream PR. `WorkloadPlan` is a container only; the CSE -> pass that populates `bindings` is deferred until the L3 algebra grows -> `LetBinding` / `Ref`. - -One public type for every kind of input: - -```rust -pub enum QueryWorkload { - /// A single QuerySpec from an HTTP POST. DC's current entry point. - Single(QuerySpec), - /// A hand-authored YAML workload file. asap-planner-rs's current entry point. - AuthoredSet(Vec), - /// A Prometheus query log replayed. asap-planner-rs's alt entry point. - QueryLog(QueryLogReplay), -} - -pub struct QuerySpec { - /// Stable identifier — preserved across `replan` cycles so the runtime - /// can correlate plan outputs with the originating spec, and L4 reuse - /// rules can name shared producers across consumers in the same workload. - pub id: QueryId, - - /// Source-language query string + which language it's written in. - /// L1 parses `query` against `language`; no schema lookup yet. - pub query: String, - pub language: QueryLanguage, // PromQL | Sql | DataFusion | ElasticDsl - - /// Evaluation time range. Some languages (PromQL) carry their own ranges - /// inside `query`; this field wins when both are set. - pub time_range: TimeRange, - - /// Per-target SLA. Drives L4 binding (which sketch / which params) and - /// L5 stage placement (push down or not). The three are independent — - /// `accuracy: Exact` disables all `Bind*` rules; an unset latency / - /// dollars target lets the cost model pick freely. - pub accuracy: AccuracyTarget, // Exact | Epsilon(f64) | EpsilonDelta { ε, δ } - pub latency: Option, // p99 evaluation latency target - pub dollars: Option, // per-evaluation budget - - /// Routing hint: which deployment model should plan this query. When - /// unset, the runtime defaults to the deployment model bound to the - /// inbound HTTP route (`POST /plan/lifecycle` vs `POST /plan/query`). - pub deployment_model: Option, - - /// How the query is *evaluated*: one-shot, continuous, or scheduled. - /// Distinct from `data` below — a one-shot query against a streaming - /// source is "evaluate now over the latest window"; a streaming query - /// against a batch source doesn't make sense and is rejected at L1. - /// Drives L4 binding (mergeable vs one-shot sketch family) and the - /// L5 wire format (`OneShot` emits config + result; `Streaming` and - /// `Periodic` emit a config that keeps running). - pub shape: QueryShape, - - /// Shape of the *data* feeding the query. Workload-level summary — - /// per-leaf detail rides on `Source::data_shape` (§6 L3). For a join - /// over streaming + batch this is `Mixed`, and the planner reads the - /// per-leaf shape during L4. Drives binding choices: an - /// `AppendOnlyStream` unlocks incremental, mergeable sketches and - /// retraction-free aggregation; `Batch` lets the planner pick a - /// non-mergeable estimator (e.g. exact percentile over a sort) that - /// wouldn't survive a distributed streaming setting; `Mutable` - /// requires retraction-aware operators (out of scope today — the - /// planner refuses sketch binding and falls back to re-scan). - pub data: DataShape, -} - -pub enum QueryLanguage { PromQL, Sql, DataFusion, ElasticDsl } -pub enum AccuracyTarget { Exact, Epsilon(f64), EpsilonDelta { eps: f64, delta: f64 } } - -pub enum QueryShape { - /// Evaluate once. Plan, execute, return result, discard state. - /// SQL ad-hoc queries; one-off PromQL via `POST /plan`. - OneShot, - /// Continuous query — output stream that the executor keeps emitting - /// as new data arrives. No fixed cadence; the runtime emits whenever - /// the underlying state changes. Streaming dashboards, alerting - /// expressions evaluated by the agent rather than by a poller. - Streaming, - /// Re-evaluated at a fixed cadence — Prometheus recording rules, - /// scheduled dashboard panels, alerting evaluation cycles. The - /// planner amortises sketch / aggregate build cost across evaluations - /// within the cadence and reuses state between adjacent windows. - Periodic { every: Duration }, -} - -pub enum DataShape { - /// Bounded relation, fully materialised at plan time. SQL tables, - /// Parquet / CSV files, DataFusion in-process tables. - Batch, - /// Append-only stream — events arrive over time, never updated or - /// deleted. Metrics, logs, event streams. The common case for - /// asaplifecycle and asapquery. - AppendOnlyStream, - /// Mutable relation — inserts + updates + deletes. Operational - /// databases, CRUD-style tables. Sketch binding is currently refused - /// for this shape (no retraction-aware sketches in the catalog yet). - Mutable, - /// Join across sources of differing shape. The planner consults - /// `Source::data_shape` per leaf during L4; this variant exists so - /// callers don't have to flatten a workload-level summary. - Mixed, -} -``` - -All three deployment models take `&QueryWorkload`. Same type, different planners. Fields are deployment-model-agnostic — anything deployment-model-specific (DC's stage-budget overrides, fusion's `SessionContext` handle) rides on `DeploymentConstraints`, not on `QuerySpec`. - -The `shape` × `data` cross-product isn't fully populated. Combinations the planner accepts and rejects: - -| `shape` \ `data` | `Batch` | `AppendOnlyStream` | `Mutable` | `Mixed` | -|---|---|---|---|---| -| `OneShot` | ✓ ad-hoc SQL, fusion | ✓ "evaluate now over latest window" | ✓ via re-scan, no sketch binding | ✓ per-leaf shape decides | -| `Streaming` | ✗ rejected at L1 (no stream over a static dataset) | ✓ canonical streaming case | ✗ rejected (no retraction support) | ✓ if streaming leaves dominate | -| `Periodic { every }` | ✓ scheduled batch report | ✓ recording rules | ✓ via re-scan | ✓ | - -### `core::cost` - -Extracted from DC's `planner/*` (delta, online, pareto, tco). Deployment models pick the models they need. - -```rust -pub trait CostModel { - fn accuracy(&self, plan: &Plan) -> Accuracy; - fn latency(&self, plan: &Plan) -> Duration; - fn dollars(&self, plan: &Plan) -> Dollars; - - /// Workload-level total cost. A straight sum of per-plan costs - /// under-estimates how good a plan is when multiple queries share - /// computation — a sketch or a precomputed aggregate built for - /// one query serves the others for free. Implementations must - /// identify reusable sub-expressions across `plans` and credit - /// their build cost once, so the planner prefers plans that - /// maximise reuse when the total-cost objective allows. - fn workload_cost(&self, plans: &[Plan]) -> WorkloadCost; -} - -pub struct WorkloadCost { - pub total_latency: Duration, - pub total_dollars: Dollars, - /// Per-plan contribution, for EXPLAIN / observability. - pub per_plan: Vec, - /// Sub-expressions built once, consumed by ≥2 plans. Drives - /// decisions like "build a KLL for p99 that q1 and q2 both read" - /// vs "run exact select-n per query". - pub reused: Vec, -} -``` - -The reuse model is not sketch-specific: any primitive that is costly to build once and cheap to query (sketches, materialized aggregates, cached scan results, future wavelet summaries) plugs into the same `ReusedComponent` accounting. - -#### Implementation status - -Phase F (in `controller/src/planner/cost_model.rs`) lands the workload-cost shape that pairs with the `Schema::unique_keys`-based CSE legality gate above: - -- `workload_cost(plan: &WorkloadCostPlan<'_>) -> Result` — walks the L3 IR DAG (`intent_algebra::QueryExpr`) post-order, memoises by `LetBinding` name, credits each shared producer once across consumers. Returns `WorkloadCost { total_dollars, per_root_breakdown, reused_savings }` — the `reused_savings` field exposes the gap between the bundled total and the naive sum-over-roots, so EXPLAIN can show what shared-producer credit was worth. -- `WorkloadCostPlan { bindings, roots }` — the cost-model's view of `types_v2::WorkloadPlan` carrying real `&QueryExpr` references rather than the `QueryExprPlaceholder` JSON-wire string. Collapses into `types_v2::WorkloadPlan` when the placeholder is swapped for live `QueryExpr` downstream. - -Per-node cost primitives at L3 (`node_cost_scan`, `node_cost_window`, `node_cost_aggregate`, `intent_cost`) are coarse-but-monotonic placeholders calibrated against schema width and intent kind. Calibration against real benchmarks is downstream; what Phase F pins is the *shape* — costs are positive, additive over sub-trees, and the savings invariant `bundled_total ≤ naive_sum` holds with `savings = naive_sum − bundled_total`. - -Not yet shipped at L3 cost (deferred): the `total_latency` field, the per-plan `Contribution` split (Phase F's `per_root_breakdown` carries one `f64` per root, not the full latency / dollars / accuracy tuple), and the `ReusedComponent` enumeration (savings are reported as a scalar, not as a per-component vector — the per-component breakdown is downstream when more producer kinds (sketches, materialized aggregates) become candidates for sharing). The L4 `score` / `score_with` path above remains the per-plan dollars / latency / memory cost; the L3 `workload_cost` is the bundled-plan credit on top. - -### `core::emit` - -```rust -pub trait PlanEmitter { - type PlanInput; - type Output; // YAML bytes, OpAMP RemoteConfig, rewritten LogicalPlan - fn emit(&self, plan: &Self::PlanInput) -> Result; -} -``` - -Concrete emitters live in deployment model crates: - -- `deployment-model-asapquery::yaml::StreamingConfigEmitter` → `streaming_config.yaml` bytes -- `deployment-model-asapquery::yaml::InferenceConfigEmitter` → `inference_config.yaml` bytes -- `deployment-model-asaplifecycle::emit::OpAmpRemoteConfigEmitter` → `opamp::RemoteConfig` protobuf -- `deployment-model-asapfusion::emit::DataFusionPlanEmitter` → rewritten `datafusion::LogicalPlan` - -### `core::registry` - -The extension point. The runtime binary does: - -```rust -let mut reg = DeploymentModelRegistry::new(); -reg.register::(); -reg.register::(); -reg.register::(); -// add more... -``` - -Registration is by-type; the runtime looks up by `DeploymentModelId` (either from the `QuerySpec.deployment_model` field or from an HTTP route). A 4th deployment model is added by: - -1. New crate `deployment-model-/` -2. Implement `DeploymentModel` trait (wraps a `Planner` + its `PlanEmitter`s) -3. Register in `bin/asap-controller/main.rs` - -No core change. - -### End-to-end example — one query through all five layers - -A worked trace of a single PromQL query as it flows L1 → L5. The query is intentionally simple (one metric, one window, one aggregate) so the IR shapes stay readable; production workloads have larger DAGs but the per-layer transformation is the same. - -**Input.** A `QuerySpec` arrives at `POST /plan` carrying: - -```text -quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m]) -``` - -with `accuracy: AccuracyTarget::Epsilon(0.01)` and `language: PromQL`. - -#### L1 — query language (parse to `PromqlAst`) - -`core::query_language::promql::parse` wraps `promql-parser`. Output (sketch — actual fields come from the upstream crate): - -```rust -PromqlAst::Call { - func: BuiltinFn::QuantileOverTime, - args: vec![ - Expr::Number(0.99), - Expr::MatrixSelector { - name: "http_request_duration_seconds", - matchers: vec![LabelMatcher::Eq("service", "api")], - range: Duration::from_secs(300), - }, - ], -} -``` - -Pure language-level parse — no schema lookup, no sketch awareness. Same call returns the same AST regardless of deployment model. - -#### L2 — language logical plan (`PromqlLogicalPlan` tree) - -`core::lower::promql::lower_to_logical` walks the AST against a `MetricSchema` resolved from Prometheus's `/api/v1/labels`. The result preserves PromQL semantics — `quantile_over_time` is a range-vector aggregate, distinct from a generic `Aggregate { Quantile }` over a `Window`: - -```rust -PromqlLogicalPlan::RangeAggregate { - func: PromqlRangeAggFunc::QuantileOverTime { q: 0.99 }, - range: Duration::from_secs(300), - matrix: Box::new(PromqlLogicalPlan::MatrixSelector { - metric: "http_request_duration_seconds", - matchers: vec![("service", LabelOp::Eq, "api")], - }), -} -``` - -Per language: the SQL form `SELECT approx_percentile(latency, 0.99) FROM events WHERE service='api' AND ts > now() - INTERVAL '5 min'` lands in `SqlLogicalPlan` with a different shape. L2 is per-language; the shapes converge at L3. - -#### L3 — intent algebra (`QueryExpr` + `AggIntent`) - -`core::lower::promql::lower_to_intent` strips PromQL-specific shapes. `quantile_over_time` does **not** survive — the canonical L3 form for a windowed quantile is `Window` over `Aggregate{Quantile}` (see §6 design rule 1, "no `WindowedAgg`"; see §6 "Why no `QuantileOverTime` intent?"). The accuracy target threads through from the `QuerySpec`: - -```rust -QueryExpr::Aggregate { - by: vec![], - aggs: vec![AggIntent::Quantile { - q: 0.99, - accuracy: AccuracyTarget::Epsilon(0.01), - }], - having: None, - child: Box::new(QueryExpr::Window { - kind: WindowKind::Sliding, - size: Duration::from_secs(300), - slide: None, - child: Box::new(QueryExpr::Scan { - source: Source::TimeSeries { - metric: MetricRef::from("http_request_duration_seconds"), - time: TimeRange::default(), // supplied by QuerySpec at evaluation - labels: LabelFilter::eq("service", "api"), - }, - predicates: vec![], - }), - }), -} -``` - -Per-edge schemas (derived per the §6 input/output spec): - -| Edge | Schema | -|---|---| -| `Scan` → `Window` | `{value: Float64, ts: Int64@time_index, service: Utf8}` | -| `Window` → `Aggregate` | the above + `{window_id: Int64, window_start: Int64, window_end: Int64}` | -| `Aggregate` → root | `{p99_value: Float64}` (one column per `AggIntent::Quantile`, typed via `output_type`) | - -This IR is identical across deployment models — same DAG, same intent, same accuracy target. Below this point each deployment model picks its own L4 rules. - -#### L4 — sketch algebra (`SketchExpr`) - -The shared rule `core::optimizer::rules::BindKllOnQuantile` matches `Aggregate { aggs: [Quantile{q, accuracy}] }`, consults the sketch catalog (KLL has `supported_intents: [Quantile]`, is mergeable, satisfies `ε=0.01` at `k=200`), and rewrites the matched sub-DAG into a `SketchAgg` wrapped in a `SketchEstimate` (the readout). Everything not rewritten passes through in `SketchExpr::Logical(...)`: - -```rust -SketchExpr::SketchEstimate { - sketch_input: Box::new(SketchExpr::SketchAgg { - child: Box::new(SketchExpr::Logical(/* the L3 Window+Scan subtree */)), - sketch: SketchKind::Kll, - params: SketchParams::Kll(KllParams { k: 200 }), - col: ColumnRef::value(), - by: vec![], - }), - query: SketchQuery::Quantile { q: 0.99 }, -} -``` - -Catalog-derived edge schemas: - -| Edge | Schema | -|---|---| -| inner `Logical(Window)` → `SketchAgg` | as L3 above (logical pass-through) | -| `SketchAgg` → `SketchEstimate` | `{kll: Sketch(Kll, KllParams{k:200})}` | -| `SketchEstimate` → root | `{p99_value: Float64}` (the `Sketch(...)` field type does not propagate past `SketchEstimate`) | - -The `Sketch(Kll, KllParams{k:200})` field type is the L4 type-system invariant — a downstream `SketchMerge` over a mismatched `Sketch(Cms, …)` input would fail at plan time, before L5 ever sees it. `BindKllOnQuantile` lives once in core and fires for all three deployment models; deployment-model-specific rules (lifecycle's `StageAwarePushDown`, fusion's `HashModeRule`) chain before or after. - -#### L5 — physical plan (stage allocation + emission, per deployment model) - -L4 chose the sketch family + params. L5 colors the DAG by `StageId` and emits per-executor configs. Same `SketchExpr` input; topology and emitter differ per deployment model. - -##### deployment-model-asaplifecycle (3-stage, OpAMP + backend POST) - -`StageAllocator` colors against `topology::ThreeStage`: - -| Node | StageId | Why | -|---|---|---| -| `Scan` + `Window` + `SketchAgg` | `"edge"` | scrape happens on the agent host; KLL build is mergeable so it's safe to run early | -| `SketchMerge` (inserted on cut edge) | `"gateway"` | catalog says `KLL.mergeable = true`; reduces N edge streams to 1 | -| `SketchEstimate` | `"backend"` | readout where users query | - -`PhysicalPlanner` then fans out via `DeploymentConstraints::executors()`. Given a deployment with 3 edge agents + 1 gateway + 1 backend: - -```rust -vec![ - (ExecutorId("edge-001"), OpAmpRemoteConfig { /* OTel YAML: scrape http_request_duration_seconds{service=api}, - sliding 5m window, KLL k=200, forward to gateway-001 */ }), - (ExecutorId("edge-002"), OpAmpRemoteConfig { /* identical OTel YAML; differs only by agent_id */ }), - (ExecutorId("edge-003"), OpAmpRemoteConfig { /* … */ }), - (ExecutorId("gateway-001"), OpAmpRemoteConfig { /* receive 3 KLL streams, SketchMerge, forward to backend */ }), - (ExecutorId("backend-001"), AsapqueryBackendConfig { /* read merged KLL, SketchEstimate q=0.99 */ }), -] -``` - -The 3 edge configs are byte-identical except for `agent_id` — that's the §3 "one stage may have N executors all materialised from the same per-stage sub-DAG" property. The `AsapqueryBackendConfig` is produced by calling `deployment-model-asapquery::yaml::StreamingConfigEmitter` (see §8 asymmetric dep). - -##### deployment-model-asapquery (1-stage, YAML) - -`StageAllocator` against `topology::SingleStage` returns everything on `"backend"`. `PhysicalPlanner` produces one config per backend executor (typically one): - -```yaml -# streaming_config.yaml — emitted by deployment-model-asapquery::yaml::StreamingConfigEmitter -operators: - - name: kll_p99_request_duration - kind: DatasketchesKLL - params: { k: 200 } - input: - metric: http_request_duration_seconds - label_filter: { service: api } - window: { kind: sliding, size: 5m } -estimates: - - sketch: kll_p99_request_duration - query: { kind: quantile, q: 0.99 } -``` - -This is the byte-for-byte format ASAPQuery-backend already consumes — `inference_config.yaml` is emitted alongside by `InferenceConfigEmitter` for the readout side. Both are wire invariants (see §10). - -##### deployment-model-asapfusion (0-stage, in-process `LogicalPlan`) - -`StageAllocator` against `topology::ZeroStage` returns the whole DAG in-process. `PhysicalPlanner` rewrites the caller's `datafusion::LogicalPlan`, replacing the original `Aggregate(quantile(0.99, ...))` node with an `Extension` wrapping the KLL sketch op: - -```rust -LogicalPlan::Extension(Extension { - node: Arc::new(SketchAggExt { - kind: SketchKind::Kll, - params: KllParams { k: 200 }, - input: /* original Window+Filter+Scan subtree, unchanged */, - estimate: SketchQuery::Quantile { q: 0.99 }, - }), -}) -``` - -Returned to the caller's `SessionContext` for execution by `asap-fusion`'s in-process operator. No wire format, no external executor; the data plane is the caller's process. Note also that `asap-fusion`'s entry point is `Source::Table` (not `Source::TimeSeries`) — this PromQL example is only illustrative for the fusion deployment model, which in practice consumes a pre-built DataFusion `LogicalPlan` and skips L1 (see §8). - -#### What this example demonstrates - -- **L1 → L3 are deployment-model-independent.** All three deployment models see the same `QueryExpr` for this query. -- **L4 binding is a shared rule.** `BindKllOnQuantile` lives once in `core::optimizer::rules` and fires for all three. -- **L5 is where deployment models diverge.** Same `SketchExpr` → three topologies → three emitter outputs. The topology is a parameter, not an axis of code (§3). -- **The sketch path is selected, not mandated.** If the `QuerySpec` had `accuracy: AccuracyTarget::Exact`, no `Bind*` rule would fire; L4 would pass `SketchExpr::Logical(QueryExpr::Aggregate{…})` through, and L5 would target an exact `HashAgg` (§1 goal 6). - -### End-to-end example — batched queries with shared sub-DAGs - -The single-query trace above shows one `QueryExpr` flowing through all five layers. This example shows what changes when the `QueryWorkload` is an `AuthoredSet` of related queries — the case `CostModel::workload_cost` and §6 design rule 5 (DAG fan-in) are designed for. The point is that reuse is expressed as **shared nodes in a single DAG**, not as side-channel caching. - -**Input.** A `QueryWorkload::AuthoredSet` arrives with three PromQL queries on the same metric: - -```text -q1: quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m]) -q2: quantile_over_time(0.95, http_request_duration_seconds{service="api"}[5m]) -q3: max_over_time(http_request_duration_seconds{service="api"}[5m]) -``` - -All three carry `accuracy: Epsilon(0.01)`. Planning them independently would scan + window the same metric three times. The DAG-shaped IR collapses the shared work. - -#### L3 — common sub-expression elimination produces fan-in - -After per-query L1→L2→L3 lowering, the three `QueryExpr` trees are identical below the `Aggregate` node. A workload-level CSE pass (`core::lower::workload::dedupe_subtrees`) hoists the shared sub-DAG behind a `LetBinding`, leaving three `Aggregate` nodes that fan in to one producer: - -```text - Scan{http_request_duration_seconds, service="api"} - │ - Window{Sliding, 5m} ◄── shared producer - ╱ │ ╲ - Aggregate Aggregate Aggregate - [Quantile{0.99,ε}] [Quantile{0.95,ε}] [Max] - │ │ │ - q1 q2 q3 -``` - -Multi-root DAGs live one level above `QueryExpr`. `QueryExpr` stays single-root (one query in, one root out) — the workload-level container holds N roots plus the hoisted bindings they share: - -```rust -// core::workload — output of L1→L2→L3 lowering for a QueryWorkload -pub struct WorkloadPlan { - /// Named shared producers, hoisted out of individual queries by the CSE - /// pass. Each binding is referenced by ≥2 roots via `QueryExpr::Ref`. - pub bindings: Vec<(BindingName, QueryExpr)>, - /// One root per QuerySpec in the workload, in input order. - pub roots: Vec<(QueryId, QueryExpr)>, -} -``` - -For the example above: - -```rust -WorkloadPlan { - bindings: vec![ - ("windowed_latency".into(), /* Scan → Window subtree */), - ], - roots: vec![ - (q1_id, QueryExpr::Aggregate { - by: vec![], - aggs: vec![AggIntent::Quantile { q: 0.99, accuracy: Epsilon(0.01) }], - child: Box::new(QueryExpr::Ref("windowed_latency".into())), - having: None, - }), - (q2_id, /* same shape, q=0.95, same Ref */), - (q3_id, /* Aggregate { aggs: [AggIntent::Max] } over the same Ref */), - ], -} -``` - -Three `QueryExpr::Ref` nodes, one binding; the fan-in is explicit. `CostModel::workload_cost` credits the Scan + Window build cost once across {q1, q2, q3} via `WorkloadCost::reused`, which is what makes the bundled plan beat three independent plans on `total_dollars` (§6 `core::cost`). Within-query CTE fan-in (SQL `WITH`, PromQL recording rules) keeps using `QueryExpr::LetBinding`/`Ref` unchanged — `WorkloadPlan` only adds the cross-query layer. - -CSE legality leans on `Schema::unique_keys` (§6 Schema flow): two `QueryExpr::Ref` consumers can share a producer only when its output schema is provably stable across reads — the unique-key metadata is what lets the deduper assert that without re-running the producer's logic. - -#### L4 — sketch reuse across q1 and q2 - -`BindKllOnQuantile` fires three times (once per `Quantile` intent in the workload), but a follow-on rule `MergeKllSketches` recognises that q1 and q2 read from the same input at the same accuracy. KLL state is independent of `q` — one sketch serves any quantile readout — so the two `SketchAgg{KLL}` nodes collapse into one, with two `SketchEstimate` parents reading `q=0.99` and `q=0.95`. q3 (`Max`) takes a separate path: KLL doesn't expose max, so the rule emits an exact `Aggregate{Max}` over the shared `Window`. - -```text - Scan - │ - Window - ╱ │ ╲ - SketchAgg{KLL,k=200} Aggregate{Max} - ╱ ╲ │ - SketchEstimate SketchEstimate q3 - q=0.99 q=0.95 - │ │ - q1 q2 -``` - -`SketchExpr::LetBinding` carries the two-tier fan-in: the outer let names the `Window` output (read by all three branches), an inner let names the `SketchAgg{KLL}` output (read by both `SketchEstimate` parents). The L4 type system rejects mismatched merges before L5 sees them — both `SketchEstimate` parents must declare `Sketch(Kll, KllParams{k:200})` on their input edge, which is checked locally per the §6.4 sketch-state schema rules. - -#### L5 — colored DAG, one config per executor - -`StageAllocator` colors the bound DAG by `StageId` exactly as for the single query, but the shared nodes are colored once. Under `topology::ThreeStage`: - -| Node | StageId | -|---|---| -| `Scan` + `Window` + `SketchAgg{KLL}` + `Aggregate{Max}` | `"edge"` | -| `SketchMerge` (over KLL streams), `Merge` (over Max streams) | `"gateway"` | -| `SketchEstimate{q=0.99}`, `SketchEstimate{q=0.95}`, root of q3 | `"backend"` | - -The OpAMP config emitted to each edge agent describes one scrape, one window operator, one KLL builder, one max accumulator — feeding two output streams (KLL state, Max state) toward gateway. Per-edge memory drops from `3× scan + 3× window + 2× KLL + 1× Max` (independent plans) to `1× scan + 1× window + 1× KLL + 1× Max`; bandwidth across the edge→gateway cut drops correspondingly. - -#### What this example demonstrates - -- **Reuse is a DAG property, not a sketch property.** The Scan + Window collapse is L3 CSE; the KLL collapse is L4 sketch-aware; both expressed as fan-in in the same DAG. No new IR surface, no caching layer. -- **`Schema::unique_keys` becomes load-bearing.** The CSE pass uses it to prove two `Ref`s read from a stable producer. Without it, the deduper has to be conservative and reuse drops on the floor — which is why §6 keeps the field on the `Schema` struct even though single-query plans don't read it (§6 Schema flow). -- **`CostModel::workload_cost` decides whether to share.** Reuse isn't free — when q1 demands a tighter accuracy than q2, the smaller-budget sketch may not satisfy both, and keeping them separate can be cheaper. `MergeKllSketches` is gated by `workload_cost`, not unconditional. -- **L5 is unchanged.** Same allocator, same emitter, same wire format — the only difference is that the DAG has multiple roots and shared interior nodes. The single-query path is the degenerate case where `outputs.len() == 1` and no fan-in fires. - -## 7. Runtime crate details - -Lifted from DC controller with zero semantic change: - -- `http/` — axum server on `:8080`. Routes: `POST /plan`, `POST /replan`, `GET /plans/:id`, `GET /status`, `GET /metrics` -- `opamp/` — WebSocket OpAMP server on `:4320`. Same protocol DC speaks today. -- `monitor/` — `Scraper` polls Prometheus, emits `Violation` -- `replan/` — subscribes to violations + expiry ticks, re-invokes the deployment model registry -- `store/` — in-memory `PlanStore` + `WorkloadStore`. Pluggable backend later. -- `backend_client/` — pushes `StreamingConfig` YAML to ASAPQuery-backend's `/api/v1/streaming-config` endpoint. Factored out so new deployment models can push to other backends. - -Runtime depends on `core` but NOT on any deployment model crate directly. It talks to deployment models via the registry. - -## 8. Deployment model crate details (each owns its L4 + L5) - -Each deployment model crate is a library with: - -- A `DeploymentModel` impl that registers its planner, its emitters, and its HTTP routes (if any). -- Its own config types — no shared config crate. -- Its own cost model — maybe using `core::cost` primitives, maybe not. -- Its own integration tests. - -### `deployment-model-asaplifecycle` (thin) - -- **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **L4 rules**: picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules that require stage awareness (`StageAwarePushDown`, `TransmissionCostRewrite`). Adding a stage-specific rule = a new file in `deployment-model-asaplifecycle/src/rules.rs`, impl `OptimizerRule`. Unchanged rules come from core. -- **L5 topology**: `core::physical::topology::ThreeStage` (edge → gateway → backend). No deployment-model-specific allocator logic — `StageAllocator` handles the tree walk; lifecycle only declares what the topology looks like. -- **Cost models**: `delta / online-EMA / Pareto / TCO` — these live in `deployment-model-asaplifecycle/src/cost.rs` because they're specific to the DC deployment's network/compute assumptions. Implement `core::optimizer::cost::CostModel`. -- **L5 emitters**: `OpAmpRemoteConfigEmitter` (per-role OTel YAML over OpAMP WebSocket) and `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery`'s `StreamingConfigEmitter` for the YAML bytes, then POSTs to backend). -- **HTTP route**: `POST /plan` for full-lifecycle planning, `POST /replan` for SLA-triggered. -- **Size estimate**: ~1500 LOC (was ~5000 pre-refactor). Cost models are the bulk; rule selection + topology + emitter are each a few hundred lines. - -### `deployment-model-asapquery` (thin) - -- **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. -- **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. -- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::intent_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). -- **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this deployment model covers most sketch types) + a deployment-model-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. -- **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. -- **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `deployment-model-asaplifecycle` calls them when it needs to POST to ASAPQuery-backend. -- **Extra L1 inputs**: `query_log/` for Prometheus-query-log replay (unique to this deployment model); `schema/` for `PromQLSchema` discovery from a live Prometheus URL. -- **HTTP route**: `POST /plan/query` (JSON `QuerySpec` in, YAML stream out). Also backs the `bin/asap-query` one-shot CLI. -- **Size estimate**: ~2500 LOC (was ~6000 pre-refactor — saved by picking from shared rule library; still pays the L2 tree + L3/L4 split refactor cost, which is one-time). - -### `deployment-model-asapfusion` (thin) - -- **Data model**: tabular. L2→L3 lowering produces `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when fusion extends to multi-table). Crucially **not time-indexed** — fusion works on arbitrary DataFusion relations; time is just another column if present. -- **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode deployment model. -- **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so deployment models that want to take a DataFusion plan as input have a canonical name for it. -- **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::intent_algebra::AggIntent` subset. -- **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + deployment-model-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. -- **L5 topology**: `core::physical::topology::ZeroStage` (in-process). -- **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this deployment model is library-mode. -- **Executor**: `ASAPExecutor` wraps a DataFusion `SessionContext`. Users construct `deployment-model-asapfusion` in-process. -- **HTTP route**: none by default. -- **Conformance cost**: near zero. Fusion's `SketchConfigRule` logic is replaced with picks from `core::optimizer::rules::*`; its `HashModeRule` stays deployment-model-specific. -- **Size estimate**: ~1800 LOC (was ~3000 pre-refactor; the `SketchConfigRule` code folds into core's shared rule library). - -The sketch microbenchmarks (KLL/CMS) move with the crate and keep running. The TODO items from `asap-fusion/TODO.md` (batch/multi-query execution, time semantics, distributed model) remain open but are now filed against `deployment-model-asapfusion/TODO.md` in-repo. - -### Data plane communication - -Control plane (ASAPController) and data plane (OTel agents / ASAPQuery-backend / DataFusion runtimes) always talk **over wire**, never via in-process calls. This is unchanged from today: - -| Deployment model | Data plane lives in | Wire protocol | -|---|---|---| -| lifecycle | DataCollector (OTel collectors, agent + backend roles) | OpAMP WebSocket (config push) + HTTP POST (`StreamingConfig` → ASAPQuery-backend) + Prometheus scrape (metrics in) | -| query | ASAPQuery-backend (query engine, SketchStore) | HTTP POST `/api/v1/streaming-config` + `/api/v1/plan` (capability-miss callback in) + YAML file on disk (init-container mode) | -| fusion | Caller's DataFusion `SessionContext` | in-process library call (no wire) | - -**Data plane code stays in its original repo.** ASAPController only owns the control plane. The merger doesn't move OTel collectors out of DataCollector, doesn't move the query engine out of ASAPQuery-backend, and doesn't move DataFusion out of asap-fusion's users. Each data plane keeps its own release cadence. - -### Deployment model placement: in-repo or out-of-repo - -Because core owns the L4/L5 infrastructure (not just L1-3), a deployment model crate is small and largely self-contained. That means deployment models can live either: - -- **Inside ASAPController workspace** — `crates/deployment-model-/`, lockstep release with core, cross-deployment-model changes are one PR. -- **In their own downstream repo** — declares `asap-control-core` as a git-tagged dep, releases independently, owns its own CI. - -Both produce functionally identical artifacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. - -Default recommendation: -- **deployment-model-asaplifecycle** and **deployment-model-asapquery** in ASAPController (they share `deployment-model-asapquery`'s YAML emitter via a workspace `path = "../deployment-model-asapquery"` dep — trivial in-workspace). -- **deployment-model-asapfusion** out-of-tree in `asap-fusion` repo (research project with independent benchmark cadence; depends on `asap-control-core` + `asap-control-optimizer` as published git tags). - -Future deployment models choose whichever placement fits the team that owns them. - -### Asymmetric dependency - -`deployment-model-asaplifecycle` depends on `deployment-model-asapquery` because the `StreamingConfig` YAML emitter is authoritative there. When both live in ASAPController workspace this is a trivial `path =` dep. If one is ever moved out-of-tree, we lift the emitter into core to avoid a cross-repo Cargo dep. - -## 9. Extension point — a 4th deployment model - -With the L4/L5 framework in core, adding a new deployment model is mostly picking + a bit of glue. A hypothetical "edge caching" deployment model that decides which queries to cache at the edge vs. backend would land as: - -``` -crates/deployment-model-edge-cache/ # or your own repo -├── Cargo.toml # depends on asap-control-core -├── src/ -│ ├── lib.rs # impl DeploymentModel for EdgeCacheDeploymentModel -│ ├── rules.rs # pick core rules + add EdgeCacheBindRule -│ ├── topology.rs # TwoStage { edge, backend } TopologyDescriptor -│ ├── cost.rs # impl CostModel — hit ratio × bandwidth -│ └── emit/ -│ └── edge_agent_config.rs # impl PlanEmitter — emits edge-agent YAML -└── tests/ -``` - -Total touch outside the new crate: -- 1 line in `bin/asap-controller/main.rs` to register (if in-workspace) OR a published binary in your own repo -- 1 line in workspace `Cargo.toml` (if in-workspace) -- optional: add `DeploymentModelId::EdgeCache` to `core::registry` - -Typical size: ~500-2000 LOC depending on how deployment-model-specific the rules / cost model / emitter are. If the deployment model accepts core's defaults everywhere, ~300 LOC is realistic. - -### Standalone binary for one deployment model - -If you want a slim binary that runs only one deployment model: - -``` -bin/asap-edge-cache/ -├── Cargo.toml # depends only on runtime + deployment-model-edge-cache, not other deployment models -└── src/main.rs # registers only EdgeCacheDeploymentModel; no DataFusion, no query YAML -``` - -Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql-parser` unless this deployment model needs it. Useful when a deployment is dedicated to one deployment model (e.g. an edge-caching-only service). - -## 10. Wire protocols — what changes, what doesn't - -**Unchanged** (hard contract with other systems): -- `POST /api/v1/streaming-config` on ASAPQuery-backend — consumed YAML format -- `POST /api/v1/plan` on DC controller (from backend capability-miss) — request JSON -- OpAMP `ServerToAgent.remote_config` payload — OTel collector YAML -- Prometheus scrape format - -**Internal** (controller's own surface, still HTTP+JSON for now): -- `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artifacts (URIs). `/api/v1/plan` proxies to this. -- `GET /plans/:id` — plan inspection -- `GET /status` — runtime + deployment model health - -**New** (internal-only, proto): -- `proto/asap_control.proto` defines `Plan`, `PlanNode`, `Expr` for persistence + store-internal serialization. Not on the wire between services. Optional; initial migration skips this and persists via `serde_json`. - -## 11. Dependencies and Cargo surface - -Workspace `Cargo.toml`: - -```toml -[workspace] -resolver = "2" -members = [ - "crates/core", - "crates/runtime", - "crates/deployment-model-asaplifecycle", - "crates/deployment-model-asapquery", - "crates/deployment-model-asapfusion", - "crates/control-proto", - "crates/testing", - "bin/asap-controller", - "bin/asap-query", -] - -[workspace.dependencies] -tokio = "1" -axum = "0.7" -reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } -prost = "0.13" -tonic = "0.12" -serde = { version = "1", features = ["derive"] } -serde_yaml = "0.9" -tracing = "0.1" -thiserror = "1" -# deployment-model-specific deps kept in deployment model crates -``` - -Core depends only on `serde`, `tracing`, `thiserror`, and small utility crates. No HTTP, no DataFusion, no OpAMP. - -Runtime depends on core + `axum` + `reqwest` + `tokio-tungstenite` + `prost` (OpAMP proto). - -Deployment models depend on core. deployment-model-asapquery also pulls in `promql-parser`, `sqlparser`. deployment-model-asapfusion pulls in `datafusion` + `arrow`. deployment-model-asaplifecycle pulls in `sqlparser` + `promql-parser` + the cost-model math crates. - -This matters: a user who only wants `deployment-model-asapfusion` (e.g., an offline benchmark) gets DataFusion but NOT axum/OpAMP. - -## 12. Open questions - -1. **Do we need a cross-deployment-model cost model?** Today DC's lifecycle planner and asap-planner-rs's query planner have overlapping but not identical cost models. Answer for now: keep them separate in deployment model crates; let them re-converge organically. If a third deployment model needs the same model, lift at that point. - -2. **Where does the backend's `ControllerClient.create_plan` call land?** Initially: HTTP `POST /api/v1/plan` on the controller, handled by `deployment-model-asaplifecycle` (same as today). Long-term: could route by `QuerySpec.deployment_model` to a different planner, but that's a follow-up. - -3. **Which deployment model owns `StreamingConfig` YAML emission?** Both `deployment-model-asaplifecycle` and `deployment-model-asapquery` emit it today (DC's `config/generate_streaming_config_yaml` and `output/generator.rs` in `asap-planner-rs`). Plan: **one emitter in `deployment-model-asapquery`**, called from both. This is why `deployment-model-asaplifecycle` depends on `deployment-model-asapquery` in the dependency graph (asymmetric — query does not depend on lifecycle). - -4. **Do we vendor DataFusion's IR into core?** No. `deployment-model-asapfusion` owns its DataFusion-flavored plan; `core::plan::Plan` stays an enum with a variant that wraps a `datafusion::LogicalPlan` behind a feature flag. Core itself never reaches into DataFusion types. - -5. **OpAMP proto: vendored, or from crates.io?** Today DC vendors. Recommend: keep vendored in `proto/opamp.proto`, generate via `prost-build` in `crates/control-proto`. Same thing DC does today, just moved. - -6. **What happens to ASAPQuery-backend's `asap-planner-rs` directory post-migration?** Deleted. ASAPQuery-backend's docker-compose drops the `asap-planner-rs` init container; the controller's `deployment-model-asapquery` now runs in-process (service mode) or as the `asap-query` CLI (one-shot mode). The ASAPQuery-backend repo shrinks by two directories. - -7. **Versioning?** Start at `0.1.0` on the workspace. Deployment models can rev independently later via per-crate versions, but initially lockstep. - -8. **What if a future deployment model can't fit a tree L2?** L2 is currently a per-language tree, mandatory for every deployment model. asap-planner-rs's Phase 4 conformance cost (reverse-engineering PromQL pattern templates into a `PromqlLogicalPlan` tree) was taken deliberately to keep the architecture uniform. If a future deployment model's source language doesn't map naturally onto a tree (e.g. a constraint-based or dataflow-graph query language), that's the moment to re-examine the L2 contract — the `core::logical_plan` module is a single Rust trait + per-language types, not a deep assumption baked across the codebase. Until then, L2 = tree. - -9. **Deployment model placement — in ASAPController workspace, or in own repo?** Both supported, same architecture either way (see §8 "Deployment model placement"). Default: `deployment-model-asaplifecycle` and `deployment-model-asapquery` in ASAPController (easy cross-deployment-model changes, shared YAML emitter); `deployment-model-asapfusion` in the `asap-fusion` repo (research cadence). A new deployment model picks based on team ownership and release cadence preferences. - -10. **How far do shared L4/L5 rules live in core before they become deployment-model-specific?** The rule of thumb: if ≥2 current deployment models would use it, it lives in `core::optimizer::rules::*`. If only 1 deployment model uses it *and* it depends on deployment-model-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the deployment model crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `deployment-model-asaplifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the deployment model; lift to core once a second deployment model wants it. Don't pre-emptively generalize. - -11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP deployment models)?** Already handled — see §3 "Data-model support" and §6 `core::intent_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP deployment model picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. - -### Resolved during the L3 IR cleanup (see §6 `core::intent_algebra`) - -The following questions came up during review of the previous `QueryExpr` draft and are now settled. Listed here so the trail is visible: - -- **Output of `SketchAgg` vs `WindowedAgg`?** Neither node exists at L3 anymore. `SketchAgg` is an L4 sketch-bound node (`SketchExpr::SketchAgg`) emitted by binding rules; `WindowedAgg` was redundant with `Window` over `Aggregate` and removed. Output of `SketchAgg` is a sketch-typed column carrying the partial state; `SketchEstimate` reads it out into a scalar / vector. -- **Is `TopK` different from `Sort + Limit`?** They overlap on heavy-hitter queries but are different concepts. `TopK` is now an `AggIntent` at L3 (heavy-hitter intent — SpaceSaving / CMS-with-heap / Misra-Gries serve it as a single primitive), while generic `Sort + Limit` survives as `QueryExpr` operators for non-heavy-hitter cases (`ORDER BY name LIMIT 10`). L1→L2→L3 lowering picks the intent form when it recognises a heavy-hitter pattern (`ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`) and the operator form otherwise. -- **What is `JoinSketch`?** A sketch-aware join (KMV / theta / join-sample). Moved to L4 (`SketchExpr::SketchJoin`) so the choice "exact join vs sketch join" is an L4 cost decision against `Join`, not a competing L3 surface. -- **`HistogramQuantile` and `PromQLSubquery` feel out of place** — they were. Both removed from L3. `histogram_quantile` lowers in PromQL L1→L2 to bucket reads + a `Quantile` intent. `[range:resolution]` is a driver that expands into multiple queries during PromQL L1→L2 lowering, not a DAG node. -- **Sketch subtract / delete / estimate operators?** Added: `SketchExpr::SketchSubtract`, `SketchDelete`, `SketchEstimate`. Catalog flags (`subtractable`, `deletable`) gate which sketch families admit them. -- **What's `Dedup` exactly?** SQL `DISTINCT`. Renamed to `Distinct { cols }` and generalised from one column to N. -- **Why differentiate `Quantile` and `QuantileOverTime`?** No reason — the surrounding `Window` already encodes the temporal axis. `QuantileOverTime` removed from `AggIntent`. -- **`WindowedAgg` vs `Window` + `Agg`?** Equivalent; `WindowedAgg` removed. `WindowKind::{Tumbling, Sliding, Session}` lives on the `Window` node so the streaming-window kind is explicit; SQL analytic `OVER (...)` stays on the separate `WindowFunc` node. -- **Need input/output specs more fine-grained than `QueryExpr`?** Done: every L3 edge carries a typed `Schema` (fields + time index + unique-key sets), and §6 lists per-node input/output schemas. - -## 13. Future work - -### Extending the primitive set beyond sketches - -The L4 rule engine + `OptimizerRule` trait are **primitive-agnostic** — nothing in the framework is sketch-specific above the rule library. Future primitive classes land as additional rules against the same trait + additional entries in `CostModel`, not as new translation phases or parallel IRs. Candidates we explicitly anticipate: - -1. **Wavelets** — a sibling approximation family (Haar / DWT + coefficient thresholding). Strong on smooth, low-entropy signals where thresholded coefficient sets dramatically out-compress randomized sketches. Slots in as a new physical alternative for the same `AggIntent` variants sketches serve today (range-sum, heavy-hitter, quantile). No change to L3. - -2. **Reuse / precomputation as first-class primitives.** Materialized views, cached scan results, shared sub-expressions. The cost-model hook already exists (`CostModel::workload_cost` + `ReusedComponent`); the missing piece is rules that *introduce* a reuse node — e.g. "build this aggregate once for `q1`, rewrite `q2` to read from it". Orthogonal to approximation: you can reuse an exact aggregate or an approximate one. - -3. **Other approximation algorithms** — sampling, coresets, online PCA / linear-regression normal equations, naive-Bayes-with-conjugate-priors. Anything with a monoid-shaped build + bounded error fits the same contract a sketch does. - -The guiding principle: **same framework, different rule.** A new primitive class is never a translation-layer change — only a new rule, a new cost-model entry, and (if it introduces a new runtime op) a new physical operator in L5. - -## 14. Glossary - -Terms that are project-specific or that get conflated. Where the term has a Rust counterpart, the type is shown in backticks. - -### Architecture roles - -- **Deployment model** — A concrete bundle of (L4 rule choices) + (L5 topology) + (emitter), packaged as a `deployment-model-*` crate. Three exist today: `asaplifecycle`, `asapquery`, `asapfusion`. -- **Stage** (`StageId`) — A *categorical tier* in the data lifecycle: edge / gateway / backend / in-process. Roles, not instances. Declared by the deployment model's `TopologyDescriptor`. -- **Executor** (`Executor`) — A *concrete runtime instance* occupying a stage: a specific OTel agent, the ASAPQuery-backend process, a DataFusion `SessionContext`. One stage may have N executors (e.g. 50 edge agents). Carries `id`, `stage: StageId`, `capabilities`, `address`. -- **Topology** (`TopologyDescriptor`) — Declares which stages exist + how they connect (3-stage / 1-stage / 0-stage). Categorical, not instance-level. -- **DeploymentConstraints** — Trait object owned by each deployment model. Carries memory budgets, network topology, available sketch backends, and the registered `Executor` list. Threaded through L4 rules and the stage allocator. - -### Layer drivers - -- **RuleEngine** — Generic L4 driver in core (fixed-point iteration + cycle detection + priority ordering). One implementation; same code for every deployment model. -- **OptimizerRule** — Trait for an L4 rewrite. Categories: `PushDown | Fusion | Elim | Bind | StageRouting`. Deployment models pick which rules from `core::optimizer::rules` + add their own. -- **CostModel** — Trait scoring a plan's accuracy / latency / dollars. Generic implementations in core; deployment-model-specific cost models (DC's delta / online / pareto / TCO) live in their crates. -- **StageAllocator** — Generic L5 algorithm in core. Colors the L4-bound DAG by `StageId`. **Stage-granularity only — no executor knowledge.** One implementation; same code for every deployment model. -- **PhysicalPlanner** — Per-deployment-model L5 *driver* (one impl per deployment model). Calls `StageAllocator`, then fans the per-stage sub-DAGs out to executors via `DeploymentConstraints::executors()`, then produces the deployment-specific output type. Compare to allocator: planner = whole L5 driver, allocator = the stage-coloring step it uses. -- **PlanEmitter** — Per-deployment-model trait that serialises the planner's output to its wire format (OpAMP `RemoteConfig` / `streaming_config.yaml` / rewritten DataFusion `LogicalPlan`). - -### IR by layer - -- **L1 — query language** — Raw query string + parser (PromQL / SQL / DataFusion / ElasticDSL). -- **L2 — logical plan** — Per-language relational algebra tree (`PromqlLogicalPlan`, `SqlLogicalPlan`, etc.). -- **L3 — intent algebra** (`QueryExpr`, `core::intent_algebra`) — Symbolic, intent-only IR. No sketch type, no params. `AggIntent` carries accuracy targets only. The "algebra" is the operator surface (`Scan`, `Filter`, `Aggregate`, `Window`, …); the algebra is *intent-only* because no sketches have been bound yet. -- **L4 — sketch algebra** (`SketchExpr`, `core::sketch_algebra`) — Same DAG shape as L3, but sketches now committed (kind + params). Produced by L4 binding rules; consumed by L5 emitters. Note the naming inversion vs. earlier drafts: "sketch algebra" is L4 (where sketches actually live), not L3. -- **L5 — physical plan** — Stage-assigned, executor-targeted, ready to serialize. Produced by `PhysicalPlanner`, written out by `PlanEmitter`. - -### Metadata sources (see §6 "DAG schema, DB schema, sketch catalog") - -- **DAG schema** (`Schema`) — Columns + types + `unique_keys` carried on every L3/L4/L5 edge. Type-checked locally at each node. -- **DB schema** / **source schema** (`SchemaCatalog`) — Data-plane metadata (Prometheus TSDB / SQL `information_schema` / DataFusion catalog). Read by `core::lower::*` for L1→L2 symbol resolution. -- **Sketch catalog** (`SketchCatalog`) — Runtime registry of available primitives: what sketches exist, mergeability, deletability, parameter ranges, accuracy model. Consulted by L4 binding rules. **Not a schema** — a catalog of available primitives, not a description of a stream. - -### Workload + identity - -- **QueryWorkload** — Top-level controller input: one or more `QuerySpec`s plus workload features (batch vs streaming, reuse opportunity, data source). Arrives via HTTP POST, OpAMP capability-miss callback, YAML file, or query-log replay. -- **QuerySpec** — A single query: source-language string + accuracy / latency / cost target. - -## 15. Success criteria - -The migration is done when: - -1. `asap-controller` binary runs and passes DC controller's existing integration tests (OpAMP push, backend config POST, SLA replan). -2. `asap-query` binary takes the same YAML input asap-planner-rs does today and produces byte-identical `streaming_config.yaml` + `inference_config.yaml` (fuzz-test against a corpus of fixtures). -3. ASAPQuery-backend's docker-compose no longer starts `asap-planner-rs`; the controller handles both shapes. -4. `asap-fusion`'s microbenchmarks still run under `deployment-model-asapfusion` with identical numbers. -5. The `DataCollector/controller/`, `ASAPQuery/asap-planner-rs/`, `ASAPQuery-backend/asap-planner-rs/`, and `asap-fusion/` directories are deletable (or already deleted) without breaking any currently-running deployment. -6. A new hypothetical deployment model can be added with zero changes outside its crate + one line in `bin/asap-controller/main.rs`. - -## 16. ADR — Capability consolidation + `algebra/`/`planner/`/`config/` retirement (2026-05) - -This section records two migration steps that brought the controller into its current shape. - -### 16.1 PR #129 — Four-merged-then-cleaned capability tables - -Before PR #129 there were **four** overlapping capability tables in the controller: - -| Source | Location | Shape | -|---|---|---| -| YAML at `controller/sketch_capabilities.yml` | filesystem | per-sketch perf profile, runtime-loaded | -| Compiled-in defaults | `algebra/optimizer.rs::sketch_capability` | per-sketch perf profile, hard-coded | -| `SketchKind` enum | `sketch_algebra/params.rs` | sketch type tag | -| `Capability` / `SketchKindHandle` | `asap_tier_analysis.rs` (PR #128) | query-side dispatch tag invented per-query | - -All four collapsed into one module: `controller/src/sketch_algebra/capability.rs`. The four-way map is now: - -- `SketchCapability` + `SupportedIntent` — per-sketch perf profile, read by L4 cost model + L5 physical planner (formerly the YAML + compiled-in copies). -- `Capability` + `SketchKindHandle` — query-side capability tag, used by `data_plane`'s ASAP-tier reducer (formerly the per-query invention in PR #128). -- `capability_for(intent: &AggIntent) -> Option` — the **semantic** intent → ASAP-tier dispatch bridge. The new signature replaces the older `capability_for(query_func: &str)` string-keyed lookup. PromQL → `intent_algebra::lower` → `AggIntent` → (this fn) → `Capability`. The ASAP-tier analyzer at `controller/src/asap_tier_analysis.rs` is now a thin facade around this single function. -- `default_capability_table()` / `load_capability_overrides()` — compiled-in defaults + YAML override loader. - -The four-table fragmentation reflected partial consolidations that never finished; once `Capability` was the canonical query-side tag (PR #128) the perf-profile + override-loader story had a natural home next to it (PR #129). - -### 16.2 Refactor 2026-05 (`refactor/controller-layered-cleanup`) — `algebra/` / `planner/` / `config/` retirement - -The pre-refactor layout grew organically as the controller absorbed three legacy code paths (the DC `algebra/` IR, the `planner/` cost models, the `config/` per-deployment emitters). Each had its own naming convention and module structure. This refactor restructures `controller/src/` to mirror design.md §5's target module split without splitting into multiple crates. The retirements: - -| Retired path | Replacement | -|---|---| -| `controller/src/algebra/expr.rs` | `controller/src/intent_algebra/relational.rs` | -| `controller/src/algebra/lower.rs` | `controller/src/intent_algebra/lower.rs` (the L2→L3 lowering + converter) | -| `controller/src/algebra/directory.rs` | `controller/src/physical/sketch_catalog.rs` | -| `controller/src/algebra/physical.rs` | `controller/src/physical/planner.rs` | -| `controller/src/algebra/allocator.rs` | `controller/src/physical/allocator.rs` | -| `controller/src/algebra/plan.rs` | `controller/src/physical/plan.rs` | -| `controller/src/algebra/optimizer.rs` | `controller/src/optimizer/engine.rs` | -| `controller/src/planner/cost_model.rs` | `controller/src/optimizer/cost/mod.rs` | -| `controller/src/planner/{delta,online}_cost_model.rs` | `controller/src/optimizer/cost/{delta,online}.rs` | -| `controller/src/planner/{pareto,tco,wire_cost}.rs` | `controller/src/optimizer/cost/{pareto,tco,wire}.rs` | -| `controller/src/planner/rules.rs` | `controller/src/optimizer/rules/mod.rs` | -| `controller/src/planner/baseline_planner.rs` | `controller/src/optimizer/baseline.rs` | -| `controller/src/planner/stage_split.rs` | `controller/src/physical/stage_split.rs` | -| `controller/src/analyzer.rs` | `controller/src/pipeline.rs` | -| `controller/src/stage_split/` | `controller/src/physical/colored_dag/` | -| `controller/src/query_language/` | *(deleted)* — was moved to `controller/src/query_parser/language/`, then removed as unwired scaffolding; the real L1 parsers are `query_parser/{promql,sql}.rs` | -| `controller/src/config/workloads.rs` | `controller/src/workload.rs` | -| `controller/src/config/{stage_config*,otap,telegraf,agent,backend,asapquery_backend,precompute}.rs` | `controller/src/emit/{stage_config,otap,telegraf,agent,backend,asapquery_backend,precompute}.rs` | -| (new) | `controller/src/emit/trait_def.rs` — `PlanEmitter` trait placeholder | -| (new) | `controller/src/optimizer/trait_def.rs` — `OptimizerRule` trait placeholder | -| (new) | `controller/src/deployment_model.rs` — `DeploymentModelRegistry` + `DeploymentModelId` placeholder | -| (new) | `controller/src/physical/topology.rs` — `Topology` descriptor re-export | - -`controller/src/lib.rs` keeps thin `pub use` aliases (`pub use pipeline as analyzer;`, `pub use emit as config;`, `pub mod algebra { ... }`, `pub mod planner { ... }`, `pub use physical::colored_dag as stage_split;`) so the historical paths `controller::analyzer::*`, `controller::config::*`, `controller::algebra::*`, `controller::planner::*`, `controller::stage_split::*` keep resolving for external consumers (the `controller` bin's `use controller::algebra;` etc.) without further source churn. Internal source code is migrated to the new paths. - -**TODOs left from the refactor:** - -- `controller/src/emit/stage_config.rs` (3,020 lines, formerly `config/stage_config.rs`) was moved whole rather than split into `emit/opamp.rs` + `emit/streaming_config.rs` + `emit/inference_config.rs` per design.md §5. The monolith mixes OTel-collector YAML emit, ASAPQuery-backend JSON emit, storage-routing JSON emit, and shared internals; a clean split needs ownership reorganisation, not file renames. Tracked for a follow-up. -- `controller/src/intent_algebra/relational.rs` carries the **L2 relational** `QueryExpr` the `query_parser` front ends emit; `lower.rs` is the single pass that converts it (sketch-fusion folded in) to the canonical `intent_algebra::{query_expr,agg_intent}` L3 types. The sketch-fused `SketchAgg` / `WindowedAgg` variants and the standalone sketch-lowering pass that once produced them have been retired. Several controller modules (query_parser, physical/, optimizer/, emit/) still reference `relational` for shared leaf types; fully *deleting* the L2 tree is gated on the canonical `Predicate` covering the remaining `ScalarExpr` variants — but it is the real, current L2 IR (formerly misnamed `legacy_expr`), not removable debt. -- `controller/src/optimizer/trait_def.rs` (`OptimizerRule`), `controller/src/emit/trait_def.rs` (`PlanEmitter`), `controller/src/deployment_model.rs` (`DeploymentModelRegistry`) ship as placeholders — the existing free-function emitters and concrete rule loops still drive behaviour. Migrating them onto the trait surfaces lands when the per-deployment-model crate split lands. - diff --git a/control_plane/docs/design-compiled-plan-collector-backend-split.md b/control_plane/docs/physical-planning.md similarity index 90% rename from control_plane/docs/design-compiled-plan-collector-backend-split.md rename to control_plane/docs/physical-planning.md index eb67a9c5..33b5e398 100644 --- a/control_plane/docs/design-compiled-plan-collector-backend-split.md +++ b/control_plane/docs/physical-planning.md @@ -1,7 +1,10 @@ -# Compiling one Planner decision into collector and backend plans +# Physical planning for collector and backend execution > Status: proposed > +> MVP relation: required to turn one Planner selection into matching collector +> and backend runtime plans. +> > Scope: the ASAPQuery-backend physical-planning step between ASAPPlanner's > selected post-ASAP workload DAG and the two runtime executors: > ASAPCollector and the ASAPQuery data plane. @@ -175,7 +178,7 @@ collector's complete bootstrap configuration. ## 6. Backend subplan The backend subplan follows -[`design-backend-plan-wire-format.md`](design-backend-plan-wire-format.md). +[`backend-plan.md`](backend-plan.md). It specifies: @@ -229,6 +232,28 @@ Delta is legal only when collector and backend advertise the same state, sequence, and checkpoint semantics. Every payload identifies its plan, materialization, producer, window, sequence, and base/checkpoint. +### Aggregation placement + +Planner's reduction and grouping are logical requirements. The physical +compiler decides where that reduction runs without changing them. For example, +`sum by (region) (rate(http_requests_total[5m]))` may maintain one summary per +`region` at collectors, while a query with no grouping may use one +whole-workload materialization. A per-series or per-group result must never be +silently collapsed into a global result. + +### State representation + +Dense versus sparse state is a physical representation choice, not a new +summary choice. For example, an HLL materialization may use sparse state for +low-cardinality groups and promote to dense state as cardinality grows, but +both representations must preserve the same HLL parameters, merge semantics, +wire compatibility, and accuracy contract. + +The compiler may select a representation only when both producer and consumer +advertise compatible support. Otherwise it uses the declared fallback or +rejects the plan. Representation details such as collector configuration field +names belong in the collector interface, not in this design. + ## 9. Compile and activation sequence 1. Validate the selected DAG against both capability snapshots. diff --git a/control_plane/docs/query-to-edge-scope-and-sparse.md b/control_plane/docs/query-to-edge-scope-and-sparse.md deleted file mode 100644 index 22cbf54a..00000000 --- a/control_plane/docs/query-to-edge-scope-and-sparse.md +++ /dev/null @@ -1,163 +0,0 @@ -# Edge Aggregation Scope (`mode`) and Sparse HLL (`hll_sparse`) - -This note documents how the control plane maps query/workload signals to two -per-metric knobs it emits into the edge agent's fused `asap_edge` processor -config (`processors.asap_edge.metrics[]`): - -1. **`mode`** — the aggregation **SCOPE**: `per_series` (default) vs - `whole_stream`. -2. **`hll_sparse`** — opt-in to the in-memory **sparse HLL** base (bool, - default `false` = dense). - -The edge side of both knobs landed in ASAPCollector: - -- **#471** added `MetricFamily.mode` (operator-facing string `per_series` / - `whole_stream`), plumbed into `precompute.PrecomputeConfig.Scope` - (`AggMode` = `ModePerSeries` / `ModeWholeStream`). #471 *folded the legacy - `GlobalAggregation` bool INTO this scope*: `whole_stream` is the unified - `effectiveScope`, semantically identical to the old empty-`aggregate_by` → - `GlobalAggregation` behaviour. -- **#472** added the sparse-HLL base (`NewHLLWrapperSparse`), selected per-HLL - by `hll_sparse`. The sparse base is *lossless*: it auto-promotes to dense - once enough registers are set, and its serialized output is byte-identical to - dense for the same inputs. Default `false` (dense) = today's behaviour. - -Both knobs are emitted from a single site: `control_plane/src/emit/stage_config.rs`, -in `emit_edge_yaml_asap_edge`'s sketch-family loop (the `e: Mapping` per-metric -entry assembly). The legacy single-processor emitter in `emit/agent.rs` -(`build_processor_block`) does **not** emit a `metrics[]` family map — its -`mode` field is the unrelated `ProcessorMode` (Window/Batch windowing strategy), -not this aggregation scope — so it needs **no** change. - -## `mode` (aggregation scope) - -### The signal: effective grouping × family - -At the emit site each sketch entry already computes its **effective -`aggregate_by`** = the metric's `metric_to_grouping_labels` minus its -`metric_to_item_label` (the inner heavy-hitter dimension is the sketch -*subject*, not a grouping key). Call this `effective_by`. - -- `effective_by` **non-empty** ⇒ the edge keys ONE sketch per group (e.g. per - `zone`). This is `per_series` (per-group) scope. -- `effective_by` **empty** ⇒ the edge factory collapses grouping into a single - attr-less sketch. Per #471 that *is* `whole_stream`. - -But "empty grouping" alone is **not** sufficient: the per-series quantile -families (DDSketch / KLL) also carry no grouping (a `quantile_over_time` reduces -*within* a single series), and an empty grouping there means "no extra keying", -**not** "collapse the whole stream". Emitting `whole_stream` for them would be -wrong. - -So the rule combines grouping **and family role**: - -``` -whole_stream ⇔ effective_by.is_empty() AND family ∈ {HLL, CMS, CountSketch} -``` - -The item-counting / frequency families (HLL = distinct-count, CMS = frequency, -CountSketch = top-k / frequency) with an empty effective grouping are exactly -the *genuinely global* aggregates the planner emits for `count(distinct …)` -(no `by`), global top-k, and global frequency — #471's own `WholeStream` -examples ("distinct-count / global-quantile / grand-total / global-top-k"). - -### What we emit - -- **whole_stream**: emit `mode: whole_stream`. -- **per_series**: emit **nothing**. `per_series` is the edge default (an - omitted/empty `mode` parses to `ModePerSeries`), so omitting keeps the YAML - byte-identical to the pre-#471 emit for every metric that is not a genuine - whole-stream global aggregate. This is the deliberate choice: explicit - `whole_stream` where it matters, implicit `per_series` everywhere else. - -### Back-compat / safety - -`whole_stream` ≡ the folded `GlobalAggregation` (#471's `effectiveScope`), so -emitting it for an HLL/CMS/CountSketch that **today** already produces an empty -`aggregate_by` is semantically identical to current behaviour — no wire-level -change, just an explicit name for the existing collapse. - -Crucially we emit `whole_stream` **only** where `effective_by` is empty. A -heap-bearing CountSketch (warm top-k) or any item-counting family that *does* -carry per-group keying (non-empty `effective_by`) stays `per_series`, so we -never newly collapse a metric that needs per-group sid minting at the backend -(the failure mode the long comment in `emit_edge_yaml_asap_edge` warns about: -empty `aggregate_by` + GlobalAggregation breaks backend sid minting for -attr-less series). We do not *introduce* empty-grouping; we only *name* the -scope the existing emit already produces. - -## `hll_sparse` (sparse HLL base) - -Emitted **only** on HLL-family entries (`SketchKind::Hll`); never on any other -family. - -### The rule (scope-driven) - -``` -hll_sparse = !whole_stream (i.e. true for per_series HLL, false for whole_stream HLL) -``` - -- **whole_stream HLL** → a single high-cardinality instance per metric (distinct - values over the whole stream). It promotes to dense almost immediately, so the - sparse base buys nothing and only adds promotion churn → `hll_sparse: false` - (dense). -- **per_series HLL** → one HLL per group; most groups are low-cardinality (e.g. - distinct `user_id`s per `region`), where the sparse base is a large memory win - and auto-promotes the few hot groups → `hll_sparse: true`. - -Because the sparse base is lossless and auto-promoting, the scope-based default -carries **zero** accuracy or wire risk. - -### Cardinality hint (deferred follow-up) - -The ideal refinement is a per-metric cardinality hint: -`WorkloadCharacteristics.distinct_keys_per_window: Option` -(`control_plane/src/types.rs:45`). When present and **large** — above the -dense-crossover (~4096 non-zero registers ≈ the in-memory promotion point) — even -a `per_series` HLL should go dense (`hll_sparse: false`) to skip the -sparse→dense promotion. When small or absent, sparse. - -**This hint is not reachable at the emit site today.** The only input to -`emit_edge_yaml_asap_edge` is `EdgeStageConfig` -(`control_plane/src/physical/colored_dag/emitter.rs`), which carries no -`WorkloadCharacteristics` and no per-metric cardinality map. Plumbing one -through the L5 stage config is a follow-up. Until then we use the scope-based -default above, which is safe (sparse is lossless). Per the default-OFF safety -rule, we never make a metric sparse when scope can't be determined — but for the -HLL family scope is always determinable here (empty vs non-empty -`effective_by`), so HLL entries always carry an explicit `hll_sparse`. - -## Worked examples - -| Query | Family | grouping_labels | item_label | `effective_by` | `mode` emitted | `aggregate_by` | `hll_sparse` | -|---|---|---|---|---|---|---|---| -| `count(distinct user_id)` (no `by`) | HLL | — | `user_id` | empty | `whole_stream` | (omitted) | `false` (dense) | -| `count by (region)(distinct user_id)` | HLL | `[region]` | `user_id` | `[region]` | (omitted ⇒ per_series) | `[region]` | `true` (sparse) | -| `quantile_over_time(0.99, latency[5m])` (per-series) | DDSketch / KLL | — | — | empty | (omitted ⇒ per_series) | (omitted) | n/a (not HLL) | -| `quantile by (zone)(latency)` | DDSketch | `[zone]` | — | `[zone]` | (omitted ⇒ per_series) | `[zone]` | n/a | -| `topk(10, endpoint_qps)` (global) | CountSketch | — | `endpoint` | empty | `whole_stream` | (omitted) | n/a | -| `topk(10, sum by (zone)(...))` | CountSketch + heap | `[zone]` | `endpoint` | `[zone]` | (omitted ⇒ per_series) | `[zone]` | n/a | -| global frequency `endpoint_request_freq` | CMS | — | `endpoint` | empty | `whole_stream` | (omitted) | n/a | -| `sum by (zone)(http_requests_total)` | Sum (not a sketch) | `[zone]` | — | `[zone]` | (no mode on sum entries) | `[zone]` | n/a | - -Note the byte-stability column: every row that emits no `mode` (the per-series -quantile and per-group cases) keeps its `metrics[]` entry byte-identical to the -pre-#471 emit; only the genuine whole-stream globals gain a `mode:` key, and -only HLL entries gain `hll_sparse:`. - -## Tests - -In `control_plane/src/emit/stage_config.rs`: - -- `fused_asap_edge_emits_single_pipeline_and_metrics_list` (extended): asserts - the global item-counting families (HLL / CountSketch / CMS, no grouping) emit - `mode: whole_stream` and no `aggregate_by`; the per-series quantile families - (DDSketch / KLL) and the sum entry emit no `mode`; the whole-stream HLL emits - `hll_sparse: false`; no non-HLL family carries `hll_sparse`. -- `fused_asap_edge_per_group_hll_is_per_series_and_sparse`: a per-group HLL - (`grouping=[region]`, `item_label=user_id`) emits `aggregate_by: [region]`, - no `mode` (per_series), and `hll_sparse: true`. -- `fused_asap_edge_quantile_only_omits_mode_and_sparse`: byte-stability guard — - a quantile-only (DDSketch + KLL) plan emits neither `mode:` nor `hll_sparse`. -- `fused_asap_edge_keys_are_a_subset_of_asapedgeprocessor_config_go`: extended - to allow `mode` and `hll_sparse` as known `MetricFamily` keys. diff --git a/control_plane/docs/query-to-sketch-translation.md b/control_plane/docs/query-to-sketch-translation.md deleted file mode 100644 index 4d0f0a5a..00000000 --- a/control_plane/docs/query-to-sketch-translation.md +++ /dev/null @@ -1,754 +0,0 @@ -# Query-to-Sketch Translation: How QL Maps to Sketch Execution - -This document explains how a PromQL or SQL query is translated through the -control plane's five-layer architecture and ultimately mapped to sketch-based -distributed execution. - -## 1. Five-Layer Architecture - -The control plane is structured as a five-layer pipeline. Each layer has a -clear input, output, and responsibility: - -``` -Query workloads - │ - │ Layer 1 — Query Language - │ (PromQL, SQL, DataFusion, ElasticDSL, ...) - ▼ -Language-specific AST - │ - │ Layer 2 — Language Logical Plan - │ (each language's own relational/query algebra) - ▼ -Language Logical Plan - │ - │ Layer 3 — Sketch Logical Plan (Sketch Algebra) - │ (language-independent, implementation-independent) - ▼ -Sketch Logical Plan - │ - │ Layer 4 — Sketch Optimizer - │ (rewrite rules on the sketch logical plan) - ▼ -Optimised Sketch Logical Plan - │ - │ Layer 5 — Physical Execution Plan - │ (concrete implementations for a specific deployment) - ▼ -Physical Plan (edge processors, backend sketchDB, backend original DB, object store) -``` - -### What each layer owns - -| Layer | Input | Output | Responsibility | -|---|---|---|---| -| **1. Query Language** | query string | language AST | grammar, parsing | -| **2. Language Logical Plan** | AST | language-specific relational plan | language semantics (PromQL instant/range vectors, SQL frames, Elastic buckets) | -| **3. Sketch Logical Plan** | language plan | sketch algebra tree (`QueryExpr`) | **what** to compute: aggregation intent + accuracy requirement + window semantics — no sketch names, no implementation details | -| **4. Sketch Optimizer** | sketch plan + deployment constraints | optimised sketch plan | cost-aware rewrites: push-down, fusion, elimination, budget-driven deferral — considers physical deployment constraints (memory budgets, network topology, available backends) | -| **5. Physical Plan** | optimised plan + deployment config | executable plan | **how** to execute: edge processors (sketch build), backend sketchDB (merge + query), backend original DB (exact), object store (raw backup) | - -### Key design principle - -**Layers 1–3 are query-language-independent and workload-independent.** -They define *what* to compute without reference to any specific query language, -sketch implementation, or deployment topology. A `Quantile { φ=0.99, accuracy=0.01 }` -intent is the same whether it came from PromQL, SQL, DataFusion, or ElasticDSL, -and whether the deployment is a single node or a 1000-agent fleet. - -**Layer 4 is deployment-constraint-aware.** -The optimizer considers physical deployment constraints — memory budgets per stage, -network bandwidth, available backends — when applying cost-based rewrite rules -(e.g., deferring a sketch from Agent to Backend when the agent memory budget is -exceeded, or fusing TopK when the downstream merge is expensive). - -**Layer 5 is deployment-specific.** -The physical planner commits to concrete implementations based on the specific -setup: edge processors, backend sketchDB, backend original DB, or object store. - -### `AggIntent` — the Layer 3 aggregation vocabulary - -| `AggIntent` variant | Meaning | Physical candidates (Layer 5) | -|---|---|---| -| `Quantile { quantiles, accuracy }` | "I need quantile estimates at these φ values within this error" | DDSketch, KLL, t-digest, PromSketch EHKLL | -| `Cardinality { accuracy }` | "I need a distinct-count estimate within this error" | HLL, UnivMon, PromSketch EHUniv | -| `Frequency { accuracy }` | "I need frequency estimates within this error" | CountSketch, CountMinSketch | -| `Extrema { min, max }` | "I need exact min/max" | ExactMinMax, DDSketch at φ=0/1 | -| `PerPartition { inner, keys }` | "Run inner once per distinct key tuple" | Hydra, per-key sketch instances | -| `Exact(Sum\|Count\|Avg\|Min\|Max)` | "No sketch benefit — exact computation" | Raw passthrough, DB-side | - -The flow: -- **Layers 1–2** (parsers): "this query needs a quantile at φ=0.99" → `Aggregate { Quantile(0.99) }` -- **Layer 3** (lowering): `Aggregate` → `SketchAgg { AggIntent::Quantile }` (shared by all languages) -- **Layer 4** (optimizer): rewrites the plan considering deployment constraints -- **Layer 5** (physical planner): "for this deployment, DDSketch at the edge is cheapest" or "KLL at the backend sketchDB is better for this workload" - -## 2. Sketch Logical Plan: `QueryExpr` (Layer 3) - -`QueryExpr` (`algebra/expr.rs`) is the sketch algebra IR — a **logical plan** that -normalises all query languages into a common algebraic form. - -| | AST (syntax tree) | Logical Plan (QueryExpr) | -|---|---|---| -| **Structure** | Mirrors the grammar | Mirrors relational algebra operators | -| **Semantics** | Preserves syntactic details | Preserves only operator semantics | -| **Sketch types** | N/A | Implementation-independent intents (`AggIntent`) | -| **Language** | Language-specific | Language-independent (shared by SQL, PromQL, etc.) | - -QueryExpr has **25 operator variants** organized into categories: - -**Relational core** — standard relational algebra: -- `Source` — base metric / table (leaf node) -- `Filter { pred, input }` — selection (σ) -- `Project { cols, input }` — projection (π) -- `Aggregate { keys, aggs, having, input }` — grouping + aggregation (γ) -- `Join { kind, pred, left, right }` — relational join (⋈) -- `SetOp { kind, all, left, right }` — UNION / INTERSECT / EXCEPT -- `Sort { keys, input }` — ORDER BY -- `Limit { n, offset, input }` — LIMIT / OFFSET - -**Sketch-specific** — operators that express sketch computation intent: -- `SketchAgg { op: AggIntent, col, input }` — sketch aggregation intent (what, not how) -- `WindowedAgg { agg: AggIntent, window: WindowSpec, col, input }` — bundled window + sketch agg (window defines sketch lifecycle) -- `Partition { keys, input }` — GROUP BY distribution for distributed sketches -- `Dedup { col, input }` — deduplication (absorbed by cardinality sketches) -- `TopK { k, by, input }` — top-K heavy-hitter query -- `Merge { inputs }` — sketch merge (linearity: sketch(A∪B) = merge(sketch(A), sketch(B))) -- `JoinSketch { join_key, outer, inner }` — sketch-aware join push-down - -**Time / streaming** — window operators: -- `Window { duration, slide, input }` — standalone time window (batching) - -**PromQL-specific** — operators that preserve PromQL semantics: -- `HistogramQuantile { phi, input }` — `histogram_quantile(φ, …)` -- `PromQLSubquery { range, resolution, input }` — `expr[range:resolution]` -- `BinaryOp { op, lhs, rhs, vector_match }` — vector binary arithmetic with matching - -**Structural** — subqueries and bindings: -- `Subquery`, `LetBinding`, `Ref`, `WindowFunc` - -### Window operators: `Window` vs `WindowedAgg` - -| Operator | Use | Why separate | -|---|---|---| -| `Window { duration, slide }` | Standalone time batching (no sketch) | Used when the sketch op is a separate `SketchAgg` child node | -| `WindowedAgg { agg, window, col }` | Bundled window + sketch aggregation | In sketch systems the window defines the sketch lifecycle (when to flush/reset). Bundling lets the physical planner choose the best implementation (edge tumbling flush vs backend sketchDB EH vs original DB time_bucket). | - -`WindowSpec` supports five window kinds: - -| `WindowKind` | Semantics | Example | -|---|---|---| -| `Tumbling { size }` | Fixed-size, non-overlapping | PromQL implicit, SQL `TUMBLE(ts, '5m')`, Elastic `fixed_interval` | -| `Sliding { size, slide }` | Fixed-size, overlapping | PromQL `[5m]` range vector, SQL `HOP(ts, '1m', '5m')` | -| `Unbounded` | All samples, no time dimension | SQL `GROUP BY key` without time | -| `Landmark` | From epoch to now (cumulative) | Running aggregates | -| `Session { gap }` | Gap-based, closes after inactivity | Elastic session windows | - -## 3. Layers 1–3: Query Language → Language Plan → Sketch Algebra - -### 3.1 Layer 1→2: Language AST → Language Logical Plan - -Each parser takes a language-specific AST (from an external crate) and produces -a **language logical plan** using relational operators (`Aggregate`, `Window`, -`Filter`, `Sort`, `Limit`, etc.) with generic `AggFunc` variants — no sketch -names at this layer. - -**PromQL** (`query_parser/promql.rs`): - -``` -PromQL AST (promql-parser crate) - ↓ walk_qe(ast_node, ctx) -Language Logical Plan (Aggregate + AggFunc + Window) -``` - -The walker carries context downward: `partition` (GROUP BY keys), `topk` (K value), -`outer_count` (whether wrapped in `count(…)`). - -| PromQL construct | Layer 2 output | -|---|---| -| `quantile_over_time(φ, m[5m])` | `Aggregate { Quantile(φ), input: Window { 5m, Filter(Source) } }` | -| `histogram_quantile(φ, rate(…))` | `HistogramQuantile { φ, Aggregate { Quantile(φ), Window(...) } }` | -| `count_over_time(m[5m])` | `Aggregate { Count, input: Window { 5m, Source } }` | -| `avg_over_time(m[5m])` | `Aggregate { Avg, input: Window { 5m, Source } }` | -| `topk(k, …) by (dims)` | `TopK { k, Partition { dims, inner } }` | -| `a + b` | `BinaryOp { Add, lhs, rhs, VectorMatch }` | -| `m[5m:1m]` | `PromQLSubquery { range: 5m, step: 1m, inner }` | - -**SQL** (`query_parser/sql.rs`): - -``` -SQL AST (sqlparser crate) - ↓ extract_select_qe(select, order_by, limit, offset) -Language Logical Plan (Aggregate + AggFunc + Sort + Limit) -``` - -The SQL parser builds the plan bottom-up from SELECT clauses: - -| SQL construct | Layer 2 output | -|---|---| -| `FROM table` | `Source(table)` | -| `WHERE pred` | `Filter(ScalarExpr, Source)` | -| `JOIN … ON` | `Join(kind, pred, left, right)` | -| `GROUP BY keys` + agg functions | `Aggregate { keys, aggs: [AggItem { func }] }` | -| `TUMBLE(ts, INTERVAL '5m')` | `Aggregate { input: Window { 5m, Source } }` | -| `ORDER BY … DESC` | `Sort(keys, input)` | -| `LIMIT n` | `Limit(n, input)` | -| `UNION ALL` | `SetOp(Union, all, left, right)` | - -### 3.2 Layer 2→3: Language Logical Plan → Sketch Algebra (lowering) - -The shared `lower_to_sketch_algebra()` pass (`algebra/lower.rs`) converts -language-independent `Aggregate { AggFunc }` nodes into sketch algebra -`SketchAgg { AggIntent }` nodes. This is the same pass for both PromQL and SQL. - -**Algorithm**: - -``` -lower_to_sketch_algebra(expr): - Recursively walk the QueryExpr tree. - For each single-agg Aggregate node: - - 1. Map AggFunc → AggIntent (implementation-independent): - Quantile(φ) → AggIntent::Quantile { [φ], accuracy } - CountDistinct → AggIntent::Cardinality { accuracy } - Count (w/ GROUP BY) → AggIntent::Frequency { accuracy } - Avg → AggIntent::Quantile { [0.5], accuracy } (median proxy) - Min → AggIntent::Extrema { min: true } - Max → AggIntent::Extrema { max: true } - StdDev → AggIntent::Quantile { [0.25, 0.75], accuracy } (IQR proxy) - Sum/Rate/Delta → AggIntent::Exact(Sum) - Count (no GROUP BY) → stays as Aggregate (no sketch benefit) - - 2. If the Aggregate's input is a Window, fuse into WindowedAgg: - Aggregate { AggFunc, input: Window { duration } } - → WindowedAgg { AggIntent, WindowSpec { Tumbling(duration) }, input } - - 3. If the Aggregate had GROUP BY keys, wrap with Partition: - → Partition { keys, input: SketchAgg/WindowedAgg } - - Multi-agg Aggregates and HAVING clauses pass through unchanged. -``` - -| Layer 2 input | Layer 3 output | -|---|---| -| `Aggregate { Quantile(0.99), Window { 5m, Source } }` | `WindowedAgg { Quantile([0.99]), Tumbling(5m), Source }` | -| `Aggregate { CountDistinct, Source }` | `SketchAgg { Cardinality, Source }` | -| `Aggregate { Count, keys: [region], Source }` | `Partition { [region], SketchAgg { Frequency, Source } }` | -| `Aggregate { Avg, keys: [symbol], Source }` | `Partition { [symbol], SketchAgg { Quantile([0.5]), Source } }` | -| `Aggregate { Sum, Source }` | `SketchAgg { Exact(Sum), Source }` | -| `Aggregate { Count (no GROUP BY), Source }` | unchanged (no sketch benefit) | - -**SQL function → AggFunc mapping**: - -| SQL function | AggFunc | Sketch candidate | -|---|---|---| -| `COUNT(*)` with GROUP BY | `Count` | CountSketch / CountMinSketch | -| `COUNT(*)` without GROUP BY | `Count` | Exact (no sketch benefit) | -| `COUNT(DISTINCT col)` | `CountDistinct` | HLL | -| `SUM(col)` | `Sum` | Exact (not sketchable) | -| `AVG(col)` | `Avg` | DDSketch (p50 proxy) or Exact(Avg) | -| `MIN(col)` | `Min` | DDSketch (φ=0.0) or ExactMinMax | -| `MAX(col)` | `Max` | DDSketch (φ=1.0) or ExactMinMax | - -Note: the parser emits `Aggregate { func: Avg }` — it does **not** emit sketch ops. -Sketch assignment happens later in the optimizer (R9 HydraConversion) and allocator. -The SQL parser only produces relational operators; the PromQL parser is more aggressive -and emits `SketchAgg` nodes directly because PromQL functions like `quantile_over_time` -have a 1-to-1 mapping to sketch types. - -## 4. Concrete Example: PromQL (all 5 layers) - -### Query -```promql -quantile_over_time(0.99, http_request_duration{env="prod"}[5m]) -``` - -### Layer 1 — Language AST - -The `promql-parser` crate parses the string into a PromQL AST: -`Call("quantile_over_time", [NumberLiteral(0.99), MatrixSelector("http_request_duration", {env="prod"}, 5m)])` - -### Layer 2 — Language Logical Plan (parser output) - -The PromQL parser emits **relational operators only** — `Aggregate { AggFunc }` + `Window`, -no sketch names: - -``` -Aggregate { - keys: [], - aggs: [AggItem { func: Quantile(0.99), col: SampleValue }], - input: Window { - duration: 5m, - input: Filter { - pred: Column("env") = Literal("prod"), - input: Source("http_request_duration") - } - } -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -The shared `lower_to_sketch_algebra()` pass converts `Aggregate { Quantile }` to -`AggIntent::Quantile` and fuses with `Window` into `WindowedAgg`: - -``` -WindowedAgg { - agg: Quantile { quantiles: [0.99], accuracy: 0.01 }, - window: WindowSpec { kind: Tumbling { size: 5m } }, - col: SampleValue, - input: Filter { - pred: Column("env") = Literal("prod"), - input: Source("http_request_duration") - } -} -``` - -Note: no sketch implementation names — just "I need a quantile at φ=0.99 with ≤1% error." - -### Layer 4 — Optimizer - -R1 (PredicatePushDown): filter is already below the window — no change. Tree is returned as-is. - -### Layer 5 — Physical Plan - -`physical::plan(expr, config)` produces a `PhysicalNode` tree. For this simple -query, all nodes are at the Agent — no Exchange boundaries: - -``` -OtelSketchBuild { DDSketch, OtelTumblingFlush(5m) } [AgentCollector] - └── Filter { env="prod" } [AgentCollector] - └── OtlpScan [AgentCollector] -``` - -Resolution: `Quantile([0.99], 0.01)` → `DDSketch { relative_accuracy: 0.01, quantiles: [0.99] }`, -`Tumbling(5m)` at AgentCollector → `OtelTumblingFlush { 5m }`. - -### Execution - -1. **Agent** receives raw samples → filters `env="prod"` → batches 5m windows → DDSketch → emit -2. **Backend** merges DDSketches from N agents -3. **Query time**: extract 0.99 quantile from merged DDSketch - ---- - -## 5. Concrete Example: PromQL with Top-K (all 5 layers) - -### Query -```promql -topk by (service) (10, count_over_time(requests{env="prod"}[1m])) -``` - -### Layer 1 — Language AST - -The `promql-parser` crate parses this as: -`Aggregate(op="topk", param=10, modifier=By(["service"]), expr=Call("count_over_time", MatrixSelector("requests", {env="prod"}, 1m)))` - -### Layer 2 — Language Logical Plan (parser output) - -The PromQL parser emits relational operators. The `by (service)` partition keys -are propagated into the inner `Aggregate`'s GROUP BY keys, so the lowering pass -can see `Count WITH GROUP BY` → `Frequency`: - -``` -TopK { - k: 10, - by: ["service"], - input: Aggregate { - keys: ["service"], - aggs: [AggItem { func: Count, col: SampleValue }], - input: Window { - duration: 1m, - input: Filter { - pred: Column("env") = Literal("prod"), - input: Source("requests") - } - } - } -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -`lower_to_sketch_algebra()` converts `Aggregate { Count, keys: ["service"] }` → -`Partition { ["service"], WindowedAgg { Frequency } }`. The `Window + Aggregate` -fuses into `WindowedAgg`: - -``` -TopK { - k: 10, - by: ["service"], - input: Partition { - keys: By(["service"]), - input: WindowedAgg { - agg: Frequency { accuracy: 0.001 }, - window: WindowSpec { kind: Tumbling { size: 1m } }, - col: SampleValue, - input: Filter { - pred: env = "prod", - input: Source("requests") - } - } - } -} -``` - -Note: `Frequency`, not `CountSketch` — implementation-independent. Both CountSketch -and CountMinSketch are valid candidates; the physical planner decides. - -### Layer 4 — Optimizer - -R1 (PredicatePushDown): filter already below window — no change. - -### Layer 5 — Physical Plan - -`physical::plan(expr, config)` produces a multi-stage `PhysicalNode` tree with -Exchange nodes at stage boundaries: - -``` -TopK { k: 10 } [QueryEngine] - └── Exchange { SketchBinary } [QueryEngine] - └── HashAggregate { keys: ["service"] } [BackendCollector] - └── Exchange { Otlp } [BackendCollector] - └── OtelSketchBuild { CountSketch, [AgentCollector] - OtelTumblingFlush(1m) } - └── Filter { env="prod" } [AgentCollector] - └── OtlpScan [AgentCollector] -``` - -Three stages, two Exchange boundaries: -- **Agent → Backend** (Otlp): sketch data flows from agent collectors to merge tier -- **Backend → QueryEngine** (SketchBinary): merged sketches flow to query engine for top-K - -### Execution - -1. **Agent** → filters → builds CountSketch per 1m window → emits via OTLP -2. **Backend** → merges CountSketches per service -3. **QueryEngine** → extracts top-10 services by frequency - ---- - -## 6. Concrete Example: SQL (all 5 layers) - -### Query -```sql -SELECT symbol, AVG(price) FROM trades GROUP BY symbol -``` - -### Layer 1 — Language AST - -`sqlparser` produces: `Select { projection: [Identifier("symbol"), Function(AVG, "price")], from: [Table("trades")], group_by: [Identifier("symbol")] }` - -### Layer 2 — Language Logical Plan - -Both parsers emit the same kind of output — relational `Aggregate { AggFunc }`: - -``` -Aggregate { - keys: ["symbol"], - aggs: [AggItem { func: Avg, col: Named("price"), alias: "avg" }], - having: None, - input: Source("trades") -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -`lower_to_sketch_algebra()` converts `Avg` → `Quantile { [0.5], 0.01 }` (median proxy): - -``` -Partition { - keys: By(["symbol"]), - input: SketchAgg { - op: Quantile { quantiles: [0.5], accuracy: 0.01 }, - col: Named("price"), - input: Source("trades") - } -} -``` - -However, `Avg` is **non-mergeable** (`avg(A∪B) ≠ merge(avg(A), avg(B))`). -The stage-split will route this to DB for exact computation. - -### Layer 4 — Optimizer - -No rewrites applicable. - -### Layer 5 — Physical Plan - -`physical::plan()` assigns the non-mergeable Aggregate to the Database: - -``` -DbQuery { GROUP BY ["symbol"] } [Database] - └── Exchange { RawSamples } [Database] - └── OtlpScan [AgentCollector] -``` - -The Agent passes raw samples through to the Database, which computes exact AVG. - ---- - -## 7. Concrete Example: SQL with TUMBLE window (all 5 layers) - -### Query -```sql -SELECT region, COUNT(DISTINCT user_id) AS cnt -FROM sessions -GROUP BY region, TUMBLE(ts, INTERVAL '5' MINUTE) -ORDER BY cnt DESC LIMIT 10 -``` - -### Layer 1–2 — Parse to relational operators - -The SQL parser detects `TUMBLE(ts, INTERVAL '5' MINUTE)` in GROUP BY and emits -a `Window` node. `COUNT(DISTINCT user_id)` becomes `AggFunc::CountDistinct`: - -``` -Limit { - n: 10, - input: Sort { - keys: [{ col: "cnt", desc: true }], - input: Aggregate { - keys: ["region"], - aggs: [AggItem { func: CountDistinct, col: Named("user_id"), alias: "cnt" }], - input: Window { - duration: 5m, - input: Source("sessions") - } - } - } -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -`lower_to_sketch_algebra()` converts `CountDistinct` → `Cardinality { 0.01 }` and -fuses `Window + Aggregate` → `WindowedAgg`: - -``` -Limit { - n: 10, - input: Sort { - input: Partition { - keys: By(["region"]), - input: WindowedAgg { - agg: Cardinality { accuracy: 0.01 }, - window: WindowSpec { kind: Tumbling { size: 5m } }, - col: Named("user_id"), - input: Source("sessions") - } - } - } -} -``` - -### Layer 4 — Optimizer - -**R5 (TopKFusion)**: `Limit(10, Sort(desc, ...))` → fused into `TopK { k: 10 }` - -### Layer 5 — Physical Plan - -`physical::plan()` produces a multi-stage tree: - -``` -TopK { k: 10 } [QueryEngine] - └── Exchange { SketchBinary } [QueryEngine] - └── HashAggregate { keys: ["region"] } [BackendCollector] - └── Exchange { Otlp } [BackendCollector] - └── OtelSketchBuild { HLL, [AgentCollector] - OtelTumblingFlush(5m) } - └── OtlpScan [AgentCollector] -``` - -Resolution: `Cardinality(0.01)` → `HLL { precision: 14 }`, `Tumbling(5m)` → `OtelTumblingFlush`. - -### Execution - -1. **Agent**: builds one HLL per region per 5m window → emits via OTLP -2. **Backend**: merges HLLs from N agents (HLL merge = set union) -3. **QueryEngine**: extracts cardinality per region → top 10 - ---- - -## 8. Optimizer: Formulation of the Sketch Placement Problem - -### Optimization Goal - -Given a set of query workloads Q = {q₁, q₂, …, qₙ} and a deployment with -pipeline stages S = {Agent, BackendCollector, BackendDB, OriginalDB, ObjectStore}, -the optimizer solves: - -``` -minimize TotalCost(P) -subject to Accuracy(qᵢ, P) ≤ accuracy_sla(qᵢ) ∀ qᵢ ∈ Q - Latency(qᵢ, P) ≤ latency_sla(qᵢ) ∀ qᵢ ∈ Q - Throughput(qᵢ, P) ≥ throughput_sla(qᵢ) ∀ qᵢ ∈ Q - ResourceUsage(s, P) ≤ Budget(s) ∀ s ∈ S -``` - -where P is the physical plan (sketch type assignment + stage placement + window -configuration for each query operator). - -### Cost Model - -The total cost decomposes into per-stage costs: - -``` -TotalCost(P) = Σ_s [ BandwidthCost(s) + MemoryCost(s) + CPUCost(s) + StorageCost(s) ] -``` - -Each term is the aggregate resource consumption across all queries assigned to -that stage: - -| Cost component | Formula | -|---|---| -| `BandwidthCost(s)` | Σ_q transmission_bytes(sketch(q)) × flush_rate(q) | -| `MemoryCost(s)` | Σ_q memory_per_series(sketch(q)) × series_count(q) | -| `CPUCost(s)` | Σ_q cpu_per_insert(sketch(q)) × samples_per_sec(q) | -| `StorageCost(s)` | Σ_q transmission_bytes(sketch(q)) × retention(q) | - -### Constraints - -**Per-stage resource budgets** — each stage has memory, CPU, disk, and bandwidth limits: - -``` -∀ s ∈ S: - Σ_q memory_per_series(sketch(q, s)) × series_count(q) ≤ s.memory_bytes - Σ_q cpu_per_insert(sketch(q, s)) × samples_per_sec(q) ≤ s.cpu_budget - Σ_q transmission_bytes(sketch(q, s)) × flush_rate(q) ≤ s.bandwidth_budget -``` - -**Accuracy constraint** — sketch error must be within the query's SLA: - -``` -∀ qᵢ: - error(sketch_type(qᵢ), sketch_params(qᵢ)) ≤ accuracy_sla(qᵢ) -``` - -For example: DDSketch with `relative_accuracy = 0.01` guarantees ≤1% relative error -on quantile queries. HLL with `precision = 14` guarantees ≤0.8% relative error -on cardinality. - -**Functional constraint** — the sketch must support the query's aggregation intent: - -``` -∀ qᵢ: - intent(qᵢ) ∈ sketch_capability(sketch_type(qᵢ)).supported_intents -``` - -For example: a `Cardinality` intent can only be served by a sketch with -`SupportedIntent::Cardinality` (HLL, UnivMon), not by DDSketch. - -### Decision Variables - -For each query operator `op` in the plan: - -1. **Sketch type selection**: `sketch_type(op) ∈ candidates(intent(op))` - - Quantile → {DDSketch, KLL} - - Cardinality → {HLL} - - Frequency → {CountSketch, CountMinSketch} - -2. **Stage placement**: `stage(op) ∈ S` - - Subject to `stage_budget(stage(op)).fits(sketch_capability(sketch_type(op)))` - - Deferral chain: Agent → BackendCollector → BackendDB - -3. **Window configuration**: `window(op) ∈ {Tumbling(d), Sliding(d, s), Unbounded}` - - Subject to sketch capability: `sketch_capability(type).supports_sliding_window` - -4. **Delta encoding**: `delta(op) ∈ {true, false}` - - Subject to: `sketch_capability(type).supports_delta` - - Reduces bandwidth at the cost of reconstruction at the receiver - -### Cross-Query Optimization: What to Precompute - -When multiple queries share overlapping time series or aggregation patterns, the -optimizer can amortise costs: - -**Shared sketch reuse**: if q₁ = `quantile_over_time(0.99, m[5m])` and -q₂ = `quantile_over_time(0.5, m[5m])`, a single DDSketch serves both -(DDSketch can answer any quantile from one structure). - -**Precomputation decision**: a query should be precomputed (sketch maintained -continuously) rather than computed on-demand when: - -``` -precompute(q) = true iff repeat_interval(q) < query_latency_sla(q) -``` - -i.e., the query fires more often than the system can recompute it from raw data. -Precomputed sketches are maintained at the Agent and merged at the Backend, -with the Precompute Engine answering queries against the merged state. - -**Multi-query sketch sharing matrix**: for N queries over the same metric, the -optimizer builds a sharing matrix: - -| | DDSketch | HLL | CountSketch | -|---|---|---|---| -| q₁: quantile(0.99) | ✓ serves | ✗ | ✗ | -| q₂: quantile(0.5) | ✓ **shared with q₁** | ✗ | ✗ | -| q₃: count_distinct | ✗ | ✓ serves | ✗ | -| q₄: topk(10) | ✗ | ✗ | ✓ serves | - -One DDSketch instance serves both q₁ and q₂ → memory cost counted once, not twice. - -### Current Implementation - -The optimizer currently solves a simplified version: - -1. **Per-query greedy**: each query is optimised independently (no cross-query sharing yet) -2. **Sketch selection**: `CostModelPlanner` scores all candidates per query, picks cheapest meeting accuracy SLA -3. **Stage placement**: `physical::decide_sketch_placement()` checks `StageBudget::fits(SketchCapability)` per stage in order: Agent → Backend → QueryEngine -4. **Precomputation**: `should_precompute(q)` checks `repeat_interval < latency_sla` - -Future work: -- Global optimisation across queries (shared sketch instances) -- Joint sketch+stage+window optimisation (currently done greedily per dimension) -- Workload-adaptive re-optimisation (replan when query patterns change) - ---- - -## 9. Sketch Directory: Which Sketch for Which Operation? - -The sketch directory (`algebra/directory.rs`) maps aggregation types to candidate -sketch families. The `CostModelPlanner` scores all candidates and picks the -cheapest that meets the accuracy SLA. - -### Candidates per Aggregation Type - -| Aggregation | Candidates (default first) | When non-default is chosen | -|---|---|---| -| Quantile | **DDSketch**, KLL | KLL when memory-constrained | -| Cardinality | **HLL** | Single candidate | -| Frequency | **CountSketch**, CountMinSketch | Based on cost model scoring | - -### Sketch Type → OTel Collector Processor - -| SketchType | Go processor | Key parameters | -|---|---|---| -| DDSketch | `ddsketch` | relative_accuracy, quantiles | -| KLL | `KLL` | k, quantiles | -| HLL | `HLL` | (fixed precision in Go code) | -| CountSketch | `countsketch` | epsilon, delta | -| CountMinSketch | `countmin` | rows, cols, metric_name | - -### Stage Assignment Rules - -| QueryExpr node | Default stage | Deferral trigger | -|---|---|---| -| Source, Filter, Window, SketchAgg | **Agent** | Memory budget exceeded → Backend | -| Partition, Merge, Dedup, Exact(Sum/Count/Min/Max) | **Backend** | Memory exceeded → Precompute | -| TopK, HistogramQuantile, BinaryOp, PromQLSubquery | **Precompute** | — | -| Exact(Avg) | **DB** | Non-mergeable — cannot distribute | - -When an Agent sketch exceeds the memory budget, it is deferred to Backend. -If it also exceeds the Backend budget, it moves to Precompute. Every deferral -is logged in `StagedPlan.deferral_log` for observability. - -## 10. Edge Aggregation Scope (`mode`) and Sparse HLL (`hll_sparse`) - -Once a sketch family is chosen for a metric, two further per-metric knobs the -control plane emits into the edge `asap_edge.metrics[]` config control *how* the -edge aggregates: the aggregation **scope** (`mode`: `per_series` vs -`whole_stream`) and the in-memory **sparse-HLL** opt-in (`hll_sparse`). See -[Edge aggregation scope + sparse HLL](query-to-edge-scope-and-sparse.md) for the -signal-to-config mapping, worked examples, and back-compat reasoning. diff --git a/docs/adding-a-new-sketch.md b/docs/adding-a-new-sketch.md index 31f751a5..06814c0f 100644 --- a/docs/adding-a-new-sketch.md +++ b/docs/adding-a-new-sketch.md @@ -422,7 +422,7 @@ next person trying to add another one: | `ASAPQuery-backend docs/design-sketch-db.md §19.9` | The theoretical accuracy bound formula for `Foo` | | `ASAPQuery-backend docs/design-sketch-db.md §19.10` | The merge propagation rule for `Foo` | | Shared design doc for `Foo` | A short rationale: what query class, why this sketch over alternatives, what the trade-off is | -| `DataCollector controller/docs/query-to-sketch-translation.md` | Where in the five-layer plan `Foo` shows up as a candidate | +| [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) | How `Foo` is represented and selected as a logical summary candidate | ---