diff --git a/control_plane/docs/README.md b/control_plane/docs/README.md new file mode 100644 index 00000000..0c46724c --- /dev/null +++ b/control_plane/docs/README.md @@ -0,0 +1,32 @@ +# ASAPQuery-backend control-plane design + +> Status: active + +## TL;DR + +These documents describe only the design owned by ASAPQuery-backend: its +integration boundary with ASAPPlanner, physical compilation for the collector +and backend, and the plan consumed by the ASAPQuery data plane. + +ASAPPlanner owns PromQL parsing, pre-ASAP and post-ASAP IR, query-to-summary +mapping, accuracy reasoning, workload sharing, candidate generation, and +logical plan selection. Refer to the +[ASAPPlanner repository](https://github.com/ProjectASAP/ASAPPlanner) for those +designs; they are intentionally not repeated here. + +## Documents + +| Document | ASAPQuery-backend-owned scope | +| --- | --- | +| [ASAPPlanner integration](asapplanner-integration.md) | Ownership boundary and integration contract for consuming a selected logical workload plan. | +| [Physical planning](physical-planning.md) | Placement, windows, representation, transmission, and compilation into matching collector and backend plans. | +| [BackendPlan](backend-plan.md) | Versioned contract installed and executed by the ASAPQuery data plane. | + +The corresponding collector-facing plan is documented by +[ASAPCollector](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). + +## Scope rule + +A design belongs here only when ASAPQuery-backend owns the decision or must +enforce the contract at runtime. If ASAPPlanner owns the decision, this +directory links to Planner instead of maintaining a second description. diff --git a/control_plane/docs/asapplanner-integration.md b/control_plane/docs/asapplanner-integration.md new file mode 100644 index 00000000..95a658d4 --- /dev/null +++ b/control_plane/docs/asapplanner-integration.md @@ -0,0 +1,177 @@ +# ASAPQuery-backend integration with ASAPPlanner + +> Status: proposed +> +> MVP relation: required for query planning and control-plane decisions. + +## TL;DR + +ASAPPlanner chooses a logical plan for a query workload. ASAPQuery-backend +turns that selected plan into deployable collector and backend plans, installs +them, and routes queries to the resulting state. + +ASAPQuery-backend does not maintain a second design for parsing PromQL, +building Planner IR, mapping queries to summaries, reasoning about accuracy, +sharing work across queries, or ranking logical candidates. Those designs are +owned by [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner). + +```text +PromQL workload + | + v +ASAPPlanner +selected logical workload plan + | + v +ASAPQuery-backend control plane +physical compilation and activation + | + +-------------------------+ + | | + v v +CollectorPlan BackendPlan +ASAPCollector ASAPQuery data plane +``` + +## Ownership boundary + +### ASAPPlanner + +Planner is the source of truth for: + +- query parsing and semantic IR; +- logical exact and summary-based alternatives; +- summary family, parameters, grouping, and readout semantics; +- accuracy constraints and logical result guarantees; +- workload-wide reuse and common subexpressions; +- logical cost comparison and candidate selection; and +- logical rejection and explanation. + +See ASAPPlanner's own design documents for those semantics. Their types and +rules must be consumed from the pinned Planner revision, not copied into this +repository. + +### ASAPQuery-backend control plane + +The control plane owns: + +- supplying workload context, runtime statistics, and executor capabilities; +- invoking the pinned Planner revision and consuming one selected workload + plan; +- choosing collector/backend placement, physical panes, transmission mode, + and storage routes; +- compiling matching CollectorPlan and BackendPlan artifacts; +- staging, activating, retiring, and rolling back plan versions; and +- exposing planning and activation status to operators. + +These decisions are described in [physical planning](physical-planning.md). + +### ASAPQuery data plane + +The data plane owns: + +- installing [BackendPlan](backend-plan.md); +- validating, ingesting, storing, and merging summary state; +- applying the selected readout and remaining backend-side operators; +- enforcing readiness, freshness, and plan identity; and +- executing an explicit exact fallback when the selected plan requires one. + +### ASAPCollector + +ASAPCollector owns validation and execution of CollectorPlan: observing the +selected metrics, maintaining the requested state, transmitting raw data or +full/delta summaries, and reporting which plan is actually active. Its +interface is documented in the +[ASAPCollector repository](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). + +## Integration contracts + +### Workload request + +ASAPQuery-backend passes the complete workload to Planner so that sharing and +global selection remain possible. A request includes the query expressions, +evaluation shape, requested accuracy, and relevant schema or cost context. +Protocol-specific scheduling and alert state remain outside Planner. + +For example, two dashboard queries that read the same five-minute latency +distribution are planned together. ASAPQuery-backend must not reduce them to +two independent `(metric, sketch)` requests before Planner can identify shared +state. + +### Selected logical plan + +The returned boundary is one selected post-ASAP workload plan from the pinned +Planner API. Shared logical nodes remain shared. Exact fallback is an explicit +part of that plan; absence of a supported summary is not permission for the +backend to invent one. + +For example, Planner may select one DDSketch producer shared by: + +```promql +quantile_over_time(0.50, request_duration_seconds[5m]) +quantile_over_time(0.95, request_duration_seconds[5m]) +``` + +The backend consumes the shared producer and two readouts. It must not select +DDSketch again from the query strings. + +### Physical plans + +One physical compile produces CollectorPlan and BackendPlan from the same +selected logical plan. Both artifacts carry the same plan version and +materialization identities. Independent compilation is invalid because it can +make the collector and data plane disagree about family, parameters, grouping, +windows, or state representation. + +### Runtime evidence + +Delivery acknowledgement is not activation evidence. Query routing changes +only after the backend has installed BackendPlan and each required collector +has reported that it applied the matching CollectorPlan. Emitted state must +also carry identities that the backend can validate. + +## Planner version policy + +All Planner crates used by one backend build are pinned to one immutable +revision. The revision is recorded in compiled plans and diagnostic output. + +An upstream Planner change is adopted only after the backend has deliberately +handled any new IR variant, guarantee, capability, or cost-model requirement. +Unknown variants fail as unsupported; the backend must not pre-copy proposed +Planner types or infer semantics from explain/viewer output. + +Open Planner pull requests are compatibility inputs, not backend design. Their +details stay in Planner and in the implementation change that updates the pin, +instead of becoming a second long-lived specification here. + +## Failure behavior + +Planning or activation fails closed when: + +- the selected logical operation is unsupported by its assigned executor; +- collector and backend plans disagree on any materialization contract; +- a required accuracy guarantee is missing or insufficient; +- full/delta state semantics are incompatible; +- a plan is stale, expired, partially applied, or from another run; or +- exact fallback is required but unavailable. + +The system must not silently choose another summary, loosen an accuracy +requirement, remove grouping, or serve state from a different plan. + +## MVP requirements + +The integration is MVP-ready when one end-to-end run demonstrates that: + +- the submitted PromQL workload reaches Planner as one workload; +- the selected logical plan is preserved through physical compilation; +- matching collector and backend plans are captured as artifacts; +- ASAPCollector reports semantic application of the collector plan; +- the data plane serves supported summary-backed queries from BackendPlan; +- unsupported queries are rejected or use the selected exact fallback; and +- missing, stale, or incompatible plan evidence makes the run fail. + +## Non-goals + +This document does not define Planner IR, query-to-summary mappings, accuracy +algebra, summary algorithms, collector configuration fields, state byte +encoding, or query-engine implementation details. diff --git a/control_plane/docs/backend-plan.md b/control_plane/docs/backend-plan.md new file mode 100644 index 00000000..303c90b7 --- /dev/null +++ b/control_plane/docs/backend-plan.md @@ -0,0 +1,313 @@ +# BackendPlan: control-plane to data-plane contract + +> Status: proposed +> +> MVP relation: required for the ASAPQuery data plane to ingest and query the +> state selected by the control plane. +> +> Scope: the typed runtime contract by which ASAPQuery-backend's control +> plane tells its data plane what summary materializations exist, how they +> are ingested, and which query capabilities they serve. + +## TL;DR + +Planning chooses once; serving reuses that exact decision. + +`BackendPlan` is the backend half of a `CompiledPlan`. It describes the state +that matching CollectorPlans produce and the readout/routing decisions the +data plane must apply. It prevents the data plane from independently guessing +summary family, parameters, grouping, windows, accuracy, or storage. + +```text +selected Planner DAG + | + v +physical compile + / \ +CollectorPlan BackendPlan + | | +summary state ingest + route + readout +``` + +## 1. Why BackendPlan exists + +Control-plane planning and data-plane serving happen in different processes +and at different times. They must nevertheless agree on: + +- which materializations should exist; +- their exact summary state types; +- where their payloads come from; +- which windows and groups they represent; +- which queries/readouts they can answer; +- which guarantees apply; and +- which plan version is active. + +Without a typed plan, serving tends to reconstruct decisions from query text, +hard-coded defaults, or observed storage metadata. That can silently select a +different algorithm or parameters, miss valid state, merge incompatible +state, or serve under the wrong accuracy contract. + +BackendPlan makes the control-plane decision authoritative. + +## 2. Relationship to other plans + +ASAPPlanner supplies the selected logical post-ASAP DAG. ASAPQuery's physical +compiler creates: + +- CollectorPlan, which constructs and transmits materializations; and +- BackendPlan, which validates, stores, routes, merges, and reads them. + +The two are defined together in +[`physical-planning.md`](physical-planning.md). + +BackendPlan is not: + +- a copy of ASAPPlanner's internal DAG; +- a query-language AST; +- a collector configuration; +- a storage inventory discovered after planning; or +- an explain/viewer artifact. + +## 3. Plan envelope + +Every BackendPlan contains: + +| Field | Meaning | +| --- | --- | +| `plan_id` | Identity shared with every CollectorPlan compiled from the same decision. | +| `plan_version` | Ordered version within that plan identity. | +| `activation` | Earliest time this plan may serve queries. | +| `expiry` | Optional time after which this plan is invalid. | +| `backend_compat` | Backend-plan and summary-state schema compatibility identity. | +| `generated_at` | Time the control plane emitted the plan. | +| `planner_revision` | Immutable Planner revision that produced the logical selection. | + +The data plane rejects stale, conflicting, premature, expired, or +incompatible plans. Reapplying the same plan/version/content is idempotent. + +## 4. Materializations + +A materialization describes one logical summary state expected by the +backend. It contains: + +- content-addressed materialization identity; +- bound metric/source and canonical filters; +- summarized value or item label; +- physical/logical window contract; +- reduction and grouping layout; +- exact `SummaryFamilyType`, including concrete algorithm and parameters; +- collector producer references; +- state encoding/schema compatibility; +- storage destination and retention; and +- selected result guarantee when supplied by Planner. + +Exact accumulators and approximate summaries use the same materialization +concept. The family discriminator determines which state and parameters are +valid. An algorithm/parameter mismatch is a decoding or validation failure, +not an untyped configuration bag. + +The materialization identity includes every semantic property required for +safe reuse and merge. Two states with the same metric name but different +filters, reductions, grouping, windows, families, parameters, or encoding are +different materializations. + +## 5. Routing and readout + +Materializations describe what state exists. Routing entries describe what +queries that state can answer. These are separate because one materialization +may serve several readouts or queries. + +For example, one sufficiently accurate quantile materialization may support +p50, p95, and p99 readouts. It remains one maintained state with several +routing entries. + +A routing entry identifies: + +- the query capability/readout it satisfies; +- the materialization used; +- any remaining backend-side operation; +- required grouping/window compatibility; +- the storage tier; and +- exact fallback behavior on a miss. + +Routing never changes the materialization's summary semantics. A capability +match can choose among already valid planned routes; it cannot reinterpret +stored state as another family or parameterization. + +## 6. Two levels of matching + +Serving needs two different matching strengths. + +### Capability matching + +Capability matching answers: + +> Is there a planned materialization that can answer this logical query shape? + +It may allow a compatible family-level or readout-level relationship, such as +using one quantile summary for several ranks or rolling up a mergeable finer +grouping when the plan explicitly permits it. + +### State compatibility matching + +State compatibility answers: + +> Can these exact payloads be decoded, merged, and read under this +> materialization contract? + +This comparison is strict. Family, algorithm, parameters, grouping layout, +window, encoding, plan identity, and materialization identity must agree. + +Capability compatibility never implies state compatibility. The backend may +route a query to a compatible planned materialization, but it may merge only +strictly compatible states. + +## 7. Accuracy guarantees + +`AccuracyTarget` is the requested constraint used during planning. When the +pinned Planner revision provides `ResultGuarantee`, BackendPlan preserves the +selected result's: + +- error metric; +- bound expression; +- failure probability; +- provenance and budget allocation; and +- any unavailable statistic that kept the guarantee unknown. + +The data plane does not recompute or weaken that guarantee. Unknown is not +zero and not exact. A route that requires a guarantee fails when the stored +plan lacks a sufficient one. + +For an explicitly permitted approximate realization of an Exact-requested +TopK query, BackendPlan records the effective approximation target and +guarantee. It must never label that realization exact. + +## 8. Source and payload validation + +Each ingested summary payload identifies: + +- plan ID and version; +- backend compatibility identity; +- materialization ID; +- collector/producer ID; +- window identity; +- state schema version; and +- full/delta sequencing and checkpoint identity when applicable. + +The backend rejects payloads that are unknown, stale, incompatible, out of +sequence, or addressed to another active plan. It does not place them into a +best-effort metric-name bucket. + +A plan may reference several collector producers for one sharded +materialization. The backend merges them only if the materialization contract +and family algebra allow it. + +## 9. Installation and lifecycle + +BackendPlan installation is staged and atomic: + +1. Decode and validate the complete plan. +2. Validate every family, readout, route, source, and storage capability. +3. Build a new routing/index view without mutating the active view. +4. Confirm compatibility with the matching collector subplan. +5. Mark the plan staged until its activation time and collector application + evidence are available. +6. Atomically switch query routing to the new view. +7. Retain the previous view until in-flight readers and the configured drain + horizon finish. + +An invalid update never partially changes routing. The previous unexpired +plan remains available for rollback. + +## 10. Query-time behavior + +At query time, the data plane: + +1. parses/canonicalizes the query only as needed to identify its planned + logical shape; +2. looks up routes installed from BackendPlan; +3. verifies readiness, freshness, grouping, window, and guarantee; +4. fetches strictly compatible states; +5. merges and reads them according to the planned operation; and +6. returns the result or takes the explicit exact fallback. + +It does not invoke Planner candidate search, run a planning cost model, resize +a sketch, or infer a missing parameter. + +## 11. Example + +Suppose the workload contains: + +```promql +quantile_over_time(0.95, request_duration_seconds[5m]) +``` + +The selected and compiled plan may contain one DDSketch materialization with +one-minute panes and a p95 routing/readout entry. BackendPlan records the +DDSketch parameter, per-entity reduction, independent grouping layout, +producer collectors, storage route, window composition, and selected +guarantee. + +When the query arrives, the data plane finds that route, fetches five +compatible panes, merges DDSketch state, and reads p95. It does not run a cost +model to reconsider KLL or choose a new DDSketch parameter. + +## 12. Wire-format principles + +BackendPlan is a typed protobuf contract. + +The schema follows these principles: + +- family-specific values use typed discriminated variants; +- invalid family/parameter combinations are unrepresentable or rejected; +- additive optional fields support controlled rollout; +- unknown required variants fail closed; +- compatibility is explicit through `backend_compat`; +- materialization identity is stable and content-addressed; and +- debug/explain fields are not runtime identity. + +Exact protobuf field numbers and generated-language types belong in the +protocol definition and implementation review, not this design document. + +## 13. Fail-closed behavior + +The plan or query fails when: + +- the plan envelope is stale or incompatible; +- a materialization family/parameter/grouping/window is unsupported; +- a collector source does not match the expected contract; +- state payload identities or sequences are invalid; +- a requested route has no valid materialization; +- a required guarantee is absent or insufficient; +- required state is empty, stale, or incomplete; or +- exact fallback is required but unavailable. + +The backend must not return a plausible summary result from mismatched state, +silently drop missing series, treat missing data as zero, or hide a failure as +a routing miss. + +## 14. Non-goals + +This document does not define: + +- Planner candidate generation or ranking; +- CollectorPlan; +- summary-state byte encoding; +- storage-engine implementation; +- query parser implementation; +- routing-index data structures or performance optimizations; or +- protobuf field numbering. + +## 15. Definition of done + +The BackendPlan design is satisfied when: + +- control plane and data plane share one typed contract; +- every expected collector materialization has an exact backend declaration; +- one materialization can serve several explicit routes/readouts; +- state merge uses strict compatibility; +- guarantees remain Planner-derived and fail closed; +- plan installation and routing switch atomically; +- query serving performs no independent summary planning; +- stale or mismatched payloads are rejected; and +- end-to-end tests prove matching plans serve and mismatched plans fail. diff --git a/control_plane/docs/design-backend-plan-wire-format.md b/control_plane/docs/design-backend-plan-wire-format.md deleted file mode 100644 index 2fb81439..00000000 --- a/control_plane/docs/design-backend-plan-wire-format.md +++ /dev/null @@ -1,281 +0,0 @@ -# `BackendPlan`: the control-plane → data-plane wire contract - -> Scope: the wire format and query-time structure connecting `control_plane` -> (planning) to `data_plane` (serving) in this deployment. This document -> designs the target shape directly — it does not narrate what currently -> exists or what a migration path looks like; see PR history for that. - -## 0. One-sentence version - -`BackendPlan` is the typed message `control_plane` pushes to `data_plane` -describing every materialization it has decided on and what query -capabilities each one answers; `RoutingIndex` is the structure `data_plane` -builds from it and consults at query time — both planning time and serving -time end up reading the *same* decision, instead of serving time -re-deriving one independently. - -## 1. Why this exists: planning decides once, serving must reuse it exactly - -Planning (`control_plane`) and serving (`data_plane`) are different -processes with different jobs — planning picks a summary family and sizes -its parameters from a query shape and an accuracy target, symbolically; -serving walks already-materialized state and answers queries against it. -Those two steps must agree on **exactly** which `(SummaryKind, -SummaryParams)` a given metric's materialization uses, because -`SummaryExecutor::find_candidates` matches on that pair *by strict -equality*, on purpose: this deployment chose exact agreement over silently -serving an answer under a looser accuracy guarantee than what was -actually planned (see `summary_executor.rs::summary_params_match`'s -rationale). - -The wire format is the mechanism that keeps those two sides honest. If -serving time ever has to *guess* what was planned — e.g. re-deriving a -family/params choice from a hardcoded default accuracy, independent of -what a specific metric's workload actually requested — that guess will -occasionally be wrong, and the two sides drift apart silently (a query -that should be servable warm capability-misses to archive for no -query-shape reason, only because the guess didn't match reality). The -correct shape is: `control_plane` decides once, writes the decision into -`BackendPlan`, and `data_plane` (via `RoutingIndex`) reads that same -decision back — never re-derives it. - -This has one direct implication for `data_plane`'s serving-time L4 -lowering: it should resolve a query's `(SummaryKind, SummaryParams)` by -looking it up in `RoutingIndex` (built from the `BackendPlan` `control_plane` -already pushed), not by invoking a `CostModel` a second time at query time. -`CostModel::rank_candidates`/`size_params` are a **planning-time-only** -concern — they run once, when `BackendPlan` is built, never again per -query. - -## 2. Goals - -- One wire message, typed end to end, that both sketch-based and exact - materializations populate identically — no "exact means unplanned" - loophole. -- A query-time structure in `data_plane` that resolves "which - materialization answers this query" by direct lookup against what was - actually planned, not by independently re-classifying the query and - hoping the classification matches reality. -- Reuse this deployment's existing canonical vocabulary - (`asap_ir`/`asap_sketch`'s `QueryExpr`/`AggIntent`/`SummaryKind`/ - `SummaryParams`, and `control_plane`'s own `Capability`/`PolicyFingerprint`) - directly. No parallel, wire-specific re-encoding of concepts that already - have a canonical type. - -## 3. `Materialization`: exact and approximate are already the same shape - -`asap_sketch::SummaryKind`/`SummaryParams` already unify "approximate -sketch" and "exact accumulator" into one vocabulary — `Sum`/`Count`/ -`MinMax`/`Increase`/`Rate` are `SummaryKind` variants exactly like `Kll`/ -`DDSketch`/`Hll`/`Cms`, distinguished only by `SummaryKind::is_exact()`. -There is no reason for the wire format to reintroduce a -`Sketch`-vs-`ExactAggregate` split on top of a vocabulary that has already -closed that split. A `Materialization`'s payload is a `(SummaryKind, -SummaryParams)` pair, full stop — whichever family it names decides -whether readout is exact or approximate; the wire schema doesn't need to -know or care. - -```rust -pub struct BackendPlan { - pub plan_id: PlanId, // observability only, not identity - pub generated_at: DateTime, - - /// Every materialization this backend should build/maintain, - /// keyed by content-addressed identity. - pub materializations: HashMap, - - /// Query-time capability routing — see §4 for why this is a - /// separate table rather than 1:1 with `materializations`. - pub routing: Vec, - - pub monitors: Vec, // CDM, unchanged concept -} - -pub struct Materialization { - pub fingerprint: PolicyFingerprint, - - // Reuses asap_ir/asap_sketch types directly — no re-flattening into - // ad-hoc string/HashMap fields. - pub source: Source, // metric + label filter - pub window: WindowSpec, // kind / size / slide - pub group_by: Vec, - pub rollup: Vec, - - /// What this materialization actually is. `kind.is_exact()` tells - /// readers whether this is an exact accumulator or an approximate - /// sketch — no separate enum arm needed for that distinction. - pub kind: SummaryKind, - pub params: SummaryParams, - pub col: ColumnRef, - - pub retention: Option, -} - -pub struct RoutingEntry { - pub satisfies: Capability, // control_plane's own coarse, - // family-level routing vocabulary - pub materialization: PolicyFingerprint, - pub storage_backend: StorageBackend, -} -``` - -**Why `routing` is a separate table from `materializations`, not a 1:1 -map:** one materialization can satisfy several query capabilities — a -single KLL(k=200) sketch built for p99 can also answer p50/p95, and -`Capability::is_satisfied_by` already does family-wildcard matching -(`QuantileApprox(Any)` vs `QuantileApprox(DDSketch)`). Binding "what's -built" to "what it can answer" 1:1 can't express that reuse; a separate -routing table can — adding a new answerable shape for an existing -materialization is one new `RoutingEntry`, not a change to the -materialization itself. - -**Why `Capability` still exists as its own type, distinct from -`(SummaryKind, SummaryParams)`:** `Capability` is the *family-level* -question ("does anything at all answer a `QuantileApprox` shape for this -metric") used for coarse routing decisions — miss detection, archive -fallback, "should I even try the sketch tier." `(SummaryKind, -SummaryParams)` is the *exact* question `SummaryExecutor::find_candidates` -needs — because two candidates must agree exactly to be legally mergeable -via `merge_states`, family-level compatibility alone isn't enough to -decide that. These are genuinely different match precisions for genuinely -different callers (see §4) — collapsing them into one would either make -routing too strict (rejecting a real family match on an incidental params -difference) or make serving too loose (merging things that don't actually -agree). - -## 4. `RoutingIndex`: query-time structure in `data_plane` - -Built fresh from each `BackendPlan` push, atomically swapped -(`arc-swap`) so a config update never serves from a half-updated index. - -**Tier 1 — exact fingerprint, O(1).** Compute the incoming query's own -shape fingerprint using the same hash function as `PolicyFingerprint` -(metric + agg shape + params + grouping/rollup labels + window + -normalized filter) and look it up directly in `materializations`. Covers -the common case — a query structurally identical to something -`control_plane` already planned (the majority of fixed-dashboard traffic). - -**Tier 2 — structural/capability match, on Tier-1 miss.** - -```rust -pub struct RoutingIndex { - exact: HashMap, // Tier 1 - by_metric: HashMap, // Tier 2 -} - -/// Columnar, not `Vec` nested in a `HashMap` — see §4.1. -struct MetricBucket { - group_shape_ids: Vec, - filter_ids: Vec, - windows: Vec, - capabilities: Vec, - kinds: Vec, - params: Vec, - fingerprints: Vec, - storage_backends: Vec, -} -``` - -Match algorithm on Tier-1 miss: - -1. `by_metric.get(metric_id)`. -2. Filter `group_shape_ids` for rows equal to, or a rollup-derivable parent - of, the query's requested group-by. -3. Filter remaining rows by window compatibility — exact match, or the - existing window-merge/closest-pane logic - (`storage_engines/sketch_db/query/window_merger.rs`). -4. Filter remaining rows for filter *subsumption* (materialization's - baked-in filter ⊆ query's filter — not equality). -5. **Match precision depends on the caller** — this is the one place - `RoutingIndex` has two genuinely different read modes, not a single - shared one: - - **Whole-query resolution** (deciding whether this query can be - served warm at all, and which storage backend to route to): match - `Capability::is_satisfied_by(query_agg_intent, accuracy)` — - family-level. - - **`SummaryExecutor::find_candidates`** (per-`L4Node`-leaf, called - during `asap_sketch::exec::execute()`'s walk): match exact - `(SummaryKind, SummaryParams)` — required for anything that can feed - a `SummaryMerge`, and the reason `l4_lowering.rs` no longer needs to - independently observe or guess this (see §5). -6. **Whole-query resolution only:** if more than one candidate survives, - rank by error bound / storage-tier cost and return the single winner. - **`find_candidates` only:** skip ranking — return every row that - survived step 5's exact match, so `execute()` can fold them all via - `merge_states`. -7. No survivors → archive/Thanos fallback routing (whole-query mode), or - `ExecError::NoCandidates` (`find_candidates` mode) — two callers of the - same "empty" outcome. - -### 4.1 Why columnar + interned ids, not nested `HashMap, Vec<_>>` - -Consistency with an existing, already production-validated pattern one -layer down: `storage_engines/sketch_db/index/epoch_columnar.rs` already -does exactly this for the physical sketch index — `LabelValuesId = u32` -interns each group-by label-values vector so the hot loop compares a -4-byte int instead of walking a `BTreeMap`, and the epoch -store itself is parallel arrays ("range scan touches only `windows_col`"). -`RoutingIndex` should use `GroupShapeId`/`FilterId` interning and -parallel-array storage per `MetricBucket` for the same reason: at the -scale of thousands of planned materializations per metric, Tier-2 lookup -is effectively a small in-memory OLAP scan, not a handful of hash lookups, -and should be built that way rather than reinvented as nested hash maps. - -## 5. What this replaces at serving time - -Today, `data_plane`'s serving-time L4 lowering (`l4_lowering.rs`) parses -the raw query string down to a canonical `QueryExpr`, then has to -*independently reconstruct* which `(SummaryKind, SummaryParams)` a -metric's registered sid actually uses by inspecting the `SketchStore`'s -own metadata (`ObservedFamilyCostModel`) before it can bind an `L4Node` -that `find_candidates` will actually match. That's a real, working -mechanism, but it's inherently a *reconstruction* — it infers the plan -from its side effect (what got registered), rather than reading the plan -directly. - -Once `RoutingIndex` exists, serving-time lowering simplifies to: parse to -`QueryExpr` (L1-L3, still genuinely needed — a query's *shape* has to be -recovered from its text regardless of any wire format), then resolve the -query's `(SummaryKind, SummaryParams)` via `RoutingIndex`'s -`find_candidates`-mode lookup directly, and construct the `L4Node` from -that pair — no `CostModel::rank_candidates`/`size_params` call at serving -time at all. `CostModel` becomes exactly what its name says: a -planning-time cost model, invoked once when `BackendPlan` is built, never -re-invoked per query. `ObservedFamilyCostModel`'s SketchStore-introspection -approach was always a stopgap for the absence of this lookup, not a -replacement for it. - -## 6. Transport - -**Proto, not YAML/JSON.** `control_plane` and `data_plane` deploy -atomically in this deployment (no independent rollout, no coordinated -upgrade window), so there's no wire-compatibility constraint across -versions to protect. `SummaryParams`/`SummaryKind` become proper `oneof`s, -checked at compile time on both ends — no untyped `parameters: -HashMap` bag to fall back into. The project already has -proto precedent (`asap_otel_proto` for OTLP ingest). - -## 7. Open questions - -- **Query-side fingerprinting.** Does Tier-1 matching reuse - `PolicyFingerprint::from_config` verbatim, or does an incoming query - need a distinct fingerprint function (it carries no `original_yaml` or - `num_aggregates_to_retain` — both already excluded from the hash per - `policy_fingerprint.rs`'s own doc comment, so likely fine as-is, but - worth verifying rather than assuming). -- **Rollup algebra.** Step 2 above assumes "a materialization grouped by - `(zone, region)` can answer a query grouped by `(zone)` alone" is a - known-safe operation gated by the `rollup` field. The precise algebra — - which `SummaryKind`s roll up safely (`Sum`/`Count`-family: yes; - `Quantile`: generally no without re-estimation error) — needs its own - short design pass before this step can be implemented as described. -- **Exact top-k has no `SummaryKind` to name.** `asap_sketch::SummaryKind` - has no exact (non-approximate) top-k variant — `TopK` only exists as an - approximate family (`CmsWithHeap`/`CountSketchWithHeap`). A - `Materialization` for `topk(k, metric)` at `accuracy: Exact` can't be - expressed in this schema until that vocabulary gap closes upstream (see - the tracked issue for this — an ASAPController-side vocabulary - extension, not a `BackendPlan` schema question). -- **`RoutingIndex` performance.** New query-time hot path in a - latency-sensitive service — needs a benchmark pass against the current - lookup, not just a correctness pass, before it can replace anything. diff --git a/control_plane/docs/design-target-architecture.md b/control_plane/docs/design-target-architecture.md deleted file mode 100644 index 3c3c5b09..00000000 --- a/control_plane/docs/design-target-architecture.md +++ /dev/null @@ -1,303 +0,0 @@ -# `control_plane` target architecture — planning-time and serving-time - -> **Status.** Target-state design, written independent of what's currently -> implemented in `control_plane`/`data_plane`. Grounded entirely in -> ASAPController's own current interfaces — specifically -> [`docs/l1-query-language.md`](https://github.com/ProjectASAP/ASAPController/blob/main/docs/l1-query-language.md) -> through -> [`l5-physical-plan.md`](https://github.com/ProjectASAP/ASAPController/blob/main/docs/l5-physical-plan.md)'s -> `## Interface` sections, added in -> [ASAPController#169](https://github.com/ProjectASAP/ASAPController/pull/169) -> (every signature there verified against ASAPController `main` at -> `cc18c98`, 2026-07-28) — not carried over from this repo's own history or -> assumed from memory. Where this doc says "should," it is describing a -> target, not certifying that the current code already matches it; see §4 -> for the gap. - -## 0. The one-sentence version - -`control_plane` should be a **thin planning-time shell** around -ASAPController's `asap-ir`/`asap-l2`/`asap-plan`/`asap-sketch` crates, -contributing exactly two things ASAPController doesn't ship: an L5 -physical-planner implementation (ASAPController has no `asap-physical` -crate — L5 is speculative there, real here) and deployment-specific L4 -extension points (`CostModel`, `Matcher`). `data_plane` should be a thin -**serving-time shell**, contributing exactly one thing: a `SummaryExecutor` -implementation. Planning and serving are two separate interfaces per -ASAPController's own `design.md` §"Serving-time execution" — this doc -treats them as two separate sections for the same reason. - -## 1. Planning vs. serving — the split ASAPController's own design draws - -```mermaid -flowchart LR - Q["query string"] --> P["PLANNING\nL1 → L2 → L3 → L4"] - P --> L4N["L4Node\n(symbolic plan)"] - L4N --> E["SERVING\nSummaryExecutor::execute"] - E --> V["answer"] -``` - -- **Planning** turns a query string into a plan. It has, by design, no - reference to what's actually materialized anywhere — it symbolically - picks a summary family and parameters from the query shape and an - accuracy target alone. -- **Serving** walks an already-decided plan against whatever is actually - materialized *right now* and produces an answer. Reality can diverge - from the plan in ways planning never sees — missing data, multiple - instances needing a merge, instances that disagree on parameters — - which is why serving needs its own error vocabulary, distinct from - planning's. -- In this deployment, `control_plane` runs **in-process** with - `data_plane` (same binary), so calling `control_plane`'s planning entry - point from `data_plane` at request time is a same-binary library call, - not a network hop or a second planning implementation living in - `data_plane`. That's a deployment-specific convenience this topology - affords, not something ASAPController assumes of every deployment. - -## 2. Planning-time — `control_plane`'s job - -### L1 — query language - -ASAPController's L1 interface (no shared trait; converging free functions): - -```rust -pub fn lower_promql(query: &str, accuracy: AccuracyTarget) -> Result; -pub async fn lower_sql(query: &str, catalog: &SqlCatalog, accuracy: AccuracyTarget) -> Result; -``` - -**Target: `control_plane` calls `asap_frontend_promql::lower_promql` directly.** -No local PromQL parser. SQL support (currently 0% adopted in this repo) is -`asap_frontend_sql::lower_sql` the same way, gated behind whatever schema -catalog this deployment can supply (`SqlCatalog` — see L2 below; PromQL has -none, SQL needs a real one). - -### L2 — logical plan / schema binding - -```rust -pub trait SchemaCatalog { fn columns_for(&self, source: &str) -> Option>; } -pub struct Binder; -pub fn convert_root(legacy: &QueryExpr, accuracy: &AccuracyTarget) -> Result; -``` - -**Target: no `control_plane`-local `SchemaCatalog` for PromQL** — this -deployment's metrics have no real catalog (a metric's label set is only -knowable from what a query references), so the default `Binder::default()` -(`UsageDerivedCatalog`) is correct as-is, same as ASAPController's own -PromQL front end uses. `convert_root` is called *inside* -`lower_promql`/`lower_sql` already — `control_plane` never calls it -directly. - -### L3 — intent algebra - -The canonical `QueryExpr`/`AggIntent`/`Schema` vocabulary (see PR #169's -`l3-intent-algebra.md` diff for the full ~30-variant `AggIntent` list this -deployment should treat as canonical, not its own subset). - -**Target: `control_plane` has zero local L3 type definitions.** Every -`QueryExpr`/`AggIntent`/`Schema` reference is `asap_ir::intent_algebra::*`, -used directly — no wrapper types, no local re-derivation of accuracy-math -helpers (`hll_accuracy`, `default_cardinality`, etc. — those already live -in `asap_ir` and should be called there, not duplicated locally). - -**One acknowledged, deliberate exception**, matching ASAPController's own -design rule ("core treats `Extension` opaquely; the owning deployment -defines and interprets `payload` itself"): this deployment's `Frequency` -capability (`AggIntent::Extension{ext_kind: "frequency", ..}` — per-series -sample-count estimation via CMS/CountSketch, with no upstream equivalent) -is genuinely deployment-specific vocabulary layered *on top of* the -canonical `AggIntent`, not a competing definition of it. It's implemented -as free functions (`frequency()`/`as_frequency()`) wrapping the foreign -`AggIntent` type (Rust's orphan rules require this), not a fork of the -type itself. - -### L4 — cost-aware binding + summary-bound IR - -```rust -pub fn implement_tree_with(expr: &QueryExpr, cost_model: &dyn CostModel) -> Result, ImplementError>; -pub trait CostModel { - fn rank_candidates(&self, intent: &AggIntent, candidates: &[SummaryKind]) -> Vec; - fn size_params(&self, kind: SummaryKind, intent: &AggIntent, eps: f64, delta: f64) -> SummaryParams { .. } - fn realize_extension(&self, ext_kind: &str, payload: &serde_json::Value) -> Implementation { .. } - fn readout_extension(&self, ext_kind: &str, payload: &serde_json::Value, col: &ColumnRef) -> SketchQuery { .. } -} -``` - -**Target: `control_plane` owns exactly one `CostModel` impl and nothing -else at this layer.** `implement_tree_with` + the `L4Node`/`SummaryExpr` -IR are pure ASAPController types, used directly, not wrapped. The -`CostModel` is the single deployment-specific extension point this layer -grants: - -- `rank_candidates` — this deployment's accuracy-bound-driven family - preference (KLL vs. DDSketch for quantiles, etc.). -- `size_params` — real parameter sizing from an accuracy target, not the - crate's built-in default formulas (this deployment has its own - calibrated cost table). -- `realize_extension`/`readout_extension` — the two hooks that make the - `Frequency` extension (§ above) actually work: turning - `Extension{"frequency", ..}` into a real `Implementation::Summary{Cms/CountSketch,..}` - and answering a readout against it. This is *the* mechanism, not a - workaround — ASAPController's `CostModel` trait exists specifically so a - deployment can plug in exactly this kind of opaque-extension realization - without touching `asap-plan`'s binding pass. - -No local `PhysicalExpr`/`L4Plan` wrapper type is needed for the IR itself -— `Rc` is the L4 output, used directly by L5 below. (A thin L5 -*placement* wrapper is still legitimate — see next section — but it should -wrap `Rc`, not reinvent the L4 algebra inside it.) - -### L5 — physical plan (control_plane's real, load-bearing contribution) - -ASAPController has **no `asap-physical` crate** — confirmed: `crates/` on -`main` is `{ir, l2, sketch, plan, frontend-promql, frontend-sql, lower, -e2e}`, no fifth entry. L5's interface section in `l5-physical-plan.md` is -explicitly marked speculative — "a concrete target to design against, not -an API to depend on." **This is the one layer `control_plane` should own -in full**, implemented *against* that speculative shape so a future real -`asap-physical` crate absorbs it with minimal churn, not built to some -other, incompatible pattern: - -```rust -pub trait PhysicalPlanner { - type Topology: TopologyDescriptor; - type Output; - fn lower(&self, l4: Rc, t: &Self::Topology) -> Result; -} -pub trait TopologyDescriptor { - fn stages(&self) -> &[StageDescriptor]; - fn edges(&self) -> &[StageEdge]; -} -pub struct StageAllocator; -impl StageAllocator { - pub fn allocate(&self, l4: Rc, topology: &T, c: &DeploymentConstraints) - -> Result, PlanError>; -} -``` - -Concretely, for this deployment: - -- **`Topology`**: a 3-stage topology (`edge` / `gateway` / `backend`) — - this repo's existing `physical::colored_dag` machinery is structurally - the right shape (`StageId`, per-stage allocator, per-stage emitter); - it should be read as *this deployment's* `TopologyDescriptor` + - `StageAllocator` + `PhysicalPlanner` impl, not a bespoke thing outside - ASAPController's pattern. -- **`Executor`**: an OpAMP-addressed agent/gateway process or an HTTP-addressed - backend — `address: ExecutorAddr` maps onto this repo's existing - OpAMP-connection / backend-HTTP-endpoint machinery. -- **Output** (`PhysicalPlanner::Output`): per-executor configuration — - OpAMP `RemoteConfig` (OTel-collector YAML) for edge/gateway executors, - `StreamingConfig` JSON POST for the backend executor. This is exactly - what `emit::{emit_edge_yaml, emit_gateway_yaml, emit_backend_streaming_config_json}` - already produce — L5's job is *stage allocation + emission format*, and - that's genuinely this repo's own concern, not something to strip out or - route through ASAPController. - -**This is the load-bearing implication of "no `asap-physical` crate -exists": `control_plane`'s physical-planning code is not legacy debt to -retire — it's the actual product.** Everything upstream of it (L1-L4) -should shrink toward zero local code; L5 should not. - -## 3. Serving-time — `data_plane`'s job - -```rust -pub trait SummaryExecutor { - type Handle: Clone; type State; type Value; type Error; - type GroupKey: Clone + Ord + Default; - fn find_candidates(&self, summary: &SummaryKind, params: &SummaryParams, col: &ColumnRef, - reduction: &Reduction, child: &L4Node) -> Result, Self::Error>; - fn fetch_state(&self, handle: &Self::Handle) -> Result; - fn merge_states(&self, states: Vec) -> Result; - fn readout(&self, state: &Self::State, query: &SketchQuery) -> Result; - fn logical(&self, expr: &QueryExpr) -> Result; -} -pub fn execute(node: &L4Node, exec: &E) -> Result, ExecError>; -``` - -**Target: `data_plane` implements `SummaryExecutor` once, and that -implementation is the sole live serving path.** `execute()` (upstream, -not deployment code) owns every *structural* rule generically — which -nestings are valid, what must agree for a merge to be legal, how deep -recursion composes. The deployment supplies only storage, summary math, -and readout, through the five trait methods: - -- `find_candidates` — resolve `(SummaryKind, SummaryParams, group_by)` - against this deployment's `SketchStore` index, using `reduction` - (`Reduce(GroupKeys)` vs. `PerEntity`) as the actual, structural signal - for grouping — **not** inferred from an empty key list (this is where - the "empty `by` is ambiguous" problem this repo's own - `data_plane/docs/l4node-plan-executor-design.md` documents gets solved - for real: `reduction` carries the distinction `L4Node` alone doesn't). - Family-compatibility inside this method's own matching logic is exactly - what `sketch_algebra::matcher::SummaryFamilyMatcher` (the `asap_plan::Matcher` - impl) exists for. -- `fetch_state`/`merge_states` — decode + merge sketch/exact-accumulator - state from storage; this repo's existing per-family merge logic - (`AggregateCore::merge_with`, the sketch-family merge helpers) is the - right home, generalized to run through this one trait method instead of - as one-off special cases. -- `readout` — evaluate a `SketchQuery` against merged state; this repo's - existing per-sketch-family readout code is the right home. -- `logical` — evaluate a raw, unrewritten subtree directly (the - `SummaryExpr::Logical` escape hatch) — this deployment's fallback path - for anything that reaches serving time without a summary decision. - -**`storage_engines/sketch_db/query/sketch_reducer.rs` and -`asap_query_engine/shadow_compare.rs` are retired.** Neither was any more -"ground truth" than `SummaryExecutor` itself — `shadow_compare.rs`'s job -(validate `SummaryExecutor` against the legacy reducer before cutover) -was done once the cutover landed (Part A, #427), and keeping the legacy -reducer around after that only meant two independently-planned answering -mechanisms could silently disagree with each other, not that either was -more trustworthy. Shapes `SummaryExecutor` self-excludes before binding -(`rate()`/`irate()`, `topk(K, sum by(...)(rate(...)))`, keyed-CMS -point-estimate, and the outer-exact/summary composition gaps below) now -fail over to archive directly — there is no legacy fallback left, by -design, not because a rollout step is still pending. - -## 4. What this means for current code — gap against this target - -| Layer | Target | Current gap | -|---|---|---| -| L1 | `asap_frontend_promql::lower_promql` called directly; no local parser | `query_parser::parse_query_expr_canonical`/`parse_query` call `lower_promql` directly; `query_parser/promql.rs` (the local parser, ~1187 lines) is deleted. No reconciliation pass — classification (e.g. bare selectors no longer implying `Aggregate{Sum}`) follows `asap-l2`'s lowering as-is. **Closed** (#428). | -| L2 | `Binder::default()` / `convert_root` used via L1, no local schema logic | Already true in substance — `intent_algebra/{binder,column_resolution}.rs` are thin re-export shims. **Effectively closed.** | -| L3 | Zero local `QueryExpr`/`AggIntent`/`Schema` definitions | Already true — `intent_algebra/{agg_intent,query_expr,relational,schema,expr_ir}.rs` are thin re-export shims with only genuinely-local residues (`Frequency` extension helpers, `PerPartitionWrap`, PromQL-ergonomic `LabelFilter`). `intent_algebra/lower.rs` (~1000 lines) remains real local code — deliberately, for two documented reasons with no ASAPController equivalent (multi-agg fusion, the windowed-Count-as-Frequency heuristic). **Effectively closed modulo `lower.rs`'s two documented exceptions.** | -| L4 | One `CostModel` impl; `Rc` used directly | `sketch_algebra::cost_model::ControlPlaneCostModel` + `sketch_algebra::lower::bind_query_expr` (delegating to `implement_tree_in_with`) already match this shape. `sketch_algebra::matcher::SummaryFamilyMatcher` is the `Matcher` impl this section's serving-time §3 depends on. **Effectively closed** — `PhysicalExpr`/`L4Plan` is a thin, acceptable L5-placement wrapper around `Rc`, not a competing L4 algebra. | -| L5 | Full local `PhysicalPlanner`/`TopologyDescriptor`/`StageAllocator` impl | `physical/colored_dag/*` + `emit/*` already implement this shape structurally, just not against the trait names above (no literal `PhysicalPlanner` trait exists in this repo — the free functions/structs are the de facto impl). Low-priority gap: naming/trait-alignment, not missing functionality. | -| Serving | Single `SummaryExecutor` impl is the live path | **Closed.** `data_plane`'s `summary_executor.rs` implements the trait fully and is the default-on, *sole* live path (`ASAP_SUMMARY_EXECUTOR_LIVE` default flipped from off to on, #427; the legacy `sketch_reducer.rs` and diagnostic `shadow_compare.rs` are both retired). Shapes it self-excludes before binding (`rate()`/`irate()`, `topk(K, sum by(...)(rate(...)))`, keyed-CMS point-estimate) and shapes it structurally can't realize yet (composed exact/summary aggregation in either nesting order — [ASAPController#171](https://github.com/ProjectASAP/ASAPController/issues/171), e.g. `max/avg by (zone) (quantile_over_time(...))`) fail over to archive directly, with no local workaround. | - -**Net reading**: L1–L4 and the serving-time cutover are all now at -target. The earlier instinct that "`intent_algebra`/`sketch_algebra` -should be unnecessary once connected to ASAPController" is correct and -largely *already true* for L2–L4; L1 has since closed the same way -(#428), and the serving-time cutover is fully done (#427) — `sketch_reducer.rs` -and `shadow_compare.rs` are both retired, not just superseded. The -remaining serving-time gaps (rate/topk-over-rate/keyed-CMS, -ASAPController#171's composed exact/summary shapes) are genuine upstream -L4 limitations tracked in ASAPController, not something this repo routes -around locally — same category as the already-tracked `TopK { accuracy: -Exact }` gap (ASAPController#151). L5 should **not** shrink — it's this -deployment's own, permanent responsibility per ASAPController's own "no -`asap-physical` crate" status; its only remaining gap is the low-priority -naming/trait-alignment noted above. - -## 5. Open questions (carried from `data_plane/docs/l4node-plan-executor-design.md`, mostly resolved) - -Listed here because both docs describe the same target and shouldn't -drift into two different pictures of what's still open: - -1. **Grouping ambiguity for empty, sketch-family `by`. RESOLVED.** PR - #169's `l3-intent-algebra.md` interface section resolves this at the - type level — `Reduction::{Reduce(GroupKeys), PerEntity}` is exactly - the upstream IR signal this repo's design doc flagged as missing. - `data_plane`'s `find_candidates` implementation (`summary_executor.rs`'s - `resolve_group_key`) already branches on the real `Reduction` value, - not an empty-key heuristic — confirmed by reading the current code, - not assumed. This was the stated blocker for the serving-time cutover - in §3/Part A; it's why that cutover was safe to default-on already. -2. **Outer-fold family of gaps** (`topk(K, sum by (...) (rate(m[r])))`, - stacking an outer exact statistic on a sketch/exact-agg readout) — - still open, still a cross-repo IR design question per the original doc. -3. **Sizing drift** — how often a freshly-planned `SummaryParams` fails to - match what's actually registered — still only measurable empirically - once serving-time re-planning runs against real traffic. diff --git a/control_plane/docs/design.md b/control_plane/docs/design.md deleted file mode 100644 index 5dd9d097..00000000 --- a/control_plane/docs/design.md +++ /dev/null @@ -1,1888 +0,0 @@ -# ASAPController — design - -> **Mirror.** This file mirrors `docs/design.md` from -> [`ProjectASAP/ASAPController#docs/workload-plan-and-queryspec`](https://github.com/ProjectASAP/ASAPController/tree/docs/workload-plan-and-queryspec/docs) -> as of 2026-05-05. -> -> ASAPController upstream is currently doc-only. The actively-running -> controller implementation lives next to this doc in -> [`controller/src/`](../src/) — the file you're reading describes the -> **target architecture** the implementation is converging toward. -> Where this doc and the current `analyzer::QuerySpec` shape diverge, -> the doc is the design intent. - -Target repo: **`github.com/ProjectASAP/ASAPController`** (currently doc-only). - -Merges three existing codebases into one: - -1. **`DataCollector/controller`** (Rust) — service-shaped; end-to-end data lifecycle planner (collection → transmission → storage → analytics query); OpAMP + HTTP; SLA-driven replanning loop. -2. **`ASAPQuery[-backend]/asap-planner-rs`** (Rust) — CLI-shaped; analytics-query-only planner; YAML-in → YAML-out. Two divergent copies today. -3. **`asap-fusion`** (Rust) — library-shaped; DataFusion operator-level rewrite rules with sketch awareness; multi-query batch fusion is aspirational, executor is a thin wrapper. - -## 1. Goals - -1. **One repo, one workspace.** One `Cargo.toml` at root, one place to file issues, one release cadence. -2. **Common core, pluggable deployment models.** The three existing codebases each solve a *different* planning problem. Factor out what's actually shared; keep the deployment-model-specific parts as crates so each deployment model can evolve independently. -3. **Extensible for future deployment models** (4th, 5th, …). A new deployment model should land as a new crate that implements well-defined traits — not by touching the core or the runtime. -4. **Preserve the two deployment shapes:** - - **Service**: long-running HTTP+OpAMP process that accepts live `QuerySpec`s, replans on SLA violations, pushes configs to agents/backends. (DC controller today.) - - **CLI**: one-shot "read workload YAML → emit `streaming_config.yaml` + `inference_config.yaml`". (`asap-planner-rs` today.) - Both should be thin shells over the same core. -5. **No regression in today's wire contracts.** `POST /api/v1/streaming-config` to ASAPQuery-backend and OpAMP push to agents must keep working byte-for-byte through the migration. The backend's capability-miss callback `ControllerClient.create_plan` must keep working. -6. **Sketch is a primitive, not a mandate.** The optimizer selects among physical alternatives for each logical operator — e.g. `HashJoin` / `SortMergeJoin` / `SketchJoin`, or `SortAgg` / `HashAgg` / `SketchAgg` — the same way a traditional DB optimizer picks a join algorithm. A plan may come back with zero sketch operators when an exact path Pareto-dominates for the query's accuracy target. Sketches are one primitive class the optimizer can reach for; the framework does not privilege them. - -## 2. Non-goals - -- **Not** redesigning the `Plan` IR end-to-end. DC's `algebra/` + fusion's `translator/optimizer/executor` stay where they are semantically; we merge them into a shared `Plan` crate with a clean trait seam, not a green-field rewrite. -- **Not** unifying DataFusion's `LogicalPlan` with DC's custom algebra at the type level. Those are different universes. Deployment models own their IR; core owns the *staged dispatch contract*. -- **Not** in-scope: changing the existing wire protocol between controller and backend. ASAPQuery-backend keeps consuming `streaming_config.yaml` and keeps responding on `/api/v1/plan`. - -## 3. The organizing spine: DC controller's 5-layer pipeline - -DataCollector/controller already documents its query→sketch translation as a 5-layer pipeline (`DataCollector/controller/docs/query-to-sketch-translation.md`). That's the spine of this merger: - -| # | Layer | What it does | Today's locations | -|---|-------|--------------|-------------------| -| 1 | **Query Language** | Parse raw strings (PromQL, SQL, DataFusion, ElasticDSL, …) into a language-specific AST | DC `controller/src/query_parser/{promql,sql}.rs` (each parses straight to the L2 relational tree below); asap-planner-rs pulls `promql-parser` + `sqlparser` directly; asap-fusion consumes a pre-built DataFusion `LogicalPlan` (its L1 happens upstream) | -| 2 | **Language Logical Plan** | Per-language algebra tree (`Aggregate` / `Window` / `Filter` / `Sort` / `Limit`) preserving language semantics, **no sketch names, no sketch binding** | DC `controller/src/intent_algebra/relational.rs` — the L2 relational `QueryExpr` tree the `query_parser` front ends emit (one shared tree for PromQL + SQL today; a per-language split is future work); asap-fusion inherits DataFusion's `LogicalPlan` as its L2; asap-planner-rs has no L2 today (uses a template-pattern catalogue) — **Phase 4 builds one** | -| 3 | **Intent algebra** | Language- and deployment-independent IR: `QueryExpr` + `AggIntent`. Describes **intent only** — *what* to compute, with accuracy target. **No sketch type, no sketch parameters, no sketch-bound nodes** (`SketchAgg` / `SketchJoin` / `SketchSubtract` etc. live in the L4 IR `SketchExpr`). **No language-shaped operators** (no `HistogramQuantile`, no `PromQLSubquery` — those are PromQL L2 nodes that lower to data-model-agnostic shapes here). **One canonical form per plan** — no `WindowedAgg` (use `Window` over `Aggregate`). Heavy-hitter intents are first-class (`AggIntent::TopK`) so heavy-hitter sketches bind directly on the intent rather than on a generic `Sort + Limit` shape; generic `Sort + Limit` survives in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). Every edge carries a typed `Schema`. Data-model-agnostic — `QueryExpr::Scan` wraps a `Source` sum with `TimeSeries` / `Table` / `Join` variants so the same L3 IR covers ASAPQuery's time-series queries and asap-fusion's tabular queries. | DC `controller/src/intent_algebra/{agg_intent,query_expr,schema,lower,cse}.rs` (canonical L3) + `controller/src/intent_algebra/lower.rs` (the L2→L3 converter: folds the `relational` L2 tree straight to the canonical IR, single-statistic sketchable `Aggregate` fusion included — no intermediate fused L3 IR); asap-fusion's 3-variant `SubPopulationAnalyticsType` maps to a subset; asap-planner-rs's 9-variant `Statistic` maps to a subset — **Phase 4 splits sketch binding out of planner's current fused L3+L4**. *Refactor 2026-05 absorbed `controller/src/algebra/expr.rs` into `intent_algebra/relational.rs`.* | -| 4 | **Sketch algebra + optimizer** | Cost-aware algebraic rewrite rules under deployment constraints. **This is where sketch binding happens** — L4 rules take intent-only L3 (`QueryExpr`) and emit the sketch-bound IR (`SketchExpr`). ~12 rules in DC; a smaller targeted subset in planner; `SketchConfigRule` + `HashModeRule` in fusion. | Core provides the **rule engine driver** + `OptimizerRule` trait + a shared rule library + the sketch-bound IR `core::sketch_algebra::SketchExpr`; deployment models **pick** which rules to enable + supply their own deployment constraints. DC `controller/src/sketch_algebra/` (IR + binding rules) + `controller/src/optimizer/{engine,trait_def,baseline,rules/,cost/}.rs` (rule engine + cost models); fusion `src/optimizer/rules/`; planner's `map_statistic_to_precompute_operator`. *Refactor 2026-05 absorbed `controller/src/algebra/optimizer.rs` (→ `optimizer/engine.rs`) and `controller/src/planner/{cost_model,delta_cost_model,online_cost_model,pareto,tco,wire_cost,rules,baseline_planner}.rs` (→ `optimizer/{cost/,rules/,baseline.rs}`).* | -| 5 | **Physical Execution Plan** | Assign ops to pipeline stages (edge / gateway / backend / object store); produce the deployment-specific artifact (OpAMP YAML, `streaming_config.yaml`, rewritten DataFusion `LogicalPlan`). **Sketch binding is already committed by L4**; L5 is about stage allocation + emission. | Core provides the **stage allocator framework** + `PhysicalPlanner` trait + the sketch catalogue; deployment models supply their own **topology** (3-stage / 1-stage / 0-stage) + their own **emitter** for the output format. DC `controller/src/physical/{allocator,planner,plan,sketch_catalog,stage_split,topology,colored_dag/}.rs` (allocator framework + sketch catalogue + typed three-stage colouring) + `controller/src/emit/{stage_config,otap,telegraf,agent,backend,asapquery_backend,precompute,trait_def}.rs` (per-deployment-model emitters + `PlanEmitter` trait) + `controller/src/pipeline.rs` (L1→…→L5 driver, formerly `analyzer.rs`); asap-planner-rs `output/generator.rs`; asap-fusion `src/executor/`. *Refactor 2026-05 absorbed `controller/src/algebra/{physical,allocator,plan,directory}.rs` and `controller/src/planner/stage_split.rs` and the legacy `controller/src/stage_split/` framework into `physical/`; absorbed `controller/src/config/` into `emit/` + `workload.rs`; renamed `analyzer.rs` to `pipeline.rs`.* | - -**The doc's key claim: layers 1–3 are query-language-independent and workload-independent.** That makes them the natural **common core**. Every deployment model reads PromQL (or SQL, or …) the same way, lowers it to the same per-language algebra, and lowers THAT to the same intent-algebra IR (intent only, no sketch binding). - -**L4 and L5 also have substantial common infrastructure.** Initially we assumed deployment models owned L4/L5 wholesale; on closer inspection what's actually deployment-model-specific is *which rules fire* (L4) and *what topology + output format* (L5) — not the rule engine, not the allocator, not the sketch catalogue. Those frameworks belong in core. This makes deployment models significantly thinner: each becomes a small crate that picks rules from a shared library, declares a deployment topology, and writes an emitter. - -### Sketch binding lives in L4, not L3 - -A key clarification after cross-checking the three source repos: **L3 is intent-only**. DC's `AggIntent` names *what* to compute (`Quantile(0.99, ε=0.01)`, `Cardinality(δ=0.001)`, …) without committing to a sketch type. Picking KLL vs DDSketch, CMS vs CMS-with-heap, parameter sizes — all of that is L4's job, driven by deployment constraints. - -More generally: **L1–L3 lower into a logical representation + `AggIntent`; L4 and L5 choose the concrete execution plan.** That choice is a standard physical-operator selection — `HashJoin` vs `SortMergeJoin` vs `SketchJoin`; `SortAgg` vs `HashAgg` vs `SketchAgg`. "Use a sketch" is one option among several; the same L4 rule framework that picks sketch parameters also picks between sketch and non-sketch operators when a rule is registered for the intent. This keeps the existing 5-layer split intact: nothing about the layering presupposes the output contains a sketch. - -Today: -- DC: correctly separated (L3 has `AggIntent`, L4 picks sketch via cost model). -- asap-fusion: correctly separated — `SketchConfigRule` at L4 fills `SketchConfig::NULL` with concrete `CountMinSketch{5,4096}` / `KLL{k=200,m=8}`. -- asap-planner-rs: **L3+L4 fused today**. `map_statistic_to_precompute_operator` jumps from `Statistic` straight to `AggregationType::DatasketchesKLL{k=200}` in one call. **Phase 4 splits this** — `Statistic → AggIntent` at L3, `AggIntent + DeploymentConstraints → AggregationType + SketchParams` at L4. - -### Intent vocabulary: DC's `AggIntent` is a superset; deployment models use subsets - -DC's pre-cleanup superset had ~25 variants; after L3 normalisation (see §6) it's smaller because language-flavored synonyms (`QuantileOverTime → Window + Quantile`) no longer earn their own intent. The post-cleanup core is 9 (`Count, Sum, Min, Max, Quantile, TopK, Cardinality, Rate, Increase`); the long-term ceiling is bounded by genuinely-distinct operations (stddev, variance, approximate-join-cardinality, …), not by language-flavored synonyms. Planner's 9-variant `Statistic` maps directly: `Topk` keeps its own intent (heavy-hitter sketches like SpaceSaving / CMS-with-heap compute it as a single primitive, so the intent earns L3 visibility). Fusion's 3 variants (`Count, Sum, Quantile`) map directly. Adding a new intent (e.g. stddev) is a core change that deployment models opt into. - -### Data-model support: both time-series and tabular - -ASAPQuery-backend / DC controller operate on time-series data (metrics + labels + timestamp); asap-fusion operates on tabular data (DataFusion `LogicalPlan` over relations) that may or may not be time-indexed. These two data models differ fundamentally in their leaf shape (`metric + labels + time` vs. `table + columns`), but they share everything above the leaf — filter semantics, aggregation semantics, sketches themselves. - -Core handles this with: -- **`QueryExpr::Scan { source: Source, ... }`** where `Source` is a sum type (`TimeSeries`, `Table`, `Join`, …). Deployment models' L1→L2→L3 lowering produces the appropriate variant; L4 rules that care about the data model gate on `source.data_model()`. -- **`AggIntent::requires() -> DataModel`** — each intent variant tags whether it's data-model-agnostic (`Count`, `Sum`, `Min`, `Max`, `Quantile`, `Cardinality`), time-series-only (`Rate`, `Increase` — both carry PromQL counter-reset semantics), or tabular-only (future additions for joins, correlated subqueries). -- **Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values; that stream can come from a time-series window or a table column, the sketch does not know or care. So `BindKllOnQuantile` and siblings work uniformly across both. - -Practically, this means: -- `deployment-model-asapquery` + `deployment-model-asaplifecycle` lower into `QueryExpr` with `Source::TimeSeries` leaves. -- `deployment-model-asapfusion` lowers into `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when it extends to multi-table queries). -- A hypothetical OLAP deployment model that runs approximate queries over tabular data reuses `Source::Table` + the same `AggIntent` subset fusion uses, plus any OLAP-specific intents it adds. - -See §6 `core::intent_algebra` for the concrete type sketches. - -### Scope: start single-query, grow into workload-aware - -The initial implementation can operate on **one query at a time** — L4 picks physical operators per query against per-query constraints, and `CostModel::workload_cost` degenerates to a sum of per-plan costs. This matches today's three source repos (all single-query planners) and is the minimum bar for parity during the migration. - -**Workload-awareness is an extension, not a rewrite.** When ≥2 queries are planned together, `workload_cost` credits shared sub-expressions (sketches, precomputed aggregates) so the planner can pick a plan for `q1` that lets `q2` read its output for free. Nothing in the L1–L5 spine changes — only the cost objective widens and the rule engine gains cross-plan visibility. See §6 `core::cost` and §13 future work. - -### L2 is mandatory; the tree shape is an evolvable contract - -Every deployment model must produce an L2 tree, even when the source language didn't originally come as one. asap-planner-rs's current approach (PromQL pattern catalogue → `IntermediateAggConfig`) skips L2; Phase 4 will reverse-engineer the five PromQL pattern shapes into a `PromqlLogicalPlan` tree so the L1→L2→L3 pipeline is uniform. - -A future deployment model whose source semantics genuinely don't fit a tree (e.g. a constraint-based query language) would motivate revisiting the L2 contract at that time. Until then, L2 = per-language tree, mandatory, no elision. - -### System I/O contract — what the controller takes in, what it emits - -The controller is a **planner**, not an executor. It does not run queries; it decides where each piece of a query runs. The data plane (OTel collectors / ASAPQuery-backend / DataFusion `SessionContext`) runs them. - -**System input — what the controller takes in.** A `QueryWorkload` (one or more `QuerySpec`s) plus deployment context (available executors, their capabilities, current SLA targets, telemetry of recent violations). The `QuerySpec` carries the raw query string in its source language (PromQL / SQL / DataFusion / ElasticDSL) and the accuracy / latency / cost target. Workloads can arrive via four entry points (HTTP `POST /plan`, OpAMP capability-miss callback, YAML file for the CLI shell, query-log replay); they all normalise to `QueryWorkload` before L1. - -**Workload features that the controller cares about, beyond the queries themselves:** -- Execution model: **batch** (one-shot YAML, query-log replay) vs **streaming** (live pipeline that must keep producing results as data arrives). -- Data input source: time-series scrape, tabular relation, query log replay, etc. Drives `Source` variant choice in L3. -- Reuse opportunity: ≥2 queries planned together → `CostModel::workload_cost` credits shared sub-expressions. - -**System output — what the controller emits.** For each registered executor in the deployment, a sub-DAG of the optimized plan plus the configuration that lets that executor run it. Concretely: - -| Output | Consumer | Wire shape | -|---|---|---| -| Per-executor sub-DAG assignment | The executor (edge agent / gateway / backend / DataFusion session) | OpAMP `RemoteConfig` (OTel YAML) / `streaming_config.yaml` POST / rewritten `LogicalPlan` | -| Cut-edges between executors | The transport between executors (OTel pipeline, HTTP, sketch-merge / compute-from-raw over the precompute engine, …) | Implied by the per-executor configs; not a separate artifact | -| Plan ID + provenance metadata | The controller's own `PlanStore` for replan / EXPLAIN / observability | JSON / proto | - -**Premise behind the staged dispatch.** Two distinct concepts: - -- **Stage** (`StageId`) — a categorical *tier* in the data lifecycle (edge / gateway / backend / in-process). The topology declares which stages exist (3-stage / 1-stage / 0-stage). Stages are roles, not instances. -- **Executor** (`Executor`, defined in `core::physical::executor`) — a *concrete runtime instance* that occupies a stage. Carries `id`, `stage: StageId`, `capabilities`, and `address` (OpAMP agent / HTTP endpoint / in-process handle). One stage may have N executors — e.g. a 50-host edge fleet is 50 `Executor`s all with `stage = StageId("edge")`; a singleton backend is one `Executor` at `stage = StageId("backend")`. - -Every stage is in principle capable of running the entire query tree — all stages speak the same physical operators. The controller's job is to decide *which stage* runs *which sub-tree / sub-DAG* under the deployment's constraints (memory budget per stage, network bandwidth between stages, sketch backends available at each stage). A "stage assignment" is a colouring of the L4-bound `SketchExpr` DAG by `StageId`, with sketch-merge / data-shipping nodes inserted on the cut edges. L5's `StageAllocator` does the colouring at stage granularity; the per-deployment-model `PhysicalPlanner` then materialises each stage's sub-DAG into one config per `Executor` at that stage (varying only by per-executor connection / identity details). The executor list comes from `DeploymentConstraints::executors()`. - -This is what makes the topology a *parameter* rather than an axis of code: the same `SketchExpr` plus a different `TopologyDescriptor` produces edge-only / 1-stage / 3-stage / 0-stage placements without rewriting the plan. - -**Symbolic plan vs concrete plan.** L3 `QueryExpr` and L4 `SketchExpr` are symbolic — they describe operations and bindings without committing to *where* anything runs. L5 produces the concrete plan: stage-assigned, executor-targeted, ready to serialize into the executor's configuration format. The split is what lets the controller swap topologies (single-stage backend → three-stage edge/gateway/backend) without re-running L1-L4. - -This drives the core/deployment model split: - -- **`crates/core/`** owns all 5 layers of **shared infrastructure**: L1-3 end-to-end (parsers + lowering + intent-only IR), plus L4's rule engine driver + rule library + cost-model traits, plus L5's stage-allocator framework + `PhysicalPlanner` trait + sketch catalogue. -- **Each deployment model is a thin crate** that: (1) picks which of core's L4 rules to enable + adds any deployment-model-specific rules, (2) declares its deployment topology (how many stages, where data flows), (3) provides an emitter for its output format. That's usually a few hundred lines, not thousands. - -## 4. Principles - -### P1. Core owns shared infrastructure across all 5 layers; deployment models own choices - -See §3. Core is NOT just types + trait stubs — it ships working parsers + lowering passes (L1-3), a rule engine + rule library + cost-model traits (L4 framework), and a stage allocator + physical-plan framework + sketch catalogue (L5 framework). What deployment models plug in is **which rules fire** (picking from core's library + adding their own), **deployment topology** (how many stages; DC=3, query=1, fusion=0), and an **emitter** for the output format. A deployment model that accepts all core's default L4 rules and uses `core::physical::single_stage_topology` is maybe 200 lines of code. - -### P2. Core has no I/O - -No HTTP, no OpAMP, no YAML, no Prometheus scrape, no `tokio::spawn`. Pure algorithms over in-memory data. This is what makes deployment models unit-testable without running the runtime. - -### P3. Runtime is a thin binary, deployment models are libraries - -The `asap-controller` binary is assembled from: runtime (HTTP/OpAMP/replanner/store) + N deployment model crates registered as plugins. Swapping deployment models is a build-time feature flag or a runtime registry entry. No deployment model may reach directly into another. - -### P4. One input boundary, one output boundary - -**Input**: everything that enters the controller (HTTP `QuerySpec`, Prometheus query log replay, YAML workload, capability-miss callback) normalizes into a single `QueryWorkload` type in core — a collection of `QuerySpec`s each feeding L1→L2→L3. - -**Output**: L5 emitters (OpAMP `RemoteConfig`, backend `StreamingConfig` POST, one-shot YAML file, rewritten DataFusion `LogicalPlan`) all implement a `PlanEmitter` trait. Deployment models supply emitter implementations; core doesn't know which emitters exist. - -This is the "extension point for future deployment models" — a new deployment model adds an L4 rule set + an L5 emitter and registers them. - -### P5. No feature-flag spaghetti - -If something must be optional (e.g. OpAMP for deployments that don't run OTel collectors), it's a separate crate, not a `cfg` block in core. - -## 5. Target repo layout - -> **Refactor 2026-05 — current state vs target.** The per-deployment-model crate -> split below (`crates/deployment-model-{asaplifecycle,asapquery,asapfusion}/`) -> is **deferred** until ≥2 deployment models actually ship. Today's -> implementation is a single `controller/` crate whose internal module -> structure mirrors design.md §5's modular split: -> -> | Target §5 path | Current single-crate path | -> |---|---| -> | `crates/core/query_language/` | `controller/src/query_parser/{promql,sql}.rs` (L1 parsers — emit the L2 tree directly) | -> | `crates/core/logical_plan/` | `controller/src/intent_algebra/relational.rs` (the L2 relational `QueryExpr` tree) | -> | `crates/core/intent_algebra/` | `controller/src/intent_algebra/` (with `relational` carrying the L2 relational IR and `lower` lowering it to the canonical L3 types) | -> | `crates/core/sketch_algebra/` | `controller/src/sketch_algebra/` | -> | `crates/core/optimizer/{engine,trait,rules,cost}/` | `controller/src/optimizer/{engine.rs,trait_def.rs,rules/,cost/,baseline.rs}` | -> | `crates/core/physical/{planner_trait,stage_allocator,topology,executor,sketch_catalog}/` | `controller/src/physical/{planner,allocator,plan,stage_split,sketch_catalog,topology,colored_dag/}.rs` | -> | `crates/core/pipeline/` | `controller/src/pipeline.rs` (single-file driver — was `analyzer.rs`) | -> | `crates/core/workload/` | `controller/src/workload.rs` | -> | `crates/core/emit/` | `controller/src/emit/` | -> | `crates/core/registry/` | `controller/src/deployment_model.rs` | -> | `crates/runtime/{http,opamp,monitor,replan,store,backend_client}/` | `controller/src/{main.rs,opamp/,monitor/,replan.rs,store/,backend_client.rs,metrics_exposer.rs}` | -> | `crates/deployment-model-asaplifecycle/` | folded into `controller/src/emit/{agent,backend,asapquery_backend,otap,telegraf,stage_config,precompute}.rs` + the three-stage topology in `controller/src/physical/colored_dag/` | -> -> The target multi-crate layout below remains the long-term goal but is **not a -> current migration**. The single-crate is structurally aligned to the §5 -> module boundaries so future extraction is largely `git mv` + `Cargo.toml` -> edits. - -``` -ASAPController/ -├── Cargo.toml # workspace -├── README.md -├── docs/ -│ ├── design.md # this file -│ ├── migration-plan.md # the companion file -│ ├── deployment-models/ # per-deployment-model design notes -│ └── adr/ # architecture decision records -├── proto/ -│ ├── asap_control.proto # Plan IR + QueryWorkload on the wire -│ └── opamp.proto # vendored from DataCollector -├── crates/ -│ ├── core/ # Shared infrastructure across all 5 layers; no I/O -│ │ ├── query_language/ # L1: per-language parsers — promql/, sql/, datafusion/, elasticdsl/ -│ │ ├── logical_plan/ # L2: per-language algebra tree (Aggregate/Window/Filter/…) -│ │ ├── intent_algebra/ # L3 IR: QueryExpr + AggIntent + Schema/HasSchema (intent only) -│ │ ├── sketch_algebra/ # L4 IR: SketchExpr (sketch-bound — kind + params committed) -│ │ ├── lower/ # L1→L2→L3 lowering passes (one entry per language) -│ │ ├── optimizer/ # L4 framework — produces SketchExpr from QueryExpr: -│ │ │ ├── engine/ # rule driver — fixed-point iteration, cycle detection, priority -│ │ │ ├── trait/ # OptimizerRule + RuleCategory (PushDown / Fusion / Elim / Bind) -│ │ │ ├── rules/ # shared rule library (e.g. sketch-binding rules; stream-vs-batch picker) -│ │ │ └── cost/ # CostModel trait + generic impls (memory budget, accuracy degradation) -│ │ ├── physical/ # L5 framework: -│ │ │ ├── planner_trait/ # PhysicalPlanner trait -│ │ │ ├── stage_allocator/ # generic topology-driven allocator -│ │ │ ├── topology/ # Topology descriptor types (edge/gateway/backend, single, zero) -│ │ │ ├── executor/ # Executor type — concrete runtime instance occupying a stage -│ │ │ └── sketch_catalog/ # candidate sketch types + parameter constraints -│ │ ├── pipeline/ # orchestrates L1→L2→L3→L4→L5, parameterized on deployment model -│ │ ├── workload/ # QueryWorkload wrapper around QuerySpecs -│ │ ├── emit/ # PlanEmitter trait — implemented by deployment model L5 -│ │ ├── registry/ # DeploymentModelRegistry, DeploymentModelId -│ │ └── telemetry/ # tracing macros, metric names (no exporter) -│ ├── runtime/ # service skeleton — HTTP, OpAMP, replanner -│ │ ├── http/ # axum surface — /plan, /replan, /metrics, /status -│ │ ├── opamp/ # WebSocket OpAMP server -│ │ ├── monitor/ # Scraper, Thresholds, Violation -│ │ ├── replan/ # Replanner — SLA + expiry triggers -│ │ ├── store/ # PlanStore, WorkloadStore -│ │ └── backend_client/ # HTTP client (pushes to ASAPQuery-backend, etc.) -│ ├── deployment-model-asaplifecycle/ # thin — picks rules, 3-stage topology, OTel/backend emitters -│ │ ├── rules.rs # L4 rule selection + DC-specific rules (stage-aware push-down) -│ │ ├── topology.rs # 3-stage: edge / gateway / backend -│ │ ├── cost.rs # deployment-model-specific cost impls — delta / online / pareto / tco -│ │ └── emit/ # OpAmpRemoteConfig + AsapqueryBackendConfig emitters -│ ├── deployment-model-asapquery/ # thin — rules, 1-stage topology, YAML emitters + query-log input -│ │ ├── rules.rs # L4 rule selection + sketch-binding rule (split from map_statistic_*) -│ │ ├── topology.rs # 1-stage: backend-only -│ │ ├── emit/ # StreamingConfig.yaml + InferenceConfig.yaml -│ │ ├── query_log/ # extra L1 input: Prometheus query-log replay -│ │ └── schema/ # PromQLSchema discovery from Prometheus (feeds L1) -│ ├── deployment-model-asapfusion/ # thin — DF-flavored rules, 0-stage, in-process emit -│ │ ├── rules.rs # L4 rules for DataFusion LogicalPlan (sketch-aware rewrites) -│ │ ├── topology.rs # 0-stage: in-process -│ │ ├── emit/ # rewritten DataFusion LogicalPlan -│ │ ├── executor/ # DataFusion SessionContext wrapper (library-mode execution) -│ │ └── sketch_support/ # asap_sketchlib-backed rewrites -│ ├── control-proto/ # generated from proto/ (tonic/prost) -│ └── testing/ # test fixtures + harness shared across deployment models -└── bin/ - ├── asap-controller/ # long-running service with all deployment models - │ └── main.rs # axum + OpAMP + replanner + registered deployment models - ├── asap-query/ # one-shot CLI for deployment-model-asapquery (what asap-planner-rs is today) - │ └── main.rs # clap — read workload YAML, emit two YAMLs - ├── asap-lifecycle/ # OPTIONAL: standalone service with only deployment-model-asaplifecycle - │ └── main.rs # slimmer image — no DataFusion, no query YAML emitters - └── asap-fusion-bench/ # OPTIONAL: benchmark harness over deployment-model-asapfusion - └── main.rs # criterion entry; used by researchers -``` - -**Per-deployment-model standalone binaries** are first-class. Each `bin//` is a thin shell that `use`s only the deployment model crates it needs — so `bin/asap-lifecycle/` doesn't pull `datafusion` into its dep tree, and `bin/asap-fusion-bench/` doesn't pull `axum`/`opamp`. Feature flags on the workspace root let you `cargo build -p asap-lifecycle` and get a minimal binary. - -### Why three deployment model crates today, not one monolithic `deployment-models/` - -Each deployment model has a different problem shape: - -| | **lifecycle** | **query** | **fusion** | -|---|---|---|---| -| Input | workload + live metrics | workload YAML / query log | DataFusion `LogicalPlan` | -| Decision unit | end-to-end staged pipeline | per-aggregation YAML | per-operator rewrite | -| Output | OpAMP OTel config + `StreamingConfig` YAML | `streaming_config.yaml` + `inference_config.yaml` | rewritten `LogicalPlan` | -| Trigger | QuerySpec, SLA violation, expiry | one-shot CLI invocation | a DataFusion session constructing a query | -| Cost model | accuracy × latency × $ staged | single-query accuracy/latency | operator selectivity / sketch feasibility | - -Mashing them into one crate means the union of all their dependencies (sqlparser + promql-parser + DataFusion + OpAMP proto + Prometheus client) bleeds into every downstream user. Separating them means a user who only needs `deployment-model-asapfusion` (an offline query-optimization benchmark, say) can depend on it without pulling OpAMP. - -## 6. Core crate details (layers 1–3 + driver) - -Core is not a trait-stubs library. It ships real L1/L2/L3 code lifted from DC's `controller/src/query_parser/` + `controller/src/algebra/` and exposes a small set of traits for L4/L5 plugin points. - -> **§6 / `core::*` paths vs current single-crate paths.** §6 below names -> targets like `core::query_language`, `core::sketch_algebra`, -> `core::optimizer` from the design.md §5 target layout. Today's -> single-crate (refactor 2026-05) is structurally aligned to those names -> via a 1:1 mapping. The lookup table: -> -> | `core::*` reference in §6 | Current single-crate path | -> |---|---| -> | `core::query_language` | `controller/src/query_parser/{promql,sql}.rs` | -> | `core::logical_plan` | `controller/src/intent_algebra/relational.rs` (the L2 relational `QueryExpr` tree) | -> | `core::intent_algebra` | `controller/src/intent_algebra/` (canonical `query_expr` / `agg_intent`) + `controller/src/intent_algebra/relational.rs` (the L2 relational IR) | -> | `core::sketch_algebra` | `controller/src/sketch_algebra/` | -> | `core::lower` | per-layer: `controller/src/intent_algebra/{lower,lower}.rs` + `controller/src/sketch_algebra/lower.rs` | -> | `core::optimizer::engine` | `controller/src/optimizer/engine.rs` | -> | `core::optimizer::trait` | `controller/src/optimizer/trait_def.rs` (placeholder) | -> | `core::optimizer::rules` | `controller/src/optimizer/rules/` | -> | `core::optimizer::cost` | `controller/src/optimizer/cost/` | -> | `core::physical::planner_trait` | `controller/src/physical/planner.rs` (legacy `physical_plan_to_staged` impl pending trait extraction) | -> | `core::physical::stage_allocator` | `controller/src/physical/{allocator,stage_split,colored_dag/allocator}.rs` | -> | `core::physical::topology` | `controller/src/physical/topology.rs` | -> | `core::physical::executor` | not yet materialised (placeholder absent) | -> | `core::physical::sketch_catalog` | `controller/src/physical/sketch_catalog.rs` | -> | `core::pipeline` | `controller/src/pipeline.rs` (single file — was `analyzer.rs`) | -> | `core::workload` | `controller/src/workload.rs` | -> | `core::emit` | `controller/src/emit/` | -> | `core::registry` | `controller/src/deployment_model.rs` | -> | `core::telemetry` | `controller/src/metrics_exposer.rs` (out-of-core today — drives Prometheus metrics) | -> -> Every §6 reference below should be read with this table as the -> concrete pointer. Where §6 says "lifted from DC's -> `controller/src/algebra/`", read "structurally aligned to -> `intent_algebra/relational.rs` + `physical/` + `optimizer/engine.rs` -> per the §16 ADR." - -### `core::query_language` — Layer 1 - -Per-language parsers, one module each: - -- `promql/` — wraps `promql-parser`. Input: `&str` PromQL. Output: `PromqlAst`. -- `sql/` — wraps `sqlparser`. Input: `&str` SQL. Output: `SqlAst`. -- `datafusion/` — wraps DataFusion's own parser. Output: `DfAst`. -- `elasticdsl/` — wraps `elastic_dsl_utilities`. Output: `EsAst`. - -Each returns a language-flavored AST type. No sketch awareness. - -> **Implementation status.** L1 + L2 are realised concretely, without -> the `Language`-trait / `LanguageAst` indirection the target design -> sketches. L1 lives in `controller/src/query_parser/{promql,sql}.rs` — -> each parser walks its language's AST and emits the **L2 relational -> tree directly** (`intent_algebra::relational::QueryExpr`: `Aggregate` -> / `Window` / `Filter` / `Join` / `Sort` / `Limit` / …). That tree -> *is* L2. `query_parser::parse_query_expr_canonical` then lowers it to -> the canonical L3 IR via `intent_algebra::lower` -> (sketch-fusion + the Binder folded in); `parse_query` projects the -> flat `ParsedQuery` summary the legacy analyzer / pipeline consume. -> -> An earlier `Language`-trait + `LanguageAst` + `LanguageLogicalPlan` -> scaffold (a per-language enum mirroring the target design) was built -> ahead of its call site, never wired into the pipeline, and drifted -> out of sync — its "L2" actually held the L3 tree. It was removed; the -> per-language `Language`-trait extension point gets rebuilt when a -> second real language backend (SQL beyond the current direct parser, -> DataFusion, ElasticDSL) actually needs it. - -### `core::logical_plan` — Layer 2 - -A **per-language** algebra tree — one `enum LogicalPlan` per language. Preserves language-specific semantics (PromQL instant vs range vector, SQL window frames, Elastic buckets) that would be lossy to collapse this early. Types are symmetric: `Aggregate { AggFunc }`, `Window`, `Filter`, `Sort`, `Limit`. No sketch names yet. - -### `core::intent_algebra` — Layer 3 - -The language- and deployment-independent IR. **Pure intent at this layer**: no language-specific operators (no `HistogramQuantile`, no `PromQLSubquery` — those are L2 PromQL nodes), no sketch types, no sketch parameters, no physical operator choice. Data-model-agnostic: supports both **time-series** inputs (ASAPQuery-backend, DC lifecycle) and **tabular** inputs (asap-fusion, future OLAP deployment models) via a `Source` sum type inside `QueryExpr::Scan`. - -#### Design rules for L3 - -1. **One canonical form per plan.** No redundant variants whose semantics decompose into other variants. `Window` over `Aggregate` is the canonical windowed-aggregate shape; there is no separate `WindowedAgg`. This keeps L4 rule matching unambiguous (a rule fires on one shape, not on N synonyms). The exception is when an "intent" is its own physical primitive: heavy-hitter top-k is served by sketches (SpaceSaving, CMS-with-heap) as a single operation, so `AggIntent::TopK` is a first-class intent at L3 — distinct from the generic `Sort + Limit` operator pair, which still appears in `QueryExpr` for non-heavy-hitter cases (e.g. `ORDER BY name LIMIT 10`). -2. **Language-orthogonal.** No PromQL-shaped or SQL-shaped operators leak into L3. `HistogramQuantile` is a PromQL artifact (it consumes Prometheus's specific bucketed-histogram exposition format and produces a quantile from buckets) — it lives in `core::logical_plan::promql` (L2) and lowers to bucket reads + a regular `Quantile` intent over those bucket counts. `PromQLSubquery` (`[range:resolution]`) is a *driver* construct — it asks the engine to evaluate the inner expression at N timestamps — and is expanded by the PromQL L1→L2 lowering into a set of independent queries, not preserved as an L3 DAG node. -3. **Intent at L3, sketch at L4.** L3 carries `AggIntent` ("compute a quantile to ε=0.01 accuracy"). The choice between `HashAgg` / `SortAgg` / `SketchAgg(KLL{k=200})` is made by L4 cost-aware rules, not encoded in L3. -4. **One physical-choice node per logical operator.** L3 has `Aggregate` (logical) and `Join` (logical). L4 produces the sketch-bound physical alternatives (`SketchAgg`, `SketchJoin`, `SketchSubtract`, `SketchDelete`, `SketchEstimate`, `SketchMerge`) into an extended IR — see "L4 sketch-bound IR" below. Mixing logical and physical at L3 (the previous draft did this with `SketchAgg` / `JoinSketch` at L3) creates ambiguity about which layer owns which decision. -5. **DAG, not tree.** Edges between nodes carry typed schemas (see "Schema flow" below). A node's output schema is a function of its inputs and parameters and is verifiable independently of the surrounding context. The shape is a DAG, not a tree, because **a producer node can have multiple parents that share its precomputed intermediate state** — instead of recomputing the same sub-expression once per consumer, the consumers fan in to a single node and read its output. Three sources of fan-in: - 1. **Explicit, in-query.** SQL CTEs (`WITH name AS (expr) SELECT ... FROM name JOIN name AS n2 ON ...`) and PromQL recording rules name a sub-expression and reference it N times; each reference becomes a `QueryExpr::Ref(name)` parent of the named producer. - 2. **L4 reuse rules.** When two queries planned together both need (e.g.) a p99 quantile of the same series, the optimizer can introduce a shared sketch node whose output feeds both — even though neither query author wrote a CTE. - 3. **L4 stage / shard structure.** A pre-aggregate computed once on the edge can feed multiple downstream gateway-stage operators. - - `CostModel::workload_cost` credits a shared producer's build cost once across all consumers, which is what makes reuse a Pareto win. Tree IRs lose this — they have to duplicate the producer for every consumer. - -```rust -pub enum QueryExpr { - // ── Base relations ──────────────────────────────────────────────────── - /// A metric stream / table / join. Outermost leaf. - Scan { source: Source, predicates: Vec }, - /// Reference to a CTE / let-binding by name; resolved at plan time. - Ref(String), - - // ── Filtering & projection ──────────────────────────────────────────── - /// σ — row-level filter (WHERE / PromQL label matchers). - Filter { child: Box, pred: Predicate }, - /// π — column projection (SELECT list). - Project { child: Box, cols: Vec }, - - // ── Aggregation (logical, intent-only) ──────────────────────────────── - /// γ + α — GROUP BY + aggregate intents, with optional HAVING. - /// `aggs` carry `AggIntent`; concrete sketch / non-sketch operator - /// is chosen by L4 and lives in the L4-extended IR (`SketchExpr`). - Aggregate { child: Box, by: Vec, - aggs: Vec, having: Option }, - - // ── Time / streaming windows ────────────────────────────────────────── - /// ψ — tumbling / sliding / session window over the time axis. Defines - /// the lifecycle (flush / reset bounds) of any aggregate in its sub-tree / sub-DAG. - /// PromQL `[5m]` and streaming windows lower here. SQL `OVER (...)` - /// analytic frames are a different node — see `WindowFunc` below. - Window { child: Box, kind: WindowKind, - size: Duration, slide: Option }, - - // ── Distributed-execution structure ─────────────────────────────────── - /// Partition the stream by key tuple (`GROUP BY` / PromQL `by (dims)`). - Partition { child: Box, keys: PartitionKeys }, - /// δ — SQL `DISTINCT` / row deduplication on `cols`. - Distinct { child: Box, cols: Vec }, - /// ⊕ — union of sub-results from independent stages or shards (the - /// exact-merge case). Sketch unions are a separate node in `SketchExpr` - /// because they carry sketch-family / params type constraints. - Merge { children: Vec }, - - // ── Joins (logical) ─────────────────────────────────────────────────── - /// Logical join. L4 picks the physical alternative — `HashJoin` / - /// `SortMergeJoin` / `SketchJoin` (e.g. KMV / theta-sketch / join-sample) - /// — based on selectivity, memory budget, and accuracy target. The - /// sketch-aware variant lives in `SketchExpr::SketchJoin`. - Join { kind: JoinKind, left: Box, right: Box, - pred: Option }, - - // ── Set operators ───────────────────────────────────────────────────── - /// UNION / INTERSECT / EXCEPT, with or without ALL. - SetOp { kind: SetOpKind, all: bool, - left: Box, right: Box }, - - // ── Ordering & limiting ─────────────────────────────────────────────── - /// Generic order-by — survives L3 for non-heavy-hitter cases - /// (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`). - Sort { child: Box, keys: Vec }, - /// `LIMIT n OFFSET k`. The heavy-hitter shape (`ORDER BY count DESC - /// LIMIT k`, PromQL `topk(k, …)`) is recognised at L1→L2→L3 lowering - /// and produces `AggIntent::TopK` rather than generic `Sort + Limit`, - /// so heavy-hitter sketches (SpaceSaving, CMS-with-heap) bind on the - /// intent. Generic `Sort + Limit` flows through unchanged. - Limit { child: Box, n: u64, offset: u64 }, - - // ── Subquery / CTE ──────────────────────────────────────────────────── - Subquery { child: Box, alias: String }, - /// SQL `WITH name AS (expr) IN body`; lowering target for PromQL - /// recording-rule bindings. - LetBinding { name: String, expr: Box, body: Box }, - - // ── Analytic (OVER) window functions ────────────────────────────────── - /// SQL `OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)`. - /// Distinct from `Window` above — that is a streaming/tumbling window - /// over the time axis; this is an analytic frame over already-grouped rows. - WindowFunc { child: Box, func: WindowFuncKind, - partition_by: Vec, order_by: Vec, - frame: Option }, - - // ── Binary composition ──────────────────────────────────────────────── - /// Arithmetic / comparison / boolean composition between two relational - /// sub-expressions (PromQL binary ops including `and`/`or`/`unless`, - /// SQL boolean composition). - BinaryOp { op: BinaryOpKind, lhs: Box, rhs: Box, - vector_match: Option }, -} - -pub enum WindowKind { Tumbling, Sliding, Session } - -pub enum Source { - /// Time-series input — deployment-model-asapquery / deployment-model-asaplifecycle shape. - TimeSeries { metric: MetricRef, time: TimeRange, labels: LabelFilter }, - /// Tabular input — deployment-model-asapfusion / future-OLAP shape. - Table { table_ref: TableRef, columns: Vec }, - /// Join over Sources composes leaf shapes recursively. - Join { left: Box, right: Box, on: JoinKey }, - // Future: WindowedStream, Subquery — added by deployment models that need them. -} - -pub enum DataModel { TimeSeries, Tabular, Any } - -impl Source { - pub fn data_model(&self) -> DataModel { /* … */ } - /// Output schema produced by this leaf (see "Schema flow"). - pub fn schema(&self, catalog: &SchemaCatalog) -> Schema { /* … */ } -} -``` - -#### What was removed during cleanup, and why - -| Removed | Reason | Replacement | -|---|---|---| -| `TopK { k, by }` *as a `QueryExpr` node* | A `QueryExpr`-level top-k operator collapses two distinct concepts: (a) the *intent* of "compute heavy hitters", which has its own sketch primitive, and (b) the generic operator pair `Sort + Limit`, which doesn't. Splitting them puts heavy-hitter logic at the right level | Heavy-hitter intent → `AggIntent::TopK` (L3, recognised by L1→L2→L3 lowering of `ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`); generic ordering+limit → `Sort + Limit` `QueryExpr` nodes (unchanged). Both retained — they describe different things | -| `WindowedAgg { intent, window }` | Equivalent to `Window` over `Aggregate`; redundant. Tumbling/sliding kind moves onto `Window::kind` | `Window { kind: …, … } → Aggregate { aggs: [intent] }` | -| `SketchAgg { intent, col }` | Sketch-bound; L3 must be intent-only | `Aggregate { aggs: [intent] }` at L3; L4 emits `SketchExpr::SketchAgg` | -| `JoinSketch { outer, inner, key }` | Sketch-bound physical alternative; L3 must be intent-only. Several papers describe sketch-of-join (KMV, theta, join-sample) — picking one is an L4 cost decision, not an L3 surface choice | `Join { … }` at L3; L4 emits `SketchExpr::SketchJoin` when a `Bind*OnJoin` rule fires | -| `HistogramQuantile { phi }` | PromQL artifact — consumes Prometheus's specific bucketed-histogram format. Language-specific | Lowers in PromQL L1→L2 to bucket reads + `Aggregate { aggs: [Quantile{q: phi, …}] }` over the bucket counts | -| `PromQLSubquery { range, resolution }` | Driver construct — asks the engine to evaluate the inner expression at N timestamps; not a single DAG node | Expanded by PromQL L1→L2 lowering into a set of independent `QueryExpr` instances, not preserved at L3 | -| `Dedup { col }` | Single-column-only spelling of SQL `DISTINCT` | Renamed to `Distinct { cols }`, generalised to N columns | - -#### Schema flow — every L3 edge carries a typed schema - -Every node has a derivable output schema given its input schemas and parameters. The DAG is type-checked: a `Filter` whose predicate references a column not in its child's output schema fails at plan time. - -```rust -pub struct Schema { - pub fields: Vec, - /// Index into `fields` for the time axis, if any. PromQL leaves carry one; - /// SQL leaves may or may not. - pub time_index: Option, - /// Optional metadata for reuse-aware planning. Each inner `Vec` is - /// a set of column indices that together uniquely identify rows; the outer - /// `Vec` allows multiple unique-key sets (e.g. primary key + another unique - /// constraint). - /// - /// **Populated by**: the per-node input/output spec (e.g. `Aggregate { by, .. }` - /// emits `unique_keys = [by]`; `Distinct { cols }` adds `cols`; `Project` - /// carries forward the retained columns; most other nodes pass through). - /// - /// **Consumed by**: `CostModel::workload_cost` only — the reuse-aware path - /// that credits shared sub-expressions across multiple queries. Single-query - /// plans, the `Bind*` rules, push-down rules, and L5 emitters do not read - /// this field. If the reuse path is deferred (see §13 future work), this - /// field is dead weight; it lives here so the metadata is available the - /// moment workload-aware planning lands without requiring an L3-wide - /// schema change. - pub unique_keys: Vec>, -} - -pub struct Field { - pub name: String, - pub dtype: DataType, // Int64 / Float64 / Utf8 / Map / … - pub nullable: bool, -} - -pub trait HasSchema { - fn input_schemas(&self) -> Vec<&Schema>; - fn output_schema(&self, inputs: &[&Schema], cat: &SchemaCatalog) -> Schema; -} -``` - -Per-node input/output spec — the stable contract for L3 nodes (full implementation in `core/src/intent_algebra/schema.rs`). Each row reads independently: every column position, type, and constraint is named explicitly rather than carried by a shorthand like `S` or `L` / `R`. - -| Node | Input schemas | Output schema | -|---|---|---| -| `Scan { source }` | none — leaf node | `source.schema(catalog)` — derived from the source's catalog metadata (TimeSeries metric labels + value + timestamp; or Table columns from `information_schema`; or recursive `Source::Join`) | -| `Ref(name)` | none — pointer node | the output schema of the `LetBinding` whose `name` matches; resolved at plan time | -| `Filter { pred }` | one input schema (the child's output) | the child's input schema unchanged — `Filter` is a row-level refinement; no columns added, removed, or re-typed | -| `Project { cols }` | one input schema | the input schema projected to `cols` — fields filtered and reordered to match `cols`; `time_index` and `unique_keys` carried over for retained columns | -| `Aggregate { by, aggs }` | one input schema | the `by` columns (carried verbatim from input) followed by one new column per entry in `aggs`, each named and typed by `AggIntent::output_type(input_field)`; `unique_keys = [by]` | -| `Window { kind, size, slide }` | one input schema, **must** contain a `time_index` field | the input schema extended with synthetic `window_id` and `window_start` / `window_end` metadata fields | -| `Partition { keys }` | one input schema | the input schema unchanged — logical-only marker; carries a sharding hint for L5's stage allocator | -| `Distinct { cols }` | one input schema | the input schema with `unique_keys` tightened to include `cols`; field types unchanged | -| `Merge` | N input schemas, all union-compatible (same field names, same types, same nullability, same `time_index` position if any) | the first input's schema (representative; checked union-compatible with the rest) | -| `Join { kind, pred }` | two input schemas — left and right children | the concatenation of left's fields and right's fields, minus columns that USING / NATURAL deduplicates; nullability widened on the OUTER side for outer joins | -| `SetOp { kind, all }` | two input schemas, must be union-compatible (as for `Merge`) | the left input's schema | -| `Sort { keys }` | one input schema; every field referenced in `keys` must be present in it | the input schema unchanged | -| `Limit { n, offset }` | one input schema | the input schema unchanged | -| `Subquery { alias }` | one input schema | the input schema with `alias` applied as the table alias to all field names | -| `LetBinding { name, expr, body }` | `expr` produces an intermediate schema bound to `name`; `body` consumes it (and any other in-scope bindings) | the `body`'s output schema | -| `WindowFunc { func, partition_by, order_by, frame }` | one input schema; every field in `partition_by` / `order_by` must be present | the input schema extended with one new column carrying the analytic-function output (named after `func`, typed per `func`) | -| `BinaryOp { op, vector_match }` | two input schemas — left and right operands. PromQL vector-match constraints (`on`/`ignoring` + `group_left`/`group_right`) govern label-set compatibility | for arithmetic/comparison `op`: the left input's schema with the value column re-typed to the result of `op`; for boolean `op` (`and`, `or`, `unless`): the left input's schema with a boolean value column | - -##### Implementation status - -Phase F (in `controller/src/intent_algebra/schema.rs` + `controller/src/intent_algebra/cse.rs`) lands the load-bearing consumer of `Schema::unique_keys`: - -- `cse_reuse_is_legal(producer_schema, consumer_count) -> Result<(), CseError>` — the gatekeeper. Two `QueryExpr::Ref` consumers may share a producer only when the producer's output schema has a non-empty `unique_keys` set and the consumer count is ≥ 2. This is the proof point that `unique_keys` is load-bearing — without it, the deduper conservatively refuses to share and reuse "drops on the floor" (this section, line ~1356). -- `dedupe_subtrees(roots) -> CseWorkloadPlan` — basic implementation of the workload-level CSE pass for the literal "≥2 root queries with identical sub-expressions" case (the batched-queries example, line ~1256). Detects shared `Aggregate` children, gates on `cse_reuse_is_legal`, hoists into a `LetBinding`. Richer detection (alpha-equivalence, schema-merge across compatible-but-not-identical shapes, recursive nested CSE) is downstream — Phase F lands the gate + the basic case so the cost-model side has something to credit. - -The full general CSE algorithm (the optimisation half) remains future work. - -#### DAG schema, DB schema, sketch catalog — three distinct metadata sources - -These three are sometimes conflated and shouldn't be. Only the first two are *schemas* (descriptions of stream / table shape); the sketch catalog is a *registry* of available primitives, not a description of a stream: - -| Source | Where it lives | What it describes | Who reads it | -|---|---|---|---| -| **DAG schema** | On every edge of the L3 / L4 / L5 DAG (`Schema` above) | Columns + types flowing between operators | L4 rules (selectivity estimation, push-down legality), L5 emitter | -| **DB / source schema** | The query target (Prometheus TSDB metric metadata, SQL `information_schema`, DataFusion catalog) | What metrics / tables / columns exist in the data plane, with their types and indexing | The **Binder** (below), via the `SchemaCatalog` interface (`core::intent_algebra::binder`) | -| **Sketch catalog** | `core::physical::sketch_catalog` (built at startup; static) | What sketches the runtime can build; what intents each one serves; mergeability, accuracy / confidence guarantees, supported aggregation keys, parameter ranges | L4 binding rules to choose a sketch for an `AggIntent`; L5 to instantiate the sketch | - -The **Binder** reads the **DB schema** to resolve symbols. L3 onward, every edge carries a **DAG schema** that is type-checked locally. L4 binding rules consult the **sketch catalog** to map an intent to a concrete sketch under the deployment's constraints. They are three separate inputs to three distinct decisions. - -#### The Binder — name resolution as an explicit pass - -Every mature query engine has exactly one explicit boundary where -symbolic column / table *names* are resolved against a schema source, -and everything downstream of that boundary is fully resolved: - -| Engine | Resolution pass | Resolved column identity | -|---|---|---| -| ClickHouse | `QueryAnalyzer` rewrites `IdentifierNode` → `ColumnNode` against `StorageSnapshot` | name + type + source pointer | -| Trino | `Analyzer` → `Analysis` side-table, against catalog `Metadata` | opaque, plan-local `Symbol` | -| RisingWave | `Binder` resolves against its `Catalog` | positional `InputRef { index, data_type }` | -| **ASAP control plane** | **`core::intent_algebra::binder::Binder`** | positional `ColumnId` (index into `Schema`) | - -Our canonical L3 IR already commits to positional column identity — -`Aggregate.by: Vec`, exactly RisingWave's `InputRef`. The -[`Binder`] is the pass that *produces* it: given a query tree, it walks -it, collects every referenced column / group-key name, and builds the -complete, **self-contained** [`Schema`] every `ColumnId` in the lowered -tree indexes into — the IR's own "RelationType" (Trino) / bind scope -(RisingWave). Resolution downstream is then **total** — it cannot fail -on a well-formed tree. - -The schema source is the **`SchemaCatalog`** seam: - -- The default `UsageDerivedCatalog` knows nothing — every schema is - derived purely from what the query references. This is the honest - state for the observability domain: metric label sets are open-ended - and data-dependent, there is no closed catalog to resolve against - (unlike a SQL `information_schema`). -- A registry-backed `SchemaCatalog` (a metric-schema registry) is future - work. Crucially, **the `Binder` pass does not change when it lands** — - only the catalog impl swaps. - -Placement: today the Binder runs at the L2→L3 (relational → canonical) -conversion boundary (`lower::convert_root` calls it). -Once the `relational` L2 IR is retired it moves into the `core::lower` L1→L2→L3 -passes proper — the `lower_*(ast, schema)` signatures below already -anticipate a schema parameter at that point. - -#### `AggIntent` — what to compute, not how - -```rust -pub enum AggIntent { - // Data-model-agnostic - Count { accuracy: AccuracyTarget }, - Sum, - Min, Max, - Quantile { q: f64, accuracy: AccuracyTarget }, - /// Heavy-hitter top-k. Distinct from generic `Sort + Limit` because a - /// dedicated sketch primitive (SpaceSaving, CMS-with-heap, Misra-Gries) - /// computes it as a single operation. L1→L2→L3 lowering produces this - /// when it recognises a heavy-hitter shape (`ORDER BY count DESC LIMIT k`, - /// PromQL `topk(k, …)`); other ordering+limit cases stay as - /// `QueryExpr::Sort + QueryExpr::Limit`. - TopK { k: usize, by: Vec, accuracy: AccuracyTarget }, - Cardinality { accuracy: AccuracyTarget }, - - // Time-series streaming derivatives — specific operations, not just - // "Sum / Count over a Window". `Rate` is the per-second average derivative - // computed with PromQL's counter-reset adjustment, not a generic windowed - // mean. Kept distinct because (a) they have counter-reset semantics that - // exact `Sum` does not, and (b) sketch backends specialised for - // derivatives (e.g. delta-set aggregator) bind on these intents directly. - Rate { window: Duration }, - Increase { window: Duration }, - - // Tabular / OLAP — added as deployment models demand - // CorrelatedSubqueryCount { … }, ApproxJoinCardinality { … }, -} - -impl AggIntent { - /// Which data-model this intent semantically requires. L4 rules - /// consult this to skip non-applicable intents (e.g. `Rate` over - /// a `Source::Table` is nonsense). - pub fn requires(&self) -> DataModel { /* … */ } - /// Output column type — used by L3 schema derivation for `Aggregate`. - pub fn output_type(&self, input: &Field) -> DataType { /* … */ } - /// Which sketch families in the catalog can serve this intent. - /// Read by L4 binding rules. - pub fn candidate_sketches(&self) -> &'static [SketchKind] { /* … */ } -} -``` - -**Why `TopK` *is* an intent (and `Sort + Limit` is not collapsed into it).** Heavy-hitter top-k has a dedicated sketch primitive — SpaceSaving / CMS-with-heap / Misra-Gries compute it in a single pass with sub-linear memory. L4 binding rules want to fire on the *intent* "give me the top-k frequent items" rather than on a syntactic shape, the same argument that makes `Quantile` an intent rather than a `Sort` + "pick the φ-th element" pattern. So `AggIntent::TopK` lives at L3. Generic `QueryExpr::Sort + QueryExpr::Limit` *also* survives, because not every order-by-limit query is heavy-hitter (`ORDER BY name LIMIT 10`, `ORDER BY ts DESC LIMIT 1`); these have no sketch alternative and stay as generic operators. The canonical-form invariant is preserved because L1→L2→L3 lowering picks one or the other deterministically based on whether it recognises a heavy-hitter pattern. - -**Why no `QuantileOverTime` intent?** It duplicated `Quantile` over a `Window`. The window — its kind, its size, its slide — is fully captured by the surrounding `Window { … }` node; the quantile *operation* is the same regardless of whether the input was a windowed time-series or a row-grouped table. PromQL's `quantile_over_time(0.99, m[5m])` lowers cleanly to `Window{size=5m} → Aggregate{aggs:[Quantile{q=0.99}]}`. One intent halves the L4 rule surface (one bind rule per operation, not per language-flavor of an operation). - -**Why `Rate` and `Increase` survive that argument.** They are not "Sum / Count over a Window with a different name" — they include PromQL's counter-reset adjustment, which is a non-trivial transformation an exact `Sum` does not perform. They earn distinct intent variants because they parameterise different physical operators (delta-set aggregators bind on these intents directly). If a non-PromQL streaming language has the same notion (e.g. SQL `RATE() OVER (RANGE)`), it lowers to the same intent — the intent vocabulary names the operation, not the language. - -#### Implementation status - -Phase B (this PR) ships the L3 IR in `controller/src/intent_algebra/`: - -- `intent_algebra::AggIntent` — vocabulary above (`Count`, `Sum`, `Min`, `Max`, `Avg`, `Quantile`, `TopK`, `Cardinality`, `Frequency`, `Rate`, `Increase`). -- `intent_algebra::QueryExpr` — variant subset for the DC + PromQL deployment: `Scan`, `Window`, `Aggregate`, `LetBinding`, `Ref`. The remaining variants (`Filter`, `Project`, `Partition`, `Distinct`, `Merge`, `Join`, `SetOp`, `Sort`, `Limit`, `Subquery`, `WindowFunc`, `BinaryOp`) are deferred so each lands with a planner consumer rather than as dead code; adding them is purely additive. -- `intent_algebra::Schema` — typed schema flow with `unique_keys` (the load-bearing CSE-legality field, `cse_substitution_legal_only_with_unique_keys` test pins the invariant). -- `intent_algebra::lower_parsed_query` — `query_parser::ParsedQuery` → `QueryExpr` single-query lowering, exposed as a standalone function. The analyzer is **not** yet wired to emit `QueryExpr`; that wiring is the follow-up phase's job, kept separate so the IR rev and the consumer rev land independently. - -Follow-up phases: - -- **Phase C** — wire `Analyzer::analyze` to also produce a `QueryExpr` alongside `QueryWorkload`, populate `WorkloadPlan::roots` from the lowered roots. -- **Phase D** — workload-level CSE pass (`core::lower::workload::dedupe_subtrees`) that hoists shared sub-DAGs into `WorkloadPlan::bindings`, leaning on `Schema::unique_keys` for legality (the batched-queries example above). -- **Phase E** — L4 `SketchExpr` IR (`core::sketch_algebra`) + L4 binding rules. -- **Phase F** *(this PR)* — `CostModel::workload_cost` + `Schema::unique_keys`-based CSE legality. See the per-section "Implementation status" notes under `core::cost` and the "Schema flow" table below for what shipped / what's deferred. - -### `core::sketch_algebra` — Layer 4 IR (`SketchExpr`) - -L4 binding rules consume L3 `QueryExpr` (in `core::intent_algebra`) and produce `SketchExpr` (in `core::sketch_algebra`). This is the IR L5 emitters consume. The two-IR split — intent-only L3 (`QueryExpr`) and sketch-bound L4 (`SketchExpr`), in two separate modules — gives L4 rule application a clean type signature: `fn apply(&QueryExpr, &Constraints) -> Option`, and the boundary cannot be silently violated. - -```rust -pub enum SketchExpr { - /// Any logical L3 node passes through unchanged when no L4 rule rewrote - /// it — a `Filter` doesn't need a sketch counterpart. - Logical(QueryExpr), - - /// Sketch aggregation. L4 picked the sketch type and parameters from the - /// catalog given the `AggIntent` and `DeploymentConstraints`. - SketchAgg { - child: Box, - sketch: SketchKind, // Kll, Cms, Hll, DDSketch, CmsWithHeap, … - params: SketchParams, // catalog-validated - col: ColumnRef, - by: Vec, - }, - - /// Sketch-aware join (KMV / theta-sketch for join cardinality; - /// join-sample for join sampling). Emitted only when a `Bind*OnJoin` - /// rule fires — L3 always presents the logical `Join` for L4 to choose. - SketchJoin { - outer: Box, - inner: Box, - key: ColumnRef, - sketch: SketchKind, - params: SketchParams, - }, - - /// Subtract one sketch from another. Valid only for sketches with a - /// linear-inverse property (CMS, theta, count-based). Lets the planner - /// compute "all-A minus all-B" cardinality / count without re-scanning. - SketchSubtract { left: Box, right: Box }, - - /// Delete a key from a sketch (CMS update with -1, deletable Bloom - /// filter, …). Valid only for deletion-supporting sketches. - SketchDelete { sketch_input: Box, key: ColumnRef }, - - /// Read out a query result from a built sketch. Inverse of `SketchAgg`. - /// `query` says what to extract — quantile φ, count for key k, cardinality. - SketchEstimate { sketch_input: Box, query: SketchQuery }, - - /// ⊕ — union of sketches across stages / shards. Distinct from L3 - /// `Merge` because sketch union has type constraints (same family, - /// same params). L5 stage allocator emits this when distributing. - SketchMerge { children: Vec }, -} -``` - -The optimizer's job is to selectively replace logical aggregates / joins with their sketch-bound variants when a binding rule fires; everything else stays inside `SketchExpr::Logical(…)`. - -#### Per-node input/output spec for `SketchExpr` - -L4 introduces a new field type into `Schema`: - -```rust -pub enum DataType { - // … the L3 types (Int64, Float64, Utf8, Map<…>, …) … - /// Sketch state. Carries the sketch family + params so the type system - /// rejects merges of incompatible sketches at plan time. - Sketch(SketchKind, SketchParams), -} -``` - -A "sketch-state schema" is a regular `Schema` whose value-bearing field has a `DataType::Sketch(...)` dtype. Reading rules: - -| Node | Input schemas | Output schema | -|---|---|---| -| `Logical(qe)` | whatever the inner L3 node `qe` consumes (per the L3 table above) | whatever `qe` produces — straight pass-through | -| `SketchAgg { child, sketch, params, col, by }` | one input schema; must contain `col` (the column being summarised) and every field referenced in `by` (group keys) | the `by` columns carried over verbatim, followed by one synthetic field of dtype `Sketch(sketch, params)` carrying the partial sketch state per group; `unique_keys = [by]` | -| `SketchJoin { outer, inner, key, sketch, params }` | two input schemas (outer + inner); both must contain `key` with compatible types | one field of dtype `Sketch(sketch, params)` carrying the join-cardinality / join-sample state — read out by a downstream `SketchEstimate` | -| `SketchSubtract { left, right }` | two input schemas, each with exactly one `Sketch(s, p)` field; **`s` and `p` must match** between the two inputs (catalog rejects mismatches at plan time); the `s` family must have `subtractable = true` in the catalog | one `Sketch(s, p)` field carrying the subtracted state (same family + params as inputs) | -| `SketchDelete { sketch_input, key }` | one input schema with a `Sketch(s, p)` field whose catalog entry has `deletable = true`; plus the `key` column to delete | the input schema unchanged in type — sketch state is mutated logically (the key's contribution removed) but the field's `(sketch, params)` signature is preserved | -| `SketchEstimate { sketch_input, query }` | one input schema with a `Sketch(s, p)` field; `query` (e.g. `Quantile(φ)`, `PointCount(k)`, `Cardinality`) must appear in the catalog entry's `supported_intents` for `s` | a regular row-shaped schema carrying the answer — `Float64` for quantile, `Int64` for count / cardinality, an array of `(key, count)` for top-k. The `Sketch(...)` field type does *not* propagate downstream of an Estimate | -| `SketchMerge { children }` | N input schemas, each with a `Sketch(s, p)` field; **all N must agree on `(s, p)`**; the `s` family must have `mergeable = true` in the catalog | one `Sketch(s, p)` field carrying the unioned state (same family + params as inputs) | - -Two type-system invariants make L4 robust: - -1. **Sketch-family mismatch is a plan-time error.** `SketchSubtract` over `Sketch(KLL, …)` and `Sketch(CMS, …)` fails type-checking before L5 ever sees it. -2. **Catalog capability flags gate which nodes can fire.** `SketchSubtract` requires `subtractable`, `SketchDelete` requires `deletable`, `SketchMerge` requires `mergeable`. The catalog (see §6 `core::physical::sketch_catalog`) is the single source of truth for these flags; binding rules consult it before producing the node. - -**Why a `Source` sum instead of two parallel `QueryExpr` trees:** most `QueryExpr` nodes (`Filter`, `Aggregate`) are data-model-agnostic — filter semantics are the same whether the input is a time-series window or a table scan. Only the leaf `Scan` differs. Keeping one tree with a polymorphic leaf means L4 rules like `BindKllOnQuantile` work uniformly across both data models; rules that care about data-model specifics (stage-aware push-down for TS; join-selectivity for tabular) gate on `source.data_model()` + `intent.requires()`. - -**Sketches are data-model-agnostic by construction.** KLL / CMS / HLL / DDSketch ingest a stream of values. That stream can come from a time-series window (`Source::TimeSeries`) or a table column (`Source::Table`); the sketch doesn't know or care. So `BindKllOnQuantile` works identically regardless of `Source`. - -#### Implementation status - -**Phase C** (`feat(controller): Phase C — sketch_algebra L4 IR (SketchExpr + Bind* rules)`) lands the typed `SketchExpr` IR + `Bind*` rules in `controller/src/sketch_algebra/`. Module layout: - -- `sketch_expr.rs` — `SketchExpr` enum (`Logical` / `SketchAgg` / `SketchEstimate` / `SketchMerge` / `LetBinding` / `Ref`). -- `params.rs` — `SketchKind` + per-family `SketchParams` (`KllParams{k}`, `DDSketchParams{alpha}`, `HllParams{precision}`, `CmsParams{w,d}`, `CountSketchParams{w,d,with_heap}`). -- `schema.rs` — `SketchStateSchema` with the `(SketchKind, SketchParams)` field-type and the catalog-capability flags (`mergeable` / `subtractable` / `deletable`). -- `rules/{bind_kll_quantile, bind_ddsketch_quantile, bind_cms_count, bind_cms_topk, bind_hll_cardinality}.rs` — five `Bind*` rules implementing the `Rule` trait. -- `lower.rs` — `bind_query_expr(&QueryExpr, AccuracyTarget) -> Result`. - -**Scope reduction.** The variant set ships the subset DC + PromQL needs. `SketchJoin`, `SketchSubtract`, `SketchDelete` from the spec above are intentionally *not* surfaced yet — they're gated on rules that haven't landed. Adding them is purely additive. - -**Planner-side migration is opt-in.** `controller/src/planner/rules.rs` exposes `bind_workload_typed(&QueryWorkload) -> Option` and an env-var gate `USE_TYPED_SKETCH_ALGEBRA=1`. Existing call sites continue to use the legacy untyped binding path (`algebra::directory::sketch_type_for_agg`); the typed path runs in parallel for callers that opt in. **Phase E** (stage_split refactor) is the natural migration point at which the typed path becomes the only path. - -### `core::lower` — L1 → L2 → L3 passes - -One pass per language, each producing the same `intent_algebra::QueryExpr`: - -```rust -pub fn lower_promql(ast: PromqlAst, schema: &MetricSchema) -> Result; -pub fn lower_sql(ast: SqlAst, schema: &TableSchema) -> Result; -pub fn lower_datafusion(ast: DfAst, ctx: &DfContext) -> Result; -pub fn lower_elasticdsl(ast: EsAst, schema: &IndexSchema) -> Result; -``` - -Once a query hits L3 it's language-agnostic. All deployment models downstream see the same IR. - -The `schema` parameter on each pass is supplied by the **Binder** — see -§6 "The Binder — name resolution as an explicit pass". The Binder is the -single place symbolic names become positional `ColumnId`s; the `lower_*` -passes consume its output and never resolve names ad-hoc. Today the -Binder runs at the L2→L3 boundary inside `lower`; it folds -into these `lower_*` signatures once the `relational` L2 IR is retired. - -### `core::pipeline` — orchestration - -The L1→…→L5 driver. Parameterised on a deployment model's optimizer rules (L4) + emitter (L5): - -```rust -pub struct Pipeline { - deployment_model: S, -} - -impl Pipeline { - pub fn run(&self, workload: &QueryWorkload) -> Result { - let l3: Vec = workload.queries() - .iter() - .map(|q| self.parse_and_lower(q)) // L1→L2→L3 - .collect::>()?; - let l4 = self.deployment_model.optimizer().optimize(l3)?; // L4 - let l5 = self.deployment_model.physical().lower(l4)?; // L5 - self.deployment_model.emitter().emit(&l5) // L5 → bytes/protobuf/DF plan - } -} -``` - -Core owns the driver. Deployment models own what goes into each deployment-model-specific seam. - -### `core::optimizer` — Layer 4 framework - -Core ships the **rule engine + trait surface + a shared rule library**. Deployment models pick which rules to enable. - -```rust -// trait (in core) -pub trait OptimizerRule: Send + Sync { - fn name(&self) -> &'static str; - fn category(&self) -> RuleCategory; // PushDown | Fusion | Elim | Bind | ... - fn priority(&self) -> u16; - fn apply(&self, expr: &QueryExpr, c: &DeploymentConstraints) -> Option; -} - -// engine (in core) — fixed-point iteration, cycle detection, priority ordering -pub struct RuleEngine { /* ... */ } -impl RuleEngine { - pub fn new(rules: Vec>) -> Self { /* ... */ } - pub fn run(&self, exprs: Vec, c: &DeploymentConstraints) - -> Result, OptError>; -} - -// shared rule library (in core::optimizer::rules) — opt-in from deployment models -pub mod rules { - pub struct BindKllOnQuantile; // AggIntent::Quantile → bind KLL(k by accuracy) - pub struct BindCmsOnCount; // AggIntent::Count → bind CMS(w, d) - pub struct BindHllOnCardinality; // AggIntent::Cardinality → bind HLL(p) - pub struct FusionPassthrough; // Aggregate over Filter → push Filter under Aggregate - pub struct ElimNoopFilter; // Filter(true) → child - // ... etc - impl OptimizerRule for BindKllOnQuantile { /* ... */ } -} -``` - -Deployment models compose rule sets by picking from the shared library + adding their own: - -```rust -// in deployment-model-asaplifecycle -use asap_control_core::optimizer::{RuleEngine, rules::*}; -fn rule_set() -> Vec> { - vec![ - Box::new(BindKllOnQuantile), - Box::new(BindCmsOnCount), - Box::new(FusionPassthrough), - // DC-specific additions: - Box::new(StageAwarePushDown), // pushes ops to edge when possible - Box::new(TransmissionCostRewrite), // uses TCO model to defer aggregation - ] -} -``` - -Core also ships `DeploymentConstraints` as a trait object; each deployment model supplies a concrete impl with its deployment's memory budgets, network topology, available sketch backends, and the registered `Executor` list (one entry per concrete runtime instance, each tagged with its `StageId`). The executor list is populated from the deployment model's discovery channel — OpAMP for DC lifecycle, static config for single-backend query, the in-process `SessionContext` itself for fusion. - -### `core::physical` — Layer 5 framework - -```rust -pub trait PhysicalPlanner { - type Topology: TopologyDescriptor; - type Output; - fn lower(&self, l4: Vec, t: &Self::Topology) -> Result; -} - -pub trait TopologyDescriptor { - fn stages(&self) -> &[StageDescriptor]; - fn edges(&self) -> &[StageEdge]; -} - -// pre-baked topologies in core -pub mod topology { - pub struct ThreeStage { /* edge → gateway → backend */ } - pub struct SingleStage { /* backend-only */ } - pub struct ZeroStage; /* in-process */ -} - -/// Identifier for a stage role (categorical tier in the data lifecycle). -pub struct StageId(pub String); // "edge" / "gateway" / "backend" / "in-process" - -/// A **concrete runtime instance** that executes a sub-DAG of the plan. -/// One stage may have many executors — e.g. a 50-host edge fleet is 50 -/// executors all with `stage = StageId("edge")`; a singleton backend is -/// one executor at `stage = StageId("backend")`. The stage allocator -/// works at stage granularity; the deployment-model planner / emitter -/// materialises each stage's sub-DAG into one concrete config per executor. -pub struct Executor { - pub id: ExecutorId, // stable identifier across replans - pub stage: StageId, // which stage this executor occupies - pub capabilities: ExecutorCaps, // memory budget, available sketch backends, network neighbours - pub address: ExecutorAddr, // OpAMP agent / HTTP endpoint / in-process handle -} - -pub struct ExecutorId(pub String); - -pub enum ExecutorAddr { - OpAmpAgent(AgentId), // OTel collector managed via OpAMP (DC lifecycle) - HttpEndpoint(Url), // ASAPQuery-backend, etc. - InProcess, // asap-fusion library-mode SessionContext -} - -// generic stage allocator — given a QueryExpr tree + a topology, decide which -// ops land on which stage subject to constraints. Stage-level only; per-executor -// fan-out happens in the deployment model's PhysicalPlanner using the executor -// list from `DeploymentConstraints::executors()`. -pub struct StageAllocator; -impl StageAllocator { - pub fn allocate( - &self, exprs: &[QueryExpr], topology: &T, c: &DeploymentConstraints, - ) -> Result, PlanError>; -} - -// sketch catalogue — what sketches exist, what they support, what params they -// accept. Built at startup from the registered sketch backends; queried by L4 -// binding rules to map an `AggIntent` → `SketchKind` + `SketchParams`, and by -// L5 to instantiate the chosen sketch. -pub struct SketchCatalog { - pub entries: Vec, -} - -pub struct SketchEntry { - pub kind: SketchKind, // Kll, Cms, Hll, DDSketch, KMV, Theta, … - /// Which `AggIntent` variants this sketch can serve. - pub supported_intents: &'static [IntentTag], // Quantile, Count, Cardinality, JoinCardinality, … - /// Mergeability — sketches built on disjoint inputs combine without re-scan. - /// (CMS / HLL / KLL / theta = mergeable; SpaceSaving = approximately; - /// some heavy-hitter variants = no.) - pub mergeable: Mergeability, - /// Whether the sketch supports point deletion (CMS update with -1, - /// deletable Bloom filter) — gates `SketchExpr::SketchDelete`. - pub deletable: bool, - /// Whether the sketch admits a linear inverse (CMS, theta, count-based) - /// — gates `SketchExpr::SketchSubtract`. - pub subtractable: bool, - /// Accuracy / confidence model: error bounds as a function of params. - /// `(eps, delta)` for randomised sketches; absolute error for KLL; etc. - pub accuracy: AccuracyModel, - /// Aggregation keys this sketch supports natively. Some sketches are - /// keyed (CMS over (label, value)); others are unkeyed (HLL). - pub aggregated_keys: KeyShape, // Unkeyed | KeyedScalar | KeyedTuple - /// Parameter ranges + defaults. Catalog rejects out-of-range params at - /// L4 bind time so L5 never sees an unsupported configuration. - pub param_ranges: ParamRanges, - /// Memory + CPU model used by `CostModel`. Function of params. - pub cost_model: SketchCostModel, -} -``` - -Deployment models use these pieces: - -```rust -// in deployment-model-asaplifecycle -use asap_control_core::physical::{StageAllocator, topology::ThreeStage, Executor}; - -impl PhysicalPlanner for LifecyclePlanner { - type Topology = ThreeStage; - type Output = Vec<(ExecutorId, ExecutorPlan)>; // one entry per executor - fn lower(&self, l4: Vec, t: &ThreeStage) -> Result<_, _> { - // 1. Stage-level: colour the DAG by StageId. - let assignments = StageAllocator.allocate(&l4, t, &self.constraints)?; - // 2. Per-executor fan-out: for each Executor in the deployment, - // pick the sub-DAG assigned to its stage and produce its config. - let executors: &[Executor] = self.constraints.executors(); - // deployment-model-specific post-processing: DC's delta_cost logic, - // backend_client push preparation, etc. - Ok(/* ... */) - } -} -``` - -> **Implementation status (Phase E).** The typed L5 framework lives in -> `controller/src/stage_split/` (this PR): -> - `stage_id.rs` — `StageId` enum (`Edge`/`Gateway`/`Backend`) + -> `Topology` enum. Phase E ships only `Topology::ThreeStage`; the -> `SingleStage` / `ZeroStage` variants surface as future-proofing -> stubs that error cleanly via `AllocateError::UnsupportedTopology`. -> - `colored_dag.rs` — `ColoredDag { topology, nodes, edges }` IR, the -> allocator's output. `cut_edges()` exposes cross-stage edges for -> future wire-format insertion (Phase G+). -> - `allocator.rs` — `StageAllocator::allocate(expr, topology) -> -> Result`. Implements the §6 -> batched-queries colouring rules: `Logical(Scan/Window/Aggregate)` -> + `SketchAgg` → Edge; `SketchMerge` → Gateway; `SketchEstimate` → -> Backend; `LetBinding`/`Ref` colour by their bound expression's -> stage. The L3 `Aggregate{exact}` (e.g. `Max`) reachable through -> `Logical` colours Edge — the design.md "root of q3" backend -> placement is exercised when the same exact aggregation appears -> *above* a SketchMerge sibling structure (a Phase G enhancement -> that adds an explicit `Logical(Merge)` SketchExpr variant for the -> gateway hop). -> - `emitter.rs` — `Emitter` trait + `ThreeStageEmitter` that lowers -> `ColoredDag` → `HashMap`. `StageConfig` -> carries the structural facts each downstream consumer needs: -> - `Edge` → `EdgeStageConfig { source_metric, label_filters, -> window_secs, sketch_processors, exporter_target }`. Sketch -> processor names follow the catalog (`KLL`, -> `ddsketch`, `HLL`, `countmin`, -> `countsketch`). -> - `Gateway` → `GatewayStageConfig { otlp_receiver_port, -> merge_processors, exporter_target }`. -> - `Backend` → `BackendStageConfig { aggregations, readouts }` — -> the aggregation_id ↔ (sketch_kind, params) mapping the backend's -> `OtlpReceiver` + readout catalog need. -> -> The typed path is opt-in via the `USE_TYPED_STAGE_SPLIT` env var -> consulted by `controller/src/planner/stage_split.rs::split_typed_three_stage`; -> existing untyped callers (`split_expr_by_stage`) continue to run -> unchanged. OpAMP push (`crate::opamp::OpampServer::push_to_role`) -> and backend `StreamingConfig` POST (`crate::backend_client`) wiring -> against the typed `StageConfig` is downstream (Phase G+) — Phase E -> ships only the structured per-stage output, not the wire push. - -### `core::plan` — shared traits bridging layers - -```rust -pub trait DeploymentModel { - type Topology: TopologyDescriptor; - type EmitterOutput; - fn rules(&self) -> Vec>; - fn topology(&self) -> &Self::Topology; - fn physical(&self) -> &dyn PhysicalPlanner; - fn emitter(&self) -> &dyn PlanEmitter; - fn constraints(&self) -> &dyn DeploymentConstraints; -} -``` - -### `core::workload` - -> **Implementation status (this PR).** The new types `QueryLanguage`, -> `AccuracyTarget`, `QueryShape`, `DataShape`, `QueryId`, `BindingName`, -> and a `WorkloadPlan` container live in `controller/src/types_v2.rs`. -> `analyzer::QuerySpec` carries the new fields (`id`, `language`, -> `accuracy`, `dollars`, `deployment_model`, `shape`, `data`) as -> `#[serde(default)]` additions; defaults preserve legacy behaviour for -> existing `POST /api/v1/plan` callers and the `workloads.yaml` -> pre-population path. `Analyzer::analyze` enforces the L1 cross-product -> rejections (`Streaming × Batch`, `Streaming × Mutable`) from the table -> below and resolves typed `accuracy` over legacy `accuracy_sla` with -> the typed form taking precedence. The fields are not yet load-bearing -> in `replan.rs` / `planner/` cost or binding decisions — that's a -> separate downstream PR. `WorkloadPlan` is a container only; the CSE -> pass that populates `bindings` is deferred until the L3 algebra grows -> `LetBinding` / `Ref`. - -One public type for every kind of input: - -```rust -pub enum QueryWorkload { - /// A single QuerySpec from an HTTP POST. DC's current entry point. - Single(QuerySpec), - /// A hand-authored YAML workload file. asap-planner-rs's current entry point. - AuthoredSet(Vec), - /// A Prometheus query log replayed. asap-planner-rs's alt entry point. - QueryLog(QueryLogReplay), -} - -pub struct QuerySpec { - /// Stable identifier — preserved across `replan` cycles so the runtime - /// can correlate plan outputs with the originating spec, and L4 reuse - /// rules can name shared producers across consumers in the same workload. - pub id: QueryId, - - /// Source-language query string + which language it's written in. - /// L1 parses `query` against `language`; no schema lookup yet. - pub query: String, - pub language: QueryLanguage, // PromQL | Sql | DataFusion | ElasticDsl - - /// Evaluation time range. Some languages (PromQL) carry their own ranges - /// inside `query`; this field wins when both are set. - pub time_range: TimeRange, - - /// Per-target SLA. Drives L4 binding (which sketch / which params) and - /// L5 stage placement (push down or not). The three are independent — - /// `accuracy: Exact` disables all `Bind*` rules; an unset latency / - /// dollars target lets the cost model pick freely. - pub accuracy: AccuracyTarget, // Exact | Epsilon(f64) | EpsilonDelta { ε, δ } - pub latency: Option, // p99 evaluation latency target - pub dollars: Option, // per-evaluation budget - - /// Routing hint: which deployment model should plan this query. When - /// unset, the runtime defaults to the deployment model bound to the - /// inbound HTTP route (`POST /plan/lifecycle` vs `POST /plan/query`). - pub deployment_model: Option, - - /// How the query is *evaluated*: one-shot, continuous, or scheduled. - /// Distinct from `data` below — a one-shot query against a streaming - /// source is "evaluate now over the latest window"; a streaming query - /// against a batch source doesn't make sense and is rejected at L1. - /// Drives L4 binding (mergeable vs one-shot sketch family) and the - /// L5 wire format (`OneShot` emits config + result; `Streaming` and - /// `Periodic` emit a config that keeps running). - pub shape: QueryShape, - - /// Shape of the *data* feeding the query. Workload-level summary — - /// per-leaf detail rides on `Source::data_shape` (§6 L3). For a join - /// over streaming + batch this is `Mixed`, and the planner reads the - /// per-leaf shape during L4. Drives binding choices: an - /// `AppendOnlyStream` unlocks incremental, mergeable sketches and - /// retraction-free aggregation; `Batch` lets the planner pick a - /// non-mergeable estimator (e.g. exact percentile over a sort) that - /// wouldn't survive a distributed streaming setting; `Mutable` - /// requires retraction-aware operators (out of scope today — the - /// planner refuses sketch binding and falls back to re-scan). - pub data: DataShape, -} - -pub enum QueryLanguage { PromQL, Sql, DataFusion, ElasticDsl } -pub enum AccuracyTarget { Exact, Epsilon(f64), EpsilonDelta { eps: f64, delta: f64 } } - -pub enum QueryShape { - /// Evaluate once. Plan, execute, return result, discard state. - /// SQL ad-hoc queries; one-off PromQL via `POST /plan`. - OneShot, - /// Continuous query — output stream that the executor keeps emitting - /// as new data arrives. No fixed cadence; the runtime emits whenever - /// the underlying state changes. Streaming dashboards, alerting - /// expressions evaluated by the agent rather than by a poller. - Streaming, - /// Re-evaluated at a fixed cadence — Prometheus recording rules, - /// scheduled dashboard panels, alerting evaluation cycles. The - /// planner amortises sketch / aggregate build cost across evaluations - /// within the cadence and reuses state between adjacent windows. - Periodic { every: Duration }, -} - -pub enum DataShape { - /// Bounded relation, fully materialised at plan time. SQL tables, - /// Parquet / CSV files, DataFusion in-process tables. - Batch, - /// Append-only stream — events arrive over time, never updated or - /// deleted. Metrics, logs, event streams. The common case for - /// asaplifecycle and asapquery. - AppendOnlyStream, - /// Mutable relation — inserts + updates + deletes. Operational - /// databases, CRUD-style tables. Sketch binding is currently refused - /// for this shape (no retraction-aware sketches in the catalog yet). - Mutable, - /// Join across sources of differing shape. The planner consults - /// `Source::data_shape` per leaf during L4; this variant exists so - /// callers don't have to flatten a workload-level summary. - Mixed, -} -``` - -All three deployment models take `&QueryWorkload`. Same type, different planners. Fields are deployment-model-agnostic — anything deployment-model-specific (DC's stage-budget overrides, fusion's `SessionContext` handle) rides on `DeploymentConstraints`, not on `QuerySpec`. - -The `shape` × `data` cross-product isn't fully populated. Combinations the planner accepts and rejects: - -| `shape` \ `data` | `Batch` | `AppendOnlyStream` | `Mutable` | `Mixed` | -|---|---|---|---|---| -| `OneShot` | ✓ ad-hoc SQL, fusion | ✓ "evaluate now over latest window" | ✓ via re-scan, no sketch binding | ✓ per-leaf shape decides | -| `Streaming` | ✗ rejected at L1 (no stream over a static dataset) | ✓ canonical streaming case | ✗ rejected (no retraction support) | ✓ if streaming leaves dominate | -| `Periodic { every }` | ✓ scheduled batch report | ✓ recording rules | ✓ via re-scan | ✓ | - -### `core::cost` - -Extracted from DC's `planner/*` (delta, online, pareto, tco). Deployment models pick the models they need. - -```rust -pub trait CostModel { - fn accuracy(&self, plan: &Plan) -> Accuracy; - fn latency(&self, plan: &Plan) -> Duration; - fn dollars(&self, plan: &Plan) -> Dollars; - - /// Workload-level total cost. A straight sum of per-plan costs - /// under-estimates how good a plan is when multiple queries share - /// computation — a sketch or a precomputed aggregate built for - /// one query serves the others for free. Implementations must - /// identify reusable sub-expressions across `plans` and credit - /// their build cost once, so the planner prefers plans that - /// maximise reuse when the total-cost objective allows. - fn workload_cost(&self, plans: &[Plan]) -> WorkloadCost; -} - -pub struct WorkloadCost { - pub total_latency: Duration, - pub total_dollars: Dollars, - /// Per-plan contribution, for EXPLAIN / observability. - pub per_plan: Vec, - /// Sub-expressions built once, consumed by ≥2 plans. Drives - /// decisions like "build a KLL for p99 that q1 and q2 both read" - /// vs "run exact select-n per query". - pub reused: Vec, -} -``` - -The reuse model is not sketch-specific: any primitive that is costly to build once and cheap to query (sketches, materialized aggregates, cached scan results, future wavelet summaries) plugs into the same `ReusedComponent` accounting. - -#### Implementation status - -Phase F (in `controller/src/planner/cost_model.rs`) lands the workload-cost shape that pairs with the `Schema::unique_keys`-based CSE legality gate above: - -- `workload_cost(plan: &WorkloadCostPlan<'_>) -> Result` — walks the L3 IR DAG (`intent_algebra::QueryExpr`) post-order, memoises by `LetBinding` name, credits each shared producer once across consumers. Returns `WorkloadCost { total_dollars, per_root_breakdown, reused_savings }` — the `reused_savings` field exposes the gap between the bundled total and the naive sum-over-roots, so EXPLAIN can show what shared-producer credit was worth. -- `WorkloadCostPlan { bindings, roots }` — the cost-model's view of `types_v2::WorkloadPlan` carrying real `&QueryExpr` references rather than the `QueryExprPlaceholder` JSON-wire string. Collapses into `types_v2::WorkloadPlan` when the placeholder is swapped for live `QueryExpr` downstream. - -Per-node cost primitives at L3 (`node_cost_scan`, `node_cost_window`, `node_cost_aggregate`, `intent_cost`) are coarse-but-monotonic placeholders calibrated against schema width and intent kind. Calibration against real benchmarks is downstream; what Phase F pins is the *shape* — costs are positive, additive over sub-trees, and the savings invariant `bundled_total ≤ naive_sum` holds with `savings = naive_sum − bundled_total`. - -Not yet shipped at L3 cost (deferred): the `total_latency` field, the per-plan `Contribution` split (Phase F's `per_root_breakdown` carries one `f64` per root, not the full latency / dollars / accuracy tuple), and the `ReusedComponent` enumeration (savings are reported as a scalar, not as a per-component vector — the per-component breakdown is downstream when more producer kinds (sketches, materialized aggregates) become candidates for sharing). The L4 `score` / `score_with` path above remains the per-plan dollars / latency / memory cost; the L3 `workload_cost` is the bundled-plan credit on top. - -### `core::emit` - -```rust -pub trait PlanEmitter { - type PlanInput; - type Output; // YAML bytes, OpAMP RemoteConfig, rewritten LogicalPlan - fn emit(&self, plan: &Self::PlanInput) -> Result; -} -``` - -Concrete emitters live in deployment model crates: - -- `deployment-model-asapquery::yaml::StreamingConfigEmitter` → `streaming_config.yaml` bytes -- `deployment-model-asapquery::yaml::InferenceConfigEmitter` → `inference_config.yaml` bytes -- `deployment-model-asaplifecycle::emit::OpAmpRemoteConfigEmitter` → `opamp::RemoteConfig` protobuf -- `deployment-model-asapfusion::emit::DataFusionPlanEmitter` → rewritten `datafusion::LogicalPlan` - -### `core::registry` - -The extension point. The runtime binary does: - -```rust -let mut reg = DeploymentModelRegistry::new(); -reg.register::(); -reg.register::(); -reg.register::(); -// add more... -``` - -Registration is by-type; the runtime looks up by `DeploymentModelId` (either from the `QuerySpec.deployment_model` field or from an HTTP route). A 4th deployment model is added by: - -1. New crate `deployment-model-/` -2. Implement `DeploymentModel` trait (wraps a `Planner` + its `PlanEmitter`s) -3. Register in `bin/asap-controller/main.rs` - -No core change. - -### End-to-end example — one query through all five layers - -A worked trace of a single PromQL query as it flows L1 → L5. The query is intentionally simple (one metric, one window, one aggregate) so the IR shapes stay readable; production workloads have larger DAGs but the per-layer transformation is the same. - -**Input.** A `QuerySpec` arrives at `POST /plan` carrying: - -```text -quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m]) -``` - -with `accuracy: AccuracyTarget::Epsilon(0.01)` and `language: PromQL`. - -#### L1 — query language (parse to `PromqlAst`) - -`core::query_language::promql::parse` wraps `promql-parser`. Output (sketch — actual fields come from the upstream crate): - -```rust -PromqlAst::Call { - func: BuiltinFn::QuantileOverTime, - args: vec![ - Expr::Number(0.99), - Expr::MatrixSelector { - name: "http_request_duration_seconds", - matchers: vec![LabelMatcher::Eq("service", "api")], - range: Duration::from_secs(300), - }, - ], -} -``` - -Pure language-level parse — no schema lookup, no sketch awareness. Same call returns the same AST regardless of deployment model. - -#### L2 — language logical plan (`PromqlLogicalPlan` tree) - -`core::lower::promql::lower_to_logical` walks the AST against a `MetricSchema` resolved from Prometheus's `/api/v1/labels`. The result preserves PromQL semantics — `quantile_over_time` is a range-vector aggregate, distinct from a generic `Aggregate { Quantile }` over a `Window`: - -```rust -PromqlLogicalPlan::RangeAggregate { - func: PromqlRangeAggFunc::QuantileOverTime { q: 0.99 }, - range: Duration::from_secs(300), - matrix: Box::new(PromqlLogicalPlan::MatrixSelector { - metric: "http_request_duration_seconds", - matchers: vec![("service", LabelOp::Eq, "api")], - }), -} -``` - -Per language: the SQL form `SELECT approx_percentile(latency, 0.99) FROM events WHERE service='api' AND ts > now() - INTERVAL '5 min'` lands in `SqlLogicalPlan` with a different shape. L2 is per-language; the shapes converge at L3. - -#### L3 — intent algebra (`QueryExpr` + `AggIntent`) - -`core::lower::promql::lower_to_intent` strips PromQL-specific shapes. `quantile_over_time` does **not** survive — the canonical L3 form for a windowed quantile is `Window` over `Aggregate{Quantile}` (see §6 design rule 1, "no `WindowedAgg`"; see §6 "Why no `QuantileOverTime` intent?"). The accuracy target threads through from the `QuerySpec`: - -```rust -QueryExpr::Aggregate { - by: vec![], - aggs: vec![AggIntent::Quantile { - q: 0.99, - accuracy: AccuracyTarget::Epsilon(0.01), - }], - having: None, - child: Box::new(QueryExpr::Window { - kind: WindowKind::Sliding, - size: Duration::from_secs(300), - slide: None, - child: Box::new(QueryExpr::Scan { - source: Source::TimeSeries { - metric: MetricRef::from("http_request_duration_seconds"), - time: TimeRange::default(), // supplied by QuerySpec at evaluation - labels: LabelFilter::eq("service", "api"), - }, - predicates: vec![], - }), - }), -} -``` - -Per-edge schemas (derived per the §6 input/output spec): - -| Edge | Schema | -|---|---| -| `Scan` → `Window` | `{value: Float64, ts: Int64@time_index, service: Utf8}` | -| `Window` → `Aggregate` | the above + `{window_id: Int64, window_start: Int64, window_end: Int64}` | -| `Aggregate` → root | `{p99_value: Float64}` (one column per `AggIntent::Quantile`, typed via `output_type`) | - -This IR is identical across deployment models — same DAG, same intent, same accuracy target. Below this point each deployment model picks its own L4 rules. - -#### L4 — sketch algebra (`SketchExpr`) - -The shared rule `core::optimizer::rules::BindKllOnQuantile` matches `Aggregate { aggs: [Quantile{q, accuracy}] }`, consults the sketch catalog (KLL has `supported_intents: [Quantile]`, is mergeable, satisfies `ε=0.01` at `k=200`), and rewrites the matched sub-DAG into a `SketchAgg` wrapped in a `SketchEstimate` (the readout). Everything not rewritten passes through in `SketchExpr::Logical(...)`: - -```rust -SketchExpr::SketchEstimate { - sketch_input: Box::new(SketchExpr::SketchAgg { - child: Box::new(SketchExpr::Logical(/* the L3 Window+Scan subtree */)), - sketch: SketchKind::Kll, - params: SketchParams::Kll(KllParams { k: 200 }), - col: ColumnRef::value(), - by: vec![], - }), - query: SketchQuery::Quantile { q: 0.99 }, -} -``` - -Catalog-derived edge schemas: - -| Edge | Schema | -|---|---| -| inner `Logical(Window)` → `SketchAgg` | as L3 above (logical pass-through) | -| `SketchAgg` → `SketchEstimate` | `{kll: Sketch(Kll, KllParams{k:200})}` | -| `SketchEstimate` → root | `{p99_value: Float64}` (the `Sketch(...)` field type does not propagate past `SketchEstimate`) | - -The `Sketch(Kll, KllParams{k:200})` field type is the L4 type-system invariant — a downstream `SketchMerge` over a mismatched `Sketch(Cms, …)` input would fail at plan time, before L5 ever sees it. `BindKllOnQuantile` lives once in core and fires for all three deployment models; deployment-model-specific rules (lifecycle's `StageAwarePushDown`, fusion's `HashModeRule`) chain before or after. - -#### L5 — physical plan (stage allocation + emission, per deployment model) - -L4 chose the sketch family + params. L5 colors the DAG by `StageId` and emits per-executor configs. Same `SketchExpr` input; topology and emitter differ per deployment model. - -##### deployment-model-asaplifecycle (3-stage, OpAMP + backend POST) - -`StageAllocator` colors against `topology::ThreeStage`: - -| Node | StageId | Why | -|---|---|---| -| `Scan` + `Window` + `SketchAgg` | `"edge"` | scrape happens on the agent host; KLL build is mergeable so it's safe to run early | -| `SketchMerge` (inserted on cut edge) | `"gateway"` | catalog says `KLL.mergeable = true`; reduces N edge streams to 1 | -| `SketchEstimate` | `"backend"` | readout where users query | - -`PhysicalPlanner` then fans out via `DeploymentConstraints::executors()`. Given a deployment with 3 edge agents + 1 gateway + 1 backend: - -```rust -vec![ - (ExecutorId("edge-001"), OpAmpRemoteConfig { /* OTel YAML: scrape http_request_duration_seconds{service=api}, - sliding 5m window, KLL k=200, forward to gateway-001 */ }), - (ExecutorId("edge-002"), OpAmpRemoteConfig { /* identical OTel YAML; differs only by agent_id */ }), - (ExecutorId("edge-003"), OpAmpRemoteConfig { /* … */ }), - (ExecutorId("gateway-001"), OpAmpRemoteConfig { /* receive 3 KLL streams, SketchMerge, forward to backend */ }), - (ExecutorId("backend-001"), AsapqueryBackendConfig { /* read merged KLL, SketchEstimate q=0.99 */ }), -] -``` - -The 3 edge configs are byte-identical except for `agent_id` — that's the §3 "one stage may have N executors all materialised from the same per-stage sub-DAG" property. The `AsapqueryBackendConfig` is produced by calling `deployment-model-asapquery::yaml::StreamingConfigEmitter` (see §8 asymmetric dep). - -##### deployment-model-asapquery (1-stage, YAML) - -`StageAllocator` against `topology::SingleStage` returns everything on `"backend"`. `PhysicalPlanner` produces one config per backend executor (typically one): - -```yaml -# streaming_config.yaml — emitted by deployment-model-asapquery::yaml::StreamingConfigEmitter -operators: - - name: kll_p99_request_duration - kind: DatasketchesKLL - params: { k: 200 } - input: - metric: http_request_duration_seconds - label_filter: { service: api } - window: { kind: sliding, size: 5m } -estimates: - - sketch: kll_p99_request_duration - query: { kind: quantile, q: 0.99 } -``` - -This is the byte-for-byte format ASAPQuery-backend already consumes — `inference_config.yaml` is emitted alongside by `InferenceConfigEmitter` for the readout side. Both are wire invariants (see §10). - -##### deployment-model-asapfusion (0-stage, in-process `LogicalPlan`) - -`StageAllocator` against `topology::ZeroStage` returns the whole DAG in-process. `PhysicalPlanner` rewrites the caller's `datafusion::LogicalPlan`, replacing the original `Aggregate(quantile(0.99, ...))` node with an `Extension` wrapping the KLL sketch op: - -```rust -LogicalPlan::Extension(Extension { - node: Arc::new(SketchAggExt { - kind: SketchKind::Kll, - params: KllParams { k: 200 }, - input: /* original Window+Filter+Scan subtree, unchanged */, - estimate: SketchQuery::Quantile { q: 0.99 }, - }), -}) -``` - -Returned to the caller's `SessionContext` for execution by `asap-fusion`'s in-process operator. No wire format, no external executor; the data plane is the caller's process. Note also that `asap-fusion`'s entry point is `Source::Table` (not `Source::TimeSeries`) — this PromQL example is only illustrative for the fusion deployment model, which in practice consumes a pre-built DataFusion `LogicalPlan` and skips L1 (see §8). - -#### What this example demonstrates - -- **L1 → L3 are deployment-model-independent.** All three deployment models see the same `QueryExpr` for this query. -- **L4 binding is a shared rule.** `BindKllOnQuantile` lives once in `core::optimizer::rules` and fires for all three. -- **L5 is where deployment models diverge.** Same `SketchExpr` → three topologies → three emitter outputs. The topology is a parameter, not an axis of code (§3). -- **The sketch path is selected, not mandated.** If the `QuerySpec` had `accuracy: AccuracyTarget::Exact`, no `Bind*` rule would fire; L4 would pass `SketchExpr::Logical(QueryExpr::Aggregate{…})` through, and L5 would target an exact `HashAgg` (§1 goal 6). - -### End-to-end example — batched queries with shared sub-DAGs - -The single-query trace above shows one `QueryExpr` flowing through all five layers. This example shows what changes when the `QueryWorkload` is an `AuthoredSet` of related queries — the case `CostModel::workload_cost` and §6 design rule 5 (DAG fan-in) are designed for. The point is that reuse is expressed as **shared nodes in a single DAG**, not as side-channel caching. - -**Input.** A `QueryWorkload::AuthoredSet` arrives with three PromQL queries on the same metric: - -```text -q1: quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m]) -q2: quantile_over_time(0.95, http_request_duration_seconds{service="api"}[5m]) -q3: max_over_time(http_request_duration_seconds{service="api"}[5m]) -``` - -All three carry `accuracy: Epsilon(0.01)`. Planning them independently would scan + window the same metric three times. The DAG-shaped IR collapses the shared work. - -#### L3 — common sub-expression elimination produces fan-in - -After per-query L1→L2→L3 lowering, the three `QueryExpr` trees are identical below the `Aggregate` node. A workload-level CSE pass (`core::lower::workload::dedupe_subtrees`) hoists the shared sub-DAG behind a `LetBinding`, leaving three `Aggregate` nodes that fan in to one producer: - -```text - Scan{http_request_duration_seconds, service="api"} - │ - Window{Sliding, 5m} ◄── shared producer - ╱ │ ╲ - Aggregate Aggregate Aggregate - [Quantile{0.99,ε}] [Quantile{0.95,ε}] [Max] - │ │ │ - q1 q2 q3 -``` - -Multi-root DAGs live one level above `QueryExpr`. `QueryExpr` stays single-root (one query in, one root out) — the workload-level container holds N roots plus the hoisted bindings they share: - -```rust -// core::workload — output of L1→L2→L3 lowering for a QueryWorkload -pub struct WorkloadPlan { - /// Named shared producers, hoisted out of individual queries by the CSE - /// pass. Each binding is referenced by ≥2 roots via `QueryExpr::Ref`. - pub bindings: Vec<(BindingName, QueryExpr)>, - /// One root per QuerySpec in the workload, in input order. - pub roots: Vec<(QueryId, QueryExpr)>, -} -``` - -For the example above: - -```rust -WorkloadPlan { - bindings: vec![ - ("windowed_latency".into(), /* Scan → Window subtree */), - ], - roots: vec![ - (q1_id, QueryExpr::Aggregate { - by: vec![], - aggs: vec![AggIntent::Quantile { q: 0.99, accuracy: Epsilon(0.01) }], - child: Box::new(QueryExpr::Ref("windowed_latency".into())), - having: None, - }), - (q2_id, /* same shape, q=0.95, same Ref */), - (q3_id, /* Aggregate { aggs: [AggIntent::Max] } over the same Ref */), - ], -} -``` - -Three `QueryExpr::Ref` nodes, one binding; the fan-in is explicit. `CostModel::workload_cost` credits the Scan + Window build cost once across {q1, q2, q3} via `WorkloadCost::reused`, which is what makes the bundled plan beat three independent plans on `total_dollars` (§6 `core::cost`). Within-query CTE fan-in (SQL `WITH`, PromQL recording rules) keeps using `QueryExpr::LetBinding`/`Ref` unchanged — `WorkloadPlan` only adds the cross-query layer. - -CSE legality leans on `Schema::unique_keys` (§6 Schema flow): two `QueryExpr::Ref` consumers can share a producer only when its output schema is provably stable across reads — the unique-key metadata is what lets the deduper assert that without re-running the producer's logic. - -#### L4 — sketch reuse across q1 and q2 - -`BindKllOnQuantile` fires three times (once per `Quantile` intent in the workload), but a follow-on rule `MergeKllSketches` recognises that q1 and q2 read from the same input at the same accuracy. KLL state is independent of `q` — one sketch serves any quantile readout — so the two `SketchAgg{KLL}` nodes collapse into one, with two `SketchEstimate` parents reading `q=0.99` and `q=0.95`. q3 (`Max`) takes a separate path: KLL doesn't expose max, so the rule emits an exact `Aggregate{Max}` over the shared `Window`. - -```text - Scan - │ - Window - ╱ │ ╲ - SketchAgg{KLL,k=200} Aggregate{Max} - ╱ ╲ │ - SketchEstimate SketchEstimate q3 - q=0.99 q=0.95 - │ │ - q1 q2 -``` - -`SketchExpr::LetBinding` carries the two-tier fan-in: the outer let names the `Window` output (read by all three branches), an inner let names the `SketchAgg{KLL}` output (read by both `SketchEstimate` parents). The L4 type system rejects mismatched merges before L5 sees them — both `SketchEstimate` parents must declare `Sketch(Kll, KllParams{k:200})` on their input edge, which is checked locally per the §6.4 sketch-state schema rules. - -#### L5 — colored DAG, one config per executor - -`StageAllocator` colors the bound DAG by `StageId` exactly as for the single query, but the shared nodes are colored once. Under `topology::ThreeStage`: - -| Node | StageId | -|---|---| -| `Scan` + `Window` + `SketchAgg{KLL}` + `Aggregate{Max}` | `"edge"` | -| `SketchMerge` (over KLL streams), `Merge` (over Max streams) | `"gateway"` | -| `SketchEstimate{q=0.99}`, `SketchEstimate{q=0.95}`, root of q3 | `"backend"` | - -The OpAMP config emitted to each edge agent describes one scrape, one window operator, one KLL builder, one max accumulator — feeding two output streams (KLL state, Max state) toward gateway. Per-edge memory drops from `3× scan + 3× window + 2× KLL + 1× Max` (independent plans) to `1× scan + 1× window + 1× KLL + 1× Max`; bandwidth across the edge→gateway cut drops correspondingly. - -#### What this example demonstrates - -- **Reuse is a DAG property, not a sketch property.** The Scan + Window collapse is L3 CSE; the KLL collapse is L4 sketch-aware; both expressed as fan-in in the same DAG. No new IR surface, no caching layer. -- **`Schema::unique_keys` becomes load-bearing.** The CSE pass uses it to prove two `Ref`s read from a stable producer. Without it, the deduper has to be conservative and reuse drops on the floor — which is why §6 keeps the field on the `Schema` struct even though single-query plans don't read it (§6 Schema flow). -- **`CostModel::workload_cost` decides whether to share.** Reuse isn't free — when q1 demands a tighter accuracy than q2, the smaller-budget sketch may not satisfy both, and keeping them separate can be cheaper. `MergeKllSketches` is gated by `workload_cost`, not unconditional. -- **L5 is unchanged.** Same allocator, same emitter, same wire format — the only difference is that the DAG has multiple roots and shared interior nodes. The single-query path is the degenerate case where `outputs.len() == 1` and no fan-in fires. - -## 7. Runtime crate details - -Lifted from DC controller with zero semantic change: - -- `http/` — axum server on `:8080`. Routes: `POST /plan`, `POST /replan`, `GET /plans/:id`, `GET /status`, `GET /metrics` -- `opamp/` — WebSocket OpAMP server on `:4320`. Same protocol DC speaks today. -- `monitor/` — `Scraper` polls Prometheus, emits `Violation` -- `replan/` — subscribes to violations + expiry ticks, re-invokes the deployment model registry -- `store/` — in-memory `PlanStore` + `WorkloadStore`. Pluggable backend later. -- `backend_client/` — pushes `StreamingConfig` YAML to ASAPQuery-backend's `/api/v1/streaming-config` endpoint. Factored out so new deployment models can push to other backends. - -Runtime depends on `core` but NOT on any deployment model crate directly. It talks to deployment models via the registry. - -## 8. Deployment model crate details (each owns its L4 + L5) - -Each deployment model crate is a library with: - -- A `DeploymentModel` impl that registers its planner, its emitters, and its HTTP routes (if any). -- Its own config types — no shared config crate. -- Its own cost model — maybe using `core::cost` primitives, maybe not. -- Its own integration tests. - -### `deployment-model-asaplifecycle` (thin) - -- **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **L4 rules**: picks from `core::optimizer::rules::*` (Bind*, Fusion*, Elim*) + adds DC-specific rules that require stage awareness (`StageAwarePushDown`, `TransmissionCostRewrite`). Adding a stage-specific rule = a new file in `deployment-model-asaplifecycle/src/rules.rs`, impl `OptimizerRule`. Unchanged rules come from core. -- **L5 topology**: `core::physical::topology::ThreeStage` (edge → gateway → backend). No deployment-model-specific allocator logic — `StageAllocator` handles the tree walk; lifecycle only declares what the topology looks like. -- **Cost models**: `delta / online-EMA / Pareto / TCO` — these live in `deployment-model-asaplifecycle/src/cost.rs` because they're specific to the DC deployment's network/compute assumptions. Implement `core::optimizer::cost::CostModel`. -- **L5 emitters**: `OpAmpRemoteConfigEmitter` (per-role OTel YAML over OpAMP WebSocket) and `AsapqueryBackendConfigEmitter` (calls `deployment-model-asapquery`'s `StreamingConfigEmitter` for the YAML bytes, then POSTs to backend). -- **HTTP route**: `POST /plan` for full-lifecycle planning, `POST /replan` for SLA-triggered. -- **Size estimate**: ~1500 LOC (was ~5000 pre-refactor). Cost models are the bulk; rule selection + topology + emitter are each a few hundred lines. - -### `deployment-model-asapquery` (thin) - -- **Data model**: time-series. L2→L3 lowering produces `QueryExpr` with `Source::TimeSeries` leaves. -- **Inherited L1**: uses `core::query_language::promql` and `core::query_language::sql`. -- **NEW L2 tree** (Phase 4 work): defines `PromqlLogicalPlan` in `core::logical_plan::promql` that expresses the five pattern shapes asap-planner-rs currently template-matches as first-class L2 nodes. Replaces the pattern-catalogue approach with a proper L1→L2 tree rewrite. SQL side gets a matching `SqlLogicalPlan`. -- **L3 intent**: maps `Statistic` enum (9 variants) onto `core::intent_algebra::AggIntent` subset. Planner's `Topk` maps directly to `AggIntent::TopK` (heavy-hitter intent, served by SpaceSaving / CMS-with-heap at L4). -- **L4 rules**: picks from `core::optimizer::rules::*` (all `Bind*` rules are relevant since this deployment model covers most sketch types) + a deployment-model-specific sketch-binding rule for the precompute engine's flavor (which sketches are available, what params, `DeltaSetAggregator` auto-injection before CMS/HydraKLL). This rule absorbs `map_statistic_to_precompute_operator`'s sketch-binding half. -- **L5 topology**: `core::physical::topology::SingleStage` (backend-only). No stage-split; `StageAllocator` returns everything on one stage trivially. -- **L5 emitters**: `StreamingConfigEmitter` + `InferenceConfigEmitter` (YAML bytes). **Authoritative** for these two formats — `deployment-model-asaplifecycle` calls them when it needs to POST to ASAPQuery-backend. -- **Extra L1 inputs**: `query_log/` for Prometheus-query-log replay (unique to this deployment model); `schema/` for `PromQLSchema` discovery from a live Prometheus URL. -- **HTTP route**: `POST /plan/query` (JSON `QuerySpec` in, YAML stream out). Also backs the `bin/asap-query` one-shot CLI. -- **Size estimate**: ~2500 LOC (was ~6000 pre-refactor — saved by picking from shared rule library; still pays the L2 tree + L3/L4 split refactor cost, which is one-time). - -### `deployment-model-asapfusion` (thin) - -- **Data model**: tabular. L2→L3 lowering produces `QueryExpr` with `Source::Table` leaves (and, in future, `Source::Join` when fusion extends to multi-table). Crucially **not time-indexed** — fusion works on arbitrary DataFusion relations; time is just another column if present. -- **L1 opt-out**: fusion consumes a pre-built DataFusion `LogicalPlan` from its caller. `core::query_language` is not invoked. Library-mode deployment model. -- **L2 inherited from DataFusion**: DataFusion's `LogicalPlan` *is* fusion's L2 tree. `core::logical_plan::datafusion` is a thin re-export of `datafusion::logical_expr::LogicalPlan` so deployment models that want to take a DataFusion plan as input have a canonical name for it. -- **L3 intent**: `SubPopulationAnalyticsType` (3 variants: `Count`, `Sum`, `Quantile`) maps to `core::intent_algebra::AggIntent` subset. -- **L4 rules**: picks `BindCmsOnCount` and `BindKllOnQuantile` from `core::optimizer::rules::*` (which already cover fusion's `SketchConfigRule` semantics) + deployment-model-specific `HashModeRule`. Rules operate on DataFusion `LogicalPlan` (via Extension wrapping), not on `QueryExpr` trees — this lets DataFusion's `context.state().optimize` run *after* fusion's rewrites, keeping free reuse of DF's standard optimizer passes. -- **L5 topology**: `core::physical::topology::ZeroStage` (in-process). -- **L5 emitter**: rewritten `datafusion::LogicalPlan`. Not a wire format — this deployment model is library-mode. -- **Executor**: `ASAPExecutor` wraps a DataFusion `SessionContext`. Users construct `deployment-model-asapfusion` in-process. -- **HTTP route**: none by default. -- **Conformance cost**: near zero. Fusion's `SketchConfigRule` logic is replaced with picks from `core::optimizer::rules::*`; its `HashModeRule` stays deployment-model-specific. -- **Size estimate**: ~1800 LOC (was ~3000 pre-refactor; the `SketchConfigRule` code folds into core's shared rule library). - -The sketch microbenchmarks (KLL/CMS) move with the crate and keep running. The TODO items from `asap-fusion/TODO.md` (batch/multi-query execution, time semantics, distributed model) remain open but are now filed against `deployment-model-asapfusion/TODO.md` in-repo. - -### Data plane communication - -Control plane (ASAPController) and data plane (OTel agents / ASAPQuery-backend / DataFusion runtimes) always talk **over wire**, never via in-process calls. This is unchanged from today: - -| Deployment model | Data plane lives in | Wire protocol | -|---|---|---| -| lifecycle | DataCollector (OTel collectors, agent + backend roles) | OpAMP WebSocket (config push) + HTTP POST (`StreamingConfig` → ASAPQuery-backend) + Prometheus scrape (metrics in) | -| query | ASAPQuery-backend (query engine, SketchStore) | HTTP POST `/api/v1/streaming-config` + `/api/v1/plan` (capability-miss callback in) + YAML file on disk (init-container mode) | -| fusion | Caller's DataFusion `SessionContext` | in-process library call (no wire) | - -**Data plane code stays in its original repo.** ASAPController only owns the control plane. The merger doesn't move OTel collectors out of DataCollector, doesn't move the query engine out of ASAPQuery-backend, and doesn't move DataFusion out of asap-fusion's users. Each data plane keeps its own release cadence. - -### Deployment model placement: in-repo or out-of-repo - -Because core owns the L4/L5 infrastructure (not just L1-3), a deployment model crate is small and largely self-contained. That means deployment models can live either: - -- **Inside ASAPController workspace** — `crates/deployment-model-/`, lockstep release with core, cross-deployment-model changes are one PR. -- **In their own downstream repo** — declares `asap-control-core` as a git-tagged dep, releases independently, owns its own CI. - -Both produce functionally identical artifacts because they pick from the same `core::optimizer::rules` library and use the same traits. The placement is a **deployment / team-ownership decision**, not an architectural fork. - -Default recommendation: -- **deployment-model-asaplifecycle** and **deployment-model-asapquery** in ASAPController (they share `deployment-model-asapquery`'s YAML emitter via a workspace `path = "../deployment-model-asapquery"` dep — trivial in-workspace). -- **deployment-model-asapfusion** out-of-tree in `asap-fusion` repo (research project with independent benchmark cadence; depends on `asap-control-core` + `asap-control-optimizer` as published git tags). - -Future deployment models choose whichever placement fits the team that owns them. - -### Asymmetric dependency - -`deployment-model-asaplifecycle` depends on `deployment-model-asapquery` because the `StreamingConfig` YAML emitter is authoritative there. When both live in ASAPController workspace this is a trivial `path =` dep. If one is ever moved out-of-tree, we lift the emitter into core to avoid a cross-repo Cargo dep. - -## 9. Extension point — a 4th deployment model - -With the L4/L5 framework in core, adding a new deployment model is mostly picking + a bit of glue. A hypothetical "edge caching" deployment model that decides which queries to cache at the edge vs. backend would land as: - -``` -crates/deployment-model-edge-cache/ # or your own repo -├── Cargo.toml # depends on asap-control-core -├── src/ -│ ├── lib.rs # impl DeploymentModel for EdgeCacheDeploymentModel -│ ├── rules.rs # pick core rules + add EdgeCacheBindRule -│ ├── topology.rs # TwoStage { edge, backend } TopologyDescriptor -│ ├── cost.rs # impl CostModel — hit ratio × bandwidth -│ └── emit/ -│ └── edge_agent_config.rs # impl PlanEmitter — emits edge-agent YAML -└── tests/ -``` - -Total touch outside the new crate: -- 1 line in `bin/asap-controller/main.rs` to register (if in-workspace) OR a published binary in your own repo -- 1 line in workspace `Cargo.toml` (if in-workspace) -- optional: add `DeploymentModelId::EdgeCache` to `core::registry` - -Typical size: ~500-2000 LOC depending on how deployment-model-specific the rules / cost model / emitter are. If the deployment model accepts core's defaults everywhere, ~300 LOC is realistic. - -### Standalone binary for one deployment model - -If you want a slim binary that runs only one deployment model: - -``` -bin/asap-edge-cache/ -├── Cargo.toml # depends only on runtime + deployment-model-edge-cache, not other deployment models -└── src/main.rs # registers only EdgeCacheDeploymentModel; no DataFusion, no query YAML -``` - -Cargo builds this with its own minimal dep tree — no `datafusion`, no `promql-parser` unless this deployment model needs it. Useful when a deployment is dedicated to one deployment model (e.g. an edge-caching-only service). - -## 10. Wire protocols — what changes, what doesn't - -**Unchanged** (hard contract with other systems): -- `POST /api/v1/streaming-config` on ASAPQuery-backend — consumed YAML format -- `POST /api/v1/plan` on DC controller (from backend capability-miss) — request JSON -- OpAMP `ServerToAgent.remote_config` payload — OTel collector YAML -- Prometheus scrape format - -**Internal** (controller's own surface, still HTTP+JSON for now): -- `POST /plan` — new unified entry point that takes `QueryWorkload` in, returns a plan ID + list of emitted artifacts (URIs). `/api/v1/plan` proxies to this. -- `GET /plans/:id` — plan inspection -- `GET /status` — runtime + deployment model health - -**New** (internal-only, proto): -- `proto/asap_control.proto` defines `Plan`, `PlanNode`, `Expr` for persistence + store-internal serialization. Not on the wire between services. Optional; initial migration skips this and persists via `serde_json`. - -## 11. Dependencies and Cargo surface - -Workspace `Cargo.toml`: - -```toml -[workspace] -resolver = "2" -members = [ - "crates/core", - "crates/runtime", - "crates/deployment-model-asaplifecycle", - "crates/deployment-model-asapquery", - "crates/deployment-model-asapfusion", - "crates/control-proto", - "crates/testing", - "bin/asap-controller", - "bin/asap-query", -] - -[workspace.dependencies] -tokio = "1" -axum = "0.7" -reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } -prost = "0.13" -tonic = "0.12" -serde = { version = "1", features = ["derive"] } -serde_yaml = "0.9" -tracing = "0.1" -thiserror = "1" -# deployment-model-specific deps kept in deployment model crates -``` - -Core depends only on `serde`, `tracing`, `thiserror`, and small utility crates. No HTTP, no DataFusion, no OpAMP. - -Runtime depends on core + `axum` + `reqwest` + `tokio-tungstenite` + `prost` (OpAMP proto). - -Deployment models depend on core. deployment-model-asapquery also pulls in `promql-parser`, `sqlparser`. deployment-model-asapfusion pulls in `datafusion` + `arrow`. deployment-model-asaplifecycle pulls in `sqlparser` + `promql-parser` + the cost-model math crates. - -This matters: a user who only wants `deployment-model-asapfusion` (e.g., an offline benchmark) gets DataFusion but NOT axum/OpAMP. - -## 12. Open questions - -1. **Do we need a cross-deployment-model cost model?** Today DC's lifecycle planner and asap-planner-rs's query planner have overlapping but not identical cost models. Answer for now: keep them separate in deployment model crates; let them re-converge organically. If a third deployment model needs the same model, lift at that point. - -2. **Where does the backend's `ControllerClient.create_plan` call land?** Initially: HTTP `POST /api/v1/plan` on the controller, handled by `deployment-model-asaplifecycle` (same as today). Long-term: could route by `QuerySpec.deployment_model` to a different planner, but that's a follow-up. - -3. **Which deployment model owns `StreamingConfig` YAML emission?** Both `deployment-model-asaplifecycle` and `deployment-model-asapquery` emit it today (DC's `config/generate_streaming_config_yaml` and `output/generator.rs` in `asap-planner-rs`). Plan: **one emitter in `deployment-model-asapquery`**, called from both. This is why `deployment-model-asaplifecycle` depends on `deployment-model-asapquery` in the dependency graph (asymmetric — query does not depend on lifecycle). - -4. **Do we vendor DataFusion's IR into core?** No. `deployment-model-asapfusion` owns its DataFusion-flavored plan; `core::plan::Plan` stays an enum with a variant that wraps a `datafusion::LogicalPlan` behind a feature flag. Core itself never reaches into DataFusion types. - -5. **OpAMP proto: vendored, or from crates.io?** Today DC vendors. Recommend: keep vendored in `proto/opamp.proto`, generate via `prost-build` in `crates/control-proto`. Same thing DC does today, just moved. - -6. **What happens to ASAPQuery-backend's `asap-planner-rs` directory post-migration?** Deleted. ASAPQuery-backend's docker-compose drops the `asap-planner-rs` init container; the controller's `deployment-model-asapquery` now runs in-process (service mode) or as the `asap-query` CLI (one-shot mode). The ASAPQuery-backend repo shrinks by two directories. - -7. **Versioning?** Start at `0.1.0` on the workspace. Deployment models can rev independently later via per-crate versions, but initially lockstep. - -8. **What if a future deployment model can't fit a tree L2?** L2 is currently a per-language tree, mandatory for every deployment model. asap-planner-rs's Phase 4 conformance cost (reverse-engineering PromQL pattern templates into a `PromqlLogicalPlan` tree) was taken deliberately to keep the architecture uniform. If a future deployment model's source language doesn't map naturally onto a tree (e.g. a constraint-based or dataflow-graph query language), that's the moment to re-examine the L2 contract — the `core::logical_plan` module is a single Rust trait + per-language types, not a deep assumption baked across the codebase. Until then, L2 = tree. - -9. **Deployment model placement — in ASAPController workspace, or in own repo?** Both supported, same architecture either way (see §8 "Deployment model placement"). Default: `deployment-model-asaplifecycle` and `deployment-model-asapquery` in ASAPController (easy cross-deployment-model changes, shared YAML emitter); `deployment-model-asapfusion` in the `asap-fusion` repo (research cadence). A new deployment model picks based on team ownership and release cadence preferences. - -10. **How far do shared L4/L5 rules live in core before they become deployment-model-specific?** The rule of thumb: if ≥2 current deployment models would use it, it lives in `core::optimizer::rules::*`. If only 1 deployment model uses it *and* it depends on deployment-model-specific types (DataFusion's `LogicalPlan`, OTel YAML shape), it lives in the deployment model crate. `SketchConfigRule`-style "bind intent to concrete sketch" rules belong in core (shared). `StageAwarePushDown` (which needs DC's stage graph) belongs in `deployment-model-asaplifecycle`. When a rule straddles — e.g. a fusion rewrite that *could* be generalized to any L4-compatible plan — start it in the deployment model; lift to core once a second deployment model wants it. Don't pre-emptively generalize. - -11. **How does the design support non-time-series data (asap-fusion's tabular queries, future OLAP deployment models)?** Already handled — see §3 "Data-model support" and §6 `core::intent_algebra`. `QueryExpr::Scan` wraps a `Source` sum (`TimeSeries` / `Table` / `Join`); `AggIntent::requires() -> DataModel` tags which intents apply to which data models; sketches themselves are data-model-agnostic. asap-fusion uses `Source::Table` exclusively; ASAPQuery uses `Source::TimeSeries`; a future OLAP deployment model picks whichever fits. The only implementation cost is that L4 rules which genuinely don't apply across data models (e.g. "merge overlapping time windows") must gate on `source.data_model()`; data-model-agnostic rules (the `Bind*` family) need no changes. - -### Resolved during the L3 IR cleanup (see §6 `core::intent_algebra`) - -The following questions came up during review of the previous `QueryExpr` draft and are now settled. Listed here so the trail is visible: - -- **Output of `SketchAgg` vs `WindowedAgg`?** Neither node exists at L3 anymore. `SketchAgg` is an L4 sketch-bound node (`SketchExpr::SketchAgg`) emitted by binding rules; `WindowedAgg` was redundant with `Window` over `Aggregate` and removed. Output of `SketchAgg` is a sketch-typed column carrying the partial state; `SketchEstimate` reads it out into a scalar / vector. -- **Is `TopK` different from `Sort + Limit`?** They overlap on heavy-hitter queries but are different concepts. `TopK` is now an `AggIntent` at L3 (heavy-hitter intent — SpaceSaving / CMS-with-heap / Misra-Gries serve it as a single primitive), while generic `Sort + Limit` survives as `QueryExpr` operators for non-heavy-hitter cases (`ORDER BY name LIMIT 10`). L1→L2→L3 lowering picks the intent form when it recognises a heavy-hitter pattern (`ORDER BY count DESC LIMIT k`, PromQL `topk(k, …)`) and the operator form otherwise. -- **What is `JoinSketch`?** A sketch-aware join (KMV / theta / join-sample). Moved to L4 (`SketchExpr::SketchJoin`) so the choice "exact join vs sketch join" is an L4 cost decision against `Join`, not a competing L3 surface. -- **`HistogramQuantile` and `PromQLSubquery` feel out of place** — they were. Both removed from L3. `histogram_quantile` lowers in PromQL L1→L2 to bucket reads + a `Quantile` intent. `[range:resolution]` is a driver that expands into multiple queries during PromQL L1→L2 lowering, not a DAG node. -- **Sketch subtract / delete / estimate operators?** Added: `SketchExpr::SketchSubtract`, `SketchDelete`, `SketchEstimate`. Catalog flags (`subtractable`, `deletable`) gate which sketch families admit them. -- **What's `Dedup` exactly?** SQL `DISTINCT`. Renamed to `Distinct { cols }` and generalised from one column to N. -- **Why differentiate `Quantile` and `QuantileOverTime`?** No reason — the surrounding `Window` already encodes the temporal axis. `QuantileOverTime` removed from `AggIntent`. -- **`WindowedAgg` vs `Window` + `Agg`?** Equivalent; `WindowedAgg` removed. `WindowKind::{Tumbling, Sliding, Session}` lives on the `Window` node so the streaming-window kind is explicit; SQL analytic `OVER (...)` stays on the separate `WindowFunc` node. -- **Need input/output specs more fine-grained than `QueryExpr`?** Done: every L3 edge carries a typed `Schema` (fields + time index + unique-key sets), and §6 lists per-node input/output schemas. - -## 13. Future work - -### Extending the primitive set beyond sketches - -The L4 rule engine + `OptimizerRule` trait are **primitive-agnostic** — nothing in the framework is sketch-specific above the rule library. Future primitive classes land as additional rules against the same trait + additional entries in `CostModel`, not as new translation phases or parallel IRs. Candidates we explicitly anticipate: - -1. **Wavelets** — a sibling approximation family (Haar / DWT + coefficient thresholding). Strong on smooth, low-entropy signals where thresholded coefficient sets dramatically out-compress randomized sketches. Slots in as a new physical alternative for the same `AggIntent` variants sketches serve today (range-sum, heavy-hitter, quantile). No change to L3. - -2. **Reuse / precomputation as first-class primitives.** Materialized views, cached scan results, shared sub-expressions. The cost-model hook already exists (`CostModel::workload_cost` + `ReusedComponent`); the missing piece is rules that *introduce* a reuse node — e.g. "build this aggregate once for `q1`, rewrite `q2` to read from it". Orthogonal to approximation: you can reuse an exact aggregate or an approximate one. - -3. **Other approximation algorithms** — sampling, coresets, online PCA / linear-regression normal equations, naive-Bayes-with-conjugate-priors. Anything with a monoid-shaped build + bounded error fits the same contract a sketch does. - -The guiding principle: **same framework, different rule.** A new primitive class is never a translation-layer change — only a new rule, a new cost-model entry, and (if it introduces a new runtime op) a new physical operator in L5. - -## 14. Glossary - -Terms that are project-specific or that get conflated. Where the term has a Rust counterpart, the type is shown in backticks. - -### Architecture roles - -- **Deployment model** — A concrete bundle of (L4 rule choices) + (L5 topology) + (emitter), packaged as a `deployment-model-*` crate. Three exist today: `asaplifecycle`, `asapquery`, `asapfusion`. -- **Stage** (`StageId`) — A *categorical tier* in the data lifecycle: edge / gateway / backend / in-process. Roles, not instances. Declared by the deployment model's `TopologyDescriptor`. -- **Executor** (`Executor`) — A *concrete runtime instance* occupying a stage: a specific OTel agent, the ASAPQuery-backend process, a DataFusion `SessionContext`. One stage may have N executors (e.g. 50 edge agents). Carries `id`, `stage: StageId`, `capabilities`, `address`. -- **Topology** (`TopologyDescriptor`) — Declares which stages exist + how they connect (3-stage / 1-stage / 0-stage). Categorical, not instance-level. -- **DeploymentConstraints** — Trait object owned by each deployment model. Carries memory budgets, network topology, available sketch backends, and the registered `Executor` list. Threaded through L4 rules and the stage allocator. - -### Layer drivers - -- **RuleEngine** — Generic L4 driver in core (fixed-point iteration + cycle detection + priority ordering). One implementation; same code for every deployment model. -- **OptimizerRule** — Trait for an L4 rewrite. Categories: `PushDown | Fusion | Elim | Bind | StageRouting`. Deployment models pick which rules from `core::optimizer::rules` + add their own. -- **CostModel** — Trait scoring a plan's accuracy / latency / dollars. Generic implementations in core; deployment-model-specific cost models (DC's delta / online / pareto / TCO) live in their crates. -- **StageAllocator** — Generic L5 algorithm in core. Colors the L4-bound DAG by `StageId`. **Stage-granularity only — no executor knowledge.** One implementation; same code for every deployment model. -- **PhysicalPlanner** — Per-deployment-model L5 *driver* (one impl per deployment model). Calls `StageAllocator`, then fans the per-stage sub-DAGs out to executors via `DeploymentConstraints::executors()`, then produces the deployment-specific output type. Compare to allocator: planner = whole L5 driver, allocator = the stage-coloring step it uses. -- **PlanEmitter** — Per-deployment-model trait that serialises the planner's output to its wire format (OpAMP `RemoteConfig` / `streaming_config.yaml` / rewritten DataFusion `LogicalPlan`). - -### IR by layer - -- **L1 — query language** — Raw query string + parser (PromQL / SQL / DataFusion / ElasticDSL). -- **L2 — logical plan** — Per-language relational algebra tree (`PromqlLogicalPlan`, `SqlLogicalPlan`, etc.). -- **L3 — intent algebra** (`QueryExpr`, `core::intent_algebra`) — Symbolic, intent-only IR. No sketch type, no params. `AggIntent` carries accuracy targets only. The "algebra" is the operator surface (`Scan`, `Filter`, `Aggregate`, `Window`, …); the algebra is *intent-only* because no sketches have been bound yet. -- **L4 — sketch algebra** (`SketchExpr`, `core::sketch_algebra`) — Same DAG shape as L3, but sketches now committed (kind + params). Produced by L4 binding rules; consumed by L5 emitters. Note the naming inversion vs. earlier drafts: "sketch algebra" is L4 (where sketches actually live), not L3. -- **L5 — physical plan** — Stage-assigned, executor-targeted, ready to serialize. Produced by `PhysicalPlanner`, written out by `PlanEmitter`. - -### Metadata sources (see §6 "DAG schema, DB schema, sketch catalog") - -- **DAG schema** (`Schema`) — Columns + types + `unique_keys` carried on every L3/L4/L5 edge. Type-checked locally at each node. -- **DB schema** / **source schema** (`SchemaCatalog`) — Data-plane metadata (Prometheus TSDB / SQL `information_schema` / DataFusion catalog). Read by `core::lower::*` for L1→L2 symbol resolution. -- **Sketch catalog** (`SketchCatalog`) — Runtime registry of available primitives: what sketches exist, mergeability, deletability, parameter ranges, accuracy model. Consulted by L4 binding rules. **Not a schema** — a catalog of available primitives, not a description of a stream. - -### Workload + identity - -- **QueryWorkload** — Top-level controller input: one or more `QuerySpec`s plus workload features (batch vs streaming, reuse opportunity, data source). Arrives via HTTP POST, OpAMP capability-miss callback, YAML file, or query-log replay. -- **QuerySpec** — A single query: source-language string + accuracy / latency / cost target. - -## 15. Success criteria - -The migration is done when: - -1. `asap-controller` binary runs and passes DC controller's existing integration tests (OpAMP push, backend config POST, SLA replan). -2. `asap-query` binary takes the same YAML input asap-planner-rs does today and produces byte-identical `streaming_config.yaml` + `inference_config.yaml` (fuzz-test against a corpus of fixtures). -3. ASAPQuery-backend's docker-compose no longer starts `asap-planner-rs`; the controller handles both shapes. -4. `asap-fusion`'s microbenchmarks still run under `deployment-model-asapfusion` with identical numbers. -5. The `DataCollector/controller/`, `ASAPQuery/asap-planner-rs/`, `ASAPQuery-backend/asap-planner-rs/`, and `asap-fusion/` directories are deletable (or already deleted) without breaking any currently-running deployment. -6. A new hypothetical deployment model can be added with zero changes outside its crate + one line in `bin/asap-controller/main.rs`. - -## 16. ADR — Capability consolidation + `algebra/`/`planner/`/`config/` retirement (2026-05) - -This section records two migration steps that brought the controller into its current shape. - -### 16.1 PR #129 — Four-merged-then-cleaned capability tables - -Before PR #129 there were **four** overlapping capability tables in the controller: - -| Source | Location | Shape | -|---|---|---| -| YAML at `controller/sketch_capabilities.yml` | filesystem | per-sketch perf profile, runtime-loaded | -| Compiled-in defaults | `algebra/optimizer.rs::sketch_capability` | per-sketch perf profile, hard-coded | -| `SketchKind` enum | `sketch_algebra/params.rs` | sketch type tag | -| `Capability` / `SketchKindHandle` | `asap_tier_analysis.rs` (PR #128) | query-side dispatch tag invented per-query | - -All four collapsed into one module: `controller/src/sketch_algebra/capability.rs`. The four-way map is now: - -- `SketchCapability` + `SupportedIntent` — per-sketch perf profile, read by L4 cost model + L5 physical planner (formerly the YAML + compiled-in copies). -- `Capability` + `SketchKindHandle` — query-side capability tag, used by `data_plane`'s ASAP-tier reducer (formerly the per-query invention in PR #128). -- `capability_for(intent: &AggIntent) -> Option` — the **semantic** intent → ASAP-tier dispatch bridge. The new signature replaces the older `capability_for(query_func: &str)` string-keyed lookup. PromQL → `intent_algebra::lower` → `AggIntent` → (this fn) → `Capability`. The ASAP-tier analyzer at `controller/src/asap_tier_analysis.rs` is now a thin facade around this single function. -- `default_capability_table()` / `load_capability_overrides()` — compiled-in defaults + YAML override loader. - -The four-table fragmentation reflected partial consolidations that never finished; once `Capability` was the canonical query-side tag (PR #128) the perf-profile + override-loader story had a natural home next to it (PR #129). - -### 16.2 Refactor 2026-05 (`refactor/controller-layered-cleanup`) — `algebra/` / `planner/` / `config/` retirement - -The pre-refactor layout grew organically as the controller absorbed three legacy code paths (the DC `algebra/` IR, the `planner/` cost models, the `config/` per-deployment emitters). Each had its own naming convention and module structure. This refactor restructures `controller/src/` to mirror design.md §5's target module split without splitting into multiple crates. The retirements: - -| Retired path | Replacement | -|---|---| -| `controller/src/algebra/expr.rs` | `controller/src/intent_algebra/relational.rs` | -| `controller/src/algebra/lower.rs` | `controller/src/intent_algebra/lower.rs` (the L2→L3 lowering + converter) | -| `controller/src/algebra/directory.rs` | `controller/src/physical/sketch_catalog.rs` | -| `controller/src/algebra/physical.rs` | `controller/src/physical/planner.rs` | -| `controller/src/algebra/allocator.rs` | `controller/src/physical/allocator.rs` | -| `controller/src/algebra/plan.rs` | `controller/src/physical/plan.rs` | -| `controller/src/algebra/optimizer.rs` | `controller/src/optimizer/engine.rs` | -| `controller/src/planner/cost_model.rs` | `controller/src/optimizer/cost/mod.rs` | -| `controller/src/planner/{delta,online}_cost_model.rs` | `controller/src/optimizer/cost/{delta,online}.rs` | -| `controller/src/planner/{pareto,tco,wire_cost}.rs` | `controller/src/optimizer/cost/{pareto,tco,wire}.rs` | -| `controller/src/planner/rules.rs` | `controller/src/optimizer/rules/mod.rs` | -| `controller/src/planner/baseline_planner.rs` | `controller/src/optimizer/baseline.rs` | -| `controller/src/planner/stage_split.rs` | `controller/src/physical/stage_split.rs` | -| `controller/src/analyzer.rs` | `controller/src/pipeline.rs` | -| `controller/src/stage_split/` | `controller/src/physical/colored_dag/` | -| `controller/src/query_language/` | *(deleted)* — was moved to `controller/src/query_parser/language/`, then removed as unwired scaffolding; the real L1 parsers are `query_parser/{promql,sql}.rs` | -| `controller/src/config/workloads.rs` | `controller/src/workload.rs` | -| `controller/src/config/{stage_config*,otap,telegraf,agent,backend,asapquery_backend,precompute}.rs` | `controller/src/emit/{stage_config,otap,telegraf,agent,backend,asapquery_backend,precompute}.rs` | -| (new) | `controller/src/emit/trait_def.rs` — `PlanEmitter` trait placeholder | -| (new) | `controller/src/optimizer/trait_def.rs` — `OptimizerRule` trait placeholder | -| (new) | `controller/src/deployment_model.rs` — `DeploymentModelRegistry` + `DeploymentModelId` placeholder | -| (new) | `controller/src/physical/topology.rs` — `Topology` descriptor re-export | - -`controller/src/lib.rs` keeps thin `pub use` aliases (`pub use pipeline as analyzer;`, `pub use emit as config;`, `pub mod algebra { ... }`, `pub mod planner { ... }`, `pub use physical::colored_dag as stage_split;`) so the historical paths `controller::analyzer::*`, `controller::config::*`, `controller::algebra::*`, `controller::planner::*`, `controller::stage_split::*` keep resolving for external consumers (the `controller` bin's `use controller::algebra;` etc.) without further source churn. Internal source code is migrated to the new paths. - -**TODOs left from the refactor:** - -- `controller/src/emit/stage_config.rs` (3,020 lines, formerly `config/stage_config.rs`) was moved whole rather than split into `emit/opamp.rs` + `emit/streaming_config.rs` + `emit/inference_config.rs` per design.md §5. The monolith mixes OTel-collector YAML emit, ASAPQuery-backend JSON emit, storage-routing JSON emit, and shared internals; a clean split needs ownership reorganisation, not file renames. Tracked for a follow-up. -- `controller/src/intent_algebra/relational.rs` carries the **L2 relational** `QueryExpr` the `query_parser` front ends emit; `lower.rs` is the single pass that converts it (sketch-fusion folded in) to the canonical `intent_algebra::{query_expr,agg_intent}` L3 types. The sketch-fused `SketchAgg` / `WindowedAgg` variants and the standalone sketch-lowering pass that once produced them have been retired. Several controller modules (query_parser, physical/, optimizer/, emit/) still reference `relational` for shared leaf types; fully *deleting* the L2 tree is gated on the canonical `Predicate` covering the remaining `ScalarExpr` variants — but it is the real, current L2 IR (formerly misnamed `legacy_expr`), not removable debt. -- `controller/src/optimizer/trait_def.rs` (`OptimizerRule`), `controller/src/emit/trait_def.rs` (`PlanEmitter`), `controller/src/deployment_model.rs` (`DeploymentModelRegistry`) ship as placeholders — the existing free-function emitters and concrete rule loops still drive behaviour. Migrating them onto the trait surfaces lands when the per-deployment-model crate split lands. - diff --git a/control_plane/docs/physical-planning.md b/control_plane/docs/physical-planning.md new file mode 100644 index 00000000..33b5e398 --- /dev/null +++ b/control_plane/docs/physical-planning.md @@ -0,0 +1,339 @@ +# Physical planning for collector and backend execution + +> Status: proposed +> +> MVP relation: required to turn one Planner selection into matching collector +> and backend runtime plans. +> +> Scope: the ASAPQuery-backend physical-planning step between ASAPPlanner's +> selected post-ASAP workload DAG and the two runtime executors: +> ASAPCollector and the ASAPQuery data plane. + +## TL;DR + +ASAPPlanner selects a logical plan. That plan says which summaries and exact +operations answer a workload, but it intentionally does not choose machines, +shards, runtime windows, transport modes, or storage routes. + +ASAPQuery-backend performs one physical compile that produces both runtime +views of the decision: + +```text +selected post-ASAP workload DAG + | + v + physical compilation + / \ + v v +CollectorSubplan BackendSubplan +CollectorPlan(s) BackendPlan +``` + +The two subplans share the same plan and materialization identities. They are +never derived independently and are never considered active unless both sides +confirm a compatible decision. + +## 1. Why this layer exists + +ASAPPlanner owns logical choices such as: + +- exact accumulator versus approximate summary; +- summary family, algorithm, and parameters; +- reduction and grouping strategy; +- logical sharing and composition; +- summary readout; and +- exact fallback. + +The runtime still needs deployment decisions that do not belong in Planner: + +- which collector or backend stage runs each operation; +- how logical state is sharded or shared; +- which streaming panes materialize a query time range; +- whether state is sent as raw observations, full summaries, or deltas; +- where state is stored and queried; +- when a new plan becomes active; and +- how incompatible or failed updates are rolled back. + +Skipping this layer causes two common failures: + +1. treating a logical `SummaryAgg` as though it already names a collector + process and runtime configuration; and +2. deriving collector and backend plans separately, allowing family, + parameters, grouping, windows, or identities to drift. + +The physical compiler closes both gaps in one operation. + +## 2. Input contract + +The compiler receives: + +- the selected post-ASAP DAG for the whole workload; +- stable workload/query correlation information; +- collector and backend capability snapshots; +- deployment topology and stage boundaries; +- workload statistics and resource constraints; +- runtime window, freshness, and retention policy; and +- transmission and storage policy. + +Shared logical nodes remain shared at this boundary. The compiler must not +first flatten the workload into independent per-query or per-metric rows. + +The selected DAG may contain summary producers, estimates, merges, +subtractions, deletes, joins, exact operations, shared sub-DAGs, and +`KeepPreAsap` fallback. A pinned Planner revision may also carry explicit +execution phases and result guarantees. The compiler consumes those canonical +values rather than defining local equivalents. + +## 3. Output contract + +One compile returns one `CompiledPlan` with: + +- a collector subplan containing one `CollectorPlan` for every targeted + collector; +- a backend subplan containing one `BackendPlan` for the ASAPQuery data + plane; +- a shared plan envelope; +- content-addressed materialization identities; and +- a validation record showing that both subplans were produced from the same + selected DAG and capability snapshots. + +### Shared plan envelope + +| Field | Meaning | +| --- | --- | +| `plan_id` | Content-addressed identity of the logical selection, topology identity, and semantic constraints. | +| `plan_version` | Ordered update within the same plan identity, such as changed sizing or lifecycle policy. | +| `activation` | Earliest time at which both subplans may become authoritative. | +| `expiry` | Optional time after which the plan may no longer produce or serve state. | +| `backend_compat` | Compatibility identity for BackendPlan and emitted summary-state schemas. | +| `planner_revision` | Immutable ASAPPlanner revision used for the selection. | + +Mutable lifecycle or sizing fields do not silently change the identity of an +existing version. Reusing the same `(plan_id, plan_version)` for different +content is invalid. + +### Materialization identity + +A materialization is the physical state built for one selected logical +summary producer. Its identity includes every property required for safe +reuse and merge, including: + +- bound source and filters; +- summarized value/item; +- summary family, algorithm, and parameters; +- reduction and grouping layout; +- logical/physical window contract; and +- state schema compatibility. + +Placement, transport cadence, or storage location may change without +pretending a semantically different summary is the same materialization. + +Explain or viewer node IDs are traceability metadata, not materialization +identity. + +## 4. Partitioning the selected DAG + +The baseline partition follows data availability: + +- operations that consume new observations and construct maintained state + belong on the update side; +- operations that read, merge, or transform maintained state into query + answers belong on the readout side; and +- `KeepPreAsap` remains exact backend/archive execution. + +In the current Planner vocabulary, `SummaryAgg` is the principal update-side +boundary and `SummaryEstimate` is a readout. Summary merge and other state +composition are placed according to topology, capabilities, and transport +cost without changing their logical semantics. + +If the pinned Planner revision provides explicit execution availability or +phase assignments, those validated phases are authoritative. The compiler +must not infer a conflicting phase from node names. + +An operation may be assigned only to an executor that advertises its full +semantics. A nameable Planner alternative is not automatically deployable. + +## 5. Collector subplan + +The collector subplan follows ASAPCollector's +[collection-plan interface](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). + +For each assigned collector, it specifies: + +- target collector and capability snapshot; +- the shared plan envelope; +- source metrics and matchers; +- materialization identities; +- summary family, algorithm, parameters, and accuracy requirement; +- reduction and grouping semantics; +- concrete streaming windows and lateness; +- local sharding; +- raw, full, or delta transmission; and +- backend endpoint/schema compatibility. + +OpAMP is the delivery transport. The payload is the versioned +`asap-collector-plan.yaml` document, not ASAPPlanner's Rust IR and not the +collector's complete bootstrap configuration. + +## 6. Backend subplan + +The backend subplan follows +[`backend-plan.md`](backend-plan.md). + +It specifies: + +- the same plan envelope; +- every expected materialization; +- the collector assignments producing its state; +- exact summary family, algorithm, parameters, grouping, and windows; +- ingestion/storage destination; +- query capabilities and readouts satisfied by the materialization; +- remaining backend-side operators; and +- selected result guarantees when provided by Planner. + +The backend does not reconstruct the chosen summary from query text or stored +state. It installs and executes the control plane's exact decision. + +## 7. Sharing, sharding, and merge + +One shared logical producer remains one logical materialization even when it +serves several queries. The backend may associate several routing/readout +entries with that one materialization. + +A materialization may have several physical producers when sharded across +collectors. The compiler records all producers and inserts a compatible merge +at the selected boundary. A merge is legal only when every input agrees on +the materialization contract and the selected family supports merge. + +Sharding does not create several unrelated logical summaries, and sharing +does not allow consumers with incompatible filters, reductions, grouping, +windows, parameters, or guarantees to reuse state. + +## 8. Window and transmission decisions + +Planner time ranges express query semantics. The physical compiler chooses +streaming panes capable of answering those ranges. + +For the MVP: + +- panes are anchored and tumbling; +- pane composition must exactly cover each claimed query range; +- allowed lateness and watermark behavior are explicit; +- incompatible alignment is rejected; and +- freshness policy is shared with the backend readiness check. + +Transmission is independent of logical summary choice: + +- `raw` forwards selected observations for exact/backend execution; +- `full` sends complete summary state; and +- `delta` sends ordered state changes plus periodic full checkpoints. + +Delta is legal only when collector and backend advertise the same state, +sequence, and checkpoint semantics. Every payload identifies its plan, +materialization, producer, window, sequence, and base/checkpoint. + +### Aggregation placement + +Planner's reduction and grouping are logical requirements. The physical +compiler decides where that reduction runs without changing them. For example, +`sum by (region) (rate(http_requests_total[5m]))` may maintain one summary per +`region` at collectors, while a query with no grouping may use one +whole-workload materialization. A per-series or per-group result must never be +silently collapsed into a global result. + +### State representation + +Dense versus sparse state is a physical representation choice, not a new +summary choice. For example, an HLL materialization may use sparse state for +low-cardinality groups and promote to dense state as cardinality grows, but +both representations must preserve the same HLL parameters, merge semantics, +wire compatibility, and accuracy contract. + +The compiler may select a representation only when both producer and consumer +advertise compatible support. Otherwise it uses the declared fallback or +rejects the plan. Representation details such as collector configuration field +names belong in the collector interface, not in this design. + +## 9. Compile and activation sequence + +1. Validate the selected DAG against both capability snapshots. +2. Allocate update/readout operators and physical producers. +3. Choose compatible windows, transmission, and storage routes. +4. Construct materialization identities. +5. Emit both subplans from the same in-memory decision. +6. Validate cross-subplan equality for all shared contracts. +7. Stage both subplans before `activation`. +8. Require backend installation and collector semantic application reports. +9. Route queries to the new plan only after both sides report compatible + active identities. +10. Retire old state after its readers and lateness horizon drain. + +If any step fails, the previous unexpired plan remains authoritative. A +partial push, OpAMP delivery acknowledgement, file write, or process restart +does not constitute plan activation. + +## 10. Fail-closed rules + +The compiler or runtime rejects the plan when: + +- an assigned executor lacks a required family, algorithm, grouping, phase, + readout, window, or transmission capability; +- collector and backend materialization contracts differ; +- a required accuracy guarantee is unknown or insufficient; +- a merge combines incompatible state; +- delta sequencing/checkpoint semantics do not match; +- plan versions conflict or lifecycle conditions disallow activation; or +- semantic application evidence is missing. + +It must never substitute another family, parameter, grouping layout, +accuracy target, or transmission semantics to make an invalid plan appear +deployable. + +## 11. Example + +For: + +```promql +quantile_over_time(0.95, request_duration_seconds{region="us-east"}[5m]) +``` + +Planner may select a per-entity DDSketch summary and a p95 readout. The +physical compiler may then: + +- assign DDSketch construction to selected collectors; +- choose one-minute panes that compose into the five-minute query range; +- transmit deltas every ten seconds with periodic full checkpoints; +- declare one shared DDSketch materialization in BackendPlan; and +- route the p95 query readout to that materialization. + +The compiler does not change DDSketch to KLL, change the accuracy parameter, +or aggregate series together merely because another physical layout would be +cheaper. Such a change requires selection of a different valid Planner +candidate. + +## 12. Non-goals + +This document does not define: + +- query parsing or summary selection; +- internal Planner serialization; +- exact protobuf field numbers; +- summary-state byte encoding; +- collector bootstrap configuration; +- storage-engine implementation; or +- query-engine implementation details. + +## 13. Definition of done + +The compiled-plan design is satisfied when: + +- one compile produces both subplans; +- all shared identities and semantic fields match; +- shared Planner nodes remain shared materializations; +- sharded producers merge only under a valid contract; +- unsupported shapes fail before activation; +- both plans stage and activate atomically from the user's perspective; +- emitted state carries the active identities; +- the data plane serves without replanning; and +- a deliberately mismatched or partially applied plan is rejected in an + end-to-end test. diff --git a/control_plane/docs/query-to-edge-scope-and-sparse.md b/control_plane/docs/query-to-edge-scope-and-sparse.md deleted file mode 100644 index 22cbf54a..00000000 --- a/control_plane/docs/query-to-edge-scope-and-sparse.md +++ /dev/null @@ -1,163 +0,0 @@ -# Edge Aggregation Scope (`mode`) and Sparse HLL (`hll_sparse`) - -This note documents how the control plane maps query/workload signals to two -per-metric knobs it emits into the edge agent's fused `asap_edge` processor -config (`processors.asap_edge.metrics[]`): - -1. **`mode`** — the aggregation **SCOPE**: `per_series` (default) vs - `whole_stream`. -2. **`hll_sparse`** — opt-in to the in-memory **sparse HLL** base (bool, - default `false` = dense). - -The edge side of both knobs landed in ASAPCollector: - -- **#471** added `MetricFamily.mode` (operator-facing string `per_series` / - `whole_stream`), plumbed into `precompute.PrecomputeConfig.Scope` - (`AggMode` = `ModePerSeries` / `ModeWholeStream`). #471 *folded the legacy - `GlobalAggregation` bool INTO this scope*: `whole_stream` is the unified - `effectiveScope`, semantically identical to the old empty-`aggregate_by` → - `GlobalAggregation` behaviour. -- **#472** added the sparse-HLL base (`NewHLLWrapperSparse`), selected per-HLL - by `hll_sparse`. The sparse base is *lossless*: it auto-promotes to dense - once enough registers are set, and its serialized output is byte-identical to - dense for the same inputs. Default `false` (dense) = today's behaviour. - -Both knobs are emitted from a single site: `control_plane/src/emit/stage_config.rs`, -in `emit_edge_yaml_asap_edge`'s sketch-family loop (the `e: Mapping` per-metric -entry assembly). The legacy single-processor emitter in `emit/agent.rs` -(`build_processor_block`) does **not** emit a `metrics[]` family map — its -`mode` field is the unrelated `ProcessorMode` (Window/Batch windowing strategy), -not this aggregation scope — so it needs **no** change. - -## `mode` (aggregation scope) - -### The signal: effective grouping × family - -At the emit site each sketch entry already computes its **effective -`aggregate_by`** = the metric's `metric_to_grouping_labels` minus its -`metric_to_item_label` (the inner heavy-hitter dimension is the sketch -*subject*, not a grouping key). Call this `effective_by`. - -- `effective_by` **non-empty** ⇒ the edge keys ONE sketch per group (e.g. per - `zone`). This is `per_series` (per-group) scope. -- `effective_by` **empty** ⇒ the edge factory collapses grouping into a single - attr-less sketch. Per #471 that *is* `whole_stream`. - -But "empty grouping" alone is **not** sufficient: the per-series quantile -families (DDSketch / KLL) also carry no grouping (a `quantile_over_time` reduces -*within* a single series), and an empty grouping there means "no extra keying", -**not** "collapse the whole stream". Emitting `whole_stream` for them would be -wrong. - -So the rule combines grouping **and family role**: - -``` -whole_stream ⇔ effective_by.is_empty() AND family ∈ {HLL, CMS, CountSketch} -``` - -The item-counting / frequency families (HLL = distinct-count, CMS = frequency, -CountSketch = top-k / frequency) with an empty effective grouping are exactly -the *genuinely global* aggregates the planner emits for `count(distinct …)` -(no `by`), global top-k, and global frequency — #471's own `WholeStream` -examples ("distinct-count / global-quantile / grand-total / global-top-k"). - -### What we emit - -- **whole_stream**: emit `mode: whole_stream`. -- **per_series**: emit **nothing**. `per_series` is the edge default (an - omitted/empty `mode` parses to `ModePerSeries`), so omitting keeps the YAML - byte-identical to the pre-#471 emit for every metric that is not a genuine - whole-stream global aggregate. This is the deliberate choice: explicit - `whole_stream` where it matters, implicit `per_series` everywhere else. - -### Back-compat / safety - -`whole_stream` ≡ the folded `GlobalAggregation` (#471's `effectiveScope`), so -emitting it for an HLL/CMS/CountSketch that **today** already produces an empty -`aggregate_by` is semantically identical to current behaviour — no wire-level -change, just an explicit name for the existing collapse. - -Crucially we emit `whole_stream` **only** where `effective_by` is empty. A -heap-bearing CountSketch (warm top-k) or any item-counting family that *does* -carry per-group keying (non-empty `effective_by`) stays `per_series`, so we -never newly collapse a metric that needs per-group sid minting at the backend -(the failure mode the long comment in `emit_edge_yaml_asap_edge` warns about: -empty `aggregate_by` + GlobalAggregation breaks backend sid minting for -attr-less series). We do not *introduce* empty-grouping; we only *name* the -scope the existing emit already produces. - -## `hll_sparse` (sparse HLL base) - -Emitted **only** on HLL-family entries (`SketchKind::Hll`); never on any other -family. - -### The rule (scope-driven) - -``` -hll_sparse = !whole_stream (i.e. true for per_series HLL, false for whole_stream HLL) -``` - -- **whole_stream HLL** → a single high-cardinality instance per metric (distinct - values over the whole stream). It promotes to dense almost immediately, so the - sparse base buys nothing and only adds promotion churn → `hll_sparse: false` - (dense). -- **per_series HLL** → one HLL per group; most groups are low-cardinality (e.g. - distinct `user_id`s per `region`), where the sparse base is a large memory win - and auto-promotes the few hot groups → `hll_sparse: true`. - -Because the sparse base is lossless and auto-promoting, the scope-based default -carries **zero** accuracy or wire risk. - -### Cardinality hint (deferred follow-up) - -The ideal refinement is a per-metric cardinality hint: -`WorkloadCharacteristics.distinct_keys_per_window: Option` -(`control_plane/src/types.rs:45`). When present and **large** — above the -dense-crossover (~4096 non-zero registers ≈ the in-memory promotion point) — even -a `per_series` HLL should go dense (`hll_sparse: false`) to skip the -sparse→dense promotion. When small or absent, sparse. - -**This hint is not reachable at the emit site today.** The only input to -`emit_edge_yaml_asap_edge` is `EdgeStageConfig` -(`control_plane/src/physical/colored_dag/emitter.rs`), which carries no -`WorkloadCharacteristics` and no per-metric cardinality map. Plumbing one -through the L5 stage config is a follow-up. Until then we use the scope-based -default above, which is safe (sparse is lossless). Per the default-OFF safety -rule, we never make a metric sparse when scope can't be determined — but for the -HLL family scope is always determinable here (empty vs non-empty -`effective_by`), so HLL entries always carry an explicit `hll_sparse`. - -## Worked examples - -| Query | Family | grouping_labels | item_label | `effective_by` | `mode` emitted | `aggregate_by` | `hll_sparse` | -|---|---|---|---|---|---|---|---| -| `count(distinct user_id)` (no `by`) | HLL | — | `user_id` | empty | `whole_stream` | (omitted) | `false` (dense) | -| `count by (region)(distinct user_id)` | HLL | `[region]` | `user_id` | `[region]` | (omitted ⇒ per_series) | `[region]` | `true` (sparse) | -| `quantile_over_time(0.99, latency[5m])` (per-series) | DDSketch / KLL | — | — | empty | (omitted ⇒ per_series) | (omitted) | n/a (not HLL) | -| `quantile by (zone)(latency)` | DDSketch | `[zone]` | — | `[zone]` | (omitted ⇒ per_series) | `[zone]` | n/a | -| `topk(10, endpoint_qps)` (global) | CountSketch | — | `endpoint` | empty | `whole_stream` | (omitted) | n/a | -| `topk(10, sum by (zone)(...))` | CountSketch + heap | `[zone]` | `endpoint` | `[zone]` | (omitted ⇒ per_series) | `[zone]` | n/a | -| global frequency `endpoint_request_freq` | CMS | — | `endpoint` | empty | `whole_stream` | (omitted) | n/a | -| `sum by (zone)(http_requests_total)` | Sum (not a sketch) | `[zone]` | — | `[zone]` | (no mode on sum entries) | `[zone]` | n/a | - -Note the byte-stability column: every row that emits no `mode` (the per-series -quantile and per-group cases) keeps its `metrics[]` entry byte-identical to the -pre-#471 emit; only the genuine whole-stream globals gain a `mode:` key, and -only HLL entries gain `hll_sparse:`. - -## Tests - -In `control_plane/src/emit/stage_config.rs`: - -- `fused_asap_edge_emits_single_pipeline_and_metrics_list` (extended): asserts - the global item-counting families (HLL / CountSketch / CMS, no grouping) emit - `mode: whole_stream` and no `aggregate_by`; the per-series quantile families - (DDSketch / KLL) and the sum entry emit no `mode`; the whole-stream HLL emits - `hll_sparse: false`; no non-HLL family carries `hll_sparse`. -- `fused_asap_edge_per_group_hll_is_per_series_and_sparse`: a per-group HLL - (`grouping=[region]`, `item_label=user_id`) emits `aggregate_by: [region]`, - no `mode` (per_series), and `hll_sparse: true`. -- `fused_asap_edge_quantile_only_omits_mode_and_sparse`: byte-stability guard — - a quantile-only (DDSketch + KLL) plan emits neither `mode:` nor `hll_sparse`. -- `fused_asap_edge_keys_are_a_subset_of_asapedgeprocessor_config_go`: extended - to allow `mode` and `hll_sparse` as known `MetricFamily` keys. diff --git a/control_plane/docs/query-to-sketch-translation.md b/control_plane/docs/query-to-sketch-translation.md deleted file mode 100644 index 4d0f0a5a..00000000 --- a/control_plane/docs/query-to-sketch-translation.md +++ /dev/null @@ -1,754 +0,0 @@ -# Query-to-Sketch Translation: How QL Maps to Sketch Execution - -This document explains how a PromQL or SQL query is translated through the -control plane's five-layer architecture and ultimately mapped to sketch-based -distributed execution. - -## 1. Five-Layer Architecture - -The control plane is structured as a five-layer pipeline. Each layer has a -clear input, output, and responsibility: - -``` -Query workloads - │ - │ Layer 1 — Query Language - │ (PromQL, SQL, DataFusion, ElasticDSL, ...) - ▼ -Language-specific AST - │ - │ Layer 2 — Language Logical Plan - │ (each language's own relational/query algebra) - ▼ -Language Logical Plan - │ - │ Layer 3 — Sketch Logical Plan (Sketch Algebra) - │ (language-independent, implementation-independent) - ▼ -Sketch Logical Plan - │ - │ Layer 4 — Sketch Optimizer - │ (rewrite rules on the sketch logical plan) - ▼ -Optimised Sketch Logical Plan - │ - │ Layer 5 — Physical Execution Plan - │ (concrete implementations for a specific deployment) - ▼ -Physical Plan (edge processors, backend sketchDB, backend original DB, object store) -``` - -### What each layer owns - -| Layer | Input | Output | Responsibility | -|---|---|---|---| -| **1. Query Language** | query string | language AST | grammar, parsing | -| **2. Language Logical Plan** | AST | language-specific relational plan | language semantics (PromQL instant/range vectors, SQL frames, Elastic buckets) | -| **3. Sketch Logical Plan** | language plan | sketch algebra tree (`QueryExpr`) | **what** to compute: aggregation intent + accuracy requirement + window semantics — no sketch names, no implementation details | -| **4. Sketch Optimizer** | sketch plan + deployment constraints | optimised sketch plan | cost-aware rewrites: push-down, fusion, elimination, budget-driven deferral — considers physical deployment constraints (memory budgets, network topology, available backends) | -| **5. Physical Plan** | optimised plan + deployment config | executable plan | **how** to execute: edge processors (sketch build), backend sketchDB (merge + query), backend original DB (exact), object store (raw backup) | - -### Key design principle - -**Layers 1–3 are query-language-independent and workload-independent.** -They define *what* to compute without reference to any specific query language, -sketch implementation, or deployment topology. A `Quantile { φ=0.99, accuracy=0.01 }` -intent is the same whether it came from PromQL, SQL, DataFusion, or ElasticDSL, -and whether the deployment is a single node or a 1000-agent fleet. - -**Layer 4 is deployment-constraint-aware.** -The optimizer considers physical deployment constraints — memory budgets per stage, -network bandwidth, available backends — when applying cost-based rewrite rules -(e.g., deferring a sketch from Agent to Backend when the agent memory budget is -exceeded, or fusing TopK when the downstream merge is expensive). - -**Layer 5 is deployment-specific.** -The physical planner commits to concrete implementations based on the specific -setup: edge processors, backend sketchDB, backend original DB, or object store. - -### `AggIntent` — the Layer 3 aggregation vocabulary - -| `AggIntent` variant | Meaning | Physical candidates (Layer 5) | -|---|---|---| -| `Quantile { quantiles, accuracy }` | "I need quantile estimates at these φ values within this error" | DDSketch, KLL, t-digest, PromSketch EHKLL | -| `Cardinality { accuracy }` | "I need a distinct-count estimate within this error" | HLL, UnivMon, PromSketch EHUniv | -| `Frequency { accuracy }` | "I need frequency estimates within this error" | CountSketch, CountMinSketch | -| `Extrema { min, max }` | "I need exact min/max" | ExactMinMax, DDSketch at φ=0/1 | -| `PerPartition { inner, keys }` | "Run inner once per distinct key tuple" | Hydra, per-key sketch instances | -| `Exact(Sum\|Count\|Avg\|Min\|Max)` | "No sketch benefit — exact computation" | Raw passthrough, DB-side | - -The flow: -- **Layers 1–2** (parsers): "this query needs a quantile at φ=0.99" → `Aggregate { Quantile(0.99) }` -- **Layer 3** (lowering): `Aggregate` → `SketchAgg { AggIntent::Quantile }` (shared by all languages) -- **Layer 4** (optimizer): rewrites the plan considering deployment constraints -- **Layer 5** (physical planner): "for this deployment, DDSketch at the edge is cheapest" or "KLL at the backend sketchDB is better for this workload" - -## 2. Sketch Logical Plan: `QueryExpr` (Layer 3) - -`QueryExpr` (`algebra/expr.rs`) is the sketch algebra IR — a **logical plan** that -normalises all query languages into a common algebraic form. - -| | AST (syntax tree) | Logical Plan (QueryExpr) | -|---|---|---| -| **Structure** | Mirrors the grammar | Mirrors relational algebra operators | -| **Semantics** | Preserves syntactic details | Preserves only operator semantics | -| **Sketch types** | N/A | Implementation-independent intents (`AggIntent`) | -| **Language** | Language-specific | Language-independent (shared by SQL, PromQL, etc.) | - -QueryExpr has **25 operator variants** organized into categories: - -**Relational core** — standard relational algebra: -- `Source` — base metric / table (leaf node) -- `Filter { pred, input }` — selection (σ) -- `Project { cols, input }` — projection (π) -- `Aggregate { keys, aggs, having, input }` — grouping + aggregation (γ) -- `Join { kind, pred, left, right }` — relational join (⋈) -- `SetOp { kind, all, left, right }` — UNION / INTERSECT / EXCEPT -- `Sort { keys, input }` — ORDER BY -- `Limit { n, offset, input }` — LIMIT / OFFSET - -**Sketch-specific** — operators that express sketch computation intent: -- `SketchAgg { op: AggIntent, col, input }` — sketch aggregation intent (what, not how) -- `WindowedAgg { agg: AggIntent, window: WindowSpec, col, input }` — bundled window + sketch agg (window defines sketch lifecycle) -- `Partition { keys, input }` — GROUP BY distribution for distributed sketches -- `Dedup { col, input }` — deduplication (absorbed by cardinality sketches) -- `TopK { k, by, input }` — top-K heavy-hitter query -- `Merge { inputs }` — sketch merge (linearity: sketch(A∪B) = merge(sketch(A), sketch(B))) -- `JoinSketch { join_key, outer, inner }` — sketch-aware join push-down - -**Time / streaming** — window operators: -- `Window { duration, slide, input }` — standalone time window (batching) - -**PromQL-specific** — operators that preserve PromQL semantics: -- `HistogramQuantile { phi, input }` — `histogram_quantile(φ, …)` -- `PromQLSubquery { range, resolution, input }` — `expr[range:resolution]` -- `BinaryOp { op, lhs, rhs, vector_match }` — vector binary arithmetic with matching - -**Structural** — subqueries and bindings: -- `Subquery`, `LetBinding`, `Ref`, `WindowFunc` - -### Window operators: `Window` vs `WindowedAgg` - -| Operator | Use | Why separate | -|---|---|---| -| `Window { duration, slide }` | Standalone time batching (no sketch) | Used when the sketch op is a separate `SketchAgg` child node | -| `WindowedAgg { agg, window, col }` | Bundled window + sketch aggregation | In sketch systems the window defines the sketch lifecycle (when to flush/reset). Bundling lets the physical planner choose the best implementation (edge tumbling flush vs backend sketchDB EH vs original DB time_bucket). | - -`WindowSpec` supports five window kinds: - -| `WindowKind` | Semantics | Example | -|---|---|---| -| `Tumbling { size }` | Fixed-size, non-overlapping | PromQL implicit, SQL `TUMBLE(ts, '5m')`, Elastic `fixed_interval` | -| `Sliding { size, slide }` | Fixed-size, overlapping | PromQL `[5m]` range vector, SQL `HOP(ts, '1m', '5m')` | -| `Unbounded` | All samples, no time dimension | SQL `GROUP BY key` without time | -| `Landmark` | From epoch to now (cumulative) | Running aggregates | -| `Session { gap }` | Gap-based, closes after inactivity | Elastic session windows | - -## 3. Layers 1–3: Query Language → Language Plan → Sketch Algebra - -### 3.1 Layer 1→2: Language AST → Language Logical Plan - -Each parser takes a language-specific AST (from an external crate) and produces -a **language logical plan** using relational operators (`Aggregate`, `Window`, -`Filter`, `Sort`, `Limit`, etc.) with generic `AggFunc` variants — no sketch -names at this layer. - -**PromQL** (`query_parser/promql.rs`): - -``` -PromQL AST (promql-parser crate) - ↓ walk_qe(ast_node, ctx) -Language Logical Plan (Aggregate + AggFunc + Window) -``` - -The walker carries context downward: `partition` (GROUP BY keys), `topk` (K value), -`outer_count` (whether wrapped in `count(…)`). - -| PromQL construct | Layer 2 output | -|---|---| -| `quantile_over_time(φ, m[5m])` | `Aggregate { Quantile(φ), input: Window { 5m, Filter(Source) } }` | -| `histogram_quantile(φ, rate(…))` | `HistogramQuantile { φ, Aggregate { Quantile(φ), Window(...) } }` | -| `count_over_time(m[5m])` | `Aggregate { Count, input: Window { 5m, Source } }` | -| `avg_over_time(m[5m])` | `Aggregate { Avg, input: Window { 5m, Source } }` | -| `topk(k, …) by (dims)` | `TopK { k, Partition { dims, inner } }` | -| `a + b` | `BinaryOp { Add, lhs, rhs, VectorMatch }` | -| `m[5m:1m]` | `PromQLSubquery { range: 5m, step: 1m, inner }` | - -**SQL** (`query_parser/sql.rs`): - -``` -SQL AST (sqlparser crate) - ↓ extract_select_qe(select, order_by, limit, offset) -Language Logical Plan (Aggregate + AggFunc + Sort + Limit) -``` - -The SQL parser builds the plan bottom-up from SELECT clauses: - -| SQL construct | Layer 2 output | -|---|---| -| `FROM table` | `Source(table)` | -| `WHERE pred` | `Filter(ScalarExpr, Source)` | -| `JOIN … ON` | `Join(kind, pred, left, right)` | -| `GROUP BY keys` + agg functions | `Aggregate { keys, aggs: [AggItem { func }] }` | -| `TUMBLE(ts, INTERVAL '5m')` | `Aggregate { input: Window { 5m, Source } }` | -| `ORDER BY … DESC` | `Sort(keys, input)` | -| `LIMIT n` | `Limit(n, input)` | -| `UNION ALL` | `SetOp(Union, all, left, right)` | - -### 3.2 Layer 2→3: Language Logical Plan → Sketch Algebra (lowering) - -The shared `lower_to_sketch_algebra()` pass (`algebra/lower.rs`) converts -language-independent `Aggregate { AggFunc }` nodes into sketch algebra -`SketchAgg { AggIntent }` nodes. This is the same pass for both PromQL and SQL. - -**Algorithm**: - -``` -lower_to_sketch_algebra(expr): - Recursively walk the QueryExpr tree. - For each single-agg Aggregate node: - - 1. Map AggFunc → AggIntent (implementation-independent): - Quantile(φ) → AggIntent::Quantile { [φ], accuracy } - CountDistinct → AggIntent::Cardinality { accuracy } - Count (w/ GROUP BY) → AggIntent::Frequency { accuracy } - Avg → AggIntent::Quantile { [0.5], accuracy } (median proxy) - Min → AggIntent::Extrema { min: true } - Max → AggIntent::Extrema { max: true } - StdDev → AggIntent::Quantile { [0.25, 0.75], accuracy } (IQR proxy) - Sum/Rate/Delta → AggIntent::Exact(Sum) - Count (no GROUP BY) → stays as Aggregate (no sketch benefit) - - 2. If the Aggregate's input is a Window, fuse into WindowedAgg: - Aggregate { AggFunc, input: Window { duration } } - → WindowedAgg { AggIntent, WindowSpec { Tumbling(duration) }, input } - - 3. If the Aggregate had GROUP BY keys, wrap with Partition: - → Partition { keys, input: SketchAgg/WindowedAgg } - - Multi-agg Aggregates and HAVING clauses pass through unchanged. -``` - -| Layer 2 input | Layer 3 output | -|---|---| -| `Aggregate { Quantile(0.99), Window { 5m, Source } }` | `WindowedAgg { Quantile([0.99]), Tumbling(5m), Source }` | -| `Aggregate { CountDistinct, Source }` | `SketchAgg { Cardinality, Source }` | -| `Aggregate { Count, keys: [region], Source }` | `Partition { [region], SketchAgg { Frequency, Source } }` | -| `Aggregate { Avg, keys: [symbol], Source }` | `Partition { [symbol], SketchAgg { Quantile([0.5]), Source } }` | -| `Aggregate { Sum, Source }` | `SketchAgg { Exact(Sum), Source }` | -| `Aggregate { Count (no GROUP BY), Source }` | unchanged (no sketch benefit) | - -**SQL function → AggFunc mapping**: - -| SQL function | AggFunc | Sketch candidate | -|---|---|---| -| `COUNT(*)` with GROUP BY | `Count` | CountSketch / CountMinSketch | -| `COUNT(*)` without GROUP BY | `Count` | Exact (no sketch benefit) | -| `COUNT(DISTINCT col)` | `CountDistinct` | HLL | -| `SUM(col)` | `Sum` | Exact (not sketchable) | -| `AVG(col)` | `Avg` | DDSketch (p50 proxy) or Exact(Avg) | -| `MIN(col)` | `Min` | DDSketch (φ=0.0) or ExactMinMax | -| `MAX(col)` | `Max` | DDSketch (φ=1.0) or ExactMinMax | - -Note: the parser emits `Aggregate { func: Avg }` — it does **not** emit sketch ops. -Sketch assignment happens later in the optimizer (R9 HydraConversion) and allocator. -The SQL parser only produces relational operators; the PromQL parser is more aggressive -and emits `SketchAgg` nodes directly because PromQL functions like `quantile_over_time` -have a 1-to-1 mapping to sketch types. - -## 4. Concrete Example: PromQL (all 5 layers) - -### Query -```promql -quantile_over_time(0.99, http_request_duration{env="prod"}[5m]) -``` - -### Layer 1 — Language AST - -The `promql-parser` crate parses the string into a PromQL AST: -`Call("quantile_over_time", [NumberLiteral(0.99), MatrixSelector("http_request_duration", {env="prod"}, 5m)])` - -### Layer 2 — Language Logical Plan (parser output) - -The PromQL parser emits **relational operators only** — `Aggregate { AggFunc }` + `Window`, -no sketch names: - -``` -Aggregate { - keys: [], - aggs: [AggItem { func: Quantile(0.99), col: SampleValue }], - input: Window { - duration: 5m, - input: Filter { - pred: Column("env") = Literal("prod"), - input: Source("http_request_duration") - } - } -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -The shared `lower_to_sketch_algebra()` pass converts `Aggregate { Quantile }` to -`AggIntent::Quantile` and fuses with `Window` into `WindowedAgg`: - -``` -WindowedAgg { - agg: Quantile { quantiles: [0.99], accuracy: 0.01 }, - window: WindowSpec { kind: Tumbling { size: 5m } }, - col: SampleValue, - input: Filter { - pred: Column("env") = Literal("prod"), - input: Source("http_request_duration") - } -} -``` - -Note: no sketch implementation names — just "I need a quantile at φ=0.99 with ≤1% error." - -### Layer 4 — Optimizer - -R1 (PredicatePushDown): filter is already below the window — no change. Tree is returned as-is. - -### Layer 5 — Physical Plan - -`physical::plan(expr, config)` produces a `PhysicalNode` tree. For this simple -query, all nodes are at the Agent — no Exchange boundaries: - -``` -OtelSketchBuild { DDSketch, OtelTumblingFlush(5m) } [AgentCollector] - └── Filter { env="prod" } [AgentCollector] - └── OtlpScan [AgentCollector] -``` - -Resolution: `Quantile([0.99], 0.01)` → `DDSketch { relative_accuracy: 0.01, quantiles: [0.99] }`, -`Tumbling(5m)` at AgentCollector → `OtelTumblingFlush { 5m }`. - -### Execution - -1. **Agent** receives raw samples → filters `env="prod"` → batches 5m windows → DDSketch → emit -2. **Backend** merges DDSketches from N agents -3. **Query time**: extract 0.99 quantile from merged DDSketch - ---- - -## 5. Concrete Example: PromQL with Top-K (all 5 layers) - -### Query -```promql -topk by (service) (10, count_over_time(requests{env="prod"}[1m])) -``` - -### Layer 1 — Language AST - -The `promql-parser` crate parses this as: -`Aggregate(op="topk", param=10, modifier=By(["service"]), expr=Call("count_over_time", MatrixSelector("requests", {env="prod"}, 1m)))` - -### Layer 2 — Language Logical Plan (parser output) - -The PromQL parser emits relational operators. The `by (service)` partition keys -are propagated into the inner `Aggregate`'s GROUP BY keys, so the lowering pass -can see `Count WITH GROUP BY` → `Frequency`: - -``` -TopK { - k: 10, - by: ["service"], - input: Aggregate { - keys: ["service"], - aggs: [AggItem { func: Count, col: SampleValue }], - input: Window { - duration: 1m, - input: Filter { - pred: Column("env") = Literal("prod"), - input: Source("requests") - } - } - } -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -`lower_to_sketch_algebra()` converts `Aggregate { Count, keys: ["service"] }` → -`Partition { ["service"], WindowedAgg { Frequency } }`. The `Window + Aggregate` -fuses into `WindowedAgg`: - -``` -TopK { - k: 10, - by: ["service"], - input: Partition { - keys: By(["service"]), - input: WindowedAgg { - agg: Frequency { accuracy: 0.001 }, - window: WindowSpec { kind: Tumbling { size: 1m } }, - col: SampleValue, - input: Filter { - pred: env = "prod", - input: Source("requests") - } - } - } -} -``` - -Note: `Frequency`, not `CountSketch` — implementation-independent. Both CountSketch -and CountMinSketch are valid candidates; the physical planner decides. - -### Layer 4 — Optimizer - -R1 (PredicatePushDown): filter already below window — no change. - -### Layer 5 — Physical Plan - -`physical::plan(expr, config)` produces a multi-stage `PhysicalNode` tree with -Exchange nodes at stage boundaries: - -``` -TopK { k: 10 } [QueryEngine] - └── Exchange { SketchBinary } [QueryEngine] - └── HashAggregate { keys: ["service"] } [BackendCollector] - └── Exchange { Otlp } [BackendCollector] - └── OtelSketchBuild { CountSketch, [AgentCollector] - OtelTumblingFlush(1m) } - └── Filter { env="prod" } [AgentCollector] - └── OtlpScan [AgentCollector] -``` - -Three stages, two Exchange boundaries: -- **Agent → Backend** (Otlp): sketch data flows from agent collectors to merge tier -- **Backend → QueryEngine** (SketchBinary): merged sketches flow to query engine for top-K - -### Execution - -1. **Agent** → filters → builds CountSketch per 1m window → emits via OTLP -2. **Backend** → merges CountSketches per service -3. **QueryEngine** → extracts top-10 services by frequency - ---- - -## 6. Concrete Example: SQL (all 5 layers) - -### Query -```sql -SELECT symbol, AVG(price) FROM trades GROUP BY symbol -``` - -### Layer 1 — Language AST - -`sqlparser` produces: `Select { projection: [Identifier("symbol"), Function(AVG, "price")], from: [Table("trades")], group_by: [Identifier("symbol")] }` - -### Layer 2 — Language Logical Plan - -Both parsers emit the same kind of output — relational `Aggregate { AggFunc }`: - -``` -Aggregate { - keys: ["symbol"], - aggs: [AggItem { func: Avg, col: Named("price"), alias: "avg" }], - having: None, - input: Source("trades") -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -`lower_to_sketch_algebra()` converts `Avg` → `Quantile { [0.5], 0.01 }` (median proxy): - -``` -Partition { - keys: By(["symbol"]), - input: SketchAgg { - op: Quantile { quantiles: [0.5], accuracy: 0.01 }, - col: Named("price"), - input: Source("trades") - } -} -``` - -However, `Avg` is **non-mergeable** (`avg(A∪B) ≠ merge(avg(A), avg(B))`). -The stage-split will route this to DB for exact computation. - -### Layer 4 — Optimizer - -No rewrites applicable. - -### Layer 5 — Physical Plan - -`physical::plan()` assigns the non-mergeable Aggregate to the Database: - -``` -DbQuery { GROUP BY ["symbol"] } [Database] - └── Exchange { RawSamples } [Database] - └── OtlpScan [AgentCollector] -``` - -The Agent passes raw samples through to the Database, which computes exact AVG. - ---- - -## 7. Concrete Example: SQL with TUMBLE window (all 5 layers) - -### Query -```sql -SELECT region, COUNT(DISTINCT user_id) AS cnt -FROM sessions -GROUP BY region, TUMBLE(ts, INTERVAL '5' MINUTE) -ORDER BY cnt DESC LIMIT 10 -``` - -### Layer 1–2 — Parse to relational operators - -The SQL parser detects `TUMBLE(ts, INTERVAL '5' MINUTE)` in GROUP BY and emits -a `Window` node. `COUNT(DISTINCT user_id)` becomes `AggFunc::CountDistinct`: - -``` -Limit { - n: 10, - input: Sort { - keys: [{ col: "cnt", desc: true }], - input: Aggregate { - keys: ["region"], - aggs: [AggItem { func: CountDistinct, col: Named("user_id"), alias: "cnt" }], - input: Window { - duration: 5m, - input: Source("sessions") - } - } - } -} -``` - -### Layer 3 — Sketch Logical Plan (after lowering) - -`lower_to_sketch_algebra()` converts `CountDistinct` → `Cardinality { 0.01 }` and -fuses `Window + Aggregate` → `WindowedAgg`: - -``` -Limit { - n: 10, - input: Sort { - input: Partition { - keys: By(["region"]), - input: WindowedAgg { - agg: Cardinality { accuracy: 0.01 }, - window: WindowSpec { kind: Tumbling { size: 5m } }, - col: Named("user_id"), - input: Source("sessions") - } - } - } -} -``` - -### Layer 4 — Optimizer - -**R5 (TopKFusion)**: `Limit(10, Sort(desc, ...))` → fused into `TopK { k: 10 }` - -### Layer 5 — Physical Plan - -`physical::plan()` produces a multi-stage tree: - -``` -TopK { k: 10 } [QueryEngine] - └── Exchange { SketchBinary } [QueryEngine] - └── HashAggregate { keys: ["region"] } [BackendCollector] - └── Exchange { Otlp } [BackendCollector] - └── OtelSketchBuild { HLL, [AgentCollector] - OtelTumblingFlush(5m) } - └── OtlpScan [AgentCollector] -``` - -Resolution: `Cardinality(0.01)` → `HLL { precision: 14 }`, `Tumbling(5m)` → `OtelTumblingFlush`. - -### Execution - -1. **Agent**: builds one HLL per region per 5m window → emits via OTLP -2. **Backend**: merges HLLs from N agents (HLL merge = set union) -3. **QueryEngine**: extracts cardinality per region → top 10 - ---- - -## 8. Optimizer: Formulation of the Sketch Placement Problem - -### Optimization Goal - -Given a set of query workloads Q = {q₁, q₂, …, qₙ} and a deployment with -pipeline stages S = {Agent, BackendCollector, BackendDB, OriginalDB, ObjectStore}, -the optimizer solves: - -``` -minimize TotalCost(P) -subject to Accuracy(qᵢ, P) ≤ accuracy_sla(qᵢ) ∀ qᵢ ∈ Q - Latency(qᵢ, P) ≤ latency_sla(qᵢ) ∀ qᵢ ∈ Q - Throughput(qᵢ, P) ≥ throughput_sla(qᵢ) ∀ qᵢ ∈ Q - ResourceUsage(s, P) ≤ Budget(s) ∀ s ∈ S -``` - -where P is the physical plan (sketch type assignment + stage placement + window -configuration for each query operator). - -### Cost Model - -The total cost decomposes into per-stage costs: - -``` -TotalCost(P) = Σ_s [ BandwidthCost(s) + MemoryCost(s) + CPUCost(s) + StorageCost(s) ] -``` - -Each term is the aggregate resource consumption across all queries assigned to -that stage: - -| Cost component | Formula | -|---|---| -| `BandwidthCost(s)` | Σ_q transmission_bytes(sketch(q)) × flush_rate(q) | -| `MemoryCost(s)` | Σ_q memory_per_series(sketch(q)) × series_count(q) | -| `CPUCost(s)` | Σ_q cpu_per_insert(sketch(q)) × samples_per_sec(q) | -| `StorageCost(s)` | Σ_q transmission_bytes(sketch(q)) × retention(q) | - -### Constraints - -**Per-stage resource budgets** — each stage has memory, CPU, disk, and bandwidth limits: - -``` -∀ s ∈ S: - Σ_q memory_per_series(sketch(q, s)) × series_count(q) ≤ s.memory_bytes - Σ_q cpu_per_insert(sketch(q, s)) × samples_per_sec(q) ≤ s.cpu_budget - Σ_q transmission_bytes(sketch(q, s)) × flush_rate(q) ≤ s.bandwidth_budget -``` - -**Accuracy constraint** — sketch error must be within the query's SLA: - -``` -∀ qᵢ: - error(sketch_type(qᵢ), sketch_params(qᵢ)) ≤ accuracy_sla(qᵢ) -``` - -For example: DDSketch with `relative_accuracy = 0.01` guarantees ≤1% relative error -on quantile queries. HLL with `precision = 14` guarantees ≤0.8% relative error -on cardinality. - -**Functional constraint** — the sketch must support the query's aggregation intent: - -``` -∀ qᵢ: - intent(qᵢ) ∈ sketch_capability(sketch_type(qᵢ)).supported_intents -``` - -For example: a `Cardinality` intent can only be served by a sketch with -`SupportedIntent::Cardinality` (HLL, UnivMon), not by DDSketch. - -### Decision Variables - -For each query operator `op` in the plan: - -1. **Sketch type selection**: `sketch_type(op) ∈ candidates(intent(op))` - - Quantile → {DDSketch, KLL} - - Cardinality → {HLL} - - Frequency → {CountSketch, CountMinSketch} - -2. **Stage placement**: `stage(op) ∈ S` - - Subject to `stage_budget(stage(op)).fits(sketch_capability(sketch_type(op)))` - - Deferral chain: Agent → BackendCollector → BackendDB - -3. **Window configuration**: `window(op) ∈ {Tumbling(d), Sliding(d, s), Unbounded}` - - Subject to sketch capability: `sketch_capability(type).supports_sliding_window` - -4. **Delta encoding**: `delta(op) ∈ {true, false}` - - Subject to: `sketch_capability(type).supports_delta` - - Reduces bandwidth at the cost of reconstruction at the receiver - -### Cross-Query Optimization: What to Precompute - -When multiple queries share overlapping time series or aggregation patterns, the -optimizer can amortise costs: - -**Shared sketch reuse**: if q₁ = `quantile_over_time(0.99, m[5m])` and -q₂ = `quantile_over_time(0.5, m[5m])`, a single DDSketch serves both -(DDSketch can answer any quantile from one structure). - -**Precomputation decision**: a query should be precomputed (sketch maintained -continuously) rather than computed on-demand when: - -``` -precompute(q) = true iff repeat_interval(q) < query_latency_sla(q) -``` - -i.e., the query fires more often than the system can recompute it from raw data. -Precomputed sketches are maintained at the Agent and merged at the Backend, -with the Precompute Engine answering queries against the merged state. - -**Multi-query sketch sharing matrix**: for N queries over the same metric, the -optimizer builds a sharing matrix: - -| | DDSketch | HLL | CountSketch | -|---|---|---|---| -| q₁: quantile(0.99) | ✓ serves | ✗ | ✗ | -| q₂: quantile(0.5) | ✓ **shared with q₁** | ✗ | ✗ | -| q₃: count_distinct | ✗ | ✓ serves | ✗ | -| q₄: topk(10) | ✗ | ✗ | ✓ serves | - -One DDSketch instance serves both q₁ and q₂ → memory cost counted once, not twice. - -### Current Implementation - -The optimizer currently solves a simplified version: - -1. **Per-query greedy**: each query is optimised independently (no cross-query sharing yet) -2. **Sketch selection**: `CostModelPlanner` scores all candidates per query, picks cheapest meeting accuracy SLA -3. **Stage placement**: `physical::decide_sketch_placement()` checks `StageBudget::fits(SketchCapability)` per stage in order: Agent → Backend → QueryEngine -4. **Precomputation**: `should_precompute(q)` checks `repeat_interval < latency_sla` - -Future work: -- Global optimisation across queries (shared sketch instances) -- Joint sketch+stage+window optimisation (currently done greedily per dimension) -- Workload-adaptive re-optimisation (replan when query patterns change) - ---- - -## 9. Sketch Directory: Which Sketch for Which Operation? - -The sketch directory (`algebra/directory.rs`) maps aggregation types to candidate -sketch families. The `CostModelPlanner` scores all candidates and picks the -cheapest that meets the accuracy SLA. - -### Candidates per Aggregation Type - -| Aggregation | Candidates (default first) | When non-default is chosen | -|---|---|---| -| Quantile | **DDSketch**, KLL | KLL when memory-constrained | -| Cardinality | **HLL** | Single candidate | -| Frequency | **CountSketch**, CountMinSketch | Based on cost model scoring | - -### Sketch Type → OTel Collector Processor - -| SketchType | Go processor | Key parameters | -|---|---|---| -| DDSketch | `ddsketch` | relative_accuracy, quantiles | -| KLL | `KLL` | k, quantiles | -| HLL | `HLL` | (fixed precision in Go code) | -| CountSketch | `countsketch` | epsilon, delta | -| CountMinSketch | `countmin` | rows, cols, metric_name | - -### Stage Assignment Rules - -| QueryExpr node | Default stage | Deferral trigger | -|---|---|---| -| Source, Filter, Window, SketchAgg | **Agent** | Memory budget exceeded → Backend | -| Partition, Merge, Dedup, Exact(Sum/Count/Min/Max) | **Backend** | Memory exceeded → Precompute | -| TopK, HistogramQuantile, BinaryOp, PromQLSubquery | **Precompute** | — | -| Exact(Avg) | **DB** | Non-mergeable — cannot distribute | - -When an Agent sketch exceeds the memory budget, it is deferred to Backend. -If it also exceeds the Backend budget, it moves to Precompute. Every deferral -is logged in `StagedPlan.deferral_log` for observability. - -## 10. Edge Aggregation Scope (`mode`) and Sparse HLL (`hll_sparse`) - -Once a sketch family is chosen for a metric, two further per-metric knobs the -control plane emits into the edge `asap_edge.metrics[]` config control *how* the -edge aggregates: the aggregation **scope** (`mode`: `per_series` vs -`whole_stream`) and the in-memory **sparse-HLL** opt-in (`hll_sparse`). See -[Edge aggregation scope + sparse HLL](query-to-edge-scope-and-sparse.md) for the -signal-to-config mapping, worked examples, and back-compat reasoning. diff --git a/docs/adding-a-new-sketch.md b/docs/adding-a-new-sketch.md index 31f751a5..06814c0f 100644 --- a/docs/adding-a-new-sketch.md +++ b/docs/adding-a-new-sketch.md @@ -422,7 +422,7 @@ next person trying to add another one: | `ASAPQuery-backend docs/design-sketch-db.md §19.9` | The theoretical accuracy bound formula for `Foo` | | `ASAPQuery-backend docs/design-sketch-db.md §19.10` | The merge propagation rule for `Foo` | | Shared design doc for `Foo` | A short rationale: what query class, why this sketch over alternatives, what the trade-off is | -| `DataCollector controller/docs/query-to-sketch-translation.md` | Where in the five-layer plan `Foo` shows up as a candidate | +| [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) | How `Foo` is represented and selected as a logical summary candidate | ---