Figure 1. Integrated ASAPPlanner–ASAPQuery-backend architecture and workflow.
+This design splits one selected ASAPPlanner semantic DAG into two executable +backend plans: + +- **PrecomputePlan** produces and maintains stored summary state. +- **QueryPlan** reads stored state and computes query results. + +Both plans use identities and state contracts from the +[SDS design](summary-catalog-sds-architecture.md) and install as one plan version. +The [migration plan](asapplanner-migration-plan.md) defines delivery steps. +CollectorPlan, TransmissionPlan and distributed activation are deferred; this +migration must not introduce a backend dependency on ASAPCollector. + +## Document map + +1. [Architecture at a glance](#architecture-at-a-glance) +2. [Design definitions and selection](#design-definitions-and-selection) + - [Worked example](#worked-example) +3. [Core concepts and ownership](#core-concepts-and-ownership) +4. [Compiler contract](#compiler-contract) +5. [Compilation rules](#compilation-rules) +6. [Runtime contract](#runtime-contract) +7. [Validation and acceptance](#validation-and-acceptance) +8. [Decisions and deferred work](#decisions-and-deferred-work) + +## Architecture at a glance + +The current `PrecomputePlan.executable_dags` can contain a complete semantic DAG, +including query-time nodes such as `SummaryEstimate`. Bindings may prevent those +nodes from running during maintenance, but the artifact and its visualization do +not express that ownership clearly. + +This is a representation defect tracked by +[issue #740](https://github.com/ProjectASAP/ASAPQuery-backend/issues/740). +The target design requires separate executable projections. + +The compiler instead binds stored summaries once and cuts the DAG at each +materialization boundary: ```mermaid -flowchart TD - subgraph PlannerBoundary["ASAPPlanner boundary — reusable optimization"] - Canonical[Canonical QueryExpr and workload semantics] - Canonical --> Strategies[CSE and reusable replacement strategies] - Strategies --> Candidates[Candidate post-ASAP workload DAGs] - Candidates --> Ranking[Semantic legality, accuracy and evidence-based ranking] - end - - subgraph BackendBoundary["ASAPQuery-backend boundary — observability application"] - Inputs[PromQL registrations, QueryWorkload and DataWorkload] - Evidence[Runtime capabilities and complete deployment cost evidence] - Commit[Control plane commits a feasible post-ASAP workload DAG] - Compile[Physical binding and deployment selection] - Bundle[One versioned physical plan bundle] - Activate[Validate, stage and activate] - Precompute[Ingest, precompute and summary store] - Serve[Bound query execution and explicit exact fallback] - Feedback[Readiness, accuracy and resource observations] - Inputs --> Commit - Commit --> Compile --> Bundle --> Activate - Activate --> Precompute - Activate --> Serve - Precompute --> Serve - Precompute --> Feedback - Serve --> Feedback - Feedback --> Evidence - end - - Inputs -->|Planning request| Canonical - Candidates -->|Implementation evaluation request| Evidence - Evidence -->|Feasibility and cost evidence| Ranking - Ranking -->|Legal ranked post-ASAP alternatives| Commit - Bundle -->|CollectorPlan in distributed profile| Collector[ASAPCollector — external runtime] - Collector -->|Planned data or summary frames| Precompute - Clients[PromQL clients] --> Serve - Serve -->|Configured exact route| Exact[Prometheus or archive query service] +flowchart LR + D[Selected Planner DAG] --> C[Physical compiler] + C --> P[PrecomputePlan] + C --> S[Summary Catalog / SDS] + C --> Q[QueryPlan] + P -->|write state| Store[Summary store] + Q -->|bound state read| Store + P -->|definition ID| S + Q -->|definition ID| S ``` -**ASAPPlanner's selected post-ASAP workload DAG is the authoritative semantic -plan. ASAPQuery-backend binds and executes that decision through its control -plane and data plane.** Backend physical plans remain necessary, but must be -traceable projections of that DAG, not independently optimized replacements -for its dependencies, shared state, or query-result semantics. +Semantic provenance remains available, but query-only operators are not +PrecomputePlan executable content. -Planner provides reusable legal alternatives and ranking. The backend owns -deployment commitment, concrete realization, and operational policy. A -deployment choice cannot silently change Planner-owned grouping, statistic, -summary parameters, logical window, accuracy, or lifecycle: it must return to -the legal candidate-selection boundary. +## Design definitions and selection -## Architecture boundaries and reuse +Audience: developers implementing the Planner/backend boundary. The definitions +below describe the target design; the YAML that follows illustrates that design +and is not a serialized Rust API. Implementations should adapt existing types +where they express these requirements rather than introduce duplicate models. -ASAPQuery-backend is the observability downstream application, including the -MetricsObservabilityQuery use case. DQC (the proposed name for the current -asap-fusion repository) is a separate downstream application, not an execution -dependency of this backend. +### Existing representation and target boundary -| Responsibility | ASAPPlanner | ASAPQuery-backend | -| --- | --- | --- | -| Query semantics | Canonical expressions, equivalence, grouping and time semantics | PromQL API, workload registration and profile restrictions | -| Optimization | CSE, legal sharing, rollup, decomposition, summary and accuracy alternatives | Feasibility evidence, deployment commitment and concrete assignments | -| Time and state | Logical windows, abstract window framework and maintenance lifecycle | Panes, retention layout, update implementation and placement | -| Plan identity | Logical producer identities and result dependencies | Plan versions, physical materializations, SID bindings and runtime handles | -| Execution | Deployment-independent semantic contract | Ingest, precompute, store, serving, readiness and fallback | -| Operations | Reusable models consuming scoped evidence | Activation, rollback, telemetry, freshness and resource enforcement | - -Reuse works in both directions. The backend consumes Planner strategies; -general-purpose rules discovered while optimizing repeated observability -queries belong in Planner so DQC and other applications can reuse them. -Prometheus staleness handling, SID resolution, Collector placement, and OpAMP -publication remain downstream responsibilities. - -## Inspection: what already exists - -Inspected backend main at -[`95131d83972bb7a07d338e2a5af925a20c15ddce`](https://github.com/ProjectASAP/ASAPQuery-backend/tree/95131d83972bb7a07d338e2a5af925a20c15ddce), -using its pinned Planner revision -[`cb50219c582d43f53ab77d3a595bd1ea4a9aa119`](https://github.com/ProjectASAP/ASAPPlanner/tree/cb50219c582d43f53ab77d3a595bd1ea4a9aa119). -The baseline is merged code, not the completion of open PRs. - -| Area | Existing foundation | Consolidation needed | -| --- | --- | --- | -| Frontend and selection | Planner dependency, canonical query parsing, backend selection from Planner alternatives | Make workload-wide sharing and strategy composition explicit across supported entry points | -| Physical compilation | One bundle with precompute, transmission, backend and query projections; Collector projections when applicable | Preserve all selected shared producers and provenance through every projection | -| Serving | Bound QueryPlan execution, exact materialization identities and explicit fallback | Audit remaining compatibility paths; serving must not make a new summary choice | -| Deployment | Versioned staging and activation, runtime capability and evidence checks | Verify profile-specific failure and readiness behavior end to end | -| Compatibility | Backend-local ASAPQuery profile alongside distributed collection | Keep distinct deployment profiles on the same semantic contract | - -Evidence: -[selection adapter](../../control_plane/src/planner_selection.rs), -[physical compiler](../../control_plane/src/physical/compiler.rs), -[legacy workload adapter](../../control_plane/src/physical/workload_planner.rs), -[shared QueryPlan](../../crates/asap_types/src/query_plan.rs), -[query lowering](../../control_plane/src/query_plan.rs), and -[bound serving executor](../../data_plane/src/query_engines/asap_query_engine/post_asap_readout.rs). -The selection adapter explicitly commits a ranked Planner candidate downstream. -Consequently, the figure does not imply that the Planner library deploys or -commits a complete backend configuration by itself. - -This is an extension of existing integration, not a proposal to replace it -wholesale. Implementation guides sometimes describe a broader target than an -individual runtime path supports; migration acceptance must be demonstrated -against executable paths, not inferred from interface names. - -## One authoritative semantic DAG, derived runtime plans - -The shared contract must preserve sources and filters, label/grouping identity, -exact operators surrounding summaries, summary build/merge/readout, shared -producers, query roots, logical time coverage, accuracy, and maintenance -requirements. Audit the pinned post-ASAP representation for genuine gaps; -extend Planner semantics where necessary. - -Do not put concrete engine or implementation IDs into Planner IR. The backend -retains a binding from logical producer identity to implementation, placement, -materialization, state schema, and active generation. This follows the -[Planner/downstream boundary](https://github.com/ProjectASAP/ASAPPlanner/blob/cb50219c582d43f53ab77d3a595bd1ea4a9aa119/docs/design_docs/asapplanner-downstream-boundary.md). - -One selected DAG can produce several execution projections: - -- PrecomputePlan: how the selected state is built and maintained. -- TransmissionPlan and optional CollectorPlan: how distributed producers - implement and deliver that state. -- SummaryCatalog: canonical summary/data descriptors and stable materialization identities. -- QueryPlan: executable reads, merges, readouts and remaining exact operations. - -These projections may expand one semantic node into several physical tasks. -They must not invent a different semantic sharing graph. QueryPlan need not be -a byte-for-byte serialization of post-ASAP IR, nor should ingestion and query -serving literally run an identical task schedule. They implement different -phases of the same selected computation. - -Sharing has explicit scope: maintain a shared producer once per compatible -source/window/plan generation; reuse its state across query roots. Memoizing a -query DAG within one request is useful but does not, by itself, prove -cross-query or cross-request sharing. - -## End-to-end workflow - -1. **Register demand.** Collect canonical queries, evaluation cadence, time - windows, accuracy scope, source arrival facts and optimization horizon. -2. **Generate alternatives.** Planner applies legal rewrites and sharing, - choosing among summary, abstract-window and lifecycle alternatives. -3. **Evaluate implementations.** The backend checks runtime feasibility and - supplies complete, fresh costs over the same workload horizon. -4. **Commit and bind.** The control plane selects a legal workload alternative, - retains its concrete realization, and compiles one coherent plan bundle. -5. **Publish.** Validate and stage matching projections. For distributed - deployment, require the corresponding Collector application evidence - before activation. A failed rollout preserves the prior active generation. -6. **Maintain and serve.** Ingest updates the selected state; a request uses one - active snapshot and exact bindings. Warm execution requires complete, - fresh coverage. Otherwise follow the configured exact route or return an - explicit failure if that route is unavailable. -7. **Observe and replan.** Attribute cost, readiness and accuracy evidence to - the plan generation and producer. Semantic changes require a new planning - decision and activation, not an ad-hoc serving-time substitution. - -The backend-local profile uses Remote Write, local precompute and Prometheus -fallback without requiring Collector/OpAMP. The distributed profile may use -Collector-maintained summaries and configured archive services. Neither -profile's optional infrastructure becomes a prerequisite for the other. - -## Example: repeated dashboard queries sharing one state producer - -Consider a gauge `request_size_bytes`, one scalar series per -`(service, instance)`, without extra labels. Register these instant-query -expressions repeatedly at the same evaluation cadence: - -```promql -# Q1: sum of observed sample values per service over the last five minutes -sum by (service) (sum_over_time(request_size_bytes[5m])) - -# Q2: sample-weighted mean per service over that same interval -sum by (service) (sum_over_time(request_size_bytes[5m])) -/ -sum by (service) (count_over_time(request_size_bytes[5m])) -``` +The selected post-ASAP DAG describes the selected computation: source operations, +summary producers, shared dependencies and query readouts. Maintenance decisions +are associated with its summary producers through plan-scoped node identities. + +Planner's `SummaryMaintenanceLifecyclePlan` contains a materialized DAG `root` +and a `deployments` collection, with one entry per unique reachable `SummaryAgg`. +Each deployment identifies its `post_asap_node_id` and carries an optional +`SummaryMaintenanceLifecycleGuarantee`, considered alternatives and a selected +window framework. The plan also carries workload demand and costing context. +Thus the lifecycle plan already refers to the computation DAG; it is not a +separate query representation, nor is one whole lifecycle plan required per +producer. A missing guarantee is not an executable maintenance commitment. -Q2 is deliberately not the unweighted mean of per-instance means. Its -denominator counts actual observations, which matters when instances have -different sample counts. These are gauge samples, not counter increases. +See the Planner +[lifecycle plan types](https://github.com/ProjectASAP/ASAPPlanner/blob/ba1c4436a3410dc03a133363ab5b75649e70f97a/crates/asap-aware-mapping/src/summary_maintenance_lifecycle.rs) +and [guarantee vocabulary](https://github.com/ProjectASAP/ASAPPlanner/blob/ba1c4436a3410dc03a133363ab5b75649e70f97a/crates/types/src/post_asap/summary_maintenance_lifecycle.rs). +These describe the referenced Planner revision, not a claim that every field +below is already supported by the backend's pinned dependency. -A legal target alternative is: +The backend currently records physical node ownership with +[`BackendExecutableBinding`](../../crates/asap_types/src/executable_plan.rs). +The target compiler consumes the selected computation and its maintenance +decisions together, validates them against backend support, and emits the two +physical plans plus catalog bindings. A shared producer is maintained once for +all compatible consumers. + +### Lifecycle commitment + +A **lifecycle commitment** is the selected maintenance promise for one logical +summary producer in a particular selected plan. This is a design term for the +selected guarantee and its concrete scheduling/retention binding, not a proposed +replacement for `SummaryMaintenanceLifecyclePlan`. + +| Field in the example | Definition and constraint | +| --- | --- | +| `producer` | Node identity in the selected DAG; must resolve to a stored summary producer. | +| `mode` | Selected construction/update method. `batch_rebuild_from_data_at_rest` reads persisted input and constructs replacement state for each required coverage interval. | +| `refresh.every` | Spacing of scheduled evaluation endpoints, not elapsed time after the preceding build finishes. | +| `refresh.anchor` | Origin of that schedule; `unix_epoch` with `every: 1m` yields UTC minute boundaries. | +| `retention.completed_state_for` | Minimum duration to retain each completed output snapshot after publication. It is independent of input coverage and raw-data retention. | +| `implementation` | Backend implementation selected to fulfill this commitment. | + +For each endpoint `T`, a rebuild reads exactly the logical input interval for +`T` and publishes state labeled with that coverage. Publication after `T` does +not change the interval. Retention expiry makes a snapshot eligible for cleanup +only after readers and dependent producers release it. A missed or unfinished +build leaves that endpoint unready; the configured fallback/unavailability +policy applies. Reusing an older snapshot requires an explicit query freshness +policy and must not silently change query time semantics. + +Planner supplies legal maintenance alternatives. The backend supplies executable +implementations and evidence; the control plane commits a feasible selection. +The compiler validates that commitment without silently changing its mode, +coverage or sharing. A changed commitment is installed through a new plan +plan version. It need not change the semantic summary definition when only the +physical maintenance policy changes. + +### Backend capability + +A **backend capability** is an implementation provider's declaration of a +supported combination of algorithm, parameters, maintenance mode, input kind, +window behavior and state schema. It answers whether a proposed realization can +execute faithfully. Independent global lists of algorithms and modes would +incorrectly imply support for every combination. + +Each capability record has an `implementation` identity, an `algorithm` +configuration, `maintenance_modes`, `input_kind`, `window_support`, and +`state_schema`. The compiler must match the whole record. The example declares +only KLL with `k: 200`, batch rebuilding from stored rows, and complete snapshots +for the requested logical range. It does not establish incremental maintenance +or arbitrary parameter support. A readout implementation alone does not prove +the corresponding producer is supported. + +### Physical cost evidence + +**Physical cost evidence** is a scoped estimate or measurement for one +implementation/configuration and maintenance mode. It is supplied by the backend +provider and used when comparing feasible alternatives over the same planning +horizon. It is separate from both capability and the final commitment. + +An evidence record identifies the implementation, algorithm parameters, mode, +input range, sample count, group count and execution profile. It declares whether +numbers are measured or modeled, their provenance and applicability period. +Measured evidence needs a benchmark identity/time; modeled evidence needs a model +version and assumptions. Missing or stale evidence is not zero cost. + +`state_bytes_per_group` measures one completed summary payload; +`rebuild_cpu_ms_total` measures CPU time for one rebuild across all declared +groups. CPU time is not wall-clock completion latency. Memory, temporary build +space, retained snapshots, I/O and query readout must also be costed before +claiming a complete deployment cost. A five-minute range alone does not determine +sample count or CPU cost. + +### Selection and validation ```text -Selected samples and logical five-minute coverage - | - Shared state per (service, instance) - SUM(value), COUNT(observations) - | - Merge/reduce by service - SUM(sum), SUM(count) - | - +------+------+ - | | - sum -> Q1 sum / count -> Q2 +Selected computation and lifecycle alternatives + + backend capabilities: supported combinations + + scoped cost evidence: resource costs of those combinations + -> control-plane commitment per selected producer + -> physical compiler validation + -> PrecomputePlan + QueryPlan + catalog bindings ``` -Planner recognizes the common sum computation and can propose aggregate-state -fusion with per-consumer readouts. The backend implements the selected window -framework with compatible runtime state and binds both query roots to the -same producer. It must preserve PromQL range boundaries, labels, absent-series -behavior and division semantics; a missing denominator is not invented as -zero. Physical panes may be used only when their coverage matches the selected -logical interval, including boundary handling. - -This diagram is a target acceptance example, not a claim that today's compiler -already fuses these complete PromQL expressions. If an operator or window -cannot be realized end to end, the current supported behavior is explicit -fallback rather than partial warm execution with changed semantics. - -For the first milestone, use exact sum/count state and compare against -Prometheus at identical timestamps. Verify both numerical/label equivalence -and one maintained producer shared by the two roots. Exact aggregate state -does not eliminate the separate requirement to verify data completeness. - -Approximate extensions must declare what epsilon measures and what delta -covers. For a whole 20-row result with failure probability at most 0.05, -20 valid per-row failure bounds of at most 0.0025 suffice by the union bound; -independence is not required. Per-row 95% intervals alone do not establish -95% confidence for the complete result. Multiple dashboard evaluations need -their own declared scope; a result-level guarantee is not automatically -session-wide. Shared state also does not make separate errors independent. - -## Capabilities, costs and feedback - -Capabilities answer **can this deployment faithfully execute this alternative?** -Costs answer **which feasible alternative is preferable?** - -| Capability question | Why it constrains selection | -| --- | --- | -| Can the producer build/update the selected family and parameters? | A readout implementation alone does not make a state maintainable | -| Can storage and readout preserve the selected windows and labels? | A tumbling-only path cannot silently implement arbitrary sliding coverage | -| Are merge operations and full/delta encodings compatible? | Distributed producers must construct the same logical state | -| Can the runtime perform every exact operator after readout? | A supported sketch is insufficient for an unsupported full expression | -| Can readiness, staleness and exact fallback be enforced? | Mathematical legality does not establish runtime answerability | - -Costs include initialization, ingestion updates, overlapping/retained state, -transmission, storage, merges, readouts, recurring queries, and shared producer -construction once. Compare alternatives over the same data and demand scope. -Missing evidence is not zero cost; stale or incomplete implementation evidence -cannot justify selection. - -Runtime observations reference the concrete binding and selected semantic -producer. Physical controls may vary only within already-authorized -guardrails. Changing grouping, family, parameters, windows or sharing returns -to planning. - -## Reuse across various ASAP workload scenarios - -| Scenario | Reusable Planner strategy | Application-specific responsibility | +Before installation, validate producer identity, supported algorithm/mode/schema, +schedule and coverage, retention sufficient for dependent reads, accuracy and +query requirements, and the scope/completeness of cost evidence. Reject an +inconsistent binding instead of inventing missing maintenance policy. Where the +planning interface supports exact fallback, select that explicitly. + +The existing binding/compiler path is the migration starting point. Adapters +must map existing Planner guarantees and backend capabilities into these +requirements, reporting unsupported fields. The plan split must preserve those +decisions in writer and reader bindings. New wire schemas and concrete scheduling +support are implementation work; this document defines their required behavior. + +## Worked example + +Query `p99-api-latency` asks for the 99th percentile of five minutes of latency, +grouped by `service` and evaluated every minute. The YAML below is conceptual; it +is not the current serialized API schema. Resource numbers are fictional, +illustrating units and scope only; they are not benchmark evidence or proof that +this candidate meets accuracy, cost or latency requirements. + +### Compiler input + +```yaml +selected_planner_dag: + query_id: p99-api-latency + query_language: clickhouse_sql + query_expression: >- + SELECT service, quantile(0.99)(request_latency_seconds) + FROM metrics + WHERE timestamp > :evaluation_time - INTERVAL 5 MINUTE + AND timestamp <= :evaluation_time + GROUP BY service + root: estimate-p99 + nodes: + - input: request_latency_seconds + - group_by: [service] + - id: build-kll + build_summary: {algorithm: kll, k: 200} + - estimate: {quantile: 0.99} + +query_requirements: + accuracy: supplied_by_selected_planner_guarantee + response_latency_ms: 200 + +lifecycle_commitment: + producer: build-kll + implementation: local-kll-batch-v1 + mode: batch_rebuild_from_data_at_rest + refresh: {every: 1m, anchor: unix_epoch} + retention: {completed_state_for: 10m} + +backend_capabilities: + - implementation: local-kll-batch-v1 + algorithm: {kind: kll, k: 200} + maintenance_modes: [batch_rebuild_from_data_at_rest] + input_kind: stored_rows + window_support: complete_snapshot_for_requested_range + state_schema: kll-v1 + +physical_cost_evidence: + - implementation: local-kll-batch-v1 + algorithm: {kind: kll, k: 200} + mode: batch_rebuild_from_data_at_rest + workload: {input_range: 5m, samples_per_group: 300, groups: 100} + execution_profile: illustrative-local-worker + provenance: {kind: illustrative, usable_for_selection: false} + costs: {state_bytes_per_group: 4096, rebuild_cpu_ms_total: 35} + +installation_context: + catalog_version: 12 + state_schema: kll-v1 + plan_version: 42 +``` + +### Compiler output + +```yaml +summary_catalog: + definitions: + - id: def-api-latency-kll + input: request_latency_seconds + group_by: [service] + range: 5m + algorithm: {kind: kll, k: 200} + +precompute_plan: + plan_version: 42 + nodes: + - {id: read-samples, op: ReadInput, metric: request_latency_seconds} + - {id: group-service, op: GroupBy, labels: [service]} + - {id: build-kll, op: BuildKll, k: 200} + - id: write-kll + op: WriteState + reference: {state_slot_id: latency-kll, definition_id: def-api-latency-kll} + schema: kll-v1 + encoding: kll-binary-v1 + partition_by: [service, window_end] + edges: + - [read-samples, group-service] + - [group-service, build-kll] + - [build-kll, write-kll] + +query_plan: + plan_version: 42 + query_id: p99-api-latency + query_language: clickhouse_sql + query_expression: >- + SELECT service, quantile(0.99)(request_latency_seconds) + FROM metrics + WHERE timestamp > :evaluation_time - INTERVAL 5 MINUTE + AND timestamp <= :evaluation_time + GROUP BY service + nodes: + - id: read-kll + op: ReadState + reference: {state_slot_id: latency-kll, definition_id: def-api-latency-kll} + expected_schema: kll-v1 + expected_encoding: kll-binary-v1 + partition: {service: all_requested_services, window_end: evaluation_time} + - {id: estimate-p99, op: SummaryEstimate, quantile: 0.99} + - {id: result, op: QueryResult} + edges: + - [read-kll, estimate-p99] + - [estimate-p99, result] + +provenance: + planner.build_summary: [precompute.build-kll, precompute.write-kll] + planner.estimate-p99: [query.read-kll, query.estimate-p99] +``` + +`latency-kll` is the state slot shared by the writer and reader in plan version +42. The catalog defines its summary semantics; the matching executable bindings +declare format and partition rules. There is no separate catalog materialization +object. Provenance relates +both physical projections to the selected DAG without making that DAG executable +inside PrecomputePlan. + +Here `range: 5m` denotes logical coverage `(T - 5m, T]`, not pane size, +refresh cadence, state retention or scrape interval. `ReadInput` is parameterized +by the scheduled endpoint and that range; `WriteState` publishes a completed +snapshot per service and endpoint. `ReadState` selects the snapshot matching the +requested endpoint and checks readiness. The ten-minute retention keeps older +completed snapshots available; it does not turn the summary into a ten-minute +aggregate. The illustrative KLL parameters alone do not establish a particular +accuracy guarantee, and CPU cost alone does not establish the 200 ms latency +requirement. + +## Core concepts and ownership + +“Maintenance” is the execution phase that constructs or updates state, including +batch construction, rebuilding, merging and derived summaries. “Precompute” names +the plan and engine responsible for that work; it does not imply incremental +maintenance. + +Bindings describe the semantic-to-physical mapping: + +| Binding | Meaning | Example | | --- | --- | --- | -| Repeated dashboards (MetricsObservabilityQuery) | Shared aggregates and prepared/maintained state | PromQL semantics, freshness and serving | -| Multiple dashboard resolutions | Legal rollup and window alternatives | Compatible retention and exact time coverage | -| Distributed telemetry aggregation | Mergeable summary and grouping alternatives | Collector placement, transmission and activation | -| DQC analytical workloads | CSE, aggregate fusion and rollup | DQC engine adapters and batch execution policy | +| `Materialization` | PrecomputePlan stores this node's output | `KLL` in `KLL(sum(data))` | +| `MaintenanceInput` | PrecomputePlan executes this input/intermediate without storing it independently | `sum(data)` feeding KLL | +| `Query` | The node maps to an explicit QueryPlan operation | `SummaryEstimate` | +| `QueryInput` | Query semantics are absorbed into another physical operation | A quantile parameter compiled into `SummaryEstimate` | -General semantic rules belong in Planner. Backend-local metric-name fixtures, -SID lookup or deployment-specific placement must not become universal Planner -rules. No dependency on DQC is needed to reuse strategies contributed by it. +`Materialization` above is the existing backend node-binding variant marking +stored output. It does not create a separate catalog object. The compiler assigns +that output a state slot and emits matching writer/reader bindings; see +[field ownership and migration](summary-catalog-sds-architecture.md#core-objects). -## High-level migration +| Layer | Owns | +| --- | --- | +| ASAPPlanner | Semantic candidates, legality, accuracy reasoning and selection among advertised capabilities | +| Physical compiler | Concrete implementation, subgraph split, catalog bindings and plan version | +| Precompute runtime | Installed maintenance nodes and state publication | +| Query runtime | Bound state reads, query operators, exact residuals and fallback | +| Catalog | Summary definitions | +| Plan read/write bindings | State references, format, partition rules and writer ownership | +| Runtime inventory/store | Actual state instances, coverage, readiness, location and payloads | -See the [migration delivery plan](asapplanner-migration-plan.md) for PR-sized -implementation slices, dependencies, regression fixtures and completion gates. +## Compiler contract -| Milestone | System outcome | Acceptance | -| --- | --- | --- | -| 1. Audit the shared contract and entry points | Current canonical compilation and compatibility paths have explicit ownership | Document supported operators, sharing scope, profile limits and true IR gaps | -| 2. Complete one workload-wide semantic path | Registered queries use Planner alternatives with preserved shared producers | The two-query example has one selected producer and both result roots | -| 3. Preserve bindings through all projections | Precompute, storage and serving implement the same selected decision | No duplicate maintenance; exact state/schema/window and generation agreement | -| 4. Consolidate reusable strategies | Missing general fusion/rollup rules extend Planner | Rules work without backend metric names, SID objects or placement assumptions | -| 5. Close capability and cost feedback | Only fully executable, properly costed alternatives are committed | Unsupported or stale evidence fails closed; estimated and observed costs are traceable | -| 6. Validate profiles and retire redundant selection paths | Serving executes installed bindings without independent semantic planning | Prometheus parity, sharing, readiness, fallback and activation-failure tests pass | -| 7. Broaden coverage (ProjectASAP-wide; not required for this repository) | Other applications, engines, sketches and lifecycles reuse the contract | Each participating provider demonstrates capability and semantic conformance | - -The first milestone demonstration should use backend-local ingestion and the -exact two-query example. Distributed rollout follows the same contract with -additional producer and activation checks. Existing paths may remain as -comparison baselines until parity is established; remove duplicate semantic -selection, not necessary physical plans or profile-specific runtime adapters. - -Step 7 is an ecosystem extension, not a prerequisite for completing this -backend's scoped consolidation through steps 1–6. - -## Related contracts and implementation guides - -- [Physical compiler](../developer_docs/control-plane/physical-compiler.md) -- [Plan publication](../developer_docs/control-plane/plan-publication.md) -- [Catalog-backed physical-plan runtime](../developer_docs/query-engine/catalog-physical-plan-runtime.md) -- [ASAPQuery compatibility profile](asapquery-compatibility-profile.md) -- [Runtime accuracy feedback](../developer_docs/control-plane/runtime-accuracy-feedback.md) +The compiler consumes: + +- selected Planner DAG roots and query associations; +- query accuracy and response requirements; +- complete lifecycle commitments for the supported backend mode; +- backend capabilities and concrete implementation evidence; +- catalog, schema and plan-version inputs. + +Capabilities constrain Planner choices. A data-at-rest-only backend advertises +only batch construction; recurring query demand does not imply incremental +support. + +| Output | Responsibility | +| --- | --- | +| Catalog entries | Summary definitions referenced by the plans | +| PrecomputePlan | Maintenance subgraphs ending in state writes | +| QueryPlan | Bound state reads, query operators and exact residuals | +| Provenance | Physical-to-semantic node mapping | + +The compiler derives all four outputs from the same bindings. They cannot choose +summary semantics, grouping, time ranges or schemas independently. + +## Compilation rules + +### Executable subgraphs and materialization boundaries + +For every selected stored summary, the compiler: + +1. Creates or reuses a compatible summary definition and assigns a state slot + within the plan version. No standalone catalog materialization is created. +2. Places source reads, maintenance operators, derived-state reads and the state + sink in PrecomputePlan. +3. Replaces the stored-summary edge in QueryPlan with an explicit state read + referencing the same slot and definition, with matching format and partition + rules. Writer identity belongs to the PrecomputePlan binding. +4. Places `SummaryEstimate`, merges, exact residuals and result composition in + QueryPlan. +5. Records provenance for semantic nodes absorbed into larger physical nodes. + +Two queries may share a producer only when their definition and state partition +are compatible. Sharing does not multiply maintenance updates; each query keeps +its own readout operators. + +A summary built from completed stored summaries uses explicit source reads and +a separate destination slot. For example, five compatible one-minute KLL states +can be merged into a stored five-minute KLL if coverage and accuracy permit it. +A merge used only to answer a query belongs in QueryPlan and creates no stored +destination: + +```text +PrecomputePlan: Read state A -> derive state B -> store B +QueryPlan: Read state B -> estimate -> result +``` + +## Runtime contract + +The backend stages the catalog and both plans as one plan version and exposes them +atomically. Failed staging leaves the previous plan version active. + +Installation and readiness are distinct. Until required state coverage exists, +QueryPlan uses its configured exact fallback or returns explicit unavailability. +The query runtime follows installed state references; it does not search the +catalog for alternative summaries. + +Visualization renders PrecomputePlan and QueryPlan separately, connected by +labeled state references. Legacy full-DAG artifacts may use a projected +view, but it must label maintenance-owned and query-owned nodes. + +## Validation and acceptance + +Compilation and installation reject unresolved state references, schema/encoding +mismatches, incompatible grouping or time partitions, wrong plan versions, cycles, +unsupported phase operators and unsatisfied derived-state completeness. + +Acceptance tests demonstrate: + +1. Summary construction executes only in PrecomputePlan and estimation only in + QueryPlan. +2. One query can read multiple summaries and two queries can share one producer. +3. Derived summaries honor completion and schema requirements. +4. Invalid cross-plan bindings fail before activation. +5. Staging failure, restart and plan version switching preserve consistency and + documented fallback behavior. +6. The backend builds and runs these cases without ASAPCollector. + +## Decisions and deferred work + +The full semantic DAG is retained only as provenance or diagnostic metadata; +bindings alone do not make it valid PrecomputePlan executable content. The two +physical plans are not compiled independently because that permits identity and +schema drift. + +Deferred work includes CollectorPlan and TransmissionPlan compilation, distributed +activation, new transport/checkpoint protocols, Collector adoption of neutral +libraries and a broader ASAPPlanner API redesign. diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index babe652c..34557a72 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -1,313 +1,171 @@ -# ASAPPlanner integration: migration delivery plan +# PrecomputePlan and QueryPlan migration plan -Status: implementation sequence for the -[system architecture proposal](asapplanner-integration.md). A checked milestone -requires executable evidence; publishing this plan or opening a PR does not -complete migration. +Status: proposed delivery sequence. Audience: backend implementers. -## Baseline and completion definition +Terminology: [Planner/backend glossary](planner-backend-glossary.md). -The inspected baseline is backend `95131d83972bb7a07d338e2a5af925a20c15ddce`. -The compiler already deduplicates backend PrecomputePlan state by physical -fingerprint and binds QueryPlan leaves explicitly. It still builds Collector -materialization declarations per query, and lifecycle selection builds a -single-query demand. Therefore, do not describe all sharing as absent, or -treat existing fingerprint deduplication as workload-wide optimization. +## Goal and scope -Migration is complete for a declared supported workload/profile when: +Replace complete semantic DAGs stored under PrecomputePlan with separate +PrecomputePlan and QueryPlan executable subgraphs connected by SDS state +references. Also remove the backend build/runtime dependency on ASAPCollector by +moving shared contracts and reconstruction code to neutral libraries. -- one Planner-authorized semantic decision governs all result roots; -- compatible shared producers have one physical maintenance path per source - partition and generation; -- unsupported sharing or operators are rejected or explicitly fall back; -- activation, readiness, query execution and feedback refer to matching - bindings and generations; -- supported entry points no longer independently select a different summary; -- parity and producer-update tests pass for the promised deployment profile. +CollectorPlan, TransmissionPlan, distributed activation, new transport behavior +and a general ASAPPlanner API redesign are deferred. -Backend-local and distributed profiles have separate acceptance evidence. -Neither arbitrary PromQL coverage nor ProjectASAP-wide engine coverage is a -completion prerequisite. +## Document map -## Delivery sequence and dependencies +1. [Migration at a glance](#migration-at-a-glance) +2. [Worked example](#worked-example) +3. [Stage 1: inventory and fixtures](#stage-1-inventory-and-fixtures) +4. [Stage 2: extract common code](#stage-2-extract-common-code) +5. [Stage 3: bind and split plans](#stage-3-bind-and-split-plans) +6. [Stage 4: validate and install](#stage-4-validate-and-install) +7. [Stage 5: migrate and retire](#stage-5-migrate-and-retire) +8. [Completion evidence](#completion-evidence) -Implementation tracking (PRs are not merged automatically): +## Migration at a glance -| PR | Implemented scope | +| Stage | Change | Exit gate | +| --- | --- | --- | +| 1. Inventory | Freeze current contracts and behavior as fixtures | Every supported path has a fixture or explicit unsupported result | +| 2. Extract | Move neutral contracts/codecs out of Collector | Backend dependencies and tests contain no ASAPCollector | +| 3. Split | Derive catalog, maintenance DAGs and query DAGs from one binding | Ownership and state references match selected semantics | +| 4. Install | Validate and atomically activate one plan version | Invalid snapshots fail without disturbing the active plan version | +| 5. Retire | Normalize old artifacts and remove superseded paths | Compatibility and end-to-end gates pass | + +Do not combine payload-format changes with dependency extraction. Version the new +plan representation separately from any later wire/schema change. + +## Worked example + +The current artifact may store this complete DAG under PrecomputePlan: + +```text +Input -> BuildKLL -> SummaryEstimate -> Result +``` + +The migration produces: + +```yaml +plan_version: 42 +summary_catalog: + definition: {id: def-9, algorithm: kll, k: 200} + +precompute_plan: + nodes: [Input, BuildKLL, 'WriteState(slot-17)'] + write_binding: {state_slot_id: slot-17, definition_id: def-9, schema: kll-v1} + +query_plan: + nodes: ['ReadState(slot-17)', SummaryEstimate, Result] + read_binding: {state_slot_id: slot-17, definition_id: def-9, expected_schema: kll-v1} + +provenance: + selected_dag: Input -> BuildKLL -> SummaryEstimate -> Result +``` + +During rollout, the backend normalizes a supported legacy artifact into this +internal form. Old and new forms must produce the same update count and query +result. After compatibility gates pass, the complete-DAG execution path can be +removed while its versioned reader remains for the supported window. + +## Stage 1: inventory and fixtures + +Inventory Planner output, plan/SDS types, state schemas, envelopes, +`asap_precompute_rs` imports, Cargo patches, build scripts and tests that invoke +Collector. + +Capture fixtures for: + +- full, delta and legacy bare-state decoding; +- summary reconstruction, maintenance updates and query readout; +- completion, restart and recovery; +- staging, activation, readiness and fallback. + +Fixtures may originate from Collector but must run without a Collector checkout +or process. Record source revision and schema provenance; use semantic assertions +when randomized sketch bytes are unstable. + +Preserve complete lifecycle commitments from Planner selection. A backend that +only supports batch construction from data at rest must not infer incremental +support from recurring query demand. + +## Stage 2: extract common code + +| Neutral responsibility | Excludes | | --- | --- | -| [Backend #513](https://github.com/ProjectASAP/ASAPQuery-backend/pull/513) | A: compatible physical producer deduplication and conflicting deployment-contract rejection | -| [Backend #514](https://github.com/ProjectASAP/ASAPQuery-backend/pull/514) | B prerequisite: port the backend from its divergent historical pin to merged Planner APIs, including typed summary inputs | -| [Planner #356](https://github.com/ProjectASAP/ASAPPlanner/pull/356) | B: reusable, scope-local typed post-ASAP subtree interning; includes schemas and guarantees in equivalence | -| [Backend #515](https://github.com/ProjectASAP/ASAPQuery-backend/pull/515) | B: workload search, shared producer bindings and persistent query-root mapping | -| [Backend #516](https://github.com/ProjectASAP/ASAPQuery-backend/pull/516) | C: backend-local packed SUM/observation-count state, exact readouts, additive reductions and constrained arithmetic; production HTTP acceptance | -| [Backend #517](https://github.com/ProjectASAP/ASAPQuery-backend/pull/517) | E: current distributed publication/frame protocol, actual Collector validator, two shared readouts, failed staging and inactive-generation rejection | -| [Backend #518](https://github.com/ProjectASAP/ASAPQuery-backend/pull/518) | B/F: one workload-selection adapter for canonical startup and compile-and-publish; query-scoped accuracy certificates | -| [Backend #519](https://github.com/ProjectASAP/ASAPQuery-backend/pull/519) | D component: joint producer lifecycle demand, incompatible-evidence rejection and identity-keyed lifecycle estimates | -| [Backend #520](https://github.com/ProjectASAP/ASAPQuery-backend/pull/520) | E: published config drives the actual Collector Rust update/window/emission loop; N raw observations yield N updates and one shared output | -| [Backend #521](https://github.com/ProjectASAP/ASAPQuery-backend/pull/521) | E: failed staging cleanup permits retry; concurrent readers survive successful same-semantic generation cutover; retired frames are rejected | -| [Backend #522](https://github.com/ProjectASAP/ASAPQuery-backend/pull/522) | D: provider-priced complete bound-workload selection, strict v2 startup evidence, read-only quote preparation, live publication/reporting and process acceptance | - -The backend PRs form a sequential review stack from #513 through #522; -#515 uses merged Planner #356 at revision -`378a7547ede629a64e84c9f7c810226ce196cce9`. #516 includes the fail-closed -arithmetic regression fix, propagated through its dependent branches. -The backend-local dashboard and distributed single-partition quantile examples -have executable acceptance evidence, including complete cost-based selection -and same-semantic generation cutover. The supported-profile implementation -is in the review stack, not yet merged or deployed. Production calibration, -platform-specific rollout and broader semantic workload replacement are not -claimed complete by these fixtures. - -Local verification of the original combined migration stack: 654 control-plane -library tests, 28 control-plane binary tests, one control-plane integration -test, 977 data-plane library tests and three production-process tests passed. Planner -#356 passed its 156 type-library tests and GitHub formatting/lint/test checks. -The backend process tests cover the actual binaries and Collector Rust library, -not production traffic or every Collector platform adapter. Local passes do -not replace PR CI, review or the remaining migration gates. - -| Slice | Repository | Depends on | Deliverable and acceptance | -| --- | --- | --- | --- | -| A. Safe physical state sharing | ASAPQuery-backend | Existing compiler | Deduplicate Collector declarations for compatible state; reject conflicting implementation/layout/lifecycle contracts; keep both query roots bound to one backend state | -| B. Workload semantic planning adapter | ASAPQuery-backend, with Planner changes only for demonstrated gaps | A and Planner API audit | Batch registered canonical roots through reusable Planner search; preserve root mapping and producer identity; do not implement backend-local semantic CSE | -| C. Aggregate-state fusion and readouts | ASAPPlanner for rules; backend for execution | B | SUM/COUNT example with per-consumer projections, label/time equivalence and fully executable division; reuse existing decomposition/rollup rules | -| D. Workload-wide implementation evidence | ASAPQuery-backend and Planner evidence boundary | B; C for fused states | Compare complete alternatives with shared build/update cost once and per-consumer read costs; joint state lifecycle/implementation agreement | -| E. Bound execution and lifecycle acceptance | ASAPQuery-backend; Collector only where public runtime gaps require it | A–D | Producer update counts, readiness/fallback, generation isolation, failed rollout, and distributed projection tests | -| F. Compatibility-path retirement | ASAPQuery-backend | E for each affected profile | Route supported entry points through the validated path; remove duplicate selection only after call-site and parity audit | - -Slices are reviewable PR units, not an instruction to open empty placeholder -PRs. If a slice spans semantic changes and physical execution, split by -repository and stack the dependent PR explicitly. Do not merge automatically -or make one unverified pin bump cover unrelated Planner changes. - -## A. Safe physical state sharing - -The immediate regression fixture is two different quantile readouts over the -same source, parameters and window. It exercises existing supported operations -without depending on future SUM/COUNT fusion. - -Implementation scope: - -1. Compare concrete contracts when multiple selected leaves resolve to the - same physical fingerprint. Include algorithm/parameters, grouping, window - framework, implementation, pane layout and lifecycle. Runtime transmission - policies must also agree. -2. Emit one Collector producer declaration for a compatible shared state while - preserving every query's binding and readout. -3. Keep evidence conservative: differing evidence cannot silently disappear - during deduplication. A future certificate-union design is a separate step. -4. Reject conflicting contracts before any plan is published. Do not pick - whichever query happened to be visited first. - -> Historical note: this acceptance text predates the SummaryCatalog migration; -> the former BackendPlan state is now represented by a catalog materialization -> and its execution-plan references. - -Acceptance: both query roots exist; one catalog materialization and one PrecomputePlan -state exist; each Collector has one producer declaration; both bindings point -to that state. A different implementation/layout for the same fingerprint -fails compilation. Distinct source/window/parameters must remain distinct. - -This slice establishes deployment consistency, not workload search or a claim -that all query-time computations execute once across separate HTTP requests. - -## B. Workload semantic planning adapter - -Audit the pinned Planner workload/search APIs before defining another backend -plan representation. Inputs must preserve canonical query identity, source -selection, requirements, recurrence and time scope. - -The result must retain all original roots and shared logical producers. -Backend bindings must be keyed by workload-scoped producer identity, not only -a per-query pointer. Physical IDs stay downstream. Preserve explicit mappings -from each query root to its required materializations and fallback. - -Acceptance fixtures: - -- identical producers used by two different roots; -- a diamond within one query and sharing across queries; -- incompatible filters, grouping, windows or accuracy do not share; -- round-trip compilation retains roots and sharing; -- unsupported alternatives cannot become partially executable warm routes. - -Pointer sharing in memory alone is not persistent identity. A serialized -execution projection must preserve the relationship explicitly. - -## C. Aggregate-state fusion and complete readouts - -Use the system document's sample-weighted mean example as the target. -Planner owns the equivalence rule: union compatible SUM/COUNT states and -project the needed results to consumers. The backend owns physical state -implementations and exact output operators. - -First inspect existing AVG decomposition, CSE and rollup rules. Add only missing -semantics upstream; do not copy DQC transformation objects or hard-code metric -names in Planner. - -Acceptance includes uneven per-instance sample counts, missing/stale series, -multiple services, exact interval endpoints, range evaluation steps and -denominator edge cases. Query results must match Prometheus labels, timestamps -and numeric semantics. Until the whole expression is supported, preserve -explicit fallback rather than claiming partial integration. - -The implemented backend-local example uses one raw accumulator that retains -both sum and observation count. This is native physical packing of selected -Planner operations, not a new backend semantic rewrite. The process test has -two services: observations `[10]` and `[2, 4, 8]` across two API instances give -SUM = 24, COUNT = 4 and weighted mean = 6; worker observations `[9, 15]` give -SUM = 24, COUNT = 2 and mean = 12. Three registered consumers still configure -one producer; a Remote Write retry does not double the counts. Range steps, -output labels/timestamps and unaligned-window fallback are checked. - -Do not generalize that execution contract to `sum(sum_over_time(m) / -count_over_time(m))`: summing per-instance means cannot pool samples first. -Non-additive entity reduction, mismatched operand grouping/windows, shifted -selectors and unverified instantaneous/temporal combinations remain explicit -fallbacks. Unknown legacy observation counts also fail closed. Distributed -observation-count readout is not advertised by this implementation. - -## D. Workload-wide evidence and selection - -Today per-query lifecycle inputs are not proof of joint workload costing. -Aggregate demand for each shared producer while retaining consumer-specific -requirements. Compare alternatives over one horizon and data scope. - -Charge shared initialization and maintenance once, account for all consumer -readouts and live/retained state, and include applicable placement and -transmission costs. Feasibility checks cover the entire selected DAG, not -only a summary family. The winning evidence must resolve to the same concrete -implementation that compilation installs. - -Acceptance: a shared alternative wins when its complete cost is lower, loses -when retention/materialization overhead dominates, and is unavailable when -any required capability/evidence is absent or stale. Adding another consumer -must not double-count the producer's update stream. - -Implemented component: #519 gives each unique physical producer a -`WorkloadDemand` containing all its consuming query entries. For a 300-second -horizon, 100 updates/second and two consumers reading every 10 and 20 seconds, -the demand is 30,000 updates and 45 reads. With build = 10, update = 0.001, -read = 0.1, retention/second = 0.001 and retirement = 1, the lifecycle cost is -45.8. Adding the second consumer increases cost by 1.5, not another build and -update stream. Publication reports this component against the materialization -and implementation identities; it is not a complete-plan total. - -Implemented selection: #522 compares complete bound alternatives before -commitment. A provider prices source upkeep, each shared state's build/update/ -residency/retirement per location, transport, every reachable query operator, -and results over one common horizon. Query work is multiplied by recurrence; -shared maintenance is not multiplied by consumer count. Native exact fallback -includes its service's input upkeep as well as full native query execution. - -The default inventory is the Planner-selected continuously maintained workload -and its whole-workload exact alternative. The comparison interface also accepts -additional Planner-authorized, bindable forests; this is not exhaustive search -over all engines or lifecycle variants. Tests prove both the sharing win and -high-retention loss, and reject missing, stale, mismatched or infeasible quotes. - -Implementation refinement: pricing uses a flat coverage manifest over the -existing bound physical projection, not another semantic DAG. It does not -populate `PlannerPhysicalPlanProvider` with guessed source statistics or split -the older opaque per-query window scalar into fabricated components. Providers -must quote the actual source scope, state layout, implementation and capability -generation. The selected plan and report retain those identities. - -Version-2 canonical snapshots require complete evidence. Live requests can -obtain requirements from the read-only `cost-manifests` endpoint before -publication. Version 1 and live requests without quotes remain explicitly -uncosted compatibility paths. See the [provider workflow in #522](https://github.com/ProjectASAP/ASAPQuery-backend/blob/feat/complete-workload-cost-selection/docs/examples/workload-cost-evidence.md). - -Production calibration still requires evidence from the intended deployment; -the deterministic fixture costs are not production measurements. The provider -attests exact-backend access and resource feasibility; a low cost alone does -not establish either. - -## E. Runtime and deployment acceptance - -Start backend-local, then validate the distributed profile independently. - -- Replay deterministic raw samples through production ingestion. -- Count state creation and updates: one compatible producer per generation, - with no duplicated updates when a second query subscribes. -- Query both roots through HTTP and compare with an exact reference. -- Test incomplete coverage, stale state, absent routes and unavailable fallback. -- Stage a successor while requests run; each request observes one generation. -- Fail staging or producer acknowledgement and verify the active generation - remains unchanged. -- For distributed collection, decode emitted plans through the actual Collector - validator and assert one producer per source partition, not one producer - globally across independent sources. - -Unit-level declaration counts do not replace runtime update-count tests. - -Current evidence combines real backend executables with the actual Collector -Rust runtime library. The test's host adapter supplies OpAMP acknowledgements -and frame metadata; it does not launch a platform-specific Collector binary. -In #521, failed Collector staging is discarded without touching the active -snapshot; the same successor version can then be retried successfully while -queries run. Old-generation frames are rejected after cutover and successor -frames become queryable. #522 exercises this flow with costed publication. -This verifies same-semantic runtime generation replacement, not arbitrary -semantic workload replacement or a platform-specific production rollout. -Platform adapter rollout remains a deployment acceptance step. - -## F. Retire duplicate selection safely - -Inventory canonical startup compilation, explicit compile-and-publish, -legacy workload adapters and serving-time binding helpers. Distinguish dead -code from intentionally supported profiles using call-site inspection. - -For each path, either route it through the selected workload contract, retain -it as an explicitly unsupported/fallback adapter, or remove it after parity. -Parsing and canonicalization at serving time are fine; family/parameter, -grouping or lifecycle reselection is not. - -Do not remove QueryPlan, PrecomputePlan, physical deployment selection, -exact fallback, or profile-specific adapters merely because their types are -different from post-ASAP IR. - -Call-site audit: production instant/range serving already requires an active -physical QueryPlan and declines absent or unregistered routes. The old -summary-selection serving branches in `engine.rs` are `cfg(test)` fixtures. -#518 unifies the two first-class compilation entry points. Legacy flat-workload -demo/configuration adapters remain separate compatibility paths; they must not -be presented as migrated canonical-workload entry points or removed without -their own parity/retirement decision. - -## Existing PR coordination - -At the baseline inspection, open PRs -[#505](https://github.com/ProjectASAP/ASAPQuery-backend/pull/505), -[#506](https://github.com/ProjectASAP/ASAPQuery-backend/pull/506), -[#509](https://github.com/ProjectASAP/ASAPQuery-backend/pull/509) and -[#511](https://github.com/ProjectASAP/ASAPQuery-backend/pull/511) cover PromQL, -process-E2E and TopK-related work. Re-check their status and changed files -before touching overlapping paths. Their presence is not evidence that the -workload-sharing migration is complete. - -Review follow-up (2026-09-08): #505 is now stacked on #522 and uses the merged -Planner revision above. Typed TopK update weights belong to the selected -producer, not its readout. Its multi-series fixture distinguishes count ranking -(`api=4`) from value ranking (`worker=200`). #509 compares complete vectors at -each range step, including changing winners. #506 tests unregistered-query -fallback; it is not evidence that registered arithmetic is unsupported. - -#515 preserves duplicate algorithm candidates during cost ranking; removing -them violates Planner's candidate-multiset contract and can panic. #522 quote -preparation enumerates bindable alternatives without requiring the default -warm alternative to compile, so missing warm implementations do not hide an -available exact quote. Publication still requires a selected, validated plan. - -#511 retains evidence-aware legacy binding and preserves count update semantics -in emitted heap configuration. Its two heap TopK acceptance tests now use -registered `topk(3, count_over_time(top_endpoint_qps[5s]))`, a compiled physical -QueryPlan, and the production backend-local Remote Write path. Both CMS-with-heap -and CountSketch-with-heap return gamma=200, zeta=150 and alpha=100 over two -windows, with exact item identities, timestamps and retry deduplication checked. -Unregistered instantaneous TopK still follows the explicit exact fallback. -This replaces the two obsolete no-QueryPlan tests; it does not restore that -serving contract or claim migration of other legacy OTLP fixtures. - -The [architecture PR #512](https://github.com/ProjectASAP/ASAPQuery-backend/pull/512) -tracks the design and this delivery plan. Implementation PRs should report the -slice they complete, tests actually run, and remaining acceptance gaps. +| Envelope metadata, shared IDs/schema references and validation | Planner optimization and runtime executors | +| Sketch schemas, encode/decode/reconstruction and supported state operations | Window scheduling, host adapters and backend storage | + +Prefer existing sketch-library APIs. Move reusable DDSketch/KLL reconstruction +out of Collector wrappers and remove reconstruct-serialize-decode round trips. +Keep legacy readers and family-specific backend paths until replacements have +parity evidence. + +Remove `asap-precompute-rs` and Collector-specific Cargo patches. Inspect +manifests, lockfiles, dependency graphs, scripts and required tests for direct or +transitive Collector dependencies. + +## Stage 3: bind and split plans + +Create compiler bindings for semantic nodes, summary definitions, +version-scoped state slots, schemas and state references. Derive the catalog and both plans +from those bindings using the +[materialization-boundary rules](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries): + +- PrecomputePlan contains maintenance inputs/operators and state sinks. +- QueryPlan contains state reads, `SummaryEstimate`, exact residuals and results. +- Derived maintenance uses explicit completed-state references. +- Shared producers retain one identity and update path. +- Provenance records semantic operations absorbed into physical nodes. + +Version the split representation. Do not reinterpret an old field under an +unchanged schema version. + +Do not introduce a standalone catalog `Materialization` object. Keep definitions +in the catalog, format/partition/writer configuration in executable bindings, +and actual coverage/location/readiness in instance inventory. Normalize legacy +stored-output identities into state slots while preserving payload locators; +validate all consumers against the same writer configuration. The existing +`BackendNodeBinding::Materialization` remains a placement marker for stored output. + +## Stage 4: validate and install + +Validate definition, state slot, schema, encoding, grouping, time partition, +coverage and plan version across the catalog and both plans. Then perform local +resource checks. + +Stage and activate the three artifacts as one snapshot. Readiness remains +separate: until coverage is ready, QueryPlan follows its configured fallback or +explicit unavailability. Failed staging preserves the previous plan version. + +Render PrecomputePlan and QueryPlan separately, joined by state references. +Legacy projected views label maintenance-owned and query-owned nodes. + +## Stage 5: migrate and retire + +Release pinned neutral-library versions and rollback artifacts. Migrate +backend-local publications first and retain versioned adapters for the supported +compatibility window. + +Remove complete-DAG precompute execution and Collector adapter code only after +fixtures and end-to-end tests pass. State reuse across plan versions requires an +explicit SDS compatibility decision independently of binary rollback. + +## Completion evidence + +Completion requires: + +- summary construction runs only in PrecomputePlan and estimation only in + QueryPlan; +- one query can read multiple summaries and two queries can share one producer; +- derived state observes completion and schema requirements; +- invalid bindings fail before activation; +- restart and plan version switching preserve consistency and fallback; +- legacy and split artifacts produce equivalent results and update counts; +- backend builds and required tests do not fetch, build or run ASAPCollector. + +Record tested revisions, supported state families, fixture results and dependency +checks. Trace one query from its selected semantic root through the state +writer, SDS reference and QueryPlan reader. diff --git a/docs/design_docs/planner-backend-glossary.md b/docs/design_docs/planner-backend-glossary.md new file mode 100644 index 00000000..158dec5c --- /dev/null +++ b/docs/design_docs/planner-backend-glossary.md @@ -0,0 +1,61 @@ +# Planner/backend design glossary + +For developers reading the [integration](asapplanner-integration.md), +[SDS](summary-catalog-sds-architecture.md), and +[migration](asapplanner-migration-plan.md) designs. Definitions describe the +proposed boundary; they do not imply that every proposed field already exists +in the serialized API. + +## Computation and execution + +| Term | Meaning | +| --- | --- | +| Selected post-ASAP DAG | Planner-selected computation graph, including summary producers, shared dependencies and query readouts. Called the “semantic DAG” in earlier discussion. | +| Summary producer | An operation or subgraph that builds summary state. Multiple queries may share its stored output. | +| `SummaryMaintenanceLifecyclePlan` | Planner result associating a post-ASAP root with deployment decisions for its unique reachable summary producers, plus workload and costing context. | +| Lifecycle commitment | Selected maintenance promise for one producer, with its scheduling and retention binding. A deployment's `SummaryMaintenanceLifecycleGuarantee` carries the Planner-level commitment. | +| Maintenance | Work that constructs, refreshes or derives stored summary state, including batch rebuilds and incremental updates. | +| `PrecomputePlan` | Backend executable plan for maintenance and state writes. | +| `QueryPlan` | Backend executable plan for state reads, query readouts and remaining query operations. | +| Readout / `SummaryEstimate` | Operation that obtains a query value from summary state, such as p99 from KLL. | +| Derived summary state | Stored summary state computed from existing summary states. Earlier discussion calls this a “derived materialization”; it does not require a separate catalog object. | +| Exact residual | Part of the selected query computed exactly around summary operations, such as supported filtering or arithmetic after readout. It does not make the whole approximate result exact. | +| Exact fallback | Configured execution of the original query through an exact route when the summary plan cannot serve it. | + +For example, merging five compatible one-minute KLL summaries and storing the +five-minute result produces derived summary state in a separate destination +slot. Merging them only to answer a query is a query-time operation. Both require +compatible grouping, coverage and accuracy. + +## State and identity + +| Term | Meaning | +| --- | --- | +| Summary Catalog | Metadata registry of summary definitions; payload bytes live in the summary store. | +| SDS (Self-Describing Summary) | The description and metadata needed to interpret and validate stored summary state. It is not a separate execution engine or payload store. | +| `SummaryDefinition` | What a summary represents: source/filter, input value, grouping, time semantics, algorithm and parameters. | +| `state_slot_id` | Compiler-assigned identifier for a stored producer output within one plan version. Shared readers use the same slot; it has no independent catalog object. | +| `StateReference` | Plan reference identifying a slot and summary definition within the enclosing plan version. Reader configuration selects the required state instances and constrains format and coverage. | +| `SummaryStateInstance` | A concrete stored state, such as one service's completed five-minute KLL snapshot, with partition, coverage, format and location metadata. | +| Summary store | Storage for actual summary payloads. Runtime inventory records their existence, coverage and readiness. | +| `plan_version` | Version shared by an installed plan bundle and its catalog bindings. Creating or updating state instances does not itself change this version. Previously called “generation” in this proposal. | +| Schema / encoding | Schema describes the state structure; encoding describes how that structure is represented as bytes. | +| Provenance | Mapping from physical plan operations back to the selected Planner computation. | + +The existing `BackendNodeBinding::Materialization` marks a node whose output is +stored. It remains a node binding; this design has no standalone catalog +`Materialization` object. A materialization boundary is simply where a producer +writes stored state and a consumer reads it. + +## Time, selection and validation + +| Term | Meaning | +| --- | --- | +| Logical range | Input interval required by the computation. In the example, `range: 5m` means `(T - 5m, T]` at evaluation time `T`. | +| Pane | Physical time partition of stored state. Several compatible panes may serve one logical range; pane size need not equal that range. | +| Refresh cadence | How often the producer is scheduled to build or refresh state. | +| Retention | How long state remains available; distinct from its input range and refresh cadence. | +| Readiness | Whether the required state is available with valid format and sufficient coverage/completeness for a read. Plan installation alone does not establish readiness. | +| Backend capability | Declaration of supported implementation combinations: algorithm/parameters, maintenance mode, input kind, window behavior and format. | +| Physical cost evidence | Scoped measurements or estimates used to compare executable alternatives; includes workload and implementation context. | +| Compiler contract | Required inputs, outputs, validation rules and guarantees, including matching writer/reader definitions, formats, partitions and plan versions. | diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index 2522bb02..ccd18e7b 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -1,578 +1,252 @@ -# Summary Catalog and Self-Describing Summary Architecture +# Summary Catalog and Self-Describing Summary architecture -This design defines three logical layers for summary producers and consumers. +Status: proposed contract with current-backend migration notes. Audience: +developers compiling, storing, recovering or reading summary state. -| Layer | Describes | Changes when | -| --- | --- | --- | -| **Summary Descriptor** | Summary operator and fidelity guarantees | Algorithm, configuration or guarantee contract changes | -| **Data Descriptor** | Summarized source and population | Source binding or population definition changes | -| **Summary Instance** | Instance metadata and summary state | A concrete materialization is created or updated | +Terminology: [Planner/backend glossary](planner-backend-glossary.md). -Separating these layers lets many materialized instances reuse the same operator -configuration and data scope. A new time interval creates a new instance without -copying or redefining either descriptor. +## Purpose and scope -## Proposed ownership +The Summary Catalog and Self-Describing Summary (SDS) model defines what persisted +summary state means. It connects PrecomputePlan writers to QueryPlan readers +without requiring either runtime to reinterpret Planner IR. -The descriptor vocabulary is a shared contract in `asap_types`. The control -plane owns the authoritative `SummaryCatalog`; Collector and backend receive the -same immutable catalog snapshot. Planner reasons about operators, fidelity, -source and population semantics, while runtime components bind catalog identities -to producers and stored instances. +This document owns summary identity, schema, state references, readiness and +lifecycle. The [integration design](asapplanner-integration.md) owns executable +plan splitting; the [migration plan](asapplanner-migration-plan.md) owns delivery. +Cost ranking, operator scheduling and transmission policy are outside SDS. -| Layer | Responsibility | -| --- | --- | -| Summary Descriptor | Shared semantic definition used by Planner and backend | -| Data Descriptor | Shared source/population definition; backend resolves concrete runtime bindings | -| Summary Instance | Backend owns metadata, state, updates, storage and retirement | - -Planner may observe instance availability, covered time ranges and descriptor -references as planning evidence. It does not need the encoded summary state. -SDS describes summaries; an installed QueryPlan specifies how to execute a query -using them. The current backend fields are an incremental implementation of this -model. They must converge on the identities and invariants below rather than add -operator-specific stores beside `SketchStore`. - -## Target semantic model - -The target model has descriptor registries plus pane instances. Descriptor IDs -are derived from canonical semantic content; display names and runtime SIDs are -not descriptor identities. `SummaryDescriptorId` and `DataDescriptorId` currently -contain versioned canonical semantic strings. `SummaryDefinitionId` is a distinct -typed policy fingerprint, and `CatalogGeneration` identifies a publication using -its digest and plan version. A physical `SeriesId` identifies one storage lifetime -of a definition/group; it is neither a descriptor ID nor a pane instance ID. -Changing descriptor encoding to a hash must preserve content identity and handle -collisions explicitly. - -```rust -struct SummaryDescriptor { - id: SummaryDescriptorId, - operator: SummaryOperator, - fidelity: Vec