From ac30f3214df03ad27cfbaed029c515af4dcca99c Mon Sep 17 00:00:00 2001 From: zz_y Date: Thu, 17 Sep 2026 18:23:13 +0000 Subject: [PATCH 01/15] docs: clarify Planner physical plan and SDS architecture --- docs/design_docs/README.md | 42 +- docs/design_docs/asapplanner-integration.md | 876 ++++++++++++------ .../design_docs/asapplanner-migration-plan.md | 495 ++++------ .../summary-catalog-sds-architecture.md | 292 +++--- 4 files changed, 896 insertions(+), 809 deletions(-) diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 721bd0aa8..433e26663 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -1,26 +1,26 @@ -# System design location +# Design documents -ASAPQuery-backend does not maintain a second copy of the system design. -The canonical component design is in -[ASAPCollector/docs/design_docs](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs/design_docs). +These documents are for architects and developers. The integration proposal and +SDS model below define the target Planner-to-runtime boundary; their current-code +notes and migration gates distinguish implemented behavior from proposed changes. -Backend-specific implementation design notes are organized by component under -[`../developer_docs`](../developer_docs/README.md). They explain current Rust -internals and are subordinate to the shared system contracts. +- [Planner, physical plans, SDS, and runtime architecture](asapplanner-integration.md) + owns semantic/physical compilation, common bindings, the four plan projections, + policy ownership, codec boundaries, and publication/activation requirements. +- [Summary Catalog and SDS](summary-catalog-sds-architecture.md) owns descriptors, + definition/instance identity, state references, inventory and lifecycle semantics. +- [Architecture migration delivery plan](asapplanner-migration-plan.md) defines + compatibility fixtures, implementation stages, rollout and retirement gates. +- [Accepted-input completeness](continuous-summary-completeness.md) describes + the backend's bounded admission, publication and recovery behavior. -Proposals for shared-contract review: +Existing [Collector system contracts](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs/design_docs) +remain the cross-component compatibility baseline until coordinated migrations +land. These proposals do not silently change those interfaces. Current backend +implementation guides live under [developer docs](../developer_docs/README.md). -- [ASAPPlanner integration architecture](asapplanner-integration.md) proposes - the Planner/backend responsibility boundary, shared semantic DAG workflow, - and high-level consolidation milestones. -- [Summary Catalog and SDS Architecture](summary-catalog-sds-architecture.md) defines the proposed - Summary Descriptor, Data Descriptor and Summary Instance layers. +Other designs and profiles: -These proposals complement the canonical cross-component contracts above. - -Backend-specific operating profiles: - -- [ASAPQuery compatibility profile](asapquery-compatibility-profile.md) defines - the smaller target configuration for Prometheus Remote Write, backend-local - precompute, and PromQL serving without ASAPCollector. It becomes a strict - configuration subset after its currently missing Remote Write adapter lands. +- [ASAPQuery compatibility profile](asapquery-compatibility-profile.md) +- [Shape-aware ERP](shape-aware-erp-v1.md) +- [Empirical observability execution plan](empirical-o11y-execution-plan.md) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index c6cc83be6..9284f6e3b 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -1,308 +1,596 @@ -# ASAPPlanner and ASAPQuery-backend: integrated architecture +# Planner, physical plans, SDS, and runtime architecture -Status: proposed system-level consolidation and high-level migration, grounded -in existing integration. This is not a claim that every target capability is -implemented. No repository rename is proposed. +## Audience, status, and scope -This document owns the Planner/backend integration proposal, not a second copy -of the [shared ASAP system contracts](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs/design_docs). -The existing physical-plan, collection, transmission, and storage contracts -remain authoritative for their respective interfaces. +Audience: architects and developers of ASAPPlanner, ASAPQuery-backend, and +ASAPCollector. This document defines the target integration architecture. +The current-code baseline below is separate from the proposed changes; writing +this design does not establish runtime support or change a wire contract. -## Design decision +This document owns the integration boundary and compilation flow. The +[SDS design](summary-catalog-sds-architecture.md) owns descriptor, instance, and +state-lifecycle semantics. The [delivery plan](asapplanner-migration-plan.md) +owns implementation gates. Existing +[Collector system contracts](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs/design_docs) +remain the compatibility baseline until corresponding changes land in both +consumers. Conflicts require a versioned migration, not unilateral reinterpretation. -

Figure 1. Integrated ASAPPlanner–ASAPQuery-backend architecture and workflow.

+## Problem and current baseline -```mermaid -flowchart TD - subgraph PlannerBoundary["ASAPPlanner boundary — reusable optimization"] - Canonical[Canonical QueryExpr and workload semantics] - Canonical --> Strategies[CSE and reusable replacement strategies] - Strategies --> Candidates[Candidate post-ASAP workload DAGs] - Candidates --> Ranking[Semantic legality, accuracy and evidence-based ranking] - end - - subgraph BackendBoundary["ASAPQuery-backend boundary — observability application"] - Inputs[PromQL registrations, QueryWorkload and DataWorkload] - Evidence[Runtime capabilities and complete deployment cost evidence] - Commit[Control plane commits a feasible post-ASAP workload DAG] - Compile[Physical binding and deployment selection] - Bundle[One versioned physical plan bundle] - Activate[Validate, stage and activate] - Precompute[Ingest, precompute and summary store] - Serve[Bound query execution and explicit exact fallback] - Feedback[Readiness, accuracy and resource observations] - Inputs --> Commit - Commit --> Compile --> Bundle --> Activate - Activate --> Precompute - Activate --> Serve - Precompute --> Serve - Precompute --> Feedback - Serve --> Feedback - Feedback --> Evidence - end - - Inputs -->|Planning request| Canonical - Candidates -->|Implementation evaluation request| Evidence - Evidence -->|Feasibility and cost evidence| Ranking - Ranking -->|Legal ranked post-ASAP alternatives| Commit - Bundle -->|CollectorPlan in distributed profile| Collector[ASAPCollector — external runtime] - Collector -->|Planned data or summary frames| Precompute - Clients[PromQL clients] --> Serve - Serve -->|Configured exact route| Exact[Prometheus or archive query service] -``` - -**ASAPPlanner's selected post-ASAP workload DAG is the authoritative semantic -plan. ASAPQuery-backend binds and executes that decision through its control -plane and data plane.** Backend physical plans remain necessary, but must be -traceable projections of that DAG, not independently optimized replacements -for its dependencies, shared state, or query-result semantics. - -Planner provides reusable legal alternatives and ranking. The backend owns -deployment commitment, concrete realization, and operational policy. A -deployment choice cannot silently change Planner-owned grouping, statistic, -summary parameters, logical window, accuracy, or lifecycle: it must return to -the legal candidate-selection boundary. +Collector and backend must agree on what a summary means, how it is produced, +how updates travel, and how queries consume it. Sharing an envelope decoder +alone does not guarantee agreement across these boundaries. -## Architecture boundaries and reuse +The inspected backend baseline is `b06385d1c155986c05ccbd011978e43bf3786deb`. +The following are current implementation facts, not the desired dependency graph: -ASAPQuery-backend is the observability downstream application, including the -MetricsObservabilityQuery use case. DQC (the proposed name for the current -asap-fusion repository) is a separate downstream application, not an execution -dependency of this backend. - -| Responsibility | ASAPPlanner | ASAPQuery-backend | -| --- | --- | --- | -| Query semantics | Canonical expressions, equivalence, grouping and time semantics | PromQL API, workload registration and profile restrictions | -| Optimization | CSE, legal sharing, rollup, decomposition, summary and accuracy alternatives | Feasibility evidence, deployment commitment and concrete assignments | -| Time and state | Logical windows, abstract window framework and maintenance lifecycle | Panes, retention layout, update implementation and placement | -| Plan identity | Logical producer identities and result dependencies | Plan versions, physical materializations, SID bindings and runtime handles | -| Execution | Deployment-independent semantic contract | Ingest, precompute, store, serving, readiness and fallback | -| Operations | Reusable models consuming scoped evidence | Activation, rollback, telemetry, freshness and resource enforcement | - -Reuse works in both directions. The backend consumes Planner strategies; -general-purpose rules discovered while optimizing repeated observability -queries belong in Planner so DQC and other applications can reuse them. -Prometheus staleness handling, SID resolution, Collector placement, and OpAMP -publication remain downstream responsibilities. - -## Inspection: what already exists - -Inspected backend main at -[`95131d83972bb7a07d338e2a5af925a20c15ddce`](https://github.com/ProjectASAP/ASAPQuery-backend/tree/95131d83972bb7a07d338e2a5af925a20c15ddce), -using its pinned Planner revision -[`cb50219c582d43f53ab77d3a595bd1ea4a9aa119`](https://github.com/ProjectASAP/ASAPPlanner/tree/cb50219c582d43f53ab77d3a595bd1ea4a9aa119). -The baseline is merged code, not the completion of open PRs. - -| Area | Existing foundation | Consolidation needed | +| Area | Existing foundation | Remaining coupling | | --- | --- | --- | -| Frontend and selection | Planner dependency, canonical query parsing, backend selection from Planner alternatives | Make workload-wide sharing and strategy composition explicit across supported entry points | -| Physical compilation | One bundle with precompute, transmission, backend and query projections; Collector projections when applicable | Preserve all selected shared producers and provenance through every projection | -| Serving | Bound QueryPlan execution, exact materialization identities and explicit fallback | Audit remaining compatibility paths; serving must not make a new summary choice | -| Deployment | Versioned staging and activation, runtime capability and evidence checks | Verify profile-specific failure and readiness behavior end to end | -| Compatibility | Backend-local ASAPQuery profile alongside distributed collection | Keep distinct deployment profiles on the same semantic contract | - -Evidence: -[selection adapter](../../control_plane/src/planner_selection.rs), -[physical compiler](../../control_plane/src/physical/compiler.rs), -[legacy workload adapter](../../control_plane/src/physical/workload_planner.rs), -[shared QueryPlan](../../crates/asap_types/src/query_plan.rs), -[query lowering](../../control_plane/src/query_plan.rs), and -[bound serving executor](../../data_plane/src/query_engines/asap_query_engine/post_asap_readout.rs). -The selection adapter explicitly commits a ranked Planner candidate downstream. -Consequently, the figure does not imply that the Planner library deploys or -commits a complete backend configuration by itself. - -This is an extension of existing integration, not a proposal to replace it -wholesale. Implementation guides sometimes describe a broader target than an -individual runtime path supports; migration acceptance must be demonstrated -against executable paths, not inferred from interface names. - -## One authoritative semantic DAG, derived runtime plans - -The shared contract must preserve sources and filters, label/grouping identity, -exact operators surrounding summaries, summary build/merge/readout, shared -producers, query roots, logical time coverage, accuracy, and maintenance -requirements. Audit the pinned post-ASAP representation for genuine gaps; -extend Planner semantics where necessary. - -Do not put concrete engine or implementation IDs into Planner IR. The backend -retains a binding from logical producer identity to implementation, placement, -materialization, state schema, and active generation. This follows the -[Planner/downstream boundary](https://github.com/ProjectASAP/ASAPPlanner/blob/cb50219c582d43f53ab77d3a595bd1ea4a9aa119/docs/design_docs/asapplanner-downstream-boundary.md). - -One selected DAG can produce several execution projections: - -- PrecomputePlan: how the selected state is built and maintained. -- TransmissionPlan and optional CollectorPlan: how distributed producers - implement and deliver that state. -- SummaryCatalog: canonical summary/data descriptors and stable materialization identities. -- QueryPlan: executable reads, merges, readouts and remaining exact operations. - -These projections may expand one semantic node into several physical tasks. -They must not invent a different semantic sharing graph. QueryPlan need not be -a byte-for-byte serialization of post-ASAP IR, nor should ingestion and query -serving literally run an identical task schedule. They implement different -phases of the same selected computation. - -Sharing has explicit scope: maintain a shared producer once per compatible -source/window/plan generation; reuse its state across query roots. Memoizing a -query DAG within one request is useful but does not, by itself, prove -cross-query or cross-request sharing. - -## End-to-end workflow - -1. **Register demand.** Collect canonical queries, evaluation cadence, time - windows, accuracy scope, source arrival facts and optimization horizon. -2. **Generate alternatives.** Planner applies legal rewrites and sharing, - choosing among summary, abstract-window and lifecycle alternatives. -3. **Evaluate implementations.** The backend checks runtime feasibility and - supplies complete, fresh costs over the same workload horizon. -4. **Commit and bind.** The control plane selects a legal workload alternative, - retains its concrete realization, and compiles one coherent plan bundle. -5. **Publish.** Validate and stage matching projections. For distributed - deployment, require the corresponding Collector application evidence - before activation. A failed rollout preserves the prior active generation. -6. **Maintain and serve.** Ingest updates the selected state; a request uses one - active snapshot and exact bindings. Warm execution requires complete, - fresh coverage. Otherwise follow the configured exact route or return an - explicit failure if that route is unavailable. -7. **Observe and replan.** Attribute cost, readiness and accuracy evidence to - the plan generation and producer. Semantic changes require a new planning - decision and activation, not an ad-hoc serving-time substitution. - -The backend-local profile uses Remote Write, local precompute and Prometheus -fallback without requiring Collector/OpAMP. The distributed profile may use -Collector-maintained summaries and configured archive services. Neither -profile's optional infrastructure becomes a prerequisite for the other. - -## Example: repeated dashboard queries sharing one state producer - -Consider a gauge `request_size_bytes`, one scalar series per -`(service, instance)`, without extra labels. Register these instant-query -expressions repeatedly at the same evaluation cadence: - -```promql -# Q1: sum of observed sample values per service over the last five minutes -sum by (service) (sum_over_time(request_size_bytes[5m])) - -# Q2: sample-weighted mean per service over that same interval -sum by (service) (sum_over_time(request_size_bytes[5m])) -/ -sum by (service) (count_over_time(request_size_bytes[5m])) -``` +| Compilation | `CompiledPhysicalPlan` contains catalog, query, precompute, collector, and transmission plans | Transmission compilation reads producers/schemas from PrecomputePlan; catalog is constructed from materializations and then bound back into plans | +| Publication | `PhysicalPlanPublication` validates related plans; backend supports staging/activation | Shared publication validation and runtime installation repeat some cross-plan checks | +| Contracts | `asap_types` contains SDS and installed plan types | Types still depend on Planner representations; Collector maintains separate Go/Rust DTOs | +| Semantic DAG | Planner exports a versioned DAG; backend retains node bindings | `OwnedPostAsapDag` serializes payloads into JSON to avoid process-local `Rc` ownership | +| Runtime policy | Transmission rules carry sampling, delta/GOS, and adaptation | Production semantics and transport controls share one policy structure | +| Sketch ingest | Shared sketch library plus an edge-runtime adapter | Backend imports Collector wrappers for DDSketch/KLL reconstruction; other reconstruction and delta paths remain local | + +Implementation references: [compiler](../../control_plane/src/physical/compiler.rs), +[publication](../../crates/asap_types/src/plan_publication.rs), +[producer contracts](../../crates/asap_types/src/producer_plan.rs), +[installed DAG](../../crates/asap_types/src/executable_plan.rs), and +[edge adapter](../../data_plane/src/precompute_engine/operators/edge_runtime_adapter.rs). + +## Goals and non-goals + +The minimum outcome is one selected semantic decision, one set of physical +bindings, and four consistent runtime projections. Both backend-local and +Collector-produced summaries must use this boundary. Backend ingest must no +longer depend on the Collector execution runtime for shared state codecs. + +Preserve supported query semantics, sharing, legacy decoding, completeness, +and recovery behavior during extraction. Neither arbitrary PromQL coverage, +a new optimizer, a universal execution engine, an arbitrary network topology, +nor a repository reorganization is required. A missing capability remains an +explicit rejection or configured exact fallback. + +## Inputs, outputs, and end-to-end behavior + +Inputs are canonical workload roots, query accuracy and freshness requirements, +Planner alternatives, and scoped deployment evidence: capabilities, topology, +source bindings, retained-state availability, and complete cost estimates. +The output is one validated `PhysicalPlanPublication` and target-specific +installation artifacts derived from it. -Q2 is deliberately not the unweighted mean of per-instance means. Its -denominator counts actual observations, which matters when instances have -different sample counts. These are gauge samples, not counter increases. - -A legal target alternative is: - -```text -Selected samples and logical five-minute coverage - | - Shared state per (service, instance) - SUM(value), COUNT(observations) - | - Merge/reduce by service - SUM(sum), SUM(count) - | - +------+------+ - | | - sum -> Q1 sum / count -> Q2 +```mermaid +flowchart TD + W[Canonical workload and requirements] --> P[ASAPPlanner semantic alternatives] + E[Capabilities, topology, costs, observed SDS] --> C[Control-plane physical compiler] + P --> C + C -->|Feasibility and costs for candidate selection| P + C --> B[Selected decision: catalog and common physical bindings] + B --> Q[QueryPlan] + B --> M[PrecomputePlan] + B --> L[CollectorPlan per target] + B --> T[TransmissionPlan] + Q --> U[PhysicalPlanPublication] + M --> U + L --> U + T --> U + U --> V[Validate, stage, coordinate activation] + V --> R[Collector and backend runtimes] + R --> O[Observed inventory, readiness, accuracy, costs] + O --> E ``` -Planner recognizes the common sum computation and can propose aggregate-state -fusion with per-consumer readouts. The backend implements the selected window -framework with compatible runtime state and binds both query roots to the -same producer. It must preserve PromQL range boundaries, labels, absent-series -behavior and division semantics; a missing denominator is not invented as -zero. Physical panes may be used only when their coverage matches the selected -logical interval, including boundary handling. - -This diagram is a target acceptance example, not a claim that today's compiler -already fuses these complete PromQL expressions. If an operator or window -cannot be realized end to end, the current supported behavior is explicit -fallback rather than partial warm execution with changed semantics. - -For the first milestone, use exact sum/count state and compare against -Prometheus at identical timestamps. Verify both numerical/label equivalence -and one maintained producer shared by the two roots. Exact aggregate state -does not eliminate the separate requirement to verify data completeness. - -Approximate extensions must declare what epsilon measures and what delta -covers. For a whole 20-row result with failure probability at most 0.05, -20 valid per-row failure bounds of at most 0.0025 suffice by the union bound; -independence is not required. Per-row 95% intervals alone do not establish -95% confidence for the complete result. Multiple dashboard evaluations need -their own declared scope; a result-level guarantee is not automatically -session-wide. Shared state also does not make separate errors independent. - -## Capabilities, costs and feedback - -Capabilities answer **can this deployment faithfully execute this alternative?** -Costs answer **which feasible alternative is preferable?** - -| Capability question | Why it constrains selection | +1. Planner produces legal semantic alternatives, retaining shared producers and + distinct query roots. Physical evaluation supplies feasibility and costs. +2. The control plane commits a feasible alternative and its concrete realization. +3. The compiler assigns catalog identities and binds semantic nodes, state, + producers, consumers, and data-flow edges once. +4. It projects those bindings into the four plans and validates the publication. +5. Targets stage their projections and required catalog content. The coordinator + authorizes activation only after the required target acknowledgements. +6. Producers maintain state; receivers apply authorized frames; queries use one + active plan snapshot and states with sufficient coverage and provenance. +7. Runtime evidence is attributed to those bindings and generations. A new + semantic choice returns to planning rather than changing query behavior locally. + +Plan installation and state readiness are separate. A query with missing or +incomplete state follows its configured exact route or returns an explicit +unavailable result; it cannot interpret missing state as an empty population. + +## Planner and compiler ownership + +Planner owns semantic equivalence, source/population semantics, grouping, +logical windows, summary families and parameters, result guarantees, lifecycle +choices, and maintenance-time versus read-time dependencies. Reusable sharing, +fusion, and rollup rules belong there. + +The physical compiler owns concrete implementations, placement, input routing, +state layout, retention realization, runtime identifiers, codecs, transmission +configuration, and deployment commitment. It must prove that an implementation +preserves the selected semantic decision. An unsupported choice returns to +candidate selection or fails explicitly; lowering cannot silently change its +window, sampling semantics, statistic, or guarantees. + +Capabilities and costs are distinct. A cheap implementation is not necessarily +feasible. Costs include shared construction once, maintenance, retained memory, +network, storage, recovery/checkpoints, per-consumer merges and readouts, and +query demand over the same horizon. Missing or stale evidence is not zero cost. + +The semantic IR export must be typed, versioned, and independent of internal +search ownership such as `Rc`. The target is one export contract shared by +Planner and consumers, with backend physical bindings alongside it. Migrate +`OwnedPostAsapDag` only after round-trip and runtime compatibility are proven; +do not introduce another operator language or require runtimes to import the +optimizer. Runtime evaluation of installed operators remains legitimate. + +## Caller contract and lifecycle completeness + +[Planner issue #438](https://github.com/ProjectASAP/ASAPPlanner/issues/438) +identifies a separate interface requirement: callers need to know the required +inputs, the consequences of omissions, and the promises of each output. A shared +DAG format alone does not meet that requirement. + +At the inspected Planner revision +[`e7fdb2492c42c9f5b34760706a5162aa586d3025`](https://github.com/ProjectASAP/ASAPPlanner/tree/e7fdb2492c42c9f5b34760706a5162aa586d3025), +plain materialization and lifecycle-aware selection/materialization are separate +library operations. `materialize_with_summary_maintenance_lifecycles` attaches +state deployments; `export_summary_maintenance_plan` exports their decisions, +alternatives and costs alongside the graph. Thus, the existence of an exported +DAG does not certify that lifecycle selection or complete deployment costing ran. +This observation does not imply that the backend's pinned Planner revision +already exposes every API from that revision. + +### Current public API audit + +The following describes the inspected Planner revision above, rather than the +proposed facade. These are library operations, not equivalent end-user workflows. +See [replacement APIs](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/asap-aware-mapping/src/replacement.rs), +[lifecycle APIs](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/asap-aware-mapping/src/summary_maintenance_lifecycle.rs), +[workload types](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/types/src/workload.rs), and +[cost model](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/asap-aware-mapping/src/cost_model.rs). + +| Operation | Input and output | What it does not establish by itself | +| --- | --- | --- | +| `search_workload` / `search_workload_with` | Canonical roots, default/explicit strategies -> `PlanSpace` | A selected deployment, workload lifecycle, or application-specific end-to-end accuracy target | +| `search_workload_with_targets` | Roots, strategies, per-root targets and accuracy model -> target-checked candidate space | Physical feasibility, lifecycle commitment or measured deployment cost | +| `PlanSpace::global_selection` | Candidate space and cost model -> structural `GlobalSelection` | Recurrence-aware or lifecycle-aware selection | +| `global_selection_with_recurrence` | Candidate space, cost model, recurrence profiles and optional horizon -> selection/error | Selected state lifecycle commitments | +| `global_selection_with_summary_maintenance_lifecycles` | Candidate space, workload/root associations, time, horizon, capabilities and cost model -> selection/error | Successful physical installation or ready state | +| `GlobalSelection::materialize` | A selected target -> optional semantic summary root/error | Executed summary data or a lifecycle deployment record; “materialize” here constructs IR | +| `plan_summary_maintenance_lifecycles` | An already materialized root plus demand/context -> lifecycle plan/error | Re-ranking all original semantic alternatives | +| `materialize_with_summary_maintenance_lifecycles` | Selection, target and lifecycle context -> optional lifecycle plan/error | A backend physical publication; callers must inspect decisions and available evidence | +| `export_summary_maintenance_plan` | Lifecycle plan -> serializable graph plus deployment/cost information | Any additional optimization, validation or runtime execution | + +A late lifecycle pass can evaluate a fixed root but does not retroactively make +an earlier structural selection lifecycle-optimal. A deployment flow must include +lifecycle feasibility/costs before its final candidate commitment. Likewise, +constructing `QueryRequirements` is not enough if a caller then invokes a low-level +search function that never receives those requirements. The orchestrator must +thread per-root targets into the target-aware path. + +Concrete defaults have different meanings: + +| Current Rust default/omission | Actual behavior | Consequence for integration | +| --- | --- | --- | +| `QueryRequirements::default()` | Implicit exact accuracy; unspecified response latency | Approximation requires explicit permission; no response-time bound is supplied | +| `DataWorkload::default()` | Unknown arrival and unknown evidence values | Does not assume data at rest, zero updates or a measured distribution | +| `Evidence::default()` | No value; unknown source | Missing/freshness-invalid evidence cannot establish a cost or empirical guarantee | +| `SummaryMaintenanceLifecycleCapabilities::default()` | All four runtime lifecycle flags true | This is not capability detection; adapters must pass truthful support explicitly | +| Default per-summary maintenance capabilities | Incremental update, merge and delete flags false | Runtime lifecycle support does not imply the algorithm/state representation supports its required operations | +| Default lifecycle cost inputs | All primitive costs unknown | Default structural costing does not supply a fully costed lifecycle deployment | +| Lifecycle horizon `None` | Horizon-dependent alternatives remain unselectable | One-time/rate comparisons cannot assume an arbitrary amortization horizon | +| `search_workload()` | Built-in strategies and `DefaultCostModel` | Useful for candidate exploration; ranking is not calibrated to the target deployment | + +`DefaultCostModel` preserves built-in algorithm order/sizing and uses structural +cost hooks. Custom models and evidence must be supplied for deployment-specific +claims, including candidate generation where strategies consume them, not only +for a final sort. Rust `Default` implementations are not automatically JSON/YAML +omission defaults: several required fields have no `serde(default)`. API/adapter +normalization must document serialized omission behavior separately. + +### Who controls what + +Application users control query meaning, permitted approximation, workload intent, +and any latency/resource objectives. They should not select internal passes or +assert unsupported runtime capabilities. An explicit application profile can +supply documented defaults, but normalization must report them. + +Runtime integrators supply source/type binding, capabilities, available lifecycle +actions, current state inventory, measured cost/evidence providers and the planning +time/horizon policy. They implement physical lowering and execution. Planner +extension developers supply replacement strategies, cost/accuracy models and +capability implementations. Restricting strategies narrows search opportunity; +it must not bypass semantic/accuracy checks. These are distinct control surfaces, +not a requirement for every user to configure every library parameter. + +### User guide for entry points, exit points, and controls + +The API audit above is architecture evidence, not a replacement for a Planner +user guide. Partial workflows are legitimate uses: a frontend author may need only +pre-ASAP IR, a strategy author may inspect candidate alternatives, and an embedding +application may consume a selected semantic DAG. None must invoke deployment +planning merely to make its intermediate output useful. + +ASAPPlanner should own a user guide organized by intended result, with one worked +example for each supported path: + +| User intent / exit artifact | What the guide must establish | | --- | --- | -| Can the producer build/update the selected family and parameters? | A readout implementation alone does not make a state maintainable | -| Can storage and readout preserve the selected windows and labels? | A tumbling-only path cannot silently implement arbitrary sliding coverage | -| Are merge operations and full/delta encodings compatible? | Distributed producers must construct the same logical state | -| Can the runtime perform every exact operator after readout? | A supported sketch is insufficient for an unsupported full expression | -| Can readiness, staleness and exact fallback be enforced? | Mathematical legality does not establish runtime answerability | - -Costs include initialization, ingestion updates, overlapping/retained state, -transmission, storage, merges, readouts, recurring queries, and shared producer -construction once. Compare alternatives over the same data and demand scope. -Missing evidence is not zero cost; stale or incomplete implementation evidence -cannot justify selection. - -Runtime observations reference the concrete binding and selected semantic -producer. Physical controls may vary only within already-authorized -guardrails. Changing grouping, family, parameters, windows or sharing returns -to planning. - -## Reuse across various ASAP workload scenarios - -| Scenario | Reusable Planner strategy | Application-specific responsibility | +| Parse and bind a workload into pre-ASAP IR | Supported frontend entry point, source/schema inputs, normalization and semantic checks actually performed | +| Generate post-ASAP candidates | Input IR, strategy configuration, automatically added passes, models consulted during generation, and candidate/rejection output | +| Rank/select and materialize a semantic DAG | Applicable cost/accuracy models, legality checks, selection scope and assumptions; distinguish structural from recurrence-aware selection | +| Plan summary lifecycles | Runtime and per-family capabilities, workload/time evidence, fixed versus searched lifecycle choices, and deployment commitments returned | +| Export a result | Which export preserves which decisions/evidence, schema version and what serialization does not validate | +| Compile and deploy in ASAPQuery | The handoff to the separate physical compiler and its completeness requirements; not another Planner execution API | + +For each path, document exact callable APIs at a supported revision, required and +optional inputs, Rust versus serialized defaults, customization points, returned +artifacts, checks performed, checks not performed, and valid next steps. Include +examples that stop at that exit point. Do not present every technically callable +combination as a supported workflow or infer guarantees from a type's name. + +Explain four separate control surfaces: optimization strategy policy (which +alternatives to explore), model/evidence providers (how to estimate and compare), +runtime capabilities (what is executable), and requirements (what is acceptable). +Strategy configuration must disclose automatically applied behavior: the current +`search_workload_with` also derives workload-dependent rollup internally, so its +explicit strategy list is not a complete enable/disable switch. Models used during +candidate generation must be distinguished from models supplied only at selection. +Disabling an optimization narrows opportunities; it does not disable correctness +checks or relax requirements. Missing evidence must remain explicit. + +Document today's composable APIs first. A unified application facade is a separate +interface improvement, not a prerequisite for explaining existing entry/exit +points. Its eventual explain output should identify effective strategies, automatic +passes, model versions, resolved defaults, unsupported choices, and rejected +candidates. Keep the current API reference, user recipes, and proposed facade +clearly separated so a design proposal is never mistaken for runnable guidance. + +### One supported application workflow + +For this backend, the target is one application-facing deployment-planning +request/result contract. This is a proposed orchestration boundary, not an +existing new Planner API. Its orchestrator +normalizes inputs, enumerates semantic and lifecycle alternatives, obtains physical +feasibility/cost evidence, validates guarantees, and returns the selected decision +with its evidence. Callers should not need to assemble those stages manually. +Planner supplies reusable semantic search, legality and ranking; the backend +owns application orchestration, physical evaluation and deployment commitment. +Planner's primary output remains `PlanSpace` plus ranked candidates, as defined +in its [design overview](https://github.com/ProjectASAP/ASAPPlanner/blob/main/docs/design_docs/README.md). +The downstream system may feed complete physical evidence back into Planner and +use `global_selection*` as a compatible-choice helper. A selected decision is +required at the physical compilation boundary, not at every legitimate Planner +exit point. Publication remains a separate backend operation, not a side effect +of invoking Planner. The user-facing entry/exit guide is tracked separately in +[Planner PR #440](https://github.com/ProjectASAP/ASAPPlanner/pull/440). + +Required stages are semantic normalization/validation, constraint checking, +capability filtering, and recording a complete selected decision (including +applicable lifecycle). Alternative search and ranking can collapse to validation +when only one candidate is legal. Empirical evidence, extra rewrite strategies, +and inventory reuse can be omitted only with the documented reduction in search +or guarantees. Serialization is needed only at a process/persistence boundary. + +Low-level APIs may remain available for research, candidate inspection and tests. +Their intermediate results must be distinguished from a complete planning result +and rejected by the production compilation boundary when commitments are missing. +This is one supported deployment workflow with explicit diagnostics, not several +undocumented combinations of optional optimization passes. + +### Inputs and omission rules + +The following are target normalization rules. They do not document current Rust +field defaults, which must be audited during migration. Every resolved default, +its source, and its effect on the available alternatives must appear in diagnostics. + +| Input | Supplied by | Requirement and consequence of omission | | --- | --- | --- | -| Repeated dashboards (MetricsObservabilityQuery) | Shared aggregates and prepared/maintained state | PromQL semantics, freshness and serving | -| Multiple dashboard resolutions | Legal rollup and window alternatives | Compatible retention and exact time coverage | -| Distributed telemetry aggregation | Mergeable summary and grouping alternatives | Collector placement, transmission and activation | -| DQC analytical workloads | CSE, aggregate fusion and rollup | DQC engine adapters and batch execution policy | - -General semantic rules belong in Planner. Backend-local metric-name fixtures, -SID lookup or deployment-specific placement must not become universal Planner -rules. No dependency on DQC is needed to reuse strategies contributed by it. - -## High-level migration - -See the [migration delivery plan](asapplanner-migration-plan.md) for PR-sized -implementation slices, dependencies, regression fixtures and completion gates. - -| Milestone | System outcome | Acceptance | +| Query roots and resolved source/type semantics | Caller/frontend | Required; ambiguous source or type information is an error | +| Accuracy requirement and evaluation scope | Caller or named application profile | Must resolve explicitly; omission grants no permission for approximate answers. A profile may specify exactness as its default | +| Query demand: one-time/repeating/unknown, cadence and time scope | Caller/workload registry | Required for workload-dependent decisions; unknown demand cannot be treated as zero demand or assumed future reuse | +| Optimization horizon | Caller or explicit profile | Required when comparing one-time costs with rates or amortized reuse; absent horizon prevents those comparisons, not semantic DAG inspection | +| Data arrival/update facts and cost evidence | Deployment evidence provider | Required for affected lifecycle/cost comparisons; missing evidence cannot be priced as zero or infer continuous ingestion from repeating queries | +| Runtime capabilities and allowed lifecycle actions | Physical provider | Required for a deployment candidate; absence cannot mean universal support | +| Existing summary inventory | Runtime/provider | Optional for considering new construction; omission means no existing-state reuse may be assumed | +| Empirical distribution/accuracy evidence | Optional evidence provider | Without it, consider only alternatives justified by available theoretical guarantees and costs; do not invent an empirical fit | +| Latency/resource limits | Caller or profile | Omission establishes no numerical bound or compliance claim; runtime feasibility checks still apply | + +The user controls workload intent and requirements. Runtime capabilities and +observed evidence are supplied by their authoritative providers, not arbitrary +user overrides. An unavailable optional optimization may reduce the candidate +set; an unavailable required guarantee or deployment fact yields an explicit +incomplete/infeasible result. No omission silently weakens correctness. + +### Output and lifecycle obligations + +A complete selected result includes the semantic DAG and query roots, stable +references to shared summary producers, a lifecycle commitment for each stateful +materialization, declared guarantees/assumptions, capability and cost evidence +references, normalized input/default diagnostics, and structured rejection reasons +for relevant alternatives. These may be separate typed fields in one result; +do not overload the semantic DAG with placement or wire-delivery configuration. + +Lifecycle completeness specifies whether state is built on demand, prepared, +reused, or maintained, together with its maintenance mode, evaluation schedule, +and output representation. Every stateful deployment needs this commitment. +Planner models possible lifecycles; it does not require every runtime to +implement them. For a particular deployment, the candidate set is the intersection +of modeled lifecycles, runtime capabilities, workload legality, and application +policy. Unsupported modes are excluded before ranking, not merely assigned a +higher cost. An application profile may further restrict runtime support but +cannot grant capabilities the runtime lacks. + +For example, a backend may support only building a summary directly from data at +rest, with no incremental maintenance. Planner then considers only compatible +direct-build alternatives. It cannot select continuously maintained incremental +state, even for a recurring query. Recurrence may justify repeated full builds, +but does not create an incremental-update capability. Prepared or retained reuse +is eligible only if the runtime separately supports those actions and the workload +permits them; direct-build support alone does not imply either. + +If these constraints leave one legal lifecycle, selection is degenerate: validate +and record that commitment, without searching other lifecycle modes. This remains +a complete lifecycle decision, not an incomplete plan. Build/update mode, execution +schedule, and retention/reuse are distinct dimensions, so direct build alone does +not specify the whole lifecycle. The result records the applicable choices and +assumptions; required cost comparisons use only eligible alternatives. An empty +candidate set produces an explicit infeasible result or a separately supported +raw-execution alternative. A stateless or selected raw-recomputation path can mark +state lifecycle as not applicable. Neither case means an unresolved stateful DAG +is deployable. + +Lifecycle choices affect cost ranking and phase legality, so they must participate +before final selection; attaching an arbitrary lifecycle after choosing a winner +cannot establish that the winner is feasible or cost-preferred. A diagnostic DAG +without this step promises only the checks actually performed. It does not promise +state readiness, maintenance cost, deployment feasibility, or an optimized lifecycle. +Even a complete Planner result is not an installed physical publication: the +compiler must preserve its commitments and validate all runtime projections. + +Acceptance for #438 requires a documented input/default matrix, one supported +application workflow, and examples for a one-shot query, a recurring query, an +unknown-demand request, a data-at-rest-only runtime with a singleton legal +lifecycle, and a missing-cost/capability case. Each example must show +the returned status, decisions, omissions and guarantees. Compilation must reject +an unresolved lifecycle for stateful deployment. No new facade is claimed to +exist until these examples exercise the actual public API. + +## Bind once, project four plans + +Use a compiler-internal common binding structure to record: + +- Semantic node to physical task mappings, including expansion into multiple tasks. +- Summary definitions, producer partitions, state schemas, window implementations, + and storage/input/output bindings. +- Data-flow edges with their endpoints and transmission requirements. +- Selected production and transmission policies with guarantee evidence. + +This structure addresses repeated decisions currently inferred from a backend +plan. It is not a fifth public plan or a second optimizer IR. Preserve semantic +node provenance and shared producers; one physical producer can serve multiple +query roots without inheriting a particular query's identity. + +| Projection | Responsibility | Principal contents | | --- | --- | --- | -| 1. Audit the shared contract and entry points | Current canonical compilation and compatibility paths have explicit ownership | Document supported operators, sharing scope, profile limits and true IR gaps | -| 2. Complete one workload-wide semantic path | Registered queries use Planner alternatives with preserved shared producers | The two-query example has one selected producer and both result roots | -| 3. Preserve bindings through all projections | Precompute, storage and serving implement the same selected decision | No duplicate maintenance; exact state/schema/window and generation agreement | -| 4. Consolidate reusable strategies | Missing general fusion/rollup rules extend Planner | Rules work without backend metric names, SID objects or placement assumptions | -| 5. Close capability and cost feedback | Only fully executable, properly costed alternatives are committed | Unsupported or stale evidence fails closed; estimated and observed costs are traceable | -| 6. Validate profiles and retire redundant selection paths | Serving executes installed bindings without independent semantic planning | Prometheus parity, sharing, readiness, fallback and activation-failure tests pass | -| 7. Broaden coverage (ProjectASAP-wide; not required for this repository) | Other applications, engines, sketches and lifecycles reuse the contract | Each participating provider demonstrates capability and semantic conformance | - -The first milestone demonstration should use backend-local ingestion and the -exact two-query example. Distributed rollout follows the same contract with -additional producer and activation checks. Existing paths may remain as -comparison baselines until parity is established; remove duplicate semantic -selection, not necessary physical plans or profile-specific runtime adapters. - -Step 7 is an ecosystem extension, not a prerequisite for completing this -backend's scoped consolidation through steps 1–6. - -## Related contracts and implementation guides - -- [Physical compiler](../developer_docs/control-plane/physical-compiler.md) -- [Plan publication](../developer_docs/control-plane/plan-publication.md) -- [Catalog-backed physical-plan runtime](../developer_docs/query-engine/catalog-physical-plan-runtime.md) -- [ASAPQuery compatibility profile](asapquery-compatibility-profile.md) -- [Runtime accuracy feedback](../developer_docs/control-plane/runtime-accuracy-feedback.md) +| CollectorPlan | Execute maintenance assigned to an edge target | Inputs, maintenance tasks, producer/partition identity, window implementation, production policy, output bindings | +| PrecomputePlan | Execute backend maintenance and manage state | Raw-input build, remote-state integration, derived summaries, storage, retention and recovery bindings | +| TransmissionPlan | Deliver state across execution locations | Producer/consumer endpoints, schema, encoding, frame semantics, sequence/epoch, checkpoints, cadence and recovery policy | +| QueryPlan | Read and compose results | State bindings, merge/readout, exact residuals, window boundary handling, completeness requirements and fallback | + +CollectorPlan and PrecomputePlan may use the same maintenance operator contract +with different executors. They do not need one shared scheduler or implementation. +Derive TransmissionPlan from remote data-flow edges, not from PrecomputePlan. +Initially support the existing Collector-to-backend edges; a backend-local +profile has no remote-summary transmission rules and requires no Collector. +Raw Remote Write ingestion remains an input adapter, not a fabricated summary flow. + +Build the catalog and common bindings before projecting runtime plans. Each plan +references immutable catalog definitions instead of independently choosing +algorithm, population, or logical window. Concrete pane layouts and execution +bindings remain physical choices constrained by those definitions. + +A self-contained Collector installation artifact can embed the relevant catalog +subset and transmission rules. These are mechanically derived copies from one +publication, validated against its identity/digest. They are not independently +editable authorities. Runtimes need no catalog network lookup on each update. + +## SDS, state codecs, and transmission + +| Contract | Authority | +| --- | --- | +| SDS descriptors and catalog | Meaning, source/population, fidelity, logical definition and compatible state schema | +| SDS instance/inventory | Concrete extent, groups, provenance, completeness, lifecycle and opaque state reference | +| TransmissionPlan | Authorized state flow between endpoints and its delivery/application rules | +| Sketch library codec | Full-state/delta byte representation, reconstruction and supported state operations | +| Runtime | Scheduling, durable admission/application, storage and serving | + +An envelope is not the entire SDS model. Keep payload bytes out of the desired +catalog and observed metadata inventory. Sketch payload schemas remain owned by +the sketch libraries; runtime contracts reference them rather than creating a +second copy. Exact aggregate state also needs an explicit versioned schema. + +The target package boundary separates lightweight semantic IR contracts, +runtime contracts, sketch libraries, the physical compiler, and executors. +Runtime contracts contain catalog, plan, publication, and frame contracts and +may use lightweight shared semantic types. They depend on neither optimizer, +Collector runtime, nor backend runtime. Go/Rust bindings must come from an +explicit schema authority, with cross-language fixtures where generation cannot +express semantic validation. Package extraction precedes any new repository. + +Move reusable reconstruction from Collector wrappers into sketch-library APIs. +Backend accumulators retain query-specific conversion but consume typed decoded +state, avoiding KLL's reconstruction/serialization/decoding detour. Supported +legacy bare-state reads remain until an explicit retirement gate. Consolidate +remaining codecs per family; the first extraction must not claim new parity for +HLL, CountSketch, or CountMinSketch. + +### Identity and update application + +Keep semantic node identity, SummaryDefinitionId, producer/partition identity, +concrete instance/physical storage lifetime, publication generation, and frame +sequence/checkpoint identity distinct. Moving a producer or changing cadence +need not change the logical definition, but does require an authorized deployment +transition. Reuse of state across generations requires explicit compatibility. + +Every remote state flow must specify: + +- Schema/codec and supported full/delta operations, including coverage/group keys. +- Producer partition and epoch, sequence scope, and replay/conflict behavior. +- Whether full state replaces a producer contribution or represents a distinct, + immutable contribution; how deltas reference and advance a checkpoint. +- Recovery after a gap, unknown checkpoint, restart, or incompatible generation. + +A full snapshot of an existing producer contribution cannot be merged into the +global result again as new observations. A receiver must replace/rebuild that +contribution using supported operations, or reject the unsupported update model. +Mergeable sketches are not necessarily subtractable. A delta is applicable only +to its authorized base; missing bases trigger resynchronization, not bare-state +fallback. A malformed framed payload must not evade validation through a legacy +unframed decoder. Duplicate/conflicting-frame decisions must be consistent with +state publication after failure; durable replay guarantees require durable +receipts or an equivalent reconstructable checkpoint protocol. + +These are target requirements. The initial migration preserves current wire +behavior and records any unmet requirement as a capability gap, rather than +changing full/delta semantics under an existing version. + +## Production, transmission, and query guarantees + +Split the responsibilities currently grouped in `RuntimeRulePolicy`: + +- Production policy controls sampling/admission and estimator semantics that + affect state construction. It is projected to the runtime producing that state. +- Transmission policy controls delta suppression, GOS where supported, emission + cadence, and full checkpoints. It is projected to both endpoints as needed. + +The compiler chooses these policies jointly and validates the resulting query +guarantee. Sketch error, sampling error, transport staleness, and incomplete +coverage are different quantities; they cannot be combined by an unconditional +sum of epsilons. State the estimator, assumptions, probability/evaluation scope, +and composition rule. Unknown evidence cannot establish a numerical guarantee. +A query guarantee shared across many outputs/evaluations must cover that declared +scope; shared state does not make errors independent. + +Changing sampling semantics requires guarantee and state-compatibility review. +A cadence-only change can retain the semantic definition but still needs an +accepted successor publication. Adaptation is bounded by installed policy and +fresh scoped evidence; it must not mutate an immutable generation in place. + +## Publication, activation, and readiness + +Keep `PhysicalPlanPublication` as the canonical artifact, rather than adding +another bundle format. Give each publication an unambiguous version/content +identity covering its plans and catalog references. A catalog digest alone does +not identify a change to transmission policy or physical placement. + +Use one shared cross-plan validation implementation at compilation and install +boundaries. Runtime-specific preparation still checks actual local resources. +Validate producer/consumer coverage, catalog references, schemas, window phase, +layout, supported codecs, selected policy guarantees, and query state bindings. + +Distributed rollout must account for partial failure: + +1. Validate and stage each required target; acknowledgements identify the exact + publication and target projection, not merely receipt of a message. +2. Prepare receivers before permitting new-generation producers to emit. Persist + the activation decision or use an explicit recoverable coordination protocol. +3. Switch each backend's local active snapshot atomically. Queries pin one + generation; a local pointer swap is not a distributed atomic commit. +4. Fence in-flight frames by generation. Accept an older frame only through an + explicitly retained compatible path; otherwise reject/resynchronize it. +5. On failure before activation, discard staged resources and retain the previous + generation. After partial activation, reconcile or publish a coordinated + successor; do not assume rolling back one process restores the whole system. + +Activation permits execution; it does not prove complete source coverage, warmed +state, or durable recovery. Readiness is derived from observed instances, +watermarks/completion proofs where supported, and pending admitted work. +[Completeness](continuous-summary-completeness.md) and the SDS lifecycle rules +remain required. The design does not assume that live Remote Write supplies +source watermarks or that existing runtimes implement global exactly-once delivery. + +## End-to-end examples and acceptance + +**Backend-local:** select a supported semantic summary and readout, bind its +maintenance to backend ingestion and its query to local state. Publish no +Collector targets or remote-summary rules. Exact fallback remains available +until the required coverage is ready. Where Planner authorizes two readouts +sharing one state, maintain it once per compatible input partition and generation. + +**Distributed:** two quantile queries over the same population, parameters and +window share a Collector sketch producer. The compiler emits one producer, +its remote-state rule, a backend integration binding, and two query readouts. +Sequence/checkpoint validation precedes state publication. Replaying a frame +must not increase the observation count. A failed target stage must not expose +new query bindings. Multiple producers require disjoint or explicitly accounted +input coverage; matching descriptor IDs alone do not prove safe merging. + +These examples define required fixtures, not new claims of implemented coverage. +Acceptance must exercise the actual supported Collector producer/decoder and +backend install/ingest/query boundaries, including Go/Rust interoperability. + +| Gate | Observable evidence | +| --- | --- | +| Semantic preservation | Selected node/root provenance survives all projections; incompatible grouping/window/lifecycle choices fail before publication | +| Shared production | N admitted observations cause N producer updates per intended partition, not N multiplied by consumer queries | +| Protocol conformance | Full, delta, duplicate, conflict, gap, epoch restart, unknown-base and legacy fixtures have explicit expected outcomes | +| State readiness | Missing or pending coverage uses configured fallback/unavailability; installation never certifies completeness | +| Generation transition | Failed stage, partial activation, delayed old frames and restart cannot mix query generations or double-apply state | +| Package boundary | Backend production dependencies exclude Collector execution runtime; protocol packages exclude optimizer/executor dependencies | +| Extension | Adding a codec uses one schema authority and endpoint capability registration, with no new plan-specific semantic definition | + +Test expectations should be specified before extraction. A reviewer other than +the implementation author should review protocol and rollout cases; this document +has not undergone independent review and reports no new executable test results. + +## Alternatives, quality attributes, and risks + +Keeping PrecomputePlan as the master representation is initially simpler but +makes edge and transport decisions depend on backend configuration. A small +internal binding stage resolves this without a new public IR. Independently +compiling four plans requires reconciliation after potentially different choices +and is rejected. A universal runtime would unnecessarily couple edge scheduling, +backend storage and query execution; share contracts/codecs instead. + +A new all-encompassing protocol repository does not resolve authority by itself. +First extract lightweight packages with one schema owner, then choose repository +placement and release tooling. Moving all of `asap_types` would also move Planner +and application coupling, so it is not the extraction unit. + +Maintainability is checked by the dependency graph and schema ownership audit. +Debuggability requires tracing a query root through semantic node, definition, +producer/partition, publication and checkpoint; validation reports the conflicting +identities and expected/actual contracts. Track staged/active versions, readiness, +frame rejection/resync counts, duplicate handling, and fallback reasons. Avoid +unbounded per-series metric labels; use structured diagnostic records for detail. + +Performance targets preserve current hot-path behavior: resolve catalog references +at installation, avoid network lookups per update, and remove redundant KLL byte +round trips. Measure compile/install time, payload size, ingest cost and retained +producer-state memory before and after; no speedup is assumed without evidence. +Only authenticated, authorized installation paths may grant producer/flow rights; +payload-provided identifiers do not authorize catalog or policy changes. + +The largest risks are codec drift, loss of provenance during binding extraction, +non-invertible sketch replacement, and partial rollout. Versioned adapters and +per-profile acceptance gates limit the rollout scope. Timeline estimates require +fixture and capability inventory first; intermediate success is unchanged wire +output from the new compiler structure, final success is both profiles passing +acceptance with the Collector dependency removed. + +Open implementation decisions are the contract schema/binding-generation tool, +publication identity encoding, durable coordinator mechanism, and supported +per-family replacement/recovery model. These must be resolved at their migration +gates; they do not justify enabling unsupported capabilities. Repository placement +can remain unchanged throughout the initial extraction. + +## Related documents + +- [Migration delivery plan](asapplanner-migration-plan.md) +- [Summary Catalog and SDS](summary-catalog-sds-architecture.md) +- [Physical compiler implementation](../developer_docs/control-plane/physical-compiler.md) +- [Plan publication implementation](../developer_docs/control-plane/plan-publication.md) +- [Catalog-backed runtime](../developer_docs/query-engine/catalog-physical-plan-runtime.md) +- [Compatibility profile](asapquery-compatibility-profile.md) diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index babe652cc..dba63acf7 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -1,313 +1,184 @@ -# ASAPPlanner integration: migration delivery plan - -Status: implementation sequence for the -[system architecture proposal](asapplanner-integration.md). A checked milestone -requires executable evidence; publishing this plan or opening a PR does not -complete migration. - -## Baseline and completion definition - -The inspected baseline is backend `95131d83972bb7a07d338e2a5af925a20c15ddce`. -The compiler already deduplicates backend PrecomputePlan state by physical -fingerprint and binds QueryPlan leaves explicitly. It still builds Collector -materialization declarations per query, and lifecycle selection builds a -single-query demand. Therefore, do not describe all sharing as absent, or -treat existing fingerprint deduplication as workload-wide optimization. - -Migration is complete for a declared supported workload/profile when: - -- one Planner-authorized semantic decision governs all result roots; -- compatible shared producers have one physical maintenance path per source - partition and generation; -- unsupported sharing or operators are rejected or explicitly fall back; -- activation, readiness, query execution and feedback refer to matching - bindings and generations; -- supported entry points no longer independently select a different summary; -- parity and producer-update tests pass for the promised deployment profile. - -Backend-local and distributed profiles have separate acceptance evidence. -Neither arbitrary PromQL coverage nor ProjectASAP-wide engine coverage is a -completion prerequisite. - -## Delivery sequence and dependencies - -Implementation tracking (PRs are not merged automatically): - -| PR | Implemented scope | -| --- | --- | -| [Backend #513](https://github.com/ProjectASAP/ASAPQuery-backend/pull/513) | A: compatible physical producer deduplication and conflicting deployment-contract rejection | -| [Backend #514](https://github.com/ProjectASAP/ASAPQuery-backend/pull/514) | B prerequisite: port the backend from its divergent historical pin to merged Planner APIs, including typed summary inputs | -| [Planner #356](https://github.com/ProjectASAP/ASAPPlanner/pull/356) | B: reusable, scope-local typed post-ASAP subtree interning; includes schemas and guarantees in equivalence | -| [Backend #515](https://github.com/ProjectASAP/ASAPQuery-backend/pull/515) | B: workload search, shared producer bindings and persistent query-root mapping | -| [Backend #516](https://github.com/ProjectASAP/ASAPQuery-backend/pull/516) | C: backend-local packed SUM/observation-count state, exact readouts, additive reductions and constrained arithmetic; production HTTP acceptance | -| [Backend #517](https://github.com/ProjectASAP/ASAPQuery-backend/pull/517) | E: current distributed publication/frame protocol, actual Collector validator, two shared readouts, failed staging and inactive-generation rejection | -| [Backend #518](https://github.com/ProjectASAP/ASAPQuery-backend/pull/518) | B/F: one workload-selection adapter for canonical startup and compile-and-publish; query-scoped accuracy certificates | -| [Backend #519](https://github.com/ProjectASAP/ASAPQuery-backend/pull/519) | D component: joint producer lifecycle demand, incompatible-evidence rejection and identity-keyed lifecycle estimates | -| [Backend #520](https://github.com/ProjectASAP/ASAPQuery-backend/pull/520) | E: published config drives the actual Collector Rust update/window/emission loop; N raw observations yield N updates and one shared output | -| [Backend #521](https://github.com/ProjectASAP/ASAPQuery-backend/pull/521) | E: failed staging cleanup permits retry; concurrent readers survive successful same-semantic generation cutover; retired frames are rejected | -| [Backend #522](https://github.com/ProjectASAP/ASAPQuery-backend/pull/522) | D: provider-priced complete bound-workload selection, strict v2 startup evidence, read-only quote preparation, live publication/reporting and process acceptance | - -The backend PRs form a sequential review stack from #513 through #522; -#515 uses merged Planner #356 at revision -`378a7547ede629a64e84c9f7c810226ce196cce9`. #516 includes the fail-closed -arithmetic regression fix, propagated through its dependent branches. -The backend-local dashboard and distributed single-partition quantile examples -have executable acceptance evidence, including complete cost-based selection -and same-semantic generation cutover. The supported-profile implementation -is in the review stack, not yet merged or deployed. Production calibration, -platform-specific rollout and broader semantic workload replacement are not -claimed complete by these fixtures. - -Local verification of the original combined migration stack: 654 control-plane -library tests, 28 control-plane binary tests, one control-plane integration -test, 977 data-plane library tests and three production-process tests passed. Planner -#356 passed its 156 type-library tests and GitHub formatting/lint/test checks. -The backend process tests cover the actual binaries and Collector Rust library, -not production traffic or every Collector platform adapter. Local passes do -not replace PR CI, review or the remaining migration gates. - -| Slice | Repository | Depends on | Deliverable and acceptance | +# Physical-plan architecture: migration delivery plan + +Audience: developers implementing the +[integration architecture](asapplanner-integration.md). Status: proposed delivery +sequence, not a record of completed implementation. This replaces historical PR +stack tracking with behavior-based gates. Existing merged behavior is the baseline; +old test totals and PR status are not evidence for this migration. + +## Completion definition + +For each declared supported deployment profile, one Planner decision is bound +once and projected into catalog, QueryPlan, PrecomputePlan, CollectorPlan and +TransmissionPlan. Publication, runtime state, and query readout agree on identity, +schema, window, guarantees and generation. Backend production code no longer +imports the Collector execution runtime for reconstruction. + +Backend-local and distributed profiles need separate acceptance. Arbitrary +PromQL, all sketch-family delta modes, general multi-hop execution, and a new +repository are outside the completion gate. Preserve supported existing behavior; +record unsupported combinations as capabilities rather than broadening claims. + +## Sequence and dependencies + +| Stage | Owner | Deliverable | Exit gate | | --- | --- | --- | --- | -| A. Safe physical state sharing | ASAPQuery-backend | Existing compiler | Deduplicate Collector declarations for compatible state; reject conflicting implementation/layout/lifecycle contracts; keep both query roots bound to one backend state | -| B. Workload semantic planning adapter | ASAPQuery-backend, with Planner changes only for demonstrated gaps | A and Planner API audit | Batch registered canonical roots through reusable Planner search; preserve root mapping and producer identity; do not implement backend-local semantic CSE | -| C. Aggregate-state fusion and readouts | ASAPPlanner for rules; backend for execution | B | SUM/COUNT example with per-consumer projections, label/time equivalence and fully executable division; reuse existing decomposition/rollup rules | -| D. Workload-wide implementation evidence | ASAPQuery-backend and Planner evidence boundary | B; C for fused states | Compare complete alternatives with shared build/update cost once and per-consumer read costs; joint state lifecycle/implementation agreement | -| E. Bound execution and lifecycle acceptance | ASAPQuery-backend; Collector only where public runtime gaps require it | A–D | Producer update counts, readiness/fallback, generation isolation, failed rollout, and distributed projection tests | -| F. Compatibility-path retirement | ASAPQuery-backend | E for each affected profile | Route supported entry points through the validated path; remove duplicate selection only after call-site and parity audit | - -Slices are reviewable PR units, not an instruction to open empty placeholder -PRs. If a slice spans semantic changes and physical execution, split by -repository and stack the dependent PR explicitly. Do not merge automatically -or make one unverified pin bump cover unrelated Planner changes. - -## A. Safe physical state sharing - -The immediate regression fixture is two different quantile readouts over the -same source, parameters and window. It exercises existing supported operations -without depending on future SUM/COUNT fusion. - -Implementation scope: - -1. Compare concrete contracts when multiple selected leaves resolve to the - same physical fingerprint. Include algorithm/parameters, grouping, window - framework, implementation, pane layout and lifecycle. Runtime transmission - policies must also agree. -2. Emit one Collector producer declaration for a compatible shared state while - preserving every query's binding and readout. -3. Keep evidence conservative: differing evidence cannot silently disappear - during deduplication. A future certificate-union design is a separate step. -4. Reject conflicting contracts before any plan is published. Do not pick - whichever query happened to be visited first. - -> Historical note: this acceptance text predates the SummaryCatalog migration; -> the former BackendPlan state is now represented by a catalog materialization -> and its execution-plan references. - -Acceptance: both query roots exist; one catalog materialization and one PrecomputePlan -state exist; each Collector has one producer declaration; both bindings point -to that state. A different implementation/layout for the same fingerprint -fails compilation. Distinct source/window/parameters must remain distinct. - -This slice establishes deployment consistency, not workload search or a claim -that all query-time computations execute once across separate HTTP requests. - -## B. Workload semantic planning adapter - -Audit the pinned Planner workload/search APIs before defining another backend -plan representation. Inputs must preserve canonical query identity, source -selection, requirements, recurrence and time scope. - -The result must retain all original roots and shared logical producers. -Backend bindings must be keyed by workload-scoped producer identity, not only -a per-query pointer. Physical IDs stay downstream. Preserve explicit mappings -from each query root to its required materializations and fallback. - -Acceptance fixtures: - -- identical producers used by two different roots; -- a diamond within one query and sharing across queries; -- incompatible filters, grouping, windows or accuracy do not share; -- round-trip compilation retains roots and sharing; -- unsupported alternatives cannot become partially executable warm routes. - -Pointer sharing in memory alone is not persistent identity. A serialized -execution projection must preserve the relationship explicitly. - -## C. Aggregate-state fusion and complete readouts - -Use the system document's sample-weighted mean example as the target. -Planner owns the equivalence rule: union compatible SUM/COUNT states and -project the needed results to consumers. The backend owns physical state -implementations and exact output operators. - -First inspect existing AVG decomposition, CSE and rollup rules. Add only missing -semantics upstream; do not copy DQC transformation objects or hard-code metric -names in Planner. - -Acceptance includes uneven per-instance sample counts, missing/stale series, -multiple services, exact interval endpoints, range evaluation steps and -denominator edge cases. Query results must match Prometheus labels, timestamps -and numeric semantics. Until the whole expression is supported, preserve -explicit fallback rather than claiming partial integration. - -The implemented backend-local example uses one raw accumulator that retains -both sum and observation count. This is native physical packing of selected -Planner operations, not a new backend semantic rewrite. The process test has -two services: observations `[10]` and `[2, 4, 8]` across two API instances give -SUM = 24, COUNT = 4 and weighted mean = 6; worker observations `[9, 15]` give -SUM = 24, COUNT = 2 and mean = 12. Three registered consumers still configure -one producer; a Remote Write retry does not double the counts. Range steps, -output labels/timestamps and unaligned-window fallback are checked. - -Do not generalize that execution contract to `sum(sum_over_time(m) / -count_over_time(m))`: summing per-instance means cannot pool samples first. -Non-additive entity reduction, mismatched operand grouping/windows, shifted -selectors and unverified instantaneous/temporal combinations remain explicit -fallbacks. Unknown legacy observation counts also fail closed. Distributed -observation-count readout is not advertised by this implementation. - -## D. Workload-wide evidence and selection - -Today per-query lifecycle inputs are not proof of joint workload costing. -Aggregate demand for each shared producer while retaining consumer-specific -requirements. Compare alternatives over one horizon and data scope. - -Charge shared initialization and maintenance once, account for all consumer -readouts and live/retained state, and include applicable placement and -transmission costs. Feasibility checks cover the entire selected DAG, not -only a summary family. The winning evidence must resolve to the same concrete -implementation that compilation installs. - -Acceptance: a shared alternative wins when its complete cost is lower, loses -when retention/materialization overhead dominates, and is unavailable when -any required capability/evidence is absent or stale. Adding another consumer -must not double-count the producer's update stream. - -Implemented component: #519 gives each unique physical producer a -`WorkloadDemand` containing all its consuming query entries. For a 300-second -horizon, 100 updates/second and two consumers reading every 10 and 20 seconds, -the demand is 30,000 updates and 45 reads. With build = 10, update = 0.001, -read = 0.1, retention/second = 0.001 and retirement = 1, the lifecycle cost is -45.8. Adding the second consumer increases cost by 1.5, not another build and -update stream. Publication reports this component against the materialization -and implementation identities; it is not a complete-plan total. - -Implemented selection: #522 compares complete bound alternatives before -commitment. A provider prices source upkeep, each shared state's build/update/ -residency/retirement per location, transport, every reachable query operator, -and results over one common horizon. Query work is multiplied by recurrence; -shared maintenance is not multiplied by consumer count. Native exact fallback -includes its service's input upkeep as well as full native query execution. - -The default inventory is the Planner-selected continuously maintained workload -and its whole-workload exact alternative. The comparison interface also accepts -additional Planner-authorized, bindable forests; this is not exhaustive search -over all engines or lifecycle variants. Tests prove both the sharing win and -high-retention loss, and reject missing, stale, mismatched or infeasible quotes. - -Implementation refinement: pricing uses a flat coverage manifest over the -existing bound physical projection, not another semantic DAG. It does not -populate `PlannerPhysicalPlanProvider` with guessed source statistics or split -the older opaque per-query window scalar into fabricated components. Providers -must quote the actual source scope, state layout, implementation and capability -generation. The selected plan and report retain those identities. - -Version-2 canonical snapshots require complete evidence. Live requests can -obtain requirements from the read-only `cost-manifests` endpoint before -publication. Version 1 and live requests without quotes remain explicitly -uncosted compatibility paths. See the [provider workflow in #522](https://github.com/ProjectASAP/ASAPQuery-backend/blob/feat/complete-workload-cost-selection/docs/examples/workload-cost-evidence.md). - -Production calibration still requires evidence from the intended deployment; -the deterministic fixture costs are not production measurements. The provider -attests exact-backend access and resource feasibility; a low cost alone does -not establish either. - -## E. Runtime and deployment acceptance - -Start backend-local, then validate the distributed profile independently. - -- Replay deterministic raw samples through production ingestion. -- Count state creation and updates: one compatible producer per generation, - with no duplicated updates when a second query subscribes. -- Query both roots through HTTP and compare with an exact reference. -- Test incomplete coverage, stale state, absent routes and unavailable fallback. -- Stage a successor while requests run; each request observes one generation. -- Fail staging or producer acknowledgement and verify the active generation - remains unchanged. -- For distributed collection, decode emitted plans through the actual Collector - validator and assert one producer per source partition, not one producer - globally across independent sources. - -Unit-level declaration counts do not replace runtime update-count tests. - -Current evidence combines real backend executables with the actual Collector -Rust runtime library. The test's host adapter supplies OpAMP acknowledgements -and frame metadata; it does not launch a platform-specific Collector binary. -In #521, failed Collector staging is discarded without touching the active -snapshot; the same successor version can then be retried successfully while -queries run. Old-generation frames are rejected after cutover and successor -frames become queryable. #522 exercises this flow with costed publication. -This verifies same-semantic runtime generation replacement, not arbitrary -semantic workload replacement or a platform-specific production rollout. -Platform adapter rollout remains a deployment acceptance step. - -## F. Retire duplicate selection safely - -Inventory canonical startup compilation, explicit compile-and-publish, -legacy workload adapters and serving-time binding helpers. Distinguish dead -code from intentionally supported profiles using call-site inspection. - -For each path, either route it through the selected workload contract, retain -it as an explicitly unsupported/fallback adapter, or remove it after parity. -Parsing and canonicalization at serving time are fine; family/parameter, -grouping or lifecycle reselection is not. - -Do not remove QueryPlan, PrecomputePlan, physical deployment selection, -exact fallback, or profile-specific adapters merely because their types are -different from post-ASAP IR. - -Call-site audit: production instant/range serving already requires an active -physical QueryPlan and declines absent or unregistered routes. The old -summary-selection serving branches in `engine.rs` are `cfg(test)` fixtures. -#518 unifies the two first-class compilation entry points. Legacy flat-workload -demo/configuration adapters remain separate compatibility paths; they must not -be presented as migrated canonical-workload entry points or removed without -their own parity/retirement decision. - -## Existing PR coordination - -At the baseline inspection, open PRs -[#505](https://github.com/ProjectASAP/ASAPQuery-backend/pull/505), -[#506](https://github.com/ProjectASAP/ASAPQuery-backend/pull/506), -[#509](https://github.com/ProjectASAP/ASAPQuery-backend/pull/509) and -[#511](https://github.com/ProjectASAP/ASAPQuery-backend/pull/511) cover PromQL, -process-E2E and TopK-related work. Re-check their status and changed files -before touching overlapping paths. Their presence is not evidence that the -workload-sharing migration is complete. - -Review follow-up (2026-09-08): #505 is now stacked on #522 and uses the merged -Planner revision above. Typed TopK update weights belong to the selected -producer, not its readout. Its multi-series fixture distinguishes count ranking -(`api=4`) from value ranking (`worker=200`). #509 compares complete vectors at -each range step, including changing winners. #506 tests unregistered-query -fallback; it is not evidence that registered arithmetic is unsupported. - -#515 preserves duplicate algorithm candidates during cost ranking; removing -them violates Planner's candidate-multiset contract and can panic. #522 quote -preparation enumerates bindable alternatives without requiring the default -warm alternative to compile, so missing warm implementations do not hide an -available exact quote. Publication still requires a selected, validated plan. - -#511 retains evidence-aware legacy binding and preserves count update semantics -in emitted heap configuration. Its two heap TopK acceptance tests now use -registered `topk(3, count_over_time(top_endpoint_qps[5s]))`, a compiled physical -QueryPlan, and the production backend-local Remote Write path. Both CMS-with-heap -and CountSketch-with-heap return gamma=200, zeta=150 and alpha=100 over two -windows, with exact item identities, timestamps and retry deduplication checked. -Unregistered instantaneous TopK still follows the explicit exact fallback. -This replaces the two obsolete no-QueryPlan tests; it does not restore that -serving contract or claim migration of other legacy OTLP fixtures. - -The [architecture PR #512](https://github.com/ProjectASAP/ASAPQuery-backend/pull/512) -tracks the design and this delivery plan. Implementation PRs should report the -slice they complete, tests actually run, and remaining acceptance gaps. +| 1. Contract and behavior inventory | Backend, Collector, Planner maintainers | Authority map, supported capability matrix, cross-language fixtures | Every existing production wire path and plan entry point has an explicit compatibility expectation | +| 2. Common physical bindings | Backend control plane | Internal binding stage, catalog construction, four projections | Existing supported inputs produce semantically equivalent publications; no repeated selection through PrecomputePlan | +| 3. Shared contracts and validation | Backend/Collector; Planner for IR export | Lightweight contracts, typed semantic export, shared publication validation | Actual Go/Rust consumers accept matching artifacts and reject incompatible ones | +| 4. Policy and deployment boundaries | Compiler and runtimes | Production/transport policy split; explicit application and activation rules | Guarantee, checkpoint, readiness and partial-rollout fixtures pass for enabled modes | +| 5. Codec extraction | Sketch libraries, Collector, backend | Typed reconstruction APIs and consumer migration | Backend excludes `asap-precompute-rs`; supported decoding and query results remain compatible | +| 6. Retirement and release | Participating repositories | Remove superseded copies/adapters, pin compatible versions | Both profiles pass end-to-end gates without retired paths | + +Stages 2 and 3 preserve existing wire formats through boundary adapters. Stage 4 +changes public contracts only with negotiated/versioned compatibility. Codec work +can proceed after stage 1, but its removal gate depends on stable contracts and +consumer coverage. Do not combine an unrelated Planner upgrade with extraction. + +## 1. Establish authority and fixtures + +Inventory Planner exports, backend installed contracts, Collector Go/Rust DTOs, +OTel carriers, sketch state/delta schemas and legacy bare-state decoders. Record +one owner for each concept and the current supported producer/consumer versions. +Compare actual field shapes, defaults, enum meanings, units and rejection behavior; +a similarly named struct is not compatibility evidence. + +Capture supported backend-local, distributed full-state, distributed delta, and +generation-transition examples. Use distinct evidence for wire equivalence and +semantic state/readout equivalence; randomized state may require persisted fixtures +and semantic assertions rather than comparing unrelated fresh encodings. + +Protocol cases include duplicate/conflicting sequences, unknown delta base, gaps, +producer restart, malformed framed payload, and legacy unframed state. Label any +currently failing target invariant as migration work, not passing baseline behavior. +Have a separate reviewer review expected outcomes before protocol changes. + +### Planner caller contract gate (#438) + +Resolve [Planner #438](https://github.com/ProjectASAP/ASAPPlanner/issues/438) +at the public workflow boundary, not only in the backend compiler. First deliver +an ASAPPlanner user guide for supported entry/exit points, including workflows +that intentionally stop at pre-ASAP IR, candidates, or a selected semantic DAG. +Each recipe must document exact APIs, controls, defaults, performed checks and +output limitations and run against the documented revision. This guide does not +depend on implementing a unified facade. Document strategy selection and automatic +passes separately from model providers, runtime capabilities and requirements. Audit actual +API defaults and low-level output guarantees, including the current all-enabled +lifecycle capability default, unknown lifecycle cost inputs, and per-root accuracy +propagation. Distinguish Rust defaults from serialized-field omission. Then specify +one application-facing +request/result contract with explicit incomplete/infeasible outcomes. Use the +[caller contract](asapplanner-integration.md#caller-contract-and-lifecycle-completeness) +as the target; its omission rules are proposed behavior, not current API facts. + +The complete output must associate each materialized state with a selected or +capability-constrained, validated lifecycle. Planner models the available +lifecycle vocabulary; runtime support and workload/policy constraints determine +which modes may enter candidate selection. A singleton legal set is a complete +selection, not a skipped lifecycle decision. Lifecycle feasibility and applicable +costs must participate in candidate selection. Keep diagnostic DAG exports accessible, but +do not allow them to masquerade as deployment-complete results. Document which +inputs callers control and which evidence/capabilities come from providers. + +Gate: executable public-API examples cover one-shot, recurring, unknown-demand, +and missing-evidence inputs; diagnostics expose defaults and their consequences. +Include a backend that can build summaries only from data at rest: no incremental +mode may enter ranking, and a singleton legal lifecycle must produce a complete +commitment. Recurring demand must not imply incremental support or permission for +retained reuse. Verify that an empty legal set is reported explicitly. +The physical compiler rejects incomplete stateful commitments. Stages 2 and 3 +must preserve this distinction while existing lower-level APIs remain compatible. + +## 2. Refactor compilation without changing semantics + +Retain candidate selection and cost/capability evaluation. Introduce only a +compiler-local structure for selected tasks, definitions, state bindings and +producer/consumer edges. Construct the catalog from the selected definitions, +then project all four plans from those bindings. + +Remove the dependency of transmission compilation on PrecomputePlan. Preserve +shared producer identity across roots and reject incompatible physical bindings. +Target artifacts may embed catalog/rule subsets but must be derived from the +same publication. Compare old/new outputs with normalization only for explicitly +nondeterministic metadata; do not normalize away semantic or identity differences. + +Gate: supported profiles retain query results, window/label semantics, producer +update counts, configured fallback and publication compatibility. New binding +provenance makes every runtime task traceable to the selected decision. + +## 3. Extract contracts and unify validation + +Separate lightweight semantic IR export from Planner search internals. Preserve +node/operator/schema/guarantee meaning while migrating `OwnedPostAsapDag`; do not +replace typed semantic validation with arbitrary JSON acceptance. + +Extract SDS, installed plan, publication and frame contracts into packages that +import neither execution runtime nor optimizer. Select a schema authority and +binding-generation approach before removing manual Go/Rust copies. Keep sketch +payload schemas in their sketch-library authority. + +Use shared cross-plan validation at compile and install boundaries, followed by +local resource checks. Versioned legacy adapters normalize once at the boundary. +Gate: fixtures run against real consumers, including Collector Go and Rust; +missing/unknown versions, catalog mismatches and unsupported capabilities fail +before activation. Package boundaries are checked through dependency inspection. + +## 4. Make production, delivery and activation explicit + +Split sampling/estimator policy from transmission suppression/cadence/checkpoint +policy. Allocate and validate them together against the selected query guarantee. +Preserve the rule that adaptive changes produce an authorized successor rather +than mutate an immutable generation. + +For each enabled state family, specify full-state replacement versus independent +contribution semantics, delta base/application rules, replay persistence, and +resynchronization. Retain current encoding until the required endpoint migration +lands. Never assume merge supports subtraction or replacement. + +Specify publication content identity and recoverable rollout coordination. Test +receiver preparation, exact target acknowledgements, failed stage cleanup, partial +activation, restart and delayed old-generation frames. Distinguish local atomic +snapshot installation from distributed convergence and state readiness. + +Gate: no duplicate application or cross-generation query mixing; insufficient +coverage uses fallback/unavailability; unsupported recovery modes remain disabled. + +## 5. Move codecs below runtimes + +Move reusable Collector wrapper reconstruction to typed sketch-library APIs. +Switch both Collector and backend to these APIs. Preserve backend-specific +accumulator/readout adaptation while removing the KLL re-encode/decode detour. +Migrate DDSketch/KLL first; retain supported local paths for other families until +their replacements have parity evidence. Remove vendored delta definitions only +when their authoritative replacement is consumed by both endpoints. + +Gate: full/delta/legacy fixtures and query results pass; dependency inspection +shows no backend production import of Collector runtime. Also remove the obsolete +Collector-specific dependency patch when no longer needed. Test-only end-to-end +fixtures may still build the actual Collector separately. + +## 6. Roll out and retire + +Roll out per supported profile with compatible pinned releases and preserved +rollback artifacts. Keep legacy readers for the agreed producer upgrade window; +remove them only after consumer inventory and replay/recovery retention permit it. +Do not reuse a codec version or descriptor identity for changed semantics. + +Before activation, failure leaves the previous plan intact and staged resources +can be discarded. After partial activation, use the specified recovery protocol +or an explicit successor; a backend-only rollback is not sufficient. State reuse +across generations must pass compatibility checks independently of binary rollback. + +Delete superseded DTO/schema copies, reconstruction paths, and stale documentation +after the replacement passes its gate. Independent query/maintenance projections, +profile adapters, and required legacy readers are not duplication to remove blindly. +Repository relocation and release automation follow stable package boundaries; +they are not prerequisites for runtime correctness. + +## Final evidence + +Record tested revisions, supported families/profiles, fixture results, dependency +graph checks, and compile/install/ingest measurements. Trace one query through its +semantic root, state definition, producer, flow and installed publication. Report +remaining capability gaps explicitly. Completion requires executable evidence, +not document publication, an open PR, or prior migration test counts. diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index 2522bb026..dd1e204d3 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -1,189 +1,115 @@ # Summary Catalog and Self-Describing Summary Architecture -This design defines three logical layers for summary producers and consumers. +## Audience and relationship to physical plans -| Layer | Describes | Changes when | -| --- | --- | --- | -| **Summary Descriptor** | Summary operator and fidelity guarantees | Algorithm, configuration or guarantee contract changes | -| **Data Descriptor** | Summarized source and population | Source binding or population definition changes | -| **Summary Instance** | Instance metadata and summary state | A concrete materialization is created or updated | - -Separating these layers lets many materialized instances reuse the same operator -configuration and data scope. A new time interval creates a new instance without -copying or redefining either descriptor. +Audience: architects and developers. This document owns SDS identity, metadata, +state compatibility, and lifecycle semantics. The +[Planner and physical-plan architecture](asapplanner-integration.md) owns +compilation, the four runtime projections, transmission policy, and publication. +The target model below is distinct from the implementation notes that follow. +Those notes describe bounded paths and do not establish support for every target +lifecycle, distributed recovery mode, or completeness proof. -## Proposed ownership +SDS describes what a summary represents and which concrete state is available. +QueryPlan describes how to answer a query using it. TransmissionPlan describes +how authorized producers deliver state updates. A sketch envelope is one payload +carrier; it is not the SDS catalog or a physical execution plan. -The descriptor vocabulary is a shared contract in `asap_types`. The control -plane owns the authoritative `SummaryCatalog`; Collector and backend receive the -same immutable catalog snapshot. Planner reasons about operators, fidelity, -source and population semantics, while runtime components bind catalog identities -to producers and stored instances. +## Semantic model and authority -| Layer | Responsibility | +| Layer | Meaning | Changes when | +| --- | --- | --- | +| Summary Descriptor | Operator, parameters, fidelity and compatible state representation | Operator/configuration or guarantee contract changes | +| Data Descriptor | Source, population, grouping and observation semantics | Input meaning or population changes | +| Summary Definition | Stable logical materialization referencing descriptors | The semantic definition changes | +| Summary Instance | Concrete extent/group, provenance, status and state reference | State is materialized, updated or retired | + +The control plane owns the desired `SummaryCatalog`. It is constructed from the +selected semantic definitions and common physical compilation decisions before +projecting CollectorPlan, PrecomputePlan, TransmissionPlan and QueryPlan. Plans +reference the same immutable catalog snapshot. They do not independently define +summary meaning, and PrecomputePlan is not the catalog's semantic authority. + +During migration, installed DTOs may repeat parameters, population or window +fields required by existing consumers. These must agree with the catalog and be +mechanically derived from the common bindings. A target artifact may include a +self-contained catalog subset; it must be verifiable against its publication. +Resolve references at installation rather than through per-update remote lookups. + +The observed `ObservedSummaryInventory` reports actual instances and their +readiness. It is not desired state and contains no encoded payloads. Planner may +use this scoped availability evidence without reading sketch bytes. Runtime +reconciliation creates, recovers, retires and expires state according to the +installed contracts; metadata declarations alone do not execute those actions. + +## Identity and state references + +Keep these identities distinct: + +| Identity | Scope and purpose | | --- | --- | -| Summary Descriptor | Shared semantic definition used by Planner and backend | -| Data Descriptor | Shared source/population definition; backend resolves concrete runtime bindings | -| Summary Instance | Backend owns metadata, state, updates, storage and retirement | - -Planner may observe instance availability, covered time ranges and descriptor -references as planning evidence. It does not need the encoded summary state. -SDS describes summaries; an installed QueryPlan specifies how to execute a query -using them. The current backend fields are an incremental implementation of this -model. They must converge on the identities and invariants below rather than add -operator-specific stores beside `SketchStore`. - -## Target semantic model - -The target model has descriptor registries plus pane instances. Descriptor IDs -are derived from canonical semantic content; display names and runtime SIDs are -not descriptor identities. `SummaryDescriptorId` and `DataDescriptorId` currently -contain versioned canonical semantic strings. `SummaryDefinitionId` is a distinct -typed policy fingerprint, and `CatalogGeneration` identifies a publication using -its digest and plan version. A physical `SeriesId` identifies one storage lifetime -of a definition/group; it is neither a descriptor ID nor a pane instance ID. -Changing descriptor encoding to a hash must preserve content identity and handle -collisions explicitly. +| Semantic node ID | Node within the selected Planner DAG; physical bindings retain provenance | +| SummaryDescriptorId / DataDescriptorId | Immutable semantic descriptor content | +| SummaryDefinitionId | Logical materialization; currently backed by a typed policy fingerprint | +| SummaryInstanceId | Concrete materialized instance identity | +| Producer / partition / epoch | Source contribution and restart lifetime | +| SeriesId | Backend physical storage lifetime, not a descriptor or plan identity | +| CatalogGeneration | Catalog publication reference, including digest and plan version | +| Publication identity | Exact installed plan content, including execution and transmission choices | +| Sequence / checkpoint | Update history and applicable delta base within a declared stream scope | + +Current descriptor IDs use versioned canonical semantic strings. Changing their +encoding must preserve semantic identity and explicitly address collisions. +A new interval/group creates an instance without redefining its descriptors. +Moving a producer or changing transmission cadence need not change its semantic +definition, but requires an authorized publication transition. Changed sampling +or observation semantics require guarantee and state-compatibility validation. +A catalog digest alone cannot identify every change to the four physical plans. + +The target instance metadata contract is: ```rust -struct SummaryDescriptor { - id: SummaryDescriptorId, - operator: SummaryOperator, - fidelity: Vec, - state_schema: StateSchema, -} - -struct DataDescriptor { - id: DataDescriptorId, - source: MetricSource, - population: PopulationDefinition, - observation_semantics: ObservationSemantics, -} - struct SummaryInstance { - id: SummaryInstanceId, + instance_id: SummaryInstanceId, summary_definition_id: SummaryDefinitionId, summary_descriptor_id: SummaryDescriptorId, data_descriptor_id: DataDescriptorId, - interval: HalfOpenInterval, - group_values: BTreeMap, - completeness: Completeness, + time_range: HalfOpenTimeRange, + group_values: GroupValues, catalog_generation: CatalogGeneration, placement: SummaryPlacement, state_reference: SummaryStateReference, status: SummaryInstanceStatus, - lifecycle: Persistent | Ephemeral(EphemeralLease), + completeness: InstanceCompleteness, + lifecycle: InstanceLifecycle, } ``` -The instance contract contains no payload bytes. `SummaryStateReference` is an -opaque storage-engine locator with state-schema version, generation, sequence -and optional checksum. `ObservedSummaryInventory` is a versioned data-plane -report keyed by `SummaryInstanceId`; it is observed state and never part of the -desired catalog snapshot. - -## Authoritative SummaryCatalog and execution plans - -The control-plane `SummaryCatalog` is the metadata authority. It stores immutable -Summary and Data Descriptors plus stable materialization identities. It does not -store pane payloads, watermarks, completeness, or observed availability; those -are data-plane instance/runtime metadata. - -The control plane reconciles two explicitly separate views: - -- **Desired SummaryCatalog:** persistent materializations selected through - workload feedback and Planner decisions. -- **Observed Summary Inventory:** instances actually building or stored, - including placement, time coverage, state reference, status and generation. - -Reconciliation creates missing desired materializations, updates instances from -old catalog generations, recovers failed or missing payloads, and retires then -garbage-collects materializations removed from desired state. A data-plane fast -path may create only an ephemeral instance with a finite lease and must report -it immediately. A matching desired materialization promotes it; otherwise it -expires and is collected. The data plane cannot promote an ephemeral instance -or create persistent desired state by itself. - -```text - ASAPPlanner post-ASAP DAG - | - v - Control-plane SummaryCatalog - SummaryDescriptor + DataDescriptor + SummaryDefinitionIdentity - | - catalog references | shared snapshot - +-----------------------+-----------------------+ - | | | - v v v - CollectorPlan PrecomputePlan QueryPlan DAG - producer placement, backend-ingest build, readout, combine, - input routing, build update and lifecycle Prometheus fallback - | | - +-----------+-----------+ - v - TransmissionPlan (when remote producers exist) - full/delta/checkpoint transport, sequence and encoding - | - v - Backend/Collector catalog replicas and SummaryStore - pane instances, completeness and lineage -``` +This is a conceptual shape, not a new wire DTO. `SummaryStateReference` is an +opaque storage locator with schema version, generation, sequence and optional +checksum. SummaryStore owns the referenced payload. Concrete frame identity +additionally records the producer stream and checkpoint context required by its +TransmissionPlan; an instance reference alone does not authorize delta application. -All four execution plans carry catalog references and use catalog materialization -IDs for cross-plan identity. During the compatibility migration, producer and -precompute DTOs still repeat fields needed by existing runtimes, including -operator parameters, source/filter/grouping, window, and state schema. Install -validation requires those fields to agree exactly with the catalog; they are not -independent semantic definitions. New interfaces should resolve them from the -catalog, allowing the copied fields to be removed as consumers migrate. +Sketch libraries own payload schemas, decoding/reconstruction and supported state +operations. Runtime contracts own catalog, plan and frame metadata. Transport +adapters map these contracts into OTLP or another supported carrier without +redefining sketch payload schemas. Full-state replacement, replay, and delta-base +rules are specified in the [integration design](asapplanner-integration.md#identity-and-update-application). +Matching bytes or descriptor IDs alone never proves safe merging or complete data. -| Component | Responsibility | -| --- | --- | -| `SummaryCatalog` | Canonical descriptor definitions, stable IDs and catalog schema/version | -| `CollectorPlan` | Collector placement, input routing, producer identity and collector-side build operations | -| `PrecomputePlan` | Backend-ingest placement, window updates, retention and lifecycle | -| `TransmissionPlan` | Optional producer-to-backend full state, delta, checkpoint, sequence and encoding contract | -| `QueryPlan` | Materialization references, readout, DAG composition and exact Prometheus boundaries | -| SummaryStore (`SketchStore` today) | Instance state, concrete intervals/groups, completeness, lineage and rebuildable rollups | - -The former `BackendPlan` has been removed. `SummaryCatalog` owns materialization -metadata, `PrecomputePlan` owns update/placement/lifecycle, `QueryPlan` owns -readout and fallback routing, and the common deployment envelope carries their -shared plan identity. Consumers atomically install one catalog snapshot with -the plans that reference it. - -`asap_types::executable_plan` owns the installed semantic-DAG representation, -physical node bindings, and `QueryNodeId`. Its `OwnedPostAsapDag` is a Send/Sync -representation for shared runtime snapshots; it is not Planner's -`PostAsapDagDocument` envelope. The owned representation preserves semantic -node IDs and typed operator tags while serializing Planner payloads that contain -process-local `Rc` pointers. The control plane constructs it and checks its -bindings against QueryPlan; precompute execution consumes the shared contract. -`PrecomputePlan`, its envelope, ingest, producer, state schema, and catalog -consistency checks live in `asap_types::precompute_plan`. The compiler chooses -materializations and placement; data-plane installation uses the shared -contract. `asap_types::query_plan` owns QueryPlan, materialization bindings, -logical operator DTOs, and activation validation. The control plane reexports -those types for existing callers and owns the `compile_bound*` and -`logical::compile_logical` functions; Planner traversal and AST lowering do not -move into the shared contract. Data-plane engines import the shared types -directly. No wrapper plan or second wire definition is introduced. - -`asap_types::producer_plan` owns the installed collector and transmission -contracts, frame identities, runtime policy bounds and their validation. The -control plane allocates sampling/GOS budgets and constructs transmission rules -through `sampling_policy_from_accuracy_budget`, `gos_policy_from_accuracy_budget` -and `compile_transmission_plan`. Producers and the data plane import the shared -contracts directly; compilation is not a runtime dependency of those contracts. - -The implemented ownership split is: - -1. Move the SDS catalog contract into `asap_types`. -2. Make the control plane own the authoritative `SummaryCatalog`. -3. Make `PrecomputePlan` reference catalog descriptors and own update, placement and lifecycle. -4. Make `QueryPlan::MaterializationBinding` reference catalog/materialization IDs directly. -5. Distribute the same catalog snapshot to Collector and backend. -6. `BackendPlan`, its protobuf and install endpoint, and duplicate validation are removed. +## Desired state and observed lifecycle + +Persistent desired materializations come from control-plane planning. A runtime +fast path may create only an authorized ephemeral instance with a finite lease, +report it, and await promotion or expiry. It cannot silently make that instance +persistent desired state. + +Reconciliation compares desired definitions with observed placement, extent, +state references, status and completeness. Catalog and plan activation authorize +execution; they do not establish source completeness, durability, or query +readiness. State reuse across generations requires explicit compatibility, and +retired physical lifetimes remain fenced from late updates. ## Implemented backend representation @@ -229,11 +155,12 @@ and timestamp projection. Its Float64 ingest boundary rejects integer constants outside the exactly representable range. This contract enables literal inputs; query lowering must still establish each aggregate's null and row semantics. -The durable `sid_metadata.json` format is versioned independently. Version 2 -contains `summary_descriptors`, `data_descriptors`, and `bindings` tables. A -binding stores only both descriptor IDs plus SID-local timestamps. Version-1 -flat SID records remain readable and are rewritten in normalized version-2 form -on the next metadata update. +The durable `sid_metadata.json` format is versioned independently of the wire +contracts. Descriptor tables and bindings avoid repeating semantic definitions; +later metadata revisions also preserve definition identity and catalog provenance. +Legacy records are interpreted by versioned recovery code and must not acquire +authoritative catalog bindings without validation. See +[completeness and recovery](continuous-summary-completeness.md). An ingest record is never an SDS instance. Raw samples can be transient inputs to the precompute engine, but the backend does not retain them as a second exact @@ -335,8 +262,9 @@ an arbitrary executable program attached to a summary. ## 3. Summary Instance -A Summary Instance combines **instance metadata** with **the actual summary -state**, referencing one Summary Descriptor and one Data Descriptor. +A Summary Instance describes a concrete materialization and references its +stored state, one Summary Descriptor and one Data Descriptor. The metadata DTO +and inventory never embed the encoded payload. | Field | Type | Definition | | --- | --- | --- | @@ -344,7 +272,7 @@ state**, referencing one Summary Descriptor and one Data Descriptor. | `summary_descriptor_id` | `QualifiedId` | Referenced operator/fidelity descriptor | | `data_descriptor_id` | `QualifiedId` | Referenced source/population descriptor | | `metadata` | `InstanceMetadata` | Concrete extent, population binding, completeness and provenance | -| `state` | `SummaryState` | Materialized state encoded according to the Summary Descriptor | +| `state_reference` | `SummaryStateReference` | Opaque locator for separately stored state and its schema/provenance | `InstanceMetadata` contains the concrete time range or dataset extent, any group values needed by the population rule, completeness (`Complete`, `Partial` or @@ -352,10 +280,10 @@ values needed by the population rule, completeness (`Complete`, `Partial` or evidence. Time ranges specify their clock, units and interval boundaries. Completeness is separate from mathematical approximation error. -`SummaryState` is the state itself, not a quantile readout or other query result. -If a transport carries a delta, it must identify its base instance/version and -the descriptor's supported apply operation; it cannot be interpreted as a full -state without that context. +The referenced payload is maintained state, not a quantile readout or other +query result. A transported delta identifies its authorized producer stream and +base checkpoint as well as the supported apply operation. A descriptor or instance +ID alone is insufficient to interpret it as a full state. ## Shared-descriptor example @@ -389,22 +317,22 @@ instances: summary_descriptor_id: example:kll-200-v1 data_descriptor_id: example:login-cpu-v1 metadata: {time_range: "[0,10)", clock: example:seconds} - state: S0 + state_reference: {store: example-store, key: S0, state_schema_version: 1} - instance_id: example:login-cpu-1 summary_descriptor_id: example:kll-200-v1 data_descriptor_id: example:login-cpu-v1 metadata: {time_range: "[10,20)", clock: example:seconds} - state: S1 + state_reference: {store: example-store, key: S1, state_schema_version: 1} - instance_id: example:login-cpu-2 summary_descriptor_id: example:kll-200-v1 data_descriptor_id: example:login-cpu-v1 metadata: {time_range: "[20,30)", clock: example:seconds} - state: S2 + state_reference: {store: example-store, key: S2, state_schema_version: 1} ``` -`S0`, `S1` and `S2` denote separate encoded KLL states. The example omits concrete -payload bytes and producer evidence; it makes no completeness or numerical error -claim. Descriptor references must resolve within the supplied context or a +`S0`, `S1` and `S2` are opaque keys for separately stored KLL states. This +conceptual example omits full state-reference provenance and producer evidence; +it is not an installable DTO and makes no completeness or numerical error claim. Descriptor references must resolve within the supplied context or a durably retained descriptor registry. Changing `k` creates a new Summary Descriptor. Changing the source or population From af61296daa3ff137d9b26549686b8ec91ae85f5b Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 02:36:09 +0000 Subject: [PATCH 02/15] docs: specify executable subplan materialization boundaries --- docs/design_docs/asapplanner-integration.md | 115 ++++++++++++++++++ .../design_docs/asapplanner-migration-plan.md | 44 ++++++- .../summary-catalog-sds-architecture.md | 6 + 3 files changed, 164 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 9284f6e3b..2e12977f4 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -396,6 +396,119 @@ subset and transmission rules. These are mechanically derived copies from one publication, validated against its identity/digest. They are not independently editable authorities. Runtimes need no catalog network lookup on each update. +### Executable subgraphs and materialization boundaries + +**Decision:** PrecomputePlan and QueryPlan each own the operations they execute. +The physical compiler explicitly splits the selected DAG at materialization +boundaries and records the state references connecting the subplans. There can +be multiple boundaries: one query can consume several summaries and several +queries can share the same summary. + +Today, `PrecomputePlan.executable_dags` stores complete `InstalledPostAsapDag` +documents, including read-time nodes such as `SummaryEstimate`. Bindings mark +execution ownership, and the maintenance runtime evaluates dependencies of +`precompute_sinks` rather than every stored node. This explains current behavior +but is a mismatch between the PrecomputePlan abstraction and its contents. +The target removes query-only operations from its executable representation. + +```mermaid +flowchart LR + subgraph PP[PrecomputePlan] + I[Input] --> B[Build or update summary] + B --> W[Materialize summary S] + end + W -. State reference S .-> R + subgraph QP[QueryPlan] + R[Read summary S] --> E[SummaryEstimate] + E --> O[Query result] + end +``` + +The dashed connection is a state dependency, not a claim that every query triggers +a synchronous precompute execution. State must satisfy the installed schema, +coverage and readiness requirements when read. + +A boundary reuses the existing catalog identities and materialization bindings: + +| Information | Purpose | +| --- | --- | +| Summary definition reference | Producer and reader identify the same logical summary | +| State schema and representation | Reader interprets the produced state correctly | +| Window, phase and grouping contract | Read covers the intended population and interval without double counting | +| Publication/catalog generation | Prevent incompatible installed plans and state from being combined | +| Semantic node provenance | Relate physical production/read operations to the selected Planner computation | + +These are required relationships, not a new duplicate identity registry. Reuse +`MaterializationBinding`, state-schema contracts and catalog references where they +already express the relationship. Concrete stored instances are resolved at +runtime from the definition, extent, group and accepted generation; compilation +does not allocate every future pane instance. + +The compiler extracts subgraphs using execution timing, dependencies and explicit +materialization bindings. It must not split by operator name alone. Precompute +subgraphs terminate at materialization sinks and can read prior materializations +to derive new summaries. Query subgraphs start at state reads or explicit exact +inputs and perform read-time operations. In the current semantic contract, +`SummaryEstimate` is read-time and belongs in QueryPlan; maintenance-time exact +finalization is a distinct permitted operation when its input contract is met. +Unsupported phase crossings fail compilation rather than silently moving work. + +The complete semantic DAG can remain as publication-level provenance or a compiler +artifact, with semantic-to-physical mappings. It is not executable content owned +by PrecomputePlan and need not be a third visualization section. Runtime plans +must contain their required execution information without traversing query-only +provenance to discover maintenance work. + +### Meaning of maintenance and current binding labels + +Precompute names the backend plan/engine that produces and maintains summary +state. Maintenance names the execution phase that builds, updates or derives that +state rather than answering a query. It includes initial batch construction and +full rebuilds; it does not imply incremental or continuous ingestion. + +The current binding enum classifies semantic nodes as follows. These names remain +unchanged by this documentation proposal: + +| Binding | Meaning | +| --- | --- | +| `Materialization` | Maintenance-time node explicitly bound to a stored summary definition | +| `MaintenanceInput` | Maintenance-time source or intermediate operation without its own stored-summary binding | +| `Query` | Read-time node explicitly mapped to a QueryPlan node | +| `QueryInput` | Read-time node without a separate explicit QueryPlan mapping, such as an operation absorbed by a larger query operation | + +`MaintenanceInput` is not a data format or necessarily a leaf. For example, in a +supported derived-summary pipeline, stored exact Sum/Count state can be finalized +into average-valued rows and then aggregated into a stored KLL. The finalization +is a maintenance intermediate without its own stored-summary binding; the stored +states have materialization bindings. An inner aggregate is not automatically a +`MaintenanceInput`: if its state is separately materialized, it is a +`Materialization`. Execution still requires the appropriate immutable-input and +runtime capability checks. + +Similarly, an ASAP-side descending Sort followed by Limit can lower to one +`TopKSelection` QueryPlan node. Limit maps to that node; the absorbed Sort can be +`QueryInput`. Absorption does not mean the sorting is omitted. Current binding +labels alone are not executable subgraphs; the new compiler projection makes +ownership and boundary reads explicit. + +### Visualization contract + +The default execution visualization has separate PrecomputePlan and QueryPlan +views, connected by labeled summary references. It shows each subplan's actual +operations, input/output boundaries, shared materializations, and generation. +Multiple query consumers must refer to the same shared summary rather than +suggesting duplicate maintenance. Derived-summary chains remain visible inside +the maintenance view with their state-read boundaries. + +While rendering the legacy serialized format, distinguish embedded semantic +context from operations executed by that plan. A read-time `SummaryEstimate` +embedded in PrecomputePlan must be visible in a faithful artifact view and marked +as query-owned context, never depicted as precompute execution. A projected +execution view may exclude that context only when it explicitly says it is showing +the execution projection. The user should not need a separate Semantic Plan page +to understand either subplan. After migration, the executable artifacts and their +two execution views should agree directly. + ## SDS, state codecs, and transmission | Contract | Authority | @@ -534,6 +647,8 @@ backend install/ingest/query boundaries, including Go/Rust interoperability. | Gate | Observable evidence | | --- | --- | | Semantic preservation | Selected node/root provenance survives all projections; incompatible grouping/window/lifecycle choices fail before publication | +| Subplan ownership | Precompute executable subgraphs contain no query-only SummaryEstimate; QueryPlan reads explicit compatible state boundaries; shared and derived summaries remain traceable | +| Visualization fidelity | Separate plan views agree with executable ownership; legacy embedded context is explicitly distinguished from executed operations | | Shared production | N admitted observations cause N producer updates per intended partition, not N multiplied by consumer queries | | Protocol conformance | Full, delta, duplicate, conflict, gap, epoch restart, unknown-base and legacy fixtures have explicit expected outcomes | | State readiness | Missing or pending coverage uses configured fallback/unavailability; installation never certifies completeness | diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index dba63acf7..eb051b50b 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -24,7 +24,7 @@ record unsupported combinations as capabilities rather than broadening claims. | Stage | Owner | Deliverable | Exit gate | | --- | --- | --- | --- | | 1. Contract and behavior inventory | Backend, Collector, Planner maintainers | Authority map, supported capability matrix, cross-language fixtures | Every existing production wire path and plan entry point has an explicit compatibility expectation | -| 2. Common physical bindings | Backend control plane | Internal binding stage, catalog construction, four projections | Existing supported inputs produce semantically equivalent publications; no repeated selection through PrecomputePlan | +| 2. Common physical bindings | Backend control plane | Internal binding stage, catalog construction, four projections, and explicit maintenance/query subgraph boundaries | Existing supported inputs retain semantics; subplan execution ownership and state references are explicit | | 3. Shared contracts and validation | Backend/Collector; Planner for IR export | Lightweight contracts, typed semantic export, shared publication validation | Actual Go/Rust consumers accept matching artifacts and reject incompatible ones | | 4. Policy and deployment boundaries | Compiler and runtimes | Production/transport policy split; explicit application and activation rules | Guarantee, checkpoint, readiness and partial-rollout fixtures pass for enabled modes | | 5. Codec extraction | Sketch libraries, Collector, backend | Typed reconstruction APIs and consumer migration | Backend excludes `asap-precompute-rs`; supported decoding and query results remain compatible | @@ -106,6 +106,48 @@ Gate: supported profiles retain query results, window/label semantics, producer update counts, configured fallback and publication compatibility. New binding provenance makes every runtime task traceable to the selected decision. +### 2a. Split maintenance and query executable subgraphs + +After establishing common bindings, implement the +[materialization boundary design](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries). +Extract maintenance subgraphs terminating at materialization sinks and query +subgraphs reading those definitions. Preserve semantic provenance without keeping +query-only nodes as executable content in `PrecomputePlan.executable_dags`. +Reuse existing catalog/materialization/schema identities rather than introducing +a second boundary registry. + +Switch maintenance execution to the extracted subgraphs and their explicit state +inputs. Validate every cross-plan boundary during installation: definition, +schema, grouping, window/phase, and accepted generation must agree. Retain all +query-side operations and maintenance intermediates needed by their respective +executors. A semantic node may be absorbed into a physical operation, but the +mapping must still explain where its work occurs. + +Update visualization to show the actual two executable subplans and their state +references. Legacy artifact inspection must label embedded query nodes as context; +do not silently render a filtered graph as the original serialized document. +No separate Semantic Plan section is required for understanding execution. + +Acceptance cases: + +- A build-summary/read-estimate pipeline places SummaryEstimate only in QueryPlan's + executable representation, with an explicit read of the produced summary. +- One query reading multiple summaries has all boundaries resolved; two queries + sharing one summary retain one compatible producer per intended partition. +- A supported derived-summary chain preserves source state reads and maintenance + intermediates, including permitted exact finalization on completed inputs. +- Incorrect schema, grouping/window phase or generation is rejected at installation. +- Old/new representations produce equivalent supported query results and preserve + maintenance update counts, completion checks, fallback and recovery behavior. +- Rendered plan views agree with executable ownership and retain provenance links. + +This changes an installed representation. Stage the work: establish common bindings +with the old wire format first, then introduce a versioned split representation +with adapters for supported older publications. Do not reinterpret the old field +under the same version. Remove the legacy full-DAG path only after producer, +consumer and recovery fixtures pass and the compatibility window closes. This +PR proposes the split; it does not claim the runtime migration is implemented. + ## 3. Extract contracts and unify validation Separate lightweight semantic IR export from Planner search internals. Preserve diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index dd1e204d3..01246b5e0 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -98,6 +98,12 @@ redefining sketch payload schemas. Full-state replacement, replay, and delta-bas rules are specified in the [integration design](asapplanner-integration.md#identity-and-update-application). Matching bytes or descriptor IDs alone never proves safe merging or complete data. +The [physical subplan boundary](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries) +connects a materialization sink to reads of the same summary definition. It does +not add a new SDS identity or require compile-time enumeration of future instances. +PrecomputePlan owns state production and QueryPlan owns query-time readout; +semantic provenance retained for tracing does not change execution ownership. + ## Desired state and observed lifecycle Persistent desired materializations come from control-plane planning. A runtime From e868424fd7b967f76a33fc7b5f8968828694e57f Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 02:40:57 +0000 Subject: [PATCH 03/15] docs: scope migration to backend precompute and query plans --- docs/design_docs/README.md | 4 +- docs/design_docs/asapplanner-integration.md | 10 + .../design_docs/asapplanner-migration-plan.md | 421 +++++++++--------- 3 files changed, 221 insertions(+), 214 deletions(-) diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 433e26663..0e461bf4b 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -10,7 +10,9 @@ notes and migration gates distinguish implemented behavior from proposed changes - [Summary Catalog and SDS](summary-catalog-sds-architecture.md) owns descriptors, definition/instance identity, state references, inventory and lifecycle semantics. - [Architecture migration delivery plan](asapplanner-migration-plan.md) defines - compatibility fixtures, implementation stages, rollout and retirement gates. + the current PrecomputePlan/QueryPlan scope, common-library extraction, removal + of ASAPCollector dependencies, and backend acceptance/retirement gates. Collector + and transmission plan changes are deferred. - [Accepted-input completeness](continuous-summary-completeness.md) describes the backend's bounded admission, publication and recovery behavior. diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 2e12977f4..d6f5508b8 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -15,6 +15,16 @@ owns implementation gates. Existing remain the compatibility baseline until corresponding changes land in both consumers. Conflicts require a versioned migration, not unilateral reinterpretation. +## Current implementation scope + +The [migration delivery plan](asapplanner-migration-plan.md) currently implements +only the backend PrecomputePlan/QueryPlan split and extraction of their common +contracts/codecs. It requires no backend build/runtime dependency on ASAPCollector. +CollectorPlan, TransmissionPlan, Collector adoption and distributed rollout are +future work, not prerequisites. The four-plan architecture below remains the +longer-term design; its distributed acceptance requirements do not enlarge this +iteration's completion gate. + ## Problem and current baseline Collector and backend must agree on what a summary means, how it is produced, diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index eb051b50b..f01ea50da 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -1,226 +1,221 @@ -# Physical-plan architecture: migration delivery plan +# PrecomputePlan and QueryPlan: migration delivery plan -Audience: developers implementing the +Audience: developers implementing the backend portion of the [integration architecture](asapplanner-integration.md). Status: proposed delivery -sequence, not a record of completed implementation. This replaces historical PR -stack tracking with behavior-based gates. Existing merged behavior is the baseline; -old test totals and PR status are not evidence for this migration. - -## Completion definition - -For each declared supported deployment profile, one Planner decision is bound -once and projected into catalog, QueryPlan, PrecomputePlan, CollectorPlan and -TransmissionPlan. Publication, runtime state, and query readout agree on identity, -schema, window, guarantees and generation. Backend production code no longer -imports the Collector execution runtime for reconstruction. - -Backend-local and distributed profiles need separate acceptance. Arbitrary -PromQL, all sketch-family delta modes, general multi-hop execution, and a new -repository are outside the completion gate. Preserve supported existing behavior; -record unsupported combinations as capabilities rather than broadening claims. +sequence, not a record of completed implementation. + +## Scope and completion definition + +This iteration handles **PrecomputePlan and QueryPlan only**, including their +shared SDS/catalog contracts, executable subgraph boundaries, backend installation, +and state decoding. CollectorPlan and TransmissionPlan compilation, policy redesign, +producer rollout and distributed activation are deferred. Their implementation or +release is not a prerequisite for completing this work. + +The backend must have **no build or runtime dependency on ASAPCollector**. Extract +the common contracts and codecs into runtime-independent libraries, then consume +those libraries from the backend. Copying Collector runtime code into a backend-only +fork or keeping a shared package hosted inside ASAPCollector does not meet this +boundary. Collector can adopt the common libraries in a separate follow-up. + +Completion means: + +- One selected Planner decision produces a coherent catalog and two executable + subplans, connected by explicit materialization/state references. +- PrecomputePlan executes state production/maintenance; QueryPlan executes reads + and query-time operations, including SummaryEstimate. +- Both subplans agree on definition identity, schema, grouping/window, guarantees + and generation, with backend-local atomic installation and separate readiness. +- Backend library/binary builds and the required test suite need no ASAPCollector + checkout, package or process. Shared reconstruction uses neutral libraries. +- Supported backend inputs, query results, recovery and legacy decoding retain + their documented behavior. No new distributed behavior is claimed. + +Existing CollectorPlan/TransmissionPlan fields may remain in legacy publication +adapters for compatibility. They are not redesigned by this migration. The local +path requires neither a Collector target nor transmission rules and must not use +CollectorPlan as the source of shared types or decisions. Do not silently accept +new distributed capabilities just because the local contract has changed. ## Sequence and dependencies | Stage | Owner | Deliverable | Exit gate | | --- | --- | --- | --- | -| 1. Contract and behavior inventory | Backend, Collector, Planner maintainers | Authority map, supported capability matrix, cross-language fixtures | Every existing production wire path and plan entry point has an explicit compatibility expectation | -| 2. Common physical bindings | Backend control plane | Internal binding stage, catalog construction, four projections, and explicit maintenance/query subgraph boundaries | Existing supported inputs retain semantics; subplan execution ownership and state references are explicit | -| 3. Shared contracts and validation | Backend/Collector; Planner for IR export | Lightweight contracts, typed semantic export, shared publication validation | Actual Go/Rust consumers accept matching artifacts and reject incompatible ones | -| 4. Policy and deployment boundaries | Compiler and runtimes | Production/transport policy split; explicit application and activation rules | Guarantee, checkpoint, readiness and partial-rollout fixtures pass for enabled modes | -| 5. Codec extraction | Sketch libraries, Collector, backend | Typed reconstruction APIs and consumer migration | Backend excludes `asap-precompute-rs`; supported decoding and query results remain compatible | -| 6. Retirement and release | Participating repositories | Remove superseded copies/adapters, pin compatible versions | Both profiles pass end-to-end gates without retired paths | - -Stages 2 and 3 preserve existing wire formats through boundary adapters. Stage 4 -changes public contracts only with negotiated/versioned compatibility. Codec work -can proceed after stage 1, but its removal gate depends on stable contracts and -consumer coverage. Do not combine an unrelated Planner upgrade with extraction. - -## 1. Establish authority and fixtures - -Inventory Planner exports, backend installed contracts, Collector Go/Rust DTOs, -OTel carriers, sketch state/delta schemas and legacy bare-state decoders. Record -one owner for each concept and the current supported producer/consumer versions. -Compare actual field shapes, defaults, enum meanings, units and rejection behavior; -a similarly named struct is not compatibility evidence. - -Capture supported backend-local, distributed full-state, distributed delta, and -generation-transition examples. Use distinct evidence for wire equivalence and -semantic state/readout equivalence; randomized state may require persisted fixtures -and semantic assertions rather than comparing unrelated fresh encodings. - -Protocol cases include duplicate/conflicting sequences, unknown delta base, gaps, -producer restart, malformed framed payload, and legacy unframed state. Label any -currently failing target invariant as migration work, not passing baseline behavior. -Have a separate reviewer review expected outcomes before protocol changes. - -### Planner caller contract gate (#438) - -Resolve [Planner #438](https://github.com/ProjectASAP/ASAPPlanner/issues/438) -at the public workflow boundary, not only in the backend compiler. First deliver -an ASAPPlanner user guide for supported entry/exit points, including workflows -that intentionally stop at pre-ASAP IR, candidates, or a selected semantic DAG. -Each recipe must document exact APIs, controls, defaults, performed checks and -output limitations and run against the documented revision. This guide does not -depend on implementing a unified facade. Document strategy selection and automatic -passes separately from model providers, runtime capabilities and requirements. Audit actual -API defaults and low-level output guarantees, including the current all-enabled -lifecycle capability default, unknown lifecycle cost inputs, and per-root accuracy -propagation. Distinguish Rust defaults from serialized-field omission. Then specify -one application-facing -request/result contract with explicit incomplete/infeasible outcomes. Use the -[caller contract](asapplanner-integration.md#caller-contract-and-lifecycle-completeness) -as the target; its omission rules are proposed behavior, not current API facts. - -The complete output must associate each materialized state with a selected or -capability-constrained, validated lifecycle. Planner models the available -lifecycle vocabulary; runtime support and workload/policy constraints determine -which modes may enter candidate selection. A singleton legal set is a complete -selection, not a skipped lifecycle decision. Lifecycle feasibility and applicable -costs must participate in candidate selection. Keep diagnostic DAG exports accessible, but -do not allow them to masquerade as deployment-complete results. Document which -inputs callers control and which evidence/capabilities come from providers. - -Gate: executable public-API examples cover one-shot, recurring, unknown-demand, -and missing-evidence inputs; diagnostics expose defaults and their consequences. -Include a backend that can build summaries only from data at rest: no incremental -mode may enter ranking, and a singleton legal lifecycle must produce a complete -commitment. Recurring demand must not imply incremental support or permission for -retained reuse. Verify that an empty legal set is reported explicitly. -The physical compiler rejects incomplete stateful commitments. Stages 2 and 3 -must preserve this distinction while existing lower-level APIs remain compatible. - -## 2. Refactor compilation without changing semantics - -Retain candidate selection and cost/capability evaluation. Introduce only a -compiler-local structure for selected tasks, definitions, state bindings and -producer/consumer edges. Construct the catalog from the selected definitions, -then project all four plans from those bindings. - -Remove the dependency of transmission compilation on PrecomputePlan. Preserve -shared producer identity across roots and reject incompatible physical bindings. -Target artifacts may embed catalog/rule subsets but must be derived from the -same publication. Compare old/new outputs with normalization only for explicitly -nondeterministic metadata; do not normalize away semantic or identity differences. - -Gate: supported profiles retain query results, window/label semantics, producer -update counts, configured fallback and publication compatibility. New binding -provenance makes every runtime task traceable to the selected decision. - -### 2a. Split maintenance and query executable subgraphs - -After establishing common bindings, implement the -[materialization boundary design](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries). -Extract maintenance subgraphs terminating at materialization sinks and query -subgraphs reading those definitions. Preserve semantic provenance without keeping -query-only nodes as executable content in `PrecomputePlan.executable_dags`. -Reuse existing catalog/materialization/schema identities rather than introducing -a second boundary registry. - -Switch maintenance execution to the extracted subgraphs and their explicit state -inputs. Validate every cross-plan boundary during installation: definition, -schema, grouping, window/phase, and accepted generation must agree. Retain all -query-side operations and maintenance intermediates needed by their respective -executors. A semantic node may be absorbed into a physical operation, but the -mapping must still explain where its work occurs. - -Update visualization to show the actual two executable subplans and their state -references. Legacy artifact inspection must label embedded query nodes as context; -do not silently render a filtered graph as the original serialized document. -No separate Semantic Plan section is required for understanding execution. +| 1. Inventory and fixtures | Backend | Contract/dependency map and backend behavior baseline | Every scoped entry point and Collector import has a documented replacement or compatibility fixture | +| 2. Extract common libraries | Backend and shared-library maintainers | Runtime-independent contracts and typed sketch reconstruction | Backend and required tests build without ASAPCollector; existing decoding remains compatible | +| 3. Bind and split two subplans | Backend compiler/runtime | Common bindings, catalog, maintenance/query subgraphs, explicit state boundaries | Executable ownership and provenance match supported semantics | +| 4. Validate, install and visualize | Backend | Shared two-plan checks, local generation switching, two execution views | Invalid boundaries fail; readiness and recovery remain correct | +| 5. Retire and release | Backend and shared-library maintainers | Remove superseded paths, pin common-library versions | Scoped end-to-end and dependency gates pass without Collector work | + +Stages 2 and 3 may be developed independently after the inventory, but both must +finish before the final gate. Extract code without changing payload bytes first; +version changes to installed executable representations separately. Do not combine +an unrelated Planner upgrade or a new public Planner facade with this work. + +## 1. Establish authority and backend fixtures + +Inventory the pinned Planner output, backend plan/SDS types, state schemas, +envelope types, all `asap_precompute_rs` imports, Cargo patches, and tests that +build or invoke Collector. Identify the smallest common API required at each +call site. Keep backend execution, storage and accumulator/readout adaptation in +the backend; do not move all of `asap_types` into a generic package indiscriminately. + +Capture backend-local raw ingestion, summary reconstruction, state maintenance, +query readout, completion, installation and recovery fixtures. For supported +existing full/delta/legacy payloads, record the bytes and expected state/readout +behavior with source revision and schema provenance. Frozen compatibility fixtures +may originate from Collector but must be usable without checking out or running it. +Randomized sketches may need persisted fixtures and semantic assertions rather +than comparing independently generated bytes. + +Input validation tests cover malformed framed payloads and currently supported +sequence/checkpoint behavior where touched by extraction. Missing target features +remain explicit gaps; do not turn this into a new transmission protocol project. +Have a separate reviewer assess boundary/replay expectations for consequential +implementation changes; independent review is not claimed by this document. + +### Planner input boundary + +Consume the existing pinned semantic contract and preserve per-query requirements, +root associations and lifecycle commitments. Runtime capabilities restrict eligible +lifecycle modes; a singleton legal lifecycle is valid. Incomplete stateful +commitments must not reach installation. A data-at-rest-only backend does not gain +incremental support merely because query demand repeats. + +[Planner #438](https://github.com/ProjectASAP/ASAPPlanner/issues/438) and its +[user/API documentation work](https://github.com/ProjectASAP/ASAPPlanner/pull/440) +remain related work, not completion prerequisites. Change Planner contracts only +for a demonstrated blocker to this two-plan split; a broad IR redesign or unified +Planner entry point is deferred. + +## 2. Extract shared contracts and codecs; remove Collector dependency + +Use two narrow ownership boundaries: + +| Common code | Owner / destination | Excluded dependencies | +| --- | --- | --- | +| Runtime envelope metadata, shared IDs/tags, schema references and required validation | Lightweight neutral contract package, outside ASAPCollector | Collector/backend executors and Planner optimizer | +| Sketch payload schemas, decode/encode/reconstruction and supported state operations | Existing sketch-library APIs, or a neutral codec package if a concrete dependency requires it | Edge windowing, scheduling, host adapters and backend storage | + +Prefer existing sketch libraries and a small contract package over a new general +framework. If a new neutral package is required, establish its independent source +and versioned consumption before removing the old imports. Shared does not mean +that both runtimes must migrate in the same PR: backend adoption is in scope; +Collector adoption is deferred. Keep one schema authority and preserve compatible +wire behavior so a later Collector migration can reuse the same implementation. + +Move reusable DDSketch/KLL reconstruction out of Collector wrappers. Backend +accumulators consume typed decoded state, removing the unnecessary KLL +reconstruction/serialization/decoding round trip. Preserve supported local paths +for other families until replacement APIs have parity evidence. Keep legacy +bare-state readers and required vendored schemas until a compatible authoritative +replacement exists; do not silently change encoding versions or delta semantics. + +Remove the `asap-precompute-rs` dependency and obsolete Collector-specific Cargo +patches. Replace tests that import/invoke Collector with neutral-library tests and +provenance-bearing compatibility fixtures. Adapt dependency-enforcement tests to +the new boundary. Backend CI must not clone/build Collector indirectly through a +test helper, script, transitive dependency or shared-package location. + +Gate: inspect manifests, lockfiles, dependency graphs, source imports, build scripts +and required tests; no ASAPCollector dependency remains. Full-state, supported +delta and legacy fixtures retain decoding/rejection and readout behavior. Shared +libraries do not depend back on the backend runtime. No Collector release is needed. + +## 3. Bind once and split the executable subplans + +Retain candidate evaluation and downstream commitment. Introduce only the +compiler-local bindings needed for selected tasks, summary definitions, state +schemas, storage and input/output references. Construct the catalog and derive +PrecomputePlan and QueryPlan from the same decisions. Preserve semantic node +provenance and shared producers; do not independently choose their meanings. + +Implement the [materialization boundary design](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries): + +- Extract maintenance subgraphs terminating at stored-summary sinks, including + explicit reads of prior summaries for supported derived-state pipelines. +- Extract query subgraphs with explicit materialization reads and read-time + operations; query-only SummaryEstimate is absent from precompute executable content. +- Reuse catalog/materialization/schema identities; do not add a parallel boundary + identity registry or enumerate future stored pane instances during compilation. +- Use execution timing, dependencies and bindings rather than operator names to + determine ownership. Preserve absorbed operations in semantic-to-physical mapping. + +Switch maintenance execution to these subgraphs instead of discovering its work +inside a complete query DAG. Full semantic provenance can remain an artifact or +shared installation metadata, but is not executable content owned by PrecomputePlan. +Preserve its current representation if replacing it is unnecessary for the split. + +Version the split installed representation and normalize supported legacy +publications at the backend boundary. Legacy CollectorPlan/TransmissionPlan fields +remain compatibility concerns, not additional projections to implement. Do not +reinterpret the old executable-DAG field under an unchanged version. + +## 4. Validate, install and visualize the two plans + +Use shared two-plan/catalog validation at compilation and backend installation, +followed by actual local resource checks. Validate every boundary's definition, +schema, grouping/window phase and accepted generation. Keep one coherent local +publication identity; two independently activated subplans must not become visible. + +Stage and activate the backend snapshot atomically for query readers. Failed +staging preserves the previous active generation. Test restart, queued old-generation +maintenance output and compatible/incompatible state recovery. State readiness +remains distinct from installation; pending or insufficient coverage uses the +configured exact fallback or explicit unavailability. Distributed acknowledgements, +Collector cutover and new transport resynchronization are outside this stage. + +Visualize PrecomputePlan and QueryPlan separately, connected by labeled state +references. Show shared materializations and supported derived chains. For legacy +artifact inspection, label embedded read-time nodes as query-owned context rather +than maintenance execution; a projected view must identify itself as such. No +separate Semantic Plan page is required to understand the two execution plans. Acceptance cases: -- A build-summary/read-estimate pipeline places SummaryEstimate only in QueryPlan's - executable representation, with an explicit read of the produced summary. -- One query reading multiple summaries has all boundaries resolved; two queries - sharing one summary retain one compatible producer per intended partition. -- A supported derived-summary chain preserves source state reads and maintenance - intermediates, including permitted exact finalization on completed inputs. -- Incorrect schema, grouping/window phase or generation is rejected at installation. -- Old/new representations produce equivalent supported query results and preserve - maintenance update counts, completion checks, fallback and recovery behavior. -- Rendered plan views agree with executable ownership and retain provenance links. - -This changes an installed representation. Stage the work: establish common bindings -with the old wire format first, then introduce a versioned split representation -with adapters for supported older publications. Do not reinterpret the old field -under the same version. Remove the legacy full-DAG path only after producer, -consumer and recovery fixtures pass and the compatibility window closes. This -PR proposes the split; it does not claim the runtime migration is implemented. - -## 3. Extract contracts and unify validation - -Separate lightweight semantic IR export from Planner search internals. Preserve -node/operator/schema/guarantee meaning while migrating `OwnedPostAsapDag`; do not -replace typed semantic validation with arbitrary JSON acceptance. - -Extract SDS, installed plan, publication and frame contracts into packages that -import neither execution runtime nor optimizer. Select a schema authority and -binding-generation approach before removing manual Go/Rust copies. Keep sketch -payload schemas in their sketch-library authority. - -Use shared cross-plan validation at compile and install boundaries, followed by -local resource checks. Versioned legacy adapters normalize once at the boundary. -Gate: fixtures run against real consumers, including Collector Go and Rust; -missing/unknown versions, catalog mismatches and unsupported capabilities fail -before activation. Package boundaries are checked through dependency inspection. - -## 4. Make production, delivery and activation explicit - -Split sampling/estimator policy from transmission suppression/cadence/checkpoint -policy. Allocate and validate them together against the selected query guarantee. -Preserve the rule that adaptive changes produce an authorized successor rather -than mutate an immutable generation. - -For each enabled state family, specify full-state replacement versus independent -contribution semantics, delta base/application rules, replay persistence, and -resynchronization. Retain current encoding until the required endpoint migration -lands. Never assume merge supports subtraction or replacement. - -Specify publication content identity and recoverable rollout coordination. Test -receiver preparation, exact target acknowledgements, failed stage cleanup, partial -activation, restart and delayed old-generation frames. Distinguish local atomic -snapshot installation from distributed convergence and state readiness. - -Gate: no duplicate application or cross-generation query mixing; insufficient -coverage uses fallback/unavailability; unsupported recovery modes remain disabled. - -## 5. Move codecs below runtimes - -Move reusable Collector wrapper reconstruction to typed sketch-library APIs. -Switch both Collector and backend to these APIs. Preserve backend-specific -accumulator/readout adaptation while removing the KLL re-encode/decode detour. -Migrate DDSketch/KLL first; retain supported local paths for other families until -their replacements have parity evidence. Remove vendored delta definitions only -when their authoritative replacement is consumed by both endpoints. - -Gate: full/delta/legacy fixtures and query results pass; dependency inspection -shows no backend production import of Collector runtime. Also remove the obsolete -Collector-specific dependency patch when no longer needed. Test-only end-to-end -fixtures may still build the actual Collector separately. - -## 6. Roll out and retire - -Roll out per supported profile with compatible pinned releases and preserved -rollback artifacts. Keep legacy readers for the agreed producer upgrade window; -remove them only after consumer inventory and replay/recovery retention permit it. -Do not reuse a codec version or descriptor identity for changed semantics. - -Before activation, failure leaves the previous plan intact and staged resources -can be discarded. After partial activation, use the specified recovery protocol -or an explicit successor; a backend-only rollback is not sufficient. State reuse -across generations must pass compatibility checks independently of binary rollback. - -Delete superseded DTO/schema copies, reconstruction paths, and stale documentation -after the replacement passes its gate. Independent query/maintenance projections, -profile adapters, and required legacy readers are not duplication to remove blindly. -Repository relocation and release automation follow stable package boundaries; -they are not prerequisites for runtime correctness. +- Build-summary/read-estimate places SummaryEstimate only in QueryPlan execution. +- One query can read multiple bound summaries; two queries can share one compatible + producer per intended partition without multiplying maintenance updates. +- Supported derived-summary chains preserve explicit state reads, completed-input + requirements and maintenance intermediates such as exact finalization. +- Wrong schema, grouping/window phase or generation fails before activation. +- Local failed-stage, generation-switch and restart cases preserve state lifetime, + completion checks, recovery, query consistency and fallback behavior. +- Old/new supported artifacts produce equivalent results and update counts. +- Visualization agrees with executable ownership and retains provenance links. +- These cases run without an ASAPCollector package, checkout or process. + +## 5. Roll out and retire + +Release the backend with pinned neutral-library versions and preserved rollback +artifacts. First migrate supported local publications; retain versioned adapters +for supported older artifacts. Remove full-DAG-in-precompute execution and obsolete +Collector adapter code only after their replacements pass the scoped fixtures. +State reuse across generations requires explicit compatibility independently of +binary rollback. Keep legacy payload readers for their supported recovery window. + +Distributed deployments continue on their supported compatibility path or receive +an explicit unsupported-version result. Do not claim a distributed migration or +require a Collector upgrade for this backend-local milestone. Repository-wide +schema consolidation and cross-language release coordination can follow separately. + +## Deferred work + +- CollectorPlan and TransmissionPlan compilation/refactoring and their runtime consumers. +- Moving production/sampling policy out of transmission policy across components. +- Collector adoption of the neutral contracts/codecs and Go/Rust binding consolidation. +- Distributed activation, new delivery/checkpoint/recovery semantics and multi-hop topology. +- A general Planner facade, broad semantic IR redesign and unrelated capability expansion. + +These remain part of the broader architecture, but are not dependencies or exit +gates for this migration. Existing supported input behavior is preserved through +backend adapters and fixtures, not through a live dependency on Collector. ## Final evidence -Record tested revisions, supported families/profiles, fixture results, dependency -graph checks, and compile/install/ingest measurements. Trace one query through its -semantic root, state definition, producer, flow and installed publication. Report -remaining capability gaps explicitly. Completion requires executable evidence, -not document publication, an open PR, or prior migration test counts. +Record tested revisions, the supported backend profile/state families, fixture +results, and dependency checks including tests/scripts. Trace a query through its +semantic root, materialization boundary, precompute producer and query reader. +Record compile/install/ingest measurements where extraction changes the path. +Completion requires the two-plan acceptance cases and zero ASAPCollector build/ +runtime dependency, not completion of the deferred distributed architecture. From daa52813d679b90cdba46197cf286120ce4ca1dc Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 02:51:42 +0000 Subject: [PATCH 04/15] docs: clarify window terminology migration --- .../design_docs/asapplanner-migration-plan.md | 52 ++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index f01ea50da..b068edb87 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -41,7 +41,7 @@ new distributed capabilities just because the local contract has changed. | Stage | Owner | Deliverable | Exit gate | | --- | --- | --- | --- | -| 1. Inventory and fixtures | Backend | Contract/dependency map and backend behavior baseline | Every scoped entry point and Collector import has a documented replacement or compatibility fixture | +| 1. Inventory, terminology and fixtures | Backend | Contract/dependency map, window-term audit and backend behavior baseline | Every scoped entry point and Collector import has a documented replacement or compatibility fixture; every window field has one stated semantic owner | | 2. Extract common libraries | Backend and shared-library maintainers | Runtime-independent contracts and typed sketch reconstruction | Backend and required tests build without ASAPCollector; existing decoding remains compatible | | 3. Bind and split two subplans | Backend compiler/runtime | Common bindings, catalog, maintenance/query subgraphs, explicit state boundaries | Executable ownership and provenance match supported semantics | | 4. Validate, install and visualize | Backend | Shared two-plan checks, local generation switching, two execution views | Invalid boundaries fail; readiness and recovery remain correct | @@ -68,6 +68,48 @@ may originate from Collector but must be usable without checking out or running Randomized sketches may need persisted fixtures and semantic assertions rather than comparing independently generated bytes. +### Window terminology and naming review + +[Issue #734](https://github.com/ProjectASAP/ASAPQuery-backend/issues/734) tracks +an existing ambiguity between the query window, the stored-state layout and the +runtime window scheduler. Treat them as three separate concepts: + +| Concept | Owner and meaning | Current representation | Migration decision | +| --- | --- | --- | --- | +| Query window semantics | Planner/query contract: which event-time range each result covers and when it is evaluated | `SummaryWindowFramework`, query lookback/window width and evaluation cadence | Preserve as semantic input to both physical subplans. A backend layout must implement it but must not redefine it. | +| Stored summary-state layout | PrecomputePlan: how state is partitioned and persisted so the query range can be reconstructed | `WindowMaterializationLayout` | Rename in code only through a versioned compatibility migration; target terminology is `SummaryStateLayout`. | +| Runtime window lifecycle | Precompute executor: when an in-memory state opens, closes, flushes or expires | `WindowKind` plus width/slide/lateness fields | Do not expose a second tumbling/sliding semantic choice in the new plan. Derive or validate runtime scheduling from the selected semantic window and state layout; retain `Session` only where it has an independently supported contract. | + +The proposed state-layout value names describe stored state rather than query +windows: + +| Current value | Precise meaning | Target code/documentation term | +| --- | --- | --- | +| `Pane { pane_secs }` | Store disjoint, mergeable states of `pane_secs`; QueryPlan combines enough panes to cover one requested result range | `DisjointPanes { pane_secs }` | +| `FullWindow` | Store one complete query-range state for each evaluation point | `PerEvaluationWindow` | +| `HierarchicalRollup { ... }` | Intended base panes plus coarser mergeable pane levels | `HierarchicalPanes`, only if an end-to-end producer, persistence and reader implementation is accepted | + +`Pane` does not mean “a tumbling query window.” A pane is a physical fragment; +a tumbling or sliding query window can use one or more panes. `FullWindow` does +not mean the query asks for a different window kind. It changes update fanout and +read composition while preserving the same query range and evaluation schedule. + +The current `HierarchicalRollup` variant has validation and rejection coverage, +but no supported end-to-end execution path. The new contracts must reject it as +an unsupported capability. Remove it if no committed implementation depends on +its serialized form; otherwise retain it only in a legacy decoding adapter. Do +not advertise it as an available physical realization. + +Before changing Rust names, inventory serialized fixtures, configuration files, +HTTP payloads and external consumers. If a rename proceeds, keep the existing +wire spelling as an input alias for the supported compatibility window, emit one +canonical spelling, and test old-input/new-output round trips. Do not combine the +rename with changes to durations, alignment, inclusivity, fanout or pane coverage. +Documentation and diagrams must label fields as **query range**, **evaluation +cadence**, **state layout**, **pane width**, **alignment origin**, **lateness** or +**retention**; the unqualified words “window” and “boundary” are insufficient +where more than one of these meanings is possible. + Input validation tests cover malformed framed payloads and currently supported sequence/checkpoint behavior where touched by extraction. Missing target features remain explicit gaps; do not turn this into a new transmission protocol project. @@ -140,6 +182,10 @@ Implement the [materialization boundary design](asapplanner-integration.md#execu identity registry or enumerate future stored pane instances during compilation. - Use execution timing, dependencies and bindings rather than operator names to determine ownership. Preserve absorbed operations in semantic-to-physical mapping. +- Bind query window semantics to exactly one supported state layout and derive a + single runtime schedule. Reject conflicting `SummaryWindowFramework`, + `WindowKind`, width, slide, pane or alignment combinations rather than choosing + one field as implicit authority. Switch maintenance execution to these subgraphs instead of discovering its work inside a complete query DAG. Full semantic provenance can remain an artifact or @@ -183,6 +229,10 @@ Acceptance cases: completion checks, recovery, query consistency and fallback behavior. - Old/new supported artifacts produce equivalent results and update counts. - Visualization agrees with executable ownership and retains provenance links. +- A tumbling and a sliding query can each use disjoint panes without changing + their query semantics; per-evaluation state produces the same covered ranges. +- Conflicting semantic-window, state-layout and runtime-schedule fields fail + validation, and unsupported hierarchical panes cannot reach activation. - These cases run without an ASAPCollector package, checkout or process. ## 5. Roll out and retire From e27bbb001bacb02be18ed02b1d3fbd545be58f20 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 02:56:50 +0000 Subject: [PATCH 05/15] Revert "docs: clarify window terminology migration" This reverts commit daa52813d679b90cdba46197cf286120ce4ca1dc. --- .../design_docs/asapplanner-migration-plan.md | 52 +------------------ 1 file changed, 1 insertion(+), 51 deletions(-) diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index b068edb87..f01ea50da 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -41,7 +41,7 @@ new distributed capabilities just because the local contract has changed. | Stage | Owner | Deliverable | Exit gate | | --- | --- | --- | --- | -| 1. Inventory, terminology and fixtures | Backend | Contract/dependency map, window-term audit and backend behavior baseline | Every scoped entry point and Collector import has a documented replacement or compatibility fixture; every window field has one stated semantic owner | +| 1. Inventory and fixtures | Backend | Contract/dependency map and backend behavior baseline | Every scoped entry point and Collector import has a documented replacement or compatibility fixture | | 2. Extract common libraries | Backend and shared-library maintainers | Runtime-independent contracts and typed sketch reconstruction | Backend and required tests build without ASAPCollector; existing decoding remains compatible | | 3. Bind and split two subplans | Backend compiler/runtime | Common bindings, catalog, maintenance/query subgraphs, explicit state boundaries | Executable ownership and provenance match supported semantics | | 4. Validate, install and visualize | Backend | Shared two-plan checks, local generation switching, two execution views | Invalid boundaries fail; readiness and recovery remain correct | @@ -68,48 +68,6 @@ may originate from Collector but must be usable without checking out or running Randomized sketches may need persisted fixtures and semantic assertions rather than comparing independently generated bytes. -### Window terminology and naming review - -[Issue #734](https://github.com/ProjectASAP/ASAPQuery-backend/issues/734) tracks -an existing ambiguity between the query window, the stored-state layout and the -runtime window scheduler. Treat them as three separate concepts: - -| Concept | Owner and meaning | Current representation | Migration decision | -| --- | --- | --- | --- | -| Query window semantics | Planner/query contract: which event-time range each result covers and when it is evaluated | `SummaryWindowFramework`, query lookback/window width and evaluation cadence | Preserve as semantic input to both physical subplans. A backend layout must implement it but must not redefine it. | -| Stored summary-state layout | PrecomputePlan: how state is partitioned and persisted so the query range can be reconstructed | `WindowMaterializationLayout` | Rename in code only through a versioned compatibility migration; target terminology is `SummaryStateLayout`. | -| Runtime window lifecycle | Precompute executor: when an in-memory state opens, closes, flushes or expires | `WindowKind` plus width/slide/lateness fields | Do not expose a second tumbling/sliding semantic choice in the new plan. Derive or validate runtime scheduling from the selected semantic window and state layout; retain `Session` only where it has an independently supported contract. | - -The proposed state-layout value names describe stored state rather than query -windows: - -| Current value | Precise meaning | Target code/documentation term | -| --- | --- | --- | -| `Pane { pane_secs }` | Store disjoint, mergeable states of `pane_secs`; QueryPlan combines enough panes to cover one requested result range | `DisjointPanes { pane_secs }` | -| `FullWindow` | Store one complete query-range state for each evaluation point | `PerEvaluationWindow` | -| `HierarchicalRollup { ... }` | Intended base panes plus coarser mergeable pane levels | `HierarchicalPanes`, only if an end-to-end producer, persistence and reader implementation is accepted | - -`Pane` does not mean “a tumbling query window.” A pane is a physical fragment; -a tumbling or sliding query window can use one or more panes. `FullWindow` does -not mean the query asks for a different window kind. It changes update fanout and -read composition while preserving the same query range and evaluation schedule. - -The current `HierarchicalRollup` variant has validation and rejection coverage, -but no supported end-to-end execution path. The new contracts must reject it as -an unsupported capability. Remove it if no committed implementation depends on -its serialized form; otherwise retain it only in a legacy decoding adapter. Do -not advertise it as an available physical realization. - -Before changing Rust names, inventory serialized fixtures, configuration files, -HTTP payloads and external consumers. If a rename proceeds, keep the existing -wire spelling as an input alias for the supported compatibility window, emit one -canonical spelling, and test old-input/new-output round trips. Do not combine the -rename with changes to durations, alignment, inclusivity, fanout or pane coverage. -Documentation and diagrams must label fields as **query range**, **evaluation -cadence**, **state layout**, **pane width**, **alignment origin**, **lateness** or -**retention**; the unqualified words “window” and “boundary” are insufficient -where more than one of these meanings is possible. - Input validation tests cover malformed framed payloads and currently supported sequence/checkpoint behavior where touched by extraction. Missing target features remain explicit gaps; do not turn this into a new transmission protocol project. @@ -182,10 +140,6 @@ Implement the [materialization boundary design](asapplanner-integration.md#execu identity registry or enumerate future stored pane instances during compilation. - Use execution timing, dependencies and bindings rather than operator names to determine ownership. Preserve absorbed operations in semantic-to-physical mapping. -- Bind query window semantics to exactly one supported state layout and derive a - single runtime schedule. Reject conflicting `SummaryWindowFramework`, - `WindowKind`, width, slide, pane or alignment combinations rather than choosing - one field as implicit authority. Switch maintenance execution to these subgraphs instead of discovering its work inside a complete query DAG. Full semantic provenance can remain an artifact or @@ -229,10 +183,6 @@ Acceptance cases: completion checks, recovery, query consistency and fallback behavior. - Old/new supported artifacts produce equivalent results and update counts. - Visualization agrees with executable ownership and retains provenance links. -- A tumbling and a sliding query can each use disjoint panes without changing - their query semantics; per-evaluation state produces the same covered ranges. -- Conflicting semantic-window, state-layout and runtime-schedule fields fail - validation, and unsupported hierarchical panes cannot reach activation. - These cases run without an ASAPCollector package, checkout or process. ## 5. Roll out and retire From 22307957605bc8641b7b4103fe64719b5b4d715a Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 03:16:20 +0000 Subject: [PATCH 06/15] docs: focus physical plan and SDS designs --- docs/design_docs/README.md | 14 +- docs/design_docs/asapplanner-integration.md | 887 ++++-------------- .../design_docs/asapplanner-migration-plan.md | 349 +++---- .../summary-catalog-sds-architecture.md | 725 +++++--------- 4 files changed, 557 insertions(+), 1418 deletions(-) diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 0e461bf4b..ad43475e4 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -4,15 +4,15 @@ These documents are for architects and developers. The integration proposal and SDS model below define the target Planner-to-runtime boundary; their current-code notes and migration gates distinguish implemented behavior from proposed changes. -- [Planner, physical plans, SDS, and runtime architecture](asapplanner-integration.md) - owns semantic/physical compilation, common bindings, the four plan projections, - policy ownership, codec boundaries, and publication/activation requirements. +- [Planner output to backend physical plans](asapplanner-integration.md) defines + how one selected semantic DAG becomes executable PrecomputePlan and QueryPlan + subgraphs joined at materialization boundaries. - [Summary Catalog and SDS](summary-catalog-sds-architecture.md) owns descriptors, - definition/instance identity, state references, inventory and lifecycle semantics. + definition/materialization/instance identity, state references, readiness and + lifecycle semantics. - [Architecture migration delivery plan](asapplanner-migration-plan.md) defines - the current PrecomputePlan/QueryPlan scope, common-library extraction, removal - of ASAPCollector dependencies, and backend acceptance/retirement gates. Collector - and transmission plan changes are deferred. + common-library extraction, removal of ASAPCollector dependencies, the two-plan + rollout, and backend acceptance/retirement gates. - [Accepted-input completeness](continuous-summary-completeness.md) describes the backend's bounded admission, publication and recovery behavior. diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index d6f5508b8..24df0a941 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -1,721 +1,206 @@ -# Planner, physical plans, SDS, and runtime architecture +# Planner output to backend physical plans -## Audience, status, and scope +Status: proposed backend architecture. Audience: developers changing the +Planner-to-backend compilation and execution boundary. -Audience: architects and developers of ASAPPlanner, ASAPQuery-backend, and -ASAPCollector. This document defines the target integration architecture. -The current-code baseline below is separate from the proposed changes; writing -this design does not establish runtime support or change a wire contract. +## Scope -This document owns the integration boundary and compilation flow. The -[SDS design](summary-catalog-sds-architecture.md) owns descriptor, instance, and -state-lifecycle semantics. The [delivery plan](asapplanner-migration-plan.md) -owns implementation gates. Existing -[Collector system contracts](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs/design_docs) -remain the compatibility baseline until corresponding changes land in both -consumers. Conflicts require a versioned migration, not unilateral reinterpretation. +This document defines how one selected ASAPPlanner semantic DAG becomes two +backend-executable plans: -## Current implementation scope +- **PrecomputePlan** produces and maintains stored summary state. +- **QueryPlan** reads stored state and computes query results. -The [migration delivery plan](asapplanner-migration-plan.md) currently implements -only the backend PrecomputePlan/QueryPlan split and extraction of their common -contracts/codecs. It requires no backend build/runtime dependency on ASAPCollector. -CollectorPlan, TransmissionPlan, Collector adoption and distributed rollout are -future work, not prerequisites. The four-plan architecture below remains the -longer-term design; its distributed acceptance requirements do not enlarge this -iteration's completion gate. +The two plans share catalog identities and state contracts defined by the +[Summary Catalog and SDS design](summary-catalog-sds-architecture.md). The +[migration plan](asapplanner-migration-plan.md) describes how to reach this +architecture from the current implementation. -## Problem and current baseline +CollectorPlan and TransmissionPlan are outside the current implementation scope. +They may become additional projections of the same selected decision later, but +the backend migration must neither redesign them nor depend on ASAPCollector. -Collector and backend must agree on what a summary means, how it is produced, -how updates travel, and how queries consume it. Sharing an envelope decoder -alone does not guarantee agreement across these boundaries. +## Problem -The inspected backend baseline is `b06385d1c155986c05ccbd011978e43bf3786deb`. -The following are current implementation facts, not the desired dependency graph: +The current `PrecomputePlan.executable_dags` can contain the complete selected +semantic DAG. For a query such as: -| Area | Existing foundation | Remaining coupling | -| --- | --- | --- | -| Compilation | `CompiledPhysicalPlan` contains catalog, query, precompute, collector, and transmission plans | Transmission compilation reads producers/schemas from PrecomputePlan; catalog is constructed from materializations and then bound back into plans | -| Publication | `PhysicalPlanPublication` validates related plans; backend supports staging/activation | Shared publication validation and runtime installation repeat some cross-plan checks | -| Contracts | `asap_types` contains SDS and installed plan types | Types still depend on Planner representations; Collector maintains separate Go/Rust DTOs | -| Semantic DAG | Planner exports a versioned DAG; backend retains node bindings | `OwnedPostAsapDag` serializes payloads into JSON to avoid process-local `Rc` ownership | -| Runtime policy | Transmission rules carry sampling, delta/GOS, and adaptation | Production semantics and transport controls share one policy structure | -| Sketch ingest | Shared sketch library plus an edge-runtime adapter | Backend imports Collector wrappers for DDSketch/KLL reconstruction; other reconstruction and delta paths remain local | - -Implementation references: [compiler](../../control_plane/src/physical/compiler.rs), -[publication](../../crates/asap_types/src/plan_publication.rs), -[producer contracts](../../crates/asap_types/src/producer_plan.rs), -[installed DAG](../../crates/asap_types/src/executable_plan.rs), and -[edge adapter](../../data_plane/src/precompute_engine/operators/edge_runtime_adapter.rs). - -## Goals and non-goals - -The minimum outcome is one selected semantic decision, one set of physical -bindings, and four consistent runtime projections. Both backend-local and -Collector-produced summaries must use this boundary. Backend ingest must no -longer depend on the Collector execution runtime for shared state codecs. - -Preserve supported query semantics, sharing, legacy decoding, completeness, -and recovery behavior during extraction. Neither arbitrary PromQL coverage, -a new optimizer, a universal execution engine, an arbitrary network topology, -nor a repository reorganization is required. A missing capability remains an -explicit rejection or configured exact fallback. - -## Inputs, outputs, and end-to-end behavior - -Inputs are canonical workload roots, query accuracy and freshness requirements, -Planner alternatives, and scoped deployment evidence: capabilities, topology, -source bindings, retained-state availability, and complete cost estimates. -The output is one validated `PhysicalPlanPublication` and target-specific -installation artifacts derived from it. - -```mermaid -flowchart TD - W[Canonical workload and requirements] --> P[ASAPPlanner semantic alternatives] - E[Capabilities, topology, costs, observed SDS] --> C[Control-plane physical compiler] - P --> C - C -->|Feasibility and costs for candidate selection| P - C --> B[Selected decision: catalog and common physical bindings] - B --> Q[QueryPlan] - B --> M[PrecomputePlan] - B --> L[CollectorPlan per target] - B --> T[TransmissionPlan] - Q --> U[PhysicalPlanPublication] - M --> U - L --> U - T --> U - U --> V[Validate, stage, coordinate activation] - V --> R[Collector and backend runtimes] - R --> O[Observed inventory, readiness, accuracy, costs] - O --> E +```text +Input -> Sum -> KLL -> SummaryEstimate -> QueryResult ``` -1. Planner produces legal semantic alternatives, retaining shared producers and - distinct query roots. Physical evaluation supplies feasibility and costs. -2. The control plane commits a feasible alternative and its concrete realization. -3. The compiler assigns catalog identities and binds semantic nodes, state, - producers, consumers, and data-flow edges once. -4. It projects those bindings into the four plans and validates the publication. -5. Targets stage their projections and required catalog content. The coordinator - authorizes activation only after the required target acknowledgements. -6. Producers maintain state; receivers apply authorized frames; queries use one - active plan snapshot and states with sufficient coverage and provenance. -7. Runtime evidence is attributed to those bindings and generations. A new - semantic choice returns to planning rather than changing query behavior locally. - -Plan installation and state readiness are separate. A query with missing or -incomplete state follows its configured exact route or returns an explicit -unavailable result; it cannot interpret missing state as an empty population. - -## Planner and compiler ownership - -Planner owns semantic equivalence, source/population semantics, grouping, -logical windows, summary families and parameters, result guarantees, lifecycle -choices, and maintenance-time versus read-time dependencies. Reusable sharing, -fusion, and rollup rules belong there. - -The physical compiler owns concrete implementations, placement, input routing, -state layout, retention realization, runtime identifiers, codecs, transmission -configuration, and deployment commitment. It must prove that an implementation -preserves the selected semantic decision. An unsupported choice returns to -candidate selection or fails explicitly; lowering cannot silently change its -window, sampling semantics, statistic, or guarantees. - -Capabilities and costs are distinct. A cheap implementation is not necessarily -feasible. Costs include shared construction once, maintenance, retained memory, -network, storage, recovery/checkpoints, per-consumer merges and readouts, and -query demand over the same horizon. Missing or stale evidence is not zero cost. - -The semantic IR export must be typed, versioned, and independent of internal -search ownership such as `Rc`. The target is one export contract shared by -Planner and consumers, with backend physical bindings alongside it. Migrate -`OwnedPostAsapDag` only after round-trip and runtime compatibility are proven; -do not introduce another operator language or require runtimes to import the -optimizer. Runtime evaluation of installed operators remains legitimate. - -## Caller contract and lifecycle completeness - -[Planner issue #438](https://github.com/ProjectASAP/ASAPPlanner/issues/438) -identifies a separate interface requirement: callers need to know the required -inputs, the consequences of omissions, and the promises of each output. A shared -DAG format alone does not meet that requirement. - -At the inspected Planner revision -[`e7fdb2492c42c9f5b34760706a5162aa586d3025`](https://github.com/ProjectASAP/ASAPPlanner/tree/e7fdb2492c42c9f5b34760706a5162aa586d3025), -plain materialization and lifecycle-aware selection/materialization are separate -library operations. `materialize_with_summary_maintenance_lifecycles` attaches -state deployments; `export_summary_maintenance_plan` exports their decisions, -alternatives and costs alongside the graph. Thus, the existence of an exported -DAG does not certify that lifecycle selection or complete deployment costing ran. -This observation does not imply that the backend's pinned Planner revision -already exposes every API from that revision. - -### Current public API audit - -The following describes the inspected Planner revision above, rather than the -proposed facade. These are library operations, not equivalent end-user workflows. -See [replacement APIs](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/asap-aware-mapping/src/replacement.rs), -[lifecycle APIs](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/asap-aware-mapping/src/summary_maintenance_lifecycle.rs), -[workload types](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/types/src/workload.rs), and -[cost model](https://github.com/ProjectASAP/ASAPPlanner/blob/e7fdb2492c42c9f5b34760706a5162aa586d3025/crates/asap-aware-mapping/src/cost_model.rs). - -| Operation | Input and output | What it does not establish by itself | -| --- | --- | --- | -| `search_workload` / `search_workload_with` | Canonical roots, default/explicit strategies -> `PlanSpace` | A selected deployment, workload lifecycle, or application-specific end-to-end accuracy target | -| `search_workload_with_targets` | Roots, strategies, per-root targets and accuracy model -> target-checked candidate space | Physical feasibility, lifecycle commitment or measured deployment cost | -| `PlanSpace::global_selection` | Candidate space and cost model -> structural `GlobalSelection` | Recurrence-aware or lifecycle-aware selection | -| `global_selection_with_recurrence` | Candidate space, cost model, recurrence profiles and optional horizon -> selection/error | Selected state lifecycle commitments | -| `global_selection_with_summary_maintenance_lifecycles` | Candidate space, workload/root associations, time, horizon, capabilities and cost model -> selection/error | Successful physical installation or ready state | -| `GlobalSelection::materialize` | A selected target -> optional semantic summary root/error | Executed summary data or a lifecycle deployment record; “materialize” here constructs IR | -| `plan_summary_maintenance_lifecycles` | An already materialized root plus demand/context -> lifecycle plan/error | Re-ranking all original semantic alternatives | -| `materialize_with_summary_maintenance_lifecycles` | Selection, target and lifecycle context -> optional lifecycle plan/error | A backend physical publication; callers must inspect decisions and available evidence | -| `export_summary_maintenance_plan` | Lifecycle plan -> serializable graph plus deployment/cost information | Any additional optimization, validation or runtime execution | - -A late lifecycle pass can evaluate a fixed root but does not retroactively make -an earlier structural selection lifecycle-optimal. A deployment flow must include -lifecycle feasibility/costs before its final candidate commitment. Likewise, -constructing `QueryRequirements` is not enough if a caller then invokes a low-level -search function that never receives those requirements. The orchestrator must -thread per-root targets into the target-aware path. - -Concrete defaults have different meanings: - -| Current Rust default/omission | Actual behavior | Consequence for integration | -| --- | --- | --- | -| `QueryRequirements::default()` | Implicit exact accuracy; unspecified response latency | Approximation requires explicit permission; no response-time bound is supplied | -| `DataWorkload::default()` | Unknown arrival and unknown evidence values | Does not assume data at rest, zero updates or a measured distribution | -| `Evidence::default()` | No value; unknown source | Missing/freshness-invalid evidence cannot establish a cost or empirical guarantee | -| `SummaryMaintenanceLifecycleCapabilities::default()` | All four runtime lifecycle flags true | This is not capability detection; adapters must pass truthful support explicitly | -| Default per-summary maintenance capabilities | Incremental update, merge and delete flags false | Runtime lifecycle support does not imply the algorithm/state representation supports its required operations | -| Default lifecycle cost inputs | All primitive costs unknown | Default structural costing does not supply a fully costed lifecycle deployment | -| Lifecycle horizon `None` | Horizon-dependent alternatives remain unselectable | One-time/rate comparisons cannot assume an arbitrary amortization horizon | -| `search_workload()` | Built-in strategies and `DefaultCostModel` | Useful for candidate exploration; ranking is not calibrated to the target deployment | - -`DefaultCostModel` preserves built-in algorithm order/sizing and uses structural -cost hooks. Custom models and evidence must be supplied for deployment-specific -claims, including candidate generation where strategies consume them, not only -for a final sort. Rust `Default` implementations are not automatically JSON/YAML -omission defaults: several required fields have no `serde(default)`. API/adapter -normalization must document serialized omission behavior separately. - -### Who controls what - -Application users control query meaning, permitted approximation, workload intent, -and any latency/resource objectives. They should not select internal passes or -assert unsupported runtime capabilities. An explicit application profile can -supply documented defaults, but normalization must report them. - -Runtime integrators supply source/type binding, capabilities, available lifecycle -actions, current state inventory, measured cost/evidence providers and the planning -time/horizon policy. They implement physical lowering and execution. Planner -extension developers supply replacement strategies, cost/accuracy models and -capability implementations. Restricting strategies narrows search opportunity; -it must not bypass semantic/accuracy checks. These are distinct control surfaces, -not a requirement for every user to configure every library parameter. - -### User guide for entry points, exit points, and controls - -The API audit above is architecture evidence, not a replacement for a Planner -user guide. Partial workflows are legitimate uses: a frontend author may need only -pre-ASAP IR, a strategy author may inspect candidate alternatives, and an embedding -application may consume a selected semantic DAG. None must invoke deployment -planning merely to make its intermediate output useful. - -ASAPPlanner should own a user guide organized by intended result, with one worked -example for each supported path: - -| User intent / exit artifact | What the guide must establish | +`Input -> Sum -> KLL` is maintenance work. `SummaryEstimate -> QueryResult` is +query-time work. Storing the complete DAG under PrecomputePlan makes ownership +unclear even when bindings prevent query-time nodes from running during +maintenance. It also makes a PrecomputePlan visualization look as though +`SummaryEstimate` executes while state is being built. + +The target design records one materialization boundary and derives two explicit +executable subgraphs. Semantic provenance remains available without placing +query-only operators in PrecomputePlan. + +## Inputs and outputs + +The physical compiler consumes: + +- selected Planner DAG roots and their query associations; +- query requirements, including accuracy and response constraints; +- complete lifecycle commitments for the supported backend mode; +- backend capabilities and concrete implementation evidence; +- catalog, schema and deployment-generation inputs. + +Capabilities restrict the choices the Planner may consider. For example, a +backend that can only build summaries from data at rest advertises only that +lifecycle. The Planner still models other lifecycle modes, but it must not select +one the backend cannot execute. + +The compiler produces one coherent backend publication: + +| Output | Responsibility | | --- | --- | -| Parse and bind a workload into pre-ASAP IR | Supported frontend entry point, source/schema inputs, normalization and semantic checks actually performed | -| Generate post-ASAP candidates | Input IR, strategy configuration, automatically added passes, models consulted during generation, and candidate/rejection output | -| Rank/select and materialize a semantic DAG | Applicable cost/accuracy models, legality checks, selection scope and assumptions; distinguish structural from recurrence-aware selection | -| Plan summary lifecycles | Runtime and per-family capabilities, workload/time evidence, fixed versus searched lifecycle choices, and deployment commitments returned | -| Export a result | Which export preserves which decisions/evidence, schema version and what serialization does not validate | -| Compile and deploy in ASAPQuery | The handoff to the separate physical compiler and its completeness requirements; not another Planner execution API | - -For each path, document exact callable APIs at a supported revision, required and -optional inputs, Rust versus serialized defaults, customization points, returned -artifacts, checks performed, checks not performed, and valid next steps. Include -examples that stop at that exit point. Do not present every technically callable -combination as a supported workflow or infer guarantees from a type's name. - -Explain four separate control surfaces: optimization strategy policy (which -alternatives to explore), model/evidence providers (how to estimate and compare), -runtime capabilities (what is executable), and requirements (what is acceptable). -Strategy configuration must disclose automatically applied behavior: the current -`search_workload_with` also derives workload-dependent rollup internally, so its -explicit strategy list is not a complete enable/disable switch. Models used during -candidate generation must be distinguished from models supplied only at selection. -Disabling an optimization narrows opportunities; it does not disable correctness -checks or relax requirements. Missing evidence must remain explicit. - -Document today's composable APIs first. A unified application facade is a separate -interface improvement, not a prerequisite for explaining existing entry/exit -points. Its eventual explain output should identify effective strategies, automatic -passes, model versions, resolved defaults, unsupported choices, and rejected -candidates. Keep the current API reference, user recipes, and proposed facade -clearly separated so a design proposal is never mistaken for runnable guidance. - -### One supported application workflow - -For this backend, the target is one application-facing deployment-planning -request/result contract. This is a proposed orchestration boundary, not an -existing new Planner API. Its orchestrator -normalizes inputs, enumerates semantic and lifecycle alternatives, obtains physical -feasibility/cost evidence, validates guarantees, and returns the selected decision -with its evidence. Callers should not need to assemble those stages manually. -Planner supplies reusable semantic search, legality and ranking; the backend -owns application orchestration, physical evaluation and deployment commitment. -Planner's primary output remains `PlanSpace` plus ranked candidates, as defined -in its [design overview](https://github.com/ProjectASAP/ASAPPlanner/blob/main/docs/design_docs/README.md). -The downstream system may feed complete physical evidence back into Planner and -use `global_selection*` as a compatible-choice helper. A selected decision is -required at the physical compilation boundary, not at every legitimate Planner -exit point. Publication remains a separate backend operation, not a side effect -of invoking Planner. The user-facing entry/exit guide is tracked separately in -[Planner PR #440](https://github.com/ProjectASAP/ASAPPlanner/pull/440). - -Required stages are semantic normalization/validation, constraint checking, -capability filtering, and recording a complete selected decision (including -applicable lifecycle). Alternative search and ranking can collapse to validation -when only one candidate is legal. Empirical evidence, extra rewrite strategies, -and inventory reuse can be omitted only with the documented reduction in search -or guarantees. Serialization is needed only at a process/persistence boundary. - -Low-level APIs may remain available for research, candidate inspection and tests. -Their intermediate results must be distinguished from a complete planning result -and rejected by the production compilation boundary when commitments are missing. -This is one supported deployment workflow with explicit diagnostics, not several -undocumented combinations of optional optimization passes. - -### Inputs and omission rules - -The following are target normalization rules. They do not document current Rust -field defaults, which must be audited during migration. Every resolved default, -its source, and its effect on the available alternatives must appear in diagnostics. - -| Input | Supplied by | Requirement and consequence of omission | -| --- | --- | --- | -| Query roots and resolved source/type semantics | Caller/frontend | Required; ambiguous source or type information is an error | -| Accuracy requirement and evaluation scope | Caller or named application profile | Must resolve explicitly; omission grants no permission for approximate answers. A profile may specify exactness as its default | -| Query demand: one-time/repeating/unknown, cadence and time scope | Caller/workload registry | Required for workload-dependent decisions; unknown demand cannot be treated as zero demand or assumed future reuse | -| Optimization horizon | Caller or explicit profile | Required when comparing one-time costs with rates or amortized reuse; absent horizon prevents those comparisons, not semantic DAG inspection | -| Data arrival/update facts and cost evidence | Deployment evidence provider | Required for affected lifecycle/cost comparisons; missing evidence cannot be priced as zero or infer continuous ingestion from repeating queries | -| Runtime capabilities and allowed lifecycle actions | Physical provider | Required for a deployment candidate; absence cannot mean universal support | -| Existing summary inventory | Runtime/provider | Optional for considering new construction; omission means no existing-state reuse may be assumed | -| Empirical distribution/accuracy evidence | Optional evidence provider | Without it, consider only alternatives justified by available theoretical guarantees and costs; do not invent an empirical fit | -| Latency/resource limits | Caller or profile | Omission establishes no numerical bound or compliance claim; runtime feasibility checks still apply | - -The user controls workload intent and requirements. Runtime capabilities and -observed evidence are supplied by their authoritative providers, not arbitrary -user overrides. An unavailable optional optimization may reduce the candidate -set; an unavailable required guarantee or deployment fact yields an explicit -incomplete/infeasible result. No omission silently weakens correctness. - -### Output and lifecycle obligations - -A complete selected result includes the semantic DAG and query roots, stable -references to shared summary producers, a lifecycle commitment for each stateful -materialization, declared guarantees/assumptions, capability and cost evidence -references, normalized input/default diagnostics, and structured rejection reasons -for relevant alternatives. These may be separate typed fields in one result; -do not overload the semantic DAG with placement or wire-delivery configuration. - -Lifecycle completeness specifies whether state is built on demand, prepared, -reused, or maintained, together with its maintenance mode, evaluation schedule, -and output representation. Every stateful deployment needs this commitment. -Planner models possible lifecycles; it does not require every runtime to -implement them. For a particular deployment, the candidate set is the intersection -of modeled lifecycles, runtime capabilities, workload legality, and application -policy. Unsupported modes are excluded before ranking, not merely assigned a -higher cost. An application profile may further restrict runtime support but -cannot grant capabilities the runtime lacks. - -For example, a backend may support only building a summary directly from data at -rest, with no incremental maintenance. Planner then considers only compatible -direct-build alternatives. It cannot select continuously maintained incremental -state, even for a recurring query. Recurrence may justify repeated full builds, -but does not create an incremental-update capability. Prepared or retained reuse -is eligible only if the runtime separately supports those actions and the workload -permits them; direct-build support alone does not imply either. - -If these constraints leave one legal lifecycle, selection is degenerate: validate -and record that commitment, without searching other lifecycle modes. This remains -a complete lifecycle decision, not an incomplete plan. Build/update mode, execution -schedule, and retention/reuse are distinct dimensions, so direct build alone does -not specify the whole lifecycle. The result records the applicable choices and -assumptions; required cost comparisons use only eligible alternatives. An empty -candidate set produces an explicit infeasible result or a separately supported -raw-execution alternative. A stateless or selected raw-recomputation path can mark -state lifecycle as not applicable. Neither case means an unresolved stateful DAG -is deployable. - -Lifecycle choices affect cost ranking and phase legality, so they must participate -before final selection; attaching an arbitrary lifecycle after choosing a winner -cannot establish that the winner is feasible or cost-preferred. A diagnostic DAG -without this step promises only the checks actually performed. It does not promise -state readiness, maintenance cost, deployment feasibility, or an optimized lifecycle. -Even a complete Planner result is not an installed physical publication: the -compiler must preserve its commitments and validate all runtime projections. - -Acceptance for #438 requires a documented input/default matrix, one supported -application workflow, and examples for a one-shot query, a recurring query, an -unknown-demand request, a data-at-rest-only runtime with a singleton legal -lifecycle, and a missing-cost/capability case. Each example must show -the returned status, decisions, omissions and guarantees. Compilation must reject -an unresolved lifecycle for stateful deployment. No new facade is claimed to -exist until these examples exercise the actual public API. - -## Bind once, project four plans - -Use a compiler-internal common binding structure to record: - -- Semantic node to physical task mappings, including expansion into multiple tasks. -- Summary definitions, producer partitions, state schemas, window implementations, - and storage/input/output bindings. -- Data-flow edges with their endpoints and transmission requirements. -- Selected production and transmission policies with guarantee evidence. - -This structure addresses repeated decisions currently inferred from a backend -plan. It is not a fifth public plan or a second optimizer IR. Preserve semantic -node provenance and shared producers; one physical producer can serve multiple -query roots without inheriting a particular query's identity. - -| Projection | Responsibility | Principal contents | +| Summary Catalog/SDS entries | Define summary semantics, materialization identity, state schema and state references | +| PrecomputePlan | Execute maintenance subgraphs that terminate in stored-state writes | +| QueryPlan | Execute materialization reads, query-time summary operators and exact residuals | +| Provenance mapping | Relate physical nodes and state references to the selected semantic DAG | + +These outputs are derived from the same compiler bindings. They must not make +independent choices about summary semantics, grouping, windows or schemas. + +## Ownership + +| Layer | Owns | Does not own | | --- | --- | --- | -| CollectorPlan | Execute maintenance assigned to an edge target | Inputs, maintenance tasks, producer/partition identity, window implementation, production policy, output bindings | -| PrecomputePlan | Execute backend maintenance and manage state | Raw-input build, remote-state integration, derived summaries, storage, retention and recovery bindings | -| TransmissionPlan | Deliver state across execution locations | Producer/consumer endpoints, schema, encoding, frame semantics, sequence/epoch, checkpoints, cadence and recovery policy | -| QueryPlan | Read and compose results | State bindings, merge/readout, exact residuals, window boundary handling, completeness requirements and fallback | - -CollectorPlan and PrecomputePlan may use the same maintenance operator contract -with different executors. They do not need one shared scheduler or implementation. -Derive TransmissionPlan from remote data-flow edges, not from PrecomputePlan. -Initially support the existing Collector-to-backend edges; a backend-local -profile has no remote-summary transmission rules and requires no Collector. -Raw Remote Write ingestion remains an input adapter, not a fabricated summary flow. - -Build the catalog and common bindings before projecting runtime plans. Each plan -references immutable catalog definitions instead of independently choosing -algorithm, population, or logical window. Concrete pane layouts and execution -bindings remain physical choices constrained by those definitions. - -A self-contained Collector installation artifact can embed the relevant catalog -subset and transmission rules. These are mechanically derived copies from one -publication, validated against its identity/digest. They are not independently -editable authorities. Runtimes need no catalog network lookup on each update. - -### Executable subgraphs and materialization boundaries - -**Decision:** PrecomputePlan and QueryPlan each own the operations they execute. -The physical compiler explicitly splits the selected DAG at materialization -boundaries and records the state references connecting the subplans. There can -be multiple boundaries: one query can consume several summaries and several -queries can share the same summary. - -Today, `PrecomputePlan.executable_dags` stores complete `InstalledPostAsapDag` -documents, including read-time nodes such as `SummaryEstimate`. Bindings mark -execution ownership, and the maintenance runtime evaluates dependencies of -`precompute_sinks` rather than every stored node. This explains current behavior -but is a mismatch between the PrecomputePlan abstraction and its contents. -The target removes query-only operations from its executable representation. +| ASAPPlanner | Semantic candidates, legality, accuracy reasoning and selection among advertised capabilities | Backend state IDs, storage schema or runtime installation | +| Physical compiler | Concrete implementation commitment, subgraph split, catalog bindings and plan generation | Re-optimizing a selected DAG at query time | +| Precompute runtime | Executing installed maintenance nodes and publishing state | Query result operators or selecting a different materialization | +| Query runtime | Reading bound state and executing installed query nodes | Creating missing summaries or searching the catalog for alternatives | +| SDS/catalog | Identity, schema, state references, readiness and lifecycle metadata | Operator scheduling or candidate ranking | + +## Executable subgraphs and materialization boundaries + +The compiler first binds every selected summary-producing node to one +materialization definition. It then cuts the selected DAG at stored-state +boundaries. ```mermaid flowchart LR - subgraph PP[PrecomputePlan] - I[Input] --> B[Build or update summary] - B --> W[Materialize summary S] - end - W -. State reference S .-> R - subgraph QP[QueryPlan] - R[Read summary S] --> E[SummaryEstimate] - E --> O[Query result] - end + subgraph P[PrecomputePlan] + I[Input] --> S[Sum] + S --> K[Build KLL] + K --> W[Write state] + end + W -->|materialization ID + schema| R + subgraph Q[QueryPlan] + R[Read state] --> E[SummaryEstimate] + E --> O[Query result] + end ``` -The dashed connection is a state dependency, not a claim that every query triggers -a synchronous precompute execution. State must satisfy the installed schema, -coverage and readiness requirements when read. +PrecomputePlan contains: -A boundary reuses the existing catalog identities and materialization bindings: +- source reads accepted by the maintenance runtime; +- exact or summary operators needed to produce stored state; +- reads of completed prior state for supported derived summaries; +- explicit stored-state sinks. -| Information | Purpose | -| --- | --- | -| Summary definition reference | Producer and reader identify the same logical summary | -| State schema and representation | Reader interprets the produced state correctly | -| Window, phase and grouping contract | Read covers the intended population and interval without double counting | -| Publication/catalog generation | Prevent incompatible installed plans and state from being combined | -| Semantic node provenance | Relate physical production/read operations to the selected Planner computation | - -These are required relationships, not a new duplicate identity registry. Reuse -`MaterializationBinding`, state-schema contracts and catalog references where they -already express the relationship. Concrete stored instances are resolved at -runtime from the definition, extent, group and accepted generation; compilation -does not allocate every future pane instance. - -The compiler extracts subgraphs using execution timing, dependencies and explicit -materialization bindings. It must not split by operator name alone. Precompute -subgraphs terminate at materialization sinks and can read prior materializations -to derive new summaries. Query subgraphs start at state reads or explicit exact -inputs and perform read-time operations. In the current semantic contract, -`SummaryEstimate` is read-time and belongs in QueryPlan; maintenance-time exact -finalization is a distinct permitted operation when its input contract is met. -Unsupported phase crossings fail compilation rather than silently moving work. - -The complete semantic DAG can remain as publication-level provenance or a compiler -artifact, with semantic-to-physical mappings. It is not executable content owned -by PrecomputePlan and need not be a third visualization section. Runtime plans -must contain their required execution information without traversing query-only -provenance to discover maintenance work. - -### Meaning of maintenance and current binding labels - -Precompute names the backend plan/engine that produces and maintains summary -state. Maintenance names the execution phase that builds, updates or derives that -state rather than answering a query. It includes initial batch construction and -full rebuilds; it does not imply incremental or continuous ingestion. - -The current binding enum classifies semantic nodes as follows. These names remain -unchanged by this documentation proposal: - -| Binding | Meaning | -| --- | --- | -| `Materialization` | Maintenance-time node explicitly bound to a stored summary definition | -| `MaintenanceInput` | Maintenance-time source or intermediate operation without its own stored-summary binding | -| `Query` | Read-time node explicitly mapped to a QueryPlan node | -| `QueryInput` | Read-time node without a separate explicit QueryPlan mapping, such as an operation absorbed by a larger query operation | - -`MaintenanceInput` is not a data format or necessarily a leaf. For example, in a -supported derived-summary pipeline, stored exact Sum/Count state can be finalized -into average-valued rows and then aggregated into a stored KLL. The finalization -is a maintenance intermediate without its own stored-summary binding; the stored -states have materialization bindings. An inner aggregate is not automatically a -`MaintenanceInput`: if its state is separately materialized, it is a -`Materialization`. Execution still requires the appropriate immutable-input and -runtime capability checks. - -Similarly, an ASAP-side descending Sort followed by Limit can lower to one -`TopKSelection` QueryPlan node. Limit maps to that node; the absorbed Sort can be -`QueryInput`. Absorption does not mean the sorting is omitted. Current binding -labels alone are not executable subgraphs; the new compiler projection makes -ownership and boundary reads explicit. - -### Visualization contract - -The default execution visualization has separate PrecomputePlan and QueryPlan -views, connected by labeled summary references. It shows each subplan's actual -operations, input/output boundaries, shared materializations, and generation. -Multiple query consumers must refer to the same shared summary rather than -suggesting duplicate maintenance. Derived-summary chains remain visible inside -the maintenance view with their state-read boundaries. - -While rendering the legacy serialized format, distinguish embedded semantic -context from operations executed by that plan. A read-time `SummaryEstimate` -embedded in PrecomputePlan must be visible in a faithful artifact view and marked -as query-owned context, never depicted as precompute execution. A projected -execution view may exclude that context only when it explicitly says it is showing -the execution projection. The user should not need a separate Semantic Plan page -to understand either subplan. After migration, the executable artifacts and their -two execution views should agree directly. - -## SDS, state codecs, and transmission - -| Contract | Authority | -| --- | --- | -| SDS descriptors and catalog | Meaning, source/population, fidelity, logical definition and compatible state schema | -| SDS instance/inventory | Concrete extent, groups, provenance, completeness, lifecycle and opaque state reference | -| TransmissionPlan | Authorized state flow between endpoints and its delivery/application rules | -| Sketch library codec | Full-state/delta byte representation, reconstruction and supported state operations | -| Runtime | Scheduling, durable admission/application, storage and serving | - -An envelope is not the entire SDS model. Keep payload bytes out of the desired -catalog and observed metadata inventory. Sketch payload schemas remain owned by -the sketch libraries; runtime contracts reference them rather than creating a -second copy. Exact aggregate state also needs an explicit versioned schema. - -The target package boundary separates lightweight semantic IR contracts, -runtime contracts, sketch libraries, the physical compiler, and executors. -Runtime contracts contain catalog, plan, publication, and frame contracts and -may use lightweight shared semantic types. They depend on neither optimizer, -Collector runtime, nor backend runtime. Go/Rust bindings must come from an -explicit schema authority, with cross-language fixtures where generation cannot -express semantic validation. Package extraction precedes any new repository. - -Move reusable reconstruction from Collector wrappers into sketch-library APIs. -Backend accumulators retain query-specific conversion but consume typed decoded -state, avoiding KLL's reconstruction/serialization/decoding detour. Supported -legacy bare-state reads remain until an explicit retirement gate. Consolidate -remaining codecs per family; the first extraction must not claim new parity for -HLL, CountSketch, or CountMinSketch. - -### Identity and update application - -Keep semantic node identity, SummaryDefinitionId, producer/partition identity, -concrete instance/physical storage lifetime, publication generation, and frame -sequence/checkpoint identity distinct. Moving a producer or changing cadence -need not change the logical definition, but does require an authorized deployment -transition. Reuse of state across generations requires explicit compatibility. - -Every remote state flow must specify: - -- Schema/codec and supported full/delta operations, including coverage/group keys. -- Producer partition and epoch, sequence scope, and replay/conflict behavior. -- Whether full state replaces a producer contribution or represents a distinct, - immutable contribution; how deltas reference and advance a checkpoint. -- Recovery after a gap, unknown checkpoint, restart, or incompatible generation. - -A full snapshot of an existing producer contribution cannot be merged into the -global result again as new observations. A receiver must replace/rebuild that -contribution using supported operations, or reject the unsupported update model. -Mergeable sketches are not necessarily subtractable. A delta is applicable only -to its authorized base; missing bases trigger resynchronization, not bare-state -fallback. A malformed framed payload must not evade validation through a legacy -unframed decoder. Duplicate/conflicting-frame decisions must be consistent with -state publication after failure; durable replay guarantees require durable -receipts or an equivalent reconstructable checkpoint protocol. - -These are target requirements. The initial migration preserves current wire -behavior and records any unmet requirement as a capability gap, rather than -changing full/delta semantics under an existing version. - -## Production, transmission, and query guarantees - -Split the responsibilities currently grouped in `RuntimeRulePolicy`: - -- Production policy controls sampling/admission and estimator semantics that - affect state construction. It is projected to the runtime producing that state. -- Transmission policy controls delta suppression, GOS where supported, emission - cadence, and full checkpoints. It is projected to both endpoints as needed. - -The compiler chooses these policies jointly and validates the resulting query -guarantee. Sketch error, sampling error, transport staleness, and incomplete -coverage are different quantities; they cannot be combined by an unconditional -sum of epsilons. State the estimator, assumptions, probability/evaluation scope, -and composition rule. Unknown evidence cannot establish a numerical guarantee. -A query guarantee shared across many outputs/evaluations must cover that declared -scope; shared state does not make errors independent. - -Changing sampling semantics requires guarantee and state-compatibility review. -A cadence-only change can retain the semantic definition but still needs an -accepted successor publication. Adaptation is bounded by installed policy and -fresh scoped evidence; it must not mutate an immutable generation in place. - -## Publication, activation, and readiness - -Keep `PhysicalPlanPublication` as the canonical artifact, rather than adding -another bundle format. Give each publication an unambiguous version/content -identity covering its plans and catalog references. A catalog digest alone does -not identify a change to transmission policy or physical placement. - -Use one shared cross-plan validation implementation at compilation and install -boundaries. Runtime-specific preparation still checks actual local resources. -Validate producer/consumer coverage, catalog references, schemas, window phase, -layout, supported codecs, selected policy guarantees, and query state bindings. - -Distributed rollout must account for partial failure: - -1. Validate and stage each required target; acknowledgements identify the exact - publication and target projection, not merely receipt of a message. -2. Prepare receivers before permitting new-generation producers to emit. Persist - the activation decision or use an explicit recoverable coordination protocol. -3. Switch each backend's local active snapshot atomically. Queries pin one - generation; a local pointer swap is not a distributed atomic commit. -4. Fence in-flight frames by generation. Accept an older frame only through an - explicitly retained compatible path; otherwise reject/resynchronize it. -5. On failure before activation, discard staged resources and retain the previous - generation. After partial activation, reconcile or publish a coordinated - successor; do not assume rolling back one process restores the whole system. - -Activation permits execution; it does not prove complete source coverage, warmed -state, or durable recovery. Readiness is derived from observed instances, -watermarks/completion proofs where supported, and pending admitted work. -[Completeness](continuous-summary-completeness.md) and the SDS lifecycle rules -remain required. The design does not assume that live Remote Write supplies -source watermarks or that existing runtimes implement global exactly-once delivery. - -## End-to-end examples and acceptance - -**Backend-local:** select a supported semantic summary and readout, bind its -maintenance to backend ingestion and its query to local state. Publish no -Collector targets or remote-summary rules. Exact fallback remains available -until the required coverage is ready. Where Planner authorizes two readouts -sharing one state, maintain it once per compatible input partition and generation. - -**Distributed:** two quantile queries over the same population, parameters and -window share a Collector sketch producer. The compiler emits one producer, -its remote-state rule, a backend integration binding, and two query readouts. -Sequence/checkpoint validation precedes state publication. Replaying a frame -must not increase the observation count. A failed target stage must not expose -new query bindings. Multiple producers require disjoint or explicitly accounted -input coverage; matching descriptor IDs alone do not prove safe merging. - -These examples define required fixtures, not new claims of implemented coverage. -Acceptance must exercise the actual supported Collector producer/decoder and -backend install/ingest/query boundaries, including Go/Rust interoperability. - -| Gate | Observable evidence | -| --- | --- | -| Semantic preservation | Selected node/root provenance survives all projections; incompatible grouping/window/lifecycle choices fail before publication | -| Subplan ownership | Precompute executable subgraphs contain no query-only SummaryEstimate; QueryPlan reads explicit compatible state boundaries; shared and derived summaries remain traceable | -| Visualization fidelity | Separate plan views agree with executable ownership; legacy embedded context is explicitly distinguished from executed operations | -| Shared production | N admitted observations cause N producer updates per intended partition, not N multiplied by consumer queries | -| Protocol conformance | Full, delta, duplicate, conflict, gap, epoch restart, unknown-base and legacy fixtures have explicit expected outcomes | -| State readiness | Missing or pending coverage uses configured fallback/unavailability; installation never certifies completeness | -| Generation transition | Failed stage, partial activation, delayed old frames and restart cannot mix query generations or double-apply state | -| Package boundary | Backend production dependencies exclude Collector execution runtime; protocol packages exclude optimizer/executor dependencies | -| Extension | Adding a codec uses one schema authority and endpoint capability registration, with no new plan-specific semantic definition | - -Test expectations should be specified before extraction. A reviewer other than -the implementation author should review protocol and rollout cases; this document -has not undergone independent review and reports no new executable test results. - -## Alternatives, quality attributes, and risks - -Keeping PrecomputePlan as the master representation is initially simpler but -makes edge and transport decisions depend on backend configuration. A small -internal binding stage resolves this without a new public IR. Independently -compiling four plans requires reconciliation after potentially different choices -and is rejected. A universal runtime would unnecessarily couple edge scheduling, -backend storage and query execution; share contracts/codecs instead. - -A new all-encompassing protocol repository does not resolve authority by itself. -First extract lightweight packages with one schema owner, then choose repository -placement and release tooling. Moving all of `asap_types` would also move Planner -and application coupling, so it is not the extraction unit. - -Maintainability is checked by the dependency graph and schema ownership audit. -Debuggability requires tracing a query root through semantic node, definition, -producer/partition, publication and checkpoint; validation reports the conflicting -identities and expected/actual contracts. Track staged/active versions, readiness, -frame rejection/resync counts, duplicate handling, and fallback reasons. Avoid -unbounded per-series metric labels; use structured diagnostic records for detail. - -Performance targets preserve current hot-path behavior: resolve catalog references -at installation, avoid network lookups per update, and remove redundant KLL byte -round trips. Measure compile/install time, payload size, ingest cost and retained -producer-state memory before and after; no speedup is assumed without evidence. -Only authenticated, authorized installation paths may grant producer/flow rights; -payload-provided identifiers do not authorize catalog or policy changes. - -The largest risks are codec drift, loss of provenance during binding extraction, -non-invertible sketch replacement, and partial rollout. Versioned adapters and -per-profile acceptance gates limit the rollout scope. Timeline estimates require -fixture and capability inventory first; intermediate success is unchanged wire -output from the new compiler structure, final success is both profiles passing -acceptance with the Collector dependency removed. - -Open implementation decisions are the contract schema/binding-generation tool, -publication identity encoding, durable coordinator mechanism, and supported -per-family replacement/recovery model. These must be resolved at their migration -gates; they do not justify enabling unsupported capabilities. Repository placement -can remain unchanged throughout the initial extraction. - -## Related documents - -- [Migration delivery plan](asapplanner-migration-plan.md) -- [Summary Catalog and SDS](summary-catalog-sds-architecture.md) -- [Physical compiler implementation](../developer_docs/control-plane/physical-compiler.md) -- [Plan publication implementation](../developer_docs/control-plane/plan-publication.md) -- [Catalog-backed runtime](../developer_docs/query-engine/catalog-physical-plan-runtime.md) -- [Compatibility profile](asapquery-compatibility-profile.md) +QueryPlan contains: + +- explicit reads of materialized state; +- `SummaryEstimate`, merge and other query-time summary operations; +- exact residual subtrees and result composition; +- the configured fallback or unavailable-result behavior. + +A semantic node may be represented inside a larger physical operation. The +provenance mapping records that relationship without requiring a one-to-one +physical node. + +## Binding meanings + +Bindings explain how semantic nodes map to the two physical plans. They do not +create a third execution phase. + +| Binding | Meaning | Example | +| --- | --- | --- | +| `Materialization` | The node's output is written as stored summary state by PrecomputePlan | `KLL` in `KLL(sum(data))` | +| `MaintenanceInput` | The node executes in PrecomputePlan as an input or intermediate, but its output is not independently stored | `sum(data)` feeding the KLL builder | +| `Query` | The node maps to an explicit QueryPlan operation | `SummaryEstimate` reading the KLL state | +| `QueryInput` | The node contributes query semantics but is absorbed into another QueryPlan operation | A scalar parameter or predicate compiled into a bound read/operator | + +“Maintenance” names an execution phase that constructs or updates state. It can +include initial batch construction, rebuilding, merging and derived-summary +construction; it does not imply incremental processing only. “Precompute” names +the backend plan and engine responsible for that work. + +## Shared and derived materializations + +Two queries may share a producer only when their bound definition and required +state partition are compatible. Sharing one producer must not multiply updates. +Each QueryPlan retains its own readout and result operators. + +A derived materialization is still maintenance work: + +```text +PrecomputePlan: Read completed state A -> derive state B -> store B +QueryPlan: Read state B -> estimate -> result +``` + +The dependency on A is an explicit state reference with completeness and schema +requirements. QueryPlan does not execute the derivation on demand unless the +selected physical plan explicitly models it as query work. + +## Validation and installation + +Compilation and backend installation apply the same cross-plan checks: + +- every state read resolves to one definition and permitted materialization; +- writer and reader agree on family, parameters, encoding and schema version; +- grouping, time partition, alignment and generation are compatible; +- every executable node is reachable from the correct plan root; +- each subgraph is acyclic and contains only operators supported in that phase; +- query fallback behavior is explicit; +- derived-state inputs satisfy their completeness requirement. + +The backend stages the catalog, PrecomputePlan and QueryPlan as one generation. +They become visible atomically. Installation success does not mean state is ready: +until required coverage exists, QueryPlan follows its exact fallback or returns +explicit unavailability. Failed staging leaves the previous generation active. + +## Visualization + +The plan viewer renders PrecomputePlan and QueryPlan separately and connects them +with labeled state references. It shows materialization ID, state family/schema +and readiness where useful. Query-only nodes never appear inside the executable +PrecomputePlan view. + +Legacy artifacts that embed complete semantic DAGs may be shown through a +projected view, but the UI must label that projection and identify which nodes +are maintenance-owned and query-owned. A separate semantic-plan page is not +required to understand the two executable plans. + +## End-to-end acceptance cases + +The design is complete when tests demonstrate: + +1. `Input -> Sum -> KLL` executes only in PrecomputePlan, while + `SummaryEstimate -> QueryResult` executes only in QueryPlan. +2. One query can read multiple bound summaries. +3. Two queries can share one compatible producer without duplicate updates. +4. A supported derived summary reads completed state and publishes a distinct + state reference. +5. Wrong schema, grouping, time partition or generation fails before activation. +6. Staging failure, restart and generation switching preserve the previous + consistent plan and documented fallback behavior. +7. The backend builds and runs these cases without ASAPCollector. + +## Decisions and deferred work + +We reject keeping the full semantic DAG as PrecomputePlan executable content: +bindings alone do not make plan ownership clear. We also reject compiling the +two plans independently because that permits identity and schema drift. + +The selected semantic DAG may remain as provenance or diagnostic metadata. It is +not a third executable plan. + +Deferred work includes CollectorPlan and TransmissionPlan compilation, distributed +activation, new transport/checkpoint protocols, Collector adoption of neutral +libraries and a broader ASAPPlanner API redesign. diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index f01ea50da..6d969b615 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -1,221 +1,142 @@ -# PrecomputePlan and QueryPlan: migration delivery plan - -Audience: developers implementing the backend portion of the -[integration architecture](asapplanner-integration.md). Status: proposed delivery -sequence, not a record of completed implementation. - -## Scope and completion definition - -This iteration handles **PrecomputePlan and QueryPlan only**, including their -shared SDS/catalog contracts, executable subgraph boundaries, backend installation, -and state decoding. CollectorPlan and TransmissionPlan compilation, policy redesign, -producer rollout and distributed activation are deferred. Their implementation or -release is not a prerequisite for completing this work. - -The backend must have **no build or runtime dependency on ASAPCollector**. Extract -the common contracts and codecs into runtime-independent libraries, then consume -those libraries from the backend. Copying Collector runtime code into a backend-only -fork or keeping a shared package hosted inside ASAPCollector does not meet this -boundary. Collector can adopt the common libraries in a separate follow-up. - -Completion means: - -- One selected Planner decision produces a coherent catalog and two executable - subplans, connected by explicit materialization/state references. -- PrecomputePlan executes state production/maintenance; QueryPlan executes reads - and query-time operations, including SummaryEstimate. -- Both subplans agree on definition identity, schema, grouping/window, guarantees - and generation, with backend-local atomic installation and separate readiness. -- Backend library/binary builds and the required test suite need no ASAPCollector - checkout, package or process. Shared reconstruction uses neutral libraries. -- Supported backend inputs, query results, recovery and legacy decoding retain - their documented behavior. No new distributed behavior is claimed. - -Existing CollectorPlan/TransmissionPlan fields may remain in legacy publication -adapters for compatibility. They are not redesigned by this migration. The local -path requires neither a Collector target nor transmission rules and must not use -CollectorPlan as the source of shared types or decisions. Do not silently accept -new distributed capabilities just because the local contract has changed. - -## Sequence and dependencies - -| Stage | Owner | Deliverable | Exit gate | -| --- | --- | --- | --- | -| 1. Inventory and fixtures | Backend | Contract/dependency map and backend behavior baseline | Every scoped entry point and Collector import has a documented replacement or compatibility fixture | -| 2. Extract common libraries | Backend and shared-library maintainers | Runtime-independent contracts and typed sketch reconstruction | Backend and required tests build without ASAPCollector; existing decoding remains compatible | -| 3. Bind and split two subplans | Backend compiler/runtime | Common bindings, catalog, maintenance/query subgraphs, explicit state boundaries | Executable ownership and provenance match supported semantics | -| 4. Validate, install and visualize | Backend | Shared two-plan checks, local generation switching, two execution views | Invalid boundaries fail; readiness and recovery remain correct | -| 5. Retire and release | Backend and shared-library maintainers | Remove superseded paths, pin common-library versions | Scoped end-to-end and dependency gates pass without Collector work | - -Stages 2 and 3 may be developed independently after the inventory, but both must -finish before the final gate. Extract code without changing payload bytes first; -version changes to installed executable representations separately. Do not combine -an unrelated Planner upgrade or a new public Planner facade with this work. - -## 1. Establish authority and backend fixtures - -Inventory the pinned Planner output, backend plan/SDS types, state schemas, -envelope types, all `asap_precompute_rs` imports, Cargo patches, and tests that -build or invoke Collector. Identify the smallest common API required at each -call site. Keep backend execution, storage and accumulator/readout adaptation in -the backend; do not move all of `asap_types` into a generic package indiscriminately. - -Capture backend-local raw ingestion, summary reconstruction, state maintenance, -query readout, completion, installation and recovery fixtures. For supported -existing full/delta/legacy payloads, record the bytes and expected state/readout -behavior with source revision and schema provenance. Frozen compatibility fixtures -may originate from Collector but must be usable without checking out or running it. -Randomized sketches may need persisted fixtures and semantic assertions rather -than comparing independently generated bytes. - -Input validation tests cover malformed framed payloads and currently supported -sequence/checkpoint behavior where touched by extraction. Missing target features -remain explicit gaps; do not turn this into a new transmission protocol project. -Have a separate reviewer assess boundary/replay expectations for consequential -implementation changes; independent review is not claimed by this document. - -### Planner input boundary - -Consume the existing pinned semantic contract and preserve per-query requirements, -root associations and lifecycle commitments. Runtime capabilities restrict eligible -lifecycle modes; a singleton legal lifecycle is valid. Incomplete stateful -commitments must not reach installation. A data-at-rest-only backend does not gain -incremental support merely because query demand repeats. - -[Planner #438](https://github.com/ProjectASAP/ASAPPlanner/issues/438) and its -[user/API documentation work](https://github.com/ProjectASAP/ASAPPlanner/pull/440) -remain related work, not completion prerequisites. Change Planner contracts only -for a demonstrated blocker to this two-plan split; a broad IR redesign or unified -Planner entry point is deferred. - -## 2. Extract shared contracts and codecs; remove Collector dependency - -Use two narrow ownership boundaries: - -| Common code | Owner / destination | Excluded dependencies | +# PrecomputePlan and QueryPlan migration plan + +Status: proposed delivery sequence. Audience: backend implementers. + +## Goal and scope + +Migrate the backend from a complete semantic DAG stored under PrecomputePlan to +separate executable PrecomputePlan and QueryPlan subgraphs connected by explicit +SDS state references. + +This migration also removes the backend build/runtime dependency on ASAPCollector. +Shared envelope, schema and sketch reconstruction code moves to neutral libraries. + +CollectorPlan, TransmissionPlan, distributed activation, new transport behavior +and a general ASAPPlanner API redesign are deferred. + +Completion requires: + +- `SummaryEstimate` and other query-only work appear only in QueryPlan; +- maintenance work terminates in explicit stored-state writes; +- both plans share one catalog/materialization/schema decision; +- the catalog and both plans install as one backend generation; +- supported legacy payloads and plans retain documented behavior; +- backend builds and required tests do not fetch, build or run ASAPCollector. + +## Delivery stages + +| Stage | Change | Exit gate | | --- | --- | --- | -| Runtime envelope metadata, shared IDs/tags, schema references and required validation | Lightweight neutral contract package, outside ASAPCollector | Collector/backend executors and Planner optimizer | -| Sketch payload schemas, decode/encode/reconstruction and supported state operations | Existing sketch-library APIs, or a neutral codec package if a concrete dependency requires it | Edge windowing, scheduling, host adapters and backend storage | - -Prefer existing sketch libraries and a small contract package over a new general -framework. If a new neutral package is required, establish its independent source -and versioned consumption before removing the old imports. Shared does not mean -that both runtimes must migrate in the same PR: backend adoption is in scope; -Collector adoption is deferred. Keep one schema authority and preserve compatible -wire behavior so a later Collector migration can reuse the same implementation. - -Move reusable DDSketch/KLL reconstruction out of Collector wrappers. Backend -accumulators consume typed decoded state, removing the unnecessary KLL -reconstruction/serialization/decoding round trip. Preserve supported local paths -for other families until replacement APIs have parity evidence. Keep legacy -bare-state readers and required vendored schemas until a compatible authoritative -replacement exists; do not silently change encoding versions or delta semantics. - -Remove the `asap-precompute-rs` dependency and obsolete Collector-specific Cargo -patches. Replace tests that import/invoke Collector with neutral-library tests and -provenance-bearing compatibility fixtures. Adapt dependency-enforcement tests to -the new boundary. Backend CI must not clone/build Collector indirectly through a -test helper, script, transitive dependency or shared-package location. - -Gate: inspect manifests, lockfiles, dependency graphs, source imports, build scripts -and required tests; no ASAPCollector dependency remains. Full-state, supported -delta and legacy fixtures retain decoding/rejection and readout behavior. Shared -libraries do not depend back on the backend runtime. No Collector release is needed. - -## 3. Bind once and split the executable subplans - -Retain candidate evaluation and downstream commitment. Introduce only the -compiler-local bindings needed for selected tasks, summary definitions, state -schemas, storage and input/output references. Construct the catalog and derive -PrecomputePlan and QueryPlan from the same decisions. Preserve semantic node -provenance and shared producers; do not independently choose their meanings. - -Implement the [materialization boundary design](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries): - -- Extract maintenance subgraphs terminating at stored-summary sinks, including - explicit reads of prior summaries for supported derived-state pipelines. -- Extract query subgraphs with explicit materialization reads and read-time - operations; query-only SummaryEstimate is absent from precompute executable content. -- Reuse catalog/materialization/schema identities; do not add a parallel boundary - identity registry or enumerate future stored pane instances during compilation. -- Use execution timing, dependencies and bindings rather than operator names to - determine ownership. Preserve absorbed operations in semantic-to-physical mapping. - -Switch maintenance execution to these subgraphs instead of discovering its work -inside a complete query DAG. Full semantic provenance can remain an artifact or -shared installation metadata, but is not executable content owned by PrecomputePlan. -Preserve its current representation if replacing it is unnecessary for the split. - -Version the split installed representation and normalize supported legacy -publications at the backend boundary. Legacy CollectorPlan/TransmissionPlan fields -remain compatibility concerns, not additional projections to implement. Do not -reinterpret the old executable-DAG field under an unchanged version. - -## 4. Validate, install and visualize the two plans - -Use shared two-plan/catalog validation at compilation and backend installation, -followed by actual local resource checks. Validate every boundary's definition, -schema, grouping/window phase and accepted generation. Keep one coherent local -publication identity; two independently activated subplans must not become visible. - -Stage and activate the backend snapshot atomically for query readers. Failed -staging preserves the previous active generation. Test restart, queued old-generation -maintenance output and compatible/incompatible state recovery. State readiness -remains distinct from installation; pending or insufficient coverage uses the -configured exact fallback or explicit unavailability. Distributed acknowledgements, -Collector cutover and new transport resynchronization are outside this stage. - -Visualize PrecomputePlan and QueryPlan separately, connected by labeled state -references. Show shared materializations and supported derived chains. For legacy -artifact inspection, label embedded read-time nodes as query-owned context rather -than maintenance execution; a projected view must identify itself as such. No -separate Semantic Plan page is required to understand the two execution plans. +| 1. Inventory and fixtures | Record current contracts, imports, payloads and execution behavior | Every scoped path has a compatibility fixture or explicit unsupported result | +| 2. Extract common code | Move runtime-independent contracts and reconstruction to neutral libraries | Backend dependency graph and required tests contain no ASAPCollector | +| 3. Bind and split plans | Compile one decision into catalog entries, maintenance subgraphs and query subgraphs | Executable ownership and state references match selected semantics | +| 4. Validate and install | Add cross-plan validation, atomic generation switching and two-plan visualization | Invalid publications fail before activation; previous generation survives failure | +| 5. Migrate and retire | Normalize old artifacts and remove superseded execution paths | Compatibility and end-to-end gates pass | + +## 1. Inventory and fixtures + +Inventory the pinned Planner output, PrecomputePlan/QueryPlan/SDS types, state +schemas, envelope definitions, all `asap_precompute_rs` imports, Cargo patches, +build scripts and tests that invoke Collector. + +Capture fixtures for supported: + +- raw input and summary reconstruction; +- full, delta and legacy bare-state payloads; +- maintenance updates and query readout; +- completion, restart and recovery; +- plan staging, activation and fallback. + +Fixtures may originate from Collector but must run without a Collector checkout or +process. Record their source revision and schema provenance. Use semantic readout +assertions where randomized sketch bytes are not stable. + +The backend capability profile is an input to Planner selection. Preserve complete +lifecycle commitments, even when the only supported choice is batch construction +from data at rest. Do not infer incremental support from recurring query demand. + +## 2. Extract common contracts and codecs + +Use narrow neutral-library boundaries: + +| Library responsibility | Must exclude | +| --- | --- | +| Envelope metadata, shared IDs/schema references and validation | Planner optimization and backend/Collector executors | +| Sketch payload schemas, encode/decode/reconstruction and supported state operations | Window scheduling, host adapters and backend storage | + +Prefer existing sketch-library APIs. Move reusable DDSketch/KLL reconstruction +out of Collector wrappers and remove unnecessary reconstruct-serialize-decode +round trips. Preserve legacy readers and other family-specific backend paths until +replacement APIs have parity evidence. + +Remove `asap-precompute-rs` and obsolete Collector-specific Cargo patches. Check +manifests, lockfiles, dependency graphs, scripts and required tests for direct and +transitive Collector dependencies. + +Do not change payload bytes during extraction. Version any later wire/schema +change separately. + +## 3. Bind once and split executable subgraphs + +Create compiler-local bindings for selected semantic nodes, summary definitions, +materializations, state schemas and read/write references. Derive the catalog and +both plans from those bindings. + +Apply the [materialization-boundary rules](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries): + +- PrecomputePlan contains maintenance inputs/operators and stored-state sinks. +- QueryPlan contains state reads, `SummaryEstimate`, exact residuals and result + composition. +- Derived maintenance uses explicit completed-state references. +- Shared producers retain one materialization identity and update path. +- Absorbed semantic operations remain visible through provenance mappings. + +The selected semantic DAG may remain diagnostic metadata, but it is not +PrecomputePlan executable content. + +Version the new installed representation. Normalize supported legacy publications +at the backend boundary; do not reinterpret an old field under an unchanged +schema version. + +## 4. Validate, install and visualize + +At compilation and installation, verify definition, materialization, schema, +encoding, grouping, time partition, coverage requirements and generation across +both plans. Then perform backend-local resource checks. + +Stage the catalog and two plans as one snapshot and activate them atomically. +State readiness remains separate from installation. Until required coverage is +ready, QueryPlan uses its configured exact fallback or explicit unavailability. + +Render PrecomputePlan and QueryPlan separately, joined by labeled state references. +Legacy full-DAG views must label maintenance-owned and query-owned projections. Acceptance cases: -- Build-summary/read-estimate places SummaryEstimate only in QueryPlan execution. -- One query can read multiple bound summaries; two queries can share one compatible - producer per intended partition without multiplying maintenance updates. -- Supported derived-summary chains preserve explicit state reads, completed-input - requirements and maintenance intermediates such as exact finalization. -- Wrong schema, grouping/window phase or generation fails before activation. -- Local failed-stage, generation-switch and restart cases preserve state lifetime, - completion checks, recovery, query consistency and fallback behavior. -- Old/new supported artifacts produce equivalent results and update counts. -- Visualization agrees with executable ownership and retains provenance links. -- These cases run without an ASAPCollector package, checkout or process. - -## 5. Roll out and retire - -Release the backend with pinned neutral-library versions and preserved rollback -artifacts. First migrate supported local publications; retain versioned adapters -for supported older artifacts. Remove full-DAG-in-precompute execution and obsolete -Collector adapter code only after their replacements pass the scoped fixtures. -State reuse across generations requires explicit compatibility independently of -binary rollback. Keep legacy payload readers for their supported recovery window. - -Distributed deployments continue on their supported compatibility path or receive -an explicit unsupported-version result. Do not claim a distributed migration or -require a Collector upgrade for this backend-local milestone. Repository-wide -schema consolidation and cross-language release coordination can follow separately. - -## Deferred work - -- CollectorPlan and TransmissionPlan compilation/refactoring and their runtime consumers. -- Moving production/sampling policy out of transmission policy across components. -- Collector adoption of the neutral contracts/codecs and Go/Rust binding consolidation. -- Distributed activation, new delivery/checkpoint/recovery semantics and multi-hop topology. -- A general Planner facade, broad semantic IR redesign and unrelated capability expansion. - -These remain part of the broader architecture, but are not dependencies or exit -gates for this migration. Existing supported input behavior is preserved through -backend adapters and fixtures, not through a live dependency on Collector. +- build-summary/read-estimate executes in the correct plan; +- one query reads multiple summaries; +- two queries share one compatible producer without duplicate updates; +- supported derived state observes completion requirements; +- wrong schema, grouping, time partition or generation fails before activation; +- failed staging, restart and generation switching preserve consistency; +- old and new supported artifacts produce equivalent results and update counts; +- all cases run without ASAPCollector. + +## 5. Migrate and retire + +Release the backend with pinned neutral-library versions and rollback artifacts. +Migrate backend-local publications first. Retain versioned adapters for the +supported compatibility window. + +Remove complete-DAG precompute execution and Collector adapter code only after +their replacements pass fixtures and end-to-end tests. Reusing state across plan +generations requires an explicit SDS compatibility decision independently of +binary rollback. ## Final evidence -Record tested revisions, the supported backend profile/state families, fixture -results, and dependency checks including tests/scripts. Trace a query through its -semantic root, materialization boundary, precompute producer and query reader. -Record compile/install/ingest measurements where extraction changes the path. -Completion requires the two-plan acceptance cases and zero ASAPCollector build/ -runtime dependency, not completion of the deferred distributed architecture. +Record tested revisions, supported state families, fixture results and dependency +checks. Trace at least one query from its selected semantic root through the +materialization boundary, PrecomputePlan writer, SDS state reference and QueryPlan +reader. Completion depends on the two-plan acceptance cases and zero backend +dependency on ASAPCollector, not on deferred distributed work. diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index 01246b5e0..5f47f418b 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -1,512 +1,245 @@ -# Summary Catalog and Self-Describing Summary Architecture +# Summary Catalog and Self-Describing Summary architecture -## Audience and relationship to physical plans +Status: proposed contract with notes on the current backend representation. +Audience: developers compiling, storing, recovering or reading summary state. -Audience: architects and developers. This document owns SDS identity, metadata, -state compatibility, and lifecycle semantics. The -[Planner and physical-plan architecture](asapplanner-integration.md) owns -compilation, the four runtime projections, transmission policy, and publication. -The target model below is distinct from the implementation notes that follow. -Those notes describe bounded paths and do not establish support for every target -lifecycle, distributed recovery mode, or completeness proof. +## Purpose and scope -SDS describes what a summary represents and which concrete state is available. -QueryPlan describes how to answer a query using it. TransmissionPlan describes -how authorized producers deliver state updates. A sketch envelope is one payload -carrier; it is not the SDS catalog or a physical execution plan. +The Summary Catalog and Self-Describing Summary (SDS) model is the authority for +persisted summary-state meaning. It connects PrecomputePlan writers to QueryPlan +readers without requiring either runtime to reinterpret Planner IR. -## Semantic model and authority +This document owns: -| Layer | Meaning | Changes when | +- stable summary semantics and materialization identity; +- state schema, encoding and partition identity; +- references from plans to stored state; +- readiness, generation and retirement metadata; +- validation required when state is written, recovered or read. + +The [integration design](asapplanner-integration.md) owns physical subgraph +splitting and execution. The [migration plan](asapplanner-migration-plan.md) +owns delivery order. Cost evidence, candidate ranking, operator scheduling and +transmission policy are outside the SDS model. + +## Core model + +| Object | Meaning | Stability | | --- | --- | --- | -| Summary Descriptor | Operator, parameters, fidelity and compatible state representation | Operator/configuration or guarantee contract changes | -| Data Descriptor | Source, population, grouping and observation semantics | Input meaning or population changes | -| Summary Definition | Stable logical materialization referencing descriptors | The semantic definition changes | -| Summary Instance | Concrete extent/group, provenance, status and state reference | State is materialized, updated or retired | - -The control plane owns the desired `SummaryCatalog`. It is constructed from the -selected semantic definitions and common physical compilation decisions before -projecting CollectorPlan, PrecomputePlan, TransmissionPlan and QueryPlan. Plans -reference the same immutable catalog snapshot. They do not independently define -summary meaning, and PrecomputePlan is not the catalog's semantic authority. - -During migration, installed DTOs may repeat parameters, population or window -fields required by existing consumers. These must agree with the catalog and be -mechanically derived from the common bindings. A target artifact may include a -self-contained catalog subset; it must be verifiable against its publication. -Resolve references at installation rather than through per-update remote lookups. - -The observed `ObservedSummaryInventory` reports actual instances and their -readiness. It is not desired state and contains no encoded payloads. Planner may -use this scoped availability evidence without reading sketch bytes. Runtime -reconciliation creates, recovers, retires and expires state according to the -installed contracts; metadata declarations alone do not execute those actions. - -## Identity and state references - -Keep these identities distinct: - -| Identity | Scope and purpose | -| --- | --- | -| Semantic node ID | Node within the selected Planner DAG; physical bindings retain provenance | -| SummaryDescriptorId / DataDescriptorId | Immutable semantic descriptor content | -| SummaryDefinitionId | Logical materialization; currently backed by a typed policy fingerprint | -| SummaryInstanceId | Concrete materialized instance identity | -| Producer / partition / epoch | Source contribution and restart lifetime | -| SeriesId | Backend physical storage lifetime, not a descriptor or plan identity | -| CatalogGeneration | Catalog publication reference, including digest and plan version | -| Publication identity | Exact installed plan content, including execution and transmission choices | -| Sequence / checkpoint | Update history and applicable delta base within a declared stream scope | - -Current descriptor IDs use versioned canonical semantic strings. Changing their -encoding must preserve semantic identity and explicitly address collisions. -A new interval/group creates an instance without redefining its descriptors. -Moving a producer or changing transmission cadence need not change its semantic -definition, but requires an authorized publication transition. Changed sampling -or observation semantics require guarantee and state-compatibility validation. -A catalog digest alone cannot identify every change to the four physical plans. - -The target instance metadata contract is: - -```rust -struct SummaryInstance { - instance_id: SummaryInstanceId, - summary_definition_id: SummaryDefinitionId, - summary_descriptor_id: SummaryDescriptorId, - data_descriptor_id: DataDescriptorId, - time_range: HalfOpenTimeRange, - group_values: GroupValues, - catalog_generation: CatalogGeneration, - placement: SummaryPlacement, - state_reference: SummaryStateReference, - status: SummaryInstanceStatus, - completeness: InstanceCompleteness, - lifecycle: InstanceLifecycle, -} +| `SummaryDefinition` | Canonical semantics of a summary: input, operation, grouping, time semantics, algorithm and parameters | Stable while those semantics remain unchanged | +| `Materialization` | A physical-plan decision to produce a definition with a particular state contract | Versioned with the installed plan generation | +| `SummaryStateInstance` | One persisted state partition, such as a series/pane or completed aggregate | Created and retired by runtime lifecycle | +| `StateReference` | A typed reference used by QueryPlan or a derived PrecomputePlan node | Valid only for compatible definition, schema and generation rules | + +The catalog stores definitions and materializations. Runtime inventory records +state instances. Plans carry state references rather than embedding payloads or +search predicates. + +```mermaid +flowchart LR + D[SummaryDefinition] --> M[Materialization] + M --> I1[State instance] + M --> I2[State instance] + P[PrecomputePlan writer] --> M + Q[QueryPlan reader] --> R[StateReference] + R --> M ``` -This is a conceptual shape, not a new wire DTO. `SummaryStateReference` is an -opaque storage locator with schema version, generation, sequence and optional -checksum. SummaryStore owns the referenced payload. Concrete frame identity -additionally records the producer stream and checkpoint context required by its -TransmissionPlan; an instance reference alone does not authorize delta application. - -Sketch libraries own payload schemas, decoding/reconstruction and supported state -operations. Runtime contracts own catalog, plan and frame metadata. Transport -adapters map these contracts into OTLP or another supported carrier without -redefining sketch payload schemas. Full-state replacement, replay, and delta-base -rules are specified in the [integration design](asapplanner-integration.md#identity-and-update-application). -Matching bytes or descriptor IDs alone never proves safe merging or complete data. - -The [physical subplan boundary](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries) -connects a materialization sink to reads of the same summary definition. It does -not add a new SDS identity or require compile-time enumeration of future instances. -PrecomputePlan owns state production and QueryPlan owns query-time readout; -semantic provenance retained for tracing does not change execution ownership. - -## Desired state and observed lifecycle - -Persistent desired materializations come from control-plane planning. A runtime -fast path may create only an authorized ephemeral instance with a finite lease, -report it, and await promotion or expiry. It cannot silently make that instance -persistent desired state. - -Reconciliation compares desired definitions with observed placement, extent, -state references, status and completeness. Catalog and plan activation authorize -execution; they do not establish source completeness, durability, or query -readiness. State reuse across generations requires explicit compatibility, and -retired physical lifetimes remain fenced from late updates. - -## Implemented backend representation - -The in-memory descriptor representation is normalized. `SummaryDescriptorRegistry` -content-interns Summary and Data Descriptors. A SID owns an `SdsBinding` with -shared `Arc` references to both descriptors. Pane rows store the SID foreign -key, `[start, end)`, interned group values and state; together these fields form -the Summary Instance. This avoids repeating descriptors in every pane and lets -catalog snapshots and query lookups clone pointers rather than descriptor data. -The registry holds weak references, so retiring the final SID also releases its -descriptors. `SketchInstanceMetadata` remains the registration and persistence -compatibility DTO while older sidecars are read. - -The implemented `SummaryDescriptor` currently contains one `SummaryOperator`, -one derived `FidelityGuarantee`, and a numeric state-schema version. The -implemented `DataDescriptor` contains typed source and value projections, a -canonical population filter, typed grouping columns and versioned observation -semantics. The shared contract now also -defines `SummaryInstance`, `ObservedSummaryInventory`, placement, completeness, -state references, catalog generation and ephemeral leases. The control-plane -reconciler emits create, update, recover, retire, garbage-collect, promote and -expire actions. Summary payloads and the application of those actions remain in -the SummaryStore runtime. - -The same `GroupingProjection` supplies source columns to precompute configuration, -`DataDescriptor` and the state-schema contract. Each column retains the Planner's -name, type and nullability; routing derives names without storing a second list. -Legacy label lists decode as non-null UTF-8 columns and keep their existing -identities. A changed type or nullability changes catalog and policy identity. -A SQL map column is one grouping value, not a set of PromQL labels. Typed -ClickHouse group transport remains a separate execution capability: the current -reader rejects non-label projections until that transport is implemented. - -`DataDescriptor`, precompute configuration and state-schema validation share -`ValueProjectionIdentity`: sample value, named column, or a finite numeric -constant using the Planner's `ScalarValue`. A constant input such as `1` does -not masquerade as a table column. Projection identity participates in catalog -and policy identity; existing column identities remain unchanged. Older -`value_column` config and state-schema fields are accepted only by wire adapters -and become the same typed projection in memory. ClickHouse backfill binds a -constant as a typed query parameter and applies the installed table population -and timestamp projection. Its Float64 ingest boundary rejects integer constants -outside the exactly representable range. This contract enables literal inputs; -query lowering must still establish each aggregate's null and row semantics. - -The durable `sid_metadata.json` format is versioned independently of the wire -contracts. Descriptor tables and bindings avoid repeating semantic definitions; -later metadata revisions also preserve definition identity and catalog provenance. -Legacy records are interpreted by versioned recovery code and must not acquire -authoritative catalog bindings without validation. See -[completeness and recovery](continuous-summary-completeness.md). - -An ingest record is never an SDS instance. Raw samples can be transient inputs to -the precompute engine, but the backend does not retain them as a second exact -query store. Exact residual subtrees run in Prometheus. - -The SDS metadata and inventory types represent the following invariants. The -current runtime enforces descriptor binding and non-overlapping pane selection. -Full runtime conformance still requires applying and durably persisting every -reconciliation action, including recovery, promotion, lease expiry, retirement, -and garbage collection: - -1. An instance references exactly one immutable Summary Descriptor and one - immutable Data Descriptor. -2. `[start, end)` plus concrete group values identifies the summarized extent; - different panes are different instances. -3. State may be merged only when the Summary Descriptor permits the operation, - Data Descriptors are compatible, and interval coverage does not double-count. -4. Completeness and approximation fidelity are independent. An exact operator - over a partial interval is still incomplete. -5. State bytes always carry a state schema version. A codec match alone does not - imply semantic compatibility. -6. Rollups never become authoritative state. `RollupCategory::ExactMax` and - future categories live below one `rollups` collection and can be discarded - and rebuilt from instances. - -## 1. Summary Descriptor - -A Summary Descriptor defines **how the data is summarized** and **which fidelity -claims the summary supports**. It does not identify a source population or a -particular time interval. - -| Field | Type | Definition | -| --- | --- | --- | -| `summary_descriptor_id` | `QualifiedId` | Immutable descriptor identity | -| `operator` | `SummaryOperator` | Algorithm, semantic version, parameters and supported operations | -| `fidelity` | `FidelityGuarantee[]` | Exactness or error guarantees, with their scope and conditions | -| `state_representation` | `StateRepresentation` | State type, codec and codec version | - -`SummaryOperator` contains an algorithm identifier, versioned semantics, -type-specific parameters, and supported build/update/merge/readout signatures. -Parameters and operation arguments depend on the summary type; `item` and -`weight` are not mandatory common fields. - -For example, a KLL operator may specify `k: 200`. The value of `k` is an -algorithm parameter, **not itself a numerical error guarantee**. Its fidelity -contract separately identifies the supported rank-error bound or versioned -bound derivation, probability of failure, readout scope and required conditions. -If that guarantee is unavailable, fidelity is explicitly `Unknown`. - -For a shared UnivMon state, `heap_size`, `sketch_rows`, `sketch_cols`, and -`layers` describe one configuration. They do not establish one error bound for -all readouts. The backend's `UnivMonFrequency` contract records these parameters -and the unit-frequency update domain: each sample value contributes one -occurrence. Total count is exact in that domain; distinct count, frequency L2, -and frequency entropy require their own accuracy evidence. Frequency L2 means -`sqrt(sum(frequency(key)^2))`; entropy is measured in bits. - -ERP evidence must state the readout's units: relative error for distinct and L2, -and absolute bits error for entropy. A measured error is not a certified failure -probability. Readouts may share state only when their configuration and data -population match and each readout's accuracy requirements are satisfied. A -small configuration suitable for L2 may therefore be unsuitable for entropy. -Completeness of the input window remains a separate requirement for every -readout, including exact count. - -A `FidelityGuarantee` contains: - -- The applicable operation and error quantity, such as quantile rank error. -- A category: `Exact`, `DeterministicBound`, `ProbabilisticBound` or `Unknown`. -- A bound or versioned bound derivation, and a failure probability when applicable. -- The population/readout/evaluation scope and required assumptions. - -A `StateRepresentation` identifies the logical state type and versioned encoding. -Compatible bytes alone do not establish that two operators have compatible -semantics or guarantees. - -## 2. Data Descriptor - -A Data Descriptor defines **which data is summarized**. It is independent of the -summary algorithm and of a particular materialized interval. - -| Field | Type | Definition | -| --- | --- | --- | -| `data_descriptor_id` | `QualifiedId` | Immutable data-scope identity | -| `source` | `SourceBinding` | Metric/series or dataset, including its versioned field definitions | -| `population` | `PopulationDefinition` | Selection predicate and grouping/entity scope | -| `observation_semantics` | `SemanticContract` | Value projection, units and handling of missing, duplicate or invalid observations | +## Summary definition -For example, the source can be the metric `cpu_usage`, and the summarized -population can be the series satisfying `container_type="login"`. +A definition contains all fields required to decide whether two summaries have +the same meaning: -`PopulationDefinition` records both selection and partitioning. It distinguishes -one summary over all selected observations, independent summaries per series, -and summaries grouped by specified label keys. Concrete group values belong in -the instance metadata when one descriptor describes a reusable grouping rule. +- canonical input source and filters; +- input value semantics; +- exact operation or sketch family and typed parameters; +- grouping and reduction semantics; +- query-range/time-partition semantics and alignment; +- accuracy contract where it affects state meaning; +- output value type. -A population predicate is a typed, resolved data-selection definition. It is not -an arbitrary executable program attached to a summary. +Display names, plan generation, readiness, storage location, retention status and +observed costs do not belong to definition identity. Changing a semantic field +creates a different definition instead of mutating an existing one. -## 3. Summary Instance +Definitions may refer to raw input or to another completed summary definition. +Derived input references are typed dependencies, not metric-name aliases. -A Summary Instance describes a concrete materialization and references its -stored state, one Summary Descriptor and one Data Descriptor. The metadata DTO -and inventory never embed the encoded payload. +## Materialization -| Field | Type | Definition | -| --- | --- | --- | -| `instance_id` | `QualifiedId` | Materialized instance identity | -| `summary_descriptor_id` | `QualifiedId` | Referenced operator/fidelity descriptor | -| `data_descriptor_id` | `QualifiedId` | Referenced source/population descriptor | -| `metadata` | `InstanceMetadata` | Concrete extent, population binding, completeness and provenance | -| `state_reference` | `SummaryStateReference` | Opaque locator for separately stored state and its schema/provenance | - -`InstanceMetadata` contains the concrete time range or dataset extent, any group -values needed by the population rule, completeness (`Complete`, `Partial` or -`Unknown`), producer/generation/sequence provenance and instance-specific fidelity -evidence. Time ranges specify their clock, units and interval boundaries. -Completeness is separate from mathematical approximation error. - -The referenced payload is maintained state, not a quantile readout or other -query result. A transported delta identifies its authorized producer stream and -base checkpoint as well as the supported apply operation. A descriptor or instance -ID alone is insufficient to interpret it as a full state. - -## Shared-descriptor example - -The following example summarizes `cpu_usage` observations from login containers -using KLL with `k=200`. All three instances reuse the same Summary Descriptor and -Data Descriptor; only the instance time range and state change. - -```yaml -summary_descriptor: - summary_descriptor_id: example:kll-200-v1 - operator: - algorithm: KLL - parameters: {k: 200} - semantics: example:kll-semantics-v1 - fidelity: - - operation: quantile - error_quantity: rank_error - category: Unknown # No numerical guarantee is inferred from k alone. - state_representation: example:kll-state-codec-v1 - -data_descriptor: - data_descriptor_id: example:login-cpu-v1 - source: {metric: cpu_usage} - population: - predicate: {container_type: {equals: login}} - grouping: global - observation_semantics: example:cpu-observations-v1 - -instances: - - instance_id: example:login-cpu-0 - summary_descriptor_id: example:kll-200-v1 - data_descriptor_id: example:login-cpu-v1 - metadata: {time_range: "[0,10)", clock: example:seconds} - state_reference: {store: example-store, key: S0, state_schema_version: 1} - - instance_id: example:login-cpu-1 - summary_descriptor_id: example:kll-200-v1 - data_descriptor_id: example:login-cpu-v1 - metadata: {time_range: "[10,20)", clock: example:seconds} - state_reference: {store: example-store, key: S1, state_schema_version: 1} - - instance_id: example:login-cpu-2 - summary_descriptor_id: example:kll-200-v1 - data_descriptor_id: example:login-cpu-v1 - metadata: {time_range: "[20,30)", clock: example:seconds} - state_reference: {store: example-store, key: S2, state_schema_version: 1} +A materialization commits a definition to a concrete state contract: + +- stable definition ID; +- materialization ID and plan generation; +- state family, schema version and encoding; +- physical grouping and partition layout; +- permitted producer/writer identity where required; +- lifecycle and readiness policy; +- provenance back to selected semantic nodes. + +Multiple materializations may implement the same definition, for example across +plan generations or storage migrations. QueryPlan reads a compiler-selected +materialization reference; the serving runtime does not search all catalog entries +for a substitute. + +## Summary state instance + +A state instance identifies one physical partition of a materialization. Its key +contains only dimensions needed to distinguish stored state, such as series or +group identity, time partition, producer/shard identity and generation. Its +metadata records: + +- materialization and definition IDs; +- exact schema/encoding used by the payload; +- coverage or completion bounds; +- producer sequence/checkpoint metadata when applicable; +- creation, readiness and retirement state; +- content location and integrity information. + +Payload bytes are stored in the summary store, not copied into the catalog +descriptor. Mutable runtime statistics do not change semantic identity. + +## State reference + +A state reference is the only normal connection between executable plans and +stored state. It identifies the required materialization and constrains the state +partition, schema and generation that may satisfy the read. + +PrecomputePlan uses state references for derived-summary inputs. QueryPlan uses +them for result-producing reads. A reference may select multiple instances, such +as the panes covering one query range, but it cannot broaden the summary +definition or silently select another algorithm. + +The runtime may resolve physical locations through an index. Resolution must be +an exact lookup under the installed reference and metadata; catalog scanning and +serving-time candidate selection are prohibited. + +## Identity rules + +The following identities have different purposes and must not be collapsed: + +| Identity | Answers | +| --- | --- | +| Definition ID | What summary semantics does this state represent? | +| Materialization ID | Which installed physical production decision created it? | +| State-instance ID | Which concrete partition/payload is this? | +| Plan generation | With which atomic installation may it be used? | +| Schema/encoding ID | How are its bytes interpreted? | + +IDs are assigned or derived once by the compiler/catalog authority and carried +through plans, storage and recovery. Human-readable names are diagnostic labels, +not join keys. A reused state instance across generations requires an explicit +compatibility decision; matching definition IDs alone is insufficient. + +## Plan boundary + +The physical-plan split uses the catalog as follows: + +```text +PrecomputePlan + Input -> Sum -> BuildKLL -> Write(materialization=mat-17) + +Catalog/SDS + mat-17 -> definition=def-9, family=KLL, schema=kll-v1, generation=42 + +QueryPlan + Read(mat-17, schema=kll-v1) -> SummaryEstimate -> Result +``` + +The writer and reader share the same compiler binding. They must agree on: + +- definition and materialization identity; +- state family, algorithm parameters, schema and encoding; +- grouping and time partition/alignment; +- generation compatibility and coverage requirements. + +For a derived summary, the destination materialization has its own identity and +the maintenance node holds a `StateReference` to its completed source state. The +source and destination are never represented as the same instance. + +## Lifecycle and readiness + +Materialization intent and observed state are separate: + +| State | Meaning | +| --- | --- | +| `Desired` | Installed plans require the materialization; usable state may not exist yet | +| `Building` | The runtime is producing or recovering required coverage | +| `Ready` | Required schema and coverage are available for the bound reads | +| `Draining` | No new work is assigned, but existing readers or writes are being completed | +| `Retired` | The materialization is unavailable to new reads and may be garbage-collected when safe | + +Activation installs intent atomically but does not manufacture readiness. A +QueryPlan read checks observed readiness and coverage, then follows its configured +fallback or unavailability behavior. Reactivation of a retired definition creates +or binds an authorized materialization; it does not make stale instances current. + +Completed finite-input state is immutable. Further additive writes require a new +authorized generation or replacement instance. Mutable streaming state publishes +monotone coverage/completion metadata according to its installed contract. + +## Validation invariants + +Compilation, installation, writes, recovery and reads enforce these invariants: + +1. Every materialization resolves to exactly one definition. +2. Every state instance resolves to one materialization and declares its actual + schema and encoding. +3. A state reference cannot change definition semantics during resolution. +4. Writer and reader grouping, time partition and schema contracts agree. +5. State from an incompatible generation is rejected before execution. +6. Ready state satisfies the reference's coverage and completion requirements. +7. Derived maintenance reads only completed input when its operator requires it. +8. Retirement prevents new bindings before physical state is reclaimed. +9. Unknown schema, malformed payload and unauthorized producer updates fail + closed; they never become catalog-visible ready state. + +## Current representation and migration boundary + +The backend already has catalog descriptors, policy fingerprints, series IDs, +state metadata and persisted payloads, but responsibilities are distributed +across `asap_types`, control-plane publication and the summary store. Some current +artifacts also embed complete semantic DAGs in PrecomputePlan. + +Migration should reuse authoritative IDs and storage metadata rather than create +a parallel registry. Legacy artifacts are normalized at the backend boundary; +new plans use explicit state references. Existing supported payloads remain +readable through versioned codecs and compatibility fixtures. + +The backend must not depend on ASAPCollector for these contracts or codecs. +Runtime-independent envelope/schema definitions and sketch reconstruction belong +in neutral libraries. Backend storage, scheduling and query execution remain +backend-owned. + +## Example + +Two queries request percentiles over the same grouped input. The compiler selects +one compatible KLL materialization and emits two QueryPlan entries: + +```text +definition def-9: + input=request_latency, group_by=[service], range=5m, algorithm=KLL(k=200) + +materialization mat-17, generation 42: + definition=def-9, schema=kll-v1 + +state instances: + mat-17/service=api/pane=12:00..12:01 + mat-17/service=api/pane=12:01..12:02 + ... + +query q50: Read(mat-17) -> Estimate(0.50) +query q99: Read(mat-17) -> Estimate(0.99) ``` -`S0`, `S1` and `S2` are opaque keys for separately stored KLL states. This -conceptual example omits full state-reference provenance and producer evidence; -it is not an installable DTO and makes no completeness or numerical error claim. Descriptor references must resolve within the supplied context or a -durably retained descriptor registry. - -Changing `k` creates a new Summary Descriptor. Changing the source or population -creates a new Data Descriptor. Advancing the time range creates a new Summary -Instance. Merge compatibility additionally requires the operator's merge rules, -compatible data scopes and valid instance coverage; sharing descriptors alone -does not authorize merging overlapping observations. - -### Catalog-scoped runtime ERP evidence - -A runtime observation describes the input of one allocated summary, not an -entire deployment. `ErpPopulationObservations` identifies its catalog generation, -summary definition, observation time, input window and separate summary-instance -populations. The control plane resolves the `DataDescriptor` from its successfully -activated catalog; a telemetry payload cannot provide replacement descriptors. -Alternative sketch parameters may use this evidence only when the compiler -verifies the same data and update semantics. - -The typed physical-plan HTTP endpoints accept `target: backend_local_remote_write` -with an empty `collector_ids` list. Omitting `target` preserves the distributed -collector deployment. Both paths use catalog publication and activation. Typed -activations are serialized, and the accepted catalog is retained only after the -backend acknowledges activation, including ClickHouse publications. - -An ERP `observed_shape_source.population_scope` supplies the expected catalog -and definition, input semantics, and explicit `max_age_ms` / -`max_future_skew_ms` bounds. Each compilation reads the latest runtime record -again. Missing, stale, malformed, foreign or incomplete observations invalidate -all population fits. This is an ERP miss handled by theoretical sizing or exact -execution; it must not restore an older fit or match the artifact's legacy -distribution descriptor. Offline single-shape inputs remain a separate path. - -The initial eligibility is deliberately limited to verified raw per-series -frequency/cardinality readouts over a complete matching window. A 30-second pane -observation does not certify a one-hour input distribution. These checks do not -implement an autonomous drift-triggered replan scheduler, continuous source -completion, or durable restoration of the control plane's active catalog. After -a control-plane restart, live evidence remains ineligible until an authoritative -catalog has been activated again. - -### Retired physical series and catalog reactivation - -A persisted removal tombstone prevents late fragments and stale metadata flushes -from reopening the same physical `SeriesId`. A later installed catalog generation -may authorize a fresh physical series for the same logical definition/group. -The resolver writes that rotation and its catalog provenance before changing its -cache; ordinary writes from the original generation cannot authorize rotation. -The original physical ID remains tombstoned so old disk parts cannot enter the -replacement's readout. - -Queued precompute inputs carry their captured catalog generation and physical -series ID separately from an optional admission receipt. Workers preserve both -on publication. A delayed output writes its original physical series, never a -newly resolved replacement. Derived materializations resolve their own target -series while retaining the source generation proof. Backfill processors capture -the catalog generation when attached to the store; old jobs cannot authorize a -new catalog's rotation. An older queued input that has not yet published its -first storage instance is conservatively rejected after a catalog change. Already -registered retained series can drain their birth generation or accept the current -generation. Seamless re-planning of unpublished old inputs requires additional -first-mint provenance; it is not guaranteed by this transition. - -This is an explicit lifetime transition, not cross-generation recovery of arbitrary -summary state. Legacy records without trustworthy catalog provenance remain -unbound. Tombstone reclamation still requires coordinated removal of old physical -parts and is not implemented by this transition. - -### Derived summary input identity - -A summary computed from another summary has a different data source from the -original raw table or metric. `PrecomputeMaterialization.derived_input` and -`DataSourceIdentity::Derived` use the same `DerivedInputIdentity`: the referenced -`SummaryDefinitionId`s and a SHA-256 of the maintenance program. The executable -program remains in `OwnedPostAsapDag`; the catalog does not retain another copy. - -The signature replaces materialized input frontiers with stable summary IDs and -hashes the remaining node payloads, schemas, guarantees, and edge semantics. It -excludes query names, plan-local node numbering, and catalog generations. Literal -leaves are hashed directly; raw input leaves still require catalog frontiers. A changed -input definition or transformation creates a new identity. Existing raw-source -identities retain their previous byte representation. Catalog validation rejects -missing input definitions and dependency cycles. - -Typed installation accepts the bounded immutable maintenance contract below -only when the complete installed DAG matches the catalog input identity. Legacy -raw YAML still rejects derived inputs; raw routing excludes them. Neither raw-table -substitution nor treating late correction fragments as new observations is valid. - -### Immutable completed windows - -Finite Remote Write completion now fences the SummaryStore append boundary, -not just the receiver queue. After all admitted outputs are published, the store -records the greatest published window end for each physical SeriesId. Sketch and -exact-state writes ending at or before that boundary are rejected, including -writes arriving through other producers. A later window remains writable. Observed SDS inventory reports only these frozen -instances as `Complete`; ordinary emitted panes remain `Unknown`. - -The boundary is monotone in the existing SeriesId metadata sidecar and is restored -before recovered identities become writable. A stale background metadata flush -cannot reopen a completed window. The guard belongs to the physical lifetime; -a catalog-authorized replacement SeriesId has its own boundary. - -With persistence enabled, completion explicitly requests the existing flusher to -make the completed prefix durable, even if it is still inside the hot tier. -Completion waits until the corresponding epochs have been evicted after part and -manifest publication; only then does it persist the immutable boundary. An -in-memory deployment provides no restart guarantee. Maintenance consumers still -must atomically publish their output identity before claiming replay-safe consumption. -The existing finite-source completeness proof still rejects untracked writes or -pending admitted work. Continuous producer watermarks and derived-state commit -transactions are separate from this finite-input boundary. -### Executing an immutable maintenance sink - -`precompute_engine::maintenance_runtime::execute_completed_maintenance` executes -one installed semantic subDAG from a physical source whose required base windows -are durably complete. SummaryStore validates the catalog generation, physical -SeriesId, population, exact window coverage, and each part read. Missing, corrupt, -or duplicate source windows are errors; this path cannot silently omit a pane as -a query fallback helper might. - -The existing maintenance operator registry preserves a collection of source -states until the DAG explicitly merges or finalizes it. Exact Sum/Count -finalization with a declared Float64 output produces one row per source window; an unkeyed SummaryAgg consumes -those rows together. Consequently `Finalize -> SummaryAgg` does not accidentally -become one complete DAG evaluation per correction fragment. Live worker fragments -remain ineligible for finalization. - -The engine resumes a matching durable pending part and looks up the stored input -digest before computing a potentially randomized sketch. The existing flusher publishes a new result through its part -reservation protocol; SummaryStore fences query reads and physical lifetime -changes during publication. A concurrent identical completion reuses the durable -result instead of comparing newly randomized bytes. Both pending recovery and a -committed lookup restore the live completion boundary. Catalog-derived definitions -reject additive sketch/precompute writes even beyond that boundary; only reserved -publication may create their output state. The latest committed window can be -retried after restart without adding another part. - -Backend-local remote-write plans can bind a selected exact accumulator followed -by an explicit maintenance-time Finalize and outer unkeyed SummaryAgg. Initial -automatic installation requires one raw source definition and identical full, -non-overlapping source/output windows. The finite drain barrier flushes source -state and schedules complete retained windows through this same entry point; -raw routing never feeds samples directly into the derived accumulator. - -Finite completion closes all raw store writes for that catalog generation, not -only its HTTP receiver. The existing admission lock issues a private publication -writer; a receipt carried in an output is not evidence that this lock is held. -The metadata writer persists one generation checkpoint before completion becomes -usable, and restores it before accepting writes after restart. A failed close -stays closed to writers until its persistence retry succeeds. Installing a new -catalog generation starts a new admission lifetime. Derived state from a previous -generation is excluded from query candidates and inventory; recomputation receives -a fresh physical SID through the existing resolver. Raw state remains independently -reusable, and retained old source populations cannot be omitted from a singleton proof. - -A per-entity source can feed global Reduce([]) only when the store proves that -its entire finite population contains exactly one physical source SID and one -stored label population. This proof unions live bindings with all nonremoved -strict durable metadata for the same summary definition, across catalog generations, -before reserving any output. The -reduction then removes source labels according to the installed output grouping. -Multiple source SIDs or stored groups fail closed; this is not general shuffle support. Physical -SID metadata retains observed per-entity label names for durable decoding while -the catalog retains the logical partitioning contract. SQL backfill job status -alone is not this all-producer completion proof and does not trigger this path. - -Synchronized multiple sources, general row operators, overlapping output-window -replacement, and continuous producer watermarks remain unsupported. In particular, the SQL -subquery's timestamp grouping and sampling predicate must not be replaced with an -arbitrary tumbling aggregate. Historical completion-metadata GC and pinning source -parts for recovery before a reserved output part exists remain lifecycle work. +The producer updates each state partition once. Both queries resolve the same +bound materialization, compose the required coverage and apply different readout +parameters. Neither query creates a second producer or searches for a different +summary at serving time. + +## Deferred work + +This design does not define CollectorPlan or TransmissionPlan, distributed +activation, a new checkpoint protocol, cost/ERP evidence, or retention policy +selection. Those systems may reference SDS identities later without becoming +part of the SDS semantic model. From 3f1830d3aab2565228b78a85ae2810ac09e2192f Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 03:43:36 +0000 Subject: [PATCH 07/15] docs: add physical compiler input example --- docs/design_docs/asapplanner-integration.md | 43 +++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 24df0a941..8eb592cc2 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -49,6 +49,49 @@ The physical compiler consumes: - backend capabilities and concrete implementation evidence; - catalog, schema and deployment-generation inputs. +For example, suppose query `p99-api-latency` asks for the 99th percentile of +`request_latency_seconds` over five minutes, grouped by `service`, every minute. +The following conceptual input shows what each category contributes; it is not a +serialized API schema: + +```yaml +selected_planner_dag: + query_id: p99-api-latency + root: estimate-p99 + nodes: + - input: request_latency_seconds + - group_by: [service] + - build_summary: {algorithm: kll, k: 200} + - estimate: {quantile: 0.99} + +query_requirements: + relative_error: 0.01 + response_latency_ms: 200 + +lifecycle_commitment: + mode: batch_rebuild_from_data_at_rest + rebuild_every: 1m + retain_for: 10m + +backend_capabilities_and_evidence: + supported_modes: [batch_rebuild_from_data_at_rest] + supported_algorithms: [kll] + kll_200_state_bytes: 4096 + five_minute_rebuild_cpu_ms: 35 + +installation_context: + catalog_version: 12 + state_schema: kll-v1 + plan_generation: 42 +``` + +The selected DAG states *what* may answer the query. Requirements state the +promises the selected implementation must meet. The lifecycle commitment states +how this backend will keep the summary available. Capabilities and evidence prove +that the concrete KLL implementation is eligible and provide its physical cost. +The installation context supplies the identities and schema needed to bind the +resulting PrecomputePlan and QueryPlan into one generation. + Capabilities restrict the choices the Planner may consider. For example, a backend that can only build summaries from data at rest advertises only that lifecycle. The Planner still models other lifecycle modes, but it must not select From dd0a8b4d8ab5f1aa1705f4d9f9626360224167c6 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 03:44:03 +0000 Subject: [PATCH 08/15] docs: add physical compiler output example --- docs/design_docs/asapplanner-integration.md | 54 +++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 8eb592cc2..94e72b401 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -109,6 +109,60 @@ The compiler produces one coherent backend publication: These outputs are derived from the same compiler bindings. They must not make independent choices about summary semantics, grouping, windows or schemas. +For the `p99-api-latency` input above, a conceptual compiler output is: + +```yaml +summary_catalog: + definitions: + - id: def-api-latency-kll + input: request_latency_seconds + group_by: [service] + range: 5m + algorithm: {kind: kll, k: 200} + materializations: + - id: mat-api-latency-kll-g42 + definition: def-api-latency-kll + schema: kll-v1 + generation: 42 + +precompute_plan: + generation: 42 + nodes: + - {id: read-samples, op: ReadInput, metric: request_latency_seconds} + - {id: group-service, op: GroupBy, labels: [service]} + - {id: build-kll, op: BuildKll, k: 200} + - id: write-kll + op: WriteState + materialization: mat-api-latency-kll-g42 + edges: + - [read-samples, group-service] + - [group-service, build-kll] + - [build-kll, write-kll] + +query_plan: + generation: 42 + query_id: p99-api-latency + nodes: + - id: read-kll + op: ReadState + materialization: mat-api-latency-kll-g42 + schema: kll-v1 + - {id: estimate-p99, op: SummaryEstimate, quantile: 0.99} + - {id: result, op: QueryResult} + edges: + - [read-kll, estimate-p99] + - [estimate-p99, result] + +provenance: + planner.build_summary: [precompute.build-kll, precompute.write-kll] + planner.estimate-p99: [query.read-kll, query.estimate-p99] +``` + +`mat-api-latency-kll-g42` is the join point: PrecomputePlan writes it, +QueryPlan reads it, and the catalog supplies its definition and schema. The +provenance mapping explains how both physical projections came from the selected +Planner DAG without making that DAG executable inside PrecomputePlan. + ## Ownership | Layer | Owns | Does not own | From c1fbfb6000ec081bed6135e14b3f376290d31a0b Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 03:48:33 +0000 Subject: [PATCH 09/15] docs: include query expression in compiler example --- docs/design_docs/asapplanner-integration.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 94e72b401..a3a8edadc 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -57,6 +57,12 @@ serialized API schema: ```yaml selected_planner_dag: query_id: p99-api-latency + query_language: clickhouse_sql + query_expression: >- + SELECT service, quantile(0.99)(request_latency_seconds) + FROM metrics + WHERE timestamp > now() - INTERVAL 5 MINUTE + GROUP BY service root: estimate-p99 nodes: - input: request_latency_seconds @@ -142,6 +148,12 @@ precompute_plan: query_plan: generation: 42 query_id: p99-api-latency + query_language: clickhouse_sql + query_expression: >- + SELECT service, quantile(0.99)(request_latency_seconds) + FROM metrics + WHERE timestamp > now() - INTERVAL 5 MINUTE + GROUP BY service nodes: - id: read-kll op: ReadState From 887c526be9e868762c9725d87308dc5e0e7dbd5f Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 03:52:45 +0000 Subject: [PATCH 10/15] docs: reorganize physical plan integration design --- docs/design_docs/asapplanner-integration.md | 300 +++++++++----------- 1 file changed, 127 insertions(+), 173 deletions(-) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index a3a8edadc..c805dd95b 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -3,56 +3,61 @@ Status: proposed backend architecture. Audience: developers changing the Planner-to-backend compilation and execution boundary. -## Scope +## Purpose and scope -This document defines how one selected ASAPPlanner semantic DAG becomes two -backend-executable plans: +This design splits one selected ASAPPlanner semantic DAG into two executable +backend plans: - **PrecomputePlan** produces and maintains stored summary state. - **QueryPlan** reads stored state and computes query results. -The two plans share catalog identities and state contracts defined by the -[Summary Catalog and SDS design](summary-catalog-sds-architecture.md). The -[migration plan](asapplanner-migration-plan.md) describes how to reach this -architecture from the current implementation. +Both plans use identities and state contracts from the +[SDS design](summary-catalog-sds-architecture.md) and install as one generation. +The [migration plan](asapplanner-migration-plan.md) defines delivery steps. +CollectorPlan, TransmissionPlan and distributed activation are deferred; this +migration must not introduce a backend dependency on ASAPCollector. -CollectorPlan and TransmissionPlan are outside the current implementation scope. -They may become additional projections of the same selected decision later, but -the backend migration must neither redesign them nor depend on ASAPCollector. +## Document map -## Problem +1. [Architecture at a glance](#architecture-at-a-glance) +2. [Worked example](#worked-example) +3. [Core concepts and ownership](#core-concepts-and-ownership) +4. [Compiler contract](#compiler-contract) +5. [Compilation rules](#compilation-rules) +6. [Runtime contract](#runtime-contract) +7. [Validation and acceptance](#validation-and-acceptance) +8. [Decisions and deferred work](#decisions-and-deferred-work) -The current `PrecomputePlan.executable_dags` can contain the complete selected -semantic DAG. For a query such as: +## Architecture at a glance -```text -Input -> Sum -> KLL -> SummaryEstimate -> QueryResult -``` +The current `PrecomputePlan.executable_dags` can contain a complete semantic DAG, +including query-time nodes such as `SummaryEstimate`. Bindings may prevent those +nodes from running during maintenance, but the artifact and its visualization do +not express that ownership clearly. -`Input -> Sum -> KLL` is maintenance work. `SummaryEstimate -> QueryResult` is -query-time work. Storing the complete DAG under PrecomputePlan makes ownership -unclear even when bindings prevent query-time nodes from running during -maintenance. It also makes a PrecomputePlan visualization look as though -`SummaryEstimate` executes while state is being built. +The compiler instead binds stored summaries once and cuts the DAG at each +materialization boundary: -The target design records one materialization boundary and derives two explicit -executable subgraphs. Semantic provenance remains available without placing -query-only operators in PrecomputePlan. +```mermaid +flowchart LR + D[Selected Planner DAG] --> C[Physical compiler] + C --> P[PrecomputePlan] + C --> S[Summary Catalog / SDS] + C --> Q[QueryPlan] + P -->|write state| S + S -->|bound state reference| Q +``` -## Inputs and outputs +Semantic provenance remains available, but query-only operators are not +PrecomputePlan executable content. -The physical compiler consumes: +## Worked example -- selected Planner DAG roots and their query associations; -- query requirements, including accuracy and response constraints; -- complete lifecycle commitments for the supported backend mode; -- backend capabilities and concrete implementation evidence; -- catalog, schema and deployment-generation inputs. +Query `p99-api-latency` asks for the 99th percentile of five minutes of latency, +grouped by `service` and evaluated every minute. The YAML below is conceptual; it +is not the current serialized API schema. -For example, suppose query `p99-api-latency` asks for the 99th percentile of -`request_latency_seconds` over five minutes, grouped by `service`, every minute. -The following conceptual input shows what each category contributes; it is not a -serialized API schema: +### Compiler input ```yaml selected_planner_dag: @@ -91,31 +96,7 @@ installation_context: plan_generation: 42 ``` -The selected DAG states *what* may answer the query. Requirements state the -promises the selected implementation must meet. The lifecycle commitment states -how this backend will keep the summary available. Capabilities and evidence prove -that the concrete KLL implementation is eligible and provide its physical cost. -The installation context supplies the identities and schema needed to bind the -resulting PrecomputePlan and QueryPlan into one generation. - -Capabilities restrict the choices the Planner may consider. For example, a -backend that can only build summaries from data at rest advertises only that -lifecycle. The Planner still models other lifecycle modes, but it must not select -one the backend cannot execute. - -The compiler produces one coherent backend publication: - -| Output | Responsibility | -| --- | --- | -| Summary Catalog/SDS entries | Define summary semantics, materialization identity, state schema and state references | -| PrecomputePlan | Execute maintenance subgraphs that terminate in stored-state writes | -| QueryPlan | Execute materialization reads, query-time summary operators and exact residuals | -| Provenance mapping | Relate physical nodes and state references to the selected semantic DAG | - -These outputs are derived from the same compiler bindings. They must not make -independent choices about summary semantics, grouping, windows or schemas. - -For the `p99-api-latency` input above, a conceptual compiler output is: +### Compiler output ```yaml summary_catalog: @@ -137,9 +118,8 @@ precompute_plan: - {id: read-samples, op: ReadInput, metric: request_latency_seconds} - {id: group-service, op: GroupBy, labels: [service]} - {id: build-kll, op: BuildKll, k: 200} - - id: write-kll - op: WriteState - materialization: mat-api-latency-kll-g42 + - {id: write-kll, op: WriteState, + materialization: mat-api-latency-kll-g42} edges: - [read-samples, group-service] - [group-service, build-kll] @@ -155,10 +135,8 @@ query_plan: WHERE timestamp > now() - INTERVAL 5 MINUTE GROUP BY service nodes: - - id: read-kll - op: ReadState - materialization: mat-api-latency-kll-g42 - schema: kll-v1 + - {id: read-kll, op: ReadState, + materialization: mat-api-latency-kll-g42, schema: kll-v1} - {id: estimate-p99, op: SummaryEstimate, quantile: 0.99} - {id: result, op: QueryResult} edges: @@ -171,144 +149,120 @@ provenance: ``` `mat-api-latency-kll-g42` is the join point: PrecomputePlan writes it, -QueryPlan reads it, and the catalog supplies its definition and schema. The -provenance mapping explains how both physical projections came from the selected -Planner DAG without making that DAG executable inside PrecomputePlan. +QueryPlan reads it, and SDS defines its meaning and schema. Provenance relates +both physical projections to the selected DAG without making that DAG executable +inside PrecomputePlan. -## Ownership +## Core concepts and ownership -| Layer | Owns | Does not own | -| --- | --- | --- | -| ASAPPlanner | Semantic candidates, legality, accuracy reasoning and selection among advertised capabilities | Backend state IDs, storage schema or runtime installation | -| Physical compiler | Concrete implementation commitment, subgraph split, catalog bindings and plan generation | Re-optimizing a selected DAG at query time | -| Precompute runtime | Executing installed maintenance nodes and publishing state | Query result operators or selecting a different materialization | -| Query runtime | Reading bound state and executing installed query nodes | Creating missing summaries or searching the catalog for alternatives | -| SDS/catalog | Identity, schema, state references, readiness and lifecycle metadata | Operator scheduling or candidate ranking | +“Maintenance” is the execution phase that constructs or updates state, including +batch construction, rebuilding, merging and derived summaries. “Precompute” names +the plan and engine responsible for that work; it does not imply incremental +maintenance. -## Executable subgraphs and materialization boundaries +Bindings describe the semantic-to-physical mapping: -The compiler first binds every selected summary-producing node to one -materialization definition. It then cuts the selected DAG at stored-state -boundaries. +| Binding | Meaning | Example | +| --- | --- | --- | +| `Materialization` | PrecomputePlan stores this node's output | `KLL` in `KLL(sum(data))` | +| `MaintenanceInput` | PrecomputePlan executes this input/intermediate without storing it independently | `sum(data)` feeding KLL | +| `Query` | The node maps to an explicit QueryPlan operation | `SummaryEstimate` | +| `QueryInput` | Query semantics are absorbed into another physical operation | A quantile parameter compiled into `SummaryEstimate` | -```mermaid -flowchart LR - subgraph P[PrecomputePlan] - I[Input] --> S[Sum] - S --> K[Build KLL] - K --> W[Write state] - end - W -->|materialization ID + schema| R - subgraph Q[QueryPlan] - R[Read state] --> E[SummaryEstimate] - E --> O[Query result] - end -``` +| Layer | Owns | +| --- | --- | +| ASAPPlanner | Semantic candidates, legality, accuracy reasoning and selection among advertised capabilities | +| Physical compiler | Concrete implementation, subgraph split, catalog bindings and plan generation | +| Precompute runtime | Installed maintenance nodes and state publication | +| Query runtime | Bound state reads, query operators, exact residuals and fallback | +| SDS/catalog | Definition, materialization, schema, state reference, readiness and lifecycle metadata | -PrecomputePlan contains: +## Compiler contract -- source reads accepted by the maintenance runtime; -- exact or summary operators needed to produce stored state; -- reads of completed prior state for supported derived summaries; -- explicit stored-state sinks. +The compiler consumes: -QueryPlan contains: +- selected Planner DAG roots and query associations; +- query accuracy and response requirements; +- complete lifecycle commitments for the supported backend mode; +- backend capabilities and concrete implementation evidence; +- catalog, schema and deployment-generation inputs. -- explicit reads of materialized state; -- `SummaryEstimate`, merge and other query-time summary operations; -- exact residual subtrees and result composition; -- the configured fallback or unavailable-result behavior. +Capabilities constrain Planner choices. A data-at-rest-only backend advertises +only batch construction; recurring query demand does not imply incremental +support. -A semantic node may be represented inside a larger physical operation. The -provenance mapping records that relationship without requiring a one-to-one -physical node. +| Output | Responsibility | +| --- | --- | +| Catalog/SDS entries | Summary semantics, materialization identity, schema and state references | +| PrecomputePlan | Maintenance subgraphs ending in state writes | +| QueryPlan | Bound state reads, query operators and exact residuals | +| Provenance | Physical-to-semantic node mapping | -## Binding meanings +The compiler derives all four outputs from the same bindings. They cannot choose +summary semantics, grouping, time ranges or schemas independently. -Bindings explain how semantic nodes map to the two physical plans. They do not -create a third execution phase. +## Compilation rules -| Binding | Meaning | Example | -| --- | --- | --- | -| `Materialization` | The node's output is written as stored summary state by PrecomputePlan | `KLL` in `KLL(sum(data))` | -| `MaintenanceInput` | The node executes in PrecomputePlan as an input or intermediate, but its output is not independently stored | `sum(data)` feeding the KLL builder | -| `Query` | The node maps to an explicit QueryPlan operation | `SummaryEstimate` reading the KLL state | -| `QueryInput` | The node contributes query semantics but is absorbed into another QueryPlan operation | A scalar parameter or predicate compiled into a bound read/operator | +### Executable subgraphs and materialization boundaries -“Maintenance” names an execution phase that constructs or updates state. It can -include initial batch construction, rebuilding, merging and derived-summary -construction; it does not imply incremental processing only. “Precompute” names -the backend plan and engine responsible for that work. +For every selected stored summary, the compiler: -## Shared and derived materializations +1. Creates or reuses one compatible summary definition and materialization. +2. Places source reads, maintenance operators, derived-state reads and the state + sink in PrecomputePlan. +3. Replaces the stored-summary edge in QueryPlan with an explicit state read. +4. Places `SummaryEstimate`, merges, exact residuals and result composition in + QueryPlan. +5. Records provenance for semantic nodes absorbed into larger physical nodes. -Two queries may share a producer only when their bound definition and required -state partition are compatible. Sharing one producer must not multiply updates. -Each QueryPlan retains its own readout and result operators. +Two queries may share a producer only when their definition and state partition +are compatible. Sharing does not multiply maintenance updates; each query keeps +its own readout operators. -A derived materialization is still maintenance work: +A derived materialization reads completed state explicitly: ```text -PrecomputePlan: Read completed state A -> derive state B -> store B +PrecomputePlan: Read state A -> derive state B -> store B QueryPlan: Read state B -> estimate -> result ``` -The dependency on A is an explicit state reference with completeness and schema -requirements. QueryPlan does not execute the derivation on demand unless the -selected physical plan explicitly models it as query work. +## Runtime contract -## Validation and installation +The backend stages the catalog and both plans as one generation and exposes them +atomically. Failed staging leaves the previous generation active. -Compilation and backend installation apply the same cross-plan checks: +Installation and readiness are distinct. Until required state coverage exists, +QueryPlan uses its configured exact fallback or returns explicit unavailability. +The query runtime follows installed state references; it does not search the +catalog for alternative summaries. -- every state read resolves to one definition and permitted materialization; -- writer and reader agree on family, parameters, encoding and schema version; -- grouping, time partition, alignment and generation are compatible; -- every executable node is reachable from the correct plan root; -- each subgraph is acyclic and contains only operators supported in that phase; -- query fallback behavior is explicit; -- derived-state inputs satisfy their completeness requirement. +Visualization renders PrecomputePlan and QueryPlan separately, connected by +labeled materialization references. Legacy full-DAG artifacts may use a projected +view, but it must label maintenance-owned and query-owned nodes. -The backend stages the catalog, PrecomputePlan and QueryPlan as one generation. -They become visible atomically. Installation success does not mean state is ready: -until required coverage exists, QueryPlan follows its exact fallback or returns -explicit unavailability. Failed staging leaves the previous generation active. +## Validation and acceptance -## Visualization +Compilation and installation reject unresolved state references, schema/encoding +mismatches, incompatible grouping or time partitions, wrong generations, cycles, +unsupported phase operators and unsatisfied derived-state completeness. -The plan viewer renders PrecomputePlan and QueryPlan separately and connects them -with labeled state references. It shows materialization ID, state family/schema -and readiness where useful. Query-only nodes never appear inside the executable -PrecomputePlan view. +Acceptance tests demonstrate: -Legacy artifacts that embed complete semantic DAGs may be shown through a -projected view, but the UI must label that projection and identify which nodes -are maintenance-owned and query-owned. A separate semantic-plan page is not -required to understand the two executable plans. - -## End-to-end acceptance cases - -The design is complete when tests demonstrate: - -1. `Input -> Sum -> KLL` executes only in PrecomputePlan, while - `SummaryEstimate -> QueryResult` executes only in QueryPlan. -2. One query can read multiple bound summaries. -3. Two queries can share one compatible producer without duplicate updates. -4. A supported derived summary reads completed state and publishes a distinct - state reference. -5. Wrong schema, grouping, time partition or generation fails before activation. -6. Staging failure, restart and generation switching preserve the previous - consistent plan and documented fallback behavior. -7. The backend builds and runs these cases without ASAPCollector. +1. Summary construction executes only in PrecomputePlan and estimation only in + QueryPlan. +2. One query can read multiple summaries and two queries can share one producer. +3. Derived summaries honor completion and schema requirements. +4. Invalid cross-plan bindings fail before activation. +5. Staging failure, restart and generation switching preserve consistency and + documented fallback behavior. +6. The backend builds and runs these cases without ASAPCollector. ## Decisions and deferred work -We reject keeping the full semantic DAG as PrecomputePlan executable content: -bindings alone do not make plan ownership clear. We also reject compiling the -two plans independently because that permits identity and schema drift. - -The selected semantic DAG may remain as provenance or diagnostic metadata. It is -not a third executable plan. +The full semantic DAG is retained only as provenance or diagnostic metadata; +bindings alone do not make it valid PrecomputePlan executable content. The two +physical plans are not compiled independently because that permits identity and +schema drift. Deferred work includes CollectorPlan and TransmissionPlan compilation, distributed activation, new transport/checkpoint protocols, Collector adoption of neutral From b8193bb32048eb2171193fad2edc14245c7c73bf Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 04:20:29 +0000 Subject: [PATCH 11/15] docs: reorganize SDS and migration designs --- .../design_docs/asapplanner-migration-plan.md | 209 +++++------ .../summary-catalog-sds-architecture.md | 325 ++++++++---------- 2 files changed, 252 insertions(+), 282 deletions(-) diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index 6d969b615..491d04aeb 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -4,139 +4,156 @@ Status: proposed delivery sequence. Audience: backend implementers. ## Goal and scope -Migrate the backend from a complete semantic DAG stored under PrecomputePlan to -separate executable PrecomputePlan and QueryPlan subgraphs connected by explicit -SDS state references. - -This migration also removes the backend build/runtime dependency on ASAPCollector. -Shared envelope, schema and sketch reconstruction code moves to neutral libraries. +Replace complete semantic DAGs stored under PrecomputePlan with separate +PrecomputePlan and QueryPlan executable subgraphs connected by SDS state +references. Also remove the backend build/runtime dependency on ASAPCollector by +moving shared contracts and reconstruction code to neutral libraries. CollectorPlan, TransmissionPlan, distributed activation, new transport behavior and a general ASAPPlanner API redesign are deferred. -Completion requires: +## Document map -- `SummaryEstimate` and other query-only work appear only in QueryPlan; -- maintenance work terminates in explicit stored-state writes; -- both plans share one catalog/materialization/schema decision; -- the catalog and both plans install as one backend generation; -- supported legacy payloads and plans retain documented behavior; -- backend builds and required tests do not fetch, build or run ASAPCollector. +1. [Migration at a glance](#migration-at-a-glance) +2. [Worked example](#worked-example) +3. [Stage 1: inventory and fixtures](#stage-1-inventory-and-fixtures) +4. [Stage 2: extract common code](#stage-2-extract-common-code) +5. [Stage 3: bind and split plans](#stage-3-bind-and-split-plans) +6. [Stage 4: validate and install](#stage-4-validate-and-install) +7. [Stage 5: migrate and retire](#stage-5-migrate-and-retire) +8. [Completion evidence](#completion-evidence) -## Delivery stages +## Migration at a glance | Stage | Change | Exit gate | | --- | --- | --- | -| 1. Inventory and fixtures | Record current contracts, imports, payloads and execution behavior | Every scoped path has a compatibility fixture or explicit unsupported result | -| 2. Extract common code | Move runtime-independent contracts and reconstruction to neutral libraries | Backend dependency graph and required tests contain no ASAPCollector | -| 3. Bind and split plans | Compile one decision into catalog entries, maintenance subgraphs and query subgraphs | Executable ownership and state references match selected semantics | -| 4. Validate and install | Add cross-plan validation, atomic generation switching and two-plan visualization | Invalid publications fail before activation; previous generation survives failure | -| 5. Migrate and retire | Normalize old artifacts and remove superseded execution paths | Compatibility and end-to-end gates pass | +| 1. Inventory | Freeze current contracts and behavior as fixtures | Every supported path has a fixture or explicit unsupported result | +| 2. Extract | Move neutral contracts/codecs out of Collector | Backend dependencies and tests contain no ASAPCollector | +| 3. Split | Derive catalog, maintenance DAGs and query DAGs from one binding | Ownership and state references match selected semantics | +| 4. Install | Validate and atomically activate one generation | Invalid snapshots fail without disturbing the active generation | +| 5. Retire | Normalize old artifacts and remove superseded paths | Compatibility and end-to-end gates pass | -## 1. Inventory and fixtures +Do not combine payload-format changes with dependency extraction. Version the new +plan representation separately from any later wire/schema change. -Inventory the pinned Planner output, PrecomputePlan/QueryPlan/SDS types, state -schemas, envelope definitions, all `asap_precompute_rs` imports, Cargo patches, -build scripts and tests that invoke Collector. +## Worked example -Capture fixtures for supported: +The current artifact may store this complete DAG under PrecomputePlan: -- raw input and summary reconstruction; -- full, delta and legacy bare-state payloads; -- maintenance updates and query readout; -- completion, restart and recovery; -- plan staging, activation and fallback. +```text +Input -> BuildKLL -> SummaryEstimate -> Result +``` -Fixtures may originate from Collector but must run without a Collector checkout or -process. Record their source revision and schema provenance. Use semantic readout -assertions where randomized sketch bytes are not stable. +The migration produces: -The backend capability profile is an input to Planner selection. Preserve complete -lifecycle commitments, even when the only supported choice is batch construction -from data at rest. Do not infer incremental support from recurring query demand. +```yaml +summary_catalog: + materialization: {id: mat-17, schema: kll-v1, generation: 42} -## 2. Extract common contracts and codecs +precompute_plan: + nodes: [Input, BuildKLL, 'WriteState(mat-17)'] -Use narrow neutral-library boundaries: +query_plan: + nodes: ['ReadState(mat-17)', SummaryEstimate, Result] -| Library responsibility | Must exclude | -| --- | --- | -| Envelope metadata, shared IDs/schema references and validation | Planner optimization and backend/Collector executors | -| Sketch payload schemas, encode/decode/reconstruction and supported state operations | Window scheduling, host adapters and backend storage | +provenance: + selected_dag: Input -> BuildKLL -> SummaryEstimate -> Result +``` -Prefer existing sketch-library APIs. Move reusable DDSketch/KLL reconstruction -out of Collector wrappers and remove unnecessary reconstruct-serialize-decode -round trips. Preserve legacy readers and other family-specific backend paths until -replacement APIs have parity evidence. +During rollout, the backend normalizes a supported legacy artifact into this +internal form. Old and new forms must produce the same update count and query +result. After compatibility gates pass, the complete-DAG execution path can be +removed while its versioned reader remains for the supported window. -Remove `asap-precompute-rs` and obsolete Collector-specific Cargo patches. Check -manifests, lockfiles, dependency graphs, scripts and required tests for direct and -transitive Collector dependencies. +## Stage 1: inventory and fixtures -Do not change payload bytes during extraction. Version any later wire/schema -change separately. +Inventory Planner output, plan/SDS types, state schemas, envelopes, +`asap_precompute_rs` imports, Cargo patches, build scripts and tests that invoke +Collector. -## 3. Bind once and split executable subgraphs +Capture fixtures for: -Create compiler-local bindings for selected semantic nodes, summary definitions, -materializations, state schemas and read/write references. Derive the catalog and -both plans from those bindings. +- full, delta and legacy bare-state decoding; +- summary reconstruction, maintenance updates and query readout; +- completion, restart and recovery; +- staging, activation, readiness and fallback. -Apply the [materialization-boundary rules](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries): +Fixtures may originate from Collector but must run without a Collector checkout +or process. Record source revision and schema provenance; use semantic assertions +when randomized sketch bytes are unstable. -- PrecomputePlan contains maintenance inputs/operators and stored-state sinks. -- QueryPlan contains state reads, `SummaryEstimate`, exact residuals and result - composition. -- Derived maintenance uses explicit completed-state references. -- Shared producers retain one materialization identity and update path. -- Absorbed semantic operations remain visible through provenance mappings. +Preserve complete lifecycle commitments from Planner selection. A backend that +only supports batch construction from data at rest must not infer incremental +support from recurring query demand. -The selected semantic DAG may remain diagnostic metadata, but it is not -PrecomputePlan executable content. +## Stage 2: extract common code -Version the new installed representation. Normalize supported legacy publications -at the backend boundary; do not reinterpret an old field under an unchanged -schema version. +| Neutral responsibility | Excludes | +| --- | --- | +| Envelope metadata, shared IDs/schema references and validation | Planner optimization and runtime executors | +| Sketch schemas, encode/decode/reconstruction and supported state operations | Window scheduling, host adapters and backend storage | -## 4. Validate, install and visualize +Prefer existing sketch-library APIs. Move reusable DDSketch/KLL reconstruction +out of Collector wrappers and remove reconstruct-serialize-decode round trips. +Keep legacy readers and family-specific backend paths until replacements have +parity evidence. -At compilation and installation, verify definition, materialization, schema, -encoding, grouping, time partition, coverage requirements and generation across -both plans. Then perform backend-local resource checks. +Remove `asap-precompute-rs` and Collector-specific Cargo patches. Inspect +manifests, lockfiles, dependency graphs, scripts and required tests for direct or +transitive Collector dependencies. + +## Stage 3: bind and split plans + +Create compiler bindings for semantic nodes, summary definitions, +materializations, schemas and state references. Derive the catalog and both plans +from those bindings using the +[materialization-boundary rules](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries): + +- PrecomputePlan contains maintenance inputs/operators and state sinks. +- QueryPlan contains state reads, `SummaryEstimate`, exact residuals and results. +- Derived maintenance uses explicit completed-state references. +- Shared producers retain one identity and update path. +- Provenance records semantic operations absorbed into physical nodes. -Stage the catalog and two plans as one snapshot and activate them atomically. -State readiness remains separate from installation. Until required coverage is -ready, QueryPlan uses its configured exact fallback or explicit unavailability. +Version the split representation. Do not reinterpret an old field under an +unchanged schema version. -Render PrecomputePlan and QueryPlan separately, joined by labeled state references. -Legacy full-DAG views must label maintenance-owned and query-owned projections. +## Stage 4: validate and install -Acceptance cases: +Validate definition, materialization, schema, encoding, grouping, time partition, +coverage and generation across the catalog and both plans. Then perform local +resource checks. -- build-summary/read-estimate executes in the correct plan; -- one query reads multiple summaries; -- two queries share one compatible producer without duplicate updates; -- supported derived state observes completion requirements; -- wrong schema, grouping, time partition or generation fails before activation; -- failed staging, restart and generation switching preserve consistency; -- old and new supported artifacts produce equivalent results and update counts; -- all cases run without ASAPCollector. +Stage and activate the three artifacts as one snapshot. Readiness remains +separate: until coverage is ready, QueryPlan follows its configured fallback or +explicit unavailability. Failed staging preserves the previous generation. -## 5. Migrate and retire +Render PrecomputePlan and QueryPlan separately, joined by state references. +Legacy projected views label maintenance-owned and query-owned nodes. -Release the backend with pinned neutral-library versions and rollback artifacts. -Migrate backend-local publications first. Retain versioned adapters for the -supported compatibility window. +## Stage 5: migrate and retire + +Release pinned neutral-library versions and rollback artifacts. Migrate +backend-local publications first and retain versioned adapters for the supported +compatibility window. Remove complete-DAG precompute execution and Collector adapter code only after -their replacements pass fixtures and end-to-end tests. Reusing state across plan -generations requires an explicit SDS compatibility decision independently of -binary rollback. +fixtures and end-to-end tests pass. State reuse across generations requires an +explicit SDS compatibility decision independently of binary rollback. + +## Completion evidence -## Final evidence +Completion requires: + +- summary construction runs only in PrecomputePlan and estimation only in + QueryPlan; +- one query can read multiple summaries and two queries can share one producer; +- derived state observes completion and schema requirements; +- invalid bindings fail before activation; +- restart and generation switching preserve consistency and fallback; +- legacy and split artifacts produce equivalent results and update counts; +- backend builds and required tests do not fetch, build or run ASAPCollector. Record tested revisions, supported state families, fixture results and dependency -checks. Trace at least one query from its selected semantic root through the -materialization boundary, PrecomputePlan writer, SDS state reference and QueryPlan -reader. Completion depends on the two-plan acceptance cases and zero backend -dependency on ASAPCollector, not on deferred distributed work. +checks. Trace one query from its selected semantic root through the materialization +writer, SDS reference and QueryPlan reader. diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index 5f47f418b..c057746b7 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -1,245 +1,198 @@ # Summary Catalog and Self-Describing Summary architecture -Status: proposed contract with notes on the current backend representation. -Audience: developers compiling, storing, recovering or reading summary state. +Status: proposed contract with current-backend migration notes. Audience: +developers compiling, storing, recovering or reading summary state. ## Purpose and scope -The Summary Catalog and Self-Describing Summary (SDS) model is the authority for -persisted summary-state meaning. It connects PrecomputePlan writers to QueryPlan -readers without requiring either runtime to reinterpret Planner IR. +The Summary Catalog and Self-Describing Summary (SDS) model defines what persisted +summary state means. It connects PrecomputePlan writers to QueryPlan readers +without requiring either runtime to reinterpret Planner IR. -This document owns: +This document owns summary identity, schema, state references, readiness and +lifecycle. The [integration design](asapplanner-integration.md) owns executable +plan splitting; the [migration plan](asapplanner-migration-plan.md) owns delivery. +Cost ranking, operator scheduling and transmission policy are outside SDS. -- stable summary semantics and materialization identity; -- state schema, encoding and partition identity; -- references from plans to stored state; -- readiness, generation and retirement metadata; -- validation required when state is written, recovered or read. +## Document map -The [integration design](asapplanner-integration.md) owns physical subgraph -splitting and execution. The [migration plan](asapplanner-migration-plan.md) -owns delivery order. Cost evidence, candidate ranking, operator scheduling and -transmission policy are outside the SDS model. +1. [Architecture at a glance](#architecture-at-a-glance) +2. [Worked example](#worked-example) +3. [Core objects](#core-objects) +4. [Identity and reference rules](#identity-and-reference-rules) +5. [Plan and storage contract](#plan-and-storage-contract) +6. [Lifecycle and readiness](#lifecycle-and-readiness) +7. [Validation and migration](#validation-and-migration) +8. [Deferred work](#deferred-work) -## Core model - -| Object | Meaning | Stability | -| --- | --- | --- | -| `SummaryDefinition` | Canonical semantics of a summary: input, operation, grouping, time semantics, algorithm and parameters | Stable while those semantics remain unchanged | -| `Materialization` | A physical-plan decision to produce a definition with a particular state contract | Versioned with the installed plan generation | -| `SummaryStateInstance` | One persisted state partition, such as a series/pane or completed aggregate | Created and retired by runtime lifecycle | -| `StateReference` | A typed reference used by QueryPlan or a derived PrecomputePlan node | Valid only for compatible definition, schema and generation rules | +## Architecture at a glance The catalog stores definitions and materializations. Runtime inventory records -state instances. Plans carry state references rather than embedding payloads or +state instances. Plans carry typed state references rather than payloads or search predicates. ```mermaid flowchart LR D[SummaryDefinition] --> M[Materialization] - M --> I1[State instance] - M --> I2[State instance] - P[PrecomputePlan writer] --> M - Q[QueryPlan reader] --> R[StateReference] + M --> I[State instances] + P[PrecomputePlan] -->|write| M + Q[QueryPlan] --> R[StateReference] R --> M ``` -## Summary definition - -A definition contains all fields required to decide whether two summaries have -the same meaning: - -- canonical input source and filters; -- input value semantics; -- exact operation or sketch family and typed parameters; -- grouping and reduction semantics; -- query-range/time-partition semantics and alignment; -- accuracy contract where it affects state meaning; -- output value type. - -Display names, plan generation, readiness, storage location, retention status and -observed costs do not belong to definition identity. Changing a semantic field -creates a different definition instead of mutating an existing one. - -Definitions may refer to raw input or to another completed summary definition. -Derived input references are typed dependencies, not metric-name aliases. - -## Materialization - -A materialization commits a definition to a concrete state contract: - -- stable definition ID; -- materialization ID and plan generation; -- state family, schema version and encoding; -- physical grouping and partition layout; -- permitted producer/writer identity where required; -- lifecycle and readiness policy; -- provenance back to selected semantic nodes. - -Multiple materializations may implement the same definition, for example across -plan generations or storage migrations. QueryPlan reads a compiler-selected -materialization reference; the serving runtime does not search all catalog entries -for a substitute. - -## Summary state instance - -A state instance identifies one physical partition of a materialization. Its key -contains only dimensions needed to distinguish stored state, such as series or -group identity, time partition, producer/shard identity and generation. Its -metadata records: - -- materialization and definition IDs; -- exact schema/encoding used by the payload; -- coverage or completion bounds; -- producer sequence/checkpoint metadata when applicable; -- creation, readiness and retirement state; -- content location and integrity information. +These objects remain distinct because “same summary semantics,” “same production +decision,” and “same stored payload” have different compatibility rules. + +## Worked example + +Two queries request different percentiles from the same five-minute KLL summary: + +```yaml +summary_definition: + id: def-api-latency-kll + input: request_latency_seconds + group_by: [service] + range: 5m + algorithm: {kind: kll, k: 200} + +materialization: + id: mat-api-latency-kll-g42 + definition: def-api-latency-kll + generation: 42 + schema: kll-v1 + +state_instances: + - id: state-api-1200 + materialization: mat-api-latency-kll-g42 + partition: {service: api, start: '12:00', end: '12:01'} + status: ready + - id: state-api-1201 + materialization: mat-api-latency-kll-g42 + partition: {service: api, start: '12:01', end: '12:02'} + status: ready + +query_state_references: + q50: {materialization: mat-api-latency-kll-g42, quantile: 0.50} + q99: {materialization: mat-api-latency-kll-g42, quantile: 0.99} +``` -Payload bytes are stored in the summary store, not copied into the catalog -descriptor. Mutable runtime statistics do not change semantic identity. +PrecomputePlan updates each state partition once. Both QueryPlans resolve the +same bound materialization and apply different readout parameters. They neither +create duplicate producers nor search the catalog for alternatives at serving +time. -## State reference +## Core objects -A state reference is the only normal connection between executable plans and -stored state. It identifies the required materialization and constrains the state -partition, schema and generation that may satisfy the read. +| Object | Meaning | Changes when | +| --- | --- | --- | +| `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change | +| `Materialization` | An installed decision to produce a definition with one state contract | Plan generation or physical contract changes | +| `SummaryStateInstance` | One stored partition, such as a series/pane or completed aggregate | Runtime creates or replaces payload state | +| `StateReference` | A typed plan reference to permitted materialized state | A compiled reader/writer binding changes | -PrecomputePlan uses state references for derived-summary inputs. QueryPlan uses -them for result-producing reads. A reference may select multiple instances, such -as the panes covering one query range, but it cannot broaden the summary -definition or silently select another algorithm. +A definition includes every field needed to decide semantic equivalence: source +and filters, input value, operation or sketch parameters, grouping, time +semantics, accuracy fields that affect state, and output type. Display names, +costs, locations, readiness and retention status are excluded. -The runtime may resolve physical locations through an index. Resolution must be -an exact lookup under the installed reference and metadata; catalog scanning and -serving-time candidate selection are prohibited. +A materialization adds definition ID, plan generation, state family, schema, +encoding, physical partition layout, permitted writer identity and provenance. +Several generations may materialize the same definition. -## Identity rules +A state instance adds its partition key, coverage/completion, producer sequence +where applicable, lifecycle status, location and integrity metadata. Payload +bytes remain in the summary store, not in catalog descriptors. -The following identities have different purposes and must not be collapsed: +## Identity and reference rules | Identity | Answers | | --- | --- | -| Definition ID | What summary semantics does this state represent? | -| Materialization ID | Which installed physical production decision created it? | -| State-instance ID | Which concrete partition/payload is this? | +| Definition ID | What semantics does the state represent? | +| Materialization ID | Which installed physical decision produced it? | +| State-instance ID | Which concrete partition/payload is it? | | Plan generation | With which atomic installation may it be used? | | Schema/encoding ID | How are its bytes interpreted? | -IDs are assigned or derived once by the compiler/catalog authority and carried -through plans, storage and recovery. Human-readable names are diagnostic labels, -not join keys. A reused state instance across generations requires an explicit -compatibility decision; matching definition IDs alone is insufficient. +The compiler/catalog authority assigns these identities once. Human-readable +names are diagnostics, not join keys. Reuse across generations requires an +explicit compatibility decision; a matching definition ID is insufficient. -## Plan boundary +A `StateReference` identifies one materialization and constrains acceptable +partition, schema, generation and coverage. It may select several instances, such +as panes covering one range, but cannot broaden semantics or substitute another +algorithm. QueryPlan and derived PrecomputePlan nodes resolve references through +exact indexed lookup, never serving-time candidate selection. -The physical-plan split uses the catalog as follows: +## Plan and storage contract ```text PrecomputePlan - Input -> Sum -> BuildKLL -> Write(materialization=mat-17) + Input -> BuildKLL -> Write(mat-17) -Catalog/SDS - mat-17 -> definition=def-9, family=KLL, schema=kll-v1, generation=42 +SDS + mat-17 -> def-9, KLL(k=200), kll-v1, generation 42 QueryPlan - Read(mat-17, schema=kll-v1) -> SummaryEstimate -> Result + Read(mat-17, kll-v1) -> SummaryEstimate -> Result ``` -The writer and reader share the same compiler binding. They must agree on: +Writer, SDS entry and reader must agree on definition, materialization, state +family, parameters, schema/encoding, grouping, time partition and generation. +The query runtime follows the installed reference instead of scanning the catalog. -- definition and materialization identity; -- state family, algorithm parameters, schema and encoding; -- grouping and time partition/alignment; -- generation compatibility and coverage requirements. +A derived materialization has a distinct destination identity and an explicit +reference to completed source state: -For a derived summary, the destination materialization has its own identity and -the maintenance node holds a `StateReference` to its completed source state. The -source and destination are never represented as the same instance. +```text +PrecomputePlan: Read state A -> derive -> Write state B +QueryPlan: Read state B -> estimate -> result +``` -## Lifecycle and readiness +Source and destination are never represented as the same instance. -Materialization intent and observed state are separate: +## Lifecycle and readiness | State | Meaning | | --- | --- | -| `Desired` | Installed plans require the materialization; usable state may not exist yet | -| `Building` | The runtime is producing or recovering required coverage | -| `Ready` | Required schema and coverage are available for the bound reads | -| `Draining` | No new work is assigned, but existing readers or writes are being completed | -| `Retired` | The materialization is unavailable to new reads and may be garbage-collected when safe | - -Activation installs intent atomically but does not manufacture readiness. A -QueryPlan read checks observed readiness and coverage, then follows its configured -fallback or unavailability behavior. Reactivation of a retired definition creates -or binds an authorized materialization; it does not make stale instances current. - -Completed finite-input state is immutable. Further additive writes require a new -authorized generation or replacement instance. Mutable streaming state publishes -monotone coverage/completion metadata according to its installed contract. - -## Validation invariants - -Compilation, installation, writes, recovery and reads enforce these invariants: - -1. Every materialization resolves to exactly one definition. -2. Every state instance resolves to one materialization and declares its actual - schema and encoding. -3. A state reference cannot change definition semantics during resolution. -4. Writer and reader grouping, time partition and schema contracts agree. -5. State from an incompatible generation is rejected before execution. -6. Ready state satisfies the reference's coverage and completion requirements. -7. Derived maintenance reads only completed input when its operator requires it. -8. Retirement prevents new bindings before physical state is reclaimed. -9. Unknown schema, malformed payload and unauthorized producer updates fail - closed; they never become catalog-visible ready state. +| `Desired` | Installed plans require the materialization | +| `Building` | Required state is being produced or recovered | +| `Ready` | Required schema and coverage are available | +| `Draining` | New work has stopped while existing use completes | +| `Retired` | New reads are prohibited; safe reclamation may follow | -## Current representation and migration boundary +Atomic activation installs intent, not ready data. A QueryPlan read checks +observed readiness and coverage, then follows its configured fallback or explicit +unavailability behavior. Reactivation does not make stale instances current. -The backend already has catalog descriptors, policy fingerprints, series IDs, -state metadata and persisted payloads, but responsibilities are distributed -across `asap_types`, control-plane publication and the summary store. Some current -artifacts also embed complete semantic DAGs in PrecomputePlan. - -Migration should reuse authoritative IDs and storage metadata rather than create -a parallel registry. Legacy artifacts are normalized at the backend boundary; -new plans use explicit state references. Existing supported payloads remain -readable through versioned codecs and compatibility fixtures. - -The backend must not depend on ASAPCollector for these contracts or codecs. -Runtime-independent envelope/schema definitions and sketch reconstruction belong -in neutral libraries. Backend storage, scheduling and query execution remain -backend-owned. - -## Example - -Two queries request percentiles over the same grouped input. The compiler selects -one compatible KLL materialization and emits two QueryPlan entries: +Completed finite-input state is immutable. Additional writes require a new +authorized generation or replacement instance. Mutable streaming state publishes +monotone coverage according to its installed contract. -```text -definition def-9: - input=request_latency, group_by=[service], range=5m, algorithm=KLL(k=200) +## Validation and migration -materialization mat-17, generation 42: - definition=def-9, schema=kll-v1 +Compilation, installation, writes, recovery and reads enforce: -state instances: - mat-17/service=api/pane=12:00..12:01 - mat-17/service=api/pane=12:01..12:02 - ... +1. Each materialization resolves to one definition and each instance to one + materialization. +2. Instance metadata declares the payload's actual schema and encoding. +3. References preserve definition semantics and compatible generation. +4. Writer and reader grouping, time partition, schema and coverage agree. +5. Derived reads meet their completion requirement. +6. Retirement blocks new bindings before state reclamation. +7. Unknown schemas, malformed payloads and unauthorized updates fail closed. -query q50: Read(mat-17) -> Estimate(0.50) -query q99: Read(mat-17) -> Estimate(0.99) -``` +The current backend distributes these responsibilities across `asap_types`, +control-plane publication and the summary store. Migration reuses authoritative +IDs and metadata rather than creating a parallel registry. Legacy artifacts are +normalized at the backend boundary and supported payloads retain versioned +readers and fixtures. -The producer updates each state partition once. Both queries resolve the same -bound materialization, compose the required coverage and apply different readout -parameters. Neither query creates a second producer or searches for a different -summary at serving time. +Runtime-independent contracts and sketch reconstruction belong in neutral +libraries. Backend storage, scheduling and query execution remain backend-owned; +the backend must not depend on ASAPCollector. ## Deferred work -This design does not define CollectorPlan or TransmissionPlan, distributed -activation, a new checkpoint protocol, cost/ERP evidence, or retention policy -selection. Those systems may reference SDS identities later without becoming -part of the SDS semantic model. +SDS does not define CollectorPlan, TransmissionPlan, distributed activation, a +new checkpoint protocol, cost/ERP evidence or retention-policy selection. Those +systems may reference SDS identities without becoming part of this model. From b24867b686fd831af20e2b87e98c6a3989c5608c Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 16:37:36 +0000 Subject: [PATCH 12/15] docs: define maintenance inputs before plan split example --- docs/design_docs/asapplanner-integration.md | 186 ++++++++++++++++++-- 1 file changed, 172 insertions(+), 14 deletions(-) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index c805dd95b..7931dc256 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -20,7 +20,8 @@ migration must not introduce a backend dependency on ASAPCollector. ## Document map 1. [Architecture at a glance](#architecture-at-a-glance) -2. [Worked example](#worked-example) +2. [Design definitions and selection](#design-definitions-and-selection) + - [Worked example](#worked-example) 3. [Core concepts and ownership](#core-concepts-and-ownership) 4. [Compiler contract](#compiler-contract) 5. [Compilation rules](#compilation-rules) @@ -35,6 +36,10 @@ including query-time nodes such as `SummaryEstimate`. Bindings may prevent those nodes from running during maintenance, but the artifact and its visualization do not express that ownership clearly. +This is a representation defect tracked by +[issue #740](https://github.com/ProjectASAP/ASAPQuery-backend/issues/740). +The target design requires separate executable projections. + The compiler instead binds stored summaries once and cuts the DAG at each materialization boundary: @@ -51,11 +56,138 @@ flowchart LR Semantic provenance remains available, but query-only operators are not PrecomputePlan executable content. +## Design definitions and selection + +Audience: developers implementing the Planner/backend boundary. The definitions +below describe the target design; the YAML that follows illustrates that design +and is not a serialized Rust API. Implementations should adapt existing types +where they express these requirements rather than introduce duplicate models. + +### Existing representation and target boundary + +The selected post-ASAP DAG describes the selected computation: source operations, +summary producers, shared dependencies and query readouts. Maintenance decisions +are associated with its summary producers through plan-scoped node identities. + +Planner's `SummaryMaintenanceLifecyclePlan` contains a materialized DAG `root` +and a `deployments` collection, with one entry per unique reachable `SummaryAgg`. +Each deployment identifies its `post_asap_node_id` and carries an optional +`SummaryMaintenanceLifecycleGuarantee`, considered alternatives and a selected +window framework. The plan also carries workload demand and costing context. +Thus the lifecycle plan already refers to the computation DAG; it is not a +separate query representation, nor is one whole lifecycle plan required per +producer. A missing guarantee is not an executable maintenance commitment. + +See the Planner +[lifecycle plan types](https://github.com/ProjectASAP/ASAPPlanner/blob/ba1c4436a3410dc03a133363ab5b75649e70f97a/crates/asap-aware-mapping/src/summary_maintenance_lifecycle.rs) +and [guarantee vocabulary](https://github.com/ProjectASAP/ASAPPlanner/blob/ba1c4436a3410dc03a133363ab5b75649e70f97a/crates/types/src/post_asap/summary_maintenance_lifecycle.rs). +These describe the referenced Planner revision, not a claim that every field +below is already supported by the backend's pinned dependency. + +The backend currently records physical node ownership with +[`BackendExecutableBinding`](../../crates/asap_types/src/executable_plan.rs). +The target compiler consumes the selected computation and its maintenance +decisions together, validates them against backend support, and emits the two +physical plans plus catalog bindings. A shared producer is maintained once for +all compatible consumers. + +### Lifecycle commitment + +A **lifecycle commitment** is the selected maintenance promise for one logical +summary producer in a particular selected plan. This is a design term for the +selected guarantee and its concrete scheduling/retention binding, not a proposed +replacement for `SummaryMaintenanceLifecyclePlan`. + +| Field in the example | Definition and constraint | +| --- | --- | +| `producer` | Node identity in the selected DAG; must resolve to a stored summary producer. | +| `mode` | Selected construction/update method. `batch_rebuild_from_data_at_rest` reads persisted input and constructs replacement state for each required coverage interval. | +| `refresh.every` | Spacing of scheduled evaluation endpoints, not elapsed time after the preceding build finishes. | +| `refresh.anchor` | Origin of that schedule; `unix_epoch` with `every: 1m` yields UTC minute boundaries. | +| `retention.completed_state_for` | Minimum duration to retain each completed output snapshot after publication. It is independent of input coverage and raw-data retention. | +| `implementation` | Backend implementation selected to fulfill this commitment. | + +For each endpoint `T`, a rebuild reads exactly the logical input interval for +`T` and publishes state labeled with that coverage. Publication after `T` does +not change the interval. Retention expiry makes a snapshot eligible for cleanup +only after readers and dependent producers release it. A missed or unfinished +build leaves that endpoint unready; the configured fallback/unavailability +policy applies. Reusing an older snapshot requires an explicit query freshness +policy and must not silently change query time semantics. + +Planner supplies legal maintenance alternatives. The backend supplies executable +implementations and evidence; the control plane commits a feasible selection. +The compiler validates that commitment without silently changing its mode, +coverage or sharing. A changed commitment is installed through a new plan +generation. It need not change the semantic summary definition when only the +physical maintenance policy changes. + +### Backend capability + +A **backend capability** is an implementation provider's declaration of a +supported combination of algorithm, parameters, maintenance mode, input kind, +window behavior and state schema. It answers whether a proposed realization can +execute faithfully. Independent global lists of algorithms and modes would +incorrectly imply support for every combination. + +Each capability record has an `implementation` identity, an `algorithm` +configuration, `maintenance_modes`, `input_kind`, `window_support`, and +`state_schema`. The compiler must match the whole record. The example declares +only KLL with `k: 200`, batch rebuilding from stored rows, and complete snapshots +for the requested logical range. It does not establish incremental maintenance +or arbitrary parameter support. A readout implementation alone does not prove +the corresponding producer is supported. + +### Physical cost evidence + +**Physical cost evidence** is a scoped estimate or measurement for one +implementation/configuration and maintenance mode. It is supplied by the backend +provider and used when comparing feasible alternatives over the same planning +horizon. It is separate from both capability and the final commitment. + +An evidence record identifies the implementation, algorithm parameters, mode, +input range, sample count, group count and execution profile. It declares whether +numbers are measured or modeled, their provenance and applicability period. +Measured evidence needs a benchmark identity/time; modeled evidence needs a model +version and assumptions. Missing or stale evidence is not zero cost. + +`state_bytes_per_group` measures one completed summary payload; +`rebuild_cpu_ms_total` measures CPU time for one rebuild across all declared +groups. CPU time is not wall-clock completion latency. Memory, temporary build +space, retained snapshots, I/O and query readout must also be costed before +claiming a complete deployment cost. A five-minute range alone does not determine +sample count or CPU cost. + +### Selection and validation + +```text +Selected computation and lifecycle alternatives + + backend capabilities: supported combinations + + scoped cost evidence: resource costs of those combinations + -> control-plane commitment per selected producer + -> physical compiler validation + -> PrecomputePlan + QueryPlan + catalog bindings +``` + +Before installation, validate producer identity, supported algorithm/mode/schema, +schedule and coverage, retention sufficient for dependent reads, accuracy and +query requirements, and the scope/completeness of cost evidence. Reject an +inconsistent binding instead of inventing missing maintenance policy. Where the +planning interface supports exact fallback, select that explicitly. + +The existing binding/compiler path is the migration starting point. Adapters +must map existing Planner guarantees and backend capabilities into these +requirements, reporting unsupported fields. The plan split must preserve those +decisions in writer and reader bindings. New wire schemas and concrete scheduling +support are implementation work; this document defines their required behavior. + ## Worked example Query `p99-api-latency` asks for the 99th percentile of five minutes of latency, grouped by `service` and evaluated every minute. The YAML below is conceptual; it -is not the current serialized API schema. +is not the current serialized API schema. Resource numbers are fictional, +illustrating units and scope only; they are not benchmark evidence or proof that +this candidate meets accuracy, cost or latency requirements. ### Compiler input @@ -66,29 +198,44 @@ selected_planner_dag: query_expression: >- SELECT service, quantile(0.99)(request_latency_seconds) FROM metrics - WHERE timestamp > now() - INTERVAL 5 MINUTE + WHERE timestamp > :evaluation_time - INTERVAL 5 MINUTE + AND timestamp <= :evaluation_time GROUP BY service root: estimate-p99 nodes: - input: request_latency_seconds - group_by: [service] - - build_summary: {algorithm: kll, k: 200} + - id: build-kll + build_summary: {algorithm: kll, k: 200} - estimate: {quantile: 0.99} query_requirements: - relative_error: 0.01 + accuracy: supplied_by_selected_planner_guarantee response_latency_ms: 200 lifecycle_commitment: + producer: build-kll + implementation: local-kll-batch-v1 mode: batch_rebuild_from_data_at_rest - rebuild_every: 1m - retain_for: 10m - -backend_capabilities_and_evidence: - supported_modes: [batch_rebuild_from_data_at_rest] - supported_algorithms: [kll] - kll_200_state_bytes: 4096 - five_minute_rebuild_cpu_ms: 35 + refresh: {every: 1m, anchor: unix_epoch} + retention: {completed_state_for: 10m} + +backend_capabilities: + - implementation: local-kll-batch-v1 + algorithm: {kind: kll, k: 200} + maintenance_modes: [batch_rebuild_from_data_at_rest] + input_kind: stored_rows + window_support: complete_snapshot_for_requested_range + state_schema: kll-v1 + +physical_cost_evidence: + - implementation: local-kll-batch-v1 + algorithm: {kind: kll, k: 200} + mode: batch_rebuild_from_data_at_rest + workload: {input_range: 5m, samples_per_group: 300, groups: 100} + execution_profile: illustrative-local-worker + provenance: {kind: illustrative, usable_for_selection: false} + costs: {state_bytes_per_group: 4096, rebuild_cpu_ms_total: 35} installation_context: catalog_version: 12 @@ -132,7 +279,8 @@ query_plan: query_expression: >- SELECT service, quantile(0.99)(request_latency_seconds) FROM metrics - WHERE timestamp > now() - INTERVAL 5 MINUTE + WHERE timestamp > :evaluation_time - INTERVAL 5 MINUTE + AND timestamp <= :evaluation_time GROUP BY service nodes: - {id: read-kll, op: ReadState, @@ -153,6 +301,16 @@ QueryPlan reads it, and SDS defines its meaning and schema. Provenance relates both physical projections to the selected DAG without making that DAG executable inside PrecomputePlan. +Here `range: 5m` denotes logical coverage `(T - 5m, T]`, not pane size, +refresh cadence, state retention or scrape interval. `ReadInput` is parameterized +by the scheduled endpoint and that range; `WriteState` publishes a completed +snapshot per service and endpoint. `ReadState` selects the snapshot matching the +requested endpoint and checks readiness. The ten-minute retention keeps older +completed snapshots available; it does not turn the summary into a ten-minute +aggregate. The illustrative KLL parameters alone do not establish a particular +accuracy guarantee, and CPU cost alone does not establish the 200 ms latency +requirement. + ## Core concepts and ownership “Maintenance” is the execution phase that constructs or updates state, including From b358074a091f43ce247c825172c901f93ec3be83 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 18:10:14 +0000 Subject: [PATCH 13/15] docs: use plan version consistently in backend design --- docs/design_docs/asapplanner-integration.md | 32 ++++++++-------- .../design_docs/asapplanner-migration-plan.md | 12 +++--- .../summary-catalog-sds-architecture.md | 38 +++++++++++-------- 3 files changed, 44 insertions(+), 38 deletions(-) diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 7931dc256..447a60054 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -12,7 +12,7 @@ backend plans: - **QueryPlan** reads stored state and computes query results. Both plans use identities and state contracts from the -[SDS design](summary-catalog-sds-architecture.md) and install as one generation. +[SDS design](summary-catalog-sds-architecture.md) and install as one plan version. The [migration plan](asapplanner-migration-plan.md) defines delivery steps. CollectorPlan, TransmissionPlan and distributed activation are deferred; this migration must not introduce a backend dependency on ASAPCollector. @@ -119,7 +119,7 @@ Planner supplies legal maintenance alternatives. The backend supplies executable implementations and evidence; the control plane commits a feasible selection. The compiler validates that commitment without silently changing its mode, coverage or sharing. A changed commitment is installed through a new plan -generation. It need not change the semantic summary definition when only the +plan version. It need not change the semantic summary definition when only the physical maintenance policy changes. ### Backend capability @@ -240,7 +240,7 @@ physical_cost_evidence: installation_context: catalog_version: 12 state_schema: kll-v1 - plan_generation: 42 + plan_version: 42 ``` ### Compiler output @@ -254,26 +254,26 @@ summary_catalog: range: 5m algorithm: {kind: kll, k: 200} materializations: - - id: mat-api-latency-kll-g42 + - id: mat-api-latency-kll-v42 definition: def-api-latency-kll schema: kll-v1 - generation: 42 + plan_version: 42 precompute_plan: - generation: 42 + plan_version: 42 nodes: - {id: read-samples, op: ReadInput, metric: request_latency_seconds} - {id: group-service, op: GroupBy, labels: [service]} - {id: build-kll, op: BuildKll, k: 200} - {id: write-kll, op: WriteState, - materialization: mat-api-latency-kll-g42} + materialization: mat-api-latency-kll-v42} edges: - [read-samples, group-service] - [group-service, build-kll] - [build-kll, write-kll] query_plan: - generation: 42 + plan_version: 42 query_id: p99-api-latency query_language: clickhouse_sql query_expression: >- @@ -284,7 +284,7 @@ query_plan: GROUP BY service nodes: - {id: read-kll, op: ReadState, - materialization: mat-api-latency-kll-g42, schema: kll-v1} + materialization: mat-api-latency-kll-v42, schema: kll-v1} - {id: estimate-p99, op: SummaryEstimate, quantile: 0.99} - {id: result, op: QueryResult} edges: @@ -296,7 +296,7 @@ provenance: planner.estimate-p99: [query.read-kll, query.estimate-p99] ``` -`mat-api-latency-kll-g42` is the join point: PrecomputePlan writes it, +`mat-api-latency-kll-v42` is the join point: PrecomputePlan writes it, QueryPlan reads it, and SDS defines its meaning and schema. Provenance relates both physical projections to the selected DAG without making that DAG executable inside PrecomputePlan. @@ -330,7 +330,7 @@ Bindings describe the semantic-to-physical mapping: | Layer | Owns | | --- | --- | | ASAPPlanner | Semantic candidates, legality, accuracy reasoning and selection among advertised capabilities | -| Physical compiler | Concrete implementation, subgraph split, catalog bindings and plan generation | +| Physical compiler | Concrete implementation, subgraph split, catalog bindings and plan version | | Precompute runtime | Installed maintenance nodes and state publication | | Query runtime | Bound state reads, query operators, exact residuals and fallback | | SDS/catalog | Definition, materialization, schema, state reference, readiness and lifecycle metadata | @@ -343,7 +343,7 @@ The compiler consumes: - query accuracy and response requirements; - complete lifecycle commitments for the supported backend mode; - backend capabilities and concrete implementation evidence; -- catalog, schema and deployment-generation inputs. +- catalog, schema and plan-version inputs. Capabilities constrain Planner choices. A data-at-rest-only backend advertises only batch construction; recurring query demand does not imply incremental @@ -386,8 +386,8 @@ QueryPlan: Read state B -> estimate -> result ## Runtime contract -The backend stages the catalog and both plans as one generation and exposes them -atomically. Failed staging leaves the previous generation active. +The backend stages the catalog and both plans as one plan version and exposes them +atomically. Failed staging leaves the previous plan version active. Installation and readiness are distinct. Until required state coverage exists, QueryPlan uses its configured exact fallback or returns explicit unavailability. @@ -401,7 +401,7 @@ view, but it must label maintenance-owned and query-owned nodes. ## Validation and acceptance Compilation and installation reject unresolved state references, schema/encoding -mismatches, incompatible grouping or time partitions, wrong generations, cycles, +mismatches, incompatible grouping or time partitions, wrong plan versions, cycles, unsupported phase operators and unsatisfied derived-state completeness. Acceptance tests demonstrate: @@ -411,7 +411,7 @@ Acceptance tests demonstrate: 2. One query can read multiple summaries and two queries can share one producer. 3. Derived summaries honor completion and schema requirements. 4. Invalid cross-plan bindings fail before activation. -5. Staging failure, restart and generation switching preserve consistency and +5. Staging failure, restart and plan version switching preserve consistency and documented fallback behavior. 6. The backend builds and runs these cases without ASAPCollector. diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index 491d04aeb..23109f0a1 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -30,7 +30,7 @@ and a general ASAPPlanner API redesign are deferred. | 1. Inventory | Freeze current contracts and behavior as fixtures | Every supported path has a fixture or explicit unsupported result | | 2. Extract | Move neutral contracts/codecs out of Collector | Backend dependencies and tests contain no ASAPCollector | | 3. Split | Derive catalog, maintenance DAGs and query DAGs from one binding | Ownership and state references match selected semantics | -| 4. Install | Validate and atomically activate one generation | Invalid snapshots fail without disturbing the active generation | +| 4. Install | Validate and atomically activate one plan version | Invalid snapshots fail without disturbing the active plan version | | 5. Retire | Normalize old artifacts and remove superseded paths | Compatibility and end-to-end gates pass | Do not combine payload-format changes with dependency extraction. Version the new @@ -48,7 +48,7 @@ The migration produces: ```yaml summary_catalog: - materialization: {id: mat-17, schema: kll-v1, generation: 42} + materialization: {id: mat-17, schema: kll-v1, plan_version: 42} precompute_plan: nodes: [Input, BuildKLL, 'WriteState(mat-17)'] @@ -121,12 +121,12 @@ unchanged schema version. ## Stage 4: validate and install Validate definition, materialization, schema, encoding, grouping, time partition, -coverage and generation across the catalog and both plans. Then perform local +coverage and plan version across the catalog and both plans. Then perform local resource checks. Stage and activate the three artifacts as one snapshot. Readiness remains separate: until coverage is ready, QueryPlan follows its configured fallback or -explicit unavailability. Failed staging preserves the previous generation. +explicit unavailability. Failed staging preserves the previous plan version. Render PrecomputePlan and QueryPlan separately, joined by state references. Legacy projected views label maintenance-owned and query-owned nodes. @@ -138,7 +138,7 @@ backend-local publications first and retain versioned adapters for the supported compatibility window. Remove complete-DAG precompute execution and Collector adapter code only after -fixtures and end-to-end tests pass. State reuse across generations requires an +fixtures and end-to-end tests pass. State reuse across plan versions requires an explicit SDS compatibility decision independently of binary rollback. ## Completion evidence @@ -150,7 +150,7 @@ Completion requires: - one query can read multiple summaries and two queries can share one producer; - derived state observes completion and schema requirements; - invalid bindings fail before activation; -- restart and generation switching preserve consistency and fallback; +- restart and plan version switching preserve consistency and fallback; - legacy and split artifacts produce equivalent results and update counts; - backend builds and required tests do not fetch, build or run ASAPCollector. diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index c057746b7..7526e46e2 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -45,6 +45,12 @@ decision,” and “same stored payload” have different compatibility rules. ## Worked example +`plan_version` identifies the coherent version of PrecomputePlan, QueryPlans +and their catalog bindings installed together. The value `42` below is an +illustrative version identifier. Updating summary contents or publishing a new +time partition does not change the plan version. State readiness is tracked +separately; installing a plan version does not make its required state ready. + Two queries request different percentiles from the same five-minute KLL summary: ```yaml @@ -56,24 +62,24 @@ summary_definition: algorithm: {kind: kll, k: 200} materialization: - id: mat-api-latency-kll-g42 + id: mat-api-latency-kll-v42 definition: def-api-latency-kll - generation: 42 + plan_version: 42 schema: kll-v1 state_instances: - id: state-api-1200 - materialization: mat-api-latency-kll-g42 + materialization: mat-api-latency-kll-v42 partition: {service: api, start: '12:00', end: '12:01'} status: ready - id: state-api-1201 - materialization: mat-api-latency-kll-g42 + materialization: mat-api-latency-kll-v42 partition: {service: api, start: '12:01', end: '12:02'} status: ready query_state_references: - q50: {materialization: mat-api-latency-kll-g42, quantile: 0.50} - q99: {materialization: mat-api-latency-kll-g42, quantile: 0.99} + q50: {materialization: mat-api-latency-kll-v42, quantile: 0.50} + q99: {materialization: mat-api-latency-kll-v42, quantile: 0.99} ``` PrecomputePlan updates each state partition once. Both QueryPlans resolve the @@ -86,7 +92,7 @@ time. | Object | Meaning | Changes when | | --- | --- | --- | | `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change | -| `Materialization` | An installed decision to produce a definition with one state contract | Plan generation or physical contract changes | +| `Materialization` | An installed decision to produce a definition with one state contract | Plan version or physical contract changes | | `SummaryStateInstance` | One stored partition, such as a series/pane or completed aggregate | Runtime creates or replaces payload state | | `StateReference` | A typed plan reference to permitted materialized state | A compiled reader/writer binding changes | @@ -95,9 +101,9 @@ and filters, input value, operation or sketch parameters, grouping, time semantics, accuracy fields that affect state, and output type. Display names, costs, locations, readiness and retention status are excluded. -A materialization adds definition ID, plan generation, state family, schema, +A materialization adds definition ID, plan version, state family, schema, encoding, physical partition layout, permitted writer identity and provenance. -Several generations may materialize the same definition. +Several plan versions may materialize the same definition. A state instance adds its partition key, coverage/completion, producer sequence where applicable, lifecycle status, location and integrity metadata. Payload @@ -110,15 +116,15 @@ bytes remain in the summary store, not in catalog descriptors. | Definition ID | What semantics does the state represent? | | Materialization ID | Which installed physical decision produced it? | | State-instance ID | Which concrete partition/payload is it? | -| Plan generation | With which atomic installation may it be used? | +| Plan version | With which atomic installation may it be used? | | Schema/encoding ID | How are its bytes interpreted? | The compiler/catalog authority assigns these identities once. Human-readable -names are diagnostics, not join keys. Reuse across generations requires an +names are diagnostics, not join keys. Reuse across plan versions requires an explicit compatibility decision; a matching definition ID is insufficient. A `StateReference` identifies one materialization and constrains acceptable -partition, schema, generation and coverage. It may select several instances, such +partition, schema, plan version and coverage. It may select several instances, such as panes covering one range, but cannot broaden semantics or substitute another algorithm. QueryPlan and derived PrecomputePlan nodes resolve references through exact indexed lookup, never serving-time candidate selection. @@ -130,14 +136,14 @@ PrecomputePlan Input -> BuildKLL -> Write(mat-17) SDS - mat-17 -> def-9, KLL(k=200), kll-v1, generation 42 + mat-17 -> def-9, KLL(k=200), kll-v1, plan version 42 QueryPlan Read(mat-17, kll-v1) -> SummaryEstimate -> Result ``` Writer, SDS entry and reader must agree on definition, materialization, state -family, parameters, schema/encoding, grouping, time partition and generation. +family, parameters, schema/encoding, grouping, time partition and plan version. The query runtime follows the installed reference instead of scanning the catalog. A derived materialization has a distinct destination identity and an explicit @@ -165,7 +171,7 @@ observed readiness and coverage, then follows its configured fallback or explici unavailability behavior. Reactivation does not make stale instances current. Completed finite-input state is immutable. Additional writes require a new -authorized generation or replacement instance. Mutable streaming state publishes +authorized plan version or replacement instance. Mutable streaming state publishes monotone coverage according to its installed contract. ## Validation and migration @@ -175,7 +181,7 @@ Compilation, installation, writes, recovery and reads enforce: 1. Each materialization resolves to one definition and each instance to one materialization. 2. Instance metadata declares the payload's actual schema and encoding. -3. References preserve definition semantics and compatible generation. +3. References preserve definition semantics and compatible plan version. 4. Writer and reader grouping, time partition, schema and coverage agree. 5. Derived reads meet their completion requirement. 6. Retirement blocks new bindings before state reclamation. From 7e4655ae08fbbf17982f6fa92de556349d17d50a Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 18:33:04 +0000 Subject: [PATCH 14/15] docs: remove standalone catalog materialization abstraction --- docs/design_docs/README.md | 2 +- docs/design_docs/asapplanner-integration.md | 59 +++++--- .../design_docs/asapplanner-migration-plan.md | 22 ++- .../summary-catalog-sds-architecture.md | 130 ++++++++++++------ 4 files changed, 145 insertions(+), 68 deletions(-) diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index ad43475e4..14d5e25d0 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -8,7 +8,7 @@ notes and migration gates distinguish implemented behavior from proposed changes how one selected semantic DAG becomes executable PrecomputePlan and QueryPlan subgraphs joined at materialization boundaries. - [Summary Catalog and SDS](summary-catalog-sds-architecture.md) owns descriptors, - definition/materialization/instance identity, state references, readiness and + definition/instance identity, version-scoped state references, readiness and lifecycle semantics. - [Architecture migration delivery plan](asapplanner-migration-plan.md) defines common-library extraction, removal of ASAPCollector dependencies, the two-plan diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 447a60054..6c6ccf9f9 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -49,8 +49,10 @@ flowchart LR C --> P[PrecomputePlan] C --> S[Summary Catalog / SDS] C --> Q[QueryPlan] - P -->|write state| S - S -->|bound state reference| Q + P -->|write state| Store[Summary store] + Q -->|bound state read| Store + P -->|definition ID| S + Q -->|definition ID| S ``` Semantic provenance remains available, but query-only operators are not @@ -253,11 +255,6 @@ summary_catalog: group_by: [service] range: 5m algorithm: {kind: kll, k: 200} - materializations: - - id: mat-api-latency-kll-v42 - definition: def-api-latency-kll - schema: kll-v1 - plan_version: 42 precompute_plan: plan_version: 42 @@ -265,8 +262,12 @@ precompute_plan: - {id: read-samples, op: ReadInput, metric: request_latency_seconds} - {id: group-service, op: GroupBy, labels: [service]} - {id: build-kll, op: BuildKll, k: 200} - - {id: write-kll, op: WriteState, - materialization: mat-api-latency-kll-v42} + - id: write-kll + op: WriteState + reference: {state_slot_id: latency-kll, definition_id: def-api-latency-kll} + schema: kll-v1 + encoding: kll-binary-v1 + partition_by: [service, window_end] edges: - [read-samples, group-service] - [group-service, build-kll] @@ -283,8 +284,12 @@ query_plan: AND timestamp <= :evaluation_time GROUP BY service nodes: - - {id: read-kll, op: ReadState, - materialization: mat-api-latency-kll-v42, schema: kll-v1} + - id: read-kll + op: ReadState + reference: {state_slot_id: latency-kll, definition_id: def-api-latency-kll} + expected_schema: kll-v1 + expected_encoding: kll-binary-v1 + partition: {service: all_requested_services, window_end: evaluation_time} - {id: estimate-p99, op: SummaryEstimate, quantile: 0.99} - {id: result, op: QueryResult} edges: @@ -296,8 +301,10 @@ provenance: planner.estimate-p99: [query.read-kll, query.estimate-p99] ``` -`mat-api-latency-kll-v42` is the join point: PrecomputePlan writes it, -QueryPlan reads it, and SDS defines its meaning and schema. Provenance relates +`latency-kll` is the state slot shared by the writer and reader in plan version +42. The catalog defines its summary semantics; the matching executable bindings +declare format and partition rules. There is no separate catalog materialization +object. Provenance relates both physical projections to the selected DAG without making that DAG executable inside PrecomputePlan. @@ -327,13 +334,20 @@ Bindings describe the semantic-to-physical mapping: | `Query` | The node maps to an explicit QueryPlan operation | `SummaryEstimate` | | `QueryInput` | Query semantics are absorbed into another physical operation | A quantile parameter compiled into `SummaryEstimate` | +`Materialization` above is the existing backend node-binding variant marking +stored output. It does not create a separate catalog object. The compiler assigns +that output a state slot and emits matching writer/reader bindings; see +[field ownership and migration](summary-catalog-sds-architecture.md#core-objects). + | Layer | Owns | | --- | --- | | ASAPPlanner | Semantic candidates, legality, accuracy reasoning and selection among advertised capabilities | | Physical compiler | Concrete implementation, subgraph split, catalog bindings and plan version | | Precompute runtime | Installed maintenance nodes and state publication | | Query runtime | Bound state reads, query operators, exact residuals and fallback | -| SDS/catalog | Definition, materialization, schema, state reference, readiness and lifecycle metadata | +| Catalog | Summary definitions | +| Plan read/write bindings | State references, format, partition rules and writer ownership | +| Runtime inventory/store | Actual state instances, coverage, readiness, location and payloads | ## Compiler contract @@ -351,7 +365,7 @@ support. | Output | Responsibility | | --- | --- | -| Catalog/SDS entries | Summary semantics, materialization identity, schema and state references | +| Catalog entries | Summary definitions referenced by the plans | | PrecomputePlan | Maintenance subgraphs ending in state writes | | QueryPlan | Bound state reads, query operators and exact residuals | | Provenance | Physical-to-semantic node mapping | @@ -365,10 +379,13 @@ summary semantics, grouping, time ranges or schemas independently. For every selected stored summary, the compiler: -1. Creates or reuses one compatible summary definition and materialization. +1. Creates or reuses a compatible summary definition and assigns a state slot + within the plan version. No standalone catalog materialization is created. 2. Places source reads, maintenance operators, derived-state reads and the state sink in PrecomputePlan. -3. Replaces the stored-summary edge in QueryPlan with an explicit state read. +3. Replaces the stored-summary edge in QueryPlan with an explicit state read + referencing the same slot and definition, with matching format and partition + rules. Writer identity belongs to the PrecomputePlan binding. 4. Places `SummaryEstimate`, merges, exact residuals and result composition in QueryPlan. 5. Records provenance for semantic nodes absorbed into larger physical nodes. @@ -377,7 +394,11 @@ Two queries may share a producer only when their definition and state partition are compatible. Sharing does not multiply maintenance updates; each query keeps its own readout operators. -A derived materialization reads completed state explicitly: +A summary built from completed stored summaries uses explicit source reads and +a separate destination slot. For example, five compatible one-minute KLL states +can be merged into a stored five-minute KLL if coverage and accuracy permit it. +A merge used only to answer a query belongs in QueryPlan and creates no stored +destination: ```text PrecomputePlan: Read state A -> derive state B -> store B @@ -395,7 +416,7 @@ The query runtime follows installed state references; it does not search the catalog for alternative summaries. Visualization renders PrecomputePlan and QueryPlan separately, connected by -labeled materialization references. Legacy full-DAG artifacts may use a projected +labeled state references. Legacy full-DAG artifacts may use a projected view, but it must label maintenance-owned and query-owned nodes. ## Validation and acceptance diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index 23109f0a1..681b44991 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -47,14 +47,17 @@ Input -> BuildKLL -> SummaryEstimate -> Result The migration produces: ```yaml +plan_version: 42 summary_catalog: - materialization: {id: mat-17, schema: kll-v1, plan_version: 42} + definition: {id: def-9, algorithm: kll, k: 200} precompute_plan: - nodes: [Input, BuildKLL, 'WriteState(mat-17)'] + nodes: [Input, BuildKLL, 'WriteState(slot-17)'] + write_binding: {state_slot_id: slot-17, definition_id: def-9, schema: kll-v1} query_plan: - nodes: ['ReadState(mat-17)', SummaryEstimate, Result] + nodes: ['ReadState(slot-17)', SummaryEstimate, Result] + read_binding: {state_slot_id: slot-17, definition_id: def-9, expected_schema: kll-v1} provenance: selected_dag: Input -> BuildKLL -> SummaryEstimate -> Result @@ -105,7 +108,7 @@ transitive Collector dependencies. ## Stage 3: bind and split plans Create compiler bindings for semantic nodes, summary definitions, -materializations, schemas and state references. Derive the catalog and both plans +version-scoped state slots, schemas and state references. Derive the catalog and both plans from those bindings using the [materialization-boundary rules](asapplanner-integration.md#executable-subgraphs-and-materialization-boundaries): @@ -118,9 +121,16 @@ from those bindings using the Version the split representation. Do not reinterpret an old field under an unchanged schema version. +Do not introduce a standalone catalog `Materialization` object. Keep definitions +in the catalog, format/partition/writer configuration in executable bindings, +and actual coverage/location/readiness in instance inventory. Normalize legacy +stored-output identities into state slots while preserving payload locators; +validate all consumers against the same writer configuration. The existing +`BackendNodeBinding::Materialization` remains a placement marker for stored output. + ## Stage 4: validate and install -Validate definition, materialization, schema, encoding, grouping, time partition, +Validate definition, state slot, schema, encoding, grouping, time partition, coverage and plan version across the catalog and both plans. Then perform local resource checks. @@ -155,5 +165,5 @@ Completion requires: - backend builds and required tests do not fetch, build or run ASAPCollector. Record tested revisions, supported state families, fixture results and dependency -checks. Trace one query from its selected semantic root through the materialization +checks. Trace one query from its selected semantic root through the state writer, SDS reference and QueryPlan reader. diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index 7526e46e2..0edd9c550 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -27,21 +27,23 @@ Cost ranking, operator scheduling and transmission policy are outside SDS. ## Architecture at a glance -The catalog stores definitions and materializations. Runtime inventory records -state instances. Plans carry typed state references rather than payloads or -search predicates. +The catalog stores summary definitions. PrecomputePlan and QueryPlan carry +matching state references and format/partition configuration. Runtime inventory +records actual state instances; payload bytes live in the summary store. +There is no separate catalog `Materialization` object. ```mermaid flowchart LR - D[SummaryDefinition] --> M[Materialization] - M --> I[State instances] - P[PrecomputePlan] -->|write| M - Q[QueryPlan] --> R[StateReference] - R --> M + P[PrecomputePlan] -->|write through StateReference| S[Summary store] + Q[QueryPlan] -->|read through StateReference| S + P -->|definition ID| D[SummaryDefinition catalog] + Q -->|definition ID| D + I[Runtime instance inventory] -->|location and readiness| S ``` -These objects remain distinct because “same summary semantics,” “same production -decision,” and “same stored payload” have different compatibility rules. +The compiler assigns a `state_slot_id` to a stored producer output within a plan +version. This is a join key in compiled bindings, not another catalog entity with +its own lifecycle. Multiple query readers can reference the same slot. ## Worked example @@ -54,6 +56,7 @@ separately; installing a plan version does not make its required state ready. Two queries request different percentiles from the same five-minute KLL summary: ```yaml +plan_version: 42 summary_definition: id: def-api-latency-kll input: request_latency_seconds @@ -61,29 +64,41 @@ summary_definition: range: 5m algorithm: {kind: kll, k: 200} -materialization: - id: mat-api-latency-kll-v42 - definition: def-api-latency-kll - plan_version: 42 - schema: kll-v1 +precompute_plan: + write_state: + node_id: write-kll + reference: {state_slot_id: latency-kll, definition_id: def-api-latency-kll} + schema: kll-v1 + encoding: kll-binary-v1 + partition_by: [service, window_end] state_instances: - - id: state-api-1200 - materialization: mat-api-latency-kll-v42 - partition: {service: api, start: '12:00', end: '12:01'} - status: ready - - id: state-api-1201 - materialization: mat-api-latency-kll-v42 - partition: {service: api, start: '12:01', end: '12:02'} + - id: state-api-1205 + plan_version: 42 + state_slot_id: latency-kll + definition_id: def-api-latency-kll + schema: kll-v1 + encoding: kll-binary-v1 + partition: {service: api, window_end: '12:05'} + coverage: {start_exclusive: '12:00', end_inclusive: '12:05'} + location: opaque-store-locator status: ready -query_state_references: - q50: {materialization: mat-api-latency-kll-v42, quantile: 0.50} - q99: {materialization: mat-api-latency-kll-v42, quantile: 0.99} +query_plans: + q50: + read_state: &shared_read + reference: {state_slot_id: latency-kll, definition_id: def-api-latency-kll} + expected_schema: kll-v1 + expected_encoding: kll-binary-v1 + partition: {service: api, window_end: evaluation_time} + estimate: {quantile: 0.50} + q99: + read_state: *shared_read + estimate: {quantile: 0.99} ``` PrecomputePlan updates each state partition once. Both QueryPlans resolve the -same bound materialization and apply different readout parameters. They neither +same bound slot and apply different readout parameters. They neither create duplicate producers nor search the catalog for alternatives at serving time. @@ -92,7 +107,6 @@ time. | Object | Meaning | Changes when | | --- | --- | --- | | `SummaryDefinition` | Canonical input, operation, grouping, time semantics, algorithm and parameters | Summary semantics change | -| `Materialization` | An installed decision to produce a definition with one state contract | Plan version or physical contract changes | | `SummaryStateInstance` | One stored partition, such as a series/pane or completed aggregate | Runtime creates or replaces payload state | | `StateReference` | A typed plan reference to permitted materialized state | A compiled reader/writer binding changes | @@ -101,11 +115,29 @@ and filters, input value, operation or sketch parameters, grouping, time semantics, accuracy fields that affect state, and output type. Display names, costs, locations, readiness and retention status are excluded. -A materialization adds definition ID, plan version, state family, schema, -encoding, physical partition layout, permitted writer identity and provenance. -Several plan versions may materialize the same definition. +The former standalone `Materialization` catalog object was an over-abstraction: +its fields already belong to the definition, executable bindings or runtime +instance metadata. Their ownership is explicit below. -A state instance adds its partition key, coverage/completion, producer sequence +| Former field | Owner in this design | +| --- | --- | +| Materialization ID | Replaced by a compiler-assigned `state_slot_id`, scoped to the plan version, in reader/writer references. | +| Definition ID | `StateReference` points to the catalog's `SummaryDefinition`. | +| Plan version | Installed plan bundle; persisted instance metadata repeats it for recovery validation. | +| State family and algorithm parameters | `SummaryDefinition`. | +| Schema and encoding | Writer configuration and matching reader expectations; instances declare the actual payload format. | +| Physical partition layout | Writer partitioning and matching reader partition selection. | +| Permitted writer | PrecomputePlan write binding; runtime validates writes against the installed binding. | +| Provenance | Compiler's physical-to-semantic node mapping. | + +The compiler emits both bindings from one decision and validates agreement +before installation. Repetition of format fields in the serialized plans does +not authorize independent selection. The catalog does not need a second registry +for those fields. Retention and refresh policy belong to the producer's selected +lifecycle and PrecomputePlan; observed readiness belongs to runtime inventory. + +A state instance records plan version, slot, definition, actual format and its +partition key, coverage/completion, producer sequence where applicable, lifecycle status, location and integrity metadata. Payload bytes remain in the summary store, not in catalog descriptors. @@ -114,7 +146,7 @@ bytes remain in the summary store, not in catalog descriptors. | Identity | Answers | | --- | --- | | Definition ID | What semantics does the state represent? | -| Materialization ID | Which installed physical decision produced it? | +| Plan version + state slot ID | Which installed producer output does this state belong to? | | State-instance ID | Which concrete partition/payload is it? | | Plan version | With which atomic installation may it be used? | | Schema/encoding ID | How are its bytes interpreted? | @@ -123,7 +155,8 @@ The compiler/catalog authority assigns these identities once. Human-readable names are diagnostics, not join keys. Reuse across plan versions requires an explicit compatibility decision; a matching definition ID is insufficient. -A `StateReference` identifies one materialization and constrains acceptable +A `StateReference` identifies a state slot and definition within the enclosing +plan version. The reader/writer binding constrains acceptable partition, schema, plan version and coverage. It may select several instances, such as panes covering one range, but cannot broaden semantics or substitute another algorithm. QueryPlan and derived PrecomputePlan nodes resolve references through @@ -133,20 +166,23 @@ exact indexed lookup, never serving-time candidate selection. ```text PrecomputePlan - Input -> BuildKLL -> Write(mat-17) + Input -> BuildKLL -> Write(slot-17, kll-v1) SDS - mat-17 -> def-9, KLL(k=200), kll-v1, plan version 42 + Catalog: def-9 -> KLL(k=200) and input semantics + Plan bundle: version 42; writer/reader bind slot-17 to def-9 + Store: instances indexed by plan version, slot and partition QueryPlan - Read(mat-17, kll-v1) -> SummaryEstimate -> Result + Read(slot-17, kll-v1) -> SummaryEstimate -> Result ``` -Writer, SDS entry and reader must agree on definition, materialization, state -family, parameters, schema/encoding, grouping, time partition and plan version. +Writer, instance metadata and reader must agree on slot, definition ID, +schema/encoding, grouping, time partition and plan version. State family and +parameters must match the referenced catalog definition. The query runtime follows the installed reference instead of scanning the catalog. -A derived materialization has a distinct destination identity and an explicit +A stored summary derived from existing state has a distinct destination slot and an explicit reference to completed source state: ```text @@ -160,7 +196,7 @@ Source and destination are never represented as the same instance. | State | Meaning | | --- | --- | -| `Desired` | Installed plans require the materialization | +| `Desired` | Installed plans require state for this slot and coverage | | `Building` | Required state is being produced or recovered | | `Ready` | Required schema and coverage are available | | `Draining` | New work has stopped while existing use completes | @@ -178,8 +214,8 @@ monotone coverage according to its installed contract. Compilation, installation, writes, recovery and reads enforce: -1. Each materialization resolves to one definition and each instance to one - materialization. +1. Each slot resolves to one definition and authorized producer binding within + its plan version; each instance identifies that version and slot. 2. Instance metadata declares the payload's actual schema and encoding. 3. References preserve definition semantics and compatible plan version. 4. Writer and reader grouping, time partition, schema and coverage agree. @@ -193,6 +229,16 @@ IDs and metadata rather than creating a parallel registry. Legacy artifacts are normalized at the backend boundary and supported payloads retain versioned readers and fixtures. +Remove the proposed `materializations` catalog collection and standalone object +from new plan examples and schemas. Preserve the existing +`BackendNodeBinding::Materialization` variant as the node-placement marker for +stored output; it does not imply a catalog object. At the compatibility boundary, +map legacy stored-output identifiers into version-scoped slots and copy their +format/partition constraints into matching bindings. Preserve payload locators +and reject unresolved or conflicting mappings; do not rename existing persisted +IDs or reinterpret legacy wire fields in place. Legacy formats keep their +versioned readers during the supported migration window. + Runtime-independent contracts and sketch reconstruction belong in neutral libraries. Backend storage, scheduling and query execution remain backend-owned; the backend must not depend on ASAPCollector. From 07580f8eb7fbd1901040500079e7a05057158f74 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 18 Sep 2026 20:11:51 +0000 Subject: [PATCH 15/15] docs: add concise planner backend glossary --- docs/design_docs/README.md | 2 + docs/design_docs/asapplanner-integration.md | 2 + .../design_docs/asapplanner-migration-plan.md | 2 + docs/design_docs/planner-backend-glossary.md | 61 +++++++++++++++++++ .../summary-catalog-sds-architecture.md | 2 + 5 files changed, 69 insertions(+) create mode 100644 docs/design_docs/planner-backend-glossary.md diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 14d5e25d0..01336929d 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -4,6 +4,8 @@ These documents are for architects and developers. The integration proposal and SDS model below define the target Planner-to-runtime boundary; their current-code notes and migration gates distinguish implemented behavior from proposed changes. +- [Planner/backend glossary](planner-backend-glossary.md) defines the terms used + by the following three designs. - [Planner output to backend physical plans](asapplanner-integration.md) defines how one selected semantic DAG becomes executable PrecomputePlan and QueryPlan subgraphs joined at materialization boundaries. diff --git a/docs/design_docs/asapplanner-integration.md b/docs/design_docs/asapplanner-integration.md index 6c6ccf9f9..72b144ed3 100644 --- a/docs/design_docs/asapplanner-integration.md +++ b/docs/design_docs/asapplanner-integration.md @@ -3,6 +3,8 @@ Status: proposed backend architecture. Audience: developers changing the Planner-to-backend compilation and execution boundary. +Terminology: [Planner/backend glossary](planner-backend-glossary.md). + ## Purpose and scope This design splits one selected ASAPPlanner semantic DAG into two executable diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index 681b44991..34557a727 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -2,6 +2,8 @@ Status: proposed delivery sequence. Audience: backend implementers. +Terminology: [Planner/backend glossary](planner-backend-glossary.md). + ## Goal and scope Replace complete semantic DAGs stored under PrecomputePlan with separate diff --git a/docs/design_docs/planner-backend-glossary.md b/docs/design_docs/planner-backend-glossary.md new file mode 100644 index 000000000..158dec5cb --- /dev/null +++ b/docs/design_docs/planner-backend-glossary.md @@ -0,0 +1,61 @@ +# Planner/backend design glossary + +For developers reading the [integration](asapplanner-integration.md), +[SDS](summary-catalog-sds-architecture.md), and +[migration](asapplanner-migration-plan.md) designs. Definitions describe the +proposed boundary; they do not imply that every proposed field already exists +in the serialized API. + +## Computation and execution + +| Term | Meaning | +| --- | --- | +| Selected post-ASAP DAG | Planner-selected computation graph, including summary producers, shared dependencies and query readouts. Called the “semantic DAG” in earlier discussion. | +| Summary producer | An operation or subgraph that builds summary state. Multiple queries may share its stored output. | +| `SummaryMaintenanceLifecyclePlan` | Planner result associating a post-ASAP root with deployment decisions for its unique reachable summary producers, plus workload and costing context. | +| Lifecycle commitment | Selected maintenance promise for one producer, with its scheduling and retention binding. A deployment's `SummaryMaintenanceLifecycleGuarantee` carries the Planner-level commitment. | +| Maintenance | Work that constructs, refreshes or derives stored summary state, including batch rebuilds and incremental updates. | +| `PrecomputePlan` | Backend executable plan for maintenance and state writes. | +| `QueryPlan` | Backend executable plan for state reads, query readouts and remaining query operations. | +| Readout / `SummaryEstimate` | Operation that obtains a query value from summary state, such as p99 from KLL. | +| Derived summary state | Stored summary state computed from existing summary states. Earlier discussion calls this a “derived materialization”; it does not require a separate catalog object. | +| Exact residual | Part of the selected query computed exactly around summary operations, such as supported filtering or arithmetic after readout. It does not make the whole approximate result exact. | +| Exact fallback | Configured execution of the original query through an exact route when the summary plan cannot serve it. | + +For example, merging five compatible one-minute KLL summaries and storing the +five-minute result produces derived summary state in a separate destination +slot. Merging them only to answer a query is a query-time operation. Both require +compatible grouping, coverage and accuracy. + +## State and identity + +| Term | Meaning | +| --- | --- | +| Summary Catalog | Metadata registry of summary definitions; payload bytes live in the summary store. | +| SDS (Self-Describing Summary) | The description and metadata needed to interpret and validate stored summary state. It is not a separate execution engine or payload store. | +| `SummaryDefinition` | What a summary represents: source/filter, input value, grouping, time semantics, algorithm and parameters. | +| `state_slot_id` | Compiler-assigned identifier for a stored producer output within one plan version. Shared readers use the same slot; it has no independent catalog object. | +| `StateReference` | Plan reference identifying a slot and summary definition within the enclosing plan version. Reader configuration selects the required state instances and constrains format and coverage. | +| `SummaryStateInstance` | A concrete stored state, such as one service's completed five-minute KLL snapshot, with partition, coverage, format and location metadata. | +| Summary store | Storage for actual summary payloads. Runtime inventory records their existence, coverage and readiness. | +| `plan_version` | Version shared by an installed plan bundle and its catalog bindings. Creating or updating state instances does not itself change this version. Previously called “generation” in this proposal. | +| Schema / encoding | Schema describes the state structure; encoding describes how that structure is represented as bytes. | +| Provenance | Mapping from physical plan operations back to the selected Planner computation. | + +The existing `BackendNodeBinding::Materialization` marks a node whose output is +stored. It remains a node binding; this design has no standalone catalog +`Materialization` object. A materialization boundary is simply where a producer +writes stored state and a consumer reads it. + +## Time, selection and validation + +| Term | Meaning | +| --- | --- | +| Logical range | Input interval required by the computation. In the example, `range: 5m` means `(T - 5m, T]` at evaluation time `T`. | +| Pane | Physical time partition of stored state. Several compatible panes may serve one logical range; pane size need not equal that range. | +| Refresh cadence | How often the producer is scheduled to build or refresh state. | +| Retention | How long state remains available; distinct from its input range and refresh cadence. | +| Readiness | Whether the required state is available with valid format and sufficient coverage/completeness for a read. Plan installation alone does not establish readiness. | +| Backend capability | Declaration of supported implementation combinations: algorithm/parameters, maintenance mode, input kind, window behavior and format. | +| Physical cost evidence | Scoped measurements or estimates used to compare executable alternatives; includes workload and implementation context. | +| Compiler contract | Required inputs, outputs, validation rules and guarantees, including matching writer/reader definitions, formats, partitions and plan versions. | diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index 0edd9c550..ccd18e7b5 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -3,6 +3,8 @@ Status: proposed contract with current-backend migration notes. Audience: developers compiling, storing, recovering or reading summary state. +Terminology: [Planner/backend glossary](planner-backend-glossary.md). + ## Purpose and scope The Summary Catalog and Self-Describing Summary (SDS) model defines what persisted