diff --git a/control_plane/docs/README.md b/control_plane/docs/README.md index 0c46724c..6b33fc8d 100644 --- a/control_plane/docs/README.md +++ b/control_plane/docs/README.md @@ -22,6 +22,11 @@ designs; they are intentionally not repeated here. | [Physical planning](physical-planning.md) | Placement, windows, representation, transmission, and compilation into matching collector and backend plans. | | [BackendPlan](backend-plan.md) | Versioned contract installed and executed by the ASAPQuery data plane. | +## Developer documentation + +- [Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md) +- [Runtime plan publication](developer_docs/runtime-plan-publication.md) + The corresponding collector-facing plan is documented by [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). diff --git a/control_plane/docs/asapplanner-integration.md b/control_plane/docs/asapplanner-integration.md index 95a658d4..087760ad 100644 --- a/control_plane/docs/asapplanner-integration.md +++ b/control_plane/docs/asapplanner-integration.md @@ -4,6 +4,8 @@ > > MVP relation: required for query planning and control-plane decisions. +Developer guide: [Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md). + ## TL;DR ASAPPlanner chooses a logical plan for a query workload. ASAPQuery-backend diff --git a/control_plane/docs/backend-plan.md b/control_plane/docs/backend-plan.md index 303c90b7..a7b36b61 100644 --- a/control_plane/docs/backend-plan.md +++ b/control_plane/docs/backend-plan.md @@ -9,6 +9,10 @@ > plane tells its data plane what summary materializations exist, how they > are ingested, and which query capabilities they serve. +Developer guides: +[runtime plan publication](developer_docs/runtime-plan-publication.md) and +[BackendPlan installation](../../data_plane/docs/developer_docs/backend-plan-runtime.md). + ## TL;DR Planning chooses once; serving reuses that exact decision. diff --git a/control_plane/docs/developer_docs/planner-and-physical-compiler.md b/control_plane/docs/developer_docs/planner-and-physical-compiler.md new file mode 100644 index 00000000..cad96f75 --- /dev/null +++ b/control_plane/docs/developer_docs/planner-and-physical-compiler.md @@ -0,0 +1,190 @@ +# Developing the Planner adapter and physical compiler + +> Interface status: target public API. Existing migration modules must converge +> on this boundary. + +## 1. Code architecture + +The control plane has three public layers: + +```text +PlanningRequest + | + v +PlannerAdapter ----------> SelectedLogicalPlan + | + v +PhysicalCompiler -------> CompiledPlanBundle + | | + v v + CollectorPlan BackendPlan +``` + +- **Planner adapter** owns the typed call to ASAPPlanner. It supplies the whole + workload and receives one selected logical plan without copying Planner IR. +- **Physical compiler** adds backend-owned placement, windows, transport, and + runtime capabilities without changing logical semantics. +- **Plan bundle** is the only output passed to publication. CollectorPlan and + BackendPlan are created together and share identities. + +Logical query parsing, summary alternatives, guarantees, and candidate search +remain public ASAPPlanner interfaces. Runtime publication is documented in +[Runtime plan publication](runtime-plan-publication.md). + +## 2. Public interfaces and definitions + +### Planner adapter + +```rust +pub trait PlannerAdapter { + type Error; + + fn select( + &self, + request: PlanningRequest, + ) -> Result; +} +``` + +`PlanningRequest` is backend-owned request context around Planner's canonical +workload value: + +```rust +pub struct PlanningRequest { + pub workload: asap_planner::Workload, + pub schema: asap_planner::SchemaCatalog, + pub constraints: asap_planner::PlanningConstraints, + pub cost_inputs: asap_planner::CostInputs, + pub planner_revision: String, +} +``` + +Input definitions: + +| Field | Definition | +| --- | --- | +| `workload` | Complete workload; shared queries must not be split into independent calls. | +| `schema` | Source/label/type information required to bind queries. | +| `constraints` | Accuracy and logical requirements supplied by the caller. | +| `cost_inputs` | Measured/declared logical cost inputs; unknown values stay unknown. | +| `planner_revision` | Immutable Planner build/revision used for reproducibility. | + +`SelectedLogicalPlan` wraps Planner's public selected post-ASAP workload plan +and correlation metadata; it does not define another DAG: + +```rust +pub struct SelectedLogicalPlan { + pub workload_plan: asap_planner::SelectedWorkloadPlan, + pub planner_revision: String, + pub query_ids: Vec, +} +``` + +Why this interface exists: it prevents adapters, protocols, and physical +planning from each implementing their own query-to-summary mapping. + +### Physical compiler + +```rust +pub trait PhysicalCompiler { + type Error; + + fn compile( + &self, + selected: SelectedLogicalPlan, + environment: DeploymentEnvironment, + policy: RuntimePolicy, + ) -> Result; +} +``` + +```rust +pub struct DeploymentEnvironment { + pub topology: DeploymentTopology, + pub collectors: Vec, + pub backend: BackendTarget, + pub capability_snapshot_id: String, +} + +pub struct RuntimePolicy { + pub activation: Timestamp, + pub expiry: Option, + pub freshness: FreshnessPolicy, + pub retention: RetentionPolicy, + pub transmission: TransmissionPolicy, +} + +pub struct CompiledPlanBundle { + pub envelope: PlanEnvelope, + pub collector_plans: Vec, + pub backend_plan: BackendPlan, +} +``` + +Supporting public types: + +| Type | Definition | +| --- | --- | +| `DeploymentTopology` | Runtime stages, network relationships, and isolation boundaries available for placement. | +| `CollectorTarget` | Collector identity, edge assignment, endpoint reference, and advertised capability snapshot. | +| `BackendTarget` | Data-plane identity, endpoint reference, storage routes, and advertised capabilities. | +| `FreshnessPolicy` | Maximum readiness lag, watermark, and allowed-lateness requirements. | +| `RetentionPolicy` | Duration and lifecycle rules for active/draining materializations. | +| `TransmissionPolicy` | Allowed raw/full/delta modes, cadence, encoding, and checkpoint limits. | +| `PlanEnvelope` | Shared `plan_id`, `plan_version`, activation/expiry, backend compatibility, and Planner revision. | +| `CollectorPlan` | Versioned public YAML execution contract owned by ASAPCollector. | +| `BackendPlan` | Versioned public data-plane materialization/routing contract defined in this repository. | + +Output definitions: + +| Output | Definition | +| --- | --- | +| `envelope` | Shared plan/version/activation/compatibility identity. | +| `collector_plans` | One plan per targeted collector, following ASAPCollector's public CollectorPlan schema. | +| `backend_plan` | Matching data-plane materialization and routing contract. | + +The compiler error must identify an unsupported capability, invalid placement, +window incompatibility, identity conflict, or invalid selected guarantee. It +must not silently substitute another logical summary. + +Why this interface exists: one compile guarantees that producer and consumer +receive the same family, parameters, grouping, window, and materialization +identity. + +## 3. Adding and verifying functionality + +### Add a deployment topology + +1. Add a public `DeploymentTopology` variant and its required target fields. +2. Teach `PhysicalCompiler::compile` how selected operators can be placed on it. +3. Reject plans requiring an unavailable stage/capability. +4. Verify the output contains one complete CollectorPlan for every producer and + one BackendPlan referencing all produced materializations. + +Interpretation: a successful bundle means both runtime views are complete and +cross-consistent; it does not mean they have been activated. + +### Add a transmission mode + +1. Extend public `TransmissionPolicy` and capability declarations. +2. Define representation, sequencing, checkpoint, and fallback requirements. +3. Emit the same compatibility identity into producer and consumer plans. +4. Verify unsupported endpoint combinations return a compile error. + +For delta, verify duplicate, missing, reordered, and recovery-checkpoint cases. + +### Add a physical window policy + +1. Add the public policy variant with anchor, size, slide, and lateness. +2. Prove it covers the selected logical range without changing semantics. +3. Include it in materialization identity. +4. Verify generated collector/backend windows are identical and incompatible + query ranges fail compilation. + +### Required output checks + +- deterministic inputs produce deterministic plan/materialization identities; +- shared logical producers remain shared materializations; +- every referenced materialization has a producer and backend declaration; +- all query IDs remain traceable; and +- unsupported inputs return structured errors, never partial bundles. diff --git a/control_plane/docs/developer_docs/runtime-plan-publication.md b/control_plane/docs/developer_docs/runtime-plan-publication.md new file mode 100644 index 00000000..701d2966 --- /dev/null +++ b/control_plane/docs/developer_docs/runtime-plan-publication.md @@ -0,0 +1,180 @@ +# Developing runtime plan publication + +> Interface status: target public API. OpAMP transport exists today; semantic +> CollectorPlan application/reporting is still incomplete. + +## 1. Code architecture + +Publication begins only after physical compilation returns a complete bundle: + +```text +CompiledPlanBundle + | + v +PlanPublisher + | | + v v +CollectorClient BackendPlanClient + | | + v v +CollectorReport BackendPlanReport + \ / + v v + ActivationResult +``` + +`CollectorClient` is the ASAPQuery-side counterpart of ASAPCollector's +authoritative +[`opamp-config-push.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). +This repository does not redefine CollectorPlan fields. + +## 2. Public interfaces and definitions + +### Runtime clients + +```rust +pub trait CollectorPlanClient { + type Error; + + async fn stage( + &self, + target: CollectorTarget, + plan: CollectorPlan, + ) -> Result; +} + +pub trait BackendPlanClient { + type Error; + + async fn stage( + &self, + target: BackendTarget, + plan: BackendPlan, + ) -> Result; +} +``` + +Collector transport requirements come directly from the corresponding +ASAPCollector interface: + +- OpAMP `AgentRemoteConfig`/`AgentConfigMap`; +- exact entry name `asap-collector-plan.yaml`; +- YAML CollectorPlan with `content_type: application/yaml`; +- OpAMP `config_hash` identifies bytes, not cross-runtime plan semantics; and +- `RemoteConfigStatus.APPLIED` is delivery/application evidence, not semantic + activation evidence. + +### Application reports + +```rust +pub enum ApplicationStatus { + Rejected, + Staged, + Active, + Expired, + Failed, +} + +pub struct ApplicationError { + pub code: String, + pub path: String, + pub message: String, +} + +pub struct CollectorApplicationReport { + pub plan_id: String, + pub plan_version: u64, + pub backend_compat: String, + pub remote_config_hash: Vec, + pub status: ApplicationStatus, + pub active_materialization_ids: Vec, + pub effective_capability_hash: String, + pub observed_at: Timestamp, + pub activated_at: Option, + pub errors: Vec, +} + +pub struct BackendApplicationReport { + pub plan_id: String, + pub plan_version: u64, + pub backend_compat: String, + pub status: ApplicationStatus, + pub active_materialization_ids: Vec, + pub observed_at: Timestamp, + pub activated_at: Option, + pub errors: Vec, +} +``` + +The collector report corresponds to capability +`io.asap.collector.plan.v1`, message type `application_report`. Unknown report +versions or missing required fields are errors. + +Why reports are separate from transport acknowledgement: the MVP must prove the +runtime applied the intended semantic plan, not merely that bytes arrived. + +### Publisher + +```rust +pub trait PlanPublisher { + type Error; + + async fn publish( + &self, + bundle: CompiledPlanBundle, + ) -> Result; + + async fn rollback( + &self, + plan_id: &str, + plan_version: u64, + ) -> Result; +} + +pub struct ActivationResult { + pub plan_id: String, + pub plan_version: u64, + pub status: ApplicationStatus, + pub collector_reports: Vec, + pub backend_report: BackendApplicationReport, +} +``` + +`publish` returns `Active` only when every required runtime reports the same +plan/version/compatibility and expected materializations. Partial staging is an +error result and keeps the prior valid plan authoritative. + +## 3. Adding and verifying functionality + +### Add another collector transport + +1. Implement `CollectorPlanClient`; keep CollectorPlan unchanged. +2. Preserve plan identity separately from transport byte identity. +3. Map transport errors to structured client errors. +4. Verify identical re-delivery is idempotent and conflicting bytes for the same + plan/version are rejected. + +Interpretation: a successful `stage` report is not global activation; only +`PlanPublisher::publish` can return an active bundle. + +### Add an application status or report field + +1. Version the public report schema/capability. +2. Define required/optional behavior and compatibility. +3. Update collector and backend clients together. +4. Verify older readers reject unknown required semantics rather than defaulting. + +### Add rollback policy + +1. Select only a retained complete bundle through `rollback`. +2. Stage both runtime sides like a normal publication. +3. Verify the result reports the restored version and all materializations. +4. Verify failed rollback leaves the current active bundle unchanged. + +### Required output checks + +- reports match bundle identity and expected materialization sets; +- stale/expired/conflicting versions fail; +- collector-only or backend-only success never returns `Active`; +- report artifacts are machine-readable by the MVP harness; and +- post-activation emitted state carries the activated identities. diff --git a/control_plane/docs/physical-planning.md b/control_plane/docs/physical-planning.md index 33b5e398..d4a55b7d 100644 --- a/control_plane/docs/physical-planning.md +++ b/control_plane/docs/physical-planning.md @@ -9,6 +9,10 @@ > selected post-ASAP workload DAG and the two runtime executors: > ASAPCollector and the ASAPQuery data plane. +Developer guides: +[Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md) +and [runtime plan publication](developer_docs/runtime-plan-publication.md). + ## TL;DR ASAPPlanner selects a logical plan. That plan says which summaries and exact diff --git a/data_plane/docs/README.md b/data_plane/docs/README.md index 0572b571..2a5bc459 100644 --- a/data_plane/docs/README.md +++ b/data_plane/docs/README.md @@ -18,6 +18,14 @@ or re-plan queries. - [Extension boundaries](developer_docs/extension-points.md) — responsibilities of protocol servers, adapters, and fallback clients. +- [BackendPlan runtime](developer_docs/backend-plan-runtime.md) — validation, + staging, atomic installation, and snapshots. +- [OTLP summary ingestion](developer_docs/otlp-summary-ingestion.md) — decoding, + plan validation, SID resolution, and full/delta handling. +- [Query routing and readout](developer_docs/query-routing-and-readout.md) — + readiness, summary execution, and exact fallback. +- [Summary storage and series identity](developer_docs/summary-storage-and-series-identity.md) + — store boundaries, identity hierarchy, lifecycle, and concurrency. - [Adding a summary family](../../docs/developer_docs/adding-summary-family.md) — cross-repository prerequisites and backend validation. diff --git a/data_plane/docs/design_docs/query-execution.md b/data_plane/docs/design_docs/query-execution.md index 761cced4..484688c8 100644 --- a/data_plane/docs/design_docs/query-execution.md +++ b/data_plane/docs/design_docs/query-execution.md @@ -4,6 +4,11 @@ > > MVP relation: required for every summary-backed query and exact fallback. +Developer guides: +[BackendPlan runtime](../developer_docs/backend-plan-runtime.md), +[OTLP summary ingestion](../developer_docs/otlp-summary-ingestion.md), and +[query routing/readout](../developer_docs/query-routing-and-readout.md). + ## TL;DR The data plane accepts only state compatible with its active BackendPlan. At diff --git a/data_plane/docs/developer_docs/backend-plan-runtime.md b/data_plane/docs/developer_docs/backend-plan-runtime.md new file mode 100644 index 00000000..9ec7bc9b --- /dev/null +++ b/data_plane/docs/developer_docs/backend-plan-runtime.md @@ -0,0 +1,127 @@ +# Developing BackendPlan installation + +> Interface status: target public API. Atomic snapshot storage exists; complete +> staging/lifecycle/cross-runtime activation remains partial. + +## 1. Code architecture + +```text +BackendPlan bytes + | + v +BackendPlanDecoder -> BackendPlanValidator -> BackendPlanRuntime + | + BackendPlanSnapshot + / \ + ingest query +``` + +The decoder owns wire decoding, the validator owns semantic/capability checks, +and the runtime owns staged/active snapshots. Ingest and query components only +consume immutable snapshots; they do not mutate plans. + +## 2. Public interfaces and definitions + +```rust +pub trait BackendPlanDecoder { + type Error; + fn decode(&self, bytes: &[u8]) -> Result; +} + +pub trait BackendPlanValidator { + type Error; + fn validate( + &self, + plan: &BackendPlan, + capabilities: &BackendCapabilities, + ) -> Result; +} +``` + +`ValidatedBackendPlan` must be constructible only through validation. It proves +schema/lifecycle ordering, unique identities, resolved routes, supported +families/parameters/windows/readouts, and compatible result guarantees. + +```rust +pub struct ValidatedBackendPlan { + pub plan: BackendPlan, + pub capability_hash: String, + pub validated_at: Timestamp, +} + +pub struct BackendCapabilities { + pub capability_hash: String, + pub ingest: Vec, + pub readouts: Vec, + pub storage: Vec, +} + +pub struct BackendPlanSnapshot { + pub plan: Arc, + pub status: PlanRuntimeStatus, + pub installed_at: Timestamp, +} + +pub enum PlanRuntimeStatus { + Staged, + Active, + Draining, + Expired, +} +``` + +Capability entry definitions: + +| Type | Definition | +| --- | --- | +| `IngestCapability` | Supported family, algorithm, parameters, encoding version, and full/delta semantics. | +| `ReadoutCapability` | Supported Planner readout/operator and guarantee kinds. | +| `StorageCapability` | Supported materialization representation, merge, window, retention, and durability behavior. | + +```rust +pub trait BackendPlanRuntime: Send + Sync { + type Error; + + fn stage(&self, plan: ValidatedBackendPlan) + -> Result; + + fn activate(&self, plan_id: &str, plan_version: u64) + -> Result; + + fn snapshot(&self) -> BackendPlanSnapshot; + + fn retire(&self, plan_id: &str, plan_version: u64) + -> Result; +} +``` + +Why these interfaces exist: decoding, validation, and activation have different +failure semantics. A decoded plan must never become queryable before validation +and matching collector evidence. + +## 3. Adding and verifying functionality + +### Add a BackendPlan field + +1. Add it to the public versioned wire/domain structure. +2. Define requiredness, identity impact, and compatibility behavior. +3. Validate it in `BackendPlanValidator`. +4. Expose it through immutable `BackendPlanSnapshot` to its consumer. +5. Verify missing/unknown/incompatible values fail before `stage`. + +### Add a runtime lifecycle state + +1. Extend `PlanRuntimeStatus` with allowed transitions. +2. Define whether ingest/query may use the state. +3. Return the effective state through `BackendApplicationReport`. +4. Verify invalid transitions do not change `snapshot()`. + +### Interpret and verify output + +- A `ValidatedBackendPlan` means the plan is deployable by this backend, not + active. +- A `Staged` report means resources/routes are prepared, not queryable. +- An `Active` report must match the requested plan/version and materializations. +- One request must observe one `BackendPlanSnapshot`, including during swap. +- Re-delivery of identical content is idempotent; conflicting content for the + same identity fails. diff --git a/data_plane/docs/developer_docs/extension-points.md b/data_plane/docs/developer_docs/extension-points.md index 9cc577d9..2829644c 100644 --- a/data_plane/docs/developer_docs/extension-points.md +++ b/data_plane/docs/developer_docs/extension-points.md @@ -1,53 +1,107 @@ -# Data-plane extension boundaries - -> Status: active -> -> MVP relation: Prometheus HTTP and the configured exact fallback are required; -> additional protocols and fallback systems are future extensions. - -## TL;DR - -The data plane separates network transport, request/response adaptation, -plan-aware execution, and exact fallback. An extension implements one boundary -without duplicating planning or bypassing BackendPlan validation. - -## Protocol server - -A protocol server owns network concerns: endpoints, authentication context, -request limits, cancellation, and transport errors. It hands a request to a -protocol adapter and returns the adapter's response. - -It does not parse Planner IR, select a summary, access summary storage directly, -or decide when fallback is allowed. - -## Protocol adapter - -An adapter converts a protocol request into the data plane's canonical query -request and converts the canonical result back into the protocol response. -Prometheus label and timestamp semantics must survive both conversions. - -An adapter may report that a language feature cannot be represented, but it -must not approximate or rewrite an unsupported query on its own. - -## Fallback client - -A fallback client executes the canonical query against the exact backend named -by BackendPlan. It preserves the logical evaluation time, range, tenant, and -error response. - -Fallback is invoked by plan-aware routing. A fallback client must not turn a -remote error into an empty successful result. - -## Adding an extension - -An extension is complete when it demonstrates: - -- request and response semantic round trips; -- cancellation, timeout, and error propagation; -- tenant and authentication context preservation; -- plan-aware routing rather than direct store access; -- no silent fallback or approximation; and -- integration coverage with one successful and one failing request. - -Implementation locations and trait signatures are intentionally left to the -code and API documentation, where they can evolve without changing this design. +# Developing protocol and fallback extensions + +> Interface status: public extension boundary. Concrete trait names may migrate +> toward the canonical interfaces below; private server helpers are not API. + +## 1. Code architecture + +```text +network request -> ProtocolServer -> ProtocolAdapter -> QueryService + | + ExactQueryClient +``` + +- `ProtocolServer` owns transport, authentication context, limits, timeout, and + cancellation. +- `ProtocolAdapter` converts protocol-specific data to/from canonical query + structures. +- `QueryService` performs plan-aware execution. +- `ExactQueryClient` is called only for an explicit fallback route. + +## 2. Public interfaces and definitions + +```rust +pub trait ProtocolAdapter: Send + Sync { + type Request; + type Response; + type Error; + + fn decode(&self, request: Self::Request) + -> Result; + + fn encode(&self, response: QueryResponse) + -> Result; + + fn encode_error(&self, error: QueryError) -> Self::Response; +} +``` + +```rust +pub trait ProtocolServer { + type Error; + async fn serve(&self, service: Arc) -> Result<(), Self::Error> + where + S: QueryService + Send + Sync + 'static; +} +``` + +```rust +pub trait ExactQueryClient: Send + Sync { + type Error; + async fn execute_exact(&self, request: &QueryRequest) + -> Result; + + fn capabilities(&self) -> ExactBackendCapabilities; +} + +pub struct ExactBackendCapabilities { + pub backend_id: String, + pub query_languages: Vec, + pub supports_instant: bool, + pub supports_range: bool, + pub maximum_range: Option, +} +``` + +`QueryRequest` and `QueryResponse` are defined in +[Query routing and readout](query-routing-and-readout.md). They preserve tenant, +query language/expression, logical evaluation range, requested accuracy, +result labels/timestamps/type, source, guarantee, and coverage. + +Why these interfaces exist: transport/protocol extensions cannot bypass +BackendPlan routing or directly access summary storage, and fallback backends +cannot silently reinterpret a request. + +## 3. Adding and verifying functionality + +### Add a protocol adapter + +1. Implement `ProtocolAdapter` for its request/response types. +2. Map every supported evaluation-time/range and tenant field. +3. Preserve Prometheus label/timestamp/result/error semantics where applicable. +4. Verify decode→canonical→encode round trips for success and error cases. + +### Add a protocol server + +1. Implement `ProtocolServer` and inject only the public `QueryService`. +2. Propagate cancellation, timeout, authentication, and request limits. +3. Never call `SummaryStore` or an exact client directly. +4. Verify cancelled requests stop downstream work and transport errors map + through `encode_error`. + +### Add an exact fallback backend + +1. Implement `ExactQueryClient` and declare `ExactBackendCapabilities`. +2. Forward the canonical logical range and tenant unchanged. +3. Return exact `QueryResponse` or a visible error. +4. Verify unsupported capability and remote failure do not return an empty + successful result. + +### Interpret and verify output + +- Adapter output is canonical input, not a routing decision. +- Server success means the response was transported, not that it was + summary-backed. +- Inspect `QueryResponse.source` to distinguish summary and exact fallback. +- End-to-end tests must include one supported request, one explicit fallback, + one malformed request, and one backend failure. diff --git a/data_plane/docs/developer_docs/otlp-summary-ingestion.md b/data_plane/docs/developer_docs/otlp-summary-ingestion.md new file mode 100644 index 00000000..ebc1c2cb --- /dev/null +++ b/data_plane/docs/developer_docs/otlp-summary-ingestion.md @@ -0,0 +1,173 @@ +# Developing OTLP summary ingestion + +> Interface status: target public API. OTLP decoding, SID resolution, and +> summary handling exist; complete BackendPlan-gated validation is partial. + +## 1. Code architecture + +```text +OTLP request + | + v +SummaryDecoder -> SeriesIdentityResolver -> SummaryValidator + | + v + SummaryStateApplier + | + v + SummaryStore +``` + +Transport decoding is separate from semantic validation. No decoder is allowed +to append directly to storage or choose a summary family. + +## 2. Public interfaces and definitions + +```rust +pub trait SummaryDecoder { + type Error; + fn decode(&self, request: OtlpMetricsRequest) + -> Result, Self::Error>; +} + +pub struct ReceivedSummary { + pub tenant: String, + pub resource: AttributeSet, + pub scope: InstrumentationScope, + pub metric_name: String, + pub attributes: AttributeSet, + pub source_timestamp: Timestamp, + pub envelope: SummaryEnvelope, +} +``` + +`SummaryEnvelope` contains plan/materialization/producer/window identity, +family/parameters/encoding, full-or-delta metadata, and payload bytes. Required +identity cannot be inferred from metric-name suffixes. + +```rust +pub struct SummaryEnvelope { + pub plan_id: String, + pub plan_version: u64, + pub materialization_id: String, + pub producer_id: String, + pub window: LogicalWindow, + pub family: SummaryFamily, + pub algorithm: SummaryAlgorithm, + pub parameters: SummaryParameters, + pub encoding: SummaryEncoding, + pub frame: SummaryFrame, + pub payload: Bytes, +} + +pub enum SummaryFrame { + Full { checkpoint_id: String }, + Delta { + base_checkpoint_id: String, + sequence: u64, + }, +} +``` + +```rust +pub trait SeriesIdentityResolver { + type Error; + fn resolve(&self, key: CanonicalSeriesKey) + -> Result; +} +``` + +`SeriesId`, `SeriesIdNamespace`, `CanonicalSeriesKey`, and `ResolvedSeries` have +one public definition in +[Summary storage and series identity](summary-storage-and-series-identity.md#sid-definition). + +```rust +pub trait SummaryValidator { + type Error; + fn validate( + &self, + received: ReceivedSummary, + plan: &BackendPlanSnapshot, + series: ResolvedSeries, + ) -> Result; +} + +pub trait SummaryStateApplier { + type Error; + fn apply(&self, summary: ValidatedSummary) + -> Result; +} + +pub struct ValidatedSummary { + pub received: ReceivedSummary, + pub series: ResolvedSeries, + pub materialization: ValidatedMaterialization, +} + +pub struct ValidatedMaterialization { + pub plan_id: String, + pub plan_version: u64, + pub materialization_id: String, + pub compatibility_fingerprint: String, +} + +pub struct IngestResult { + pub disposition: IngestDisposition, + pub plan_id: String, + pub materialization_id: String, + pub series_id: SeriesId, + pub window: LogicalWindow, + pub queryable_at: Option, +} + +pub enum IngestDisposition { + AppliedFull, + AppliedDelta, + Duplicate, + Rejected, + AwaitingCheckpoint, +} +``` + +Supporting type definitions: + +| Type | Definition | +| --- | --- | +| `OtlpMetricsRequest` | Decoded public OTLP ExportMetricsServiceRequest. | +| `AttributeSet` | Canonically typed OTel attributes with no identity-relevant loss. | +| `InstrumentationScope` | OTel scope name/version/schema identifying the producer library. | +| `LogicalWindow` | Start/end plus window identity used by plan, state, and query coverage. | +| `SummaryEncoding` | Versioned state representation shared by collector/backend capabilities. | + +Why these interfaces exist: each stage can reject invalid data without changing +queryable state, and `IngestResult` gives the MVP harness unambiguous evidence. + +## 3. Adding and verifying functionality + +### Add an OTLP summary encoding + +1. Extend public `SummaryEnvelope` encoding/version definitions. +2. Implement `SummaryDecoder` without applying state. +3. Add compatibility validation against BackendPlan/capabilities. +4. Implement full/delta application through `SummaryStateApplier`. +5. Verify corrupt bytes return an error and do not change storage. + +### Add a delta-capable family + +Define base/checkpoint, sequence scope, duplicate handling, gap behavior, and +recovery full state. Verify `AppliedDelta`, `Duplicate`, and +`AwaitingCheckpoint` are distinguishable outputs for reorder/gap tests. + +### Add series identity behavior + +Add canonical input fields to `CanonicalSeriesKey`, never to `SeriesId.value` +alone. Verify label-order independence, tenant isolation, cached-ID conflict +recovery, and stable namespace reporting. + +### Interpret and verify output + +- `Applied*` means compatible state was committed. +- `Duplicate` means idempotent replay with no second mutation. +- `AwaitingCheckpoint` means a visible delta gap and non-queryable state. +- `queryable_at` is populated only when coverage/readiness is satisfied. +- Freshness uses `source_timestamp -> queryable_at`, not receive time. diff --git a/data_plane/docs/developer_docs/query-routing-and-readout.md b/data_plane/docs/developer_docs/query-routing-and-readout.md new file mode 100644 index 00000000..88b59801 --- /dev/null +++ b/data_plane/docs/developer_docs/query-routing-and-readout.md @@ -0,0 +1,139 @@ +# Developing query routing and summary readout + +> Interface status: target public API. Summary execution and fallback exist; +> BackendPlan is still replacing legacy routing and local query-shape logic. + +## 1. Code architecture + +```text +protocol request -> QueryAdapter -> QueryService + | + BackendPlanSnapshot + / \ + SummaryReader ExactQueryClient + \ / + QueryResponse +``` + +The adapter owns protocol conversion. `QueryService` owns plan-aware route +selection. `SummaryReader` executes an already selected readout. The exact +client executes only explicit fallback routes. + +## 2. Public interfaces and definitions + +```rust +pub struct QueryRequest { + pub tenant: String, + pub language: QueryLanguage, + pub expression: String, + pub evaluation: EvaluationRange, + pub requested_accuracy: AccuracyRequirement, +} + +pub struct EvaluationRange { + pub start: Timestamp, + pub end: Timestamp, + pub step: Option, +} +``` + +```rust +pub trait QueryService { + type Error; + async fn execute(&self, request: QueryRequest) + -> Result; +} + +pub trait SummaryReader { + type Error; + fn read( + &self, + request: &QueryRequest, + route: &SummaryRoute, + plan: &BackendPlanSnapshot, + ) -> Result; +} + +pub struct SummaryRoute { + pub query_id: String, + pub materialization_ids: Vec, + pub readout: ReadoutSpec, + pub required_guarantee: AccuracyRequirement, +} + +pub struct SummaryReadout { + pub result: PrometheusResult, + pub guarantee: ResultGuarantee, + pub coverage: LogicalCoverage, +} + +pub trait ExactQueryClient { + type Error; + async fn execute_exact(&self, request: &QueryRequest) + -> Result; +} +``` + +Supporting public type definitions: + +| Type | Definition | +| --- | --- | +| `QueryLanguage` | Language identifier; MVP value is PromQL. | +| `AccuracyRequirement` | Exact, epsilon, or epsilon-delta constraint requested for the result. | +| `ReadoutSpec` | Planner-selected operation and typed parameters applied to maintained state. | +| `PrometheusResult` | Matrix/vector/scalar/string result with labels, timestamps, values, warnings, and errors. | +| `ResultGuarantee` | Effective exact/approximate guarantee of the returned result. | +| `LogicalCoverage` | Requested and actually covered time intervals plus readiness timestamp. | +| `QueryError` | Typed parse, unsupported, inactive-plan, missing/stale/gapped state, or exact-backend failure. | + +```rust +pub struct QueryResponse { + pub result: PrometheusResult, + pub source: QuerySource, + pub guarantee: ResultGuarantee, + pub coverage: LogicalCoverage, + pub plan_id: Option, + pub materialization_ids: Vec, +} + +pub enum QuerySource { + Summary, + ExactFallback, +} +``` + +Why these interfaces exist: protocol code cannot bypass plan/readiness checks, +and callers can interpret whether an answer is summary-backed or exact with its +coverage and guarantee. + +## 3. Adding and verifying functionality + +### Add a readout/operator + +1. Add the logical semantics and guarantee to ASAPPlanner. +2. Extend public backend readout capability and BackendPlan route types. +3. Implement it through `SummaryReader`; do not parse and choose a family again. +4. Return labels/timestamps/result type through `PrometheusResult`. +5. Compare with the exact backend over identical series and logical range. + +### Add a protocol adapter + +Convert protocol inputs to `QueryRequest` and `QueryResponse` back to protocol +output. Verify tenant, evaluation timestamps, labels, result type, errors, and +accuracy metadata round-trip unchanged. + +### Add an exact backend + +Implement `ExactQueryClient`, preserving the complete `QueryRequest`. Verify +remote failures remain errors and are not successful empty vectors. + +### Interpret and verify output + +- `QuerySource::Summary` requires active plan/materialization IDs and complete + coverage. +- `QuerySource::ExactFallback` must satisfy exact semantics and carry no false + summary guarantee. +- Missing/additional labels or timestamps are validation failures. +- Partial/stale/gapped summary state must not return a successful complete + `QueryResponse`. +- A plan swap during execution must not mix identities in one response. diff --git a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md new file mode 100644 index 00000000..e608803b --- /dev/null +++ b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md @@ -0,0 +1,200 @@ +# Developing summary storage and series identity + +> Interface status: target public API. Store/index and SID resolution exist; +> plan/materialization lifecycle convergence is partial. + +## 1. Code architecture + +```text +CanonicalSeriesKey -> SeriesRegistry -> SeriesId + | +ValidatedMaterialization ----------------+ + | + v + SummaryStore + write / coverage / read / retire +``` + +The series registry owns only canonical metric-series identity. The summary +store owns materialization/group/window state. Plan, materialization, and SID +identities remain distinct. + +## 2. Public interfaces and definitions + +```rust +pub struct SeriesId { + pub namespace: SeriesIdNamespace, + pub value: u64, +} + +pub struct SeriesIdNamespace { + pub tenant: String, + pub version: String, +} + +pub struct CanonicalSeriesKey { + pub tenant: String, + pub metric_name: String, + pub identifying_labels: BTreeMap, +} + +pub struct ResolvedSeries { + pub id: SeriesId, + pub canonical_key: CanonicalSeriesKey, +} + +pub trait SeriesRegistry: Send + Sync { + type Error; + + fn resolve(&self, key: CanonicalSeriesKey) + -> Result; + + fn lookup(&self, id: &SeriesId) + -> Result, Self::Error>; +} +``` + +### SID definition + +`SeriesId` (`sid`) is an opaque numeric identifier scoped by exactly one +`SeriesIdNamespace`. The namespace contains the tenant/isolation domain and a +version that changes whenever the authoritative registry is rebuilt without +preserving its previous assignments. + +```text +(SeriesIdNamespace, SeriesId.value) <-> CanonicalSeriesKey +``` + +Within one namespace this mapping is one-to-one: + +- the same canonical key always resolves to the same SID; +- two different canonical keys never resolve to the same SID; and +- the same numeric value in two namespaces is not the same SID. + +`identifying_labels` is ordered by label name before lookup or hashing, so input +label order does not affect identity. Summary family, parameters, aggregation +group, window, materialization ID, and plan ID are excluded because they +identify maintained state, not the source metric series. + +`SeriesRegistry::resolve` is idempotent. A sender-provided SID is only a lookup +shortcut; it never overrides a conflicting canonical key. + +```rust +pub struct MaterializationKey { + pub plan_id: String, + pub plan_version: u64, + pub materialization_id: String, + pub tenant: String, + pub group: MaterializationGroup, + pub window: LogicalWindow, +} + +pub enum MaterializationState { + Staged, + Queryable, + Gapped, + Draining, + Expired, + Rejected, +} +``` + +```rust +pub trait SummaryStore: Send + Sync { + type Error; + + fn register(&self, contract: ValidatedMaterialization) + -> Result; + + fn apply(&self, update: ValidatedSummary) + -> Result; + + fn coverage(&self, request: CoverageRequest) + -> Result; + + fn read(&self, request: SummaryReadRequest) + -> Result; + + fn retire(&self, materialization_id: &str, policy: RetirementPolicy) + -> Result; +} + +pub struct CoverageRequest { + pub plan: BackendPlanSnapshot, + pub materialization_ids: Vec, + pub range: EvaluationRange, +} + +pub struct SummaryReadRequest { + pub coverage: LogicalCoverage, + pub route: SummaryRoute, +} + +pub struct SummaryReadResult { + pub states: Vec, + pub coverage: LogicalCoverage, +} + +pub struct RetirementPolicy { + pub drain_until: Timestamp, + pub retain_for_rollback_until: Option, +} +``` + +```rust +pub struct StoreWriteResult { + pub key: MaterializationKey, + pub state: MaterializationState, + pub disposition: IngestDisposition, +} + +pub enum CoverageResult { + Complete(LogicalCoverage), + Missing(Vec), + Stale { newest_source_timestamp: Timestamp }, + Gapped { producer: String, expected_sequence: u64 }, + Incompatible { reason: String }, +} +``` + +Supporting types `ValidatedMaterialization`, `ValidatedSummary`, and +`IngestDisposition` are defined by +[OTLP summary ingestion](otlp-summary-ingestion.md). `EvaluationRange`, +`SummaryRoute`, and `LogicalCoverage` are defined by +[Query routing and readout](query-routing-and-readout.md). + +Why these interfaces exist: callers receive typed completeness/failure rather +than interpreting an empty collection as “no data,” and storage cannot accept +unvalidated summary bytes. + +## 3. Adding and verifying functionality + +### Add a summary family to storage + +1. Extend public materialization capability/contract types. +2. Define canonical parameters and representation compatibility. +3. Accept only `ValidatedSummary` through `SummaryStore::apply`. +4. Implement merge/read behavior through public result types. +5. Verify incompatible family/parameters/windows never merge. + +### Add a storage backend + +Implement `SummaryStore` with identical semantic outputs. Persistence or remote +transport must not change coverage, lifecycle, identity, or error behavior. +Verify restart restores metadata before returning `Complete` or `Queryable`. + +### Add SID persistence/distribution + +Implement `SeriesRegistry` while preserving deterministic canonical keys, +tenant isolation, idempotent resolve, namespace versioning, and conflict +detection. Verify cache loss/restart cannot bind an old SID to new labels. + +### Interpret and verify output + +- `Queryable` means the registered state may be considered for coverage; it is + not proof that every requested window is complete. +- Only `CoverageResult::Complete` may proceed to summary readout. +- `Missing`, `Stale`, `Gapped`, and `Incompatible` must remain distinguishable. +- `StoreWriteResult` identifies exactly which plan/materialization/window was + changed. +- Concurrent plan versions remain isolated through `MaterializationKey`. diff --git a/docs/README.md b/docs/README.md index bc1f9c36..3f3b3059 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,12 @@ - [Adding a summary family](developer_docs/adding-summary-family.md) - [Data-plane extension boundaries](../data_plane/docs/developer_docs/extension-points.md) +- [Planner adapter and physical compiler](../control_plane/docs/developer_docs/planner-and-physical-compiler.md) +- [Runtime plan publication](../control_plane/docs/developer_docs/runtime-plan-publication.md) +- [BackendPlan runtime](../data_plane/docs/developer_docs/backend-plan-runtime.md) +- [OTLP summary ingestion](../data_plane/docs/developer_docs/otlp-summary-ingestion.md) +- [Query routing and readout](../data_plane/docs/developer_docs/query-routing-and-readout.md) +- [Summary storage and series identity](../data_plane/docs/developer_docs/summary-storage-and-series-identity.md) - [Querying ASAP](../data_plane/docs/user_guide/querying-asap.md) ## Documentation ownership diff --git a/docs/design_docs/series-identity.md b/docs/design_docs/series-identity.md index a57de291..b786b063 100644 --- a/docs/design_docs/series-identity.md +++ b/docs/design_docs/series-identity.md @@ -5,6 +5,9 @@ > MVP relation: provides stable identity for ingestion, grouping, and result > labels across collector and backend boundaries. +Developer guide: +[Summary storage and series identity](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md). + ## TL;DR A series ID (`sid`) names one canonical metric series within a tenant and @@ -12,6 +15,11 @@ identity namespace. The backend registry assigns or validates this mapping; collectors may cache it, but payload labels remain the recovery evidence needed to detect stale or unknown IDs. +Formally, SID is the pair `(namespace, numeric_value)`, not a globally meaningful +integer. The namespace contains the tenant/isolation domain and a registry +version. Its public data structures and registry interfaces are defined in the +[developer guide](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md#sid-definition). + ## Identity contract The canonical series key consists of: diff --git a/docs/design_docs/summary-storage.md b/docs/design_docs/summary-storage.md index 661be3fd..924a320d 100644 --- a/docs/design_docs/summary-storage.md +++ b/docs/design_docs/summary-storage.md @@ -4,6 +4,9 @@ > > MVP relation: stores the state needed for summary-backed query execution. +Developer guide: +[Summary storage and series identity](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md). + ## TL;DR Summary storage is a plan-aware materialized-state store. It accepts only state diff --git a/docs/developer_docs/adding-summary-family.md b/docs/developer_docs/adding-summary-family.md index e4d3600d..2924a018 100644 --- a/docs/developer_docs/adding-summary-family.md +++ b/docs/developer_docs/adding-summary-family.md @@ -1,54 +1,138 @@ -# Adding a summary family to ASAPQuery-backend +# Adding a summary family -## TL;DR +> Interface status: cross-repository developer workflow. ASAPQuery-backend +> implements runtime capabilities; ASAPPlanner and summary libraries own logical +> semantics and algorithm guarantees. -ASAPQuery-backend adds runtime support for a summary family only after -ASAPPlanner defines its logical query mapping and guarantee, and the producing -collector/library defines compatible state semantics. The backend must not -invent those contracts locally. +## 1. Code architecture -## Ownership prerequisites +```text +ASAPPlanner public summary/readout types + | + v +PhysicalCompiler capability match + / \ +CollectorPlan BackendPlan + | | +ASAPCollector SummaryDecoder +update + encode -> SummaryStore -> SummaryReader +``` + +A family is supported only when the same public semantic contract crosses all +components. A decoder or enum variant by itself is not pipeline support. + +## 2. Public interfaces and definitions + +The following public structures must describe the same family/version: + +```rust +pub struct SummaryCapability { + pub family: SummaryFamily, + pub algorithm: SummaryAlgorithm, + pub parameter_schema: ParameterSchema, + pub encodings: Vec, + pub operations: SummaryOperations, + pub readouts: Vec, + pub guarantee_kinds: Vec, +} + +pub struct SummaryOperations { + pub update: bool, + pub merge: bool, + pub subtract: bool, + pub delete: bool, + pub full_state: bool, + pub delta_state: bool, +} +``` + +```rust +pub trait SummaryDecoder { + type Error; + fn decode(&self, request: OtlpMetricsRequest) + -> Result, Self::Error>; +} + +pub trait SummaryStore { + type Error; + fn register(&self, contract: ValidatedMaterialization) + -> Result; + fn apply(&self, update: ValidatedSummary) + -> Result; +} + +pub trait SummaryReader { + type Error; + fn read( + &self, + request: &QueryRequest, + route: &SummaryRoute, + plan: &BackendPlanSnapshot, + ) -> Result; +} +``` + +Definitions: + +| Interface | Input | Output | +| --- | --- | --- | +| Planner mapping | PromQL workload and constraints | Selected logical summary producer/readout and guarantee | +| Capability | Family/algorithm/version | Supported parameters, encodings, operations, readouts, guarantees | +| Decoder | OTLP request | Untrusted `ReceivedSummary` values | +| Validator | Received summary + active plan | `ValidatedSummary` or structured error | +| Store | Validated materialization/update | Lifecycle/write result | +| Reader | Query + selected route + plan snapshot | Summary readout with coverage/guarantee | + +Why these interfaces exist: every stage can compare exact typed semantics and +reject unsupported combinations instead of mapping a new family to a similar +legacy one. + +## 3. Adding and verifying a family -Before changing this repository, confirm: +### Step 1: define logical semantics outside this repository -- [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) can represent and - select the family for concrete PromQL examples; -- the summary library defines parameters, update, merge/readout, encoding, and - accuracy behavior; and -- [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector) can advertise, - configure, construct, and transmit the same family/version. +Add the query mapping, readout, composability, and guarantee to ASAPPlanner. +Add update/merge/encoding behavior and mathematical guarantee to the owning +summary library. Record concrete PromQL examples. -## Backend work +### Step 2: advertise runtime capability -Backend support covers four boundaries: +Add `SummaryCapability` values for collector and backend. Declare only the +parameter ranges, encodings, operations, and readouts actually implemented. +Verify the physical compiler rejects a candidate if either side lacks one +required capability. -1. **Capability:** advertise the exact family, algorithm, parameter, readout, - merge, representation, and full/delta support implemented. -2. **Physical compilation:** accept only selected Planner nodes that can be - assigned to compatible collector and backend executors. -3. **BackendPlan and ingestion:** preserve the selected contract and reject - incompatible payloads. -4. **Readout:** execute the declared operation and return aligned - Prometheus-compatible labels, timestamps, values, and errors. +### Step 3: ingest and store -For example, support for a new quantile family is incomplete until this query -can be planned, produced, ingested, and read end to end: +Implement decode to `ReceivedSummary`, validation to `ValidatedSummary`, and +store application through public interfaces. Include family, algorithm, +canonical parameters, encoding version, grouping, and window in compatibility +identity. + +### Step 4: execute readout + +Implement `SummaryReader::read` for the Planner-selected readout. Preserve +labels, timestamps, result type, logical coverage, and guarantee. Do not choose +the family again from query text. + +### Step 5: interpret and verify output + +For a quantile family, an end-to-end example is: ```promql quantile_over_time(0.95, request_duration_seconds[5m]) ``` -## Validation - -The cross-repository test must cover: +Verify: -- supported and deliberately unsupported parameters; -- full-state transmission and delta transmission when claimed; -- duplicate, missing, reordered, stale, and incompatible payloads; -- merge across every claimed grouping/window shape; -- aligned comparison with an identical exact input stream; -- the declared accuracy and freshness SLA; and -- capability downgrade and exact fallback behavior. +- output series align with exact results by labels and timestamps; +- reported guarantee matches the selected parameterization; +- errors satisfy the predeclared SLA over identical input; +- full and delta state have equivalent query semantics when delta is claimed; +- corrupt, mismatched, stale, gapped, or unsupported input returns a structured + failure and does not change queryable state; and +- `QueryResponse.source`, plan/materialization IDs, coverage, and freshness + prove which implementation produced the answer. -Unit tests for serialization or a local readout alone do not establish pipeline +Unit tests for serialization are necessary but do not establish cross-repository support.