From 92a1d1461465d7b37b65f6e2eea6bb0b37a12033 Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 28 Aug 2026 04:52:09 -0600 Subject: [PATCH 1/3] docs: add backend component developer guides --- control_plane/docs/README.md | 5 + control_plane/docs/asapplanner-integration.md | 2 + control_plane/docs/backend-plan.md | 4 + .../planner-and-physical-compiler.md | 110 +++++++++++++++++ .../runtime-plan-publication.md | 115 ++++++++++++++++++ control_plane/docs/physical-planning.md | 4 + data_plane/docs/README.md | 8 ++ .../docs/design_docs/query-execution.md | 5 + .../developer_docs/backend-plan-runtime.md | 82 +++++++++++++ .../developer_docs/otlp-summary-ingestion.md | 79 ++++++++++++ .../query-routing-and-readout.md | 71 +++++++++++ .../summary-storage-and-series-identity.md | 84 +++++++++++++ docs/README.md | 6 + docs/design_docs/series-identity.md | 3 + docs/design_docs/summary-storage.md | 3 + 15 files changed, 581 insertions(+) create mode 100644 control_plane/docs/developer_docs/planner-and-physical-compiler.md create mode 100644 control_plane/docs/developer_docs/runtime-plan-publication.md create mode 100644 data_plane/docs/developer_docs/backend-plan-runtime.md create mode 100644 data_plane/docs/developer_docs/otlp-summary-ingestion.md create mode 100644 data_plane/docs/developer_docs/query-routing-and-readout.md create mode 100644 data_plane/docs/developer_docs/summary-storage-and-series-identity.md diff --git a/control_plane/docs/README.md b/control_plane/docs/README.md index 0c46724c..6b33fc8d 100644 --- a/control_plane/docs/README.md +++ b/control_plane/docs/README.md @@ -22,6 +22,11 @@ designs; they are intentionally not repeated here. | [Physical planning](physical-planning.md) | Placement, windows, representation, transmission, and compilation into matching collector and backend plans. | | [BackendPlan](backend-plan.md) | Versioned contract installed and executed by the ASAPQuery data plane. | +## Developer documentation + +- [Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md) +- [Runtime plan publication](developer_docs/runtime-plan-publication.md) + The corresponding collector-facing plan is documented by [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). diff --git a/control_plane/docs/asapplanner-integration.md b/control_plane/docs/asapplanner-integration.md index 95a658d4..087760ad 100644 --- a/control_plane/docs/asapplanner-integration.md +++ b/control_plane/docs/asapplanner-integration.md @@ -4,6 +4,8 @@ > > MVP relation: required for query planning and control-plane decisions. +Developer guide: [Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md). + ## TL;DR ASAPPlanner chooses a logical plan for a query workload. ASAPQuery-backend diff --git a/control_plane/docs/backend-plan.md b/control_plane/docs/backend-plan.md index 303c90b7..a7b36b61 100644 --- a/control_plane/docs/backend-plan.md +++ b/control_plane/docs/backend-plan.md @@ -9,6 +9,10 @@ > plane tells its data plane what summary materializations exist, how they > are ingested, and which query capabilities they serve. +Developer guides: +[runtime plan publication](developer_docs/runtime-plan-publication.md) and +[BackendPlan installation](../../data_plane/docs/developer_docs/backend-plan-runtime.md). + ## TL;DR Planning chooses once; serving reuses that exact decision. diff --git a/control_plane/docs/developer_docs/planner-and-physical-compiler.md b/control_plane/docs/developer_docs/planner-and-physical-compiler.md new file mode 100644 index 00000000..40a5f611 --- /dev/null +++ b/control_plane/docs/developer_docs/planner-and-physical-compiler.md @@ -0,0 +1,110 @@ +# Planner adapter and physical compiler + +> Implementation status: target integration; current modules are migration +> substrate and do not replace ASAPPlanner ownership. + +## Purpose + +This component consumes one ASAPPlanner-selected workload plan and compiles it +into matching collector and backend runtime plans. It must preserve Planner's +logical semantics while adding only ASAPQuery-backend-owned deployment choices. + +Design sources: + +- [ASAPPlanner integration](../asapplanner-integration.md) +- [Physical planning](../physical-planning.md) + +## Component boundary + +```text +workload + schema + logical constraints + | + v + ASAPPlanner adapter + | + selected workload DAG + | + v + physical compiler + / \ + v v + CollectorPlan BackendPlan +``` + +The adapter must use the pinned ASAPPlanner API directly. It must not recreate +Planner IR, query-to-summary rules, accuracy algebra, or candidate ranking in +backend-local types. + +## Current code map + +| Responsibility | Current entry point | +| --- | --- | +| Workload pipeline coordination | [`pipeline.rs`](../../src/pipeline.rs) | +| Planner-facing workload/types during migration | [`workload.rs`](../../src/workload.rs), [`types_v2.rs`](../../src/types_v2.rs) | +| Physical allocation | [`physical/allocator.rs`](../../src/physical/allocator.rs) | +| Stage allocation and topology | [`physical/colored_dag/`](../../src/physical/colored_dag/) | +| Window realization | [`physical/window_fusion.rs`](../../src/physical/window_fusion.rs) | +| Runtime stage model and emitter | [`physical/colored_dag/emitter.rs`](../../src/physical/colored_dag/emitter.rs) | +| BackendPlan compilation | [`backend_plan/from_stage_config.rs`](../../src/backend_plan/from_stage_config.rs) | + +The legacy `intent_algebra`, `sketch_algebra`, and optimizer modules are +migration inputs, not a second canonical Planner implementation. New logical +semantics belong in ASAPPlanner. + +## Inputs + +The compile operation requires: + +- the complete selected Planner workload DAG, with shared nodes intact; +- the immutable Planner revision; +- collector and backend capability snapshots; +- deployment topology and tenant boundaries; +- physical window, freshness, retention, and transmission policy; and +- runtime statistics and resource constraints used for physical placement. + +Missing capabilities or statistics stay unknown. Do not replace unknown values +with zero cost, exact accuracy, or universal support. + +## Outputs + +One compile produces one shared plan envelope plus: + +- one CollectorPlan for each targeted collector; and +- one BackendPlan for the data plane. + +Both sides must agree on materialization identity, family, parameters, +grouping, windows, representation, and transmission semantics. Generate them +from one in-memory compiled decision, never with independent reinterpretation. + +## Implementation invariants + +- Preserve workload-wide sharing; do not flatten to independent metric rows. +- Treat Planner result guarantees and exact fallback as authoritative. +- Reject an operation if its assigned executor lacks any required capability. +- Keep logical range semantics separate from physical pane size. +- Keep aggregation placement separate from logical grouping. +- Permit delta only when both endpoints share sequencing/checkpoint semantics. +- Make plan/materialization identity deterministic from semantic content. + +## Adding a physical decision + +When adding placement, window, representation, or transport behavior: + +1. Confirm it does not change the selected logical result. +2. Add the capability input needed to make the decision safely. +3. Include every semantic effect in materialization or compatibility identity. +4. Emit the choice to both runtime plans where applicable. +5. Add a mismatch test showing incompatible plans fail before activation. + +If the change selects a different summary or changes its guarantee, implement +it in ASAPPlanner instead. + +## Required tests + +- shared Planner nodes remain one logical materialization; +- collector and backend outputs carry identical semantic fields; +- unsupported capabilities fail closed; +- incompatible windows or delta semantics are rejected; +- deterministic input produces deterministic identities; and +- a three-query PromQL workload covers within-series, across-label, and combined + time-plus-label aggregation shapes. diff --git a/control_plane/docs/developer_docs/runtime-plan-publication.md b/control_plane/docs/developer_docs/runtime-plan-publication.md new file mode 100644 index 00000000..03456b75 --- /dev/null +++ b/control_plane/docs/developer_docs/runtime-plan-publication.md @@ -0,0 +1,115 @@ +# Runtime plan publication + +> Implementation status: target contract; transport exists, but semantic +> CollectorPlan application/reporting is not complete. + +## Purpose + +Plan publication stages and activates the matching CollectorPlan and BackendPlan +created by the physical compiler. Transport success is not semantic activation. + +Design sources: + +- [Physical planning](../physical-planning.md) +- [BackendPlan](../backend-plan.md) +- [ASAPCollector plan interface](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md) + +## Current code map + +| Responsibility | Current entry point | +| --- | --- | +| Runtime-specific emission | [`emit/mod.rs`](../../src/emit/mod.rs) | +| Collector/gateway/backend payloads | [`emit/stage_config.rs`](../../src/emit/stage_config.rs) | +| Backend publication | [`emit/backend_push.rs`](../../src/emit/backend_push.rs) | +| OpAMP collector delivery/status | [`opamp/mod.rs`](../../src/opamp/mod.rs) | +| BackendPlan schema | [`backend_plan/mod.rs`](../../src/backend_plan/mod.rs) | +| BackendPlan protobuf | [`proto/backend_plan.proto`](../../proto/backend_plan.proto) | + +## Publication sequence + +1. Validate both runtime plans against the capability snapshots used to compile + them. +2. Stage BackendPlan without routing production queries to it. +3. Deliver CollectorPlan through OpAMP to every required collector. +4. Require backend installation evidence and collector semantic-application + evidence for the same plan/version. +5. Activate query routing at the declared activation boundary. +6. Drain and retire the previous version after readers and lateness expire. + +Any partial failure leaves the previous unexpired plan authoritative. + +## Collector transport contract + +Do not duplicate the CollectorPlan schema here. The authoritative field and +validation definitions are in ASAPCollector's +[`opamp-config-push.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). + +The backend publisher must preserve these corresponding requirements: + +- transport is OpAMP `AgentRemoteConfig`/`AgentConfigMap`; +- the exact entry name is `asap-collector-plan.yaml`; +- the body is the versioned YAML `CollectorPlan`, with + `content_type: application/yaml`; +- OpAMP `config_hash` identifies delivered bytes and is distinct from the + cross-runtime `metadata.plan_id`; +- CollectorPlan and BackendPlan share `plan_id`, `plan_version`, + `backend_compat`, and materialization identities; and +- an OpAMP `RemoteConfigStatus.APPLIED` response does not prove semantic + activation. + +The expected semantic application report uses the collector capability +`io.asap.collector.plan.v1`, message type `application_report`, and reports the +active plan/version, remote-config hash, backend compatibility, active +materializations, effective capability hash, timestamps, and structured +errors. Treat unknown or missing fields according to the versioned collector +contract rather than guessing defaults. + +## Evidence contract + +The publisher distinguishes: + +- transport acknowledgement: bytes reached an endpoint; +- validation acknowledgement: payload schema/capabilities were accepted; +- semantic application: the runtime reports the expected active plan and + materialization identities; and +- data evidence: emitted/ingested state carries those identities. + +Only semantic application plus compatible data evidence can make a plan +queryable. + +## Current implementation gap + +The corresponding ASAPCollector document records that current OpAMP handling +still applies a complete OTel Collector YAML, identifies it primarily by +`config_hash`, and reports `APPLIED` after syntactic validation/file write. It +does not yet provide the target CollectorPlan parser, atomic in-process apply, +or semantic application report. + +Backend code and tests must represent this honestly. Until both sides implement +the target contract, `APPLIED` is delivery evidence only and cannot satisfy the +MVP plan-application gate. + +## Versioning and retry + +- Re-sending identical `(plan_id, plan_version)` content is idempotent. +- Reusing that pair for different content is an error. +- Older or expired versions cannot replace a newer active version. +- Retry preserves activation and expiry timestamps. +- Rollback selects an explicitly retained compatible version; it does not edit + an active plan in place. + +## Failure handling + +Surface collector rejection, backend rejection, timeout, partial rollout, stale +status, and identity mismatch separately. Do not report a successful plan push +when only one runtime side applied it. + +## Required tests + +- idempotent repeated delivery; +- stale version rejection; +- collector-only and backend-only application remain inactive; +- activation succeeds only for matching identities; +- failed rollout retains the old route; +- rollback restores a complete prior pair; and +- status artifacts contain enough evidence for the MVP harness. diff --git a/control_plane/docs/physical-planning.md b/control_plane/docs/physical-planning.md index 33b5e398..d4a55b7d 100644 --- a/control_plane/docs/physical-planning.md +++ b/control_plane/docs/physical-planning.md @@ -9,6 +9,10 @@ > selected post-ASAP workload DAG and the two runtime executors: > ASAPCollector and the ASAPQuery data plane. +Developer guides: +[Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md) +and [runtime plan publication](developer_docs/runtime-plan-publication.md). + ## TL;DR ASAPPlanner selects a logical plan. That plan says which summaries and exact diff --git a/data_plane/docs/README.md b/data_plane/docs/README.md index 0572b571..2a5bc459 100644 --- a/data_plane/docs/README.md +++ b/data_plane/docs/README.md @@ -18,6 +18,14 @@ or re-plan queries. - [Extension boundaries](developer_docs/extension-points.md) — responsibilities of protocol servers, adapters, and fallback clients. +- [BackendPlan runtime](developer_docs/backend-plan-runtime.md) — validation, + staging, atomic installation, and snapshots. +- [OTLP summary ingestion](developer_docs/otlp-summary-ingestion.md) — decoding, + plan validation, SID resolution, and full/delta handling. +- [Query routing and readout](developer_docs/query-routing-and-readout.md) — + readiness, summary execution, and exact fallback. +- [Summary storage and series identity](developer_docs/summary-storage-and-series-identity.md) + — store boundaries, identity hierarchy, lifecycle, and concurrency. - [Adding a summary family](../../docs/developer_docs/adding-summary-family.md) — cross-repository prerequisites and backend validation. diff --git a/data_plane/docs/design_docs/query-execution.md b/data_plane/docs/design_docs/query-execution.md index 761cced4..484688c8 100644 --- a/data_plane/docs/design_docs/query-execution.md +++ b/data_plane/docs/design_docs/query-execution.md @@ -4,6 +4,11 @@ > > MVP relation: required for every summary-backed query and exact fallback. +Developer guides: +[BackendPlan runtime](../developer_docs/backend-plan-runtime.md), +[OTLP summary ingestion](../developer_docs/otlp-summary-ingestion.md), and +[query routing/readout](../developer_docs/query-routing-and-readout.md). + ## TL;DR The data plane accepts only state compatible with its active BackendPlan. At diff --git a/data_plane/docs/developer_docs/backend-plan-runtime.md b/data_plane/docs/developer_docs/backend-plan-runtime.md new file mode 100644 index 00000000..9eaee8f4 --- /dev/null +++ b/data_plane/docs/developer_docs/backend-plan-runtime.md @@ -0,0 +1,82 @@ +# BackendPlan installation and runtime state + +> Implementation status: partial; atomic plan storage exists, while complete +> staging, lifecycle, and cross-runtime activation checks remain target work. + +## Purpose + +This component validates, stages, and atomically installs the BackendPlan +produced by the control plane. Query and ingest operations take stable snapshots +of the active plan; they do not observe a partially written plan. + +Design source: [BackendPlan](../../../control_plane/docs/backend-plan.md). + +## Current code map + +| Responsibility | Current entry point | +| --- | --- | +| Shared BackendPlan types/decoder | [`control_plane::backend_plan`](../../../control_plane/src/backend_plan/mod.rs) | +| Atomic runtime holder | [`HotReloadBackendPlan`](../../src/storage_engines/types/hot_reload_config.rs) | +| HTTP installation surface | [`drivers/query/servers/http.rs`](../../src/drivers/query/servers/http.rs) | +| Plan-aware readout | [`l4_readout.rs`](../../src/query_engines/asap_query_engine/l4_readout.rs) | + +The current `HotReloadBackendPlan` is an `ArcSwap` holder. Atomic pointer swap +is necessary but not sufficient: target validation, version ordering, staged +activation, expiry, and matching collector evidence belong around that holder. + +## Installation pipeline + +```text +received BackendPlan + | +decode and closed-schema validation + | +capability + state-compatibility validation + | +stage materializations/routes + | +match CollectorPlan application evidence + | +atomic activation +``` + +Validation must complete before replacing the active snapshot. Readers already +holding the previous snapshot may finish, but a single request must not mix plan +versions. + +## Required validation + +- plan/version/lifecycle ordering; +- immutable content for repeated `(plan_id, plan_version)`; +- unique materialization and route identity; +- known family, algorithm, parameters, grouping, windows, and readout; +- backend capability support for ingest, merge, representation, and readout; +- route references resolve to declared materializations or exact fallback; +- result guarantees meet the route's declared requirement; and +- collector compatibility/evidence matches before activation. + +Reject the whole plan on failure. Do not drop one invalid route and activate the +rest unless a future schema explicitly defines partial activation semantics. + +## Reader contract + +Ingest and query entry points obtain one active-plan snapshot at operation +start. They pass it through validation/routing rather than consulting global +state repeatedly. State emitted under another version is not accepted merely +because its summary bytes decode. + +## Current migration boundary + +`StreamingConfig`, `BackendStorageRouting`, and BackendPlan currently coexist. +Treat the first two as legacy runtime inputs being absorbed into BackendPlan. +New semantic fields should be added to the versioned plan contract rather than +creating another independently hot-reloaded configuration source. + +## Required tests + +- malformed and unknown schema values fail without swapping; +- older/conflicting versions fail; +- concurrent readers see one complete version; +- routes cannot reference missing materializations; +- plan activation waits for matching collector evidence; and +- rollback restores a complete retained snapshot. diff --git a/data_plane/docs/developer_docs/otlp-summary-ingestion.md b/data_plane/docs/developer_docs/otlp-summary-ingestion.md new file mode 100644 index 00000000..a56b692a --- /dev/null +++ b/data_plane/docs/developer_docs/otlp-summary-ingestion.md @@ -0,0 +1,79 @@ +# OTLP summary ingestion + +> Implementation status: partial; OTLP decoding, SID resolution, and summary +> handling exist, while full BackendPlan-gated validation remains target work. + +## Purpose + +The ingestion component receives OTel metric payloads from ASAPCollector, +resolves series identity, validates summary state against the active +BackendPlan, applies full/delta semantics, and appends queryable windows to +summary storage. + +Design sources: + +- [Plan-aware query execution](../design_docs/query-execution.md) +- [Summary storage](../../../docs/design_docs/summary-storage.md) +- [Series identity](../../../docs/design_docs/series-identity.md) + +## Current code map + +| Responsibility | Current entry point | +| --- | --- | +| OTLP receivers and decoding | [`drivers/ingest/otel.rs`](../../src/drivers/ingest/otel.rs) | +| Canonical SID resolution | [`drivers/ingest/series_resolver.rs`](../../src/drivers/ingest/series_resolver.rs) | +| Full/delta evaluation helpers | [`sketch_db/query/delta_apply.rs`](../../src/storage_engines/sketch_db/query/delta_apply.rs) | +| Summary decoders | [`sketch_db/query/decoders.rs`](../../src/storage_engines/sketch_db/query/decoders.rs) | +| Store/index | [`sketch_db/index/`](../../src/storage_engines/sketch_db/index/) | + +## Processing stages + +1. Decode OTLP without discarding resource, scope, metric, point labels, or + source timestamps needed for identity and freshness. +2. Resolve tenant and canonical series key to `sid`. +3. Locate the materialization in the active BackendPlan. +4. Validate producer, family, parameters, grouping, window, encoding, and + schema compatibility. +5. Apply full or delta sequencing to an isolated candidate state. +6. Commit the window/state atomically to the store. +7. Update readiness/freshness evidence and structured ingest telemetry. + +Each batch/item must have an explicit failure policy. A decoding error must not +silently convert an incompatible summary into a raw metric or another family. + +## Full and delta invariants + +- Full state identifies its checkpoint and replaces only the compatible base. +- Delta identifies plan, materialization, producer, window, sequence, and base. +- Duplicate delivery is idempotent. +- A missing/reordered/conflicting delta creates a visible gap. +- State after a gap remains unqueryable until a compatible full checkpoint. +- State from different plan versions or materialization contracts never merges. + +## SID behavior + +Always canonicalize metric name and identifying labels deterministically. +Sender-provided numeric IDs are shortcuts, not authority. Unknown or conflicting +IDs must be resolved from canonical identity evidence or rejected. + +## Freshness evidence + +Preserve the source sample/window timestamp separately from receive and commit +timestamps. The MVP freshness measurement starts at the source timestamp and +ends at the first queryable committed state; substituting ingestion wall time +understates lag. + +## Adding an encoding + +An encoding is supported only when its decoder, compatibility identity, +full/delta behavior, checkpoint recovery, capability advertisement, collector +parity, and corrupt-input tests land together. + +## Required tests + +- full payload for every MVP family; +- delta duplicate, gap, reorder, wrong base, and recovery checkpoint; +- wrong plan/materialization/family/parameters/window rejection; +- SID cache hit, unknown ID, and conflicting identity recovery; +- timestamps retained for freshness; and +- rejected input never changes queryable state. diff --git a/data_plane/docs/developer_docs/query-routing-and-readout.md b/data_plane/docs/developer_docs/query-routing-and-readout.md new file mode 100644 index 00000000..79b686f1 --- /dev/null +++ b/data_plane/docs/developer_docs/query-routing-and-readout.md @@ -0,0 +1,71 @@ +# Query routing and summary readout + +> Implementation status: partial; summary execution and fallback exist while +> BackendPlan replaces legacy routing and local query-shape logic. + +## Purpose + +This component converts a Prometheus-compatible request into a BackendPlan route, +checks readiness, executes the declared summary readout and remaining operators, +or invokes the explicit exact fallback. + +Design source: [Plan-aware query execution](../design_docs/query-execution.md). + +## Current code map + +| Responsibility | Current entry point | +| --- | --- | +| ASAP query engine | [`asap_query_engine/engine.rs`](../../src/query_engines/asap_query_engine/engine.rs) | +| Live summary-serving gate | [`asap_query_engine/live_serve.rs`](../../src/query_engines/asap_query_engine/live_serve.rs) | +| Planner-node readout | [`asap_query_engine/l4_readout.rs`](../../src/query_engines/asap_query_engine/l4_readout.rs) | +| Summary executor contract | [`summary_exec.rs`](../../src/query_engines/asap_query_engine/summary_exec.rs), [`summary_executor.rs`](../../src/query_engines/asap_query_engine/summary_executor.rs) | +| Legacy routing during migration | [`routing/backend_storage_routing.rs`](../../src/query_engines/routing/backend_storage_routing.rs) | +| Engine/fallback dispatch | [`routing/query_engine_routing.rs`](../../src/query_engines/routing/query_engine_routing.rs) | +| Freshness probes | [`routing/freshness_probe_cache.rs`](../../src/query_engines/routing/freshness_probe_cache.rs) | + +## Request flow + +1. Adapter produces canonical query, evaluation time/range, tenant, and accuracy + requirement. +2. Snapshot one active BackendPlan. +3. Match a route by canonical query capability and bound source—not by metric + name alone. +4. Resolve referenced materializations and required logical windows. +5. Check plan identity, coverage, watermark, delta continuity, and producer + completeness. +6. Execute the declared readout and remaining backend operators. +7. Align labels/timestamps and encode the Prometheus response. +8. If the plan selects exact fallback, forward the same logical request. + +## Result contract + +- Missing or extra series are not treated as zero or dropped. +- One response does not combine incompatible materializations/plan versions. +- Approximation metadata reflects the selected result guarantee. +- A partial interval is not returned as a complete answer. +- Unsupported/missing/stale state returns a typed miss/error or explicit exact + route; it never returns a plausible summary value. + +## Current migration boundary + +`BackendStorageRouting`, backend capability matching, and some lowering helpers +still contain local query-shape logic. BackendPlan is the target authority. +Do not add a new query-to-summary rule to these legacy paths; add logical +support to ASAPPlanner and consume its selected readout. + +## Adding a readout/operator + +1. Confirm ASAPPlanner represents its exact/approximate semantics and guarantee. +2. Add backend capability advertisement. +3. Map the canonical Planner node to one executor operation without replanning. +4. Define label, timestamp, scalar/vector, and partial-coverage behavior. +5. Add exact-baseline comparison for the same PromQL and logical interval. + +## Required tests + +- the three MVP aggregation shapes; +- matching and deliberately mismatched labels/timestamps; +- incomplete/stale windows and delta gaps; +- exact versus approximate accuracy handling; +- unsupported query fallback and fallback failure; and +- concurrent plan swap does not mix versions in one response. diff --git a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md new file mode 100644 index 00000000..e6359d05 --- /dev/null +++ b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md @@ -0,0 +1,84 @@ +# Summary storage and series identity + +> Implementation status: partial; the store/index and SID resolver exist, while +> plan/materialization lifecycle convergence remains target work. + +## Purpose + +The store maintains plan-compatible summary materializations by canonical +series/group and logical window. The SID registry provides stable metric-series +identity; it does not replace plan or materialization identity. + +Design sources: + +- [Summary storage](../../../docs/design_docs/summary-storage.md) +- [Series identity](../../../docs/design_docs/series-identity.md) + +## Current code map + +| Responsibility | Current entry point | +| --- | --- | +| Store/index and metadata | [`sketch_db/index/mod.rs`](../../src/storage_engines/sketch_db/index/mod.rs) | +| Epoch data layout | [`sketch_db/index/epoch_columnar.rs`](../../src/storage_engines/sketch_db/index/epoch_columnar.rs) | +| Summary payload/config types | [`sketch_db/data/mod.rs`](../../src/storage_engines/sketch_db/data/mod.rs) | +| Window/timeline query | [`sketch_db/query/`](../../src/storage_engines/sketch_db/query/) | +| Lifecycle/reconciliation | [`sketch_db/lifecycle/`](../../src/storage_engines/sketch_db/lifecycle/) | +| SID resolution | [`drivers/ingest/series_resolver.rs`](../../src/drivers/ingest/series_resolver.rs) | +| Persistence (future/non-MVP scope) | [`sketch_db/persistence/`](../../src/storage_engines/sketch_db/persistence/) | + +## Identity hierarchy + +```text +plan_id / plan_version + materialization_id + tenant + canonical series SID or reduction group + logical window + compatible summary state/checkpoint +``` + +`sid` identifies a raw metric series. Materialization identity adds source +binding, summary family/parameters, reduction/grouping, and window semantics. +Plan identity versions deployment. Never use one identity as a substitute for +another. + +## Store operations + +The component boundary should expose semantic operations rather than internal +maps: + +- register/stage a materialization contract; +- append or replace compatible full state; +- apply compatible delta state; +- resolve exact window coverage for a readout; +- mark state active, draining, expired, gapped, or rejected; and +- retire state only after readers/lateness/rollback no longer require it. + +Lookup returns complete state or a typed unavailability reason. It never +silently skips a missing group/window. + +## Concurrency and lifecycle + +- Registration and state append validate against one plan snapshot. +- Readers get stable metadata/state for the operation duration. +- Plan transition may keep old/new versions concurrently but never merges them. +- Eviction cannot remove required active state without first changing + readiness/routing. +- Persistence recovery must restore identity/compatibility before making parts + queryable. + +## Adding a stored family + +Follow [Adding a summary family](../../../docs/developer_docs/adding-summary-family.md). +Store work includes canonical parameter identity, payload validation, supported +merge/readout, accuracy metadata, full/delta encoding identity, and lifecycle +tests. A byte decoder alone is not store support. + +## Required tests + +- canonical labels yield stable SID independent of order; +- distinct tenants/label sets never collide; +- same SID across different materializations remains isolated; +- incompatible parameters/windows/versions never merge; +- exact coverage and missing-window failure; +- concurrent append/read/retire safety; and +- recovery never exposes state before compatible metadata. diff --git a/docs/README.md b/docs/README.md index bc1f9c36..3f3b3059 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,12 @@ - [Adding a summary family](developer_docs/adding-summary-family.md) - [Data-plane extension boundaries](../data_plane/docs/developer_docs/extension-points.md) +- [Planner adapter and physical compiler](../control_plane/docs/developer_docs/planner-and-physical-compiler.md) +- [Runtime plan publication](../control_plane/docs/developer_docs/runtime-plan-publication.md) +- [BackendPlan runtime](../data_plane/docs/developer_docs/backend-plan-runtime.md) +- [OTLP summary ingestion](../data_plane/docs/developer_docs/otlp-summary-ingestion.md) +- [Query routing and readout](../data_plane/docs/developer_docs/query-routing-and-readout.md) +- [Summary storage and series identity](../data_plane/docs/developer_docs/summary-storage-and-series-identity.md) - [Querying ASAP](../data_plane/docs/user_guide/querying-asap.md) ## Documentation ownership diff --git a/docs/design_docs/series-identity.md b/docs/design_docs/series-identity.md index a57de291..162977e4 100644 --- a/docs/design_docs/series-identity.md +++ b/docs/design_docs/series-identity.md @@ -5,6 +5,9 @@ > MVP relation: provides stable identity for ingestion, grouping, and result > labels across collector and backend boundaries. +Developer guide: +[Summary storage and series identity](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md). + ## TL;DR A series ID (`sid`) names one canonical metric series within a tenant and diff --git a/docs/design_docs/summary-storage.md b/docs/design_docs/summary-storage.md index 661be3fd..924a320d 100644 --- a/docs/design_docs/summary-storage.md +++ b/docs/design_docs/summary-storage.md @@ -4,6 +4,9 @@ > > MVP relation: stores the state needed for summary-backed query execution. +Developer guide: +[Summary storage and series identity](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md). + ## TL;DR Summary storage is a plan-aware materialized-state store. It accepts only state From 8127901ce86c5d10a70e1802e48372c0272e761a Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 28 Aug 2026 05:04:44 -0600 Subject: [PATCH 2/3] docs: define public developer extension interfaces --- .../planner-and-physical-compiler.md | 242 ++++++++++----- .../runtime-plan-publication.md | 293 +++++++++++------- .../developer_docs/backend-plan-runtime.md | 169 ++++++---- .../docs/developer_docs/extension-points.md | 160 ++++++---- .../developer_docs/otlp-summary-ingestion.md | 246 ++++++++++----- .../query-routing-and-readout.md | 208 ++++++++----- .../summary-storage-and-series-identity.md | 200 ++++++++---- docs/developer_docs/adding-summary-family.md | 158 +++++++--- 8 files changed, 1124 insertions(+), 552 deletions(-) diff --git a/control_plane/docs/developer_docs/planner-and-physical-compiler.md b/control_plane/docs/developer_docs/planner-and-physical-compiler.md index 40a5f611..cad96f75 100644 --- a/control_plane/docs/developer_docs/planner-and-physical-compiler.md +++ b/control_plane/docs/developer_docs/planner-and-physical-compiler.md @@ -1,110 +1,190 @@ -# Planner adapter and physical compiler +# Developing the Planner adapter and physical compiler -> Implementation status: target integration; current modules are migration -> substrate and do not replace ASAPPlanner ownership. +> Interface status: target public API. Existing migration modules must converge +> on this boundary. -## Purpose +## 1. Code architecture -This component consumes one ASAPPlanner-selected workload plan and compiles it -into matching collector and backend runtime plans. It must preserve Planner's -logical semantics while adding only ASAPQuery-backend-owned deployment choices. +The control plane has three public layers: -Design sources: +```text +PlanningRequest + | + v +PlannerAdapter ----------> SelectedLogicalPlan + | + v +PhysicalCompiler -------> CompiledPlanBundle + | | + v v + CollectorPlan BackendPlan +``` -- [ASAPPlanner integration](../asapplanner-integration.md) -- [Physical planning](../physical-planning.md) +- **Planner adapter** owns the typed call to ASAPPlanner. It supplies the whole + workload and receives one selected logical plan without copying Planner IR. +- **Physical compiler** adds backend-owned placement, windows, transport, and + runtime capabilities without changing logical semantics. +- **Plan bundle** is the only output passed to publication. CollectorPlan and + BackendPlan are created together and share identities. -## Component boundary +Logical query parsing, summary alternatives, guarantees, and candidate search +remain public ASAPPlanner interfaces. Runtime publication is documented in +[Runtime plan publication](runtime-plan-publication.md). -```text -workload + schema + logical constraints - | - v - ASAPPlanner adapter - | - selected workload DAG - | - v - physical compiler - / \ - v v - CollectorPlan BackendPlan +## 2. Public interfaces and definitions + +### Planner adapter + +```rust +pub trait PlannerAdapter { + type Error; + + fn select( + &self, + request: PlanningRequest, + ) -> Result; +} ``` -The adapter must use the pinned ASAPPlanner API directly. It must not recreate -Planner IR, query-to-summary rules, accuracy algebra, or candidate ranking in -backend-local types. +`PlanningRequest` is backend-owned request context around Planner's canonical +workload value: + +```rust +pub struct PlanningRequest { + pub workload: asap_planner::Workload, + pub schema: asap_planner::SchemaCatalog, + pub constraints: asap_planner::PlanningConstraints, + pub cost_inputs: asap_planner::CostInputs, + pub planner_revision: String, +} +``` -## Current code map +Input definitions: -| Responsibility | Current entry point | +| Field | Definition | | --- | --- | -| Workload pipeline coordination | [`pipeline.rs`](../../src/pipeline.rs) | -| Planner-facing workload/types during migration | [`workload.rs`](../../src/workload.rs), [`types_v2.rs`](../../src/types_v2.rs) | -| Physical allocation | [`physical/allocator.rs`](../../src/physical/allocator.rs) | -| Stage allocation and topology | [`physical/colored_dag/`](../../src/physical/colored_dag/) | -| Window realization | [`physical/window_fusion.rs`](../../src/physical/window_fusion.rs) | -| Runtime stage model and emitter | [`physical/colored_dag/emitter.rs`](../../src/physical/colored_dag/emitter.rs) | -| BackendPlan compilation | [`backend_plan/from_stage_config.rs`](../../src/backend_plan/from_stage_config.rs) | +| `workload` | Complete workload; shared queries must not be split into independent calls. | +| `schema` | Source/label/type information required to bind queries. | +| `constraints` | Accuracy and logical requirements supplied by the caller. | +| `cost_inputs` | Measured/declared logical cost inputs; unknown values stay unknown. | +| `planner_revision` | Immutable Planner build/revision used for reproducibility. | + +`SelectedLogicalPlan` wraps Planner's public selected post-ASAP workload plan +and correlation metadata; it does not define another DAG: + +```rust +pub struct SelectedLogicalPlan { + pub workload_plan: asap_planner::SelectedWorkloadPlan, + pub planner_revision: String, + pub query_ids: Vec, +} +``` -The legacy `intent_algebra`, `sketch_algebra`, and optimizer modules are -migration inputs, not a second canonical Planner implementation. New logical -semantics belong in ASAPPlanner. +Why this interface exists: it prevents adapters, protocols, and physical +planning from each implementing their own query-to-summary mapping. -## Inputs +### Physical compiler -The compile operation requires: +```rust +pub trait PhysicalCompiler { + type Error; -- the complete selected Planner workload DAG, with shared nodes intact; -- the immutable Planner revision; -- collector and backend capability snapshots; -- deployment topology and tenant boundaries; -- physical window, freshness, retention, and transmission policy; and -- runtime statistics and resource constraints used for physical placement. + fn compile( + &self, + selected: SelectedLogicalPlan, + environment: DeploymentEnvironment, + policy: RuntimePolicy, + ) -> Result; +} +``` + +```rust +pub struct DeploymentEnvironment { + pub topology: DeploymentTopology, + pub collectors: Vec, + pub backend: BackendTarget, + pub capability_snapshot_id: String, +} + +pub struct RuntimePolicy { + pub activation: Timestamp, + pub expiry: Option, + pub freshness: FreshnessPolicy, + pub retention: RetentionPolicy, + pub transmission: TransmissionPolicy, +} + +pub struct CompiledPlanBundle { + pub envelope: PlanEnvelope, + pub collector_plans: Vec, + pub backend_plan: BackendPlan, +} +``` + +Supporting public types: + +| Type | Definition | +| --- | --- | +| `DeploymentTopology` | Runtime stages, network relationships, and isolation boundaries available for placement. | +| `CollectorTarget` | Collector identity, edge assignment, endpoint reference, and advertised capability snapshot. | +| `BackendTarget` | Data-plane identity, endpoint reference, storage routes, and advertised capabilities. | +| `FreshnessPolicy` | Maximum readiness lag, watermark, and allowed-lateness requirements. | +| `RetentionPolicy` | Duration and lifecycle rules for active/draining materializations. | +| `TransmissionPolicy` | Allowed raw/full/delta modes, cadence, encoding, and checkpoint limits. | +| `PlanEnvelope` | Shared `plan_id`, `plan_version`, activation/expiry, backend compatibility, and Planner revision. | +| `CollectorPlan` | Versioned public YAML execution contract owned by ASAPCollector. | +| `BackendPlan` | Versioned public data-plane materialization/routing contract defined in this repository. | + +Output definitions: + +| Output | Definition | +| --- | --- | +| `envelope` | Shared plan/version/activation/compatibility identity. | +| `collector_plans` | One plan per targeted collector, following ASAPCollector's public CollectorPlan schema. | +| `backend_plan` | Matching data-plane materialization and routing contract. | -Missing capabilities or statistics stay unknown. Do not replace unknown values -with zero cost, exact accuracy, or universal support. +The compiler error must identify an unsupported capability, invalid placement, +window incompatibility, identity conflict, or invalid selected guarantee. It +must not silently substitute another logical summary. -## Outputs +Why this interface exists: one compile guarantees that producer and consumer +receive the same family, parameters, grouping, window, and materialization +identity. -One compile produces one shared plan envelope plus: +## 3. Adding and verifying functionality -- one CollectorPlan for each targeted collector; and -- one BackendPlan for the data plane. +### Add a deployment topology -Both sides must agree on materialization identity, family, parameters, -grouping, windows, representation, and transmission semantics. Generate them -from one in-memory compiled decision, never with independent reinterpretation. +1. Add a public `DeploymentTopology` variant and its required target fields. +2. Teach `PhysicalCompiler::compile` how selected operators can be placed on it. +3. Reject plans requiring an unavailable stage/capability. +4. Verify the output contains one complete CollectorPlan for every producer and + one BackendPlan referencing all produced materializations. -## Implementation invariants +Interpretation: a successful bundle means both runtime views are complete and +cross-consistent; it does not mean they have been activated. -- Preserve workload-wide sharing; do not flatten to independent metric rows. -- Treat Planner result guarantees and exact fallback as authoritative. -- Reject an operation if its assigned executor lacks any required capability. -- Keep logical range semantics separate from physical pane size. -- Keep aggregation placement separate from logical grouping. -- Permit delta only when both endpoints share sequencing/checkpoint semantics. -- Make plan/materialization identity deterministic from semantic content. +### Add a transmission mode -## Adding a physical decision +1. Extend public `TransmissionPolicy` and capability declarations. +2. Define representation, sequencing, checkpoint, and fallback requirements. +3. Emit the same compatibility identity into producer and consumer plans. +4. Verify unsupported endpoint combinations return a compile error. -When adding placement, window, representation, or transport behavior: +For delta, verify duplicate, missing, reordered, and recovery-checkpoint cases. -1. Confirm it does not change the selected logical result. -2. Add the capability input needed to make the decision safely. -3. Include every semantic effect in materialization or compatibility identity. -4. Emit the choice to both runtime plans where applicable. -5. Add a mismatch test showing incompatible plans fail before activation. +### Add a physical window policy -If the change selects a different summary or changes its guarantee, implement -it in ASAPPlanner instead. +1. Add the public policy variant with anchor, size, slide, and lateness. +2. Prove it covers the selected logical range without changing semantics. +3. Include it in materialization identity. +4. Verify generated collector/backend windows are identical and incompatible + query ranges fail compilation. -## Required tests +### Required output checks -- shared Planner nodes remain one logical materialization; -- collector and backend outputs carry identical semantic fields; -- unsupported capabilities fail closed; -- incompatible windows or delta semantics are rejected; -- deterministic input produces deterministic identities; and -- a three-query PromQL workload covers within-series, across-label, and combined - time-plus-label aggregation shapes. +- deterministic inputs produce deterministic plan/materialization identities; +- shared logical producers remain shared materializations; +- every referenced materialization has a producer and backend declaration; +- all query IDs remain traceable; and +- unsupported inputs return structured errors, never partial bundles. diff --git a/control_plane/docs/developer_docs/runtime-plan-publication.md b/control_plane/docs/developer_docs/runtime-plan-publication.md index 03456b75..701d2966 100644 --- a/control_plane/docs/developer_docs/runtime-plan-publication.md +++ b/control_plane/docs/developer_docs/runtime-plan-publication.md @@ -1,115 +1,180 @@ -# Runtime plan publication - -> Implementation status: target contract; transport exists, but semantic -> CollectorPlan application/reporting is not complete. - -## Purpose - -Plan publication stages and activates the matching CollectorPlan and BackendPlan -created by the physical compiler. Transport success is not semantic activation. - -Design sources: - -- [Physical planning](../physical-planning.md) -- [BackendPlan](../backend-plan.md) -- [ASAPCollector plan interface](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md) - -## Current code map - -| Responsibility | Current entry point | -| --- | --- | -| Runtime-specific emission | [`emit/mod.rs`](../../src/emit/mod.rs) | -| Collector/gateway/backend payloads | [`emit/stage_config.rs`](../../src/emit/stage_config.rs) | -| Backend publication | [`emit/backend_push.rs`](../../src/emit/backend_push.rs) | -| OpAMP collector delivery/status | [`opamp/mod.rs`](../../src/opamp/mod.rs) | -| BackendPlan schema | [`backend_plan/mod.rs`](../../src/backend_plan/mod.rs) | -| BackendPlan protobuf | [`proto/backend_plan.proto`](../../proto/backend_plan.proto) | - -## Publication sequence - -1. Validate both runtime plans against the capability snapshots used to compile - them. -2. Stage BackendPlan without routing production queries to it. -3. Deliver CollectorPlan through OpAMP to every required collector. -4. Require backend installation evidence and collector semantic-application - evidence for the same plan/version. -5. Activate query routing at the declared activation boundary. -6. Drain and retire the previous version after readers and lateness expire. - -Any partial failure leaves the previous unexpired plan authoritative. - -## Collector transport contract - -Do not duplicate the CollectorPlan schema here. The authoritative field and -validation definitions are in ASAPCollector's +# Developing runtime plan publication + +> Interface status: target public API. OpAMP transport exists today; semantic +> CollectorPlan application/reporting is still incomplete. + +## 1. Code architecture + +Publication begins only after physical compilation returns a complete bundle: + +```text +CompiledPlanBundle + | + v +PlanPublisher + | | + v v +CollectorClient BackendPlanClient + | | + v v +CollectorReport BackendPlanReport + \ / + v v + ActivationResult +``` + +`CollectorClient` is the ASAPQuery-side counterpart of ASAPCollector's +authoritative [`opamp-config-push.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). - -The backend publisher must preserve these corresponding requirements: - -- transport is OpAMP `AgentRemoteConfig`/`AgentConfigMap`; -- the exact entry name is `asap-collector-plan.yaml`; -- the body is the versioned YAML `CollectorPlan`, with - `content_type: application/yaml`; -- OpAMP `config_hash` identifies delivered bytes and is distinct from the - cross-runtime `metadata.plan_id`; -- CollectorPlan and BackendPlan share `plan_id`, `plan_version`, - `backend_compat`, and materialization identities; and -- an OpAMP `RemoteConfigStatus.APPLIED` response does not prove semantic - activation. - -The expected semantic application report uses the collector capability -`io.asap.collector.plan.v1`, message type `application_report`, and reports the -active plan/version, remote-config hash, backend compatibility, active -materializations, effective capability hash, timestamps, and structured -errors. Treat unknown or missing fields according to the versioned collector -contract rather than guessing defaults. - -## Evidence contract - -The publisher distinguishes: - -- transport acknowledgement: bytes reached an endpoint; -- validation acknowledgement: payload schema/capabilities were accepted; -- semantic application: the runtime reports the expected active plan and - materialization identities; and -- data evidence: emitted/ingested state carries those identities. - -Only semantic application plus compatible data evidence can make a plan -queryable. - -## Current implementation gap - -The corresponding ASAPCollector document records that current OpAMP handling -still applies a complete OTel Collector YAML, identifies it primarily by -`config_hash`, and reports `APPLIED` after syntactic validation/file write. It -does not yet provide the target CollectorPlan parser, atomic in-process apply, -or semantic application report. - -Backend code and tests must represent this honestly. Until both sides implement -the target contract, `APPLIED` is delivery evidence only and cannot satisfy the -MVP plan-application gate. - -## Versioning and retry - -- Re-sending identical `(plan_id, plan_version)` content is idempotent. -- Reusing that pair for different content is an error. -- Older or expired versions cannot replace a newer active version. -- Retry preserves activation and expiry timestamps. -- Rollback selects an explicitly retained compatible version; it does not edit - an active plan in place. - -## Failure handling - -Surface collector rejection, backend rejection, timeout, partial rollout, stale -status, and identity mismatch separately. Do not report a successful plan push -when only one runtime side applied it. - -## Required tests - -- idempotent repeated delivery; -- stale version rejection; -- collector-only and backend-only application remain inactive; -- activation succeeds only for matching identities; -- failed rollout retains the old route; -- rollback restores a complete prior pair; and -- status artifacts contain enough evidence for the MVP harness. +This repository does not redefine CollectorPlan fields. + +## 2. Public interfaces and definitions + +### Runtime clients + +```rust +pub trait CollectorPlanClient { + type Error; + + async fn stage( + &self, + target: CollectorTarget, + plan: CollectorPlan, + ) -> Result; +} + +pub trait BackendPlanClient { + type Error; + + async fn stage( + &self, + target: BackendTarget, + plan: BackendPlan, + ) -> Result; +} +``` + +Collector transport requirements come directly from the corresponding +ASAPCollector interface: + +- OpAMP `AgentRemoteConfig`/`AgentConfigMap`; +- exact entry name `asap-collector-plan.yaml`; +- YAML CollectorPlan with `content_type: application/yaml`; +- OpAMP `config_hash` identifies bytes, not cross-runtime plan semantics; and +- `RemoteConfigStatus.APPLIED` is delivery/application evidence, not semantic + activation evidence. + +### Application reports + +```rust +pub enum ApplicationStatus { + Rejected, + Staged, + Active, + Expired, + Failed, +} + +pub struct ApplicationError { + pub code: String, + pub path: String, + pub message: String, +} + +pub struct CollectorApplicationReport { + pub plan_id: String, + pub plan_version: u64, + pub backend_compat: String, + pub remote_config_hash: Vec, + pub status: ApplicationStatus, + pub active_materialization_ids: Vec, + pub effective_capability_hash: String, + pub observed_at: Timestamp, + pub activated_at: Option, + pub errors: Vec, +} + +pub struct BackendApplicationReport { + pub plan_id: String, + pub plan_version: u64, + pub backend_compat: String, + pub status: ApplicationStatus, + pub active_materialization_ids: Vec, + pub observed_at: Timestamp, + pub activated_at: Option, + pub errors: Vec, +} +``` + +The collector report corresponds to capability +`io.asap.collector.plan.v1`, message type `application_report`. Unknown report +versions or missing required fields are errors. + +Why reports are separate from transport acknowledgement: the MVP must prove the +runtime applied the intended semantic plan, not merely that bytes arrived. + +### Publisher + +```rust +pub trait PlanPublisher { + type Error; + + async fn publish( + &self, + bundle: CompiledPlanBundle, + ) -> Result; + + async fn rollback( + &self, + plan_id: &str, + plan_version: u64, + ) -> Result; +} + +pub struct ActivationResult { + pub plan_id: String, + pub plan_version: u64, + pub status: ApplicationStatus, + pub collector_reports: Vec, + pub backend_report: BackendApplicationReport, +} +``` + +`publish` returns `Active` only when every required runtime reports the same +plan/version/compatibility and expected materializations. Partial staging is an +error result and keeps the prior valid plan authoritative. + +## 3. Adding and verifying functionality + +### Add another collector transport + +1. Implement `CollectorPlanClient`; keep CollectorPlan unchanged. +2. Preserve plan identity separately from transport byte identity. +3. Map transport errors to structured client errors. +4. Verify identical re-delivery is idempotent and conflicting bytes for the same + plan/version are rejected. + +Interpretation: a successful `stage` report is not global activation; only +`PlanPublisher::publish` can return an active bundle. + +### Add an application status or report field + +1. Version the public report schema/capability. +2. Define required/optional behavior and compatibility. +3. Update collector and backend clients together. +4. Verify older readers reject unknown required semantics rather than defaulting. + +### Add rollback policy + +1. Select only a retained complete bundle through `rollback`. +2. Stage both runtime sides like a normal publication. +3. Verify the result reports the restored version and all materializations. +4. Verify failed rollback leaves the current active bundle unchanged. + +### Required output checks + +- reports match bundle identity and expected materialization sets; +- stale/expired/conflicting versions fail; +- collector-only or backend-only success never returns `Active`; +- report artifacts are machine-readable by the MVP harness; and +- post-activation emitted state carries the activated identities. diff --git a/data_plane/docs/developer_docs/backend-plan-runtime.md b/data_plane/docs/developer_docs/backend-plan-runtime.md index 9eaee8f4..9ec7bc9b 100644 --- a/data_plane/docs/developer_docs/backend-plan-runtime.md +++ b/data_plane/docs/developer_docs/backend-plan-runtime.md @@ -1,82 +1,127 @@ -# BackendPlan installation and runtime state +# Developing BackendPlan installation -> Implementation status: partial; atomic plan storage exists, while complete -> staging, lifecycle, and cross-runtime activation checks remain target work. +> Interface status: target public API. Atomic snapshot storage exists; complete +> staging/lifecycle/cross-runtime activation remains partial. -## Purpose +## 1. Code architecture -This component validates, stages, and atomically installs the BackendPlan -produced by the control plane. Query and ingest operations take stable snapshots -of the active plan; they do not observe a partially written plan. +```text +BackendPlan bytes + | + v +BackendPlanDecoder -> BackendPlanValidator -> BackendPlanRuntime + | + BackendPlanSnapshot + / \ + ingest query +``` -Design source: [BackendPlan](../../../control_plane/docs/backend-plan.md). +The decoder owns wire decoding, the validator owns semantic/capability checks, +and the runtime owns staged/active snapshots. Ingest and query components only +consume immutable snapshots; they do not mutate plans. + +## 2. Public interfaces and definitions + +```rust +pub trait BackendPlanDecoder { + type Error; + fn decode(&self, bytes: &[u8]) -> Result; +} + +pub trait BackendPlanValidator { + type Error; + fn validate( + &self, + plan: &BackendPlan, + capabilities: &BackendCapabilities, + ) -> Result; +} +``` -## Current code map +`ValidatedBackendPlan` must be constructible only through validation. It proves +schema/lifecycle ordering, unique identities, resolved routes, supported +families/parameters/windows/readouts, and compatible result guarantees. + +```rust +pub struct ValidatedBackendPlan { + pub plan: BackendPlan, + pub capability_hash: String, + pub validated_at: Timestamp, +} + +pub struct BackendCapabilities { + pub capability_hash: String, + pub ingest: Vec, + pub readouts: Vec, + pub storage: Vec, +} + +pub struct BackendPlanSnapshot { + pub plan: Arc, + pub status: PlanRuntimeStatus, + pub installed_at: Timestamp, +} + +pub enum PlanRuntimeStatus { + Staged, + Active, + Draining, + Expired, +} +``` + +Capability entry definitions: -| Responsibility | Current entry point | +| Type | Definition | | --- | --- | -| Shared BackendPlan types/decoder | [`control_plane::backend_plan`](../../../control_plane/src/backend_plan/mod.rs) | -| Atomic runtime holder | [`HotReloadBackendPlan`](../../src/storage_engines/types/hot_reload_config.rs) | -| HTTP installation surface | [`drivers/query/servers/http.rs`](../../src/drivers/query/servers/http.rs) | -| Plan-aware readout | [`l4_readout.rs`](../../src/query_engines/asap_query_engine/l4_readout.rs) | +| `IngestCapability` | Supported family, algorithm, parameters, encoding version, and full/delta semantics. | +| `ReadoutCapability` | Supported Planner readout/operator and guarantee kinds. | +| `StorageCapability` | Supported materialization representation, merge, window, retention, and durability behavior. | -The current `HotReloadBackendPlan` is an `ArcSwap` holder. Atomic pointer swap -is necessary but not sufficient: target validation, version ordering, staged -activation, expiry, and matching collector evidence belong around that holder. +```rust +pub trait BackendPlanRuntime: Send + Sync { + type Error; -## Installation pipeline + fn stage(&self, plan: ValidatedBackendPlan) + -> Result; -```text -received BackendPlan - | -decode and closed-schema validation - | -capability + state-compatibility validation - | -stage materializations/routes - | -match CollectorPlan application evidence - | -atomic activation -``` + fn activate(&self, plan_id: &str, plan_version: u64) + -> Result; -Validation must complete before replacing the active snapshot. Readers already -holding the previous snapshot may finish, but a single request must not mix plan -versions. + fn snapshot(&self) -> BackendPlanSnapshot; -## Required validation + fn retire(&self, plan_id: &str, plan_version: u64) + -> Result; +} +``` -- plan/version/lifecycle ordering; -- immutable content for repeated `(plan_id, plan_version)`; -- unique materialization and route identity; -- known family, algorithm, parameters, grouping, windows, and readout; -- backend capability support for ingest, merge, representation, and readout; -- route references resolve to declared materializations or exact fallback; -- result guarantees meet the route's declared requirement; and -- collector compatibility/evidence matches before activation. +Why these interfaces exist: decoding, validation, and activation have different +failure semantics. A decoded plan must never become queryable before validation +and matching collector evidence. -Reject the whole plan on failure. Do not drop one invalid route and activate the -rest unless a future schema explicitly defines partial activation semantics. +## 3. Adding and verifying functionality -## Reader contract +### Add a BackendPlan field -Ingest and query entry points obtain one active-plan snapshot at operation -start. They pass it through validation/routing rather than consulting global -state repeatedly. State emitted under another version is not accepted merely -because its summary bytes decode. +1. Add it to the public versioned wire/domain structure. +2. Define requiredness, identity impact, and compatibility behavior. +3. Validate it in `BackendPlanValidator`. +4. Expose it through immutable `BackendPlanSnapshot` to its consumer. +5. Verify missing/unknown/incompatible values fail before `stage`. -## Current migration boundary +### Add a runtime lifecycle state -`StreamingConfig`, `BackendStorageRouting`, and BackendPlan currently coexist. -Treat the first two as legacy runtime inputs being absorbed into BackendPlan. -New semantic fields should be added to the versioned plan contract rather than -creating another independently hot-reloaded configuration source. +1. Extend `PlanRuntimeStatus` with allowed transitions. +2. Define whether ingest/query may use the state. +3. Return the effective state through `BackendApplicationReport`. +4. Verify invalid transitions do not change `snapshot()`. -## Required tests +### Interpret and verify output -- malformed and unknown schema values fail without swapping; -- older/conflicting versions fail; -- concurrent readers see one complete version; -- routes cannot reference missing materializations; -- plan activation waits for matching collector evidence; and -- rollback restores a complete retained snapshot. +- A `ValidatedBackendPlan` means the plan is deployable by this backend, not + active. +- A `Staged` report means resources/routes are prepared, not queryable. +- An `Active` report must match the requested plan/version and materializations. +- One request must observe one `BackendPlanSnapshot`, including during swap. +- Re-delivery of identical content is idempotent; conflicting content for the + same identity fails. diff --git a/data_plane/docs/developer_docs/extension-points.md b/data_plane/docs/developer_docs/extension-points.md index 9cc577d9..2829644c 100644 --- a/data_plane/docs/developer_docs/extension-points.md +++ b/data_plane/docs/developer_docs/extension-points.md @@ -1,53 +1,107 @@ -# Data-plane extension boundaries - -> Status: active -> -> MVP relation: Prometheus HTTP and the configured exact fallback are required; -> additional protocols and fallback systems are future extensions. - -## TL;DR - -The data plane separates network transport, request/response adaptation, -plan-aware execution, and exact fallback. An extension implements one boundary -without duplicating planning or bypassing BackendPlan validation. - -## Protocol server - -A protocol server owns network concerns: endpoints, authentication context, -request limits, cancellation, and transport errors. It hands a request to a -protocol adapter and returns the adapter's response. - -It does not parse Planner IR, select a summary, access summary storage directly, -or decide when fallback is allowed. - -## Protocol adapter - -An adapter converts a protocol request into the data plane's canonical query -request and converts the canonical result back into the protocol response. -Prometheus label and timestamp semantics must survive both conversions. - -An adapter may report that a language feature cannot be represented, but it -must not approximate or rewrite an unsupported query on its own. - -## Fallback client - -A fallback client executes the canonical query against the exact backend named -by BackendPlan. It preserves the logical evaluation time, range, tenant, and -error response. - -Fallback is invoked by plan-aware routing. A fallback client must not turn a -remote error into an empty successful result. - -## Adding an extension - -An extension is complete when it demonstrates: - -- request and response semantic round trips; -- cancellation, timeout, and error propagation; -- tenant and authentication context preservation; -- plan-aware routing rather than direct store access; -- no silent fallback or approximation; and -- integration coverage with one successful and one failing request. - -Implementation locations and trait signatures are intentionally left to the -code and API documentation, where they can evolve without changing this design. +# Developing protocol and fallback extensions + +> Interface status: public extension boundary. Concrete trait names may migrate +> toward the canonical interfaces below; private server helpers are not API. + +## 1. Code architecture + +```text +network request -> ProtocolServer -> ProtocolAdapter -> QueryService + | + ExactQueryClient +``` + +- `ProtocolServer` owns transport, authentication context, limits, timeout, and + cancellation. +- `ProtocolAdapter` converts protocol-specific data to/from canonical query + structures. +- `QueryService` performs plan-aware execution. +- `ExactQueryClient` is called only for an explicit fallback route. + +## 2. Public interfaces and definitions + +```rust +pub trait ProtocolAdapter: Send + Sync { + type Request; + type Response; + type Error; + + fn decode(&self, request: Self::Request) + -> Result; + + fn encode(&self, response: QueryResponse) + -> Result; + + fn encode_error(&self, error: QueryError) -> Self::Response; +} +``` + +```rust +pub trait ProtocolServer { + type Error; + async fn serve(&self, service: Arc) -> Result<(), Self::Error> + where + S: QueryService + Send + Sync + 'static; +} +``` + +```rust +pub trait ExactQueryClient: Send + Sync { + type Error; + async fn execute_exact(&self, request: &QueryRequest) + -> Result; + + fn capabilities(&self) -> ExactBackendCapabilities; +} + +pub struct ExactBackendCapabilities { + pub backend_id: String, + pub query_languages: Vec, + pub supports_instant: bool, + pub supports_range: bool, + pub maximum_range: Option, +} +``` + +`QueryRequest` and `QueryResponse` are defined in +[Query routing and readout](query-routing-and-readout.md). They preserve tenant, +query language/expression, logical evaluation range, requested accuracy, +result labels/timestamps/type, source, guarantee, and coverage. + +Why these interfaces exist: transport/protocol extensions cannot bypass +BackendPlan routing or directly access summary storage, and fallback backends +cannot silently reinterpret a request. + +## 3. Adding and verifying functionality + +### Add a protocol adapter + +1. Implement `ProtocolAdapter` for its request/response types. +2. Map every supported evaluation-time/range and tenant field. +3. Preserve Prometheus label/timestamp/result/error semantics where applicable. +4. Verify decode→canonical→encode round trips for success and error cases. + +### Add a protocol server + +1. Implement `ProtocolServer` and inject only the public `QueryService`. +2. Propagate cancellation, timeout, authentication, and request limits. +3. Never call `SummaryStore` or an exact client directly. +4. Verify cancelled requests stop downstream work and transport errors map + through `encode_error`. + +### Add an exact fallback backend + +1. Implement `ExactQueryClient` and declare `ExactBackendCapabilities`. +2. Forward the canonical logical range and tenant unchanged. +3. Return exact `QueryResponse` or a visible error. +4. Verify unsupported capability and remote failure do not return an empty + successful result. + +### Interpret and verify output + +- Adapter output is canonical input, not a routing decision. +- Server success means the response was transported, not that it was + summary-backed. +- Inspect `QueryResponse.source` to distinguish summary and exact fallback. +- End-to-end tests must include one supported request, one explicit fallback, + one malformed request, and one backend failure. diff --git a/data_plane/docs/developer_docs/otlp-summary-ingestion.md b/data_plane/docs/developer_docs/otlp-summary-ingestion.md index a56b692a..c8d40723 100644 --- a/data_plane/docs/developer_docs/otlp-summary-ingestion.md +++ b/data_plane/docs/developer_docs/otlp-summary-ingestion.md @@ -1,79 +1,181 @@ -# OTLP summary ingestion +# Developing OTLP summary ingestion + +> Interface status: target public API. OTLP decoding, SID resolution, and +> summary handling exist; complete BackendPlan-gated validation is partial. + +## 1. Code architecture + +```text +OTLP request + | + v +SummaryDecoder -> SeriesIdentityResolver -> SummaryValidator + | + v + SummaryStateApplier + | + v + SummaryStore +``` + +Transport decoding is separate from semantic validation. No decoder is allowed +to append directly to storage or choose a summary family. + +## 2. Public interfaces and definitions + +```rust +pub trait SummaryDecoder { + type Error; + fn decode(&self, request: OtlpMetricsRequest) + -> Result, Self::Error>; +} + +pub struct ReceivedSummary { + pub tenant: String, + pub resource: AttributeSet, + pub scope: InstrumentationScope, + pub metric_name: String, + pub attributes: AttributeSet, + pub source_timestamp: Timestamp, + pub envelope: SummaryEnvelope, +} +``` + +`SummaryEnvelope` contains plan/materialization/producer/window identity, +family/parameters/encoding, full-or-delta metadata, and payload bytes. Required +identity cannot be inferred from metric-name suffixes. + +```rust +pub struct SummaryEnvelope { + pub plan_id: String, + pub plan_version: u64, + pub materialization_id: String, + pub producer_id: String, + pub window: LogicalWindow, + pub family: SummaryFamily, + pub algorithm: SummaryAlgorithm, + pub parameters: SummaryParameters, + pub encoding: SummaryEncoding, + pub frame: SummaryFrame, + pub payload: Bytes, +} + +pub enum SummaryFrame { + Full { checkpoint_id: String }, + Delta { + base_checkpoint_id: String, + sequence: u64, + }, +} +``` + +```rust +pub trait SeriesIdentityResolver { + type Error; + fn resolve(&self, key: CanonicalSeriesKey) + -> Result; +} + +pub struct CanonicalSeriesKey { + pub tenant: String, + pub metric_name: String, + pub identifying_labels: BTreeMap, +} + +pub struct ResolvedSeries { + pub sid: u64, + pub canonical_key: CanonicalSeriesKey, + pub namespace_version: String, +} +``` + +```rust +pub trait SummaryValidator { + type Error; + fn validate( + &self, + received: ReceivedSummary, + plan: &BackendPlanSnapshot, + series: ResolvedSeries, + ) -> Result; +} + +pub trait SummaryStateApplier { + type Error; + fn apply(&self, summary: ValidatedSummary) + -> Result; +} + +pub struct ValidatedSummary { + pub received: ReceivedSummary, + pub series: ResolvedSeries, + pub materialization: ValidatedMaterialization, +} + +pub struct ValidatedMaterialization { + pub plan_id: String, + pub plan_version: u64, + pub materialization_id: String, + pub compatibility_fingerprint: String, +} + +pub struct IngestResult { + pub disposition: IngestDisposition, + pub plan_id: String, + pub materialization_id: String, + pub sid: u64, + pub window: LogicalWindow, + pub queryable_at: Option, +} + +pub enum IngestDisposition { + AppliedFull, + AppliedDelta, + Duplicate, + Rejected, + AwaitingCheckpoint, +} +``` + +Supporting type definitions: + +| Type | Definition | +| --- | --- | +| `OtlpMetricsRequest` | Decoded public OTLP ExportMetricsServiceRequest. | +| `AttributeSet` | Canonically typed OTel attributes with no identity-relevant loss. | +| `InstrumentationScope` | OTel scope name/version/schema identifying the producer library. | +| `LogicalWindow` | Start/end plus window identity used by plan, state, and query coverage. | +| `SummaryEncoding` | Versioned state representation shared by collector/backend capabilities. | -> Implementation status: partial; OTLP decoding, SID resolution, and summary -> handling exist, while full BackendPlan-gated validation remains target work. +Why these interfaces exist: each stage can reject invalid data without changing +queryable state, and `IngestResult` gives the MVP harness unambiguous evidence. -## Purpose +## 3. Adding and verifying functionality -The ingestion component receives OTel metric payloads from ASAPCollector, -resolves series identity, validates summary state against the active -BackendPlan, applies full/delta semantics, and appends queryable windows to -summary storage. +### Add an OTLP summary encoding -Design sources: +1. Extend public `SummaryEnvelope` encoding/version definitions. +2. Implement `SummaryDecoder` without applying state. +3. Add compatibility validation against BackendPlan/capabilities. +4. Implement full/delta application through `SummaryStateApplier`. +5. Verify corrupt bytes return an error and do not change storage. -- [Plan-aware query execution](../design_docs/query-execution.md) -- [Summary storage](../../../docs/design_docs/summary-storage.md) -- [Series identity](../../../docs/design_docs/series-identity.md) +### Add a delta-capable family -## Current code map +Define base/checkpoint, sequence scope, duplicate handling, gap behavior, and +recovery full state. Verify `AppliedDelta`, `Duplicate`, and +`AwaitingCheckpoint` are distinguishable outputs for reorder/gap tests. -| Responsibility | Current entry point | -| --- | --- | -| OTLP receivers and decoding | [`drivers/ingest/otel.rs`](../../src/drivers/ingest/otel.rs) | -| Canonical SID resolution | [`drivers/ingest/series_resolver.rs`](../../src/drivers/ingest/series_resolver.rs) | -| Full/delta evaluation helpers | [`sketch_db/query/delta_apply.rs`](../../src/storage_engines/sketch_db/query/delta_apply.rs) | -| Summary decoders | [`sketch_db/query/decoders.rs`](../../src/storage_engines/sketch_db/query/decoders.rs) | -| Store/index | [`sketch_db/index/`](../../src/storage_engines/sketch_db/index/) | - -## Processing stages - -1. Decode OTLP without discarding resource, scope, metric, point labels, or - source timestamps needed for identity and freshness. -2. Resolve tenant and canonical series key to `sid`. -3. Locate the materialization in the active BackendPlan. -4. Validate producer, family, parameters, grouping, window, encoding, and - schema compatibility. -5. Apply full or delta sequencing to an isolated candidate state. -6. Commit the window/state atomically to the store. -7. Update readiness/freshness evidence and structured ingest telemetry. - -Each batch/item must have an explicit failure policy. A decoding error must not -silently convert an incompatible summary into a raw metric or another family. - -## Full and delta invariants - -- Full state identifies its checkpoint and replaces only the compatible base. -- Delta identifies plan, materialization, producer, window, sequence, and base. -- Duplicate delivery is idempotent. -- A missing/reordered/conflicting delta creates a visible gap. -- State after a gap remains unqueryable until a compatible full checkpoint. -- State from different plan versions or materialization contracts never merges. - -## SID behavior - -Always canonicalize metric name and identifying labels deterministically. -Sender-provided numeric IDs are shortcuts, not authority. Unknown or conflicting -IDs must be resolved from canonical identity evidence or rejected. - -## Freshness evidence - -Preserve the source sample/window timestamp separately from receive and commit -timestamps. The MVP freshness measurement starts at the source timestamp and -ends at the first queryable committed state; substituting ingestion wall time -understates lag. - -## Adding an encoding - -An encoding is supported only when its decoder, compatibility identity, -full/delta behavior, checkpoint recovery, capability advertisement, collector -parity, and corrupt-input tests land together. - -## Required tests - -- full payload for every MVP family; -- delta duplicate, gap, reorder, wrong base, and recovery checkpoint; -- wrong plan/materialization/family/parameters/window rejection; -- SID cache hit, unknown ID, and conflicting identity recovery; -- timestamps retained for freshness; and -- rejected input never changes queryable state. +### Add series identity behavior + +Add canonical input fields to `CanonicalSeriesKey`, never to the numeric SID +alone. Verify label-order independence, tenant isolation, cached-ID conflict +recovery, and stable namespace reporting. + +### Interpret and verify output + +- `Applied*` means compatible state was committed. +- `Duplicate` means idempotent replay with no second mutation. +- `AwaitingCheckpoint` means a visible delta gap and non-queryable state. +- `queryable_at` is populated only when coverage/readiness is satisfied. +- Freshness uses `source_timestamp -> queryable_at`, not receive time. diff --git a/data_plane/docs/developer_docs/query-routing-and-readout.md b/data_plane/docs/developer_docs/query-routing-and-readout.md index 79b686f1..88b59801 100644 --- a/data_plane/docs/developer_docs/query-routing-and-readout.md +++ b/data_plane/docs/developer_docs/query-routing-and-readout.md @@ -1,71 +1,139 @@ -# Query routing and summary readout - -> Implementation status: partial; summary execution and fallback exist while -> BackendPlan replaces legacy routing and local query-shape logic. - -## Purpose - -This component converts a Prometheus-compatible request into a BackendPlan route, -checks readiness, executes the declared summary readout and remaining operators, -or invokes the explicit exact fallback. - -Design source: [Plan-aware query execution](../design_docs/query-execution.md). - -## Current code map - -| Responsibility | Current entry point | +# Developing query routing and summary readout + +> Interface status: target public API. Summary execution and fallback exist; +> BackendPlan is still replacing legacy routing and local query-shape logic. + +## 1. Code architecture + +```text +protocol request -> QueryAdapter -> QueryService + | + BackendPlanSnapshot + / \ + SummaryReader ExactQueryClient + \ / + QueryResponse +``` + +The adapter owns protocol conversion. `QueryService` owns plan-aware route +selection. `SummaryReader` executes an already selected readout. The exact +client executes only explicit fallback routes. + +## 2. Public interfaces and definitions + +```rust +pub struct QueryRequest { + pub tenant: String, + pub language: QueryLanguage, + pub expression: String, + pub evaluation: EvaluationRange, + pub requested_accuracy: AccuracyRequirement, +} + +pub struct EvaluationRange { + pub start: Timestamp, + pub end: Timestamp, + pub step: Option, +} +``` + +```rust +pub trait QueryService { + type Error; + async fn execute(&self, request: QueryRequest) + -> Result; +} + +pub trait SummaryReader { + type Error; + fn read( + &self, + request: &QueryRequest, + route: &SummaryRoute, + plan: &BackendPlanSnapshot, + ) -> Result; +} + +pub struct SummaryRoute { + pub query_id: String, + pub materialization_ids: Vec, + pub readout: ReadoutSpec, + pub required_guarantee: AccuracyRequirement, +} + +pub struct SummaryReadout { + pub result: PrometheusResult, + pub guarantee: ResultGuarantee, + pub coverage: LogicalCoverage, +} + +pub trait ExactQueryClient { + type Error; + async fn execute_exact(&self, request: &QueryRequest) + -> Result; +} +``` + +Supporting public type definitions: + +| Type | Definition | | --- | --- | -| ASAP query engine | [`asap_query_engine/engine.rs`](../../src/query_engines/asap_query_engine/engine.rs) | -| Live summary-serving gate | [`asap_query_engine/live_serve.rs`](../../src/query_engines/asap_query_engine/live_serve.rs) | -| Planner-node readout | [`asap_query_engine/l4_readout.rs`](../../src/query_engines/asap_query_engine/l4_readout.rs) | -| Summary executor contract | [`summary_exec.rs`](../../src/query_engines/asap_query_engine/summary_exec.rs), [`summary_executor.rs`](../../src/query_engines/asap_query_engine/summary_executor.rs) | -| Legacy routing during migration | [`routing/backend_storage_routing.rs`](../../src/query_engines/routing/backend_storage_routing.rs) | -| Engine/fallback dispatch | [`routing/query_engine_routing.rs`](../../src/query_engines/routing/query_engine_routing.rs) | -| Freshness probes | [`routing/freshness_probe_cache.rs`](../../src/query_engines/routing/freshness_probe_cache.rs) | - -## Request flow - -1. Adapter produces canonical query, evaluation time/range, tenant, and accuracy - requirement. -2. Snapshot one active BackendPlan. -3. Match a route by canonical query capability and bound source—not by metric - name alone. -4. Resolve referenced materializations and required logical windows. -5. Check plan identity, coverage, watermark, delta continuity, and producer - completeness. -6. Execute the declared readout and remaining backend operators. -7. Align labels/timestamps and encode the Prometheus response. -8. If the plan selects exact fallback, forward the same logical request. - -## Result contract - -- Missing or extra series are not treated as zero or dropped. -- One response does not combine incompatible materializations/plan versions. -- Approximation metadata reflects the selected result guarantee. -- A partial interval is not returned as a complete answer. -- Unsupported/missing/stale state returns a typed miss/error or explicit exact - route; it never returns a plausible summary value. - -## Current migration boundary - -`BackendStorageRouting`, backend capability matching, and some lowering helpers -still contain local query-shape logic. BackendPlan is the target authority. -Do not add a new query-to-summary rule to these legacy paths; add logical -support to ASAPPlanner and consume its selected readout. - -## Adding a readout/operator - -1. Confirm ASAPPlanner represents its exact/approximate semantics and guarantee. -2. Add backend capability advertisement. -3. Map the canonical Planner node to one executor operation without replanning. -4. Define label, timestamp, scalar/vector, and partial-coverage behavior. -5. Add exact-baseline comparison for the same PromQL and logical interval. - -## Required tests - -- the three MVP aggregation shapes; -- matching and deliberately mismatched labels/timestamps; -- incomplete/stale windows and delta gaps; -- exact versus approximate accuracy handling; -- unsupported query fallback and fallback failure; and -- concurrent plan swap does not mix versions in one response. +| `QueryLanguage` | Language identifier; MVP value is PromQL. | +| `AccuracyRequirement` | Exact, epsilon, or epsilon-delta constraint requested for the result. | +| `ReadoutSpec` | Planner-selected operation and typed parameters applied to maintained state. | +| `PrometheusResult` | Matrix/vector/scalar/string result with labels, timestamps, values, warnings, and errors. | +| `ResultGuarantee` | Effective exact/approximate guarantee of the returned result. | +| `LogicalCoverage` | Requested and actually covered time intervals plus readiness timestamp. | +| `QueryError` | Typed parse, unsupported, inactive-plan, missing/stale/gapped state, or exact-backend failure. | + +```rust +pub struct QueryResponse { + pub result: PrometheusResult, + pub source: QuerySource, + pub guarantee: ResultGuarantee, + pub coverage: LogicalCoverage, + pub plan_id: Option, + pub materialization_ids: Vec, +} + +pub enum QuerySource { + Summary, + ExactFallback, +} +``` + +Why these interfaces exist: protocol code cannot bypass plan/readiness checks, +and callers can interpret whether an answer is summary-backed or exact with its +coverage and guarantee. + +## 3. Adding and verifying functionality + +### Add a readout/operator + +1. Add the logical semantics and guarantee to ASAPPlanner. +2. Extend public backend readout capability and BackendPlan route types. +3. Implement it through `SummaryReader`; do not parse and choose a family again. +4. Return labels/timestamps/result type through `PrometheusResult`. +5. Compare with the exact backend over identical series and logical range. + +### Add a protocol adapter + +Convert protocol inputs to `QueryRequest` and `QueryResponse` back to protocol +output. Verify tenant, evaluation timestamps, labels, result type, errors, and +accuracy metadata round-trip unchanged. + +### Add an exact backend + +Implement `ExactQueryClient`, preserving the complete `QueryRequest`. Verify +remote failures remain errors and are not successful empty vectors. + +### Interpret and verify output + +- `QuerySource::Summary` requires active plan/materialization IDs and complete + coverage. +- `QuerySource::ExactFallback` must satisfy exact semantics and carry no false + summary guarantee. +- Missing/additional labels or timestamps are validation failures. +- Partial/stale/gapped summary state must not return a successful complete + `QueryResponse`. +- A plan swap during execution must not mix identities in one response. diff --git a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md index e6359d05..218cdfd4 100644 --- a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md +++ b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md @@ -1,84 +1,158 @@ -# Summary storage and series identity +# Developing summary storage and series identity -> Implementation status: partial; the store/index and SID resolver exist, while -> plan/materialization lifecycle convergence remains target work. +> Interface status: target public API. Store/index and SID resolution exist; +> plan/materialization lifecycle convergence is partial. -## Purpose +## 1. Code architecture -The store maintains plan-compatible summary materializations by canonical -series/group and logical window. The SID registry provides stable metric-series -identity; it does not replace plan or materialization identity. +```text +CanonicalSeriesKey -> SeriesRegistry -> SeriesId + | +ValidatedMaterialization ----------------+ + | + v + SummaryStore + write / coverage / read / retire +``` -Design sources: +The series registry owns only canonical metric-series identity. The summary +store owns materialization/group/window state. Plan, materialization, and SID +identities remain distinct. -- [Summary storage](../../../docs/design_docs/summary-storage.md) -- [Series identity](../../../docs/design_docs/series-identity.md) +## 2. Public interfaces and definitions -## Current code map +```rust +pub trait SeriesRegistry: Send + Sync { + type Error; -| Responsibility | Current entry point | -| --- | --- | -| Store/index and metadata | [`sketch_db/index/mod.rs`](../../src/storage_engines/sketch_db/index/mod.rs) | -| Epoch data layout | [`sketch_db/index/epoch_columnar.rs`](../../src/storage_engines/sketch_db/index/epoch_columnar.rs) | -| Summary payload/config types | [`sketch_db/data/mod.rs`](../../src/storage_engines/sketch_db/data/mod.rs) | -| Window/timeline query | [`sketch_db/query/`](../../src/storage_engines/sketch_db/query/) | -| Lifecycle/reconciliation | [`sketch_db/lifecycle/`](../../src/storage_engines/sketch_db/lifecycle/) | -| SID resolution | [`drivers/ingest/series_resolver.rs`](../../src/drivers/ingest/series_resolver.rs) | -| Persistence (future/non-MVP scope) | [`sketch_db/persistence/`](../../src/storage_engines/sketch_db/persistence/) | + fn resolve(&self, key: CanonicalSeriesKey) + -> Result; -## Identity hierarchy + fn lookup(&self, sid: u64, namespace_version: &str) + -> Result, Self::Error>; +} +``` -```text -plan_id / plan_version - materialization_id - tenant + canonical series SID or reduction group - logical window - compatible summary state/checkpoint +`CanonicalSeriesKey` and `ResolvedSeries` are defined by the ingestion public +interface. `resolve` is idempotent. A sender-provided SID never overrides a +conflicting canonical key. + +```rust +pub struct MaterializationKey { + pub plan_id: String, + pub plan_version: u64, + pub materialization_id: String, + pub tenant: String, + pub group: MaterializationGroup, + pub window: LogicalWindow, +} + +pub enum MaterializationState { + Staged, + Queryable, + Gapped, + Draining, + Expired, + Rejected, +} +``` + +```rust +pub trait SummaryStore: Send + Sync { + type Error; + + fn register(&self, contract: ValidatedMaterialization) + -> Result; + + fn apply(&self, update: ValidatedSummary) + -> Result; + + fn coverage(&self, request: CoverageRequest) + -> Result; + + fn read(&self, request: SummaryReadRequest) + -> Result; + + fn retire(&self, materialization_id: &str, policy: RetirementPolicy) + -> Result; +} + +pub struct CoverageRequest { + pub plan: BackendPlanSnapshot, + pub materialization_ids: Vec, + pub range: EvaluationRange, +} + +pub struct SummaryReadRequest { + pub coverage: LogicalCoverage, + pub route: SummaryRoute, +} + +pub struct SummaryReadResult { + pub states: Vec, + pub coverage: LogicalCoverage, +} + +pub struct RetirementPolicy { + pub drain_until: Timestamp, + pub retain_for_rollback_until: Option, +} +``` + +```rust +pub struct StoreWriteResult { + pub key: MaterializationKey, + pub state: MaterializationState, + pub disposition: IngestDisposition, +} + +pub enum CoverageResult { + Complete(LogicalCoverage), + Missing(Vec), + Stale { newest_source_timestamp: Timestamp }, + Gapped { producer: String, expected_sequence: u64 }, + Incompatible { reason: String }, +} ``` -`sid` identifies a raw metric series. Materialization identity adds source -binding, summary family/parameters, reduction/grouping, and window semantics. -Plan identity versions deployment. Never use one identity as a substitute for -another. +Supporting types `ValidatedMaterialization`, `ValidatedSummary`, and +`IngestDisposition` are defined by +[OTLP summary ingestion](otlp-summary-ingestion.md). `EvaluationRange`, +`SummaryRoute`, and `LogicalCoverage` are defined by +[Query routing and readout](query-routing-and-readout.md). -## Store operations +Why these interfaces exist: callers receive typed completeness/failure rather +than interpreting an empty collection as “no data,” and storage cannot accept +unvalidated summary bytes. -The component boundary should expose semantic operations rather than internal -maps: +## 3. Adding and verifying functionality -- register/stage a materialization contract; -- append or replace compatible full state; -- apply compatible delta state; -- resolve exact window coverage for a readout; -- mark state active, draining, expired, gapped, or rejected; and -- retire state only after readers/lateness/rollback no longer require it. +### Add a summary family to storage -Lookup returns complete state or a typed unavailability reason. It never -silently skips a missing group/window. +1. Extend public materialization capability/contract types. +2. Define canonical parameters and representation compatibility. +3. Accept only `ValidatedSummary` through `SummaryStore::apply`. +4. Implement merge/read behavior through public result types. +5. Verify incompatible family/parameters/windows never merge. -## Concurrency and lifecycle +### Add a storage backend -- Registration and state append validate against one plan snapshot. -- Readers get stable metadata/state for the operation duration. -- Plan transition may keep old/new versions concurrently but never merges them. -- Eviction cannot remove required active state without first changing - readiness/routing. -- Persistence recovery must restore identity/compatibility before making parts - queryable. +Implement `SummaryStore` with identical semantic outputs. Persistence or remote +transport must not change coverage, lifecycle, identity, or error behavior. +Verify restart restores metadata before returning `Complete` or `Queryable`. -## Adding a stored family +### Add SID persistence/distribution -Follow [Adding a summary family](../../../docs/developer_docs/adding-summary-family.md). -Store work includes canonical parameter identity, payload validation, supported -merge/readout, accuracy metadata, full/delta encoding identity, and lifecycle -tests. A byte decoder alone is not store support. +Implement `SeriesRegistry` while preserving deterministic canonical keys, +tenant isolation, idempotent resolve, namespace versioning, and conflict +detection. Verify cache loss/restart cannot bind an old SID to new labels. -## Required tests +### Interpret and verify output -- canonical labels yield stable SID independent of order; -- distinct tenants/label sets never collide; -- same SID across different materializations remains isolated; -- incompatible parameters/windows/versions never merge; -- exact coverage and missing-window failure; -- concurrent append/read/retire safety; and -- recovery never exposes state before compatible metadata. +- `Queryable` means the registered state may be considered for coverage; it is + not proof that every requested window is complete. +- Only `CoverageResult::Complete` may proceed to summary readout. +- `Missing`, `Stale`, `Gapped`, and `Incompatible` must remain distinguishable. +- `StoreWriteResult` identifies exactly which plan/materialization/window was + changed. +- Concurrent plan versions remain isolated through `MaterializationKey`. diff --git a/docs/developer_docs/adding-summary-family.md b/docs/developer_docs/adding-summary-family.md index e4d3600d..2924a018 100644 --- a/docs/developer_docs/adding-summary-family.md +++ b/docs/developer_docs/adding-summary-family.md @@ -1,54 +1,138 @@ -# Adding a summary family to ASAPQuery-backend +# Adding a summary family -## TL;DR +> Interface status: cross-repository developer workflow. ASAPQuery-backend +> implements runtime capabilities; ASAPPlanner and summary libraries own logical +> semantics and algorithm guarantees. -ASAPQuery-backend adds runtime support for a summary family only after -ASAPPlanner defines its logical query mapping and guarantee, and the producing -collector/library defines compatible state semantics. The backend must not -invent those contracts locally. +## 1. Code architecture -## Ownership prerequisites +```text +ASAPPlanner public summary/readout types + | + v +PhysicalCompiler capability match + / \ +CollectorPlan BackendPlan + | | +ASAPCollector SummaryDecoder +update + encode -> SummaryStore -> SummaryReader +``` + +A family is supported only when the same public semantic contract crosses all +components. A decoder or enum variant by itself is not pipeline support. + +## 2. Public interfaces and definitions + +The following public structures must describe the same family/version: + +```rust +pub struct SummaryCapability { + pub family: SummaryFamily, + pub algorithm: SummaryAlgorithm, + pub parameter_schema: ParameterSchema, + pub encodings: Vec, + pub operations: SummaryOperations, + pub readouts: Vec, + pub guarantee_kinds: Vec, +} + +pub struct SummaryOperations { + pub update: bool, + pub merge: bool, + pub subtract: bool, + pub delete: bool, + pub full_state: bool, + pub delta_state: bool, +} +``` + +```rust +pub trait SummaryDecoder { + type Error; + fn decode(&self, request: OtlpMetricsRequest) + -> Result, Self::Error>; +} + +pub trait SummaryStore { + type Error; + fn register(&self, contract: ValidatedMaterialization) + -> Result; + fn apply(&self, update: ValidatedSummary) + -> Result; +} + +pub trait SummaryReader { + type Error; + fn read( + &self, + request: &QueryRequest, + route: &SummaryRoute, + plan: &BackendPlanSnapshot, + ) -> Result; +} +``` + +Definitions: + +| Interface | Input | Output | +| --- | --- | --- | +| Planner mapping | PromQL workload and constraints | Selected logical summary producer/readout and guarantee | +| Capability | Family/algorithm/version | Supported parameters, encodings, operations, readouts, guarantees | +| Decoder | OTLP request | Untrusted `ReceivedSummary` values | +| Validator | Received summary + active plan | `ValidatedSummary` or structured error | +| Store | Validated materialization/update | Lifecycle/write result | +| Reader | Query + selected route + plan snapshot | Summary readout with coverage/guarantee | + +Why these interfaces exist: every stage can compare exact typed semantics and +reject unsupported combinations instead of mapping a new family to a similar +legacy one. + +## 3. Adding and verifying a family -Before changing this repository, confirm: +### Step 1: define logical semantics outside this repository -- [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) can represent and - select the family for concrete PromQL examples; -- the summary library defines parameters, update, merge/readout, encoding, and - accuracy behavior; and -- [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector) can advertise, - configure, construct, and transmit the same family/version. +Add the query mapping, readout, composability, and guarantee to ASAPPlanner. +Add update/merge/encoding behavior and mathematical guarantee to the owning +summary library. Record concrete PromQL examples. -## Backend work +### Step 2: advertise runtime capability -Backend support covers four boundaries: +Add `SummaryCapability` values for collector and backend. Declare only the +parameter ranges, encodings, operations, and readouts actually implemented. +Verify the physical compiler rejects a candidate if either side lacks one +required capability. -1. **Capability:** advertise the exact family, algorithm, parameter, readout, - merge, representation, and full/delta support implemented. -2. **Physical compilation:** accept only selected Planner nodes that can be - assigned to compatible collector and backend executors. -3. **BackendPlan and ingestion:** preserve the selected contract and reject - incompatible payloads. -4. **Readout:** execute the declared operation and return aligned - Prometheus-compatible labels, timestamps, values, and errors. +### Step 3: ingest and store -For example, support for a new quantile family is incomplete until this query -can be planned, produced, ingested, and read end to end: +Implement decode to `ReceivedSummary`, validation to `ValidatedSummary`, and +store application through public interfaces. Include family, algorithm, +canonical parameters, encoding version, grouping, and window in compatibility +identity. + +### Step 4: execute readout + +Implement `SummaryReader::read` for the Planner-selected readout. Preserve +labels, timestamps, result type, logical coverage, and guarantee. Do not choose +the family again from query text. + +### Step 5: interpret and verify output + +For a quantile family, an end-to-end example is: ```promql quantile_over_time(0.95, request_duration_seconds[5m]) ``` -## Validation - -The cross-repository test must cover: +Verify: -- supported and deliberately unsupported parameters; -- full-state transmission and delta transmission when claimed; -- duplicate, missing, reordered, stale, and incompatible payloads; -- merge across every claimed grouping/window shape; -- aligned comparison with an identical exact input stream; -- the declared accuracy and freshness SLA; and -- capability downgrade and exact fallback behavior. +- output series align with exact results by labels and timestamps; +- reported guarantee matches the selected parameterization; +- errors satisfy the predeclared SLA over identical input; +- full and delta state have equivalent query semantics when delta is claimed; +- corrupt, mismatched, stale, gapped, or unsupported input returns a structured + failure and does not change queryable state; and +- `QueryResponse.source`, plan/materialization IDs, coverage, and freshness + prove which implementation produced the answer. -Unit tests for serialization or a local readout alone do not establish pipeline +Unit tests for serialization are necessary but do not establish cross-repository support. From d49a8da47e903e0efe6e2c88772512266bbb5c3e Mon Sep 17 00:00:00 2001 From: zz_y Date: Fri, 28 Aug 2026 05:14:17 -0600 Subject: [PATCH 3/3] docs: define public series identity contract --- .../developer_docs/otlp-summary-ingestion.md | 20 +++----- .../summary-storage-and-series-identity.md | 50 +++++++++++++++++-- docs/design_docs/series-identity.md | 5 ++ 3 files changed, 57 insertions(+), 18 deletions(-) diff --git a/data_plane/docs/developer_docs/otlp-summary-ingestion.md b/data_plane/docs/developer_docs/otlp-summary-ingestion.md index c8d40723..ebc1c2cb 100644 --- a/data_plane/docs/developer_docs/otlp-summary-ingestion.md +++ b/data_plane/docs/developer_docs/otlp-summary-ingestion.md @@ -75,20 +75,12 @@ pub trait SeriesIdentityResolver { fn resolve(&self, key: CanonicalSeriesKey) -> Result; } - -pub struct CanonicalSeriesKey { - pub tenant: String, - pub metric_name: String, - pub identifying_labels: BTreeMap, -} - -pub struct ResolvedSeries { - pub sid: u64, - pub canonical_key: CanonicalSeriesKey, - pub namespace_version: String, -} ``` +`SeriesId`, `SeriesIdNamespace`, `CanonicalSeriesKey`, and `ResolvedSeries` have +one public definition in +[Summary storage and series identity](summary-storage-and-series-identity.md#sid-definition). + ```rust pub trait SummaryValidator { type Error; @@ -123,7 +115,7 @@ pub struct IngestResult { pub disposition: IngestDisposition, pub plan_id: String, pub materialization_id: String, - pub sid: u64, + pub series_id: SeriesId, pub window: LogicalWindow, pub queryable_at: Option, } @@ -168,7 +160,7 @@ recovery full state. Verify `AppliedDelta`, `Duplicate`, and ### Add series identity behavior -Add canonical input fields to `CanonicalSeriesKey`, never to the numeric SID +Add canonical input fields to `CanonicalSeriesKey`, never to `SeriesId.value` alone. Verify label-order independence, tenant isolation, cached-ID conflict recovery, and stable namespace reporting. diff --git a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md index 218cdfd4..e608803b 100644 --- a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md +++ b/data_plane/docs/developer_docs/summary-storage-and-series-identity.md @@ -22,20 +22,62 @@ identities remain distinct. ## 2. Public interfaces and definitions ```rust +pub struct SeriesId { + pub namespace: SeriesIdNamespace, + pub value: u64, +} + +pub struct SeriesIdNamespace { + pub tenant: String, + pub version: String, +} + +pub struct CanonicalSeriesKey { + pub tenant: String, + pub metric_name: String, + pub identifying_labels: BTreeMap, +} + +pub struct ResolvedSeries { + pub id: SeriesId, + pub canonical_key: CanonicalSeriesKey, +} + pub trait SeriesRegistry: Send + Sync { type Error; fn resolve(&self, key: CanonicalSeriesKey) -> Result; - fn lookup(&self, sid: u64, namespace_version: &str) + fn lookup(&self, id: &SeriesId) -> Result, Self::Error>; } ``` -`CanonicalSeriesKey` and `ResolvedSeries` are defined by the ingestion public -interface. `resolve` is idempotent. A sender-provided SID never overrides a -conflicting canonical key. +### SID definition + +`SeriesId` (`sid`) is an opaque numeric identifier scoped by exactly one +`SeriesIdNamespace`. The namespace contains the tenant/isolation domain and a +version that changes whenever the authoritative registry is rebuilt without +preserving its previous assignments. + +```text +(SeriesIdNamespace, SeriesId.value) <-> CanonicalSeriesKey +``` + +Within one namespace this mapping is one-to-one: + +- the same canonical key always resolves to the same SID; +- two different canonical keys never resolve to the same SID; and +- the same numeric value in two namespaces is not the same SID. + +`identifying_labels` is ordered by label name before lookup or hashing, so input +label order does not affect identity. Summary family, parameters, aggregation +group, window, materialization ID, and plan ID are excluded because they +identify maintained state, not the source metric series. + +`SeriesRegistry::resolve` is idempotent. A sender-provided SID is only a lookup +shortcut; it never overrides a conflicting canonical key. ```rust pub struct MaterializationKey { diff --git a/docs/design_docs/series-identity.md b/docs/design_docs/series-identity.md index 162977e4..b786b063 100644 --- a/docs/design_docs/series-identity.md +++ b/docs/design_docs/series-identity.md @@ -15,6 +15,11 @@ identity namespace. The backend registry assigns or validates this mapping; collectors may cache it, but payload labels remain the recovery evidence needed to detect stale or unknown IDs. +Formally, SID is the pair `(namespace, numeric_value)`, not a globally meaningful +integer. The namespace contains the tenant/isolation domain and a registry +version. Its public data structures and registry interfaces are defined in the +[developer guide](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md#sid-definition). + ## Identity contract The canonical series key consists of: