diff --git a/README.md b/README.md index 7c4403c4..2fc77976 100644 --- a/README.md +++ b/README.md @@ -130,7 +130,7 @@ spin up ASAPCollector + ASAPQuery-backend + Grafana together — live in [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector). The full multi-stage MVP demo (10 producers / 2 agents / 1 gateway / 1 backend / Thanos store-gateway / MinIO) is documented in its -[`docs/mvp-demo-runbook.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/mvp-demo-runbook.md). +[`mvp-demo-runbook.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/user_guide/mvp-demo-runbook.md). To build and run just this backend, see **Building from source** below. @@ -192,7 +192,7 @@ backend surfaces in every response's `infos` field. The wire format is documented in [`asap_otel_proto`](crates/asap_otel_proto/) and the cross-language byte-parity gate is described in -[ASAPCollector's edge-framework design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design-asap-edge-framework.md). +[ASAPCollector's system design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/system-overview.md). ## Query response shape @@ -231,9 +231,9 @@ that produced the current architecture: - **Sketch placement planner** — moved into [`ASAPCollector/controller/`](https://github.com/ProjectASAP/ASAPCollector/tree/main/controller) - **PromQL pattern matchers for the planner** — migrated into the controller's L3 `intent_algebra` + L4 `sketch_algebra` -- **JSONL cold-fallback path** — deleted; the archive tier replaces - it. See ASAPCollector's - [`docs/design-jsonl-deprecation-and-gorilla-promql-completeness.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design-jsonl-deprecation-and-gorilla-promql-completeness.md) +- **JSONL cold-fallback path** — deleted; the configured exact backend is the + explicit fallback described by + [`query-execution.md`](data_plane/docs/design_docs/query-execution.md). - **`StorageBackend::ColdJsonlFallback`** enum variant — removed - **Backend-local cost-model line item for cold-tier scan bytes** — removed (controller's tier-spanning cost model is the source of diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 755af3bc..00000000 --- a/TODO.md +++ /dev/null @@ -1,324 +0,0 @@ -# TODO — ASAPQuery-backend / sketchDB - -_Last updated: 2026-05-01._ - -Post-session state: PRs #43–#71 landed. This doc enumerates what's -left for sketchDB to be paper-ready (VLDB / SIGMOD) + what's -deferred to future work. - -See the design source at [`docs/design-sketch-db.md`](docs/design-sketch-db.md). - -## Warm-tier query path closes the loop (2026-05-01) - -The all-five-sketch query path from 2026-04-30 was wire-correct but -the live PromQL surface still returned empty even with data -demonstrably in the precompute store. Two PRs fixed that: - -- **OTLP gRPC `max_decoding_message_size`** ([#70](https://github.com/ProjectASAP/ASAPQuery-backend/pull/70)). - tonic's 4 MiB default rejected the gateway's first-window - full-state DDSketch batch (~17 MiB at 1k cardinality). Gateway - exporter looped on `decoded message length too large` forever. - Bumped the receiver to 64 MiB, matching the `max_recv_msg_size_mib` - value the agent and gateway already declare on their own OTLP - receivers. - -- **`range_query_into` overlap filter + closest-pane + response - annotation** ([#71](https://github.com/ProjectASAP/ASAPQuery-backend/pull/71)). - Two-part fix: - - 1. **Overlap filter (store side).** `MutableEpoch::range_query_into` - and `SealedEpoch::range_query_into` in - `simple_map_store/common.rs` (and the on-disk parts variant - in `per_key.rs:query_disk_parts`) used a "fully-contained" - filter (`tr.0 < start || tr.0 > end || tr.1 > end → skip`). - For tumbling windows of size W with a query range R, this - matches at most `floor(R/W)` panes and only when both query - endpoints land exactly on the pane grid. PromQL queries - don't align to the grid (the wall-clock fractional portion - of `query_time` is generally non-zero), so the strict filter - returned 0 panes for every realistic query. Replaced with - standard half-open overlap: keep `[tr.0, tr.1)` if `tr.1 > start - && tr.0 < end`. - - 2. **Closest-pane + annotation (engine + adapter side).** Per - follow-up review: rather than merge multiple overlapping - panes (slightly imprecise for sketch summaries), the engine - now picks a **single closest pane** (max `tr.1`, tie-break - on max `tr.0`) and threads the chosen `[start_ms, end_ms)` - up through `QueryResult::with_window_used` to the Prometheus - HTTP adapter, which adds it to the response's `infos` array - as `precompute_window: [..., ...) ms (width N ms)`. Mirrors - the existing `with_accuracy` annotation pattern. Now the - caller sees exactly which precompute time range produced - each value — important when the request range and the - answered range differ. - -### Live verification - -``` -$ curl '/api/v1/query?query=quantile_over_time(0.5, http_requests_total_latency_ms_quantile[1m])&time=$(now-90s)' -{"data":{"result":[{"metric":{"node":""}, - "value":[..., "19.493849507395904"]}], - "resultType":"vector"}, - "infos":["accuracy: ε=0.01, δ=0, kind=relative_quantile", - "precompute_window: [1777655280000, 1777655310000) ms (width 30000 ms)"]} -``` - -Pre-fix: `result: []` with `No precomputed outputs found` even -though `runtime_info.earliest_timestamp_per_aggregation_id` was -populated and worker logs showed `Worker emitting 1 sketch outputs -for group (1, )` at every flush. - -### Companion changes on the agent side - -The collector-side path needed three connected fixes for delta -transmission to round-trip -([ASAPCollector#210](https://github.com/ProjectASAP/ASAPCollector/pull/210)) -plus a windowed-processor pass-through to make multi-sketch -single-pipeline configs work -([ASAPCollector#211](https://github.com/ProjectASAP/ASAPCollector/pull/211)). -The backend-side delta apply path -(`apply_modified_otlp_delta_bytes` → -`{DDSketch,CMS,CountSketch,HLL}Accumulator::apply_proto_delta_bytes`) -was already in place; it just wasn't reachable until the agent -correctly tagged delta payloads on the typed encoding field and -stopped polluting the per-data-point attribute set with the -encoding string (which had broken the per-series snapshot cache -key). - -- **Inference-YAML pattern coverage.** Expanded - `data_plane/examples/promql/inference_config.yaml` (and the - SQL twin) with multi-quantile / wider-range / rate / increase / - topk entries; closes - [ASAPCollector PROGRESS.md follow-up #4](https://github.com/ProjectASAP/ASAPCollector/blob/main/PROGRESS.md#open-follow-ups-not-e2e-blockers) - ("Inference config breadth"). New `tests/inference_yaml_pattern_coverage.rs` - pins each family's YAML → `find_query_config` → `query_statistic` - routing. - -## All-five-sketch query path verification (2026-04-30) - -Each sketch type now has a runtime-verified PromQL → backend path -through the modified-OTLP wire format (typed `Metric.data = -{DDSketch | KLLSketch | HLLSketch | CountSketch | CountMinSketch}` -data points). Specifically: - -- **`query_statistic` for every sketch accumulator.** Implemented - on `DDSketchAccumulator` (Quantile / Sum / Count / Min / Max), - `HllSketchAccumulator` (Cardinality, with `Count` accepted as a - Cardinality alias for the existing PromQL `count(...)` path), - `CountSketchAccumulator` (Topk / Count / Sum, no-key fallback - returns row-mean total), and `CountMinSketchAccumulator` (Count - / Sum, no-key fallback returns the min-row sum — the canonical - CMS total-event estimator that's exact when each insert - increments one cell per row). -- **`accumulator_factory.rs`**: `DDSketchAccumulatorUpdater` wired - in (alongside CMS / CountSketch / KLL / HLL updaters) so the - precompute_engine recognises `AggregationType::DDSketch` from the - `streaming.yaml` schema. -- **Modified-OTLP envelope decoders** for each sketch type land via - the agent processors using sketchlib-go's `SerializePortable` / - `SerializeMsgpack`; the backend's `from_sketchlib_proto_bytes` / - `from_msgpack_bytes` constructors round-trip through - `SketchEnvelope { sketch_state: Some(SketchState::*(state)) }`. - -### Known reconciliation gap (cleanup, not a blocker) - -- ~~`compatible_agg_types` in - [`asap_types/src/capability_matching.rs`](crates/asap_types/src/capability_matching.rs) - does not list `CountMinSketch` under `Statistic::Sum`, but - [`promql_utilities/src/query_logics/logics.rs`](crates/promql_utilities/src/query_logics/logics.rs) - treats CMS as the canonical approximator for both `Sum` and - `Count`. The runtime e2e succeeds because the inference YAML's - exact-match `find_query_config` path bypasses - `find_compatible_aggregation`. Two tables → one table is the - right cleanup.~~ **Closed by - `fix/capability-matching-cms-sum-reconcile`.** `compatible_agg_types` - now lists `CountMinSketch` under `Statistic::Sum` (and `MultipleSum` - under `Statistic::Count`); both tables are kept in agreement by the - `capability_canonical_map_agreement` test, which enumerates every - `(Statistic, QueryTreatmentType)` pair and asserts the canonical map's - output is contained in `compatible_agg_types(Statistic)`. The dead - `Min/Max-Approximate → DatasketchesKLL` branch in - `map_statistic_to_precompute_operator` was removed (KLL has no - min/max query surface — that route would have produced runtime - errors). -- CMS query without a paired `SetAggregator` / - `DeltaSetAggregator` returns total volume, not per-key - frequency. To drive `topk(N, …)` over CMS-tracked keys we need - a key-aggregator processor on the agent. Tracked as a paper - follow-up; out of scope for v1. -- **`IngestState.sketch_snapshots` is RAM-only.** Per-series - snapshot cache that delta frames apply against is lost on - backend restart. After a bounce, agents continue emitting - `proto_delta` against their local snapshots, and the backend - drops them as "delta-sketch arrived before any base snapshot" - until the agent itself restarts. Persist to the existing - per-key disk layer used by `SketchStore::with_persistence_per_key`, - or add an OpAMP capability for backend → agent "send next - frame as full state" signalling. Same item lives on the - collector side - ([`PROGRESS.md` follow-up #3](https://github.com/ProjectASAP/ASAPCollector/blob/main/PROGRESS.md)); - a fix on either side closes the gap. - -## For paper submission (blocker) - -### 1. Cold-query fallback — §5.2 of the sketch-DB design — **done (local-FS cold store)** - -Initial v1 landed: [`drivers/query/fallback/s3_adapter.rs`](data_plane/src/drivers/query/fallback/s3_adapter.rs) -is a `FallbackClient` that serves capability-misses from a -hour-bucketed JSONL raw store. The format (`raw//YYYY/MM/DD/HH/part-NNNNNN.jsonl`) -is byte-identical to the S3 layout, so a future -`S3ColdStore: ColdStore` drops in with no adapter changes. - -Follow-ups (not paper-blocking): - -- **S3-backed `ColdStore` impl** next to the local-FS one; same - trait, `aws-sdk-s3` list-objects-v2 for prefix pruning. -- **Richer query surface.** Today we compute `metric{...}`, - `sum|count|avg|min|max(...)`. Regex matchers, `by (...)` - grouping, and `rate/increase` over raw samples delegate to - the chained inner fallback (typically Prometheus). Adding - grouping + regex is ~200 LOC when needed. -- **Latency target.** Paper claim is ≤2× P99 vs. warm-hot — - unverified until the multi-agent harness lands (blocker #6 - of `DataCollector/TODO.md`). The old three-way query harness - ([#66](https://github.com/ProjectASAP/ASAPQuery-backend/pull/66)) - targeted the deprecated Arroyo/Kafka stack and has been removed; a - runner for the current OTLP architecture is still needed to produce - this number. -- ~~`data_plane` `main.rs` wiring of `ASAP_COLD_STORE_ROOT`~~ - **done (P1, 2026-04-30).** `--cold-store-root` flag with - `env = "ASAP_COLD_STORE_ROOT"` plumbed into a - `build_adapter_config` helper that selects - `prometheus_promql_with_cold` when set. Four unit tests pin - the wiring matrix (cold × forward). Combine with - `--forward-unsupported-queries` to keep Prom as the tail of - the chain; without it, unsupported shapes return empty. - -### 2. Accuracy-profile library per sketch type - -`AccuracyProfile` trait landed in Phase 6.4 as a stub. Each -sketch type needs formally-derived bounds + empirical validation. - -**Reference: the sketch-bench design doc** -. - -Sketches to cover: -- **KLL** (datasketches-rs): `ε` vs `k` bound -- **DDSketch**: relative-error `α` bound -- **CMS** (count-min): `ε` bound from `w × d` -- **CMS-with-heap**: top-K error bound -- **HLL**: `δ` (std-err) bound from `p` - -Deliverable: `AccuracyProfile::derive(&AggregationConfig)` -returns concrete (ε, δ) per sketch type — not a stub. Unit tests -against the sketch-bench corpus. - -### 3. End-to-end capability-miss feedback loop test — **done (HTTP round-trip)** - -HTTP-level e2e landed in -[`data_plane/src/tests/capability_miss_http_e2e_tests.rs`](data_plane/src/tests/capability_miss_http_e2e_tests.rs). -Spins up a real backend HTTP server + mock control plane HTTP -server, fires a PromQL `sum(metric)` query that capability-misses, -and measures wall-clock `time_to_plan_ready` from query issue to -the backend observing the new `StreamingConfig` via -`GET /api/v1/streaming-config`. Localhost floor: ~20 ms. - -Also asserts: -- Mock control plane received the HTTP notify with the documented - `{kind: "capability_miss", ...}` payload -- Backend's hot-reload handle has the exact `agg_id` the - control plane pushed -- A repeat query on the same metric does **not** fire a second - notify (loop is idempotent under query replay) - -Follow-up (not paper-blocking): - -- **Cross-process test.** Today's test is single-process with - two HTTP servers. A docker-compose harness that wires a real - DataCollector controller binary against the real backend - binary is tracked by blocker #6 of `DataCollector/TODO.md`. -- **Time-to-first-hit over real data.** The "next query returns - data" half of the story needs OTLP ingestion between the - plan push and the repeat query — still tracked as an - operational follow-up in the DataCollector repo. - -### 4. Serialization format versioning tests — **done ([#65](https://github.com/ProjectASAP/ASAPQuery-backend/pull/65))** - -`mod v2_forward_compat` in -[`data_plane/src/tests/persist_format_versioning_tests.rs`](data_plane/src/tests/persist_format_versioning_tests.rs) -covers all three persistence sites with three tests that pin the contract -to `PERSIST_FORMAT_VERSION + 1` (self-updating if the version is bumped): - -- `schema_v1_with_future_version_falls_back_and_rewrites_clean` — tampers - the JSON `version` field to v_current+1, asserts safe-fallback + rewrite - at v_current with the new config's schemas and **no leakage from the - bumped blob** (the no-data-corruption claim). -- `backfill_v1_with_future_version_falls_back_and_rewrites_clean` — same - contract for `BackfillRegistry`, including `next_job_id` field presence - post-fallback. -- `part_meta_with_future_version_returns_format_error` — `SketchStore` - `meta.bin` has no fallback (parts are opaque), so the contract is a - clean `PersistError::Format("unsupported version ...")`. - -All three pass. - -Follow-up (not paper-blocking): a docker-compose harness that bumps -`PERSIST_FORMAT_VERSION` in code, rebuilds, and restarts a running -backend with on-disk v1 state to verify the live restart path. The -unit tests cover the load path which is where the version-mismatch -logic lives, so this is just defense in depth. - -### 5. Correctness proofs (for paper's theory section) — **done ([`docs/proofs.md`](docs/proofs.md))** - -All three proofs landed in [`docs/proofs.md`](docs/proofs.md) §§2–4 -(statement / setup-lemmas / proof / caveats / code-anchors per -proof; §1 reproduces the per-sketch accuracy bounds the proofs -treat as black boxes). - -1. **`combine_statistic` correctness** across schema-timeline - segments (`docs/proofs.md` §2): additive stats (Count / Sum / - Min / Max) combined over non-overlapping segments equal the - single-schema answer up to per-segment sketch error. - Non-combinable stats (Quantile / Topk / Cardinality / Rate / - Increase) return `Partial` with a bounded `covered` subset. -2. **Write-barrier safety** (`docs/proofs.md` §3): no sample - ingested at wall-clock time `t > force_expire(agg_id).ts` - appears in any query whose range includes `t'`. Follows from - the `is_writable` check + schema lifecycle monotonicity. -3. **Backfill determinism** (`docs/proofs.md` §4): the §10.5 - invariants (time-disjoint, known agg, within retention) plus - ordered raw-sample replay produce bit-identical sketches vs. - live ingest for the same underlying samples. - -## Future work (post-paper) - -### F1. Inter-window compaction - -Long-running sketchDB needs compaction — merge adjacent small -windows into larger ones when accuracy can be re-derived, drop -sub-window duplicates from overlapping backfills. Today the -store only supports write + evict-by-agg, no inter-window -merge. Impacts long-term storage cost. - -### F2. Multi-tier storage — §tier-2 read cache - -Design draft exists -(`docs/design-simple-map-store-persistence.md` §tier-2 section). -Materialize as a read-through cache on top of the S3/disk -tier. Implementation deferred — current single-tier behaviour -is fine for v1 paper. - -### F3. Cross-sketch-type combine - -Combining KLL(k=200) with KLL(k=100), or KLL with DDSketch, for -the same logical quantile across a schema boundary. Out of -scope for v1 paper — operational practice has so far been -"new sketch config ⇒ new agg_id, new time-range", so combine -never sees heterogeneous types per metric. - -### F4. OTLP ingest counter parity with other observability metrics - -PR #51 added barrier-drop counter for OTLP paths. Other -silent-drop sites (decode errors, routing errors, unconfigured -metrics) could get the same treatment. Low-pri. diff --git a/data_plane/docs/README.md b/data_plane/docs/README.md index 1fc7ce71..0572b571 100644 --- a/data_plane/docs/README.md +++ b/data_plane/docs/README.md @@ -1,114 +1,40 @@ -# QueryEngineRust Developer Documentation +# ASAPQuery data-plane documentation -Welcome to the QueryEngineRust developer documentation! This directory contains guides for extending the system with new components. +The data plane ingests state produced under an active BackendPlan, stores that +state, answers supported PromQL queries, and uses an explicit exact fallback +for unsupported queries. It executes plans; it does not choose summary families +or re-plan queries. -## Architecture Overview +## Design documents -QueryEngineRust is organized into clear, extensible layers: +- [Plan-aware query execution](design_docs/query-execution.md) — ingestion, + routing, readiness, summary readout, and exact fallback contracts. +- [Repository-wide summary storage](../../docs/design_docs/summary-storage.md) — + materialization state, lifecycle, and query consistency. +- [BackendPlan](../../control_plane/docs/backend-plan.md) — the control-plane + contract installed by the data plane. -``` -┌─────────────────────────────────────────────────────────┐ -│ Client Applications │ -└─────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────┐ -│ Protocol Servers (HTTP, etc.) │ -│ - Parse protocol-specific requests │ -│ - Route to appropriate adapter │ -│ - Handle protocol-specific endpoints │ -└─────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────┐ -│ Protocol Adapters (Prometheus, etc.) │ -│ - Parse query language (PromQL, SQL, etc.) │ -│ - Format responses for protocol │ -│ - Determine if query is supported │ -└─────────────────────────────────────────────────────────┘ - │ - ┌──────┴──────┐ - ▼ ▼ - ┌─────────────────┐ ┌──────────────────┐ - │ Query Engine │ │ Fallback Client │ - │ - Execute │ │ - Forward │ - │ queries │ │ unsupported │ - │ - Return │ │ queries │ - │ results │ │ │ - └────────┬────────┘ └──────────────────┘ - │ - ▼ - ┌─────────────────┐ - │ Store │ - │ - Data storage │ - │ - Sketches │ - └─────────────────┘ - ▲ - │ - ┌────────┴────────┐ - │ Ingest Drivers │ - │ - Kafka, etc. │ - └─────────────────┘ -``` +## Developer documentation -## Directory Structure +- [Extension boundaries](developer_docs/extension-points.md) — responsibilities + of protocol servers, adapters, and fallback clients. +- [Adding a summary family](../../docs/developer_docs/adding-summary-family.md) — + cross-repository prerequisites and backend validation. -``` -src/drivers/ -├── ingest/ # Data ingestion (Kafka, etc.) -├── query/ -│ ├── adapters/ # Protocol adapters (Prometheus HTTP, etc.) -│ ├── fallback/ # Fallback backends (Prometheus, ClickHouse, etc.) -│ └── servers/ # Protocol servers (HTTP, Flight SQL, etc.) -``` +## User guide -## Extension Guides +- [Querying ASAP](user_guide/querying-asap.md) — PromQL behavior, planned + summary execution, exact fallback, freshness, and errors. -- **[Adding a Protocol Adapter](./adding-protocol-adapter.md)** - Add support for new query protocols (e.g., ClickHouse HTTP API) -- **[Adding a Fallback Backend](./adding-fallback-backend.md)** - Add new fallback query backends (e.g., DuckDB, Elasticsearch) -- **[Adding a Protocol Server](./adding-protocol-server.md)** - Add new protocol servers (e.g., Flight SQL, gRPC) +## Ownership -## Key Concepts +- [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) owns logical query + planning, query-to-summary mapping, and accuracy reasoning. +- The ASAPQuery control plane owns physical compilation and BackendPlan. +- The data plane owns ingestion, storage, readout, query execution, and exact + fallback under the installed plan. +- [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector) owns summary + construction and transmission at the edge. -### Protocol Adapter -Handles protocol-specific request/response formatting and query parsing. Examples: Prometheus HTTP API, ClickHouse HTTP API. - -### Fallback Backend -External query system to forward unsupported queries to. Examples: Prometheus, ClickHouse, DuckDB. - -### Protocol Server -Handles network communication for a specific protocol. Examples: HTTP server, Flight SQL server. - -## Quick Reference - -### Adding a Protocol Adapter -1. Create `src/drivers/query/adapters/my_adapter.rs` -2. Implement `HttpProtocolAdapter` trait -3. Add to factory in `factory.rs` -4. Update `QueryProtocol` enum - -### Adding a Fallback Backend -1. Create `src/drivers/query/fallback/my_backend.rs` -2. Implement `FallbackClient` trait -3. Export from `fallback/mod.rs` - -### Adding a Protocol Server -1. Create `src/drivers/query/servers/my_server.rs` -2. Implement server logic with appropriate adapter -3. Export from `servers/mod.rs` - -## Testing - -Each component should include: -- Unit tests in the same file -- Integration tests in `src/tests/` -- Example usage in documentation - -## Contributing - -When adding new components: -1. Follow existing naming conventions -2. Add comprehensive documentation -3. Include tests -4. Update this documentation -5. Keep backward compatibility +Historical ingestion paths, file-by-file migrations, and configuration +walkthroughs are not data-plane design contracts. diff --git a/data_plane/docs/adding-fallback-backend.md b/data_plane/docs/adding-fallback-backend.md deleted file mode 100644 index 847a8997..00000000 --- a/data_plane/docs/adding-fallback-backend.md +++ /dev/null @@ -1,117 +0,0 @@ -# Adding a Fallback Backend - -Fallback backends allow forwarding unsupported queries to external systems. This guide shows how to add support for a new fallback backend. - -## Overview - -A fallback backend: -- Accepts queries in a specific language (SQL, PromQL, etc.) -- Makes HTTP/gRPC/native calls to external system -- Returns results in a generic format -- Optionally provides runtime/health information - -## Example: Adding DuckDB HTTP Fallback - -### Step 1: Create the Fallback Client - -Create `src/drivers/query/fallback/duckdb.rs`: - -```rust -/// Fallback client for DuckDB HTTP API -pub struct DuckDBHttpFallback { - client: Client, - base_url: String, -} - -impl DuckDBHttpFallback { - pub fn new(base_url: String) -> Self { - Self { - client: Client::new(), - base_url, - } - } -} - -#[derive(Debug, Deserialize)] -struct DuckDBResponse { - success: bool, - data: Option>>, - columns: Option>, - error: Option, -} - -#[async_trait] -impl FallbackClient for DuckDBHttpFallback { - async fn execute_query( - &self, - request: &ParsedQueryRequest, - ) -> Result, StatusCode> { - ... - } - - async fn get_runtime_info(&self) -> Result { - ... - } -} -``` - -### Step 2: Export from Module - -Update `src/drivers/query/fallback/mod.rs`: - -```rust -mod duckdb; -pub use duckdb::DuckDBHttpFallback; -``` - -### Step 3: Use in Configuration - -The fallback client can now be used in adapter configuration: - -```rust -use crate::drivers::query::adapters::AdapterConfig; -use crate::drivers::query::fallback::DuckDBHttpFallback; -use std::sync::Arc; - -// Create adapter config with DuckDB fallback -let fallback = Some(Arc::new( - DuckDBHttpFallback::new("http://localhost:8080".to_string()) -) as Arc); - -let config = AdapterConfig::new( - QueryProtocol::PrometheusHttp, // Protocol for incoming queries - QueryLanguage::sql, // Query language - fallback, // DuckDB fallback -); -``` - -### Step 4: Add Tests - -Add tests in `duckdb.rs`: - -```rust -#[cfg(test)] -mod tests { - use super::*; - - #[tokio::test] - async fn test_duckdb_fallback_creation() { - ... - } - - // Mock DuckDB server test would go here -} -``` - -## FallbackClient Trait Methods - -### Required: `execute_query()` -- Accepts a `ParsedQueryRequest` (query string + time) -- Makes external call to backend -- Returns `Json` response -- Should handle all error cases gracefully - -### Optional: `get_runtime_info()` -- Returns health/status information from backend -- Has default implementation (returns empty JSON) -- Override if backend has health endpoint diff --git a/data_plane/docs/adding-protocol-adapter.md b/data_plane/docs/adding-protocol-adapter.md deleted file mode 100644 index b2ce68fc..00000000 --- a/data_plane/docs/adding-protocol-adapter.md +++ /dev/null @@ -1,133 +0,0 @@ -# Adding a Protocol Adapter - -Protocol adapters handle protocol-specific request/response formatting and query language parsing. This guide shows how to add support for a new query protocol. - -## Overview - -A protocol adapter: -- Parses incoming requests (GET/POST parameters, headers, etc.) -- Translates queries to internal format -- Formats query results for the protocol -- Defines protocol-specific endpoints - -## Example: Adding ClickHouse HTTP Adapter - -### Step 1: Create the Adapter File - -Create `src/drivers/query/adapters/clickhouse_http.rs`: - -```rust - -/// ClickHouse HTTP protocol adapter -pub struct ClickHouseHttpAdapter { - config: AdapterConfig, -} - -impl ClickHouseHttpAdapter { - ... -} - -#[async_trait] -impl QueryRequestAdapter for ClickHouseHttpAdapter { - ... -} - -#[async_trait] -impl QueryResponseAdapter for ClickHouseHttpAdapter { - ... -} - -#[async_trait] -impl HttpProtocolAdapter for ClickHouseHttpAdapter { - ... -} -``` - -### Step 2: Add Protocol Enum Variant - -Update `src/data_model/enums.rs` to add the new protocol: - -```rust -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum QueryProtocol { - ... - ClickHouseHttp, // Add this -} -``` - -### Step 3: Export from Module - -Update `src/drivers/query/adapters/mod.rs`: - -```rust -pub mod clickhouse_http; -pub use clickhouse_http::ClickHouseHttpAdapter; -``` - -### Step 4: Add to Factory - -Update `src/drivers/query/adapters/factory.rs`: - -```rust -pub fn create_http_adapter(config: AdapterConfig) -> Arc { - match config.protocol { - ... - QueryProtocol::ClickHouseHttp => { // Add this - Arc::new(ClickHouseHttpAdapter::new(config)) - } - } -} -``` - -### Step 5: Add Convenience Constructor (Optional) - -Update `src/drivers/query/adapters/config.rs`: - -```rust -impl AdapterConfig { - pub fn clickhouse_http(fallback_url: String, forward_unsupported: bool) -> Self { - ... - } -} -``` - -### Step 6: Test the Adapter - -Add tests in `clickhouse_http.rs`: - -```rust -#[cfg(test)] -mod tests { - use super::*; - - #[tokio::test] - async fn test_parse_get_request() { - ... - } -} -``` - -## Key Traits to Implement - -### Required: `QueryRequestAdapter` -- `parse_get_request()` - Parse GET requests -- `parse_post_request()` - Parse POST requests -- `get_query_endpoint()` - Return endpoint path - -### Required: `QueryResponseAdapter` -- `format_success_response()` - Format successful query results -- `format_error_response()` - Format errors -- `format_unsupported_query_response()` - Format unsupported query errors - -### Required: `HttpProtocolAdapter` -- `adapter_name()` - Return adapter name for logging -- `get_runtime_info_path()` - Return health/status endpoint path -- `handle_runtime_info()` - Handle health/status requests - -## Common Gotchas - -- Don't implement query execution in the adapter - that's the engine's job -- Don't hard-code URLs or configuration - use `AdapterConfig` -- Handle both GET and POST requests appropriately -- Return protocol-specific error formats -- Use existing types from `traits.rs` (`ParsedQueryRequest`, `QueryExecutionResult`) diff --git a/data_plane/docs/adding-protocol-server.md b/data_plane/docs/adding-protocol-server.md deleted file mode 100644 index dd04e332..00000000 --- a/data_plane/docs/adding-protocol-server.md +++ /dev/null @@ -1,105 +0,0 @@ -# Adding a Protocol Server - -Protocol servers handle network communication for specific protocols. This guide shows how to add a new protocol server (like Flight SQL, gRPC, etc.). - -## Overview - -A protocol server: -- Listens on a network port -- Handles protocol-specific requests -- Uses adapters to process queries -- Returns protocol-specific responses - -## Example: Adding Flight SQL Server - -Flight SQL is Apache Arrow's SQL protocol over gRPC. Here's how to add it: - -### Step 1: Create the Server - -Create `src/drivers/query/servers/flight_sql.rs`: - -```rust -#[derive(Debug, Clone)] -pub struct FlightSqlServerConfig { - pub port: u16, - pub adapter_config: AdapterConfig, -} - -pub struct FlightSqlServer { - config: FlightSqlServerConfig, - query_engine: Arc, - store: Arc, -} - -impl FlightSqlServer { - pub fn new( - config: FlightSqlServerConfig, - query_engine: Arc, - store: Arc, - ) -> Self { - Self { - config, - query_engine, - store, - } - } - - pub async fn run(self) -> Result<(), Box> { - ... - } -} -``` - -### Step 3: Export from Module - -Update `src/drivers/query/servers/mod.rs`: - -```rust -pub mod flight_sql; -pub use flight_sql::{FlightSqlServer, FlightSqlServerConfig}; -``` - -### Step 4: Update Main Binary - -Update `src/main.rs` to support choosing the server: - -```rust -#[derive(Parser, Debug)] -struct Args { - // ... existing args ... - - /// Server protocol to use (http, flight_sql) - #[arg(long, default_value = "http")] - server_protocol: String, - - // ... rest of args ... -} - -#[tokio::main] -async fn main() -> Result<()> { - let args = Args::parse(); - - // ... setup engine, store, etc. ... - - match args.server_protocol.as_str() { - "http" => { - let server = HttpServer::new(http_config, engine, store); - server.run().await?; - } - "flight_sql" => { - let flight_config = FlightSqlServerConfig { - port: args.http_port, - adapter_config, - }; - let server = FlightSqlServer::new(flight_config, engine, store); - server.run().await?; - } - _ => { - eprintln!("Unknown server protocol: {}", args.server_protocol); - std::process::exit(1); - } - } - - Ok(()) -} -``` diff --git a/data_plane/docs/design_docs/query-execution.md b/data_plane/docs/design_docs/query-execution.md new file mode 100644 index 00000000..761cced4 --- /dev/null +++ b/data_plane/docs/design_docs/query-execution.md @@ -0,0 +1,122 @@ +# Plan-aware query execution + +> Status: proposed +> +> MVP relation: required for every summary-backed query and exact fallback. + +## TL;DR + +The data plane accepts only state compatible with its active BackendPlan. At +query time it matches the PromQL request to a planned readout, checks coverage +and freshness, reads the matching materialization, and returns a +Prometheus-compatible result. A query that cannot be served safely is rejected +or sent to the exact fallback selected by the plan. + +## Data flow + +```text +ASAPCollector -- OTLP summary state --> ingest validation --> SummaryStore + | +PromQL request --> protocol adapter --> planned routing + readiness + | + summary readout <----------+ + | + Prometheus-compatible response + +Unsupported/planned-exact query --> exact fallback backend +``` + +The data plane never infers a summary family from a metric name or query text. +It uses the materialization and readout declared by BackendPlan. + +## Ingestion contract + +Before accepting a payload, the data plane validates: + +- plan and plan-version identity; +- materialization, producer, and tenant identity; +- summary family, parameters, grouping, window, and representation; +- full/delta sequence and checkpoint requirements; and +- lifecycle and schema compatibility. + +Unknown, expired, reordered, or incompatible state is rejected and surfaced. +Receiving bytes is not evidence that the corresponding window is queryable. + +## Query routing + +Routing has three outcomes: + +1. **Summary readout** when BackendPlan contains a compatible route and the + required windows are fresh and complete. +2. **Exact fallback** when BackendPlan explicitly routes the query shape to an + exact backend. +3. **Explicit failure** when neither route is valid. + +A store miss, stale window, or incompatible payload must not be converted into +an empty or plausible approximate result. + +## Supported aggregation shapes + +The MVP exercises these shapes without defining Planner's query-to-summary +rules here: + +- within one series over time: + + ```promql + quantile_over_time(0.95, request_duration_seconds[5m]) + ``` + +- across label groups at an evaluation timestamp: + + ```promql + sum by (region) (http_requests_total) + ``` + +- across both a time range and label groups: + + ```promql + sum by (region) (rate(http_requests_total[5m])) + ``` + +Whether a particular expression is exact, summary-backed, or unsupported is +the selected plan's decision. The data plane only executes that decision. + +## Readiness and freshness + +A readout is ready only when all materializations required by its route: + +- belong to the active plan version; +- cover the requested logical interval; +- satisfy watermark and allowed-lateness policy; +- have no unresolved delta gap; and +- meet any declared source-completeness requirement. + +For example, a query at `12:05` over `[5m]` cannot reuse complete panes from an +earlier run merely because their labels match. The plan identity and logical +window must also match. + +## Result semantics + +The response preserves Prometheus labels, timestamps, result type, and error +behavior. Summary error guarantees come from the selected Planner result and +are carried by BackendPlan; the data plane neither tightens nor loosens them. + +When several physical shards contribute to one result, they may be merged only +if their materialization contracts match and the chosen summary supports the +declared merge. + +## Exact fallback + +Fallback is a correctness path, not a silent catch-all. BackendPlan identifies +the backend and query scope eligible for fallback. Transport failures and exact +query errors remain visible to the caller. + +Examples that may require exact fallback include an unsupported PromQL +operator, a request outside retained summary coverage, or a query whose exact +accuracy requirement has no compatible maintained state. + +## Non-goals + +This document does not define PromQL parsing, summary selection, Planner IR, +summary algorithms, state byte encoding, storage-engine implementation, or +protocol-specific server code. diff --git a/data_plane/docs/developer_docs/extension-points.md b/data_plane/docs/developer_docs/extension-points.md new file mode 100644 index 00000000..9cc577d9 --- /dev/null +++ b/data_plane/docs/developer_docs/extension-points.md @@ -0,0 +1,53 @@ +# Data-plane extension boundaries + +> Status: active +> +> MVP relation: Prometheus HTTP and the configured exact fallback are required; +> additional protocols and fallback systems are future extensions. + +## TL;DR + +The data plane separates network transport, request/response adaptation, +plan-aware execution, and exact fallback. An extension implements one boundary +without duplicating planning or bypassing BackendPlan validation. + +## Protocol server + +A protocol server owns network concerns: endpoints, authentication context, +request limits, cancellation, and transport errors. It hands a request to a +protocol adapter and returns the adapter's response. + +It does not parse Planner IR, select a summary, access summary storage directly, +or decide when fallback is allowed. + +## Protocol adapter + +An adapter converts a protocol request into the data plane's canonical query +request and converts the canonical result back into the protocol response. +Prometheus label and timestamp semantics must survive both conversions. + +An adapter may report that a language feature cannot be represented, but it +must not approximate or rewrite an unsupported query on its own. + +## Fallback client + +A fallback client executes the canonical query against the exact backend named +by BackendPlan. It preserves the logical evaluation time, range, tenant, and +error response. + +Fallback is invoked by plan-aware routing. A fallback client must not turn a +remote error into an empty successful result. + +## Adding an extension + +An extension is complete when it demonstrates: + +- request and response semantic round trips; +- cancellation, timeout, and error propagation; +- tenant and authentication context preservation; +- plan-aware routing rather than direct store access; +- no silent fallback or approximation; and +- integration coverage with one successful and one failing request. + +Implementation locations and trait signatures are intentionally left to the +code and API documentation, where they can evolve without changing this design. diff --git a/data_plane/docs/promsketch-integration.md b/data_plane/docs/promsketch-integration.md deleted file mode 100644 index 20657b82..00000000 --- a/data_plane/docs/promsketch-integration.md +++ /dev/null @@ -1,142 +0,0 @@ -# PromSketch Integration — Multi-Path Ingestion Architecture - -## 1. Overview - -QueryEngine supports two parallel data ingestion paths: - -1. **Precomputed pipeline**: A Kafka topic carrying pre-aggregated sketch buckets is consumed by `KafkaConsumer`, stored in `SketchStore`, and served through the standard query path. -2. **Raw sample pipeline (Prometheus Remote Write)**: A standalone HTTP endpoint (`/api/v1/write`) accepts standard Prometheus remote write requests (Snappy-compressed protobuf). Decoded samples are inserted into `PromSketchStore` (which maintains live EHUniv, EHKLL, and USampling sketch instances per series) and served through the sketch query path. - -When a query arrives, the engine tries the sketch path first, then falls through to the precomputed path, and finally (optionally) to a remote Prometheus server. - -## 2. Data Flow Diagram - -``` -Raw Samples Path (Prometheus Remote Write): - Prometheus / Agent --> POST /api/v1/write --> PrometheusRemoteWriteServer --> PromSketchStore - (Snappy + protobuf) decode & insert - | - sketch_insert() - (EHUniv, EHKLL, USampling) - -Precomputed Path: - Prometheus --> PrecomputeEngine --> Kafka [precomputed] --> KafkaConsumer --> SketchStore - -Query Path: - HTTP Request --> ASAPQueryEngine - |-- (1) handle_sketch_query_promql() --> PromSketchStore.eval_matching() - |-- (2) precomputed pipeline (SketchStore) - +-- (3) fallback --> Prometheus server -``` - -## 3. Query Routing - -When a PromQL query arrives, `ASAPQueryEngine` dispatches it as follows: - -1. **PromSketch path** — `handle_sketch_query_promql()` parses the query (AST first, regex fallback for custom functions). If the function name is in `promsketch_func_map` and the `PromSketchStore` has matching series data, results are returned immediately. -2. **Precomputed path** — If the sketch path returns `None` (function not sketch-backed, no store configured, or no matching series), the query falls through to `SketchStore`. -3. **Prometheus fallback** — If `--forward-unsupported-queries` is set and the precomputed path also misses, the query is forwarded to the remote Prometheus server. - -### Sketch-Backed Functions (13 total) - -These functions are routed to `PromSketchStore` first, with fallthrough to precomputed on miss: - -| Function | Sketch Type | Standard PromQL? | Description | -|-------------------------|-------------|-------------------|--------------------------------------------------| -| `entropy_over_time` | EHUniv | No (custom) | Shannon entropy of the sample distribution | -| `distinct_over_time` | EHUniv | No (custom) | Estimated number of distinct values | -| `l1_over_time` | EHUniv | No (custom) | L1 norm of the value vector | -| `l2_over_time` | EHUniv | No (custom) | L2 norm of the value vector | -| `quantile_over_time` | EHKLL | Yes | Approximate quantile (e.g., p50, p99) | -| `min_over_time` | EHKLL | Yes | Minimum value over the range | -| `max_over_time` | EHKLL | Yes | Maximum value over the range | -| `avg_over_time` | USampling | Yes | Average of sampled values | -| `count_over_time` | USampling | Yes | Count of sampled data points | -| `sum_over_time` | USampling | Yes | Sum of sampled values | -| `sum2_over_time` | USampling | No (custom) | Sum of squared values | -| `stddev_over_time` | USampling | Yes | Standard deviation over the range | -| `stdvar_over_time` | USampling | Yes | Variance over the range | - -### Non-Sketch Functions - -These functions always go directly to the precomputed pipeline (not in `promsketch_func_map`): - -| Function | Description | -|-------------|--------------------------------------| -| `rate` | Per-second rate of increase | -| `increase` | Total increase over the range | - -## 4. Configuration Reference - -### CLI Arguments - -> **NOTE:** The Prometheus / VictoriaMetrics remote-write ingest path was -> removed; backend ingest is OTLP-only now (sketch envelopes from -> asap-otel / asap-otap / asap-telegraf in ASAPCollector). Sections that -> assume a `/api/v1/write` listener on the query engine no longer apply — -> see `README.md` for the current architecture. - -| Argument | Description | Default | -|-------------------------------------|-------------------------------------------------------------------|-----------------| -| `--auto-init-sketches` | Auto-initialize all 3 sketch types for every new series | `true` | -| `--promsketch-config` | Path to a sketch configuration YAML file (optional) | (none) | - -### Sketch Config YAML - -All fields are optional; defaults are shown below. - -```yaml -eh_univ: - k: 50 # EH buckets for UnivMon - time_window: 1000000 # milliseconds - -eh_kll: - k: 50 # EH buckets for KLL - kll_k: 256 # KLL accuracy parameter - time_window: 1000000 - -sampling: - sample_rate: 0.2 # fraction of data points to sample - time_window: 1000000 -``` - -## 5. Deployment Checklist - -> **HISTORICAL:** The remote-write ingest path described below was removed. -> Drive ingest from ASAPCollector (asap-otel / asap-otap / asap-telegraf) -> over OTLP into the query engine's OTLP ports (gRPC 4317 / HTTP 4318) -> instead. The `--promsketch-config` flag is still honoured for sketch -> tuning when the precompute streaming engine is enabled. - -### Start QueryEngine - -```bash -./query_engine \ - --streaming-engine=precompute \ - --enable-otel-ingest \ - --promsketch-config promsketch_config.yaml # optional -``` - -### Verify queries - -```bash -curl 'http://localhost:8088/api/v1/query?query=quantile_over_time(0.5,metric[1m])&time=...' -``` - -### Monitor - -Use the `/metrics` endpoint for Prometheus counters (see section 6). - -## 6. `/metrics` Endpoint - -Exposed at `GET /metrics` in Prometheus exposition format. Key metrics: - -| Metric | Type | Description | -|-------------------------------------------------|-----------|-------------------------------------------------------| -| `promsketch_series_total` | Gauge | Number of live series currently tracked | -| `promsketch_samples_ingested_total` | Counter | Total raw samples ingested | -| `promsketch_ingest_errors_total` | Counter | Total ingestion errors (parse failures, etc.) | -| `promsketch_ingest_batch_duration_seconds` | Histogram | Time spent processing each ingestion batch | -| `promsketch_sketch_queries_total{result="hit"}` | Counter | Sketch queries that returned data | -| `promsketch_sketch_queries_total{result="miss"}`| Counter | Sketch queries that fell through (no matching series) | -| `promsketch_sketch_query_duration_seconds` | Histogram | End-to-end latency of sketch query evaluation | diff --git a/data_plane/docs/sketchindex-sid-unification-plan.md b/data_plane/docs/sketchindex-sid-unification-plan.md deleted file mode 100644 index 1b5072a6..00000000 --- a/data_plane/docs/sketchindex-sid-unification-plan.md +++ /dev/null @@ -1,208 +0,0 @@ -# Unification plan: SchemaRegistry → SketchIndex, agg_id → sid, MutableEpoch dedup - -This is a planning doc, not an implementation. It explains how three -in-flight migrations close out together, and what the data -plane looks like after. - -**Status as of May 2026:** all three migrations have landed their -"new side" — `SketchIndex`, `SketchInstanceMetadata`, `sid`, -generic `MutableEpoch

` — but the "old side" is still the live -production path. Concretely: - -- 45 files reference `aggregation_id`, 48 reference `agg_id` - workspace-wide. Only 12 files mention `sid`. -- Ingest barrier `SchemaRegistry::is_writable(agg_id)` is the §6.3 - write-side gate, called by every OTLP ingest at - `data_plane/src/drivers/ingest/otel.rs:526,591,615,1049,1062`. -- Query path emits `aggregation_id_for_key` / `aggregation_id_for_value` - on the wire response (asap_query_engine/engine.rs:1019,1064). -- `data_plane/src/storage_engines/sketch_db/store/{global,per_key}.rs` still - use the non-generic legacy `MutableEpoch` / `SealedEpoch` from - `store/common.rs`. The generic `MutableEpoch

` in - `index/epoch_columnar.rs` is used only by `SketchIndex`. -- There is a literal `// DEPRECATED: aggregation_id-keyed write — remove` - comment at `drivers/ingest/otel.rs:1054`, confirming the migration is - acknowledged but not finished. - -## The three overlapping migrations - -### M1 — Lifecycle metadata fold: `AggSchema` → `SketchInstanceMetadata` - -Today two structs describe overlapping per-aggregation metadata at -two granularities: - -| | `AggSchema` (sketch_db/schema/) | `SketchInstanceMetadata` (sketch_db/index/) | -|---|---|---| -| Primary key | `agg_id: u64` | `sid: u64` | -| Identity fields | metric_name, grouping_labels | metric_name, group-by KEY set, sketch_type, sketch_config, accuracy_bound | -| Lifecycle | `AggStatus { Active / Retired / Expired }`, retired_at_ms, expires_at_ms | none | -| Source of truth | `StreamingConfig` reconciliation | OTLP-ingest registration | - -After M1, `SketchInstanceMetadata` carries the lifecycle fields and -`SchemaRegistry` becomes `SketchInstanceRegistry` (or folds into -`SketchIndex::instances`). The `is_writable(sid)` gate uses sid; -the §6.3 invariant is preserved. - -### M2 — Identifier replacement: `agg_id` → `sid` - -`agg_id: u64` is a hash of `(metric, agg_type, grouping_labels)` -computed in `crates/asap_types/src/aggregation_config.rs` at -config-load time. It is stable across restarts because the inputs -are stable. - -`sid: u64` is assigned at OTLP-ingest registration time (collector -gateway), travels in the wire format, and is canonical from that -point forward. - -The migration order matters: -1. Both ids carry simultaneously through the pipeline (already - happens — wire format has both). -2. Switch the §6.3 barrier from `is_writable(agg_id)` to - `is_writable(sid)` — requires `SketchInstanceRegistry` keyed by - sid. (M1 prerequisite.) -3. Switch the store keys: `SketchStore::insert_precomputed_output_batch` - currently keys on `aggregation_id`; flip to sid. -4. Switch the query path: drop `aggregation_id_for_key / - aggregation_id_for_value` on the wire response in favor of sid. -5. Drop the `aggregation_id` field from `AggregationConfig` and - `PrecomputedOutput` (wire-format change — requires coordinated - collector release). - -### M3 — Storage primitive dedup: legacy `MutableEpoch` → generic `MutableEpoch

` - -The `index/epoch_columnar.rs::MutableEpoch

` is a generic version -of `store/common.rs::MutableEpoch` lifted from the legacy code and -parameterized on payload type `P`. Six storage optimizations preserved -(see `INDEX_DESIGN.md`). - -Today the legacy uses `P = Arc` (trait-object -dispatch). The new `SketchIndex` uses `P = SketchSampleState` (typed -bytes + encoding tag — no dyn dispatch, no Arc cloning). - -Two paths to the dedup: - -- **Eager M3:** rewrite `store/{global,per_key}.rs` to use - `MutableEpoch>` and delete the legacy copy. - ~3–5 hours of careful work on the hot ingest + query path, with - regression risk in subtle hot-path behavior. -- **Deferred M3:** add a deprecation banner to `store/common.rs`; let - the legacy version die naturally when M2 + M1 close out. After M1 - + M2, the path-of-record is `SketchIndex`-resident sketch state - (P = `SketchSampleState`), and the trait-object `AggregateCore` - payload type becomes vestigial. The `store/{global,per_key}.rs` - files either delete or become thin shims. - -## Sequencing + dependencies - -``` - +-------------------+ - | M1: AggSchema | - | → InstanceMeta | - +---------+---------+ - | - (lifecycle fields land on sid-keyed instance metadata) - | - v - +-------------------+ - | M2: agg_id → sid | - | (5 substeps) | - +---------+---------+ - | - (wire format + store keys + query path all on sid) - | - v - +-------------------+ - | M3: dedup | - | MutableEpoch | ← happens by itself - +-------------------+ - once `AggregateCore`-as-payload is gone -``` - -M1 blocks M2 (M2 needs sid-keyed lifecycle gate). M2 substeps 4 + 5 -require a coordinated ASAPCollector release because the wire format -changes. M3 happens automatically once M2 lands; or can be done -eagerly any time, at the cost of working on the live hot path twice. - -## Ordering against other in-flight chains - -From the May 12 controller_todo doc: - -- **Step Z legacy_expr retirement** (4 PRs, ~8000 LOC across ~340 - pattern-match sites) is orthogonal — operates on the control-plane- - side intent algebra, not on data-plane identifiers. Can run in - parallel. -- **Analyzer-unification α→ε** (5 PRs, 5–8 days) interacts only at - the `Capability` enum surface in `sketch_db/index/`, which both - the engine-side analyzer and the control-plane-side analyzer consume. - Coordinate the `Capability` shape once at α; downstream is - independent. - -## Concrete unification work, file-by-file - -### Step A — M1 prep (does not change the wire format) - -1. Add lifecycle fields to `SketchInstanceMetadata`: - - `status: AggStatus` - - `retired_at_ms: Option` - - `expires_at_ms: Option` -2. Add `SketchIndex::is_writable(sid: u64) -> bool` mirroring - `SchemaRegistry::is_writable(agg_id)`. Internally consult the - status field on the matched `SketchInstanceMetadata`. -3. Add `SketchIndex::list_by_status(status: AggStatus) -> Vec` - so eviction can drive off it. -4. Tests: replicate every `SchemaRegistry` test against `SketchIndex`. - -### Step B — M1 cutover - -1. Switch `SchemaEvictionService` to call `SketchIndex` methods. -2. Switch ingest barrier `is_writable(agg_id)` → `is_writable(sid)`. -3. Delete `SchemaRegistry`, `AggSchema`, `AggStatus` from - `sketch_db/schema/`. Folder remains for compat re-exports during - transition; can be deleted once all callers migrated. - -### Step C — M2.1 (parallel-write) - -Ingest emits both `agg_id` and `sid` on every precompute (already -the case today). Store accepts either as a key; internally maps -agg_id → sid via a side table. Query path resolves either. - -### Step D — M2.2 cutover - -Store keys flip to sid. Wire format drops `aggregation_id` fields. -Coordinated release with ASAPCollector. After this lands, -`aggregation_id` is dead. - -### Step E — M3 freebie - -`Arc` payloads are no longer the path-of-record; -they only exist in the legacy `store/{global,per_key}.rs` code, which -either deletes (ASAP-tier sketch state now lives in -`SketchIndex.series.windows`) or becomes a thin adapter shim. The -`store/common.rs::MutableEpoch` duplication disappears with its only -caller. - -## Estimate - -- Step A: 1 day (additive, low risk). -- Step B: 1 day (cutover; SchemaRegistry deletion is touchy but - mechanical). -- Step C: 1 day (parallel-write is already partially in place). -- Step D: 1 day + ASAPCollector PR coordination + integration - testing window. -- Step E: 0.5 day (mostly deletion). - -Total: ~5 working days end-to-end, plus coordination overhead. - -## Open questions for the reader - -1. Does the wire format have a stability commitment that constrains - Step D? (Backwards-compat shim period? Versioned acceptance?) -2. After Step E, does anything outside `SketchIndex` need to handle - the trait-object `AggregateCore` payload? E.g., the - `PrecomputeEngine` write path serializes via `SerializableToSink` - — does that path move to typed bytes too? -3. What's the agreed retention model for `sid` after a schema - transition? (Today: AggSchema lifecycle drops the agg_id when - Expired. Post-migration: SketchIndex drops the sid. Same - semantics, but verify nothing depends on the agg_id being - reusable post-eviction.) diff --git a/data_plane/docs/user_guide/querying-asap.md b/data_plane/docs/user_guide/querying-asap.md new file mode 100644 index 00000000..51ace1af --- /dev/null +++ b/data_plane/docs/user_guide/querying-asap.md @@ -0,0 +1,83 @@ +# Querying ASAP + +## TL;DR + +Send PromQL through the Prometheus-compatible API as you would to an exact +backend. ASAPQuery answers a request from planned summary state when that state +is compatible, complete, and fresh. Otherwise it uses the configured exact +fallback when the active plan permits it, or returns an explicit error. + +## Query behavior + +A request has one of three outcomes: + +1. **Summary-backed result:** the active plan contains a matching readout and + all required state is ready. +2. **Exact result:** the active plan routes the query to the configured + Prometheus/VictoriaMetrics-compatible fallback. +3. **Error:** neither route can produce a correct answer. + +The data plane never turns missing or stale state into a zero-valued result. + +## PromQL examples + +ASAP's MVP workload includes aggregation within a series over time: + +```promql +quantile_over_time(0.95, request_duration_seconds[5m]) +``` + +aggregation across label groups: + +```promql +sum by (region) (http_requests_total) +``` + +and aggregation across both time and label groups: + +```promql +sum by (region) (rate(http_requests_total[5m])) +``` + +These examples do not promise that every deployment accelerates each query. +The submitted workload, selected Planner result, available summaries, and +active BackendPlan determine the route. + +## Accuracy + +Some maintained summaries are exact; others have a declared approximation +guarantee. The query response must correspond to the guarantee selected for +that query. ASAPQuery does not silently substitute a less accurate summary. + +For validation, compare results with the exact backend over identical samples, +labels, timestamps, and logical query ranges. Missing or additional series are +errors, not values to ignore. + +## Freshness + +A summary-backed result is served only when its state: + +- belongs to the active plan and current run; +- covers the requested logical interval; +- has passed its watermark and allowed-lateness rules; and +- has no missing delta or producer state. + +A completed window from an earlier plan is not a fresh answer for the current +plan merely because its metric labels match. + +## Fallback and errors + +Fallback is explicit plan behavior. Unsupported operators, unavailable summary +coverage, or exact accuracy requirements may be routed to the exact backend. +Fallback transport and query failures remain visible; they are not returned as +successful empty vectors. + +Operational errors should identify the failed boundary where possible, such as +unsupported query shape, inactive plan, stale window, missing delta sequence, +incompatible materialization, or exact-backend failure. + +## Related documentation + +- [Data-plane design](../design_docs/query-execution.md) +- [Summary storage](../../../docs/design_docs/summary-storage.md) +- [ASAPCollector MVP demo runbook](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/user_guide/mvp-demo-runbook.md) diff --git a/data_plane/src/drivers/ingest/otel.rs b/data_plane/src/drivers/ingest/otel.rs index 14b627f2..c62a077b 100644 --- a/data_plane/src/drivers/ingest/otel.rs +++ b/data_plane/src/drivers/ingest/otel.rs @@ -1862,7 +1862,7 @@ fn derive_sketch_policy_fp( /// query (`quantile_over_time`, `count`/HLL, `topk`) capability-misses /// and the user sees `data_source: asap_query, "No result"`. /// -/// Per `docs/design-controller-into-backend.md` §1 the sketch *family* +/// Per `docs/design_docs/series-identity.md`, the summary *family* /// is a wire-level attribute (carried here in `agg_kind` / /// [`SketchKindHandle`]), NOT a name suffix; storage + query must be /// keyed on the raw SDK metric name. This helper applies that @@ -2758,7 +2758,7 @@ mod canonical_metric_name_tests { #[test] fn leaves_bare_name_untouched_idempotent() { - // Once agents stop suffixing (design-controller-into-backend + // Once agents stop suffixing (series-identity // Phase 1), the strip must be a no-op. assert_eq!( canonical_sketch_metric_name("request_size_bytes", SketchKind::Kll), diff --git a/data_plane/src/drivers/ingest/series_resolver.rs b/data_plane/src/drivers/ingest/series_resolver.rs index fb949f5b..121bd3c5 100644 --- a/data_plane/src/drivers/ingest/series_resolver.rs +++ b/data_plane/src/drivers/ingest/series_resolver.rs @@ -22,7 +22,7 @@ //! internal/series/dictionary.go`. //! //! See design doc §5.4 ("Idempotency invariant on `ResolveSeriesIDs`") -//! at `docs/design-controller-into-backend.md`. +//! in `docs/design_docs/series-identity.md`. use dashmap::DashMap; use std::fs::{File, OpenOptions}; diff --git a/data_plane/src/drivers/query/adapters/prometheus_http.rs b/data_plane/src/drivers/query/adapters/prometheus_http.rs index 63a08811..8b5083bf 100644 --- a/data_plane/src/drivers/query/adapters/prometheus_http.rs +++ b/data_plane/src/drivers/query/adapters/prometheus_http.rs @@ -48,7 +48,7 @@ pub struct PrometheusResponse { #[serde(default, skip_serializing_if = "Vec::is_empty")] pub infos: Vec, /// ASAP extension: theoretical accuracy envelope for the - /// answer (§6.4 of docs/design-sketch-db.md). Unknown to + /// answer (see docs/design_docs/summary-storage.md). Unknown to /// standard Prometheus clients (they ignore unknown fields), /// consumed by Grafana panels / paper artifacts that want /// the machine-readable (ε, δ) bound. diff --git a/data_plane/src/query_engines/asap_query_engine/engine.rs b/data_plane/src/query_engines/asap_query_engine/engine.rs index 6edc6a3c..a041f80a 100644 --- a/data_plane/src/query_engines/asap_query_engine/engine.rs +++ b/data_plane/src/query_engines/asap_query_engine/engine.rs @@ -282,7 +282,7 @@ impl ASAPQueryEngine { query_kwargs: &HashMap, ) -> Result> { // Phase 1b of the sketch DB design - // (docs/design-sketch-db.md §5.1 / §16 Phase 1): + // (docs/design_docs/summary-storage.md): // for single-subpopulation queries on additive statistics // (Count / Sum / Min / Max), serve from the typed aux // columns without deserialising the sketch payload. diff --git a/data_plane/src/storage_engines/sketch_db/accuracy.rs b/data_plane/src/storage_engines/sketch_db/accuracy.rs index b24512ab..10e0f3e9 100644 --- a/data_plane/src/storage_engines/sketch_db/accuracy.rs +++ b/data_plane/src/storage_engines/sketch_db/accuracy.rs @@ -1,8 +1,8 @@ //! `AccuracyProfile` — derived error / confidence bound for each //! `AggregationConfig`. //! -//! Implements §6.4 of the sketch DB design -//! ([`design-sketch-db.md`](../../../../../docs/design-sketch-db.md)). +//! Implements backend accuracy metadata consumed under BackendPlan. Logical +//! guarantees are owned by ASAPPlanner and family bounds by summary libraries. //! Given the `aggregation_type` + `parameters` pinned on an //! `AggSchema`, the registry can expose the theoretical accuracy //! bound of every query answer computed from it — so users and @@ -40,7 +40,7 @@ //! rather than loose textbook versions; sources are cited inline //! in each branch of [`AccuracyProfile::derive`]. //! -// See `docs/proofs.md` for formal statements of the accuracy bounds + combine_statistic / write-barrier / backfill theorems. +// See `docs/design_docs/summary-storage.md` for backend storage guarantees. use serde::{Deserialize, Serialize}; diff --git a/data_plane/src/storage_engines/sketch_db/backfill/mod.rs b/data_plane/src/storage_engines/sketch_db/backfill/mod.rs index 4ab3db95..2bc82bc5 100644 --- a/data_plane/src/storage_engines/sketch_db/backfill/mod.rs +++ b/data_plane/src/storage_engines/sketch_db/backfill/mod.rs @@ -1,7 +1,7 @@ //! `BackfillJob` lifecycle types + in-memory `BackfillRegistry`. //! //! Implements §10 (refreshable view maintenance / backfill path) of the -//! sketch DB design ([`design-sketch-db.md`](../../../../../docs/design-sketch-db.md)). +//! future storage scope ([`future-storage-and-compression.md`](../../../../../docs/design_docs/future-storage-and-compression.md)). //! //! ## Why this exists //! diff --git a/data_plane/src/storage_engines/sketch_db/backfill/processor.rs b/data_plane/src/storage_engines/sketch_db/backfill/processor.rs index b1e58a7c..4899e3f0 100644 --- a/data_plane/src/storage_engines/sketch_db/backfill/processor.rs +++ b/data_plane/src/storage_engines/sketch_db/backfill/processor.rs @@ -3,7 +3,7 @@ //! them into the store. //! //! Implements §10 (refreshable view maintenance) of the sketch DB -//! design ([`design-sketch-db.md`](../../../../../docs/design-sketch-db.md)). +//! design ([`future-storage-and-compression.md`](../../../../../docs/design_docs/future-storage-and-compression.md)). //! Reads raw samples from a [`RawSampleReader`] (Phase 5b), groups //! them by the agg's `grouping_labels` just like live ingest does, //! and writes per-(group, window) precomputes to the store. diff --git a/data_plane/src/storage_engines/sketch_db/backfill/raw_sample_reader.rs b/data_plane/src/storage_engines/sketch_db/backfill/raw_sample_reader.rs index 4de59843..a45f4e77 100644 --- a/data_plane/src/storage_engines/sketch_db/backfill/raw_sample_reader.rs +++ b/data_plane/src/storage_engines/sketch_db/backfill/raw_sample_reader.rs @@ -1,7 +1,7 @@ //! `RawSampleReader` — trait + mock implementation for reading raw //! samples from the exact DB during a [`BackfillJob`] run. //! -//! Implements §10 of the sketch DB design ([`design-sketch-db.md`](../../../../../docs/design-sketch-db.md)) +//! Supports the future backfill scope ([`future-storage-and-compression.md`](../../../../../docs/design_docs/future-storage-and-compression.md)). //! and specifically §10.2's `BackfillSource` dispatch: every //! concrete source (S3+Gorilla, Prometheus, ClickHouse, OtherSketch) //! will eventually implement this trait so the worker pool (Phase 5c) diff --git a/data_plane/src/storage_engines/sketch_db/backfill/window_builder.rs b/data_plane/src/storage_engines/sketch_db/backfill/window_builder.rs index 02132d53..0d00e57c 100644 --- a/data_plane/src/storage_engines/sketch_db/backfill/window_builder.rs +++ b/data_plane/src/storage_engines/sketch_db/backfill/window_builder.rs @@ -2,7 +2,7 @@ //! //! Implements the "real rebuild" piece of §10 (refreshable view //! maintenance) from the sketch DB design -//! ([`design-sketch-db.md`](../../../../../docs/design-sketch-db.md)). +//! ([`future-storage-and-compression.md`](../../../../../docs/design_docs/future-storage-and-compression.md)). //! Given an `AggregationConfig` and a batch of raw samples for one //! `(agg_id, window)` pair, produces the `Box` //! that would have been produced had those samples flowed through diff --git a/data_plane/src/storage_engines/sketch_db/backfill/worker.rs b/data_plane/src/storage_engines/sketch_db/backfill/worker.rs index 97ca7efa..ba09bdd4 100644 --- a/data_plane/src/storage_engines/sketch_db/backfill/worker.rs +++ b/data_plane/src/storage_engines/sketch_db/backfill/worker.rs @@ -2,7 +2,7 @@ //! registry state machine. //! //! Implements §10.3 (refresh as a separate worker pool) of the -//! sketch DB design ([`design-sketch-db.md`](../../../../../docs/design-sketch-db.md)). +//! future storage scope ([`future-storage-and-compression.md`](../../../../../docs/design_docs/future-storage-and-compression.md)). //! Phase 5c scope: one job at a time, synchronous windowing loop, //! pluggable processor. The multi-worker pool with priority + //! isolation from live ingest (§11.4) lands in a follow-up. diff --git a/data_plane/src/storage_engines/sketch_db/index/mod.rs b/data_plane/src/storage_engines/sketch_db/index/mod.rs index eaeb0964..513696f6 100644 --- a/data_plane/src/storage_engines/sketch_db/index/mod.rs +++ b/data_plane/src/storage_engines/sketch_db/index/mod.rs @@ -17,7 +17,7 @@ //! and falls through to Thanos archive (Phase 6). //! //! See design doc §4.6 ("OTLP metadata model + backend store layout") at -//! `docs/design-controller-into-backend.md`. +//! `docs/design_docs/series-identity.md`. use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::sync::{Arc, RwLock}; diff --git a/data_plane/src/storage_engines/sketch_db/mod.rs b/data_plane/src/storage_engines/sketch_db/mod.rs index 386aa20f..63baa633 100644 --- a/data_plane/src/storage_engines/sketch_db/mod.rs +++ b/data_plane/src/storage_engines/sketch_db/mod.rs @@ -1,7 +1,7 @@ -//! Sketch DB scaffolding (Phase 2 of the sketch DB design). +//! Summary-storage components. //! -//! See [`docs/design-sketch-db.md`](../../../../../docs/design-sketch-db.md) -//! for the full architecture. This module houses the components that live +//! See [`summary-storage.md`](../../../../../docs/design_docs/summary-storage.md) +//! for the semantic contract. This module houses the components that live //! "above" the existing `SketchStore` and turn it into a sketch-aware //! storage engine over time: //! diff --git a/data_plane/src/storage_engines/sketch_db/persistence/mod.rs b/data_plane/src/storage_engines/sketch_db/persistence/mod.rs index 83613fa8..349feb92 100644 --- a/data_plane/src/storage_engines/sketch_db/persistence/mod.rs +++ b/data_plane/src/storage_engines/sketch_db/persistence/mod.rs @@ -1,6 +1,6 @@ //! Persistence layer for `SketchStorePerKey`. //! -//! See `docs/design-simple-map-store-persistence.md` for the design rationale. +//! See `docs/design_docs/future-storage-and-compression.md` for the design scope. //! //! ## Structure //! diff --git a/data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs b/data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs index 47009aad..dca57823 100644 --- a/data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs +++ b/data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs @@ -1,5 +1,5 @@ //! Cross-schema result combination for the §7 schema-timeline query -//! dispatch ([`design-sketch-db.md`](../../../../docs/design-sketch-db.md)). +//! dispatch ([`summary-storage.md`](../../../../docs/design_docs/summary-storage.md)). //! //! When a metric-range query spans a reconfigure boundary, the //! `SchemaRegistry::timeline_for_metric` call returns multiple diff --git a/data_plane/src/storage_engines/types/traits.rs b/data_plane/src/storage_engines/types/traits.rs index 00a26f00..f1a8b2ad 100644 --- a/data_plane/src/storage_engines/types/traits.rs +++ b/data_plane/src/storage_engines/types/traits.rs @@ -80,7 +80,7 @@ pub trait AggregateCore: SerializableToSink + Send + Sync { /// `query_statistic` method. /// /// This is the phase-1 piece of the sketch DB design - /// (docs/design-sketch-db.md §5.1 / §16 Phase 1). + /// (docs/design_docs/summary-storage.md). fn aux_stats(&self) -> AuxStats { AuxStats::empty() } diff --git a/docs/01-getting-started/architecture.md b/docs/01-getting-started/architecture.md index 86837141..31f4ce2a 100644 --- a/docs/01-getting-started/architecture.md +++ b/docs/01-getting-started/architecture.md @@ -1,258 +1,86 @@ -# Architecture - -This document provides a comprehensive overview of ASAP's architecture, data flows, and design decisions. - -## Table of Contents -- [High-Level Architecture](#high-level-architecture) -- [Data Flows](#data-flows) -- [Component Overview](#component-overview) -- [Key Design Decisions](#key-design-decisions) -- [Technology Stack](#technology-stack) -- [Repository Structure](#repository-structure) - -## High-Level Architecture - -ASAP consists of six main components working together to accelerate Prometheus queries: - -```mermaid -graph TB - subgraph "Data Sources" - E[Prometheus Exporters] - end - - subgraph "Existing Infrastructure" - P[Prometheus] - G[Grafana] - end - - subgraph "ASAP Components" - A[Arroyo Streaming] - K[Kafka] - Q[QueryEngine] - C[Control Plane] - AS[ArroyoSketch] - end - - E -->|metrics| P - P -->|remote_write| A - A -->|sketches| K - K -->|consume| Q - G -->|PromQL| Q - Q -->|results| G - Q -.->|fallback| P - - C -->|streaming_config.yaml| AS - AS -->|create pipelines| A - - style A fill:#e1f5ff - style Q fill:#e1f5ff - style C fill:#fff4e1 - style AS fill:#fff4e1 +# ASAP system architecture + +> Status: active + +## TL;DR + +ASAP separates logical planning, physical control, edge summary maintenance, +and backend query execution. ASAPQuery-backend contains the physical control +plane and data plane; it consumes ASAPPlanner decisions and coordinates with +ASAPCollector. + +```text +Query workload + | + v +ASAPPlanner -- selected logical workload plan + | + v +ASAPQuery control plane + | physical compile + +---------------------------+ + | | + v v +CollectorPlan BackendPlan + | | + v v +ASAPCollector -- summary state --> ASAPQuery data plane + | +PromQL client ------------------------+ + | + summary readout or exact fallback ``` -## Data Flows +## Planning path -ASAP has three primary data flows: **Ingestion**, **Query Execution**, and **Configuration**. +The workload is planned as a whole so common state can be shared. ASAPPlanner +owns logical parsing, summary alternatives, accuracy reasoning, and selection. +The ASAPQuery control plane owns deployment capabilities, physical placement, +windows, transmission, matching runtime plans, and activation. -### Ingestion Path +See the [control-plane design](../../control_plane/docs/README.md). -How metrics flow from exporters to sketches: +## Ingestion path -```mermaid -sequenceDiagram - participant E as Exporters - participant P as Prometheus - participant A as Arroyo - participant K as Kafka - participant Q as QueryEngine +ASAPCollector applies CollectorPlan, maintains summaries from observed OTLP +metrics, and transmits raw observations, full state, or deltas as directed. The +data plane accepts a payload only when it matches the active BackendPlan and +stores it under the declared materialization and logical window. - E->>P: Expose metrics - P->>P: Scrape metrics - P->>A: Remote write (HTTP) - A->>A: Build sketches (SQL pipeline) - A->>K: Produce sketches - K->>Q: Consume sketches - Q->>Q: Store in SketchStore -``` +ASAPQuery-backend does not treat legacy Telegraf, OTAP, Kafka, Prometheus +remote-write, or in-backend raw precomputation paths as the MVP architecture. +Future adapters may exist behind explicit interfaces without changing the +primary OTel path. -**Step-by-step:** +## Query path -1. **Exporters** expose metrics on HTTP endpoints (e.g., `:9100/metrics`) -2. **Prometheus** scrapes metrics at a specified time interval (e.g. every 10s) -3. **Prometheus** sends metrics to **Arroyo** via remote write API -4. **Arroyo** receives raw metrics via custom connector (`prometheus_remote_write_optimized`) -5. **Arroyo** executes SQL pipelines that build sketches in real-time (configured by **ArroyoSketch**) -6. **Arroyo** produces sketches to **Kafka** output topic -7. **QueryEngine** consumes sketches from **Kafka** -8. **QueryEngine** stores sketches in **SketchStore** (in-memory) +A PromQL request enters through a protocol server and adapter. The data plane +uses BackendPlan to locate a compatible readout and checks plan identity, +coverage, freshness, and state compatibility before execution. If the selected +logical plan requires exact execution, the request is sent to the configured +exact backend. -**Data format transformations:** -- **Exporter → Prometheus**: Prometheus exposition format (text) -- **Prometheus → Arroyo**: Prometheus remote write protobuf -- **Arroyo → Kafka**: Serialized sketches (custom format) -- **Kafka → QueryEngine**: Deserialize to custom sketch objects +See [plan-aware query execution](../../data_plane/docs/design_docs/query-execution.md). -### Query Path +## State and identity -How queries are executed: +Three identities remain distinct: -```mermaid -sequenceDiagram - participant G as Grafana - participant Q as QueryEngine - participant S as SketchStore - participant P as Prometheus +- a **plan identity** versions one deployed physical decision; +- a **materialization identity** describes maintained summary semantics; and +- a **series identity (`sid`)** identifies one canonical metric series. - G->>Q: PromQL query (HTTP) - Q->>Q: Parse query (PromQL adapter) - Q->>Q: Check if supported - - alt Supported query - Q->>S: Fetch sketches - S->>Q: Return sketches - Q->>Q: Execute query (ASAPQueryEngine) - Q->>G: Approximate result - else Unsupported query - Q->>P: Forward query (fallback) - P->>Q: Exact result - Q->>G: Exact result - end -``` - -**Step-by-step:** - -1. **Grafana** sends PromQL query to **QueryEngine** (port 8088) -2. **PrometheusHttpAdapter** parses the HTTP request and extracts the query -3. **ASAPQueryEngine** checks if the query can be answered with sketches -4. **If supported:** - - Fetch relevant sketches from **SketchStore** - - Execute query using sketch operations - - Format result as Prometheus-compatible JSON -5. **If unsupported:** - - Forward query to **Prometheus** via fallback client - - Return exact result from Prometheus -6. **QueryEngine** returns result to **Grafana** - -**Query support examples:** -- ✅ Supported: `quantile(0.99, http_request_duration)`, `sum(rate(...))` -- ❌ Unsupported: `up == 1`, `label_replace(...)`, exact histograms - -### Configuration Path - -How sketches are configured: - -```mermaid -graph LR - U[User] -->|edit| CC[controller-config.yaml] - CC --> C[Control Plane] - C -->|analyze queries| C - C -->|streaming_config.yaml| AS[ArroyoSketch] - C -->|inference_config.yaml| Q[QueryEngine] - AS -->|generate SQL| AS - AS -->|Arroyo API| A[Arroyo] - A -->|running pipelines| A - - style CC fill:#fff - style C fill:#fff4e1 - style AS fill:#fff4e1 -``` +Conflating them can cause incompatible state reuse. The storage and identity +contracts are described in [summary storage](../design_docs/summary-storage.md) +and [series identity](../design_docs/series-identity.md). -**Step-by-step:** +## Component failure behavior -1. **User** creates `controller-config.yaml` with: - - List of queries to accelerate - - Metric metadata (labels, types) +- Planner or physical-compilation failures prevent a new plan from staging. +- Partial plan application leaves the previous valid plan authoritative. +- Incompatible or gapped payloads are rejected by ingestion. +- Missing or stale state prevents summary readout. +- Exact fallback errors are returned as errors, not empty results. -2. **Control Plane** analyzes the query workload: - - Determines which sketch algorithms to use (DDSketch, KLL, etc.) - - Computes sketch parameters (size, accuracy) - - Generates `streaming_config.yaml` for Arroyo - - Generates `inference_config.yaml` for QueryEngine - -3. **ArroyoSketch** reads `streaming_config.yaml`: - - Renders SQL templates using Jinja2 - - Creates Arroyo pipelines via REST API - - Configures sketch UDFs with parameters - -4. **QueryEngine** reads `inference_config.yaml`: - - Knows which sketches to expect from Kafka - - Configures deserialization logic - - Sets up query routing - -## Components - -ASAPQuery-backend is a Cargo workspace of two binaries plus shared -crates (see the repository tree above): - -| Component | Purpose | Location | -|-----------|---------|----------| -| **data plane** | Query backend: OTLP ingest, warm-tier `ASAPQueryEngine` over `SketchStore`, archive-tier `ThanosQueryEngine` forwarder | `data_plane/` | -| **control plane** | Planner: lowers PromQL/SQL to an intent algebra, plans sketches, pushes per-runtime config to agents over OpAMP | `control_plane/` | -| **shared crates** | `asap_types`, `promql_utilities`, `asap_otel_proto` | `crates/` | - -The edge side (agents, gateway, sketch processors, exporters) lives in -[ASAPCollector](https://github.com/ProjectASAP/ASAPCollector); the -archive tier uses external Thanos + object storage. - -## Key Design Decisions - -### Fallback Mechanism - -**Design decision**: Always support fallback to Prometheus - -**Rationale**: -- Not all queries can be accelerated (e.g., label manipulation) -- Users shouldn't have to know which queries are supported -- Gradual adoption - users can try ASAP without changing queries - -**Implementation**: -- QueryEngine detects unsupported queries during parsing -- Forwards to Prometheus via HTTP client -- Returns results transparently - -**Trade-off**: Added complexity vs. compatibility -- **Benefit outweighs cost**: Users can point Grafana at ASAP without modifying dashboards - -## Technology Stack - -### Core Languages -- **Rust** — `data_plane` and `control_plane` (this repo) - - Tokio for async runtime - - Axum for HTTP server - - Serde for serialization - - DataSketches (dsrs) for sketch algorithms - - Hydra for experiment config composition - -### Infrastructure -- **Apache Kafka** - Message broker (KRaft mode, no Zookeeper) -- **Prometheus** - Time-series database -- **Grafana** - Visualization (unchanged from user's existing setup) - -### Development Tools -- **Cargo** - Rust build system -- **Docker** - Containerization -- **GitHub Actions** - CI/CD -- **Pre-commit** - Git hooks for linting - -## Repository Structure - -``` -ASAPQuery-backend/ # Cargo workspace -├── crates/ # Shared workspace libraries -│ ├── asap_types/ # StorageBackend enum, accuracy envelopes -│ ├── promql_utilities/ # PromQL AST helpers -│ └── asap_otel_proto/ # OTLP protobuf bindings -├── data_plane/ # Query backend (binary) -│ └── src/ -│ ├── drivers/ # ingest, query adapters/servers, control_plane_client -│ ├── query_engines/ # ASAPQueryEngine (warm) + ThanosQueryEngine (archive) + routing -│ ├── storage_engines/ # SketchStore (sketch_db) + gorilla_object_store + types -│ ├── precompute_engine/ # Streaming pipeline (+ operators/) -│ └── tests/ # Integration tests -├── control_plane/ # In-repo control plane / planner (binary) -│ └── src/ # query_parser, intent_algebra, sketch_algebra, -│ # optimizer, physical, emit, opamp -└── docs/ # Developer documentation (this) - ├── 01-getting-started/ - └── 03-how-to-guides/ -``` +The system fails closed at every boundary where a plausible but incorrect +answer could otherwise be produced. diff --git a/docs/01-getting-started/overview.md b/docs/01-getting-started/overview.md index 9697e3a3..35965374 100644 --- a/docs/01-getting-started/overview.md +++ b/docs/01-getting-started/overview.md @@ -1,110 +1,51 @@ -# Overview & Key Concepts +# ASAP overview -Welcome to ASAP! This guide will help you understand what ASAP is, why it exists, and the key concepts you need to know. +## TL;DR -## What is ASAP? +ASAP is a summary-based metrics pipeline. ASAPCollector maintains selected +summaries near the source, and ASAPQuery-backend answers planned PromQL queries +from those summaries. An exact backend remains available for query shapes or +time ranges that cannot be served correctly from maintained state. -**ASAP** is a **drop-in query accelerator** for Prometheus that delivers: +## Why summaries -- ⚡ **Sub-second latency** for complex quantile and aggregate queries -- 💾 **100x memory reduction** compared to raw time-series data -- 🎯 **Configurable accuracy** -- 🔌 **Full Prometheus compatibility** - works with existing Grafana dashboards and PromQL queries +Exact metrics systems retain raw samples and repeatedly scan them for queries. +For selected recurring workloads, ASAP maintains compact reusable state as +samples arrive. This can reduce query work, transmission, and storage while +meeting a declared accuracy and freshness contract. -ASAP sits between Prometheus and Grafana, intercepting queries and answering them using pre-computed streaming sketches instead of scanning raw data. +Summaries may be exact accumulators or approximate structures such as DDSketch, +KLL, HLL, Count-Min Sketch, or CountSketch. The term “summary” is broader than +“sketch” and includes both exact and approximate maintained state. -## Why ASAP? +## Components -### The Problem +- **ASAPPlanner** chooses a valid logical plan for the complete query workload. +- **ASAPQuery control plane** compiles that selection into matching collector + and backend plans and coordinates activation. +- **ASAPCollector** observes metrics, maintains the requested summaries, and + transmits raw, full-summary, or delta-summary payloads. +- **ASAPQuery data plane** validates and stores those payloads, executes planned + readouts, and exposes Prometheus-compatible query responses. +- **Exact backend** serves the explicit fallback path and provides the MVP + correctness baseline. -Prometheus struggles with: -- **High-cardinality metrics** - Queries slow down as cardinality increases -- **Quantile queries** - Computing percentiles requires scanning massive amounts of data -- **Long time windows** - `quantile_over_time(...[1h])` can take seconds or fail -- **Memory pressure** - Storing raw samples for all time series +## Correctness contract -### The Solution +A summary-backed answer is valid only when: -ASAP uses **streaming sketches** to: -1. **Pre-compute approximate summaries** as data arrives -2. **Answer queries in milliseconds** using compact sketches instead of raw data -3. **Bound memory usage** - sketches are fixed-size regardless of data volume -4. **Maintain accuracy** - configurable error bounds (typically <1% error) +- it comes from the active plan and matching materialization; +- its labels and logical time range match the request; +- required windows are complete and fresh; +- its summary semantics satisfy the requested accuracy; and +- every required producer and delta sequence is compatible. -## Key Concepts +Otherwise the request fails or follows the configured exact route. Missing or +stale state is never treated as a zero-valued answer. -### Sketches +## MVP claim -**Sketches** are probabilistic data structures that provide approximate answers with bounded error. Think of them as compact summaries that capture the essential characteristics of data distributions. - -Examples: -- **DDSketch** and **KLL Sketch** - For quantiles (P50, P95, P99) -- **Count-Min Sketch** and **CountSketch** - For frequency/sum estimation - -Key properties: -- **Fixed size** - Memory usage doesn't grow with data volume or grows very slowly -- **Mergeable** - Can merge multiple sketches into a single sketch -- **Bounded error** - Guarantees on approximation quality (e.g., ±1% relative error) - -### Streaming Pipelines - -ASAP builds sketches in **real-time at the edge**, before data reaches the backend: - -1. **Exporters / SDKs** emit metrics, scraped or pushed as **OTLP** -2. **asap-otel agents** run sketch processors that encode each metric as a sketch payload (DDSketch / KLL / HLL / CountSketch / Count-Min) — the metric name is **preserved**, not rewritten -3. A **gateway** merges sketches across agents and forwards them (OTLP) to the backend -4. **ASAPQuery-backend** ingests the sketch payloads into `SketchStore` and answers queries from them via `ASAPQueryEngine` - -A **control plane** (the in-repo `control_plane/` planner) decides which sketches to compute and where, then pushes per-runtime config to the agents over **OpAMP**. - -### Query Protocol - -ASAP implements the **Prometheus HTTP API**, making it a drop-in replacement: - -``` -# Point Grafana to ASAP instead of Prometheus -Prometheus URL: http://asap:8088 - -# Use the same PromQL queries -quantile by (job) (0.99, http_request_duration_seconds) -``` - -Your existing dashboards work without modification! - -### Fallback - -Not all queries can be answered from sketches. ASAP automatically: - -1. **Detects queries the warm sketch tier can't serve** (un-planned shapes, ad-hoc queries) -2. **Forwards them to the archive tier** — exact PromQL over Gorilla-compressed raw samples on object storage (via Thanos) -3. **Returns results transparently** to the user - -This ensures full PromQL compatibility while accelerating what sketches can answer. - -## High-Level Architecture - -``` - Applications / Exporters / SDKs - │ OTLP - ▼ - ┌─────────────────────────────────────────────┐ - │ Edge (ASAPCollector) │ - │ asap-otel agents → gateway (sketch-merge) │ - │ sketch processors encode DDSketch / KLL / │ - │ HLL / CMS, preserving metric names │ - └─────────────────────────────────────────────┘ - │ OTLP (sketch payloads, raw metric names) - ▼ - ┌─────────────────────────────────────────────┐ - │ ASAPQuery-backend │ - │ control_plane: plans sketches, pushes │ - │ config to agents via OpAMP │ - │ data_plane (warm): ASAPQueryEngine │ - │ over SketchStore │ - │ data_plane (archive): ThanosQueryEngine ──┼──▶ Thanos + MinIO/S3 - │ (exact raw PromQL) │ (Gorilla-XOR) - └─────────────────────────────────────────────┘ - ▲ PromQL (Prometheus HTTP API) - │ - Grafana -``` +The MVP is not merely that the components start. One reproducible paired run +must show functional correctness, bounded query error, fresh results, lower +query latency for claimed accelerated classes, and lower collector and total +resource cost than the raw exact baseline. diff --git a/docs/03-how-to-guides/operations/bootstrap-config-from-query-log.md b/docs/03-how-to-guides/operations/bootstrap-config-from-query-log.md deleted file mode 100644 index bae2fbd3..00000000 --- a/docs/03-how-to-guides/operations/bootstrap-config-from-query-log.md +++ /dev/null @@ -1,45 +0,0 @@ -# Bootstrap Config from Prometheus Query Log - -Generate sketch configs from real query traffic instead of hand-authoring a workload YAML. - -## Steps - -### 1. Enable the Prometheus query log - -Add to your Prometheus startup flags: -``` ---query.log-file=/var/log/prometheus/query.log -``` - -Let it run for a representative period (hours to days). Each line is a JSON entry: -```json -{"params":{"query":"rate(http_requests_total[5m])","start":"2025-12-02T18:00:00Z","end":"2025-12-02T18:00:00Z","step":0},"ts":"2025-12-02T18:00:00.001Z"} -``` - -### 2. Create a metrics config - -List the metrics you want ASAP to sketch: -```yaml -# metrics.yaml -metrics: - - metric: http_requests_total - labels: [instance, job, method, status] - - metric: node_cpu_seconds_total - labels: [instance, mode] -``` - -### 3. Run the planner - -The standalone `asap-planner` CLI was removed in Phase γ; the planner -now lives inside the [ASAPCollector controller](https://github.com/ProjectASAP/ASAPCollector/tree/main/controller). -Drive it via the controller's CLI / HTTP surface (see the controller -README for the equivalent invocation). The controller writes -`streaming_config.yaml` + `inference_config.yaml` and pushes the -resulting `BackendStorageRouting` to the backend over HTTP. - -## Notes - -- Queries appearing only once are skipped (need frequency to infer repeat interval) -- Queries referencing metrics not in `metrics.yaml` are skipped with a warning -- Unsupported PromQL patterns (e.g. `absent`, complex multi-level aggregations) are skipped -- Repeat interval is inferred from median inter-arrival time, rounded to the nearest scrape interval diff --git a/docs/README.md b/docs/README.md index 11474fa5..bc1f9c36 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,22 +1,37 @@ -# ASAP Developer Documentation +# ASAPQuery-backend documentation ## Getting started -- [Overview & Key Concepts](01-getting-started/overview.md) — what ASAP is -- [Architecture](01-getting-started/architecture.md) — system design & data flows -- [Local Setup](01-getting-started/local-setup.md) — set up a dev environment +- [Overview](01-getting-started/overview.md) +- [System architecture](01-getting-started/architecture.md) +- [Local setup](01-getting-started/local-setup.md) -## How-to guides +## Component design -- [Bootstrap Config from Query Log](03-how-to-guides/operations/bootstrap-config-from-query-log.md) — auto-generate sketch configs from Prometheus query traffic +- [Control plane](../control_plane/docs/README.md) +- [Data plane](../data_plane/docs/README.md) +- [Summary storage](design_docs/summary-storage.md) +- [Series identity](design_docs/series-identity.md) +- [Future storage and compression](design_docs/future-storage-and-compression.md) -## Design docs +## Developer guides -In-depth design notes live as flat `design-*.md` files in this directory. Starting points: +- [Adding a summary family](developer_docs/adding-summary-family.md) +- [Data-plane extension boundaries](../data_plane/docs/developer_docs/extension-points.md) +- [Querying ASAP](../data_plane/docs/user_guide/querying-asap.md) -- [Controller-into-backend refactor](design-controller-into-backend.md) — the two-plane (data/control) architecture -- [Sketch DB core](design-sketch-db-core.md) — the warm-tier `SketchStore` -- [Adding a new sketch](adding-a-new-sketch.md) — end-to-end recipe -- [Correctness proofs](proofs.md) — accuracy/lifecycle invariants +## Documentation ownership -See the remaining `design-*.md` files here for sketch-DB persistence, lifecycle, the SID model, and SQL/PromQL planning. +This repository documents ASAPQuery-backend-specific physical planning, +runtime query execution, and storage contracts. + +- Logical query planning and query-to-summary mapping belong to + [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner). +- Collector processing and CollectorPlan application belong to + [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector). +- Summary algorithm implementation and mathematical guarantees belong to the + corresponding summary library. + +Design documents link to those owners instead of maintaining parallel copies. +Benchmark outputs, migration histories, file-by-file implementation plans, and +future roadmaps are not active design specifications. diff --git a/docs/adding-a-new-sketch.md b/docs/adding-a-new-sketch.md deleted file mode 100644 index 06814c0f..00000000 --- a/docs/adding-a-new-sketch.md +++ /dev/null @@ -1,497 +0,0 @@ -# How to add a new sketch type to the pipeline - -**Audience:** contributors who want to extend the sketch DB with a -new sketch family (a new `KLL`-grade or `HLL`-grade algorithm). - -**Scope:** mechanical checklist across the three repos that need to -agree on a new sketch type. **Architectural rationale lives in -[`design-sketch-db.md`](design-sketch-db.md)** — this doc is the -recipe for *implementing* a new sketch once the decision to add it -has been made. - -**Estimated effort:** ~4,000–7,000 LoC across **3 repos** (sketchlib, -DataCollector, ASAPQuery-backend), plus control plane and docs. Plan a -multi-week effort with at least one cross-repo coordination -checkpoint. - ---- - -## When to add a new sketch type vs. tune an existing one - -Before starting, exhaust the cheap alternatives: - -1. **Can existing sketches answer this query class?** The current - set covers frequency (CMS / CountSketch), top-K (CMS+heap), - cardinality (HLL), quantile (KLL / DDSketch), exact aux scalars - (Sum / Min / Max / Increase). Most observability questions map - onto one of these. -2. **Can a parameter change satisfy the SLA?** Bigger `K` for KLL, - larger `width` for CMS, more registers for HLL. The `Sketch - Profiler Library` (sketch DB design §20) tells the control plane - the cost trade-off; often this is enough. -3. **Is the new sketch genuinely novel (different statistic class, - better cost frontier, different mergeability properties), or is - it just a micro-optimisation?** Micro-optimisations belong inside - an existing sketch's implementation, not as a new top-level - sketch. - -Only proceed with this guide if the new sketch genuinely opens a -query class or cost regime no current sketch can serve. - ---- - -## The cross-repo work, in order - -This is the **critical-path order**. Doing it in any other order -will leave you with broken intermediate states. - -### Step A. sketchlib (the algorithm) - -You ship the algorithm before any consumer can use it. - -#### A.1 sketchlib-rust - -``` -sketches/Foo/ -├── mod.rs FooSketch struct + new() + insert() + estimate() -├── merge.rs merge() with associativity + commutativity tests -├── delta.rs (optional) ComputeDelta + ApplyDelta for delta wire format -├── portable.rs SerializeProtoBytes / DeserializeFromProtoBytes -├── msgpack.rs (optional) SerializeMsgpack / DeserializeMsgpack -└── tests property-based tests + accuracy vs ground truth -``` - -#### A.2 sketchlib-go - -Mirror of A.1 in Go. The proto layout MUST be byte-identical so -cross-language round-trip works. - -#### A.3 Shared proto schema - -``` -proto/foo/foo.proto: - message FooState { - // sketch-specific fields, all numbered for forward-compat - uint32 param1 = 1; - uint32 param2 = 2; - repeated uint64 buckets = 3 [packed = true]; - } - -proto/sketchlib.proto: - message SketchEnvelope { - oneof sketch_state { - CountMinState count_min = 1; - // existing variants... - FooState foo = N; // <-- new variant, never reuse a number - } - } -``` - -#### A.4 Cross-language CI test - -A test that: -1. Constructs a `FooSketch` in Go, inserts samples, serialises. -2. Reads the bytes in Rust, deserialises, queries. -3. Asserts the queried statistic is within the sketch's theoretical - error bound (§19.9 of the sketch DB design doc). - -This is non-negotiable. Without it the Rust and Go sides will drift -and bugs will only surface in production via the modified-OTLP path. - -#### A.5 Profiler entry (in same repo as sketchlib) - -Run the Sketch Profiler Library (sketch DB design §20) calibration -mode against the new sketch type with a parameter grid. Commit the -resulting `ProfilerEntry` rows to the published catalogue. The -control plane will not be able to plan with this sketch until the -catalogue knows about it. - ---- - -### Step B. DataCollector wire format - -Now the agent and the backend can agree on bytes. - -#### B.1 Modified opentelemetry-proto patch - -`opentelemetry-proto-patch/opentelemetry/proto/metrics/v1/metrics.proto`: - -```proto -message Metric { - oneof data { - Gauge gauge = 5; - // existing 7, 9, 10, 11, 13, 14, 15, 16, 17... - FooSketch foosketch = N; // <-- new tag, never reuse - } -} - -message FooSketch { - repeated FooSketchDataPoint data_points = 1; - AggregationTemporality aggregation_temporality = 2; -} - -message FooSketchDataPoint { - repeated KeyValue attributes = 1; - fixed64 start_time_unix_nano = 2; - fixed64 time_unix_nano = 3; - fixed64 count = 4; // typed aux (sketch DB design §6.4) - double sum = 5; - double min = 6; - double max = 7; - bytes sketch = 8; // serialised FooState - FooSketchEncoding encoding = 9; - uint32 flags = 10; - uint64 series_id = 11; -} - -enum FooSketchEncoding { - FOO_SKETCH_ENCODING_UNSPECIFIED = 0; - FOO_SKETCH_ENCODING_PROTO = 1; - FOO_SKETCH_ENCODING_PROTO_DELTA = 2; - FOO_SKETCH_ENCODING_MSGPACK = 3; - FOO_SKETCH_ENCODING_MSGPACK_DELTA = 4; -} -``` - -Even if you only ship one encoding initially, reserve the four -standard variants so the upgrade path matches the existing sketch -families. - -#### B.2 pmetric typed accessors - -`opentelemetry-collector-patch/pdata/pmetric/`: - -- `MetricTypeFooSketch` constant in the metric type enum -- `Metric.SetEmptyFooSketch()` / `Metric.FooSketch()` accessors -- `FooSketch.DataPoints()` / `FooSketchDataPointSlice` etc. -- `FooSketchEncoding` type alias + the four encoding constants - -Most of this is mechanical — pattern-match on what an existing -sketch family did (e.g. CountMinSketch in commit history) and -duplicate. - ---- - -### Step C. DataCollector OTel processor - -The agent-side processor that produces FooSketch metrics. - -#### C.1 New processor - -``` -opentelemetry-collector-contrib-patch/processor/foosketchprocessor/ -├── factory.go processor.NewFactory(component.MustNewType("foo"), …) -├── config.go Config { Mode, MetricName, GroupBy, sketch params, -│ TransmitSketch, Encoding, DeltaTransmission, … } -├── processor.go ConsumeMetrics(): -│ - read incoming Gauge/Sum data points -│ - update FooSketch per (group_key, window) -│ - on emission: m.SetEmptyFooSketch(); fill -│ attributes / count / sum / min / max / sketch -│ bytes / encoding via the typed pmetric accessors -├── go.mod require sketchlib-go + replace to local path -└── tests unit tests for batch and window modes -``` - -Use existing processors (countminsketchprocessor, kllprocessor) as -templates — the structure is identical aside from the sketch type. - -#### C.2 asap-otel builder config - -`opentelemetry-collector-contrib-patch/cmd/asap-otel/builder-config.yaml`: - -```yaml -processors: - # existing entries... - - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/foosketchprocessor v0.x.0 - path: ./processor/foosketchprocessor -``` - -Rebuild the asap-otel binary, confirm it accepts a config -with `processors: foo:` and emits FooSketch data points the backend -can decode. - ---- - -### Step D. ASAPQuery-backend ingest path - -Now the backend recognises the new wire variant. - -#### D.1 Vendored proto regeneration - -`crates/asap_otel_proto/`: regenerate the -vendored proto. The tonic build script will produce a new -`Data::Foosketch` variant on the `Metric.data` oneof automatically. - -#### D.2 Ingest router - -`data_plane/src/drivers/ingest/otel.rs`: - -```rust -enum SketchKind { - CountMin, CountSketch, Kll, DdSketch, Hll, - Foo, // <-- new -} - -// In route_modified_otlp_sketches_to_precompute: -Some(Data::Foosketch(f)) => f.data_points.iter().map(|dp| ModifiedOtlpSketchDp { - kind: SketchKind::Foo, - attrs: merge_point_attributes(&base_labels, &dp.attributes), - time_unix_nano: dp.time_unix_nano, - sketch: dp.sketch.clone(), - encoding: dp.encoding, -}).collect(), - -// In decode_modified_otlp_sketch_bytes: -SketchKind::Foo => Ok(Box::new( - FooSketchAccumulator::from_sketchlib_proto_bytes(bytes)?, -)), -``` - -#### D.3 Concrete accumulator - -`data_plane/src/precompute_engine/operators/foo_sketch_accumulator.rs`: - -```rust -pub struct FooSketchAccumulator { - inner: sketchlib::Foo, -} - -impl FooSketchAccumulator { - pub fn from_sketchlib_proto_bytes(buffer: &[u8]) -> Result - pub fn from_msgpack_bytes(buffer: &[u8]) -> Result - pub fn update(&mut self, value: f64) { self.inner.insert(value) } - pub fn estimate(&self, …) -> f64 { self.inner.estimate(…) } -} - -impl AggregateCore for FooSketchAccumulator { - fn clone_boxed_core(&self) -> Box { … } - fn type_name(&self) -> &'static str { "FooSketchAccumulator" } - fn as_any(&self) -> &dyn std::any::Any { self } - fn merge_with(&self, other: &dyn AggregateCore) -> Result<…> { … } - fn get_accumulator_type(&self) -> AggregationType { AggregationType::Foo } - fn get_keys(&self) -> Option> { None } - fn query_statistic(&self, statistic, key, kwargs) -> Result { … } - fn approx_memory_bytes(&self) -> usize { … } - - /// IMPORTANT (sketch DB design Phase 1, §5.1, §6.4): - /// surface count / sum / min / max via aux columns when the - /// underlying sketch tracks them. Don't default to empty. - fn aux_stats(&self) -> AuxStats { - AuxStats { - count: Some(self.inner.count()), - sum: …, // None if the sketch doesn't track it - min: …, - max: …, - } - } -} - -impl SerializableToSink for FooSketchAccumulator { … } -``` - -Plus an `AccumulatorUpdater` impl in -`precompute_engine/accumulator_factory.rs`: - -```rust -fn create_accumulator_updater(config: &AggregationConfig) -> Box { - match config.aggregation_type { - // existing arms... - AggregationType::Foo => { - let p = foo_params(&config.parameters); - Box::new(FooAccumulatorUpdater::new(p)) - } - } -} - -fn foo_params(parameters: &HashMap) -> FooParams { - FooParams { - param1: parameters.get("param1").and_then(|v| v.as_u64()).unwrap_or(default_param1()), - // ... - } -} -``` - ---- - -### Step E. Type system - -`crates/promql_utilities/src/query_logics/enums.rs`: - -```rust -pub enum AggregationType { - Sum, Increase, MinMax, DatasketchesKLL, - // existing... - Foo, // <-- new variant -} - -impl AggregationType { - pub fn as_str(self) -> &'static str { - match self { - // existing... - AggregationType::Foo => "Foo", - } - } -} - -impl FromStr for AggregationType { - fn from_str(s: &str) -> Result { - match s { - // existing... - "Foo" => Ok(AggregationType::Foo), - _ => Err(…), - } - } -} -``` - -After this, `StreamingConfig` YAML files can carry -`aggregationType: Foo`. - ---- - -### Step F. Capability matching - -`crates/asap_types/src/capability_matching.rs`: - -```rust -fn compatible_agg_types(stat: &Statistic) -> Vec { - match stat { - Statistic::WhateverFooAnswers => { - vec![AggregationType::Foo, /* alternative existing types */] - } - // existing... - } -} -``` - -Without this, even if the control plane plans a Foo sketch, -ASAPQueryEngine's capability matcher won't route queries to it. - ---- - -### Step G. Sketch DB integration (assumes sketch DB Phase 6+ has shipped) - -#### G.1 Accuracy profile - -`AccuracyProfile::for_sketch_type(SketchType::Foo, params)` (sketch -DB design §6.4) must return a real `ErrorBound` derived from -`Foo`'s theoretical formula. Add the formula to §19.9 of the -design doc. - -#### G.2 Profiler catalogue - -If you completed Step A.5, the control plane already has cost -numbers. Verify the `/api/v1/db/cost_estimate` endpoint returns -sensible numbers for `Foo` configs. - -#### G.3 Controller decision logic - -`DataCollector/controller/src/analyzer.rs` and `planner.rs` decide -which sketch type to pick for a given query. Add `Foo` to the -candidate set for the relevant query intents. The cost model -(driven by the profiler catalogue) handles the actual selection. - ---- - -### Step H. Tests - -Minimum bar before merging: - -| Test | Repo | What it proves | -|---|---|---| -| Algorithm correctness | sketchlib-rust + go | property-based tests, accuracy within theoretical bound | -| Merge associativity | sketchlib-rust + go | merge(a, merge(b, c)) == merge(merge(a, b), c) within tolerance | -| Cross-language round-trip | shared CI | Go produce → Rust consume, statistic within bound | -| Processor unit test | DataCollector | batch + window modes emit the right `*SketchDataPoint` shape | -| OTLP wire decode | ASAPQuery-backend | end-to-end: Go processor → bytes on wire → Rust accumulator → query returns expected value | -| Aux-column round-trip | ASAPQuery-backend | `aux_stats()` returns the right scalar without deserialising the sketch (sketch DB Phase 1) | -| Capability match | ASAPQuery-backend | a query naming the relevant statistic actually picks `Foo` | -| Profiler regression | shared CI | cost numbers within 10% of the committed catalogue baseline | - ---- - -### Step I. Documentation - -Without these, your sketch is invisible to operators and to the -next person trying to add another one: - -| Doc | What to add | -|---|---| -| `DataCollector docs/pipeline-query-catalog.md` | A new row in the §3 query catalog: query class → OTel op → backend accumulator → PromQL → accuracy property | -| `ASAPQuery-backend docs/design-sketch-db.md §19.9` | The theoretical accuracy bound formula for `Foo` | -| `ASAPQuery-backend docs/design-sketch-db.md §19.10` | The merge propagation rule for `Foo` | -| Shared design doc for `Foo` | A short rationale: what query class, why this sketch over alternatives, what the trade-off is | -| [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) | How `Foo` is represented and selected as a logical summary candidate | - ---- - -## Anti-patterns to avoid - -These are mistakes contributors have made or will make: - -1. **Duplicating proto field numbers**. Each new variant on - `Metric.data` and on `SketchEnvelope.sketch_state` MUST get a - fresh tag. Reusing a tag silently corrupts all wire interop. -2. **Skipping cross-language round-trip tests** because "the proto - schema is shared so it must work." It doesn't. Endianness, - floating-point representation, ordering of sub-fields, packed - vs unpacked encoding — every one of these has bitten Rust/Go - interop in the past. The CI test is the only protection. -3. **Returning `AuxStats::empty()`** because writing the override - "looks like work." Then every Count / Sum / Min / Max query on - this sketch pays sketch deserialisation cost forever (sketch DB - design §5.1). At minimum, populate `count` from the sketch's - sample counter. -4. **Hand-deriving CPU / memory numbers** for the control plane's - cost model. The Sketch Profiler Library (sketch DB design §20) - exists exactly to prevent this. Always commit a calibration - run with the new sketch. -5. **Adding the sketch only to one tier.** The sketch DB has - Tier 1 (PromSketch, in-memory EH) and Tier 2 (precompute + LSM - parts). If `Foo` only makes sense in Tier 2, that's fine — - document it. If it could serve Tier 1, implement both. -6. **Deferring the documentation**. The design doc and the query - catalog are how operators decide whether to use the sketch and - how the control plane's planner reasons about it. Undocumented - sketch types in the codebase lead to operators picking - suboptimal aggregations because they don't know the new option - exists. - ---- - -## Rough effort estimate - -| Step | LoC (approx) | Time | -|---|---|---| -| A. sketchlib (Rust + Go + cross-lang test) | 1,500–3,000 | 1–2 weeks | -| B. DataCollector proto + pmetric | 800–1,500 | 3–5 days | -| C. DataCollector processor | 700–1,000 | 4–6 days | -| D. ASAPQuery-backend ingest + accumulator | 800–1,200 | 3–5 days | -| E. Type system update | 50–100 | 1 day | -| F. Capability matching | 50–100 | 1 day | -| G. Sketch DB integration | 200–400 | 2–3 days | -| H. Tests across all repos | 500–1,000 | 3–5 days | -| I. Documentation | 100–200 | 1–2 days | -| **Total** | **~4,800–8,500 LoC** | **~4–6 weeks** for a single contributor | - -If multiple contributors can work in parallel (one on sketchlib, -one on DataCollector, one on backend), this can compress to ~2–3 -weeks of wall time, but the coordination overhead means it rarely -goes below that. - ---- - -## Quick reference: which existing sketch did each step pattern-match? - -When in doubt, look at the most recent precedent for the same step: - -| Step | Best precedent to copy | -|---|---| -| sketchlib-rust algorithm | the `KLL` family if you need quantile-shaped behaviour, `HLL` for set/cardinality, `CountMin` for frequency | -| sketchlib-go matching impl | same family on the Go side | -| Modified OTLP proto variant | `KLLSketch` or `CountMinSketch` (both have all four encoding variants) | -| pmetric typed accessors | `CountMinSketch` accessor commit history (PR #157 era) | -| OTel processor | `kllprocessor` for window-mode, `countminsketchprocessor` for batch+heap | -| Backend accumulator | `DatasketchesKLLAccumulator` (cleanest current example) | -| Cross-language wire test | the existing `e2e_modified_otlp_sketch_path.rs` test pattern in ASAPQuery-backend | diff --git a/docs/design-252-arithmetic-operators.md b/docs/design-252-arithmetic-operators.md deleted file mode 100644 index 9c7e1d9b..00000000 --- a/docs/design-252-arithmetic-operators.md +++ /dev/null @@ -1,112 +0,0 @@ -# Design: PromQL Arithmetic Operator Acceleration (Issue #252) - -## Problem - -ASAPQuery accelerates PromQL queries by pre-computing sketches over streaming data and serving answers from those sketches at query time, bypassing the underlying TSDB for supported query patterns. - -The supported patterns today are all single-expression forms: -- `rate(metric[range])`, `increase(metric[range])`, etc. (OnlyTemporal) -- `sum(metric) by (label)`, `quantile(...)`, etc. (OnlySpatial) -- Combinations like `sum by (host) (rate(metric[range]))` (OneTemporalOneSpatial) - -Binary arithmetic expressions like `rate(errors[5m]) / rate(requests[5m])` fall through entirely — `handle_query_promql` finds no matching pattern and returns `None`, causing a full fallback to Prometheus. This is a significant coverage gap: error rate, saturation, and ratio queries are extremely common in practice. - ---- - -## Approaches Considered - -### Option A: Extend the PromQL-specific execution path - -Detect binary arithmetic at the top of `handle_query_promql`, execute each arm through the existing `execute_query_pipeline`, collect two `HashMap` result sets, and combine them with a Rust-level label-matching join. - -**Pros:** Self-contained, surgical change. No DataFusion involvement in the combination step. - -**Cons:** The combination logic (label-matching join + f64 arithmetic) would be written twice — once here, and again later when the PromQL execution is migrated to DataFusion. `execute_query_pipeline` is already on a path to being replaced. - -### Option B: DataFusion JOIN + Projection - -Use the existing DataFusion execution path (`execute_plan`, already tested but not wired in for PromQL) to build a plan that looks like: - -``` -Projection (value = lhs.value OP rhs.value, label columns...) - └── Join (inner, on = label columns) - ├── SubqueryAlias("lhs") → SummaryInfer → SummaryMergeMultiple → PrecomputedSummaryRead - └── SubqueryAlias("rhs") → SummaryInfer → SummaryMergeMultiple → PrecomputedSummaryRead -``` - -For scalar-vector arithmetic (`rate(errors[5m]) * 100`), the plan is simpler — just a `Projection` on top of the single arm's plan, no join needed. - -**Pros:** `execute_plan` is tested and ready to wire in. DataFusion's `Join + Projection` replaces hand-written label-matching join logic. Wiring this in for binary arithmetic also migrates all PromQL execution to `execute_plan`, eliminating `execute_query_pipeline` as an active path. - -**Cons:** Slightly more complex plan construction (SubqueryAlias for column disambiguation). The migration of all PromQL to `execute_plan` is a broader scope, but it's the right time to do it. - ---- - -## Decision: Option B (DataFusion) - -The decisive factor: `execute_plan` is tested and just not wired in. Building the arithmetic combination in Rust (Option A) would be work done twice — the exact same join logic would need to be re-implemented in DataFusion when the migration happens. Option B does it once, correctly. - -The additional complexity of the DataFusion `Join + Projection` plan is manageable and follows patterns already established in the codebase (`SubqueryAlias`, `LogicalPlanBuilder`). - ---- - -## Design - -### Planner (`asap-planner-rs`) - -For a query like `rate(errors[5m]) / rate(requests[5m])`, the planner: - -1. Detects the top-level `BinaryExpr` in the PromQL AST -2. Checks whether each arm is individually acceleratable (matches an existing pattern) -3. If both arms are acceleratable: emits a separate `QueryConfig` entry for each arm, as if they were independent queries -4. If either arm is not acceleratable: skips both — the engine will fall back to Prometheus -5. Deduplicates: if an arm's config already exists (e.g., `rate(errors[5m])` was also configured as a standalone query), it reuses the existing entry - -For scalar-vector arithmetic (`rate(errors[5m]) * 100`): the scalar literal is not a metric expression and needs no aggregation config. The planner emits a config only for the vector arm. - -**The planner does not emit a combined config for the binary expression itself.** The engine detects the arithmetic operator at query time from the PromQL AST and handles the combination. - -### Engine (`data_plane`) - -#### Config lookup: structural PromQL matching - -To find a config for each arm at query time, the engine uses structural AST comparison (`find_query_config_promql_structural`) rather than exact string matching. This mirrors the existing `find_query_config_sql` pattern and is robust to formatting differences. - -Structural equality for PromQL compares: function name, metric name, label selectors, range duration. Evaluation timestamps are ignored. - -#### Plan construction - -For **vector op vector**: -1. Find config for each arm via structural matching -2. Build a `QueryExecutionContext` for each arm (reusing existing logic) -3. Call `to_logical_plan()` on each context to get the sketch sub-plans -4. Wrap each sub-plan in a `SubqueryAlias` (`"lhs"` / `"rhs"`) to disambiguate the `value` column name -5. Build a DataFusion inner `Join` on the shared label columns -6. Add a `Projection` computing `lhs.value OP rhs.value AS value` and projecting label columns through - -For **scalar op vector** (e.g., `rate(errors[5m]) * 100`): -1. Build a `QueryExecutionContext` for the vector arm only -2. Call `to_logical_plan()` on the context -3. Add a `Projection` computing `value OP lit(scalar) AS value` (or `lit(scalar) OP value` if scalar is on the left) - -#### Execution - -Both cases execute through `execute_logical_plan` — the refactored inner method extracted from `execute_plan`. This method takes a pre-built `LogicalPlan` and runs it through the DataFusion session. - -#### Dispatch - -In `handle_query_promql`, after parsing the PromQL AST, the engine checks whether the top node is a `BinaryExpr`. If yes, it routes to `handle_binary_expr_promql`. If no, it follows the existing single-expression path — but now calling `execute_plan` instead of `execute_query_pipeline`. - -Recursion is natural: `(A + B) / C` is a `BinaryExpr` at the top level. The LHS is itself a `BinaryExpr`, which `handle_binary_expr_promql` handles recursively when building the LHS sub-plan. - -If either arm cannot be accelerated (no matching config, or unsupported pattern), `handle_binary_expr_promql` returns `None` and the caller falls back to Prometheus for the whole query. - -### Operators supported - -All six PromQL binary arithmetic operators: `+`, `-`, `*`, `/`, `^`, `%`. - -### Out of scope - -- PromQL vector matching modifiers (`on()`, `ignoring()`, `group_left()`, `group_right()`): the join is always on all label columns. Adding vector matching is a follow-on. -- Range queries (only instant queries addressed here). -- Comparison operators (`==`, `!=`, `>`, etc.) and set operators (`and`, `or`, `unless`). diff --git a/docs/design-asap-precompute-rs.md b/docs/design-asap-precompute-rs.md deleted file mode 100644 index 2041bf11..00000000 --- a/docs/design-asap-precompute-rs.md +++ /dev/null @@ -1,174 +0,0 @@ -# Backend consumes `asap-precompute-rs` - -## Status - -Implemented. PR: `phase3/backend-consumes-asap-precompute-rs`. - -## Context - -The ASAP edge-framework migration (see -`docs/design-asap-edge-framework.md` in `ASAPCollector`, plus ADR-0002) -factors the SHARED ingest runtime — windowing, snapshot caching, -envelope encoding, sketch reconstruction, sketch merge — into a -host-neutral crate -[`asap-precompute-rs`](https://github.com/ProjectASAP/ASAPCollector/tree/main/asap-precompute-rs). -Every Rust **edge** runtime (Vector adapter, OTAP-Rust, Arrow-backed -shims, Telegraf input) consumes it, and so does this backend. - -Before this change, the backend's ingest path inlined that shared -logic: - -- `precompute_engine/operators/{ddsketch,kll,hll,countsketch,countmin}_accumulator.rs` - each had a `from_sketchlib_proto_bytes` that decoded - `SketchEnvelope` proto and dispatched the - `sketch_state` oneof by hand (the same dispatch was duplicated five - times, once per sketch). -- `drivers/ingest/otel.rs::decode_modified_otlp_sketch_bytes` routed - full-state OTLP sketch bytes through those per-accumulator - decoders. -- The merge path in `*::merge_with` (used by the - `precompute_engine`'s pane-folding logic) called into - `asap_sketchlib::sketches::*::merge_refs(...)` directly. - -Independently, `asap-precompute-rs` was building the same envelope -decode + state extraction + reconstruction logic — for agents. - -## Decision - -Backend's ingest path **consumes asap-precompute-rs** for the -shared work. Backend's QUERY-side engine (PromQL aggregation, -storage, query planning) stays put. - -### What moved into asap-precompute-rs (i.e. no longer -backend-internal) - -| Concern | Where the shared logic lives now | -| --- | --- | -| Envelope wire-format runtime view | `asap_precompute_rs::envelope::{SketchEnvelope, Encoding, SketchType}` | -| Envelope proto decode + oneof dispatch | `asap_precompute_rs::envelope::ProtoSketchEnvelope` (re-export of the prost type) plus the per-wrapper `decode_envelope` helpers | -| Sketch reconstruction from envelope payload (DDSketch + KLL) | `asap_precompute_rs::sketches::{DDSketchWrapper, KLLWrapper}::Sketch::apply_delta` | -| Sketch merge | `asap_precompute_rs::Sketch::merge` (round-trip via `snapshot` + `apply_delta`) | -| `Sketch` / `QuantileSketch` / `CardinalitySketch` / `FrequencySketch` traits | `asap_precompute_rs::precompute::*` | - -### What stays in `ASAPQuery-backend` - -- **Query-side engine**: PromQL aggregation - (`engines/{logical,physical}`), storage (`stores/`), query - planning (`asap-planner-rs`), DataFusion bridge (`tests/datafusion`). -- **Per-accumulator query-side surface**: - `precompute_engine/operators/*_accumulator.rs` keeps the `AggregateCore`, - `query_statistic`, `MergeableAccumulator`, - `SerializableToSink` impls, and the per-sketch JSON output. These - are query-side, not ingest-side. -- **Sparse-delta application**: - `precompute_engine/operators/*::apply_proto_delta_bytes` and - `drivers/ingest/otel.rs::apply_modified_otlp_delta_bytes` stay - because `asap_sketchlib` doesn't yet expose the `compute_delta` - family upstream — `asap-precompute-rs`'s wrappers fall back to - "always full" delta encoding (see - `asap-precompute-rs/src/sketches/mod.rs` "API surface caveats"). - When the upstream `compute_delta` lands, this path will collapse - into the same delegation pattern the full-state path uses today. -- **MSGPACK** (`*::from_msgpack_bytes`) — alternative wire format the - asap-precompute-rs edge runtime doesn't emit. Stays for the - Strategy-A (vendored modified-OTLP) path that hard-codes msgpack - for some sketches. - -### Bridge module - -`data_plane/src/precompute_engine/operators/edge_runtime_adapter.rs`: - -- Re-exports the asap-precompute-rs runtime view types (one canonical - `SketchEnvelope`, `Encoding`, `SketchType`, `Sketch`, - `QuantileSketch`, `CardinalitySketch`, `FrequencySketch`). -- `unwrap_envelope_state(bytes) -> Option`: the - shared envelope-decode + oneof-extract path that all five - accumulators used to inline. Single source of truth. -- `reconstruct_via_runtime(SketchType, bytes) -> ReconstructedSketch`: - uses `asap-precompute-rs`'s `Sketch::apply_delta` to decode an - envelope and reconstruct the underlying - `asap_sketchlib::sketches::*` state. Wired for DDSketch + KLL - today. -- `encode_ddsketch_envelope(&DdSketch) -> Vec`: emits the - canonical envelope shape that `asap-precompute-rs`'s wrapper - emits, byte-for-byte. -- `merge_ddsketches_via_runtime(&DdSketch, &DdSketch) -> DdSketch`: - routes through `asap-precompute-rs::Sketch::merge`. Result is - byte-identical to `DdSketch::merge_refs(&[a, b])` because both - call the same underlying merge logic. - -### Wire-up - -`drivers/ingest/otel.rs::decode_modified_otlp_sketch_bytes`'s -`ENCODING_PROTO` branch is the entry point for full-state -modified-OTLP sketch envelopes. The DDSketch and KLL arms now -delegate to `edge_runtime_adapter::reconstruct_via_runtime`. HLL / -CountSketch / CountMinSketch keep using the backend's existing -per-accumulator decoder until upstream byte parity (issue #243) -lands. - -### Cargo deps - -- `asap_sketchlib`: bumped to `branch = "main"` (post-PR-#39 module - renames; PRs #40/#41/#42 land DDSketch/KLL/CountSketch byte parity). -- `asap-precompute-rs`: new path-dep - (`{ path = "../../ASAPCollector/asap-precompute-rs" }`) — the - asap-precompute-rs crate itself path-deps `asap_sketchlib`, so - cloning ASAPCollector via cargo's git source fails to resolve the - path. Path-dep mirrors how `asap-precompute-rs` consumes - `asap_sketchlib` for the same reason. -- Workspace `[patch."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/ProjectASAP/asap_sketchlib"]` - redirects the git-sourced `asap_sketchlib` (used by backend) to - the local checkout, so the type identities at the - `asap-precompute-rs` ↔ backend boundary unify. - -## Acceptance tests - -`data_plane/tests/edge_runtime_consumes_precompute_rs.rs`: - -- **DDSketch round-trip** through asap-precompute-rs runtime is - byte-identical with the input envelope. -- **DDSketch structural** assertions (count, alpha within bounds). -- **DDSketch end-to-end**: envelope → backend `AggregateCore` → - `query_statistic(Quantile)` returns a value within the configured - `α` of the true median. -- **DDSketch back-snapshot** through asap-precompute-rs is - byte-identical with the input envelope (closes the encode side of - the round-trip). -- **KLL round-trip** via asap-precompute-rs runtime is byte-identical - with the input envelope. -- **KLL structural** assertions (k, items count). -- **HLL / CountSketch / CountMinSketch round-trips** are present - but gated `#[ignore = "blocked on ASAPCollector#243 HLL/CS/CMS - byte parity"]` — they will start passing automatically when issue - #243 lands without code changes here. - -## Consequences - -### Positive - -- Single source of truth for envelope decode + state extraction + - sketch reconstruction across agents and backend. -- Adding a new sketch only requires one place (asap-precompute-rs) - to gain envelope-handling support; the backend gets it for free - via `edge_runtime_adapter::reconstruct_via_runtime`. -- The `Sketch` / `QuantileSketch` / `CardinalitySketch` / - `FrequencySketch` trait family exposes a uniform interface for - envelope-shaped sketches, which the per-platform Strategy-B - adapters (Telegraf / Vector / OTAP) will reuse. - -### Negative - -- The path-dep on a sibling repo means CI must clone `ASAPCollector` - next to `ASAPQuery-backend`. Resolved by the README pointer; a - follow-up will revisit the dep style once the - `asap-precompute-rs` Cargo.toml flips its own `asap_sketchlib` - pin to a git URL (so cargo can resolve a single git source). - -### Deferred - -- HLL / CountSketch / CountMinSketch byte parity (issue #243). -- Sparse-delta `compute_delta` upstream (currently in `sketchlib-go` - only, not `asap_sketchlib`). Until that lands the typed-delta - apply path stays in this repo. -- MSGPACK delta encoding (`ENCODING_MSGPACK_DELTA = 4`). diff --git a/docs/design-controller-into-backend.md b/docs/design-controller-into-backend.md deleted file mode 100644 index 8cd92645..00000000 --- a/docs/design-controller-into-backend.md +++ /dev/null @@ -1,764 +0,0 @@ -# Controller-into-backend refactor - -> Status: design draft -> Author: drafted 2026-05-10 from a multi-node MVP sweep that surfaced -> the architectural mismatch between the agent-side metric-name -> rewrites (`_quantile`, `_topk`) and the "same PromQL, same metric -> name across all arms" comparison requirement. - -## 1. Motivation - -The current B0/B1/ASAP three-arm comparison fails apples-to-apples -because the ASAP arm answers a *different metric name* than B0/B1: - -| arm | metric name PromQL is fired against | who emits this name | -|---------|------------------------------------------|---------------------| -| B0 | `http_requests_total_latency_ms` | fake-exporter SDK | -| B1 | `http_requests_total_latency_ms` | fake-exporter SDK | -| ASAP | `http_requests_total_latency_ms_quantile`| `gateway-aggregate-from-raw.yaml`'s `metric_suffix: "_quantile"` rewrites the gauge name during sketch encoding | - -A user firing the same PromQL — `quantile_over_time(0.99, -http_requests_total_latency_ms[30s])` — gets results from -B0/B1 (Prometheus runs the quantile over raw gauge samples) but -nothing from ASAP, because the ASAP backend's pattern matcher is -keyed on the suffixed name `..._quantile`. - -The fix is not "add raw-name patterns to the matcher." The -fix is to remove the rewrites and design the backend so the -storage and query layers are keyed on the *raw metric name + -raw labels* that the SDK actually emits. The encoding (sketch -type, payload format) is a wire-level attribute on each sample, -not a name suffix. - -## 2. Current architecture (what we change away from) - -``` -┌──────────────────────────────────────────────────────────────────┐ -│ ASAPCollector repo │ -│ │ -│ fake-exporter ──OTLP──▶ asap-otel agent │ -│ │ (sketch processors emit │ -│ │ ..._quantile / ..._topk renamed │ -│ │ metrics) │ -│ ▼ │ -│ gateway (sketch-merge) │ -│ │ │ -│ │ │ -│ controller/ ──OpAMP──▶ agents (push per-metric sketch plan) │ -│ │ -└──────────────────────────────────────────────────────────────────┘ - │ OTLP w/ renamed metrics - ▼ -┌──────────────────────────────────────────────────────────────────┐ -│ ASAPQuery-backend repo │ -│ │ -│ data_plane │ -│ SketchStore │ -│ indexed by aggregation_id (integer assigned by │ -│ StreamingConfig::from_yaml_data on ingest) │ -│ ASAPQueryEngine query path │ -│ PromQL → find_query_config (exact pattern match) │ -│ ├─ hit: dispatch to aggregation_id │ -│ └─ miss: capability_matching by Statistic::{Sum, │ -│ Quantile, etc.} │ -│ ├─ hit: dispatch to compatible aggregation │ -│ └─ miss: fall through to BackendStorageRouting │ -│ ├─ shape ∈ [count, topk, rate_post_hoc] │ -│ │ → ThanosQueryEngine │ -│ └─ else → ASAPQueryEngine (404 if miss) │ -│ │ -│ asap-common/ │ -│ asap_types (StorageBackend, AggregationCapability, …) │ -│ promql_utilities, datafusion_summary_library │ -│ │ -└──────────────────────────────────────────────────────────────────┘ -``` - -Three structural problems: - -1. **Metric names are mutated mid-pipeline**, breaking same-PromQL - comparison across arms. -2. **Two-repo split with shared `asap-common`**: type changes need - coordinated PRs in both repos. -3. **Backend's query layer is plan-blind**: pattern matching at query - time is a cache for the controller's plan that nobody primed; when - the cache misses, the matcher reasons from `Statistic::*` - capabilities that don't carry the controller's intent. - -## 3. Target architecture - -``` -┌──────────────────────────────────────────────────────────────────┐ -│ ASAPCollector repo (edge-only after refactor) │ -│ │ -│ fake-exporter / SDK ──OTLP──▶ asap-otel agent │ -│ │ sketch processors emit │ -│ │ the SAME metric name as │ -│ │ input; payload shape is │ -│ │ sketch-typed (DDSketch / │ -│ │ KLL / HLL / CMS / CS) but │ -│ │ metric.Name is unchanged. │ -│ ▼ │ -│ gateway (sketch-merge, │ -│ name-preserving) │ -│ │ -└──────────────────────────────────────────────────────────────────┘ - │ OTLP w/ raw metric names - │ (sketch payloads attached) - ▼ -┌──────────────────────────────────────────────────────────────────┐ -│ ASAPQuery-backend repo (controller + sketch store + query) │ -│ │ -│ controller/ │ -│ L1 query_language → … → L5 stage_split │ -│ capability map: (metric_name, query_shape) → sketch_kind │ -│ OpAMP server (originating from this host, pushes to agents) │ -│ │ -│ data_plane │ -│ PrecomputeEngine │ -│ receives OTLP (sketch payloads w/ raw metric names) │ -│ may merge sketches across agents — otherwise pass-through │ -│ SketchStore │ -│ key: (raw_metric_name, raw_labels, capability_set) │ -│ value: sketch state (encoded per the controller's plan) │ -│ Query path │ -│ PromQL parse │ -│ → controller.capability_for(metric, query_shape) │ -│ ├─ Some(capability): │ -│ │ SketchStore.get(metric, labels, capability) │ -│ │ ├─ hit: return sketch result │ -│ │ └─ miss: forward to Thanos (raw fallback) │ -│ └─ None (controller doesn't plan this query): │ -│ forward to Thanos │ -│ │ -└──────────────────────────────────────────────────────────────────┘ - │ raw counter samples (Gorilla - │ XOR compressed, original - │ metric names preserved) - ▼ - MinIO / S3 - (Thanos store-gateway sees these as - TSDB blocks and answers PromQL via - thanos-query) -``` - -Key design rules: - -- **No metric-name rewriting anywhere in the data path.** The name - the SDK emits is the name on the wire is the name in the backend - index is the name the user fires PromQL against. Wire encoding - type (raw counter / DDSketch / KLL / …) lives in the OTLP - pdata.Metric variant tag, not the name. -- **MinIO holds raw samples only.** Gorilla-XOR-compressed for - efficiency, but no sketch state, no aggregations, no merged - outputs. Thanos store-gateway → thanos-query exposes them as - Prometheus-compatible TSDB blocks. -- **Warm tier holds all sketch state.** Indexed by `(metric_name, - labels, capability_set)`. Capabilities come from the controller's - per-metric plan. -- **Controller is in-process with the query engine.** No more - pattern matching at query time — the controller's capability map - IS the query dispatch table. -- **OpAMP server lives on the backend host.** It pushes plans down - to agents. Agents only need a way to reach the backend for OpAMP - + OTLP; they don't have their own controller dependency. - -## 4. Migration plan - -Each step is a minimum-merge unit: the system builds and the -multi-node demo passes after each step, with progressively more -of the new design landed. - -> **Status (2026-05):** Steps 1–5 have all landed — the system now -> runs on this design (the controller is in-repo as `control_plane/`, -> `asap-common` has collapsed into `crates/`, `SketchStore` is -> reindexed, and warm-miss → Thanos routing is live). The per-step -> detail below is kept as the original plan of record; only Step 6 -> remains future work. - -### Step 1 — Strip metric-name rewrites (1–2 hours) - -Audit every site that suffixes a metric name and remove the suffix. -Keep the wire-level encoding hint in the OTLP pdata variant. - -**Files to inspect and patch:** - -- `deploy/configs/gateway-aggregate-from-raw.yaml` — currently - `metric_suffix: "_quantile"`. Remove or set to empty. -- `opentelemetry-collector-contrib-patch/processor/{ddsketch,kll, - hll,countsketch,countminsketch}processor/` — search for - any name-suffix logic in `processor.go` / `factory.go`. -- `asap-precompute-go/` and `asap-precompute-rs/` — same audit. -- `gorillas3processor/processor.go` — verify it preserves the input - name when writing TSDB blocks. - -**Acceptance test:** ASAP backend, after one full window, has -`http_requests_total_latency_ms` (raw) in its store, NOT -`..._quantile`. Verified via `curl asap-backend:9091/api/v1/series` -or by reading the OTLP wire bytes. - -**Risk:** breaks `backend-inference.yaml` patterns (which are -keyed on `_quantile`). Step 2 replaces pattern matching anyway, -so this is intentional. - -### Step 2 — Backend routing: warm-miss → Thanos (30 min) - -`backend-storage-routing.yaml` currently routes by query shape: -`[count, topk, rate_post_hoc]` to archive, everything else to -warm. Change semantics: try warm first; on miss (no aggregation -matches the metric+labels+capability), fall through to Thanos. - -**Files:** - -- `data_plane/src/query_engines/routing/backend_storage_routing.rs` — - swap "shape allow-list" for "warm-first, archive-fallthrough". -- `data_plane/src/query_engines/routing/query_engine_routing.rs` (EngineRouter) — add - a `query_with_fallthrough` path. - -**Acceptance test:** Same PromQL `count(http_requests_total)` and -`sum_over_time(http_requests_total[1m])` works on B0, B1, and -ASAP. ASAP's response carries `data_source: thanos_archive` for -queries that fall through, `data_source: warm` for those that -hit a sketch. - -### Step 3 — Reindex SketchStore by `(metric_name, labels, capability)` (2–3 days) - -Today's `SketchStore.get_aggregation(aggregation_id: u64)` becomes -`SketchStore.get(metric_name: &str, labels: &LabelSet, capability: -&Capability)`. Streaming-config ingest no longer assigns integer -IDs; it stores under the natural tuple. - -**Files:** - -- `data_plane/src/storage_engines/sketch_db/simple_map_store/{mod, - per_key,common_state}.rs` — refactor key type. -- `data_plane/src/streaming_engine.rs` — ingest path: when - an OTLP sketch sample arrives with `(metric_name, labels)`, look - up its capability from the in-process controller's plan and - store under that triple. -- `asap-types/src/capability_matching.rs` — make `Capability` the - index key type (probably a small enum + sketch-family tag). - -**Acceptance test:** Backend's `/internal/store-dump` endpoint -shows entries like -`{metric: "http_requests_total_latency_ms", labels: {...}, -capability: "QuantileApprox(DDSketch)"}` instead of integer -aggregation IDs. - -**Migration risk:** Streaming-config YAML format changes -(`backend-streaming.yaml` no longer needs `aggregationId`). All -downstream tests under `data_plane/tests/` need updating. - -### Step 4 — Move `controller/` from ASAPCollector to ASAPQuery-backend (3–5 days) - -Physically relocate the crate. Both are Rust, both already use -prost-build for OTel proto compilation, so the build surface is -compatible. - -**Steps in order:** - -1. Copy `ASAPCollector/controller/` → `ASAPQuery-backend/controller/`. -2. Add `controller` to ASAPQuery-backend's Cargo workspace; remove - from ASAPCollector's. -3. ASAPQuery-backend's `data_plane` Cargo.toml gets `controller - = { path = "../controller" }`. -4. Backend binary embeds the controller's L4 `sketch_algebra` as a - library call (no more HTTP capability-miss notifications between - processes — same process now). -5. Move OpAMP server: `controller/src/opamp/` continues to listen, - but the listening host is now the backend node, not a separate - controller container. Update agent-config OpAMP endpoints from - `ws://controller:4320/v1/opamp` to - `ws://backend:4320/v1/opamp`. -6. Delete `ASAPCollector/controller/` after green CI. -7. Update `deploy/docker/Dockerfile.controller` to be a no-op - (or delete) — the controller is no longer a standalone image. - The `Dockerfile.backend` build context now includes - `controller/`. - -**Acceptance test:** Single backend container starts both the -query HTTP API (port 9091) and the OpAMP server (port 4320). -Agent connects to `ws://backend:4320/v1/opamp`, receives plan, -emits sketches, backend stores them. Same multi-node demo runs. - -### Step 5 — Delete `asap-common` (1–2 days) - -Audit each crate under `crates/`: - -- `asap_types`: most types are backend-internal — move into - `data_plane/src/types/`. The wire types (`SketchEnvelope`, - `Statistic`) are shared with edge processors via OTLP proto, so - no Rust-to-Go path-dep needed. -- `promql_utilities`: backend-only — move into - `data_plane/src/promql/`. -- `datafusion_summary_library`: backend-only — same. -- Anything Go-side actually used by edge processors (e.g. - sketchlib-go interop) is already in `sketchlib-go` itself, not - in `asap-common`. - -**Steps:** - -1. List every Rust file under `crates/`. - Bucket into "backend internal" vs "wire shared". -2. Move "backend internal" files into `data_plane` or - `asap-types` (a renamed minimal types-only crate kept inside - ASAPQuery-backend). -3. Delete `asap-common/`. -4. Update `Cargo.toml` path-deps in both ASAPCollector and - ASAPQuery-backend. - -**Acceptance test:** ASAPCollector's `cargo build --release` -runs without referencing `asap-common`. ASAPQuery-backend's -`cargo build --release` produces the backend binary. - -## 4.5 Capability model — what the backend index keys on - -The controller's capability map and the backend's SketchStore index -share one type: - -```rust -enum Capability { - QuantileApprox(SketchKind), // SketchKind ∈ {DDSketch, KLL} - CardinalityApprox, // single sketch family: HLL - FrequencyTopk(SketchKind), // SketchKind ∈ {CountMin, CountSketch} - // SumOverTime, RateOverTime — answered from raw counter via Thanos, - // not via ASAP-tier sketches; no Capability variant needed. -} -``` - -`CountMin` vs `CountSketch` are statistically distinct (CMS biased/cheap, -CS unbiased/slightly more expensive) but answer the SAME PromQL family -(`topk`, `frequency_of`). Treating them as alternative `SketchKind` -implementations of the same `Capability::FrequencyTopk` lets the -controller choose at plan time without affecting query routing. - -### Group-by labels are part of the index key - -A query like `topk(10, sum by (zone, service) (http_requests_total))` -folds away every label except `{zone, service}` at sketch ingest time. -The remaining label set IS the group-by keys. Backend's index entry: - -``` -key: (raw_metric_name="http_requests_total", - group_by={"zone", "service"}, - capability=FrequencyTopk(CountMin)) -value: CMS state — d×w cell matrix keyed by hash(zone||service) -``` - -A single metric can have many entries — one per -`(group_by_set, capability)` tuple the controller's plan covers: - -| (metric, group_by, capability) | sketch | -|---------------------------------------------------------------|---------| -| (`http_requests_total`, `{zone}`, FrequencyTopk(CountMin)) | CMS-1 | -| (`http_requests_total`, `{service}`, FrequencyTopk(CountMin)) | CMS-2 | -| (`http_requests_total`, `{zone}`, QuantileApprox(DDSketch)) | DD-1 | -| (`http_requests_total_latency_ms`, `{zone}`, QuantileApprox(DDSketch)) | DD-2 | - -Query path: parse PromQL → derive `(metric, group_by, capability)` → -SketchStore.get() → hit (return sketch eval) or miss (forward raw to -Thanos). - -## 4.6 OTLP metadata model + backend store layout - -### What to transmit (proto patch direction) - -Today's per-DataPoint fields duplicate state the sketch payload itself -encodes (count/sum/min/max for DD/KLL, cardinality for HLL, -sample_count for CMS). They also repeat sketch-instance config on every -DP (epsilon/delta on CS, rows/cols on CMS, precision on HLL). Both are -wasteful and create cache-invalidation bugs. - -Step 1.5 proto patch (metrics.proto): - -- **Drop precomputed values from each `*DataPoint` message**: count, - sum, min, max, cardinality, sample_count. The sketch payload (or its - msgpack-encoded state) is the source of truth. -- **Lift per-instance sketch config from DataPoint up to the parent - sketch message**: - -```protobuf -message DDSketch { - repeated DDSketchDataPoint data_points = 1; - AggregationTemporality aggregation_temporality = 2; - double relative_accuracy = 3; // DDSketch α -} -message KLLSketch { … uint32 k = 3; } -message HLLSketch { … uint32 precision = 3; } -message CountSketch { … int32 rows = 3; int32 cols = 4; } // depth, width -message CountMinSketch { … int32 rows = 3; int32 cols = 4; } // depth, width -``` - -DataPoint messages keep only the per-window mutable state: -`attributes`, `start_time_unix_nano`, `time_unix_nano`, `sketch` -(payload bytes), `encoding` (PROTO/MSGPACK ± DELTA), `flags`. - -### Backend store layout — two-level index - -``` -SketchStore -├─ instances : HashMap<(metric_name, group_by_keys, Capability), SketchInstanceMetadata> -└─ series : HashMap> - -SketchInstanceMetadata { - metric_name: String, // raw input name - group_by_keys: BTreeSet, // surviving label KEY set (not values) - capability: Capability, // derived from sketch_type - sketch_type: SketchKind, // DDSketch/KLL/HLL/CMS/CS - sketch_config: SketchConfig, // ε, δ, k, precision, rows, cols (enum per kind) - accuracy: AccuracyBound, // derived: (eps_relative, conf_1-δ) - first_seen_ts: i64, -} - -SketchTimeSeries { - instance_id: u64, // FK - series_label_values: BTreeMap, // group-by VALUES per series - samples: BTreeMap, // window_end → bytes + encoding -} -``` - -### Ingest mapping (OTLP → store) - -| OTLP source | store target | -|----------------------------------------------------|-------------------------------------------| -| `Metric.name` | `instance.metric_name` | -| `Metric.data_case` (oneof tag) | `instance.sketch_type` → `Capability` | -| Parent sketch container's config fields | `instance.sketch_config` | -| `dp.attributes.keys()` | `instance.group_by_keys` (sorted) | -| `dp.attributes` | `series.series_label_values` | -| `dp.time_unix_nano` | `series.samples` map key | -| `dp.sketch + dp.encoding` | `series.samples` map value | - -### Query mapping (PromQL → store lookup) - -``` -parse PromQL → (metric_name, query_function, group_by_keys_requested) -capability = Capability::for_function(query_function) - (e.g. quantile_over_time → QuantileApprox) -instance = instances.get((metric_name, - group_by_keys_requested, - capability)) -match instance: - Some(inst) → decode payload(s) from inst.sketch_type, evaluate - via inst.sketch_config (uses ε/δ/precision/k correctly) - None → forward query to Thanos archive -``` - -### group_by_keys: implicit via `DataPoint.attributes` (decided) - -`DataPoint.attributes` IS the OTel term for what Prometheus calls -"labels" — the dimensional identity of a data point. After the agent's -`AggregateBy` rollup at sketch-insert time: - -- All non-group-by label values are folded INTO the sketch state. -- `dp.attributes` reflects only the surviving group-by dimensions. -- Therefore `dp.attributes.keys()` = group-by KEY set (= store - instance index component); `dp.attributes` (as map) = group-by - VALUES for the per-series record. - -This matches stock-OTel and Prometheus semantics for `sum by (zone)` -— the resulting series carries only `zone`. Zero proto change. - -**Convention contract:** - -1. Agent processor MUST strip non-group-by labels via the `AggregateBy` - config before emitting the sketch DP. -2. Backend trusts `dp.attributes.keys()` as the group-by KEY set when - building the SketchInstanceMetadata index entry. -3. **Convention violations are correctness-preserving but inefficient.** - A processor that leaks extra labels causes the backend to perceive - a larger group-by set → more SketchInstanceMetadata entries than - intended (one per leaked-label combination). Query semantics stay - correct (each instance answers its own group-by); the cost is RAM - from over-fragmenting the index. - -## 5. Open questions - -- **OpAMP origination host**: the runbook's compose currently has - `controller:4320` and `backend:9091` as distinct services. After - Step 4 they collapse to one container. Do we keep two ports - (4320 OpAMP + 9091 query) or unify? -- **Stale `_quantile` patterns in `backend-inference.yaml`**: do - we keep this file at all after Step 3 (no more pattern - matching), or repurpose as the controller's bootstrap plan? -- **Edge-runtime parity**: `asap-precompute-rs` (used by the - Rust edge agent path) currently uses some `asap-common` types. - Step 5 needs to leave a thin wire-types crate accessible to - edge runtimes — name it `asap-wire-types` and put it in - ASAPCollector? Or in a third repo? -- **`_topk` and similar suffixes**: Step 1 removes `_quantile`. - Are there other suffixes (`_topk`, `_uniques`, `_count`) added - by other processors? Need to grep more thoroughly. - -## 5.4 Series-ID namespace — centralized via asap-query-backend (decided) - -The patched OTLP protocol already supports series_id minting + -SeriesAssignment response (`opentelemetry-go-patch/exporters/otlp/otlpmetric/otlpmetricgrpc/` -+ `opentelemetry-collector-patch/receiver/otlpreceiver/internal/metrics/series_cache.go`). - -Today's implementation is **per-hop**: each receiver in the chain -(agent's OTLP receiver, gateway's OTLP receiver, backend's OTLP -receiver) mints its own series_ids in its own namespace. Same series -gets registered 3× across (fake-exporter→agent, agent→gateway, -gateway→backend) and the namespaces don't share meaning. - -**New design — centralize series_id minting at asap-query-backend:** - -- The backend (which now also hosts the controller — Step 4) is the - single authoritative minter of series_ids. -- Agents and gateway DO NOT mint their own series_ids. They forward - the original Export upstream, propagate the SeriesAssignment - response back downstream, and cache the (metric, attrs_fp) → - sid_GLOBAL mapping. -- Once an agent has cached a sid for a given series, all subsequent - Exports use sid_GLOBAL directly. The gateway sees sid != 0 and - forwards verbatim to backend without any cache lookup of its own. -- Backend's SketchStore can use sid_GLOBAL directly as the index key: - `series.get(sid_GLOBAL) → SketchInstanceMetadata + per-window state`. - -Wire savings at steady state: every Export carries ~8 B `series_id` -per DP instead of the full attribute set (~12-50 B per DP for our -{zone, rack, node, pod, producer_id} schema). - -### Idempotency invariant on `ResolveSeriesIDs` - -Backend's resolution is content-addressable: same `(metric_name, -attribute_set)` input MUST produce the same `series_id` output for -the lifetime of backend's cache. No fresh mint on re-resolution of -an existing identity. Implementation is the standard -"compute-or-mint" pattern: - -```rust -fn resolve(&self, metric: &str, attrs: &AttrSet) -> u64 { - let fp = canonical_fingerprint(metric, attrs); - self.cache.entry(fp).or_insert_with(|| self.next_sid()) -} -``` - -This is what makes attribute-fallback recovery work cleanly: when an -agent re-emits its `(metric, attrs)` after a crash, backend returns -the SAME sid that was assigned before the crash. Sketch state in the -backend stored under that sid stays coherent across the agent's gap. - -Cache fingerprint contract (already in the patched proto): -- Both sender and backend MUST compute `attrs_fp` from - sorted-by-key attribute (key,value) pairs in a stable - serialization -- The proto's `SeriesAssignment.attributes_fingerprint` field locks - this contract; senders use the SAME algorithm to look up cache hits - -### Fault tolerance — attribute-carrying fallback is the universal recovery path - -Anytime ANY component loses confidence in its sid cache (cold start, -crash recovery, partition reconnect, backend restart without -persistence), it falls back to emitting `(metric_name, full_attributes)` -+ `series_id = 0`. Backend looks up `(metric, attrs_fp)` in its own -cache: -- Cache hit → return existing sid (= same as pre-failure) -- Cache miss → mint new sid (controller may also re-plan for this - series), return in SeriesAssignment - -The bootstrap path and the recovery path are identical: empty -sender-side cache → emit with attributes → resolve sids → cache → -compact emission. No special "recovery mode" code path needed. - -### Wire protocol addition for cache invalidation - -```protobuf -message ExportMetricsServiceResponse { - ExportMetricsPartialSuccess partial_success = 1; - repeated SeriesAssignment series_assignments = 2; - - // NEW: sids the backend didn't recognize in this Export. The sender - // MUST evict these from its cache; subsequent emits MUST re-attach - // attributes so backend can re-resolve. Used after backend restart - // without cache persistence, or any other sid-cache divergence. - repeated uint64 unknown_series_ids = 3; -} -``` - -Per-DP `series_id` field semantics: -- `sid != 0, attributes empty` → use cached sid (compact mode) -- `sid == 0, attributes populated` → first-time / forced-resolve path -- `sid != 0, attributes populated` → optional belt-and-suspenders - (sender wants to re-confirm) - -### Failure scenario matrix - -| failure | recovery (no special code path) | -|--------------------------------------------|--------------------------------------------------| -| Agent crash + restart | empty cache → attribute fallback → backend cache returns SAME sid → resume | -| Gateway crash + restart | same, for the gateway's rolled-up identity cache | -| Backend restart WITH persisted sid-cache | transparent; cached sids still valid | -| Backend restart WITHOUT persistence | response.unknown_series_ids → senders evict → next emit with attributes → backend re-mints (fresh sids) → resume. Old sketch state in backend was also wiped, so no orphan reference. | -| Network partition | senders buffer/retry; cache unchanged on both sides; reconnect resumes | - -### Sid-cache durability options - -- **Option A (recommended for production)**: backend persists - `(metric, attrs_fp) → sid` to its on-disk store alongside sketch - state. Restart reloads; sids preserved. Co-located with the - sketches' own durability requirement. -- **Option B (alternative)**: deterministic sid = stable_hash(metric, - sorted_attrs). Stateless backend; collisions possible at 64-bit - hash (~10⁻¹⁰ at 1M series). Mitigations: use 128-bit identifier - proto change, or backend collision-detection that returns - unknown_series_ids on rare conflicts. -- **MVP (current)**: no persistence — sid-cache and sketch state both - in-memory; both wiped consistently on backend restart; recovery - via the attribute-fallback path described above. - -Implementation cost: the gateway changes from a terminating OTLP -receiver (currently mints its own sids) to a transparent forwarder -of upstream-provided sids. This is a small patch to the gateway's -processor pipeline + a new "series-id resolver" gRPC method on the -backend. - -### "Ghost" sids — registered but never carrying state - -A subtle consequence of centralized minting + per-hop wire identity: -some sids exist only as metadata, never see sketch state. - -Example: -``` -Agent processes per-rack: - registers (lat_ms, {zone=z0, rack=r00}) → sid=42 - registers (lat_ms, {zone=z0, rack=r01}) → sid=43 - emits sid=42, sid=43 to gateway (compact mode, wire savings ✓) - -Gateway rollup: drop rack, merge by zone: - registers (lat_ms, {zone=z0}) → sid=99 - emits ONLY sid=99 (merged) to backend - -Backend metadata cache: sid=42, sid=43, sid=99 all exist -Backend sketch state: only sid=99 accumulates -``` - -`sid=42` and `sid=43` are **ghost sids** — registered identities the -backend tracks for metadata + query routing, but no sketch state ever -arrives because the gateway folded them into sid=99. - -This is by design, not a bug: -- Agent's per-rack registration is the *wire identity* on the - agent→gateway hop, where wire savings matter most. Removing it - forces attribute-carrying mode on the hottest edge. -- Backend's per-rack metadata is what makes a future user query like - `count(http_requests_total{zone=z0, rack=r00})` resolve correctly - (to a Thanos fallthrough — no warm sketch exists, but raw archive - has the answer). - -**Storage layer requirement**: backend MUST NOT assume "every sid in -metadata has sketch state": - -```rust -fn query(&self, sid: u64, ...) -> QueryResult { - let metadata = self.instances.get(sid)?; - match self.series.get(sid) { - Some(series) if !series.is_empty() => - evaluate_sketch(series, ...), // active warm hit - _ => - Forward::Thanos { // ghost — fallthrough - metric: metadata.metric_name, - labels: metadata.group_by_keys, - } - } -} -``` - -Three query outcomes per sid lookup: -- metadata + series have state → ASAP-tier hit, evaluate sketch -- metadata only (ghost) → Thanos fallthrough on the registered identity -- no metadata (unknown sid) → response.unknown_series_ids; sender re-registers; fall through with original attrs if available - -Memory cost: each SketchInstanceMetadata is ~70-100 B. At 1M ghost -sids ≈ 70-100 MB — trivial vs active sketch state (KB-MB each). No -prune needed for MVP. - -Three design alternatives rejected: -1. Don't register pre-merge identities → loses agent→gateway wire compression. -2. Gateway notifies backend "merge sid=42→sid=99" → adds protocol complexity, breaks per-rack query routing. -3. Periodic prune of ghost sids → complicates fault tolerance (delayed packet after prune mints new sid). - -### Sketch merging composes correctly with centralized sids - -Sid encodes the identity tuple `(metric, group_by_labels)`, NOT the -physical source/path. Three merge cases: - -**Case A — identity-preserving merge** (most common): -``` -Agent A emits (lat_ms, {zone=z0}) — sketch over A's data -Agent B emits (lat_ms, {zone=z0}) — sketch over B's data -Both register identity tuple → backend returns same sid=42 to both - -Gateway's sketch-merge processor receives 2× sid=42 per window, -merges payloads, emits Export(sid=42, merged) to backend. -sid passes through; gateway is transparent for the sid. -``` - -**Case B — identity-changing rollup at gateway**: -``` -Agent A emits (lat_ms, {zone=z0, rack=r00}) — sid=42 -Agent A emits (lat_ms, {zone=z0, rack=r01}) — sid=43 - -Gateway's rollup processor drops `rack`, merges by zone alone: - output identity = (lat_ms, {zone=z0}) - gateway looks up its OWN cache for this rolled-up identity - cache miss → ResolveSeriesIDs → backend mints sid=99 - emits Export(sid=99, merged) - -Different identity → different sid. Same registration flow at gateway. -``` - -**Case C — cross-metric or cross-capability synthetic merge**: same -logic as B — output identity differs from any input, gateway resolves -a fresh sid via backend. - -Single rule for gateway: -``` -output_identity = (metric, surviving_labels) -if output_identity == input_identity: - output.sid = input.sid // Case A -else: - output.sid = gateway_cache.get_or_resolve(output_identity) // B / C -``` - -Backend remains the sole minter — every fresh `(metric, group_by)` -tuple from anywhere in the pipeline asks backend; backend dedups -identical tuples to the same sid. Each LOGICAL series has exactly -one sid regardless of how many merge stages it traverses. - -Backend's per-(sid, window) storage may still receive multiple -sketches per window (e.g. redundant gateways, no merge). Storage -layer handles this with a per-(sid, window) merge step using -`SketchInstanceMetadata.sketch_type` to pick the algorithm -(DDSketch.merge, HLL.union, CMS row-add, etc.). Same problem with -or without centralized sids — sids don't worsen it. - -## 5.5 Future-work — Step 6: multi-window batching per ScopeMetrics - -Today each agent emit produces one `Metric` with one window's worth of -`DataPoint`s (the sketch state at window-close). Wire framing per -emit: ResourceMetrics + ScopeMetrics + Metric envelopes ≈ 60-100 bytes -of overhead before the first DataPoint. - -Optimisation: if an agent flushes every N windows instead of every 1 -window, it can pack N consecutive windows of the same metric into one -`Metric.data_points[]` list — N DataPoints with monotonically advancing -`time_unix_nano`. The framing overhead amortises by N×, and `Metric.name` -+ `Metric.unit` + parent sketch container's config fields -(`relative_accuracy`, `precision`, etc.) are all sent ONCE per N windows. - -Trade-off: introduces flush-latency per emit = N × window_duration. At -N=4 with window=30s, agent buffers up to 2 minutes of state before -emit — affects criterion ⑥ freshness but not correctness or query -results. Operators choose N per their freshness budget. - -Tracked as Step 6. - -## 6. What does NOT change - -- OTel data plane (OTLP gRPC, sketch payload variants in pdata). -- Gorilla XOR chunk format, MinIO bucket layout for raw archive. -- Thanos store-gateway / thanos-query / thanos-compact sidecars. -- B0 / B1 baselines (raw → Prometheus PRW). -- Multi-node deploy topology. diff --git a/docs/design-sid-lifecycle.md b/docs/design-sid-lifecycle.md deleted file mode 100644 index 54ab2439..00000000 --- a/docs/design-sid-lifecycle.md +++ /dev/null @@ -1,559 +0,0 @@ -# Sid lifecycle across asapcollector and asapquery-backend - -**Status:** in effect since PR #190 + PR #192 (May 2026). Replaces the -prior content-addressed `compute_sketch_sid` / `compute_sid` model. - -This doc describes how a **series identifier (sid)** — the compact u64 -the backend uses internally to key per-series state — is minted, -distributed, persisted, and recovered as data flows from the -asapcollector through the asapquery-backend. It also discusses how to -extend the design to a multi-shard backend without breaking the -identity contract. - ---- - -## 1. Identity contract - -> **`sid = registry-allocated u64 for the triple (metric_name, -> attrs_fingerprint, agg_kind_canonical)`** - -- **Authority:** exactly one process — the backend's `SeriesIdResolver` - — mints sids. `AtomicU64::fetch_add(1)` gives uniqueness by - construction, not as a probability. -- **Wire shape:** `u64`. `0` is reserved for "unresolved" on the wire. -- **Bandwidth optimization:** once a sender caches a sid, subsequent - emits for the same series omit the `attributes` field on the wire - and send the sid alone — the receiver disambiguates `sid → (metric, - attrs, agg_kind)` through its own cache. -- **Identity granularity:** different aggregations over the same - series get **different** sids. A DDSketch and a Sum on - `http_latency_ms{zone=z0}` mint two distinct sids because their - `agg_kind_canonical` strings differ. This matches the historical - identity model `compute_sketch_sid` enforced via its 4-input hash. -- **Stability:** for the lifetime of a resolver's cache (in-memory + - WAL), the same triple always returns the same sid. Across resolver - resets (no persistence, or fresh deployment) sids are NOT stable — - recovery is automatic via the `unknown_series_ids` eviction - primitive (§4). -- **Determinism across hosts:** the model is **deliberately not** - host-independent today. Each backend mints its own sid space. §5 - discusses extending to a deterministic distributed scheme. - -The `agg_kind_canonical` string is produced by -`AggKind::canonical_string()` in -`data_plane/src/storage_engines/sketch_db/data/mod.rs`. Stable form: - -``` -sketch:: e.g. sketch:DDSketch:D:0.01 - sketch:Kll:K:200 - sketch:Hll:H:14 -precompute:: e.g. precompute:Sum: - precompute:DatasketchesKLL:k=200; -``` - -Two `AggKind` values that compare equal MUST produce the same -canonical string; two that differ in any observable parameter MUST -produce different strings. - -## 2. Component responsibilities - -``` - asapcollector asapquery-backend - ───────────────────────────────── ────────────────────────────────────────── - - Patched OTel-Go exporter drivers/ingest/otel.rs - • per-tenant dictionary • OTLP gRPC + HTTP receivers - • canonical_attrs_fingerprint • per-DP wire dispatch (§3) - • cache hit → omit attrs - • cache miss → send attrs drivers/ingest/series_resolver.rs - • SeriesIdResolver (single mint) - Gateway (when present) • FilePersistence (WAL "ASAPSRP\x02") - • transparent forwarder for sid-bearing - DPs (same sid passes through) storage_engines/sketch_db/index/mod.rs - • for rollup-output identities, the • SketchStore - gateway calls back to the backend's instances: sid → SketchInstanceMetadata - SeriesIdResolver (same flow as the series: sid → SidStoreData (columnar) - agent — it's just a hop closer) classify(sid) → Hit / Ghost / Unknown - - precompute_engine/output_sink.rs - • SketchStoreSink (live precompute output) - • mints sid via resolver, writes to SketchStore - - storage_engines/sketch_db/backfill/processor.rs - • BackfillWindowProcessor - • mints sid via resolver, writes to SketchStore - - query_engines/asap_query_engine/engine.rs - • ASAPQueryEngine (ASAP-tier reads) - • discovers sids via - SketchStore::instances_matching(metric, gbk) -``` - -The fingerprint algorithm is shared: both sides compute -`canonical_attrs_fingerprint(attrs) = sort_by_key(attrs).join("k=v;")`, -defined in `data_plane/src/drivers/ingest/series_resolver.rs` and -mirrored in -`opentelemetry-go-patch/.../internal/series/dictionary.go::attributesFingerprint`. - -## 3. End-to-end data flow - -### 3.1 Architecture diagram - -``` - ASAPCOLLECTOR ASAPQUERY-BACKEND - ──────────────────────────────────── ────────────────────────────────────────── - - Patched OTel-Go exporter - ┌─────────────────────────────────┐ ┌────────────────────────────────────────────┐ - │ per-tenant sid dictionary │ │ drivers/ingest/otel.rs │ - │ HashMap<(metric, attrs_fp),sid> │ gRPC Export │ ┌────────────────────────────────────────┐ │ - │ │ ───────────────────▶ │ │ SeriesIdResolver (single mint auth) │ │ - │ fp = canonical_attrs_ │ resource_metrics: │ │ │ │ - │ fingerprint(attrs) │ DPs with sids │ │ next_sid : AtomicU64::fetch_add(1) │ │ - │ sort by key, "k=v;..." │ + optional attrs │ │ cache : DashMap< │ │ - │ │ + sketch bytes │ │ (metric, fp, agg_kind_canonical), │ │ - │ PER DATAPOINT: │ │ │ u64 │ │ - │ hit → send (sid, ∅, bytes) │ Export Reply │ │ > │ │ - │ miss → send (0, attrs, bytes) │ ◀─────────────────── │ │ │ │ - │ │ series_assignments │ │ resolve(m, fp, ak) -> sid │ │ - │ ON REPLY: │ unknown_series_ids │ │ idempotent; fsync-per-mint via │ │ - │ cache new assignments │ │ │ FilePersistence (WAL "ASAPSRP\x02") │ │ - │ evict unknown sids │ │ │ replay on open() restores cache │ │ - │ → next emit will send attrs │ │ └─────┬──────────────────────────────────┘ │ - └─────────────────────────────────┘ │ │ resolve(...) │ - │ │ │ - │ ┌───┴────────────┐ │ - │ ▼ ▼ │ - │ SKETCH INGEST PRECOMPUTE INGEST │ - │ otel.rs:879 output_sink.rs + │ - │ backfill/processor.rs │ - │ ak = AggKind:: ak = AggKind:: │ - │ Sketch{ Precompute{ │ - │ kind, agg_type, │ - │ config} params} │ - │ .canonical_ .canonical_ │ - │ string() string() │ - │ "sketch: "precompute:Sum:" │ - │ DDSketch: │ - │ D:0.01" │ - │ │ │ │ - │ └────────┬───────┘ │ - │ ▼ │ - │ ┌──────────────────────────────────────┐ │ - │ │ SketchStore (sid → state) │ │ - │ │ instances : HashMap │ │ - │ │ series : DashMap │ │ - │ │ classify(sid) → Hit/Ghost/Unknown │ │ - │ └──────────────┬───────────────────────┘ │ - │ │ query │ - │ ┌──────────────▼───────────────────────┐ │ - │ │ ASAPQueryEngine (ASAP tier) │ │ - │ │ analyze_promql → │ │ - │ │ ASAPTierCandidate(metric, gbk, cap) │ │ - │ │ → instances_matching → [sids] │ │ - │ │ → SketchReducer.evaluate │ │ - │ └──────────────────────────────────────┘ │ - └────────────────────────────────────────────┘ -``` - -### 3.2 Wire cases (per DP inside an Export) - -``` - sid attrs backend action backend reply - ───── ───── ────────────────────────────────────── ────────────────────────────────────────── - 0 set resolver.resolve(m, fp, ak) → mint/hit series_assignments += {fp → sid} - ≠0 set resolver.resolve → s series_assignments += {fp → s}; - if sender_sid ≠ s → stale if sender_sid ≠ s: unknown += [sender_sid] - ≠0 none SketchStore.classify(sid): - Hit → append sketch_bytes (no reply for this DP) - else → drop, signal sender unknown += [sid] - 0 none invalid wire shape drop -``` - -### 3.3 Backend-internal sources of sid mints - -| Path | File | When | `agg_kind` | -|---|---|---|---| -| OTel sketch ingest | `drivers/ingest/otel.rs::route_modified_otlp_sketches_to_precompute` | gRPC/HTTP Export with sketch DPs | `Sketch { kind, config }` | -| Live precompute output | `precompute_engine/output_sink.rs::SketchStoreSink::append_to_index` | Window completed by `PrecomputeEngine` worker | `Precompute { agg_type, params }` | -| Backfill window write | `storage_engines/sketch_db/backfill/processor.rs::BackfillWindowProcessor::process_window` | Replay from archive (Prometheus / S3 source) | `Precompute { agg_type, params }` | - -All three paths receive an `Arc` at construction and -call `resolver.resolve(metric, fp, agg_kind_canonical)` to mint. -`SketchStore::ingest_precompute_for_agg_config` takes the mint -authority as a closure parameter so the storage layer stays free of -any layer-inverted dependency on `drivers::ingest`. - -## 4. Failure recovery - -The single recovery primitive is `unknown_series_ids` in the -`ExportMetricsServiceResponse`. The collector evicts any sid listed -there; the next Export sends the corresponding attrs; the resolver -re-mints (or cache-hits) and replies via `series_assignments`. The -same primitive handles every cache-divergence failure mode. - -### 4.1 Sequence: cold start (collector cached, backend never seen this sid) - -``` - Collector Backend - │ │ - │ Export DP{ sid=42, attrs=∅ } │ (collector has cached sid 42) - │ ──────────────────────────────────▶│ - │ │ SketchStore.classify(42) = Unknown - │ │ resolver.cache has no (m, fp, ak)→42 - │ │ - │ Reply: unknown_series_ids = [42] │ - │ ◀──────────────────────────────────│ - │ │ - evict 42 │ - │ │ - │ Export DP{ sid=0, attrs=..., │ - │ sketch_bytes = ... } │ - │ ──────────────────────────────────▶│ - │ │ resolver.resolve(m, fp, ak) - │ │ → fresh mint: sid = 1 - │ │ SketchStore.register({ sid:1, ... }) - │ │ SketchStore.append_sample(1, ...) - │ │ - │ Reply: series_assignments=[(fp,1)]│ - │ ◀──────────────────────────────────│ - │ │ - cache (fp → 1) │ - (next emit can omit attrs) │ -``` - -### 4.2 Sequence: stale sender sid (e.g. after a backend wipe + restart) - -``` - Collector Backend - │ │ - │ Export DP{ sid=99, attrs=..., } │ (sender thinks 99 is right) - │ ──────────────────────────────────▶│ - │ │ resolver.resolve(m, fp, ak) → 7 - │ │ (cache miss or hit on different sid) - │ │ sender_sid (99) ≠ resolver_sid (7) - │ │ → unknown += [99] - │ │ SketchStore.register({ sid:7, ... }) - │ │ SketchStore.append_sample(7, ...) - │ │ - │ Reply: │ - │ series_assignments = [(fp, 7)] │ - │ unknown_series_ids = [99] │ - │ ◀──────────────────────────────────│ - │ │ - evict 99 │ - cache (fp → 7) │ -``` - -### 4.3 Sequence: backend restart **with** persistence (WAL replay) - -``` - Before restart: - • Collector cache: (fp → 42) - • Backend resolver: (m, fp, ak) → 42 in cache - record on disk in WAL - • SketchStore: instances[42] = MetaT{...} - - Backend restart: - SeriesIdResolver::open(path) → replay WAL → cache restored, - next_sid = max_replayed + 1 - - Collector Backend - │ │ - │ Export DP{ sid=42, attrs=∅, │ - │ sketch_bytes = ... } │ - │ ──────────────────────────────────▶│ - │ │ SketchStore.classify(42) = Hit - │ │ SketchStore.append_sample(42, ...) - │ │ - │ Reply: (no signals) │ - │ ◀──────────────────────────────────│ - │ │ - (no eviction, no extra round trip; │ - sender's cache stays valid; WAL replay │ - absorbed the restart transparently) │ -``` - -### 4.4 Sequence: backend restart **without** persistence - -Same shape as §4.1 (cold start) for every active series. With M live -identities the collector pays one extra round trip per identity on -first post-restart emit — bandwidth blip, but no data loss. - -### 4.5 Durability semantics - -- `FilePersistence::append(sid, metric, fp, agg_kind_canonical)` calls - `fsync` before returning. Callers are blocked until the record is on - stable storage. The resolver only returns the sid to its caller - after this returns successfully. -- A torn write at EOF (kernel buffered bytes but the metadata flush - was interrupted) is detected at replay via short-read on any record - field; the file is truncated to the last durable record's offset. -- WAL format v2: 8-byte magic header `b"ASAPSRP\x02"`, then a stream - of records each shaped as `(u64 sid LE, u32 metric_len LE, metric - utf8, u32 fp_len LE, fp utf8, u32 agg_kind_len LE, agg_kind_canonical - utf8)`. Append-only; sids never get rewritten, so the log grows in - proportion to live cardinality (~250-400 bytes per record). -- `xxhash_rust::xxh64` is no longer imported in - `sketch_db::data` — content-addressed sid hashing is gone from the - data plane entirely. - -## 5. Scaling to distributed asapquery-backend - -The single-backend design above relies on one `SeriesIdResolver` -process being the sole sid mint. To scale write or read throughput -beyond a single host, the system needs more than one backend. This -section sketches the extension; **none of it is implemented today**. - -### 5.1 The two coordination problems - -1. **Mint coordination.** When two backends each run a - `SeriesIdResolver::resolve(m, fp, ak)` they will produce different - sids — both correct in their own local namespace, but ambiguous to - any agent that talks to both. Without a coordination protocol, sids - are *not* globally unique. - -2. **Query fan-out.** A query for `metric` needs to consult every - backend that owns sids for that metric. The query path either - knows the shard topology, or proxies through a coordinator that - does. - -### 5.2 Recommended sharding axis: `hash(tenant, metric)` - -``` - DISTRIBUTED ASAPQUERY-BACKEND (sharded by tenant + metric) - ──────────────────────────────────────────────────────────── - - ┌───────────────────────────┐ - │ Routing layer │ - │ (collector-side or │ - │ gateway-side) │ - │ │ - │ shard_id(metric,tenant) │ - │ = consistent_hash(...) │ - │ % N_shards │ - └────────────┬──────────────┘ - │ - ┌────────────────────────────┼─────────────────────────────┐ - ▼ ▼ ▼ - ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ - │ Backend shard 0 │ │ Backend shard 1 │ ... │ Backend shard N │ - │ │ │ │ │ │ - │ SeriesIdResolver │ │ SeriesIdResolver │ │ SeriesIdResolver │ - │ sid space: │ │ sid space: │ │ sid space: │ - │ [0<<56, │ │ [1<<56, │ │ [N<<56, │ - │ (1<<56)-1] │ │ (2<<56)-1] │ │ ((N+1)<<56)-1]│ - │ │ │ │ │ │ - │ SketchStore │ │ SketchStore │ │ SketchStore │ - │ WAL (per-shard) │ │ WAL (per-shard) │ │ WAL (per-shard) │ - └─────────┬─────────┘ └─────────┬─────────┘ └─────────┬─────────┘ - │ │ │ - └───────────────────────────┴─────────────────────────────┘ - ▲ - │ PromQL / api/v1/query - │ - ┌───────────┴────────────┐ - │ Query coordinator │ - │ │ - │ • Determine shard set │ - │ for the query's │ - │ metric │ - │ • Fan out │ - │ • Per-shard reducer │ - │ • Combine results │ - └────────────────────────┘ -``` - -The sharding key is `(tenant, metric)`, not `(tenant, metric, attrs)`. -A single metric's series live on one shard, so ASAP-tier queries that -need to merge sketches across attribute values stay local to one -backend. Cross-shard queries (e.g. spanning multiple metrics) need -the coordinator to fan out. - -### 5.3 ID-space partitioning (no coordination on mint) - -Reserve the **top 8 bits** of the u64 sid for `shard_id`. The bottom -56 bits are a per-shard local counter: - -``` -sid layout (u64, little-endian on wire): - - ┌────────────────┬───────────────────────────────────────────────────────┐ - │ shard_id : 8 │ local_counter : 56 │ - └────────────────┴───────────────────────────────────────────────────────┘ - ▲ ▲ - │ │ - │ └── next_sid: AtomicU64::fetch_add(1) - │ with shard_id pre-OR'd in - │ - └── assigned at backend boot from - deployment topology (e.g. statefulset ordinal, - Kubernetes Pod label, etcd lease) -``` - -Properties: - -- **No coordination on the mint path.** Each backend's resolver runs - the same `fetch_add` it does today; the `shard_id` is OR'd in. -- **Global uniqueness by construction.** Different shards cannot - produce the same u64 because their top 8 bits differ. -- **Cap:** 256 shards × ~72 quadrillion sids each. Comfortable even - for very large multi-tenant deployments. -- **Routing-by-sid is O(1):** `shard_id = (sid >> 56)`. The query - coordinator extracts it and routes per-sid reducer queries to the - owning shard without a lookup table. -- **Backend identity:** the `shard_id` must be stable across restarts - for a given backend instance. Source it from a durable identity - (statefulset ordinal + zone, or an etcd lease). A mismatch on - restart would re-mint existing identities under a different `shard_id` - → data loss; treat `shard_id` as part of the WAL header and refuse - to open a WAL whose embedded id disagrees with the runtime id. - -### 5.4 Query-side fan-out - -Inside the query coordinator: - -1. Parse PromQL via the existing control plane analyzer (single - authority — see `control_plane/src/asap_tier_analysis.rs`). -2. Extract the candidate's `metric`. Compute `shard_set = - sharder(tenant, metric)`. For most queries this is exactly one - shard. -3. For each shard, issue a `SketchStore::instances_matching(metric, - gbk) → [sids]` RPC. Backends respond with sids in their own range, - plus the per-sid metadata the reducer needs. -4. Issue per-shard `SketchReducer.evaluate(sids, function, args, t0, - t1)` RPCs in parallel. Each shard reduces locally; coordinator - merges via the existing `combine_statistic` logic. -5. Stitch with archive via `EngineRouter` exactly as today - (capability-miss → fail over to Thanos). - -`series_assignments` returned to the collector contains sids whose top -8 bits identify the owning shard. The collector caches the binding as -opaque u64; subsequent emits for the same `(metric, fp)` always go to -the same shard because routing is keyed on `(tenant, metric)` not on -the sid value. - -### 5.5 High-availability within a shard - -Per-shard, three options: - -| Approach | Cost | RPO | RTO | -|---|---|---|---| -| **Active / passive with shared WAL on S3** | Standby replays WAL from S3 on failover. | Up to `fsync` interval | Seconds | -| **Active / passive with EBS-snapshot WAL** | Standby attaches the EBS volume. | Zero (sync replica) | Tens of seconds | -| **Raft on the WAL** | Three replicas commit each `append` via Raft. | Zero (quorum durability) | Seconds | - -The Raft option is the strongest but requires every mint to wait on a -quorum write — slower than today's local `fsync`. Active/passive on -S3 matches the pattern already used for the SketchStore's persistence -tier and is the natural first step. - -### 5.6 Migration path from single-backend to sharded - -A single-backend deployment maps cleanly to `N_shards = 1`, -`shard_id = 0`. The 8 high bits of every minted sid would be `0x00`, -which is what today's sids effectively look like (max u64 minted is -nowhere near `1<<56`). So: - -1. **Now:** ship `shard_id` into the WAL header and the resolver's - `next_sid` initialization, defaulted to `0`. No behavior change for - single-backend deploys. -2. **Two-shard step:** bring up a second backend at `shard_id = 1`. - Stand up a routing layer (collector-side first; gateway-side - later) that hashes `(tenant, metric)` and forwards. Existing - single-shard sids stay valid (top bits stay `0x00`); new mints on - the second backend land in `[1<<56, …)`. Query coordinator fans - out across both. -3. **N-shard:** scale horizontally. Each addition costs a config push - to the routing layer and a fresh backend with a fresh `shard_id`. - -The migration is fully online: no sid wire-format change, no -collector update required. - -### 5.7 Cross-shard rebalancing - -Resharding is the hard problem. Once series are minted on shard K, -moving them to shard K' requires re-minting (because `shard_id` is -embedded in the sid). Options: - -- **Live drain + re-mint:** route all new emits for a `(tenant, - metric)` to the new shard. Old shard's sids stay until their data - TTLs out (matches the existing `SchemaEvictionService` shape — sids - retire and expire on a clock). -- **Bulk re-emit:** force the collector to evict all sids for a given - `(tenant, metric)` via an out-of-band signal (e.g. a control-plane - message that pre-populates `unknown_series_ids` on the next - Export). Costs a round trip per identity but completes in one - refresh cycle. - -Neither approach is implemented; the migration path in §5.6 doesn't -need them at first since single → sharded only adds shards, it doesn't -move existing series. - -## 6. Open questions - -1. **Should `ResolveSeriesIDs` RPC stay or go?** Today's `SeriesQuery` - proto carries only `(metric, fp)` — under the new identity model - the handshake can't produce the correct sid (no `agg_kind`). The - RPC is currently stubbed to return empty `assignments` with a WARN - log. Either drop it entirely (proto change), or extend - `SeriesQuery` with `agg_kind_canonical`. - -2. **WAL compaction.** The WAL grows in proportion to live cardinality - (~25-40 GB at 100M sids). At what threshold is a snapshot + log - truncation cycle worth scheduling? The append-only shape is fine - for the foreseeable scale; the open question is operational, not - structural. - -3. **Multi-tenant isolation of `next_sid`.** Today's resolver shares - one counter across all tenants. A noisy-tenant flood could exhaust - the bottom 56 bits faster than steady-state cardinality suggests. - Reserving a per-tenant subspace inside the bottom 56 bits would - address this; the cost is reduced per-tenant headroom. Not urgent. - -4. **Collector-side multi-shard cache.** When the system sharates, the - collector's local dictionary `HashMap<(metric, fp), sid>` is still - correct (the sid is opaque to the collector), but the routing - layer needs to direct each emit to the right shard. Is the routing - table colocated with the exporter or moved into the gateway? Latency - vs. operational complexity trade-off. - -5. **Cross-shard PromQL semantics.** Queries over `metric` that's - sharded on `(tenant, metric)` stay local. Queries over multiple - metrics fan out trivially. But aggregations like `sum by (...) ( - m_a + m_b )` (binary op across two metrics that live on different - shards) need a strategy — coordinator-side join, or push-down by - sketch combinability? Outside the sid lifecycle proper but - relevant when adopting §5. - -## 7. References - -### Code - -- Resolver + WAL: `data_plane/src/drivers/ingest/series_resolver.rs` -- OTel ingest path: `data_plane/src/drivers/ingest/otel.rs` - (`route_modified_otlp_sketches_to_precompute`, - `MetricsServiceImpl::export`, `MetricsServiceImpl::resolve_series_i_ds`) -- Precompute output: `data_plane/src/precompute_engine/output_sink.rs::SketchStoreSink` -- Backfill output: `data_plane/src/storage_engines/sketch_db/backfill/processor.rs::BackfillWindowProcessor` -- Storage layer: `data_plane/src/storage_engines/sketch_db/index/mod.rs::SketchStore`, - `ingest_precompute_for_agg_config` -- Identity canonicalization: `data_plane/src/storage_engines/sketch_db/data/mod.rs::AggKind::canonical_string` -- Proto: `crates/asap_otel_proto/proto/opentelemetry/proto/collector/metrics/v1/metrics_service.proto` - -### Related design docs - -- `docs/design-controller-into-backend.md` §5.4 — original - idempotency invariant for `ResolveSeriesIDs`. -- `docs/design-sketch-db.md` — SketchStore lifecycle, ghost sids, - per-sid columnar storage. -- `docs/design-sketch-db-pluggable.md` — how `AggKind` discriminates - sketch vs precompute payloads at the storage layer. - -### Landed PRs implementing this design - -- **#190** (commits 84abac7, 3ac90cf, f72b533) — registry-allocated - sid + WAL persistence + identity contract on the OTel ingest path. -- **#192** (commit 9c8c614, merged as 8f923db) — migrate precompute - output and backfill to the same resolver; delete `compute_sid` and - helpers. diff --git a/docs/design-simple-map-store-persistence.md b/docs/design-simple-map-store-persistence.md deleted file mode 100644 index 856e5c01..00000000 --- a/docs/design-simple-map-store-persistence.md +++ /dev/null @@ -1,811 +0,0 @@ -# Design: SketchStore Persistence (Memory Limit + Disk Flush) - -## Problem - -`SketchStore` (`data_plane/src/storage_engines/simple_map_store/`) is currently an -in-memory-only store. Under long-running ingest it grows unboundedly: every sealed -window for every `(aggregation_id, group_key)` is held in `DashMap>` -until one of the three existing `CleanupPolicy` variants (`CircularBuffer`, -`ReadBased`, `NoCleanup`) either rotates it out and drops it on the floor or does -nothing at all. - -All three of those variants are **destructive** — they delete data, they do not -persist it. That creates two problems: - -1. **No memory bound that preserves data.** A deployment has to either - overprovision RAM (`NoCleanup`), throw away potentially query-relevant data - (`CircularBuffer` / `ReadBased`), or tune per-agg `num_aggregates_to_retain` - values that don't correspond to any operator-meaningful quantity. -2. **No durability.** Cold data (older than the query working set) still occupies - RAM even though most queries hit only the last few minutes. - -We want `SketchStore` to replace the existing cleanup-policy knob with a -single persistence policy driven by **two** knobs, in priority order: - -1. **Primary — memory budget.** A configurable hard ceiling on in-memory sketch - bytes. When exceeded, the oldest sealed epochs flush to disk until usage is - back under a low-water mark. This is what actually bounds RAM in production. -2. **Secondary — time watermark T.** A configurable "hot window." Any sealed - epoch whose end is older than `now - T` flushes to disk even if the store - is nowhere near the memory budget. This guarantees predictable durability - and a stable hot-set size under light load. - -Flushed sketches are read back transparently at query time. - -Scope is **single-node, single-process**. Replication, sharding, compression, -and query pushdown into segments are explicitly out of scope for v1. The three -existing destructive `CleanupPolicy` variants are removed from `SketchStore` -(the enum stays in `asap_types` for any other store that still uses it). - ---- - -## Current shape (relevant facts) - -- `SketchStorePerKey` (`per_key.rs:160`) keeps per-agg-id state in - `DashMap>>`. -- Each `StoreKeyData` has a `current_epoch` (actively being written) and - `sealed_epochs: BTreeMap` (`per_key.rs`). -- Values are `Arc`. All concrete accumulators already implement - `SerializableToSink` (`serialize_to_bytes`, `merge_with`) — so we already have - a serialization primitive and a merge primitive. -- Insert hot path: `insert_precomputed_output_batch` → `insert_for_store_key` - (`per_key.rs:261`), holding only the per-agg-id `RwLock::write`. -- Query hot path: `query_precomputed_output{,_exact}` iterates - `current_epoch` + `sealed_epochs` under `RwLock::read`. -- `CleanupPolicy` (`asap_types::enums`) currently has three destructive variants - (`CircularBuffer`, `ReadBased`, `NoCleanup`); `SketchStore` will stop - taking a `CleanupPolicy` at all and use the new persistence config instead. - The enum itself stays in `asap_types` for other stores. - -Key observation: **`current_epoch` is the only mutable region**. Sealed epochs are -append-only until cleanup. That is exactly the right unit to flush. - ---- - -## Design - -### Unit of flush vs. unit of file: the *part* - -There are two granularities to separate cleanly: - -- **Unit of flush = sealed epoch.** Same as before. Sealed epochs are - immutable, have a well-defined time range, and can be spliced out of - `sealed_epochs` under a brief per-agg `RwLock::write`. -- **Unit of file = *part*.** A part is **one flush tick's worth of sealed - epochs bundled into a single on-disk directory**, regardless of which - agg-id they came from. The flusher already assembles all the candidate - epochs for a tick before it touches the disk; instead of writing N - separate segment files and fsyncing each, it writes one part. - -This is the same pattern every mainstream TSDB converges on — Prometheus -blocks, InfluxDB TSM, VictoriaMetrics parts — for the same reasons: -file count is bounded by flush ticks (not by individual epochs), metadata -overhead is amortized across many entries, and compaction becomes a pure -directory-level merge. - -The `current_epoch` is never flushed while hot. It becomes flushable the -moment the rotator seals it, at which point it becomes a candidate for the -next flush tick's part. - -### Disk layout - -``` -/ -├── parts_manifest.log # append-only log of part additions + deletions -├── parts_manifest.snapshot # periodic binary snapshot (compaction of the log) -└── parts/ - ├── 0000000001/ # part directory, name = monotonic part_id - │ ├── meta.bin # fixed-size header: min_ts, max_ts, counts, crc - │ ├── data.bin # all epoch payloads concatenated, 8-byte aligned - │ └── index.bin # sorted array of entries, mmap-binary-search target - ├── 0000000002/ - │ ├── meta.bin - │ ├── data.bin - │ └── index.bin - └── ... -``` - -**Why parts instead of dir-per-agg with file-per-epoch:** - -- **File count scales with flush ticks, not with epochs.** On a 1-second - flush interval with 200 agg-ids and 1-minute windows, the old layout - produced ~288K files/day; the part layout produces ~86K files total - (one tick = three files: `meta.bin`, `data.bin`, `index.bin`). That is - the difference between "inode pressure is a real concern" and "we are - well within every filesystem's comfort zone." -- **Metadata is amortized.** One `fallocate` + one `fdatasync` per - `data.bin` covers all epochs in the tick, rather than N separate - allocations and N separate fsyncs. Group-commit is now an intrinsic - property of the layout, not something the flusher has to arrange. -- **Per-part in-file index.** Queries binary-search the part's `index.bin` - rather than linear-scanning a segment body. O(log N) per part instead - of O(N), and `index.bin` is mmap-friendly so the search is pure - pointer arithmetic with no syscalls. -- **Aggs are interleaved inside a part, not segregated by directory.** - No wasted directories for low-traffic aggs; the in-part index handles - agg lookup cheaply. -- **T2 retention is whole-directory.** Each part covers a tight time - range (roughly `flush_interval_ms`), so `delete_older_than_ms` - operates at part granularity — `rm -rf parts/000001234/` — instead of - touching shared files. -- **Compaction fits naturally.** A background compactor can merge N - adjacent old parts into one larger part with the same on-disk shape. - Readers don't care because the parts_manifest gets updated atomically - and old part dirs get removed only after all in-flight readers are done. - -**Part file formats** (all little-endian, fixed layouts, mmap-friendly, -8-byte aligned, written via `fallocate` + streaming CRC — same perf -details that applied to segments, now applied to `data.bin`): - -``` -meta.bin (128 bytes, fixed) - [u32 magic][u16 version][u16 flags] - [u64 part_id][u64 min_ts][u64 max_ts] - [u32 num_entries][u32 num_aggs] - [u64 data_len][u64 index_len] - [u64 created_unix_ns] - [u32 _reserved; 6] - [u32 crc32 of the above] - -data.bin (sum of padded payloads) - repeated num_entries times, in the order the index lists them: - [payload_len bytes: serialize_to_bytes()] - [0..7 bytes: tail padding to 8-byte boundary] - -index.bin (32 bytes per entry, sorted by (agg_id, start_ms)) - repeated num_entries times: - [u64 agg_id][u64 start_ms][u64 end_ms] - [u32 data_offset][u32 payload_len] - [u32 crc32 of the above][u32 _pad] -``` - -`index.bin` is the only file a query needs to traverse to locate entries -inside a part. It is small (32 B × num_entries, typically tens of KB), is -mmap'd on first access, and a binary search by `(agg_id, start_ms)` lands -on the byte range inside `data.bin` with one pointer-arithmetic step and -zero decode work. - -**Parts manifest: append-only log + periodic snapshot.** - -The manifest is the one piece of global state on disk. The previous -design had it as a JSON file rewritten on every flush tick — quadratic -over the lifetime of the deployment. We replace it with the standard -LSM-style pattern: - -- **`parts_manifest.log`** is an append-only binary file. Each flush - tick appends one record (add-part or delete-part, both fixed-size). - Appending is a single `write + fdatasync` on a file whose size is - proportional to the number of *ticks*, not the number of parts that - have ever existed. Cheap and O(1) per tick. -- **`parts_manifest.snapshot`** is a periodic binary snapshot of the - live set of parts, produced by replaying the log and emitting a flat - sorted array of `(part_id: u64, min_ts: u64, max_ts: u64, size: u64)` - = 32 bytes per live part. The snapshot is rewritten atomically (write - tmp → fsync → rename) whenever the log gets large relative to the - snapshot, and the log is truncated after. Snapshot + remaining log - is always the authoritative live state. -- **On startup**, the store loads the snapshot (mmap + direct cast, no - parse), then replays any tail of the log added since the snapshot was - taken, then verifies every live part directory's `meta.bin` CRC. - Sweep orphan part dirs (present on disk but not in the replayed - state) and treat them as a mid-flush crash — delete them. - -Binary formats throughout mean parse time is effectively zero; the -snapshot is "cast a byte slice to `&[PartEntry]`" — which works because -we declared the layout 8-byte aligned and fixed-size. - -**Durability ordering per flush tick** (the invariant a crash must not -violate: no part is referenced in the manifest until its bytes are on -disk): - -1. Assemble the tick's candidate epochs (in-memory, no I/O). -2. `fallocate` the three files in `parts//`, stream payloads - into `data.bin`, stream index into `index.bin`, write `meta.bin`. -3. `fdatasync` `data.bin`, `index.bin`, `meta.bin` (batched — one - syscall per file, not per entry). -4. `fsync` the part directory itself. -5. Append the add-part record to `parts_manifest.log` and `fdatasync` - the log. -6. `fsync` `` (the root) so the log's size update is durable. - -A crash at any step before (5) leaves an orphan part directory that -startup sweep deletes. A crash between (5) and (6) is fine — the log -record is already durable via (5). After (6), the part is officially -live and the flusher may evict the corresponding epochs from memory. - -### Memory accounting - -Add a new trait method: - -```rust -pub trait AggregateCore: ... { - fn approx_memory_bytes(&self) -> usize; -} -``` - -Implementations are cheap per-type estimates (e.g. KLL: `k * 8 + overhead`; -SumAccumulator: `size_of::()`; SetAggregator: `len * avg_entry_bytes`). -They do not call `serialize_to_bytes` — that would be too expensive on the -insert path. - -The store tracks a single `AtomicUsize` `mem_bytes_in_use`. On insert it adds -`approx_memory_bytes()` per entry; on flush it subtracts the same. This is an -estimate, not a hard guarantee — good enough to drive policy, and the alternative -(exact heap accounting) is not worth the allocator coupling. - -### Configuration - -New struct, threaded through `PrecomputeEngineConfig` and loaded from the -same YAML / control plane channel as the existing streaming config. The two -knobs match the priority order in the problem statement: **memory budget -first, time watermark second**. - -```rust -pub struct SketchStorePersistenceConfig { - // ---- Primary: memory budget ---- - // - // High-water mark. When the store's tracked in-memory sketch bytes - // exceed this, the background flusher evicts sealed epochs - // oldest-first (globally, by epoch end_ms) until usage drops below - // `memory_low_watermark_bytes`. This is the knob that bounds RAM - // in production. - pub memory_limit_bytes: usize, - pub memory_low_watermark_bytes: usize, - - // Hard ceiling. If memory usage reaches this *during* an insert - // (flusher is falling behind), the insert path blocks on a condvar - // until the flusher catches up. Set to memory_limit_bytes * 1.25 - // as a sensible default. - pub hard_cap_bytes: usize, - - // ---- Secondary: time watermark T ---- - // - // Hot-window length. Any sealed epoch whose end_ms is older than - // `now - hot_window_ms` is flushed on the next flusher tick, even - // if the store is well under `memory_limit_bytes`. This guarantees - // durability and a predictable hot-set size under light ingest. - // None disables time-based flushing (not recommended — memory - // pressure alone will still work, but cold data will linger in RAM - // until something pushes it out). - pub hot_window_ms: Option, - - // ---- Disk retention ---- - // - // Cold-tier TTL. Any segment whose end_ms is older than - // `now - delete_older_than_ms` is deleted from disk on the next - // flusher tick (after its references are removed from the manifest - // and no in-flight query is reading it). Bounds disk usage and keeps - // the manifest small enough to stay in L2/L3 cache on long-running - // deployments. Must be strictly greater than hot_window_ms; expected - // to be much greater (hours vs. days or weeks). - // None disables cold deletion entirely — disk grows unboundedly. - pub delete_older_than_ms: Option, - - // ---- Misc ---- - pub flush_interval_ms: u64, // cadence of the background flusher - pub disk_path: PathBuf, // root dir for segments + manifest -} -``` - -`SketchStorePerKey::new` now takes a `SketchStorePersistenceConfig` -instead of a `CleanupPolicy`. There is no "persistence disabled" escape -hatch — this is now the only cleanup mechanism this store has. If someone -wants the old in-memory-only behavior, they can set `hot_window_ms = None` -and `memory_limit_bytes = usize::MAX`, which degenerates to "never flush." - -### Eviction order - -**Round-robin across `agg_id`, oldest-first within each agg.** Every tick, -the flusher walks agg-ids in order, pops the oldest sealed epoch from each, -and repeats until the stopping condition (memory low-water or end of time -threshold) is met. Both triggers share the same walk order so the flusher -never has two disagreeing notions of "what to flush next." - -Rationale: - -- **Lock-spread under burst.** A strict global oldest-first ordering would - flush many epochs from the *same* hot agg-id back-to-back, hammering the - same per-agg `RwLock` repeatedly and creating brief query-latency spikes - on that one agg. Round-robin spreads the flusher's lock acquisitions - across different `RwLock`s, which is cheap for DashMap (lock-free outer) - and gives query latency a smoother profile. -- **Same total work, no complexity cost.** Round-robin does not evaluate - more epochs than strict-global would; it just reorders which epoch is - flushed next. Implementation cost is one `BTreeMap<(agg_id, end_ms), - EpochRef>` populated by walking `store` once per tick, or equivalently a - per-agg min-heap of sealed epochs with a round-robin cursor. -- **Freshness.** Small, slow-moving aggs are never starved by a burst on - a hot agg — they always get a turn in each round. -- **Still matches the time-window access pattern.** Within each agg, - oldest-first is preserved, so queries against recent windows on any agg - remain unaffected. - -### Background flusher - -A dedicated `std::thread` owned by the store, started in -`SketchStorePerKey::new`. Each tick, it checks the primary trigger -(memory) first, the secondary trigger (time watermark), then the -disk-retention sweep: - -``` -loop { - thread::sleep(flush_interval_ms); - if shutdown.load() { break; } - - let now = now_ms(); - let mut candidates: Vec = Vec::new(); - - // Phase 1 (PRIMARY): memory budget. - // Walk agg-ids round-robin, pulling the oldest sealed epoch from - // each on every pass, until projected memory drops below the - // low-water mark. This is the knob that actually bounds RAM. - if mem_bytes_in_use.load() > cfg.memory_limit_bytes { - candidates.extend(collect_round_robin_until_under_low_water( - cfg.memory_low_watermark_bytes, - )); - } - - // Phase 2 (SECONDARY): time watermark T. - // Any sealed epoch older than `now - hot_window_ms` that wasn't - // already picked up in phase 1 is flushed here. Also walked - // round-robin across aggs so a burst on one hot agg does not - // monopolize the tick. - if let Some(hot_window) = cfg.hot_window_ms { - candidates.extend(collect_older_than_round_robin( - now - hot_window, - )); - } - - // Dedup (phase 1 and phase 2 can pick the same epoch). Order is - // already interleaved across aggs; no re-sort. - candidates.dedup_by_key(|c| (c.agg_id, c.epoch_id)); - if candidates.is_empty() { /* go straight to phase 3 below */ } - - // ---- Build and persist one part for the whole tick ---- - // - // All candidate epochs from this tick land in a single part directory - // under `parts//`. File count per tick is O(1) (three - // files) instead of O(num_candidates). This is where group-commit - // stops being something the flusher explicitly arranges and starts - // being an intrinsic property of the layout. - if !candidates.is_empty() { - let part_id = next_part_id(); - let part_dir = cfg.disk_path.join("parts").join(fmt_part_id(part_id)); - - // (a) Clone each candidate's Arc under a brief read lock. - // No serialization under any lock. - let snapshots: Vec = candidates - .iter() - .map(|c| snapshot_under_read_lock(c)) - .collect(); - - // (b) Serialize to the three part files. data.bin is fallocate'd - // to `sum(approx_memory_bytes) * 1.3`; index.bin is sized - // exactly (32 B per entry). Both are 8-byte aligned and CRCs - // are computed streaming. - let (data_len, index_len) = write_part_files(&part_dir, &snapshots)?; - - // (c) Batched fdatasync of the three files + the part directory. - // One syscall per file; no per-epoch fsync. - fdatasync_file(&part_dir.join("data.bin"))?; - fdatasync_file(&part_dir.join("index.bin"))?; - fdatasync_file(&part_dir.join("meta.bin"))?; - fsync_dir(&part_dir)?; - - // (d) Append the add-part record to the manifest log and fsync it. - // Single fixed-size append — no rewrite of existing state. - manifest.append_add_part(AddPartRecord { - part_id, - min_ts: snapshots.iter().map(|s| s.min_ts).min().unwrap(), - max_ts: snapshots.iter().map(|s| s.max_ts).max().unwrap(), - size_bytes: (data_len + index_len) as u64, - })?; - fsync_dir(&cfg.disk_path)?; // log's size update is now durable - - // (e) Now that the part is officially live, evict the source - // epochs from memory. This is the only place we take the - // per-agg write lock, and we take it O(1) times per epoch. - for snapshot in &snapshots { - splice_out_of_sealed_epochs(snapshot.agg_id, snapshot.epoch_id); - mem_bytes_in_use.fetch_sub(snapshot.approx_bytes); - } - - // Maybe compact the manifest log into a fresh snapshot if the - // log has grown large relative to the current snapshot. - manifest.maybe_compact()?; - } - - // Phase 3 (disk retention sweep): delete whole parts older than T2. - // - // Because parts cover a tight time range (~flush_interval_ms), T2 - // deletion operates at part-directory granularity — we rm -rf the - // whole thing rather than touching shared files. - if let Some(ttl) = cfg.delete_older_than_ms { - let cutoff = now.saturating_sub(ttl); - let expired: Vec = manifest - .live_parts() - .filter(|p| p.max_ts < cutoff) - .map(|p| p.part_id) - .collect(); - - for part_id in expired { - // Invalidate Tier-2 cache entries that reference this part, - // append a delete-part record to the log, then remove the dir. - cache_tier2.invalidate_part(part_id); - manifest.append_delete_part(part_id)?; - let part_dir = cfg.disk_path - .join("parts") - .join(fmt_part_id(part_id)); - fs::remove_dir_all(part_dir).ok(); // orphan sweep on restart catches failures - } - if !expired.is_empty() { - fdatasync_file(&manifest.log_path())?; - fsync_dir(&cfg.disk_path)?; - } - } -} -``` - -Under memory pressure, phase 1 dominates and phase 2 usually finds nothing -left to do (the oldest epochs are already gone). Under light ingest, phase 1 -is a no-op and phase 2 does all the work. Phase 3 is independent and runs -every tick regardless; it costs one manifest scan plus one `rm -rf` per -expired part directory (usually zero). - -Group-commit is now **intrinsic to the layout**, not something the flusher -has to explicitly arrange: one tick = one part = three `fdatasync`s + one -dir fsync + one log append, independent of how many epochs the tick is -flushing. The durability ordering is spelled out in the previous section -("Durability ordering per flush tick"). - -The part-building loop above takes advantage of a property that matters a -lot for the flusher design: **sealed epochs are append-only and frozen.** -Once the rotator seals an epoch, no writer will ever touch its contents -again — it is only read (by queries) or removed wholesale (by the -flusher). That immutability is what lets the flusher stay completely off -the critical path: - -1. Take the per-agg `RwLock::read` briefly, clone the `Arc` for - each candidate out of `sealed_epochs`, drop the lock. This is the - `snapshot_under_read_lock` step. -2. Serialize all candidates into `data.bin` / `index.bin` / `meta.bin`, - fsync the three files and the part directory, and append to the - manifest log — **entirely outside any per-agg store lock**, on the - flusher's own thread. Nothing in the system is waiting on this I/O. - Inserts continue to land in `current_epoch`; queries continue to - read from the still-in-place `sealed_epochs` entries (and the cloned - `Arc`s keep bytes alive for any query that happens to hold a - reference already); the rotator continues to seal new epochs behind - us. -3. Once the part is officially live in the manifest log, take each - per-agg `RwLock::write` briefly to splice the corresponding epoch - out of `sealed_epochs` and decrement `mem_bytes_in_use`. This is - O(1) per epoch — a `BTreeMap::remove` plus an atomic subtraction — - and is the only write lock the flusher holds per epoch. - -Because steps 2 and 3 are decoupled by the `Arc` clone from step -1, **no per-agg lock is ever held across disk I/O**, and the flusher -never blocks anything on the insert or query path beyond the two brief -lock acquisitions at the start and end. - -#### Sync vs. async flush I/O — resolved - -The previous revision left this as an open question. With the append-only -property made explicit, the answer is clear: **plain `std::fs` on a -dedicated `std::thread` is what we ship.** No `tokio::fs`, no Tokio -runtime for the flusher. - -The only argument for async I/O would be "we need to yield the thread -while `fsync` is in flight so some other task on the same runtime can -make progress" — and there is no such other task. The flusher thread has -exactly one job — flush — and blocking it on `write` + `fsync` is fine -because: - -- **Inserts never wait on the flusher.** Inserts land in `current_epoch` - with no coordination with flush state; memory accounting is an atomic, - not a lock. The flusher and the insert path only share the per-agg - `RwLock`, and the flusher only holds it during the two brief windows - above. -- **Queries never wait on the flusher.** In-memory reads take the per-agg - `RwLock::read`, which contends with the flusher only during those same - brief windows; disk reads go through the manifest lock, which is - independent. -- **Back-pressure is the right answer to a slow disk.** If the flusher - genuinely cannot keep up and memory hits `hard_cap_bytes`, the insert - path blocks on a condvar until the flusher catches up. That is the - correct behavior regardless of whether the I/O underneath is sync or - async — making it async would not let more inserts through, it would - just change which thread was parked. - -Sync I/O keeps the store out of Tokio's executor entirely, keeps stack -traces readable, and eliminates a class of "why is my future not making -progress" failure modes. The flusher thread is `std::thread::spawn`'d in -`SketchStorePerKey::new` and joined in `close`, with a `shutdown` -flag checked on each loop iteration. - -### Query path - -`query_precomputed_output` becomes a three-way merge: - -1. Read from `current_epoch + sealed_epochs` as today (in-memory hits). -2. Walk the parts_manifest's live-parts list for any `part.[min_ts, - max_ts]` that overlaps the query's time range. The manifest lives in - memory as a `Vec` built from the snapshot + log replay at - startup, so this is a linear scan over a short list (tens of - thousands of entries in the worst case, all 32-byte records) — fast - and trivially parallel with inserts. -3. For each overlapping part: fetch its `DecodedPart` from the Tier-2 - segment cache (`moka::Cache>`). On miss, - `mmap` the part's `data.bin` and `index.bin`, wrap them in an - `Arc` (holding the mmap handles), and insert into the - cache. Then binary-search `index.bin` for `(agg_id, start_ms)`, walk - the matching entries forward while `start_ms <= query_end`, and for - each one hand the `&[u8]` slice of `data.bin` directly to - `AggregateCore::deserialize_from_bytes` with zero copies. - -Part reads happen under a read lock on the parts_manifest vector; they -do **not** take any per-agg store lock, so they run fully in parallel -with inserts. Merging reuses the existing `TimestampedBucketsMap` + -`AggregateCore::merge_with` that the in-memory query path already uses -— no new merge logic. - -**Why this is fast:** - -- **One file open per part hit, not per entry.** Queries that span many - aggs inside a part still only pay one `mmap`'s worth of setup cost. -- **Zero-copy deserialize.** The 8-byte alignment guarantee means - `&data.bin[offset..offset+len]` can be fed straight to the - sketch-specific decoder without a staging buffer. -- **Binary search, not linear scan.** `index.bin` is sorted by - `(agg_id, start_ms)` and is a flat mmap'd array; `partition_point` is - a few cache lines of work. -- **Page cache locality.** Adjacent entries for the same agg inside a - part are physically adjacent on disk, so a query over a time range - touches contiguous pages. - -### Read-side segment cache (two-tier memory model) - -So far the flusher treats "is this sketch in RAM?" as a pure function of -*write* state — time of ingest and write-side memory pressure. That is the -right default for a TSDB, because recency dominates query patterns, but it -leaves one real gap: **cold-but-repeatedly-queried** segments. Think of a -dashboard that scans "last Tuesday's incident" every time the on-call opens -it, or a recording rule that re-reads a fixed 24h historical range every -minute. Those queries touch segments that the time watermark has correctly -decided are cold, and under the design so far they pay full disk I/O on -every hit. - -The answer is **not** to let query frequency feed back into the flusher -policy. Doing that would couple write-path retention to read load, break -the monotonic "once cold, stays cold" invariant the flusher relies on, and -introduce unbounded-memory failure modes when a query sweeps everything. -The answer is a **second, separate memory tier** that exists purely as a -read-side cache on top of the disk layer. - -**Tier 1 — authoritative hot (write-driven).** Bounded by -`memory_limit_bytes` + `hot_window_ms`. Contains `current_epoch` and any -sealed epoch that has not yet been flushed. Source of truth for recent -data. Managed by the flusher described above. - -**Tier 2 — read-side part cache (query-driven).** Bounded by a separate -`part_cache_bytes` budget. Contains mmap handles and decoded index views -of parts pulled back from disk by the query path. Source of truth is -always the part directory on disk — the cache is a pure optimization, -drop-anytime, never dirty. Managed by the query path, not the flusher. - -```rust -pub struct SketchStorePersistenceConfig { - // ... existing fields ... - - // Read-side part cache. Bounded independently of - // `memory_limit_bytes`; this budget is for decoded parts the query - // path pulls back from disk, not for the authoritative hot set. - // - // Default: min(10% * memory_limit_bytes, 512 MiB). - // - // A fresh install should not need to know about this knob to get - // reasonable repeat-query performance. Setting to 0 disables Tier 2 - // entirely (every cold query pays disk I/O); a fixed absolute - // default would be too small on big boxes and too large on small - // ones, so the default scales with the write budget. - pub part_cache_bytes: usize, -} -``` - -**Why a TSDB specifically benefits from this shape:** - -1. **Recency and query frequency overlap ~90%.** Tier 1 already catches - everything a "rate over last 5m" workload wants pinned. The cache - only earns its budget on the residual workload — dashboards on fixed - old ranges, recording rules over long horizons. Making it a separate, - sized-independently tier means we can ship a small default (or zero) - and only budget it up for workloads that measurably need it. - -2. **Monotonic tiering.** Once a sealed epoch is flushed, it stays on - disk. A query may cache a decoded copy in Tier 2, but the flusher - never "un-flushes" it back into Tier 1. This preserves the property - that Tier 1 is purely a function of write state — which is what makes - the flusher simple enough to implement correctly. - -3. **Segment granularity, not sketch granularity.** The unit of disk I/O - is the segment file, so the cache must match that granularity. - Caching individual sketches inside a segment would mean partial reads - and complex invalidation; caching whole segments is a trivial - `Cache>` keyed on manifest metadata. - -4. **Two independent budgets are easier to tune than one unified priority - score.** Operators reason about "how much RAM does write buffering - need?" and "how much RAM does read caching need?" separately. A - unified `priority = α * recency + β * frequency` score is harder to - explain and harder to debug when it misbehaves. - -**Eviction policy for Tier 2: W-TinyLFU via `moka` (or `mini-moka`).** -Plain LRU is the obvious choice but is catastrophically scan-vulnerable — -a single long-range query sweeps the cache and evicts everything genuinely -hot, which is exactly the access pattern TSDB dashboards and recording -rules produce (hour/day/week range scans). W-TinyLFU's admission filter -rejects scan traffic from displacing hot entries and typically delivers -10–30% better hit rate than LRU at the same byte budget on skewed / -Zipfian workloads. - -The `moka` crate is the standard Rust implementation (sync and async -variants, weight-based eviction keyed on byte size, well-maintained, used -widely in the Rust ecosystem). The API is effectively a drop-in for LRU -(`get`, `insert`, `invalidate`), so we incur no additional complexity vs. -a hand-rolled LRU — just better hit rate. W-TinyLFU's per-access overhead -is a handful of CAS ops on a small count-min sketch, cheaper than LRU's -mutex-protected list reordering. - -The cache exposes hit/miss counters in `StoreDiagnostics` from day one so -we have signal for future tuning. - -**Interaction with the flusher.** None. The flusher only sees Tier 1. -The read cache has no feedback into retention decisions. This is the -whole point of splitting the tiers. - -**What this deliberately does not do:** - -- No pinning of individual sketches in Tier 1 based on read counts. The - existing `read_counts` field on `StoreKeyData` becomes purely diagnostic - for this store — it does not veto flushes. -- No promotion from Tier 2 back into Tier 1. Once cold, stays cold. -- No partial-segment loading. Segments are cached whole or not at all. - -### Recovery on startup - -1. Open `disk_path`. If `parts_manifest.snapshot` exists, mmap it and - cast the bytes to `&[PartEntry]` (no parse — the layout is 8-byte - aligned and versioned in a small header). Otherwise, start with an - empty live set. -2. Replay `parts_manifest.log` from the offset recorded in the - snapshot's footer, applying add-part and delete-part records to the - live set. -3. For every live part, stat its directory, verify `meta.bin`'s magic, - version, and CRC. Parts whose directory is missing or whose - `meta.bin` fails verification are logged and removed from the live - set (and a delete-part record is appended to the log to make the - removal durable). -4. Sweep `parts/` for directories not referenced in the live set - (orphans from a mid-flush crash before step 5 of the durability - ordering) and `rm -rf` them. -5. Build the in-memory `Vec` that the query path reads. Do - **not** mmap `data.bin` or `index.bin` eagerly — parts are mapped - lazily on first query hit and cached in Tier 2. Cold data stays - cold until a query asks for it. - -### Concurrency summary - -| Path | Lock taken | -|-------------------|---------------------------------------------| -| Insert | per-agg `RwLock::write` (unchanged) | -| Query in-memory | per-agg `RwLock::read` (unchanged) | -| Query disk | parts_manifest `RwLock::read` + moka cache internal | -| Flush: snapshot | per-agg `RwLock::read` (brief, per candidate) | -| Flush: serialize | none (operates on cloned `Arc`s) | -| Flush: commit log | parts_manifest `RwLock::write` (brief, append) | -| Flush: evict | per-agg `RwLock::write` (short, O(1) splice per epoch) | -| T2 sweep | parts_manifest `RwLock::write` (brief, delete records) | - -No lock of any kind is held across a `fsync` or disk I/O. - ---- - -## What this does **and does not** change - -Changes: - -- `SketchStorePerKey::new` no longer takes a `CleanupPolicy`; it takes a - `SketchStorePersistenceConfig`. The three destructive cleanup variants - (`CircularBuffer`, `ReadBased`, `NoCleanup`) are no longer wired into this - store at all. The code paths in `per_key.rs` that branch on - `CleanupPolicy` (`cleanup_old_aggregates`, `maybe_rotate_epoch`'s retention - logic) are deleted in favor of the flusher. -- Call sites that construct `SketchStore::new_with_strategy(..., cleanup_policy, ...)` - update to pass a `SketchStorePersistenceConfig` instead. Main.rs and any - tests that construct the store directly will need to change. - -Does not change: - -- The `CleanupPolicy` enum itself stays in `asap_types` — other stores - (`promsketch_store`, legacy paths) may still reference it. This PR only - severs `SketchStore`'s dependency on it. -- `SketchStoreGlobal` is intentionally left in-memory-only. Persistence - targets `PerKey`, which is the production path. Adding it to `Global` is a - small follow-up if anyone needs it. -- Query planning, the control plane client, and the precompute engine's output - sink are untouched. The store's `Store` trait signature does not change. - ---- - -## Phasing - -The PR this design doc accompanies will land in three commits on one branch so -the pieces can be reviewed independently: - -1. **Sizing + config plumbing + cleanup-policy removal.** Add - `approx_memory_bytes` to `AggregateCore` and all concrete accumulators. - Add `SketchStorePersistenceConfig`. Track `mem_bytes_in_use`. Rip the - `CleanupPolicy` branches out of `per_key.rs` and update call sites. No - disk I/O yet — expose the memory counter in `StoreDiagnostics` so we can - validate sizing in isolation and be confident nothing else regressed. - -2. **Parts layout, manifest log, flusher.** Add `persistence/` submodule - under `simple_map_store/` with part encode/decode (`meta.bin` + - `data.bin` + `index.bin`, 8-byte aligned, `fallocate`'d, mmap-friendly), - parts_manifest log + snapshot read/write, background flusher thread - (memory-first, then time-watermark, then T2 retention sweep, one part - per tick, group-commit implicit in the layout). Wire part-building and - eviction into the per-key store. Unit tests for round-trip, - crash-before-log-append, crash-between-log-append-and-dir-fsync, - orphan-part sweep, and T2 whole-part deletion. - -3. **Query path read-through + recovery + Tier-2 part cache.** Extend - `query_precomputed_output` to walk the parts_manifest, binary-search - `index.bin` of overlapping parts, and zero-copy deserialize payloads - out of mmap'd `data.bin`. Wire a `moka` (or `mini-moka`) - weight-bounded part cache sized to - `min(10% * memory_limit_bytes, 512 MiB)` by default, keyed on - `PartId`, with hit/miss counters exported via `StoreDiagnostics`. - Add startup recovery (snapshot mmap + log replay + CRC verify + - orphan sweep). Integration test: ingest → flush → restart → query → - same result as no-restart, plus a scan-resistance test that confirms - a long-range query does not evict a separately-hot part. - -Because phase 1 removes `CleanupPolicy` from this store, phase 1 is **not** -independently mergeable without at least the memory-pressure path from -phase 2 — otherwise the store has no bound on RAM. In practice phases 1 and -2 land together; phase 3 can land separately once the write path is stable. - ---- - -## Resolved decisions - -Every question previously flagged as open has been resolved in favor of the -performance-optimal choice. The table below is a summary; the reasoning for -each lives in the section it points to. - -| # | Question | Decision | Why | -|---|---|---|---| -| 1 | Segment file format | Custom binary, 8-byte aligned, mmap-friendly, `fallocate`d | Zero-copy deserialize into `AggregateCore`; no Parquet/Arrow overhead for data we never column-prune; smaller binary size and compile time. See **Disk layout**. | -| 2 | Sync vs. async flush I/O | Sync `std::fs` on a dedicated `std::thread`, with **group-commit fsync** batching across a tick | `tokio::fs` just routes to a blocking threadpool on Linux, so async is a wash at the syscall level; the real win is batching `fdatasync`. No task on the flusher's runtime is waiting on it to yield. See **Background flusher / Group-commit fsync**. | -| 3 | Cold data retention on disk | Add `delete_older_than_ms = T2`, run as phase 3 of the flusher tick | Unbounded segment count bloats the manifest (falls out of L2/L3), slows startup directory sweeps, and pressures Tier-2 eviction. Cheap to add now, painful to retrofit once a deployment has millions of orphans. See **Configuration** and **Background flusher phase 3**. | -| 4 | Per-agg-id flush fairness | **Round-robin across agg-ids, oldest-first within each agg** | Strict-global-oldest hammers one `RwLock` during a hot-agg burst and creates query-latency spikes on that one agg. Round-robin spreads lock acquisitions across different `RwLock`s for the same total work. See **Eviction order**. | -| 5 | Default `segment_cache_bytes` | `min(10% * memory_limit_bytes, 512 MiB)` | A fixed 64 MiB default is too small on big boxes and too large on small ones; scaling with the write budget keeps the tier sensibly sized without requiring operator tuning on a fresh install. See **Configuration**. | -| 6 | Tier-2 algorithm | **W-TinyLFU via `moka`** from day one, not plain LRU | LRU is scan-vulnerable — one long-range query evicts everything genuinely hot, which is exactly the TSDB dashboard access pattern. W-TinyLFU's admission filter rejects scan traffic and typically delivers 10–30% better hit rate at the same byte budget on skewed workloads, with a drop-in API and lower per-access CPU than LRU. See **Read-side segment cache**. | -| 7 | Disk layout | **Parts** (one directory per flush tick containing `meta.bin` + `data.bin` + `index.bin`) with an **append-only `parts_manifest.log` + periodic binary snapshot**, not one file per sealed epoch with a JSON manifest | One-file-per-epoch produces hundreds of thousands of tiny files on a real deployment (inode pressure, readdir slowdown, per-file fsync floor). A JSON manifest rewritten each tick is quadratic over the deployment's lifetime. Parts bound file count to O(flush ticks), bundle all of a tick's epochs behind one fallocate + three fdatasyncs, push per-part indexing into a binary-searchable `index.bin`, and reduce the global manifest to an append-only log whose size is proportional to ticks, not epochs. This is the layout every mainstream TSDB converges on. See **Unit of flush vs. unit of file: the *part*** and **Disk layout**. | - -If any of these decisions turn out to be wrong under real traces, the -affected sections are the natural point of revisiting — but none of them -are "temporary v1 shortcuts we'll upgrade later." This is the target -design. - -**Not resolved here, deliberately punted to v2:** background compaction -of adjacent small parts into larger ones. The parts layout accommodates -compaction cleanly (it's a pure directory-level merge with the same -on-disk shape as a regular flush tick), but v1 ships without it. With -T2 retention in place and a reasonable flush interval (seconds, not -milliseconds), the number of live parts in v1 stays bounded at -`T2 / flush_interval_ms` — a few tens of thousands at most, well within -what a binary-searched `Vec` handles with room to spare. -Compaction becomes necessary only if we lower the flush interval -significantly or extend T2 to very long horizons. diff --git a/docs/design-sketch-db-core.md b/docs/design-sketch-db-core.md deleted file mode 100644 index 99becb5d..00000000 --- a/docs/design-sketch-db-core.md +++ /dev/null @@ -1,1377 +0,0 @@ -# Sketch DB — Core Design (As-Built Contract) - -> **Scope.** Sections that describe the architecture and data contract the -> code today is built against. Subsystems that are still aspirational -> (compaction, admission control, full observability, rollout plan, -> performance envelope) live in the companion docs. -> -> **Status of sections in this file:** mostly ✅ implemented, with -> per-section notes where a design element is partial. See the -> [index](./design-sketch-db.md) for the complete status table across all -> sketch-DB docs. -> -> **Companion docs** -> - [`design-sketch-db.md`](./design-sketch-db.md) — index + status table + TL;DR -> - [`design-sketch-db-roadmap.md`](./design-sketch-db-roadmap.md) — §9, §11, §12, §13, §16, §17, §18 (not-yet-implemented) -> - [`design-sketch-db-performance.md`](./design-sketch-db-performance.md) — §19 (performance envelope) + §20 (sketch profiler) -> - [`design-sketch-db-pluggable.md`](./design-sketch-db-pluggable.md) — extracting the sketch DB into its own library / binary -> - [`design-simple-map-store-persistence.md`](./design-simple-map-store-persistence.md) — the LSM persistence layer this builds on -> - [`adding-a-new-sketch.md`](./adding-a-new-sketch.md) — cross-repo recipe for new sketch types - -**Audience:** anyone reading the code, writing PRs, or reviewing behavior -changes that touch the sketch DB today. - ---- - -## 1. Motivation - -`SketchStore` today is a KV store bucketed by `aggregation_id`. That was -enough for "write sketches, read them back by id." It is no longer enough once: - -- The control plane reconfigures sketches at runtime in response to a changing - query workload (parameter upgrades, new query patterns, retirement). -- Queries should transparently work across those reconfigurations, with no - data cliff at the upgrade boundary. -- The store needs to give the control plane back real workload observations - (bytes stored, query latency, merge cost) to drive cost-based planning. -- Persistent storage means there is no implicit cleanup mechanism — any data - written to disk stays there until something explicitly retires it. - -What's needed is a storage engine that treats sketches as first-class typed -values with per-type merge semantics, tracks schema per aggregation across -time, and supports refreshing itself from the exact-DB base relation when -the schema changes. In database terms: the sketches are **materialized -views** over raw sample data; the store is a materialized-view engine -specialized for sketch types. - -The **exact DB** (the base relation the MVs are materialized over) may be -implemented by any long-term raw-sample store: S3 with Gorilla compression, -Prometheus / VictoriaMetrics via remote-write, ClickHouse, or any other -backend the deployment picks. The design treats it as a pluggable reader -behind a single interface — the sketch DB doesn't care which backend is -actually holding the raw bytes. - ---- - -## 2. Sketches as materialized views - -This section makes the materialized-view framing explicit, because every -later section in this doc is easier to reason about if you have the -classical database analogy in mind. - -### 2.1 The view definition - -An `AggregationConfig` is a materialized-view *definition*. Expanded: - -```sql --- Pseudocode for an AggregationConfig that computes a CMS over a metric -CREATE MATERIALIZED VIEW agg_17 AS - SELECT - time_bucket(INTERVAL '60 s', ts) AS window_start, - host AS grouping_key, - cms_build(value, rows => 3, cols => 1024) AS sketch, - count(value) AS count, - sum(value) AS sum, - min(value) AS min, - max(value) AS max - FROM raw_samples - WHERE metric = 'latency' - GROUP BY 1, 2; -``` - -Read off the `AggregationConfig` fields: - -| Config field | SQL analog | -|---|---| -| `metric` | `WHERE metric = …` in the view definition | -| `aggregation_type` + `parameters` | the aggregation function (`cms_build(…)`) | -| `grouping_labels` | `GROUP BY` columns | -| `aggregated_labels` | the inner keyed dimension of multi-subpopulation aggregators | -| `window_size` + `slide_interval` | `time_bucket(…)` + windowing | -| `spatial_filter` | extra `WHERE` clause conditions | - -The base relation being materialized over is the stream of raw samples. -Incremental maintenance (§8) reads them from the live agent stream as -they arrive. Refresh (§10) reads them from the exact DB (whichever -backend is configured — S3/Gorilla, Prometheus, ClickHouse, etc.). - -### 2.2 What a materialized view framing buys us - -| Classical MV concept | Sketch DB counterpart | -|---|---| -| View definition | `AggregationConfig` | -| Base relation | Raw sample stream; exact DB for historical replay | -| Materialization | `SketchEntry` rows across Tier 1 / Tier 2 storage (§4.1) | -| View maintenance strategy | §8 (incremental) + §10 (refresh) | -| Schema evolution | §6 per-`agg_id` lifecycle, §7 schema timeline | -| `DROP MATERIALIZED VIEW` | Retirement + expiry | -| `REFRESH MATERIALIZED VIEW` | Backfill job from exact DB | -| `pg_matviews` / metadata catalog | `/api/v1/db/schemas`, §15 control plane APIs | -| Index on the MV for faster queries | Label postings index (§5.3 — roadmap), typed aux columns (§5.1 — partially implemented) | - -The rest of this doc splits into two halves, mirroring the two standard -MV maintenance strategies: - -- **§8. Incremental view maintenance** — the live ingest path, which - updates the MV as new samples arrive. This is what the precompute - engine already does today; the doc describes how it extends cleanly - into a sketch-DB-aware design. -- **§10. Refreshable view maintenance** — the backfill service, which - rebuilds the MV for a time range from the base relation (the exact DB). - This is the piece that makes schema evolution painless. - -Both strategies write to the same physical store and the same per-`agg_id` -namespace. The control plane chooses between them based on workload and -reconfigure cost (§10.6). - -### 2.3 Why not just one strategy? - -- **Incremental-only** is the status quo. It's cheap (per-sample update - cost) but it has no way to handle reconfigure: once the view - definition changes, old data in the MV is under the old schema and - new data is under the new schema, with a discontinuity at the swap - point. Sketches of different types or parameters can't be merged. -- **Refresh-only** would work but is wasteful. Rebuilding the MV from - raw samples at every ingest would throw away the per-sample - incremental cost advantage that's the whole point of having - agent-side sketching. - -Both together: incremental handles the steady state (99% of the time) -cheaply; refresh handles the rare reconfigure moment. This matches how -real warehouses deploy MVs — you never choose one maintenance strategy -forever, you choose per-view and per-situation. - ---- - -## 3. Design principles - -### 3.1 Principles - -1. **Per-`aggregation_id` namespace isolation.** Each `agg_id` is its own - logical namespace with a pinned schema (sketch type, parameters, - grouping labels, window size). Writes are validated against that - schema. Compaction never crosses `agg_id` boundaries. -2. **Schema timeline is first-class.** The store knows, for any metric, - which `aggregation_id` served it during which time range. Query - engine queries by metric; store handles the per-`agg_id` dispatch. -3. **Semantic compaction.** LSM levels represent decreasing temporal - resolution. Compaction calls the sketch type's merge operator; level - N+1 holds entries that are the merge of multiple level-N entries at - a coarser window. *(Not yet implemented — see roadmap §9.)* -4. **Typed auxiliary columns.** `count`, `sum`, `min`, `max` are stored - alongside the sketch bytes, not inside them. Queries that only need - these scalar stats never pay the sketch deserialization cost. - *(Partial — accumulators carry these; they are not yet first-class - columns on the stored `PrecomputedOutput`. See §5.1.)* -5. **Exact DB is the source of truth.** Sketches are rebuildable from - the exact-DB base relation (whichever backend is configured — S3 + - Gorilla, Prometheus, ClickHouse, VictoriaMetrics, etc.). The store - supports explicit backfill from the exact DB into any `agg_id` for - any time range. The exact DB is the same storage that also serves - the query-engine fallback path when a query cannot be answered from - sketches. -6. **Monotonic, non-reused `aggregation_id`s.** A reconfigure that - changes parameters is always "retire old id + create new id", never - "mutate in place." This is a contract on the control plane; the store - enforces it via a write-side schema barrier (§6.3). - ---- - -## 4. Architecture - -``` - ┌─────────────────────────────────────────────────┐ - │ Control plane │ - │ - owns StreamingConfig (the set of agg_ids) │ - │ - monotonic id allocator, never reuses │ - │ - triggers backfill on reconfigure │ - │ - reads workload metadata for cost model │ - └──────────┬─────────────────────┬────────────────┘ - │ StreamingConfig swap│ /api/v1/db/* - ▼ ▼ - ┌─────────────────────────────────────────────────┐ - │ Sketch DB │ - │ │ - │ ┌────────────────────────────────────────┐ │ - │ │ Schema Timeline │ │ - │ │ per metric, ordered list of agg_ids │ │ - │ │ with (created_at, retired_at, expiry) │ │ - │ └────────────────────────────────────────┘ │ - │ │ - │ ┌────────────────────────────────────────┐ │ - │ │ Per-agg_id storage (multi-tier) │ │ - │ │ Tier 1: PromSketch (in-mem EH) │ │ - │ │ Tier 2: precompute + LSM parts │ │ - │ │ pinned schema (type, params, group) │ │ - │ │ semantic compaction within a tier │ │ - │ │ label posting index, typed aux cols │ │ - │ └────────────────────────────────────────┘ │ - │ │ - │ ┌────────────────────────────────────────┐ │ - │ │ Backfill / REFRESH service │ │ - │ │ reads raw from exact DB (pluggable: │ │ - │ │ S3+Gorilla / Prometheus / CH / VM) │ │ - │ │ rebuilds sketches per (agg_id, window)│ │ - │ │ writes to per-agg_id tier storage │ │ - │ └────────────────────────────────────────┘ │ - │ │ - │ ┌────────────────────────────────────────┐ │ - │ │ Query Pushdown │ │ - │ │ timeline_for_metric(metric, range) │ │ - │ │ per-segment dispatch to agg_ids │ │ - │ │ merge + compute statistic in-store │ │ - │ └────────────────────────────────────────┘ │ - │ │ - └──────────────┬──────────────────────────────────┘ - ▲ - │ query_metric / point / range_merge - │ - ┌─────────────────────────────────────────────────┐ - │ Query Engine (ASAPQueryEngine) │ - │ - parses PromQL/SQL │ - │ - dispatches by metric, not agg_id │ - │ - assembles results from DB's per-segment │ - │ outputs │ - └─────────────────────────────────────────────────┘ -``` - -### 4.1 Storage tiers - -The sketch DB is one logical entity with multiple physical storage -tiers. PromSketch and the precompute + LSM store are **not separate -systems** — they are tiers of the same sketch DB. Which tier an -`agg_id` lives on is part of its `AggSchema`, configurable by the -control plane. - -| Tier | Implementation | Compaction mechanism | Retention | Query latency | Use case | -|---|---|---|---|---|---| -| **Tier 1** | PromSketch — in-memory EH-backed sketches over raw samples | **Continuous temporal compaction via the EH bucket structure itself** — fine-grained buckets for recent data, exponentially coarser buckets for older data, merged in place as windows age | seconds to minutes (bounded by memory) | sub-millisecond | live dashboards, alerts; **especially good when many sub-window queries target the same series** — e.g. a dashboard that asks for p50/p95/p99 over 1m, 5m, 15m, 1h all against the same metric. Tier 1 serves all of them from one EH structure with no redundant storage; Tier 2 would store one pre-merged sketch per (agg_id, window) and either duplicate the series across multiple agg_ids or rely on read-time merge. | -| **Tier 2** | Precompute engine + `SketchStore` LSM parts (memory + disk) | **Batched semantic compaction** via LSM levels — roadmap §9 — N entries at level L merged into one at level L+1 at a coarser window | minutes to weeks (bounded by `persistence_delete_older_than_secs`) | milliseconds | most production queries, longer-horizon analysis | -| **Exact DB** | S3 + Gorilla / Prometheus / VictoriaMetrics / ClickHouse | Storage-native compression (Gorilla / columnar); no sketch-level merging | months+ (configurable, bounded by raw storage cost) | seconds to tens of seconds | fallback for uncovered queries, base relation for refresh | - -The two sketch tiers implement the **same abstract concept** — "reduce -temporal resolution over time while preserving sketch-accuracy -bounds" — but at different points along a latency/complexity curve. -Tier 1 does it continuously in memory via EH bucket merges as windows -age; Tier 2 does it in background compaction with explicit LSM levels. -The control plane picks per-`agg_id` which mechanism matches the -workload: sub-ms live queries with short retention → Tier 1; weeks of -queryable history → Tier 2; both → both. - -> **Status note.** Tier 1 (PromSketch) is currently dormant in the codebase; -> most of its integration points are commented out. Tier 2 -> (`SketchStore` + LSM persistence) is the active path. Tier -> selection fields on `AggregationConfig` are partially plumbed but not -> exercised end-to-end. - -**Tier selection per `agg_id`.** A metric's `AggregationConfig` carries -a `tier` field: `Tier1Only`, `Tier2Only`, or `Both`. `Both` means -incremental maintenance writes to both tiers — Tier 1 for freshness, -Tier 2 for long retention. Query engine picks per-query based on the -query's time range and SLA. - -**Tier promotion / demotion on reconfigure.** The control plane can -upgrade an `agg_id` from Tier 1 to Tier 2 as the workload justifies -the long retention cost. Schema-wise this is a regular reconfigure -(retire old id, create new id with new tier), and the Tier 2 backfill -path (§10) reads from the exact DB to populate history. - -**The exact DB is not a tier.** It's the base relation every tier -materializes from. It's shown alongside the tiers in the table -because the query engine's fallback path also reads from it, so -operationally it looks like a third storage layer. But conceptually -it is "not sketch DB" — it holds raw samples, not sketches. - -### 4.2 Why this framing - -Before this framing: PromSketch (`stores/promsketch_store/`) and the -precompute-backed `SketchStore` looked like two independent -sketch systems that the query engine had to route between. They had -overlapping but different semantics (incremental MV in both cases, but -different schema, different retention model, different hot-reload -story). - -Unifying them as tiers of one DB means: - -- One schema lifecycle (§6) covers both tiers; a tier upgrade is a - regular reconfigure. -- One control plane API surface (§15) exposes stats across tiers. -- One refresh path (§10) can write to either tier. -- The query engine's routing logic (§7.3 per-segment dispatch) picks - tier by the same mechanism it picks `agg_id` — the schema timeline - already carries everything needed. - -The two tiers' internal storage formats differ (EH-backed arrays vs -LSM parts), and that's fine — the sketch DB abstracts over them -through a common tier-backend trait. Internally each tier keeps its -own implementation details. - ---- - -## 5. Storage schema - -### 5.1 Per-entry record - -```rust -struct SketchEntry { - // Identity - agg_id: u64, - group_key: String, // e.g. "service=auth;region=us-east" - window_start: u64, // business-time window bounds - window_end: u64, - - // Typed auxiliary columns — scalars queryable without - // deserializing sketch_bytes - count: u64, - sum: f64, - min: f64, - max: f64, - - // Sketch payload - sketch_type: SketchType, // redundant with agg_id's schema; enables - // polymorphic reads and self-describing dumps - sketch_bytes: Vec, - - // Provenance - origin: EntryOrigin, // Native | Backfilled { job_id } - ingest_ts: u64, // when this record was written -} - -enum EntryOrigin { - /// Written by the live ingest path — precompute worker flushed a - /// window close. - Native, - /// Written by the backfill service from exact-DB raw data. The - /// job_id lets the system correlate with the BackfillJob that - /// produced it (for re-runs, debugging, idempotency). - Backfilled { job_id: u64 }, -} -``` - -> **Status.** `PrecomputedOutput` in the code carries `(agg_id, window, -> key, origin)` plus the accumulator (which internally holds the -> sketch + count/sum/min/max). The typed aux columns are **not** -> materialized as first-class fields on the entry today — they live -> inside the accumulator. Promoting them is a known improvement that -> saves a deserialize on scalar queries. - -Why `count`/`sum`/`min`/`max` are not inside the sketch: - -1. `sum_over_time`, `count_over_time`, `max_over_time`, `min_over_time` - are the overwhelming majority of production queries. They should - never pay the sketch-deserialize cost. -2. These stats are already tracked losslessly by DataCollector's - processors (they live in the typed `*SketchDataPoint` proto fields - as first-class numbers). We are preserving that typing end-to-end - instead of forcing them to travel through the sketch blob. - -### 5.2 Primary key and secondary index - -Two access patterns dominate: - -| Pattern | Source | Key order | -|---|---|---| -| "all groups at time T" | batch jobs, control plane workload scans | `(agg_id, window_start, group_key)` | -| "one group over time range" | dashboard queries, `quantile_over_time(…) by (svc)` | `(agg_id, group_key, window_start)` | - -The store maintains the first as the primary index (SSTable sort order) and -the second as a secondary index. Primary order wins during compaction -(spatial locality across a single window close is the common ingest -pattern); secondary index is maintained via an auxiliary log that the -reader consults for single-group time scans. - -> **Status.** Primary order is the shape of SketchStore today -> (per-`agg_id` bucketing + per-key). A separate materialized -> secondary index is **not** implemented; single-group time scans -> iterate the primary structure. - -### 5.3 Label posting index - -For control plane workload queries like "how many distinct services does -metric `X` have?" and for label filter pushdown (`service=auth`), a -per-`agg_id` label posting index maps: - -``` -label_postings[agg_id]: HashMap<(label_name, label_value), RoaringBitmap> -``` - -Roaring bitmaps keep this compact even for high-cardinality labels. -Compared to scanning every group, a label match becomes an O(1) bitmap -lookup. - -> **Status.** Not implemented. Only label interning exists. This is a -> future performance win for high-cardinality aggs; see roadmap §16 -> "Phase 7" in [`design-sketch-db-roadmap.md`](./design-sketch-db-roadmap.md). - ---- - -## 6. Per-`agg_id` schema and its lifetime - -### 6.1 `AggSchema` — the pinned metadata - -```rust -struct AggSchema { - agg_id: u64, - metric_name: String, // which metric this agg serves - sketch_type: SketchType, // pinned for agg's lifetime - parameters: HashMap, // pinned for agg's lifetime - grouping_labels: Vec, // pinned for agg's lifetime - window_size: u64, // pinned for agg's lifetime - slide_interval: u64, // pinned for agg's lifetime - - // Lifecycle - created_at: u64, - retired_at: Option, - expires_at: Option, // = retired_at + configured retention -} - -enum AggStatus { - /// Listed in current StreamingConfig. Writes accepted. - Active, - /// Removed from StreamingConfig but within retention. Writes - /// rejected; reads allowed. - Retired { expires_in: Duration }, - /// Past retention. Scheduled for deletion. - Expired, -} -``` - -Implemented in `data_plane/src/storage_engines/sketch_db/index/mod.rs` -(`AggSchema`, `AggStatus`, `SchemaRegistry`). - -### 6.2 Status transitions - -``` -┌─────────┐ control plane POSTs new StreamingConfig -│ (none) │ with this agg_id included -└────┬────┘ - │ create_schema() - ▼ -┌─────────┐ control plane POSTs new StreamingConfig -│ Active │ that removes this agg_id -└────┬────┘ - │ retire_schema() - ▼ ┌──────────────┐ -┌─────────┐ time passes until │ Compaction │ -│ Retired │ retired_at + retention │ policy drops │ -└────┬────┘ │ to Partial / │ - │ expire() │ None as │ - ▼ │ expires_in │ -┌─────────┐ TTL sweep │ shrinks │ -│ Expired │ ─────► DELETE └──────────────┘ -└─────────┘ -``` - -Each transition is atomic on the ArcSwap that carries `StreamingConfig` -plus an explicit DB call (no polling; Active→Retired happens when the -HTTP handler that processes the config swap sees the id missing from -the new config). - -### 6.3 Write-side schema barrier - -Every write to the store is gated by `is_writable(agg_id)`: - -```rust -fn write_entry(&self, entry: SketchEntry) -> Result<(), WriteError> { - let schema = self.schemas.get(entry.agg_id).ok_or(WriteError::UnknownAgg)?; - match schema.status() { - AggStatus::Active => { /* validate entry matches schema, write */ } - AggStatus::Retired { .. } => Err(WriteError::RetiredAgg), - AggStatus::Expired => Err(WriteError::ExpiredAgg), - } -} -``` - -This is the authoritative "no writes after retirement" guarantee. -Even if a slow in-flight ingest batch routes stale data to the -retired agg_id, the store rejects it. Workers log the rejection and -drop the batch. - -Implemented at `precompute_engine/ingest_handler.rs` — the ingest -path calls `state.schemas.is_writable(config.aggregation_id)` before -every write; the `SAMPLES_BLOCKED_BY_SCHEMA_BARRIER` counter tracks -barrier rejections. - -### 6.4 Accuracy profile is part of the schema - -Every sketch type has a **mathematically proven** error bound that -depends only on its parameters and the sketch family — not on the -input data. This means as soon as the control plane commits to an -`AggregationConfig`, the resulting MV's accuracy guarantees are -fixed and known. The schema metadata captures this: - -```rust -struct AggSchema { - // ... fields from §6.1 ... - - /// Error bound this MV can serve, derived purely from - /// sketch_type + parameters. Cached on AggSchema creation so - /// the query path can return it without recomputation. - accuracy_profile: AccuracyProfile, -} - -struct AccuracyProfile { - /// Per-statistic answerability: which statistics this sketch - /// can answer at all, and what the error bound is for each. - per_statistic: HashMap, - /// Probability the bound holds (1 - δ for CMS, asymptotic 95% - /// for KLL/HLL/DDSketch unless overridden, 1.0 for typed aux - /// scalars). - confidence: f64, - /// What happens to the bound when N entries of this sketch are - /// merged. Some sketch families preserve the bound on merge - /// (CMS cell-wise sum, HLL register OR, KLL level merge with - /// modest overhead); others widen it. - merge_propagation: MergePropagation, -} - -enum ErrorBound { - /// Symmetric: |estimate − true| ≤ delta with prob ≥ confidence - AbsoluteSymmetric { delta_fn: BoundFn }, - /// One-sided over-estimator (CMS): true ≤ estimate ≤ true + δ - OneSidedOver { delta_fn: BoundFn }, - /// Multiplicative: |estimate − true| / true ≤ ε - Relative { epsilon: f64 }, - /// Standard error: σ for normal approximation; UI picks z - /// (e.g. 1.96 for 95% CI, 2.58 for 99%) - StandardError { sigma_fn: BoundFn }, - /// Rank error (KLL): the returned quantile's true rank is - /// within `epsilon` of the requested rank, with prob ≥ conf - Rank { epsilon: f64 }, - /// Exact — typed aux columns (count/sum/min/max), no error - Exact, -} - -/// Functions that compute an absolute error from per-window data -/// (e.g. CMS frequency: δ = ε·||x||₁ depends on total mass in window). -type BoundFn = Box f64>; -``` - -The actual ε / δ / σ formulas per sketch family are listed in -[`design-sketch-db-performance.md`](./design-sketch-db-performance.md) -§19.9 (theoretical bounds appendix). The point of putting them on -`AggSchema` is so the control plane can reason about -"does this sketch satisfy my query's accuracy SLA?" at plan time — -and the query path can return the bound to the user without -recomputation. - -This also ties into PromSketch (Tier 1) and Tier 2 having different -accuracy profiles for the same metric: a Tier 1 PromSketch over -recent samples might use a smaller-K KLL than the Tier 2 long-term -storage. The query engine can prefer Tier 1 for queries whose SLA -permits the looser bound, and Tier 2 for tighter SLA. - -Implemented in `data_plane/src/storage_engines/sketch_db/accuracy.rs`. - ---- - -## 7. Schema timeline — the key to query continuity - -### 7.1 What it is - -```rust -metric_timelines: HashMap< - MetricName, - BTreeMap -> -``` - -For each metric, an ordered list of `(time_range, agg_id)` segments -covering its history. Non-overlapping by construction (the control plane -contract: a metric has one Active agg_id at a time, plus zero or more -Retired overlapping within retention). - -Example for metric `latency`: - -``` -│── id=1 (CMS256) ──┤ - │── id=17 (CMS1024) ──┤ - │── id=42 (KLL200) ──► -T=0 T=day1 T=day3 now -``` - -### 7.2 `timeline_for_metric(metric, t1, t2) -> Vec<(AggId, TimeRange)>` - -The single most important query-engine-facing API. Given a metric and a -time range, returns the agg_ids that cover the range (in time order), -each clipped to the query range. - -```rust -timeline_for_metric("latency", day1-1h, day1+1h) - → [(1, [day1-1h, day1]), - (17, [day1, day1+1h])] -``` - -Query engine uses this to dispatch per-segment. DB provides it from an -in-memory BTree; cost is one HashMap lookup + BTree range scan, -nanoseconds. Implemented at `schema.rs`; used by -`storage_engines/sketch_db/query/timeline_dispatch.rs`. - -### 7.3 Per-segment query dispatch - -When the query engine processes `count_over_time(latency[24h])` at a -moment spanning multiple segments: - -```rust -fn query_metric(&self, metric: &str, range: (u64, u64), - statistic: Statistic) -> QueryResult { - let segments = self.timeline_for_metric(metric, range.0, range.1); - let mut partials = Vec::new(); - for (agg_id, seg_range) in segments { - // Check coverage — was this segment written by Native ingest, - // is a Backfill in progress, or is the range not covered? - match self.coverage(agg_id, seg_range) { - Coverage::Complete => { - partials.push(self.query_range(agg_id, seg_range, statistic)); - } - Coverage::BackfillInProgress { pct } => { - // Either wait (if eta short) or fall back - partials.push(self.fallback_exact(metric, seg_range, statistic)); - } - Coverage::Missing => { - partials.push(self.fallback_exact(metric, seg_range, statistic)); - } - } - } - combine_statistic(partials, statistic) -} -``` - -`combine_statistic` depends on the statistic: - -| Statistic | Cross-segment combinability | -|---|---| -| Count, Sum, Min, Max, Cardinality (HLL) | Combinable (addition / max / HLL-OR) | -| Quantile, TopK | **Not combinable across different sketch types / parameters.** Returns `PartialResult { covered, missing }`. | - -When a statistic is not combinable across schema changes, the user sees -a `PartialResult` that the query engine can render as a warning or a -fall-through to the exact DB for the missing segment. Crucially, this -failure mode is **explicit** — the user knows they are seeing a -schema-change artifact. - -Implemented in `storage_engines/sketch_db/query/timeline_dispatch.rs`. Coverage-driven branching -(the `match` in the snippet above) is present in skeleton form; the -`Coverage::BackfillInProgress` → wait-or-fallback policy is still a -follow-up (see roadmap §16 "Phase 5f"). - ---- - -## 8. Incremental view maintenance — the live ingest path - -This is the MV maintenance strategy that the precompute engine already -implements today. The doc is just making explicit what the design is -and what additions are needed to fit cleanly into the sketch DB. - -### 8.1 What "incremental" means for a sketch MV - -Each newly-arriving sample (or pre-built short-window sketch from a -DataCollector processor) is folded into the MV by **merging into the -accumulator of the matching `(agg_id, group_key, window_start)`**. -This is exactly the semantics of incremental MV maintenance: the -change to the base relation (one new sample) propagates to the view -(one accumulator update) without re-materializing the view. - -The key invariant that makes this tractable for sketches is that every -sketch type the pipeline supports is **mergeable** — the merge -operation is associative and commutative (modulo accuracy bounds for -probabilistic sketches), so the MV can be maintained by folding new -data into open accumulators regardless of order. - -### 8.2 The pipeline - -``` - raw sample / pre-built sketch arrives - │ - ▼ -┌────────────────────────────────┐ -│ IngestState │ -│ match by metric name │ -│ snapshot current StreamingConfig │ ← ArcSwap read, ~5ns -│ for each matching agg_id: │ -│ build (agg_id, group_key) │ -│ route to Worker │ -└─────────┬──────────────────────┘ - ▼ -┌────────────────────────────────┐ -│ Worker │ -│ get_or_create_group_state │ -│ reads schema from ArcSwap │ -│ builds Accumulator per the │ -│ MV's view definition │ -│ accumulate_one(sample) │ ← incremental update -│ or merge_with(sketch) │ ← associative fold -└─────────┬──────────────────────┘ - ▼ -┌────────────────────────────────┐ -│ Window close │ -│ emit PrecomputedOutput │ -│ write SketchEntry to store │ -│ with origin = Native │ -└────────────────────────────────┘ -``` - -Every step is O(1) per incoming sample (or O(sketch_size) per -incoming pre-built sketch). There is no scan of the base relation, -no re-materialization — pure incremental update. - -### 8.3 Window close = MV row emission - -The MV model clarifies what "flushing a window" means: it's the -*emission of an MV row*. The window manager determines when the row -is finalized; once finalized, the row is immutable in the store and -subject to further transformation only by semantic compaction -(roadmap §9) or by the full-refresh path (§10) replacing it. The MV -row carries: - -- The grouping key values (matches the MV's `GROUP BY` columns) -- The window bounds (matches `time_bucket(…)`) -- The aggregated sketch (the `agg_fn(…)` output) -- The typed aux columns `count`/`sum`/`min`/`max` (covering pre-aggregated - scalar queries without touching the sketch) -- `origin: Native` (distinguishes from Refresh-produced rows; see §10) - -### 8.4 Watermark and lateness policy - -Incremental MV maintenance against a streaming base relation has the -classical late-data problem: a sample arrives whose timestamp places -it in a window that has already been flushed. The current -`allowed_lateness_ms` and `LateDataPolicy` knobs implement a bounded -out-of-orderness policy: samples up to `watermark - allowed_lateness` -fold into open accumulators; older samples trigger the configured -policy (`Drop` or `ForwardToStore`). - -In MV terms, this is the trade-off between "eventual MV consistency -with late-arriving base data" and "bounded MV commit latency." The -default (`Drop`) chooses bounded latency; `ForwardToStore` chooses -eventual consistency. A real sketch DB should surface this trade-off -as per-`AggSchema` policy rather than a global knob — some MVs need -strict correctness (financial / regulatory), some tolerate drops -(dashboards). - -### 8.5 Interaction with reconfigure - -Because the worker's schema lookup (in `get_or_create_group_state`) -reads the current `StreamingConfig` from ArcSwap on each -first-time-seen `(agg_id, group_key)` pair, the incremental path -transparently picks up newly-created `agg_id`s the moment they are -added to the config. This is what PR #16 delivers. - -What the incremental path **cannot** do alone is fill in the MV -retroactively — once a window is closed under agg_id=1, it will never -be populated under agg_id=17, even if agg_id=17's view definition -would have produced a different row for that window. Populating the -new `agg_id` for historical windows is the job of the full-refresh -path (§10). - -### 8.6 What's missing for a sketch-DB-grade incremental path - -Most of the infrastructure is already there via the precompute -engine. The sketch-DB additions on top of it: - -- **Typed aux columns** (§5.1) on every flushed entry, not just the - sketch bytes. Already trivial to add; the processors already have - `count`/`sum`/`min`/`max` in the typed OTLP fields. -- **`origin: Native` / `origin: Backfilled { job_id }` provenance - tagging** to distinguish rows produced by incremental maintenance - from rows produced by refresh. Needed to reason about coverage - (§10.4) and for idempotent re-runs. ✅ implemented. -- **Schema-validating write barrier** (§6.3) enforcing that the - incoming write matches the target `agg_id`'s pinned schema. ✅ - implemented. -- **Per-`agg_id` label postings update** as new group_key values - appear (§5.3). Trivial — incremental update to a Roaring bitmap - per first-seen group_key. ❌ not yet. - -None of these change the fundamental data flow; they make the -incremental MV maintenance contract observable and enforceable. - ---- - - - -## 10. Refreshable view maintenance — the backfill path - -Where §8 describes how the MV is maintained as new base data arrives, -this section describes how the MV is **re-materialized** from the base -relation on demand. In SQL terms: `REFRESH MATERIALIZED VIEW agg_17 -FOR PERIOD (now - 24h, now) FROM exact_db`. - -### 10.1 The core insight - -The exact DB holds raw samples losslessly over its configured -retention. Any sketch for any time range within that retention can be -rebuilt from it. A reconfigure that breaks query continuity in the -sketch tier can be followed by a backfill that restores it. - -``` -Time → T=0 T=upgrade-horizon T=upgrade now - │ │ │ │ - │ │ │ │ -id=1 [═══════ native (CMS256) ══════════════════] │ - │ │ │ │ - │ │ │ │ -id=17 │ [▒▒▒▒ backfilled ▒▒▒▒▒▒][══ native (KLL200) ══] - │ │ │ │ - │ └── REFRESH reads from │ │ - │ exact DB for this │ │ - │ interval │ │ - │ │ │ │ - ▼ ▼ ▼ ▼ - ═══════════════ Exact DB (base relation) ══════════════════════ - (lossless raw samples — S3+Gorilla / Prometheus / CH / VM / … - feeds the refresh path for the backfilled interval and the - query engine's fallback for any query sketches can't answer) -``` - -Reading the diagram: -- **id=1** was the pre-upgrade `AggregationConfig` (CMS with width=256). - It keeps its existing data across the upgrade; IngestState stops - writing to it at `T=upgrade`. Its rows are still query-visible until - `persistence_delete_older_than_secs` elapses, but they are not used - for post-upgrade queries (the schema timeline in §7 routes queries - to id=17 after the swap). -- **id=17 (native portion)** is live incremental MV maintenance (§8): - every window close after `T=upgrade` emits a KLL(200) row with - `origin = Native`. -- **id=17 (backfilled portion)** is the refresh output (§10.1–10.5): - a backfill job reads raw samples from the exact DB for - `[T=upgrade - horizon, T=upgrade)`, rebuilds KLL rows per - `(group_key, window)`, and writes them to id=17 with - `origin = Backfilled { job_id }`. - -After backfill, id=17 covers the full query horizon. The user's query -"last 24h" is fully served from a single sketch, with a consistent -schema. - -### 10.2 Backfill job model — a REFRESH transaction - -```rust -struct BackfillJob { - job_id: u64, - agg_id: u64, - time_range: (u64, u64), - source: BackfillSource, - status: BackfillStatus, - started_at: u64, - completed_at: Option, - windows_done: u64, - windows_total: u64, -} - -/// Every variant reads raw samples for the requested metric and time -/// range and feeds them to the sketch builder. The DB picks a concrete -/// reader at job-dispatch time based on what the deployment has -/// configured as its exact DB. All variants implement a common -/// `RawSampleReader` trait internally so the rest of the backfill -/// code is source-agnostic. -enum BackfillSource { - /// Gorilla-compressed files in S3 / MinIO / GCS, produced by - /// DataCollector's gorillacol + S3 Files exporter. - S3Gorilla { bucket: String, prefix: String }, - /// Prometheus (or VictoriaMetrics / Thanos / Cortex) via the - /// HTTP range-query API. - Prometheus { url: String }, - /// ClickHouse via native HTTP / SQL. - ClickHouse { url: String, table: String }, - /// Rebuild from a different sketch (rare; only when types are - /// compatible and the source sketch is lossless w.r.t. the target). - /// Used for lossless schema widenings, e.g. CMS(256) → CMS(2048). - OtherSketch { source_agg_id: u64 }, -} - -enum BackfillStatus { - Queued, - Running, - Complete, - Failed(String), - Cancelled, -} -``` - -Implemented in `storage_engines/sketch_db/backfill.rs` (types + registry) and -`backfill_service.rs` / `backfill_worker.rs` (worker pool). - -### 10.3 Refresh is a separate worker pool - -Live ingest and backfill must not starve each other: - -- Live ingest has hard latency requirements (data must land in the - current window before it closes). -- Backfill is latency-tolerant (it's catching up historical windows - that are already closed) but CPU- and I/O-heavy. - -Implementation: a dedicated `BackfillWorkerPool` with configurable -concurrency. Priority knob on each `BackfillJob`. Backfill writes go -through the same schema barrier (`is_writable(agg_id)`) as live writes -but via a distinct `write_backfilled_window()` entry point that -bypasses `WindowManager` (the window is already closed, we're just -populating it). - -### 10.4 Coverage tracking - -```rust -enum Coverage { - /// This time range is fully covered by either Native writes or - /// completed Backfilled writes. Query proceeds against sketch. - Complete, - /// A backfill job is in progress for this range. - BackfillInProgress { job_id: u64, pct: f64 }, - /// This range is not covered at all. - Missing, -} - -fn coverage(&self, agg_id: u64, range: (u64, u64)) -> Coverage -``` - -Cached in memory keyed by `agg_id`, updated as Native writes land and -as Backfill jobs complete. `Coverage` enum exists in code; integration -with the query path is a follow-up (see roadmap §16 "Phase 5f"). - -### 10.5 Deterministic rebuild - -For backfill to produce "the same sketch we would have built live," -the sketch construction must be deterministic. That means: - -- The hash function seed for CMS / CountSketch / HLL must be part of - `AggregationConfig.parameters` and stable across Native and Backfill - paths. -- The KLL / DDSketch sampling decisions must be deterministic from - the input sample order. This constrains how raw samples are read - from the exact DB — they must be replayed in the same order they - were ingested live. -- Exact-DB writes from the live data plane therefore need to preserve - ingest order within a window (whether the exact DB is S3+Gorilla, - Prometheus, or any other backend). - -This is an invariant the Gorilla processor / S3 exporter must -guarantee. Not hard, but it has to be designed in. The rebuild -processor scaffolding is in place at `backfill_processor.rs`; -end-to-end determinism is still being validated. - -### 10.6 When to use incremental vs refresh - -The two maintenance strategies (§8, §10) are not alternatives — they -cover different situations and the control plane should pick per -situation. - -| Situation | Strategy | Why | -|---|---|---| -| Steady-state live ingest | Incremental (§8) | O(1) per sample, no scan of base relation | -| New `agg_id` created, no historical need | Incremental only | Nothing to refresh from | -| New `agg_id` created, need query continuity across the reconfigure | Incremental **+** Refresh from exact DB over the query horizon | Incremental covers \[now, future\]; refresh covers \[now-horizon, now\] | -| Recovery from a store corruption or a bug in past sketch builds | Refresh | MV is known wrong; rebuild authoritatively from base | -| Onboarding a metric with historical raw data already in the exact DB | Refresh only (until catches up), then Incremental | Much cheaper than streaming a week of historical data through the live ingest path | -| Metric with very low query rate, reconfigure | Incremental only, fall back to exact DB for historical queries | Refresh cost > fallback cost at low QPS | - -The control plane decides by comparing estimated costs: - -``` -cost_of_refresh = exact_db_bytes_to_scan * read_$_per_byte - + cpu_seconds * cpu_$ -cost_of_fallback = expected_queries_to_exact_DB_during_retention - * query_latency * query_$ -if cost_of_fallback > cost_of_refresh: trigger refresh -``` - -The `/api/v1/db/cost_estimate` endpoint (§15) is what the control plane -asks to get each side of this inequality. - -### 10.7 Relationship to classical MV refresh - -Warehouses have two common refresh modes: - -- **`REFRESH MATERIALIZED VIEW` (blocking)** — the MV is locked, its - contents replaced, readers wait. Not viable here: the MV is being - actively queried by dashboards. -- **`REFRESH MATERIALIZED VIEW CONCURRENTLY`** — rebuild into a - shadow table, atomic swap when done. Readers see the old version - until the swap. - -This design is closer to the *concurrent* model, with a twist: -refresh is scoped to a time range, not the whole MV. Until a refresh -job completes, the query engine serves the covered portion from the -not-yet-refreshed state (which may be a different `agg_id`, or may -be partial `Native` coverage) and the uncovered portion via -fallback. After the swap (marking the range as `BackfillComplete`), -queries transparently start reading the refreshed data. - -Unlike a warehouse's `CONCURRENTLY REFRESH`, the refresh here is -**incremental per window**, not all-or-nothing per MV. A job writing -24 hours of backfilled KLL entries can mark each 10-second window -complete independently. Queries that straddle the in-progress -boundary see a split coverage map (some windows Complete, some -BackfillInProgress), and the query engine handles the split per -§7.3's per-segment dispatch. - ---- - - - -## 14. The full reconfigure workflow - -``` -T=0: Config = {1: CMS(256)} - DB schemas = {1: Active} - Live: writing to id=1 - -T=swap: Control plane decides to upgrade to KLL(200). - 1. Allocates new id=17. - 2. Dual-write phase begins. - POST /api/v1/streaming-config { 1, 17 } - DB: schema 17 created Active; schema 1 still Active. - IngestState routes to both id=1 and id=17. - (Both sketches accumulating the same underlying samples.) - -T=swap+Δ: - 3. Control plane triggers backfill to close the historical gap - for id=17: - POST /api/v1/db/backfill { - agg_id: 17, - time_range: (swap - query_horizon, swap), - source: S3Gorilla, // or Prometheus, ClickHouse — whichever backs the exact DB - priority: High, - } - Backfill service reads raw from S3, rebuilds KLL per window - per group, writes to id=17 with Origin = Backfilled. - - During this phase, a query for "last 24h": - - segments for [now-24h, swap] hit id=1 (CMS, native data) - - segments for [swap, now] hit id=17 (KLL, native data) - - If the statistic is combinable (Count, Sum, Min, Max, Card), - cross-segment combine works. - - If not (Quantile, TopK), segment [now-24h, swap] returns - PartialResult; engine falls back for that segment. - -T=backfill_done: - id=17 now covers [swap - query_horizon, now] completely. - A query for "last 24h" hits id=17 end-to-end, single schema. - -T=backfill_done+ε: - 4. Control plane removes id=1 from StreamingConfig. - POST /api/v1/streaming-config { 17 } - DB: schema 1 transitions Active → Retired. - IngestState stops routing to id=1. - Worker evict_orphaned_groups reaps (agg_id=1) GroupStates. - -T=swap + retention: - 5. Schema 1 expires. - DB TTL sweep (Phase 3 of the flusher) deletes all id=1 - data from disk based on max_ts. - - Final state: Config = {17}, DB schemas = {17: Active}. - No tombstones, no dangling references, no agg_id-specific - deletion needed — time-based TTL handles cleanup on a - schema-change-oblivious index. -``` - -Every step is idempotent. If the control plane crashes mid-workflow, the -DB state at any point is a valid state; the control plane resumes from -wherever it left off by reading `/api/v1/db/schemas`. - ---- - -## 15. API surface - -### 15.1 Query-engine-facing API - -Today the query engine says "give me data for `agg_id = 1`." In the -sketch DB it says "give me `count_over_time` for metric `latency` in -this time range" and the DB does the rest. - -```rust -/// Primary query-engine entry point. Dispatches by metric, not agg_id. -fn query_metric( - &self, - metric: &str, - group_filter: Option, - time_range: (u64, u64), - statistic: Statistic, -) -> QueryResult; - -/// Point read for a known (agg_id, group_key, window). Used by -/// diagnostic tooling and by tests; not the main production path. -fn point_read( - &self, - agg_id: u64, - group_key: &str, - window_start: u64, -) -> Option; - -/// Streaming scan for batch workloads. Iterator yields entries in -/// primary-key order. -fn scan( - &self, - agg_id: u64, - range: (u64, u64), - predicate: Option, -) -> impl Iterator; -``` - -`QueryResult` carries the **estimate**, an **error bound**, the -**confidence** with which the bound holds, and **provenance** about -what data sources produced the answer. This is a first-class part of -the query contract, not an optional debug field — sketches are -approximate by design and the cost of returning the bound alongside -the value is essentially zero. - -```rust -struct QueryResult { - /// The point estimate. - estimate: ResultValue, - /// Mathematically-derived error bound for `estimate`. See §6.4 - /// for the AccuracyProfile that produces this, and - /// design-sketch-db-performance.md §19.9 for the per-sketch-type - /// formulas. - error_bound: ErrorBound, - /// Probability the bound holds. CMS uses (1 - δ); KLL/DDSketch - /// use the asymptotic confidence implied by their parameters - /// (default ~95%); HLL likewise; typed aux columns are 1.0 - /// (exact). Cross-segment combinations multiply or take the min - /// depending on independence assumptions - /// (design-sketch-db-performance.md §19.10). - confidence: f64, - /// Where the answer came from — sketch family used, how many - /// entries were merged, whether any segment was backfilled, - /// whether any segment fell back to the exact DB. - provenance: Provenance, -} - -enum ResultValue { - /// Single value (Count, Quantile, Sum, Min, Max, Cardinality). - Scalar(f64), - /// Time series — per-window value. - Series(Vec<(u64, f64)>), - /// TopK and similar multi-value answers. - Vector(Vec<(String, f64)>), - /// Some or all of the requested range could not be served from - /// sketches. Caller decides whether to fall back. - Partial { - served: Box, - missing: Vec<(u64, u64)>, - reason: PartialReason, - }, -} - -struct Provenance { - sketch_types: Vec, // single entry if homogeneous - windows_merged: u32, // how many stored entries the answer aggregated - segments_combined: u32, // schema-timeline segments crossed - backfilled_segments: u32, // 0 = no historical refresh - fallback_segments: u32, // 0 = all from sketch DB - archive_authoritative: bool, // true if sketches were rebuilt from archive -} -``` - -The `ErrorBound` enum mirrors §6.4 and is consumed without -recomputation — `AccuracyProfile.per_statistic[stat]` returns the -right variant directly. - -**Why every result carries this**: it lets clients (dashboards, -alerts, downstream systems) make informed decisions: an alert can -require `confidence ≥ 0.99` before firing; a dashboard can render -error bars without doing a second query; a downstream consumer can -decide to refetch from the exact DB if the bound is too wide for -its use case. - -**Cost**: an extra ~50 bytes per query response. Computation is -O(1) per query. Negligible compared to the actual sketch merge. - -### 15.2 Control-plane-facing API - -``` -GET /api/v1/db/schemas?status= - → list of AggSchema objects - -GET /api/v1/db/schemas/{agg_id} - → one AggSchema + current Coverage per time range - -GET /api/v1/db/timeline/{metric} - → schema timeline for a metric - -GET /api/v1/db/stats/{agg_id} - → { - bytes_stored, - entry_count, - avg_sketch_size, - queries_served, - p50_query_latency_ms, - p99_query_latency_ms, - last_queried_at, - avg_merge_cost_per_query, - } - -POST /api/v1/db/backfill - body: { agg_id, time_range, source, priority } - → { job_id } - -GET /api/v1/db/backfill/{job_id} - → { status, progress, eta } - -DELETE /api/v1/db/backfill/{job_id} - → cancel running job - -POST /api/v1/db/cost_estimate - body: { metric, sketch_type, parameters, grouping, window_size } - → estimated { bytes_per_sec, cpu_per_sec, storage_per_day } - -GET /api/v1/db/pressure - → { write_queue_depth, compaction_lag, memory_used, memory_limit } -``` - -These are the primitives the control plane uses to close its planning -loop. Without them the control plane plans in the blind; with them it -can run cost-based optimization with real observations. - -> **Status.** `GET /api/v1/db/schemas`, `/timeline`, `POST -> /api/v1/db/backfill`, `GET /api/v1/db/backfill/jobs`, -> `POST /api/v1/streaming-config` swap, retire/expire are implemented -> in `drivers/query/servers/http.rs`. `/stats`, `/cost_estimate`, and -> `/pressure` are still aspirational — the planner cost model currently -> uses hand-coded estimates rather than live store observations (see -> [`design-sketch-db-performance.md`](./design-sketch-db-performance.md) -> §20 "Sketch profiler library" for the eventual source of those -> numbers). - ---- - - - -## 21. Related approaches: wavelets and ML models as materialized views - -The MV framing in §2 treats sketches as "precomputed, -incrementally-maintainable summaries of a base relation." That -description is broader than sketches — wavelets and (some) ML models -fit it too. This section positions the sketch DB design against those -neighbours so future extensions can reason about which of them slot in -cleanly and which require contract changes. - -The framing holds whenever five properties are present: - -1. A **base relation** the summary is derived from. -2. **Deterministic derivation** (given parameters). -3. Either **incremental** or **refresh** maintenance semantics. -4. **Queries answerable without rescanning the base.** -5. A **known accuracy contract** (how wrong the answer can be). - -Sketches hit all five. Wavelets and ML models hit some but not all — -the pattern of misses determines what it would take to treat them as -first-class citizens of the sketch DB. - -### 21.1 Side-by-side comparison - -| Dimension | Sketches (CMS / HLL / KLL / DDSketch) | Wavelets (DWT / Haar + thresholding) | ML models | -|---|---|---|---| -| Base relation | Raw sample stream | Signal / time series | Training set | -| Mergeable (monoid) | **Yes, by design** — associative + commutative merge is a defining property | **Partially** — Haar on aligned dyadic intervals merges cleanly; general DWT does not | **Rarely** — only linear / moment-based things (online PCA via covariance sums, linear regression normal equations, naive Bayes w/ conjugate priors). Neural nets are not monoids: training on A then B ≠ B then A | -| Incremental update cost | O(1) per sample | Amortized O(log n) for online / sliding DWT | Variable; SGD continuations risk catastrophic forgetting | -| Refresh cost | Cheap (replay stream) | Moderate (one DWT pass) | **Huge** — full retraining is why RAG exists as a workaround | -| Accuracy bound | **Provable, closed-form** from parameters (ε, δ, K, α, m) | Provable — L2 error bounded by discarded coefficient energy (Parseval) | **Empirical**, data-dependent; PAC / conformal bounds exist but are much weaker and narrower | -| Query classes answered | Fixed at design: count, sum, quantile, top-k, cardinality | Fixed: range sums, heavy hitters, point queries, wavelet-domain features | **Open-ended** — whatever the training objective was | -| View-definition formalism | An aggregation function + parameters | A basis transform + threshold | Training objective + architecture + hyperparameters + seed | - -### 21.2 Wavelets — a sibling sketch family - -Wavelets are essentially an alternative sketch family. The classical -AQP line of work -(Garofalakis, Gibbons, Matias, Vitter — "Approximate Query Processing -via Wavelets," VLDB 1998 onward) treats them as a direct alternative -to randomized sketches for range-sum and heavy-hitter workloads. - -Compared to CMS / KLL: - -- **Strength**: wavelets exploit signal structure. On smooth or - low-entropy signals (diurnal telemetry, time-of-day patterns, - histograms that concentrate on a few modes) thresholded wavelets - produce dramatically smaller representations than sketches of - comparable accuracy. -- **Weakness**: on high-entropy / uniform data their advantage - disappears, because the thresholded coefficient set doesn't shrink. -- **Operational fit**: Haar wavelets on aligned dyadic intervals merge - cleanly, which means a Haar-based agg could use the same - `(agg_id, group_key, window)` storage as a CMS agg with only - modest changes to the merge trait. Non-Haar wavelets would require - either strict window alignment or a refresh-only maintenance - strategy. - -**Takeaway**: if a future `SketchType::Wavelet` were added, the -existing sketch-DB contracts (schema timeline, backfill, accuracy -profile, tier storage) generalize without structural change. The -`AccuracyProfile::ErrorBound` enum already has room for an -L2-energy-based variant. - -### 21.3 ML models — MVs with weaker contracts - -ML models fit the MV framing in the loose sense: they are precomputed, -queryable, compressed summaries of a base relation. But the sketch -DB's **four operational contracts** weaken or vanish: - -- **Mergeability** (§7.3 cross-segment combine) — lost for - non-linear models. Only linear or moment-based models (running - PCA, linear regression normal equations, conjugate Bayes, - streaming k-means coresets) retain a monoid structure and can - meaningfully merge across segments or time ranges. -- **Closed-form accuracy bound** (§6.4) — lost. Replaced by - empirical validation + sometimes conformal prediction bands. The - `AccuracyProfile` contract would need a new `Empirical` variant - that carries calibration data rather than a parameterized formula. -- **Deterministic rebuild** (§10.5) — weak. Training is - conditionally deterministic (seed + hyperparams + batch order + - hardware), but reproducing bit-identical outputs across hardware is - a well-known open problem in ML engineering. -- **Cheap refresh** — gone. Refresh cost is the dominant operational - concern for large models; the two-tier maintenance strategy - (incremental + refresh) that makes sketch-DB reconfigure painless - does not translate — continual training and fine-tuning are poor - substitutes for sketch refresh. - -The interesting **sub-class** that does fit is linear / additive -models: - -- **Online PCA / streaming covariance** — monoid via covariance sum; - bounded error via eigenvalue bounds. Functionally a sketch. -- **Linear regression (normal equations form)** — `XᵀX` and `Xᵀy` - accumulators merge by summation; bounds follow from standard linear - algebra. Functionally a sketch. -- **Coreset-based clustering** (BIRCH, k-means coresets) — mergeable - by construction; accuracy is a multiplicative factor on the optimal - clustering cost. -- **Naive Bayes with conjugate priors** — parameter updates are - additive in sufficient statistics. - -Each of these could be added to the sketch DB as a `SketchType` -variant without changing the core contract. They would share schema -timeline, backfill, accuracy profile, and tier storage with the -existing sketches. - -Neural / tree-ensemble / LLM models would require a parallel design -with weaker contracts: no merge, refresh-only maintenance, -empirical-only accuracy. That's closer to a **model registry** than a -sketch DB, and the open research literature -(Kraska et al. — SageDB; Hilprecht et al. — DeepDB; Yang et al. — -NeuroCard; DBEst / DBEst++) explores exactly that split. A pragmatic -integration path would be: host model artifacts beside sketches under -the same `agg_id` lifecycle and HTTP surface, but use a separate -storage engine internally — the `SketchDb` facade -(see [`design-sketch-db-pluggable.md`](./design-sketch-db-pluggable.md)) -makes this kind of backend swap mechanical. - -### 21.4 Practical implication for the sketch DB design - -The sketch DB's architecture — `agg_id` lifecycle, schema timeline, -backfill-from-base, accuracy-as-metadata — generalizes without change -to: - -1. **Sketches** (today). -2. **Wavelets** (mostly; Haar is trivial, general DWT needs window - alignment). -3. **Linear-ish ML summaries** (PCA, linear regression, coresets, - conjugate-prior Bayes). - -It does **not** generalize cleanly to neural / tree-ensemble models -without relaxing the mergeability and closed-form-bound contracts. -Keeping those relaxations out of the core, and introducing them in a -companion "model view" subsystem if the need arises, preserves the -properties that make the sketch DB's behavior predictable. diff --git a/docs/design-sketch-db-performance.md b/docs/design-sketch-db-performance.md deleted file mode 100644 index 68f027c7..00000000 --- a/docs/design-sketch-db-performance.md +++ /dev/null @@ -1,435 +0,0 @@ -# Sketch DB — Performance & Profiler - -> **Scope.** A performance analysis of the sketch DB vs. Prometheus / -> VictoriaMetrics (§19), plus the design of the measurement substrate -> — the **Sketch Profiler** library — that provides the real numbers -> other sections depend on (§20). -> -> **Audience:** anyone deciding whether to adopt the sketch DB for a -> workload, tuning parameters, or building the cost model in the -> control plane. -> -> **Status.** §19 is an analysis, not implementation. §20 describes a -> library that does not exist yet. - -Section numbers match the original monolithic `design-sketch-db.md`. - ---- - -## 19. Appendix: performance envelope vs Prometheus / VictoriaMetrics - -This section estimates the performance of the sketch DB relative to -Prometheus and VictoriaMetrics on representative queries. Numbers are -**order-of-magnitude estimates** derived from published benchmarks and -sketch complexity bounds, not measurements from this codebase. They -motivate which workloads the design targets and which it explicitly -does not. - -### 19.1 Reference workload - -A realistic production scenario that stresses TSDB scanning: - -- **10,000 hosts × 10 services = 100,000 time series** -- **100 ms scrape interval → 10 samples/sec/series** -- **Total ingest: 1 M samples/sec** -- **1 hour of data: 3.6 B samples** (one metric), or **54 B samples** - if the metric is exposed as a Prometheus histogram with 15 bucket - time series -- **1 day: 86.4 B samples** (plain) or **1.3 T samples** (histograms) - -The 100 ms scrape is not extreme for targeted high-frequency workloads -(NCCL GPU collective metrics, network packet counters, HFT, 5G radio -metrics). Standard 15 s / 30 s scrape regimes produce lower pressure -but the same relative shape. - -Baseline TSDB scan throughput used below: -- Prometheus: ~1 M samples/sec/core -- VictoriaMetrics: ~10 M samples/sec/core (published) - -Sketch DB costs: -- KLL merge: ~1 μs per merge -- CMS merge: ~microseconds (size-bounded) -- HLL register OR: ~μs -- Quantile / cardinality compute: microseconds post-merge - -### 19.2 Per-query-type estimates - -| Query | Workload | Prom | VM | Sketch DB Tier 2 | Sketch DB Tier 1 | -|---|---|---|---|---|---| -| **p99 quantile, 1h, by service** (`histogram_quantile` style) | 1.5 M bucket series, 54 B samples involved in full scan | **Infeasible cold** (15 core-hours) — requires recording rules | **~6 min on 16 cores** cold; 10–60 s with cache | **~1 ms** (600 KLL merges) | **< 1 ms** (EH lookup) | -| **Top-K over 10 M customers, 1h** | 10 M series × 3600 × 10 = 360 B samples | **Not possible** | **15–60 min** or OOM | **~5–10 ms** (60 CMS+heap merges) | N/A | -| **Cardinality / count_distinct, 1d** | 10 M series enumerated | **Not possible** | **10–60 min** or OOM | **< 1 ms** (HLL estimate) | **< 1 ms** | -| **Low-card short-range** (`rate({service="auth"}[5m])`) | 3,000 samples | ~10 ms | ~2–5 ms | ~2 ms | ~1 ms | -| **Multi-sub-window dashboard** (p99 @ 1m / 5m / 15m / 1h on same series) | 4 independent scans | 4× single-panel (seconds to minutes) | 4× single-panel | 4× independent part scans | **1× — shared EH** | -| **Point read** of one specific series at one point in time | 1 chunk | ~5–10 ms | ~2–5 ms | **Not supported** → falls back to exact DB (+ ~0.5 ms routing) | Not supported | -| **Exact count** (`count_over_time({customer="X"}[1d])`) | varies | Exact, seconds | Exact, sub-second | **Approximate** (CMS, ε ≈ 10⁻³) in ~10 ms | N/A | - -Columns where Sketch DB is "not supported" are where the design -explicitly defers to the exact DB — point reads and exact queries are -the job of the exact DB that also serves as the base relation (core -§2.1, §10.1). - -### 19.3 Why sketch DB wins grow with scrape rate - -The key observation that justifies treating 100 ms scrape as the -benchmark case: - -- **TSDB scan cost scales linearly** with sample count, which scales - linearly with scrape rate. 600× higher scrape rate → 600× more work - for every aggregate query. -- **Sketch DB cost scales with sketch parameters, not sample count.** - A KLL over 10⁶ samples and a KLL over 10⁹ samples have the same - serialized size and the same quantile-query cost. At 100 ms scrape - the ratio widens by roughly the scrape-rate ratio. - -At 1 min scrape, the p99 dashboard query is **3 s on Prom vs 1 ms -sketch** — 1,000× speedup. At 100 ms scrape, the same query becomes -**30 min → infeasible on Prom vs 1 ms sketch** — the ratio goes to -infinity because the TSDB falls off a cliff that the sketch DB -doesn't see. - -### 19.4 Storage cost - -Sketch size depends on sketch parameters, not on input sample count — -this is the most important property for high-scrape-rate deployments. - -| Storage object | 1 min scrape (100K series, 7 days) | 100 ms scrape (100K series, 7 days) | -|---|---|---| -| Prom/VM raw samples, compressed | ~100 GB | **~60 TB** | -| Sketch DB Tier 2, `grouping_labels = [service]`, 1 min windows | ~50 GB | **~50 GB** *(unchanged)* | -| Sketch DB Tier 2 + semantic compaction to 1 h windows | ~5 GB | **~5 GB** *(unchanged)* | -| Sketch DB Tier 1 (PromSketch), 5 min retention | ~100 MB | **~100 MB** *(unchanged)* | - -Storage cost ratio vs TSDB goes from **~2×** at 1 min scrape to -**~10,000×** at 100 ms scrape. Break-even — the scrape rate at which -sketch DB becomes cheaper than TSDB in storage alone — is somewhere -around 10–30 s scrape depending on sketch parameters and grouping. -Above that, sketch DB is strictly cheaper; below that, TSDB is. - -### 19.5 Qualitative shift: from "faster" to "feasible" - -The quantitative speedups above hide a more important shift. At high -scrape rates, a subset of queries becomes **impossible** on TSDB: - -| Query | 1 min scrape | 100 ms scrape | -|---|---|---| -| p99 dashboard over 1 h, 100 K series | Prom 3 s (tight) / VM 500 ms | Prom **30 min or OOM** / VM **6 min cold** | -| Top-K over 10 M customers | Prom **already infeasible** / VM 30 s | Prom impossible / VM **1–2 hours or OOM** | -| `count_distinct` over 1 day | Prom infeasible / VM 5–30 s | Prom infeasible / VM **OOM** | - -Sketch DB stays at **< 10 ms** for all of these regardless of scrape -rate. So at 100 ms scrape the sketch DB is not "an optimization" — it -is the **only way** to serve these queries interactively. - -### 19.6 Where TSDB still wins - -The sketch DB intentionally does not replace TSDB for: - -- **Point reads** of a single time series at a specific timestamp — - TSDB chunk reads are already sub-millisecond; sketch DB adds routing - overhead with no benefit. -- **Exact numerical answers** (financial reconciliation, regulatory - reporting, billing ground truth) — sketches are approximate by - design; the exact DB must serve these. -- **Ad-hoc exploration** of arbitrary label dimensions — sketches - only answer along the `grouping_labels` they were built for. A query - on an unplanned dimension falls through to the exact DB. -- **Low query rate** (few queries per day against a given metric) — - the pre-computation cost of maintaining sketches is not amortized. - The control plane's cost model should decline to materialize a sketch - for such metrics. - -### 19.7 Cost-model break-even - -Very rough formula the control plane can use to decide whether a metric -is worth sketching: - -``` -sketch_value = query_rate - × avg_series_covered_per_query - × (tsdb_query_latency − sketch_query_latency) - × retention_days -sketch_cost = agent_cpu_for_sketching - + backend_memory_for_live_sketches - + backend_storage_for_parts - + control_plane_planning_overhead - -materialize_if: sketch_value > sketch_cost -``` - -For high-QPS dashboard metrics with high cardinality, `sketch_value` -easily dominates. For cold metrics or exact-required metrics, it does -not and the control plane should leave them on the exact-DB path only. - -### 19.8 Summary - -| Dimension | Sketch DB advantage at 100 ms scrape | -|---|---| -| Aggregate queries (quantile, top-K, cardinality) | **10³–10⁶×** faster, often enabling queries TSDB can't serve | -| Storage for high-QPS metrics with grouping | **~10⁴× smaller** than raw TSDB | -| Short-window live queries with many sub-windows | **100–1000×** via Tier 1 EH locality | -| Agent → backend bandwidth | **~10–100×** via edge sketching | -| Point reads, exact queries, ad-hoc exploration | **0 or negative** — sketch DB defers to exact DB | -| Low-QPS / cold metrics | **Negative** — cost of materialization exceeds saving | - -The sketch DB is therefore best understood as an **accelerator** over -the exact DB rather than a replacement: it carries the 90 % of query -traffic that fits its design pattern at orders-of-magnitude lower -cost, and relies on the exact DB for the remaining 10 % where -sketching has no advantage. This dual-role architecture is why the -exact DB is modeled as both the refresh source (core §10) and the -query fallback target (core §7.3) — the same storage serving two -purposes. - -### 19.9 Theoretical accuracy bounds per sketch type - -These are the formulas every `AccuracyProfile` (core §6.4) is derived -from. They are mathematical guarantees from the sketch's defining -papers, not empirical estimates — given parameters and a sketch -type, the control plane and the query path know the bound exactly. - -| Sketch | Parameters | Statistic answered | Error bound | Confidence | -|---|---|---|---|---| -| **CountMin (CMS)** | width `w`, depth `d` | point frequency `f̂` | `0 ≤ f̂ − f ≤ ε · ‖x‖₁` with `ε = e/w` | `1 − δ`, `δ = e^(−d)` | -| **CountSketch** | width `w`, depth `d` | unbiased frequency | `|f̂ − f| ≤ √(‖x‖₂² / w)` (one-σ) | asymptotic ~68 % at 1σ, ~95 % at 2σ | -| **CMS + heap** | `w, d, k` | top-k by count | top-k items returned exactly when their true count exceeds `(k+1)`-th item by ≥ `ε‖x‖₁` | `1 − δ` | -| **KLL** | `K` (default 200) | quantile rank | rank error ≤ `c / √K` (c ≈ 1) | asymptotic ~99 % at default K | -| **DDSketch** | relative accuracy `α` | quantile value | `\|q̂ − q\| / q ≤ α` | deterministic if backing store is unbounded; bounded variants degrade gracefully | -| **HLL** | `m = 2^p` registers | distinct count | std error `σ ≈ 1.04 / √m` | asymptotic ~95 % at 2σ | -| **UnivMon** | levels `L`, base sketch | many statistics polymorphically | inherits from the layer-`L` base sketch's bound | per-layer | -| **Typed aux columns** (count/sum/min/max) | n/a | their respective scalar | exact (no sketch involved) | 1.0 | - -These formulas are encoded in `AccuracyProfile::per_statistic` as -`ErrorBound` variants. The query path returns them directly without -runtime computation. - -### 19.10 Merge error propagation - -When N entries of the same sketch type are merged (the common case -for range queries), the resulting bound depends on the sketch -family's algebraic properties: - -| Sketch family | Merge type | Bound after merge | -|---|---|---| -| **CMS** | cell-wise sum | `ε` unchanged; `‖x‖₁` grows to the merged stream's L1 norm — bound stays the same shape | -| **CountSketch** | cell-wise sum | `σ` unchanged on the merged distribution; bound preserved | -| **HLL** | register-wise max | bound preserved exactly; merge is the natural set-OR of the underlying multisets | -| **KLL** | level-by-level | bound preserved with small overhead (≤ 2 % typically); merging N KLLs with `K = 200` still gives ~1 % rank error | -| **DDSketch** | bucket-wise sum | relative `α` preserved exactly | -| **Typed aux** | additive (sum/count) or extremum (min/max) | exact | - -So the answer to "what is the bound after merging 60 KLL sketches -covering the last 1 hour?" is **the same bound as a single KLL** — -this is a designed property of mergeable sketches and is a major -reason to prefer them over non-mergeable approximations. - -**Cross-segment combinations** (when the schema timeline crosses -a reconfigure boundary, core §7.3) are different — segments may use -different sketch types or parameters, so the combined bound is the -**worst** of any per-segment bound, with confidence multiplied: - -``` -combined_bound = max(seg.bound for seg in segments) -combined_confidence = ∏(seg.confidence for seg in segments) -``` - -The `Provenance.segments_combined` field surfaces this so callers -know the answer is composite. For non-additive statistics (Quantile, -TopK) crossing a sketch-type boundary, the result is `Partial` with -the un-mergeable segments listed in `missing` — the user sees that -the answer covers only the segments where it could be computed, not -a silently-wrong combination. - ---- - -## 20. Sketch profiler library - -The accuracy bounds in §19.9 are mathematical guarantees, but the -**operational characteristics** (CPU, memory, latency, throughput) -of every sketch type are **measured**, not derived. Different -implementations of the same sketch family — sketchlib-rust vs -sketchlib-go, branch A vs branch B of either, different parameter -choices — have different real-world behaviour even when the -theoretical accuracy is identical. The control plane's cost model -(core §15.2 `/api/v1/db/cost_estimate`, core §10.6 incremental-vs- -refresh decision) needs real measurements to plan well. - -This section describes a separate library — the **Sketch Profiler** -— that is shared across the sketch DB, sketchlib-rust, sketchlib-go, -and the control plane. It is not part of the sketch DB itself; it is the -measurement substrate the sketch DB and the control plane both consume. - -### 20.1 What it measures - -For every `(sketch_type, parameters, sketchlib_version, hardware -profile)` tuple, the profiler collects: - -| Metric | Definition | Use | -|---|---|---| -| **CPU per insert** | nanoseconds per `update(value)` call | agent CPU budget; admission control quotas (roadmap §11.1 `max_write_qps`) | -| **CPU per merge** | nanoseconds per `merge_with(other)` call | compaction cost (roadmap §9), refresh cost (core §10.3), query merge cost (core §7.3) | -| **CPU per estimate** | nanoseconds per `query_statistic(stat)` call | query latency (§19.2) | -| **Memory per accumulator** | bytes resident, including allocator overhead | admission-control `max_bytes_in_memory`; storage-cost estimate | -| **Wire size, serialized** | bytes after `serialize_to_bytes` (proto / msgpack) | agent → backend bandwidth (§4 of DataCollector#153); backend → store | -| **Empirical accuracy** | measured `\|estimate − ground_truth\|` on synthetic and real workloads | validates §19.9 theoretical bounds; flags regressions if a sketch implementation drifts from theory | -| **Insert throughput** | samples/sec/core sustainable before backpressure | agent CPU sizing | -| **Merge throughput** | merges/sec/core | compaction sizing | -| **Cold-start cost** | first-insert latency (allocator + JIT warmup) | cold-query SLA | - -These are collected per sketch type, per parameter set, and per -target architecture (x86_64 vs arm64 vs the agent's actual CPU -model). The profiler stores results in a published catalogue that -the control plane reads at planning time and the operator inspects -when picking parameters for a new aggregation. - -### 20.2 How it runs - -The profiler is a standalone binary in its own crate -(tentatively `sketch-profiler/`). It supports three modes: - -- **Calibration** — runs all sketch types × a parameter grid against - synthetic distributions (uniform, zipf, normal, heavy-tailed) plus - a few real-world snapshots. Produces a baseline catalogue. Run - once per sketchlib release, or whenever a sketch implementation - changes. CI integration: a PR that touches sketchlib must include - a re-run that shows no significant regression. -- **Drift watch** — runs in production as a low-priority sidecar - task. Periodically samples a small subset of sketch types and - parameters, compares against the catalogue. Alerts if measured - CPU/memory/accuracy diverges by more than a threshold from the - catalogue value (catches sketchlib version mismatches, hardware - changes, allocator regressions). -- **What-if** — given a `(sketch_type, parameters, expected_qps, - expected_cardinality)` tuple, returns predicted CPU/memory/latency - numbers. The control plane calls this at plan time. It also takes a - query workload as input and returns the predicted `query_latency` - per statistic. - -### 20.3 Catalogue format - -```rust -struct ProfilerCatalogue { - /// Identity of the run that produced this catalogue. - sketchlib_versions: HashMap, // {Rust: "0.4.2", Go: "0.4.0"} - hardware: HardwareProfile, - measured_at: DateTime, - - /// One entry per (sketch_type, parameter set) tuple. - entries: Vec, -} - -struct ProfilerEntry { - sketch_type: SketchType, - parameters: HashMap, - - /// Operational characteristics - cpu_per_insert_ns: f64, - cpu_per_merge_ns: f64, - cpu_per_estimate_ns: HashMap, - memory_bytes: usize, - wire_size_bytes: WireSize, // {proto: u64, msgpack: u64, msgpack_delta: u64} - insert_throughput_per_core: f64, - merge_throughput_per_core: f64, - cold_start_us: f64, - - /// Empirical accuracy under various distributions; cross-checked - /// against the theoretical AccuracyProfile from §19.9. - measured_error: HashMap, - - /// Which workload shapes this entry was tested against. Used to - /// scope the validity of the measurement. - tested_workloads: Vec, -} -``` - -### 20.4 How the control plane uses it - -The control plane's planner (core §15.2) replaces hand-coded -constants and crude formulas with calls into the profiler: - -``` -Old: cost_model.estimate_size(CMS, width=1024, depth=5) → "12 KB" - ^ hand-coded constant - -New: profiler.lookup(CMS, width=1024, depth=5) - → ProfilerEntry { memory_bytes: 12_512, wire_size_bytes: …, - cpu_per_insert_ns: 47.3, … } -``` - -This makes cost-based plan selection actually correct: when a plan -chooses CMS over CountSketch for a frequency query, the choice -reflects measured costs on the target hardware, not extrapolated -big-O. - -The Pareto frontier endpoint (core §15.2 `/api/v1/db/cost_estimate`) -is fully driven by the profiler — every (parameter, accuracy, cost) -point on the frontier is a real measurement, and the recommended -parameter set is the one that minimises operator-weighted cost -under the user's accuracy SLA. - -### 20.5 How the sketch DB uses it - -- **Admission control quotas** (roadmap §11.1) are sized by reading - the profiler's `memory_bytes` and `insert_throughput_per_core` and - multiplying by the deployment's available headroom. -- **Compaction policy** (roadmap §9) uses `cpu_per_merge_ns` to decide - how many entries can be compacted per tick within the configured CPU - budget. -- **Backfill scheduling** (core §10.3) uses the profiler's insert/merge - throughput to estimate job duration before launching. - -### 20.6 Cross-repo placement and ownership - -The profiler is **shared infrastructure** because it has no value -unless it covers all sketch implementations the system uses: - -- **`sketch-profiler/` crate** — workspace member of ASAPQuery-backend. - Owns the catalogue format, the calibration / drift / what-if - drivers, the catalogue serializer. -- **sketchlib-rust** — exposes a `Bench` trait or similar so the - profiler can call `update / merge / estimate` uniformly across - sketch types. -- **sketchlib-go** — same story for Go-side measurements (matters - for agent-side sketching where the Go implementation runs). -- **DataCollector controller** — reads the published catalogue and - feeds it into the planner; does not run measurements itself. - -A published catalogue (e.g. JSON in the sketchlib release artifacts) -is the contract between sketchlib releases and the controller. When -sketchlib bumps a version, the catalogue updates, and the controller -picks up new cost numbers without code changes. - -### 20.7 Why this is a separate library, not part of the sketch DB - -Three reasons: - -1. **Scope**: the profiler measures sketches in isolation, not in the - context of the storage engine. It belongs alongside sketchlib, - not the sketch DB. -2. **Reuse**: the controller, the operator's parameter-tuning UI, - the sketchlib CI all need it; only one of those is the sketch DB. -3. **Cadence**: the catalogue updates on sketchlib releases (low - frequency); the sketch DB ships independently. Different release - cadences imply different repos / different versioning. - -### 20.8 Status - -This library does not exist yet. It is called out here because: - -- core §6.4 `AccuracyProfile.merge_propagation` and core §15 - `Provenance` need numbers that are most credibly produced by the - profiler, not hand-derived; -- roadmap §11.1 quotas, core §15.2 cost-estimate, and roadmap §16 - implementation phases all reference "what the profiler will - provide"; -- treating it as a separate concern with its own design surface - prevents this doc from sprawling further into measurement - infrastructure that doesn't belong here. - -A separate design doc (`design-sketch-profiler.md`) will follow. - - - - diff --git a/docs/design-sketch-db-pluggable.md b/docs/design-sketch-db-pluggable.md deleted file mode 100644 index 793c49bd..00000000 --- a/docs/design-sketch-db-pluggable.md +++ /dev/null @@ -1,457 +0,0 @@ -# Design: Sketch DB as a Pluggable Component - -> **Status: not adopted (historical).** This proposal to detach the -> sketch DB into a standalone `sketch-db` crate / network service was -> **not implemented**. The 2026-05 reorg instead kept the sketch DB as -> an in-tree module at `data_plane/src/storage_engines/sketch_db/` -> (`index/`, `data/`, `query/`, `lifecycle/`, `persistence/`, -> `backfill/`). The crate names and paths below describe the proposal -> as originally written and no longer match the tree; kept as a record -> of the alternative that was considered. - -Companion to the sketch DB design set — see -[`design-sketch-db.md`](./design-sketch-db.md) for the index and -[`design-sketch-db-core.md`](./design-sketch-db-core.md) for the -as-built contract. Those docs define *what* the sketch DB is; this -one defines *how it detaches* from -`asap-query-engine` so it can be consumed either as a **library crate** embedded -in another process or as a **standalone binary** (a sketch-store service) that -any component in the system can talk to over the network. - ---- - -## 1. Problem - -Today the sketch DB is spread across `asap-query-engine/src/stores/sketch_db/` -and reaches into the rest of the crate freely: - -- `data_model::{AggregateCore, PrecomputedOutput, KeyByLabelValues}` is in - `asap-query-engine`, not under `sketch_db`. -- Accumulator implementations live in `asap-query-engine/src/precompute_operators/`. -- `schema::SchemaRegistry` is constructed by `main.rs` and handed into both - the ingest path and the query engine. -- HTTP endpoints for `/api/v1/streaming-config`, backfill triggers, and - schema reads are registered on the query engine's Axum router. -- The `Store` trait (`stores/traits.rs`) uses `asap-query-engine` types in - its method signatures (`PrecomputedOutput`, `AggregateCore`, `KeyByLabelValues`). - -Consequences: - -1. **No way to embed the store elsewhere.** A downstream product (e.g. a - dedicated summary-serving tier, or a test harness, or a notebook) has to - pull in all of `asap-query-engine` — planner client, drivers, query - engine, the lot. -2. **No way to run it as its own process.** You can't scale ingest-write - throughput independently of query read throughput, and you can't put the - sketch store on a different fleet from the query engine. -3. **Blast radius on refactors.** A change inside `sketch_db/` can touch - types shared with query planning because the boundary isn't enforced. - -We want a single artifact — `sketch-db` — that any caller can consume in two -interchangeable shapes: - -- **Library mode**: `sketch-db` is a Rust crate you link into your process. - Zero network hops. Used by `asap-query-engine` today, by tests, by anyone - who wants an in-process store. -- **Binary mode**: `sketch-db-server` is a daemon that exposes the same API - over gRPC. Used when sketch storage needs to scale/deploy independently, - or when a non-Rust client wants to read/write sketches. - -Both shapes share **one** Rust API surface, one schema, one on-disk format. - -Scope is a refactor with no change to the sketch DB's semantics. The -design-sketch-db-core.md contract (agg_id immutability, write barrier, timeline -dispatch, accuracy profiles, backfill) is preserved verbatim. - ---- - -## 2. Goals / Non-Goals - -**Goals** -- Self-contained crate: `sketch-db` compiles without `asap-query-engine`. -- Stable public API: changes inside the crate don't ripple outward. -- Binary drop-in: the same crate + a thin `main.rs` + a gRPC adapter is a - runnable service. -- Zero semantics drift vs. today's in-process store. -- Incremental migration — no flag day. - -**Non-Goals** -- Replication, sharding, or multi-node consistency. Still single-node per - `design-simple-map-store-persistence.md`. -- Rewriting the accumulator algorithms (they already live in - `asap-common/sketch-core`). -- A new wire format for sketches (reuse the existing OTLP `SketchEnvelope`). -- A SQL/PromQL-level query interface at the sketch DB boundary — the sketch - DB remains a **storage engine**, not a query engine. PromQL stays in - `asap-query-engine`. - ---- - -## 3. Current Coupling Map - -What the boundary has to cut: - -| Piece | Today's location | Who owns it after split | -|---|---|---| -| `Store` trait | `asap-query-engine/src/stores/traits.rs` | `sketch-db` (public) | -| `SketchStore` + LSM | `asap-query-engine/src/stores/sketch_db/simple_map_store/` | `sketch-db` (public) | -| `SchemaRegistry`, `AggSchema` | `asap-query-engine/src/stores/sketch_db/schema.rs` | `sketch-db` (public) | -| Backfill types / worker / registry | `asap-query-engine/src/stores/sketch_db/backfill*.rs` | `sketch-db` (public) | -| `AggregateCore` trait | `asap-query-engine/src/data_model/` | `sketch-db` (public trait) | -| Concrete accumulators (HLL, KLL, DDSketch, CMS, …) | `asap-query-engine/src/precompute_operators/` | **new crate** `sketch-accumulators` (depends on `sketch-db` for the trait, on `sketch-core` for the algorithms) | -| `PrecomputedOutput`, `KeyByLabelValues` | `asap-query-engine/src/data_model/` | `sketch-db` (public; neutral names) | -| Streaming-config swap HTTP handler | `asap-query-engine/src/main.rs` | caller (but `SchemaRegistry::reconcile(...)` is the real entry point and stays in `sketch-db`) | -| Backfill HTTP trigger endpoints | `asap-query-engine` | callable via either crate-level API or gRPC | -| Ingest path (OTLP receivers, delta cache, series router, worker pool) | `asap-query-engine/src/drivers/`, `precompute_engine/` | **stays in `asap-query-engine`** — this is the *caller*, not the store | - -Key insight: **the sketch DB is not the ingest pipeline**. The ingest path -lives above the store and calls `insert_precomputed_output` when a window -closes. That call site is exactly where the library-vs-binary switch -happens. - ---- - -## 4. Two Deployment Shapes - -### 4.1 Library mode (default, matches today) - -``` -┌──────────────────────── asap-query-engine process ────────────────────────┐ -│ │ -│ ingest pipeline ──► SketchDb::insert(...) (Rust call, same proc) │ -│ query engine ──► SketchDb::query(...) (Rust call, same proc) │ -│ │ -│ sketch-db crate │ -│ (SchemaRegistry + Store + backfill + LSM) │ -└───────────────────────────────────────────────────────────────────────────┘ -``` - -Zero overhead, used for embedded deployments and tests. This is the shape -every call site uses today; after the refactor it keeps working, just -against a cleaner API. - -### 4.2 Binary mode - -``` -┌─ asap-query-engine ─┐ gRPC ┌─ sketch-db-server ─┐ -│ │ ◄───────────► │ │ -│ ingest pipeline │ insert/ │ SchemaRegistry │ -│ query engine │ query/ │ Store (LSM) │ -│ │ schema RPC │ Backfill workers │ -└─────────────────────┘ └────────────────────┘ - │ - local disk -``` - -The server is a thin wrapper: `main.rs` + a gRPC adapter that maps RPC -methods one-to-one onto the `SketchDb` façade. No business logic lives in -the server crate. - -The client side is a `SketchDbClient` that implements **the same public -trait as the in-process store**, so the ingest and query paths don't know -which mode they're in — it's a compile-time / config-time swap. - ---- - -## 5. Crate Layout - -Two new workspace members under a new top-level directory: - -``` -asap-common/ - sketch-db/ (new) - Cargo.toml - src/ - lib.rs ← re-exports public API - api.rs ← SketchDb trait + facade - schema/ ← moved from asap-query-engine - mod.rs - registry.rs - timeline.rs - accuracy.rs - store/ - mod.rs ← Store trait (today's traits.rs) - simple_map_store/ ← moved wholesale - global.rs - per_key.rs - persistence/ - backfill/ ← moved wholesale - mod.rs - job.rs - worker.rs - registry.rs - data_model.rs ← PrecomputedOutput, KeyByLabelValues, AggregateCore trait - metrics.rs - proto/ - sketch_db.proto ← gRPC service (only compiled under `grpc` feature) - - sketch-accumulators/ (new — concrete AggregateCore impls) - src/ - hll.rs - kll.rs - ddsketch.rs - count_min.rs - count_sketch.rs - hydra_kll.rs - set_aggregator.rs - sum_min_max.rs - factory.rs ← AggregationType → Box - -sketch-db-server/ (new — thin binary) - Cargo.toml - src/ - main.rs ← clap, tracing init, config load - service.rs ← tonic service impl; delegates to sketch_db::SketchDb -``` - -### Why split accumulators out? - -The `Store` and `SchemaRegistry` don't need to know *which* accumulators -exist — they only need `AggregateCore`. Keeping concrete implementations in -a sibling crate means the server binary can be built with a minimal set -(or an operator can swap in custom ones without forking the core). - -### Crate feature flags - -On `sketch-db`: -- `grpc` — compiles `proto/` via `tonic-build`, enables the - `SketchDbClient` gRPC client. Off by default (library users don't need - it). -- `persistence` — LSM persistence layer. On by default. - -Workspace dependency rules after the split: - -``` -sketch-core ──────────────────────► (no internal deps) -asap_types ──────────────────────► (no internal deps) -sketch-db ──► sketch-core, asap_types -sketch-accumulators ──► sketch-db, sketch-core -sketch-db-server ──► sketch-db (feat=grpc), sketch-accumulators -asap-query-engine ──► sketch-db, sketch-accumulators, ... -``` - -No cycles, and `asap-query-engine` shrinks. - ---- - -## 6. Public API (Rust) - -One façade type + the existing traits, re-exported from `sketch_db::prelude`: - -```rust -// sketch-db/src/api.rs -pub struct SketchDb { - schema: Arc, - store: Arc, - backfill: Arc, -} - -impl SketchDb { - /// Embedded construction — opens the on-disk state at `data_dir`. - pub fn open(cfg: SketchDbConfig) -> Result; - - // ── write path ───────────────────────────────────────────── - pub fn insert(&self, out: PrecomputedOutput, core: Box) - -> Result<(), SketchDbError>; - pub fn insert_batch(&self, items: Vec<(PrecomputedOutput, Box)>) - -> Result<(), SketchDbError>; - - // ── read path ────────────────────────────────────────────── - pub fn query(&self, metric: &str, agg_id: u64, t1: u64, t2: u64) - -> Result; - pub fn query_exact(&self, metric: &str, agg_id: u64, ws: u64, we: u64) - -> Result; - - // ── schema / lifecycle ───────────────────────────────────── - pub fn is_writable(&self, agg_id: u64) -> bool; - pub fn get_schema(&self, agg_id: u64) -> Option; - pub fn timeline_for_metric(&self, metric: &str, t1: u64, t2: u64) -> Vec; - pub fn reconcile(&self, cfg: &StreamingConfig) -> ReconcileSummary; - pub fn drop_agg_id(&self, agg_id: u64) -> Result; - - // ── backfill ─────────────────────────────────────────────── - pub fn submit_backfill(&self, job: BackfillJob) -> Result; - pub fn backfill_status(&self, id: BackfillJobId) -> Option; -} -``` - -This is the entire external surface. Everything in `asap-query-engine` that -talks to the store today goes through one of these methods; nothing -reaches into `schema::` or `store::` internals directly. - -### Error type - -Replace the current `Box` in the `Store` trait -with a concrete `SketchDbError` enum (unknown agg_id, schema barrier drop, -disk full, codec mismatch, …). Makes the binary-mode gRPC mapping clean -and gives callers actionable variants. - ---- - -## 7. Wire Protocol (Binary Mode) - -gRPC via `tonic`. Proto lives at `sketch-db/proto/sketch_db.proto`. One -service, methods that mirror the Rust API 1:1: - -```proto -service SketchDb { - // Write - rpc Insert(InsertRequest) returns (InsertResponse); - rpc InsertBatch(stream InsertRequest) returns (InsertBatchResponse); - - // Read - rpc Query(QueryRequest) returns (QueryResponse); - rpc QueryExact(QueryExactRequest) returns (QueryResponse); - - // Schema - rpc IsWritable(AggIdRequest) returns (BoolResponse); - rpc GetSchema(AggIdRequest) returns (AggSchemaResponse); - rpc TimelineForMetric(TimelineRequest) returns (TimelineResponse); - rpc Reconcile(StreamingConfigProto) returns (ReconcileSummaryProto); - rpc DropAggId(AggIdRequest) returns (DropResponse); - - // Backfill - rpc SubmitBackfill(BackfillJobProto) returns (BackfillJobIdProto); - rpc BackfillStatus(BackfillJobIdProto) returns (BackfillStatusProto); -} -``` - -### Sketch payload encoding - -Accumulators cross the wire as **the same `SketchEnvelope` oneof** the -ingest path already uses (see `asap-common/dependencies/rs/asap_otel_proto/`). -No new codec. `InsertRequest` carries `bytes sketch_envelope = N` plus the -typed aux columns (`count/sum/min/max`) as primitive fields. - -### Streaming - -`InsertBatch` is client-streaming so the ingest fleet can pipeline window -closes without one RTT per sketch. `Query` is unary — a typical response -fits one message. If a future query type returns large timelines, add a -server-streaming variant at that time. - -### Backpressure & retries - -- Server advertises a soft `max_inflight_writes` per connection. -- Writes are **idempotent on (agg_id, group_key, window_start)** — the - store already dedupes on that key, so clients can safely retry. -- Queries are naturally idempotent. - -### Auth & isolation - -Out of scope for v1; document the expectation that the binary runs behind -a trusted network boundary, same as today's query engine. Add mTLS as a -follow-up when someone needs it. - ---- - -## 8. Ingest & Query Path Changes - -The ingest and query paths don't care which mode the store runs in — they -hold an `Arc`. Today that's the in-process `SketchDb`. -In binary mode it's a `SketchDbClient` backed by gRPC. - -```rust -// sketch-db/src/api.rs -pub trait SketchDbHandle: Send + Sync { - fn insert(&self, ...) -> Result<(), SketchDbError>; - fn query(&self, ...) -> Result; - fn is_writable(&self, agg_id: u64) -> bool; - // ... same methods as SketchDb -} - -impl SketchDbHandle for SketchDb { /* direct */ } -impl SketchDbHandle for SketchDbClient { /* gRPC */ } -``` - -`asap-query-engine/src/main.rs` picks the impl from config: - -```rust -let handle: Arc = match cfg.sketch_db { - SketchDbTarget::Embedded(c) => Arc::new(SketchDb::open(c)?), - SketchDbTarget::Remote { endpoint } => Arc::new(SketchDbClient::connect(endpoint).await?), -}; -``` - -Nothing downstream changes. - -### Latency note - -Binary mode adds one network hop per window close on the write side and -one per query on the read side. Batching (`InsertBatch` stream) keeps the -write-side overhead amortized; query-side latency becomes noticeable only -for sub-second PromQL use cases and should be benchmarked before adoption. - ---- - -## 9. On-Disk Format & Compatibility - -Formats do not change: -- LSM parts (`part_{id}/{meta,data,index}.bin`) — unchanged. -- `SchemaRegistry` snapshot JSON — unchanged. -- Manifest append-only log — unchanged. - -So a directory written by today's in-process store opens cleanly under the -new `SketchDb::open(...)` in either deployment shape. The existing -`persist_format_versioning_tests.rs` coverage carries over. - ---- - -## 10. Migration Plan - -Six steps, each independently mergeable; the code compiles and all tests -pass between steps. - -| Step | What moves | Risk | -|---|---|---| -| **M1** | Create empty `sketch-db` crate; move `Store` trait, `PrecomputedOutput`, `KeyByLabelValues`, `AggregateCore` trait definition (not impls) into it. Re-export from `asap-query-engine` for source compat. | Low — mechanical. | -| **M2** | Move `stores/sketch_db/{schema, backfill*, simple_map_store, accuracy, metrics}` into `sketch-db`. Update imports. | Medium — large rename. | -| **M3** | Split concrete accumulators into `sketch-accumulators` crate. `asap-query-engine` depends on both. | Low — accumulators are already mostly self-contained under `precompute_operators/`. | -| **M4** | Introduce `SketchDb` façade + `SketchDbHandle` trait. Migrate call sites (`main.rs`, precompute engine, query engines, tests) to use the façade instead of reaching into the registry/store directly. | Medium — touches many files, but each edit is local. | -| **M5** | Add `grpc` feature, `proto/sketch_db.proto`, `tonic` codegen, `SketchDbClient`. Unit-test the client against an in-process server. | Medium — new dep surface. | -| **M6** | Create `sketch-db-server` binary crate. Wire through config in `asap-query-engine` for `Embedded` vs. `Remote`. Add an e2e test that runs ingest→server→query across a loopback gRPC link. | Medium — new deployment artifact. | - -At the end of M4 the library split is complete and shippable; M5 and M6 -are the binary-mode delta and can land later without blocking anything. - ---- - -## 11. Open Questions - -1. **Backfill scheduling across the boundary.** The control plane - (`asap-planner-rs`) triggers backfill jobs today via HTTP against the - query engine. Should that redirect through the new gRPC service in - binary mode (preferred, keeps state in one place), or should the - planner talk to the store directly? → proposed: planner talks to the - store; query engine proxies only if back-compat requires it. -2. **Metrics.** Prometheus `/metrics` endpoint — does `sketch-db-server` - expose its own, or does the query engine scrape both? → proposed: - server exposes its own; query engine's `/metrics` no longer reports - store internals in binary mode. -3. **Accumulator registration.** In binary mode, the server must know how - to deserialize every sketch type the ingest fleet sends. If a client - adds a new type (the `adding-a-new-sketch.md` flow), the server also - has to ship with the matching accumulator. → proposed: both build - against the same pinned `sketch-accumulators` version; bump in - lockstep. Revisit when anyone asks for hot plug-in accumulators. -4. **Config schema drift.** `StreamingConfig` is currently defined in - `asap-common/dependencies/rs/asap_types/`. That stays shared. If it - ever forks by deployment shape, we'll need versioned reconcile RPCs. - ---- - -## 12. Summary - -- The sketch DB today is semantically a separable component but - structurally embedded in `asap-query-engine`. -- Extract it into a `sketch-db` crate with one façade (`SketchDb`) and - one handle trait (`SketchDbHandle`). Concrete accumulators go into - a sibling `sketch-accumulators` crate. -- A thin `sketch-db-server` binary plus a `grpc` feature flag turns the - same crate into a standalone service; clients use `SketchDbClient` - that implements the same handle trait. -- Ingest and query paths swap between embedded and remote via one - config line. On-disk format, schema contract, and sketch semantics - are all unchanged. -- Migrate in six small steps; library extraction (M1–M4) is useful - even if the binary (M5–M6) never ships. diff --git a/docs/design-sketch-db-roadmap.md b/docs/design-sketch-db-roadmap.md deleted file mode 100644 index fbe3c30d..00000000 --- a/docs/design-sketch-db-roadmap.md +++ /dev/null @@ -1,581 +0,0 @@ -# Sketch DB — Roadmap (Not-Yet-Implemented Subsystems) - -> **Scope.** Design sections that are **intentionally ahead of the code** — -> they describe subsystems we expect to build but have not built yet, plus -> the migration/rollout thinking that applies when we do. -> -> **Status of sections in this file:** all ❌ not implemented or -> ⚠️ partial unless noted. Implemented behavior lives in -> [`design-sketch-db-core.md`](./design-sketch-db-core.md); the -> [index](./design-sketch-db.md) has the full status table. -> -> **Audience:** anyone planning future work, prioritizing, or deciding -> whether a proposal fits into the already-designed path. - -Section numbers are kept the same as the original monolithic -`design-sketch-db.md` so cross-references from other docs keep -pointing at the same §-numbers. - ---- - -## 9. Semantic compaction - -This section describes **Tier 2's** semantic compaction — the -batched, LSM-level merge process that runs on disk-backed parts. -Tier 1 (PromSketch) implements the *same concept* differently: -continuous in-memory EH bucket merging, done inline as windows age, -without an explicit level structure. Tier 1 is especially effective -when the same series is queried over many different sub-windows -(core §4.1) because the EH lets every such query reuse the same -in-memory structure. The rest of this section is Tier 2 specific. - -> **Status.** ❌ not implemented. The part format and the flusher exist -> (see [`design-simple-map-store-persistence.md`](./design-simple-map-store-persistence.md)), -> but there is no LSM-level structure, no semantic merge step, and no -> policy engine. Parts are flushed and age out via TTL; they are not -> compacted to coarser windows. Referenced only as comments in -> `stores/sketch_db/schema.rs`. - -### 9.1 Level = temporal resolution - -``` -Level 0: 10s windows (live ingest writes here) -Level 1: 1min windows (merge 6 L0 entries) -Level 2: 5min windows (merge 5 L1 entries) -Level 3: 1h windows (merge 12 L2 entries) -Level 4: 1d windows (merge 24 L3 entries) -``` - -Each compaction step takes N entries of the same `(agg_id, group_key)` -at level L and merges them into one entry at level L+1 covering a -bigger window. The merge call is the sketch type's `merge_with` trait -method. - -Query cost scales with level: "last 1h" reads a level-2 entry; "last -30 days" reads 30 level-4 entries. The coarsest level is bounded by -`retention`. - -### 9.2 Compaction policy per `AggStatus` - -```rust -fn compaction_policy(schema: &AggSchema) -> Policy { - match schema.status() { - Active => Policy::FullCompact, - Retired { expires_in: > 24h } => Policy::CompactUpTo(Level::L2), - Retired { expires_in: 1h..24h } => Policy::CompactUpTo(Level::L1), - Retired { expires_in: < 1h } => Policy::NoCompact, - Expired => Policy::Delete, - } -} -``` - -Rationale: compaction pays off long-term (less storage, faster -queries), but once an agg is within hours of expiry, compacting it -further is wasted CPU. We gracefully unwind compaction as retirement -approaches. - -### 9.3 No cross-`agg_id` compaction, ever - -This is a hard invariant. Different agg_ids have different schemas and -their sketches are not generally mergeable. Compaction always groups -entries by `(agg_id, group_key)` and merges only within that. - ---- - -## 11. Admission control and isolation - -The core doc treats the sketch DB as if memory, CPU, and I/O were -unbounded. Production deployments are the opposite — a noisy agg_id -must not be able to starve others. This section enumerates what has to -be in place for tier storage and backfill to be production-safe. - -> **Status.** ❌ not implemented. Today the write-side barrier -> (core §6.3) only checks schema status; there is no per-agg resource -> accounting. A misconfigured high-cardinality agg can currently -> exhaust memory. - -### 11.1 Per-`agg_id` quotas - -Each `AggSchema` carries quota limits that the store enforces at write -time: - -```rust -struct AggQuotas { - max_group_states: u64, // hard cap on distinct (agg_id, group_key) pairs - max_bytes_in_memory: u64, // hard cap on live sketch bytes (Tier 1 + Tier 2 memtable) - max_write_qps: u32, // write-rate limit (token bucket) -} -``` - -Today's `persistence_memory_limit_mb` is a **global** cap; the design -moves it to per-agg so one misconfigured high-cardinality agg can't -evict everyone else's hot windows. - -### 11.2 What happens when quotas are hit - -Policy is per-agg, set by the control plane: - -| Policy | Semantics | When to use | -|---|---|---| -| `Reject` | New group writes return error; existing groups continue | Exact-correctness metrics; control plane can react by upgrading quota or retiring the agg | -| `EvictLRU` | Least-recently-written group is dropped to make room | Dashboard metrics; losing old groups is acceptable | -| `Downsample` | Reduce sketch resolution in-place (e.g. halve CMS width) | When accuracy can degrade but coverage must continue | -| `Backpressure` | Return a retry-after signal to the ingest source | Coordinates with upstream — DataCollector can slow its emit cadence | - -The write barrier (`is_writable`) extends to check quota, not just -schema existence. - -### 11.3 High-cardinality GroupState idle eviction - -Even within a quota, stale `GroupState` shells accumulate. A -`group_by=[user_id]` agg might see a user once then never again; the -current code keeps that empty GroupState forever. - -Proposal: after every flush tick, evict `GroupState` entries whose -`previous_watermark_ms` is older than `idle_timeout` and whose pane -maps are empty. Next write to that `(agg_id, group_key)` recreates -a fresh state. - -```rust -fn evict_idle_groups(&mut self, now_ms: i64) { - let idle_cutoff = now_ms - self.idle_timeout_ms; - self.group_states.retain(|_, gs| { - !gs.active_panes.is_empty() - || !gs.sketch_panes.is_empty() - || gs.previous_watermark_ms > idle_cutoff - }); -} -``` - -This is cheap (one HashMap scan per flush tick) and orthogonal to -the `evict_orphaned_groups` introduced in PR #16. - -### 11.4 Live vs backfill isolation - -Core §10.3 covers separate worker pools. Additional requirements: - -- **CPU share**: live ingest has hard latency SLA; backfill is - latency-tolerant. Use OS priority or a cgroup per pool, not just - thread count. -- **Exact-DB rate limiting**: a backfill job that scans 24h of - Prometheus data can hit the Prom server hard enough to degrade - live ingest (if both use the same Prom). Backfill reader must - implement a configurable bytes/sec ceiling. -- **Backfill concurrency ceiling**: total concurrent backfill jobs - across all aggs, not just per-agg. 100 simultaneously-upgrading - metrics = 100 backfill jobs = potential write storm. - -### 11.5 Global reservations - -Some resources are inherently shared and need a global allocator: - -- **Memory budget** for Tier 2 memtable + cache: 80% for live ingest, - 20% reserved for backfill (tunable). Backfill blocked if its - share is full. -- **Disk budget** for Tier 2 parts: a high-water mark triggers - accelerated TTL sweep or refuses new writes. -- **Exact-DB read budget**: per-deployment cap on bytes-per-second - read from the exact DB across all backfill jobs. - -These are hard caps. The control plane treats them as signals for -planning (e.g. don't plan a backfill if the exact-DB read budget is -saturated by other ongoing jobs). - ---- - -## 12. Observability and debugging - -A storage engine without visibility is unusable in production. This -section enumerates what must be observable and through what surface. - -> **Status.** ⚠️ minimal. Only `SAMPLES_BLOCKED_BY_SCHEMA_BARRIER` -> exists today (`stores/sketch_db/metrics.rs`). The rest of this -> section is largely aspirational. - -### 12.1 Metrics (Prometheus endpoint) - -The sketch DB exposes a `/metrics` endpoint with per-agg_id labels: - -``` -sketch_db_writes_total{agg_id, tier, origin} -sketch_db_query_latency_seconds{agg_id, statistic} (histogram) -sketch_db_query_coverage_ratio{agg_id} (fraction served from sketch) -sketch_db_parts_total{agg_id, level} -sketch_db_bytes_stored{agg_id, tier} -sketch_db_group_states{agg_id, worker_id} -sketch_db_compaction_lag_seconds{agg_id} -sketch_db_backfill_duration_seconds{agg_id} (histogram) -sketch_db_backfill_bytes_read_total{agg_id, source} -sketch_db_quota_exceeded_total{agg_id, policy} -sketch_db_schema_transitions_total{from_status, to_status} -``` - -These feed both the Prometheus operator dashboard and the -control plane's `/api/v1/db/stats/*` endpoints (the control plane reads -aggregates across metrics; operators want per-instance detail). - -### 12.2 Distributed tracing - -A query from user → query engine → sketch DB → tier storage → -possibly exact-DB fallback crosses multiple components. Trace -context (W3C Trace Context) propagates on each hop: - -``` -Span: http_query (query engine entry) - └─ Span: find_schema_timeline - └─ Span: per_segment_dispatch - ├─ Span: tier1_query (PromSketch) - └─ Span: tier2_query (SketchStore) - ├─ Span: part_scan - └─ Span: sketch_merge - └─ Span: combine_statistic -``` - -OpenTelemetry exporter → any OTLP-compatible backend. The backend's -own ingest path is the natural target so the traces land alongside -the metrics. - -### 12.3 Structured audit log - -Lifecycle events are logged at INFO with structured fields: - -```json -{"event": "schema_create", "agg_id": 17, "metric": "latency", "sketch_type": "KLL", "params": {...}} -{"event": "schema_retire", "agg_id": 1, "retired_at": "...", "expires_at": "..."} -{"event": "config_swap", "added": [17], "removed": [1], "by": "control-plane-a"} -{"event": "backfill_start", "job_id": 5, "agg_id": 17, "time_range": [...], "source": "S3Gorilla"} -{"event": "backfill_complete", "job_id": 5, "windows_done": 8640, "duration_s": 47.2} -{"event": "quota_exceeded", "agg_id": 42, "policy": "EvictLRU", "evicted_group": "..."} -{"event": "orphan_eviction", "worker_id": 3, "agg_id": 1, "evicted_count": 1248} -``` - -This log is the forensic ground truth when something looks weird in -metrics. - -### 12.4 Debug APIs - -For on-call inspection: - -``` -GET /api/v1/db/debug/entry/{agg_id}/{group_key}/{window_start} - → raw SketchEntry (schema-validated) - -GET /api/v1/db/debug/coverage_map/{agg_id} - → full list of (range, Coverage) for this agg - (shows Native / Backfilled / BackfillInProgress / Missing) - -GET /api/v1/db/debug/parts/{agg_id} - → list of parts for this agg with (level, min_ts, max_ts, bytes) - -GET /api/v1/db/debug/backfill_diff - body: { agg_id, time_range } - → compare what's in the store vs what a refresh from the exact DB - would produce, without actually writing. Useful for catching - drift between incremental and refresh paths. -``` - -The last one is particularly valuable — if live and refresh disagree, -the diff tells you which segments and by how much. - -### 12.5 Query plan explain - -Similar to `EXPLAIN` in SQL: - -``` -POST /api/v1/query?explain=true - body: { metric, range, statistic } - → { - timeline_segments: [(agg_id, time_range, tier)], - coverage_per_segment: [...], - merge_plan: [...], - fallback_segments: [...], - estimated_latency_ms: ..., - } -``` - -Operators and the control plane use this to understand why a query -went where it went. - ---- - -## 13. Rollout and migration - -Every phase in §16 crosses a live deployment. This section lists -the invariants that let each phase roll out incrementally without -breaking existing users. - -> **Status.** ⚠️ partial. Part-format versioning is in place -> (`tests/persist_format_versioning_tests.rs`); a noop backfill -> reader factory provides a rough shadow-mode capability. A -> comprehensive feature-flag framework and full shadow-mode are not -> built. - -### 13.1 Feature flags - -Each phase gets a feature flag, off by default: - -```rust -struct SketchDbFeatures { - typed_aux_columns_enabled: bool, // Phase 1 - schema_barrier_enabled: bool, // Phase 2 - metric_routing_enabled: bool, // Phase 3 - semantic_compaction_enabled: bool, // Phase 4 - backfill_enabled: bool, // Phase 5 - metadata_apis_enabled: bool, // Phase 6 - postings_index_enabled: bool, // Phase 7 -} -``` - -Flipping a flag takes effect on the next write or query; no restart. -A flag-off rollback is always safe — the old code paths stay compiled -in and active when the corresponding flag is disabled. - -### 13.2 Shadow mode - -For Phases 1, 3, and 5 specifically, the new path can run in shadow: -compute the new answer alongside the old, diff the two, log -discrepancies, but return the old answer. Runs in production with -zero risk until enough confidence to flip the flag. - -```rust -let old = old_path.query_range(...); -if features.shadow_mode_enabled { - let new = new_path.query_range(...); - metrics.shadow_diff(old, new); // histogram of abs error -} -old -``` - -Shadow mode is also how we **validate Phase 5's deterministic -rebuild claim**: continuously compare a backfill output against live -incremental output for the same metric and assert the sketches are -within-accuracy equivalent. - -### 13.3 Part format versioning - -Phases 1, 2, 4, 7 change the on-disk part format. Each introduces a -`format_version: u16` field in the part header: - -``` -v1: count/sum/min/max as inline aux columns (Phase 1) -v2: + schema metadata + origin tag (Phase 2) -v3: + level metadata (Phase 4) -v4: + label postings section (Phase 7) -``` - -Readers dispatch on version. For forward compat: unknown fields in -a future version's part are skipped (protobuf-style). For backward -compat: at least two consecutive major versions are readable; older -parts age out via normal TTL. - -### 13.4 Migration of existing deployments - -**Schema bootstrap at Phase 2 rollout**: existing deployments don't -have `AggSchema` records. First startup after the upgrade: - -1. Read current `StreamingConfig`. -2. For each `agg_id` in config, create an `AggSchema` with - `created_at = now`, `status = Active`. -3. Persist to the schema store. - -**Historical part interpretation**: old parts don't have origin tags. -Treat missing origin as `Native` (conservative — old data was never -backfilled). - -**Coverage reconstruction** (at Phase 5 rollout): on startup, scan -existing parts per agg to populate the initial Coverage map. Slower -startup but one-time. - -### 13.5 Rollback strategy - -Each phase must be independently rollback-safe: - -- **Forward flag off → backward compat** is the baseline: the new - code doesn't run, old code still works. -- **Backward compat of persisted state**: if Phase 2 created schema - records and Phase 2 then gets rolled back, the schema records - remain on disk but are ignored. They don't corrupt the old path. -- **Phased-format parts stay readable** after rollback: a v2 part - written while Phase 2 was on is still readable by v1-only code - (the v2-specific fields are skipped). - ---- - -## 16. Implementation phases - -This design is intentionally larger than one PR. Suggested rollout: - -**Phase 1 — Typed aux columns + query pushdown (small, high-leverage).** -Add `count` / `sum` / `min` / `max` as typed columns on each -`SketchEntry`. Implement `query_range_statistic` that scans the index -and computes these scalars without deserializing sketch bytes. No -schema timeline yet, no backfill yet. - -**Phase 2 — Per-`agg_id` schema + write barrier.** ✅ done. -Introduce `AggSchema` with Active/Retired/Expired states. Wire the -ArcSwap config-swap handler to create/retire schemas. Add -`is_writable(agg_id)` barrier on the write path. - -**Phase 3 — Schema timeline + metric-based query dispatch.** ✅ done. -Build the `metric_timelines` index. Change the query engine's primary -entry point from "by agg_id" to "by metric." Implement per-segment -dispatch with `combine_statistic` for combinable cases. - -**Phase 4 — Semantic compaction.** ❌ not started. -Add level-aware compaction with sketch-type merge dispatch. Policy -table per `AggStatus`. - -**Phase 5 — Backfill service.** ⚠️ most sub-phases landed (5a–5d); -5e (real rebuild logic) is scaffolded; 5f (coverage integration with -the query path) is pending. -Independent worker pool. `write_backfilled_window` bypass of -WindowManager. `Coverage` tracking. Control-plane-facing `/backfill` -endpoints. - -**Phase 6 — Controller-facing metadata APIs.** ⚠️ partial. -`/stats`, `/timeline`, `/cost_estimate`, `/pressure`. This is what -lets the control plane's planner use real observations. Today only -`/schemas` and `/timeline` are implemented. - -**Phase 7 — Secondary indexes.** ❌ not started. -Label postings for fast filter pushdown. Roaring bitmaps per -`(agg_id, label_name, label_value)`. - -Phase 1 is largely orthogonal to the others and can ship as an -isolated improvement. Phases 2–3 are the conceptual core — everything -else builds on them. Phases 4–7 are optimizations. - ---- - -## 17. Relationship to the hot-reload PR (PR #16) - -PR #16 gets the runtime-reload machinery right: `ArcSwap` is shared, -all consumers read the current config at the same instant, orphaned -`GroupState` entries are evicted after their windows drain. It -establishes the monotonic-`aggregation_id` + time-TTL contract that -this design assumes. - -PR #16 does **not** implement any of the sketch DB architecture -described in the core doc. In PR #16: - -- There is no `AggSchema` metadata object; the store relies on - `AggregationConfig` being in the current `StreamingConfig` plus - time-based TTL to handle retirement. -- There is no schema timeline; queries that span a reconfigure - boundary see a data cliff at the swap point. This is the known - query-continuity gap that motivated Phase 3 / Phase 5 of this - design. -- There is no backfill; once a new agg_id is created, its historical - coverage grows from zero in real time. -- The storage layer still treats everything by `(agg_id, window, - group_key)` — no sketch-aware compaction, no typed aux columns, no - label postings. - -So PR #16 was the **foundation** — hot-reload is a hard prerequisite -for everything above — but the sketch DB design is a much larger -program of work that continues to be delivered phase by phase. - ---- - -## 18. Open questions - -1. **Where do backfill-source hash seeds live?** To make Backfilled - sketches byte-identical to what live ingest would have produced, - the hash function seeds need to be reproducible. Proposal: include - them in `AggregationConfig.parameters`. Requires a small - sketchlib-go / sketchlib-rust change to accept an external seed. - -2. **What does the control plane do when a backfill fails partway - through?** Proposal: job_id is idempotent; control plane retries - with exponential backoff; if persistent failure, proceed without - the backfilled range (query engine falls back for that subrange). - -3. **How much exact-DB retention is needed?** Exact-DB retention - must be ≥ the longest `backfill horizon` the control plane ever - requests, which is the longest query range users will issue that - spans a reconfigure. If exact-DB retention is 7 days and queries - never look back more than 24h, that's fine. Needs to be tracked - as a deployment-level configuration — and the control plane should - reject any upgrade plan whose backfill horizon exceeds current - exact-DB retention. - -4. **Can an in-progress backfill be query-visible with partial - coverage?** Proposal: yes — `Coverage::BackfillInProgress` is a - real state. The query engine can wait (if ETA is short), combine - partial sketch with partial fallback, or serve from fallback only. - Trade-off knob on the query level. - -5. **Does the current `SketchStore` schema support all of Phase 1 - without disk migration?** Mostly. `count`/`sum`/`min`/`max` can be - written as separate columns in the existing part format; readers - that don't know about them can skip. The label posting index - (Phase 7) needs a new on-disk structure and will require a new - part version. - -6. **Relationship to PromSketch?** *(Resolved — see core §4.1.)* - PromSketch is **Tier 1** of the sketch DB, not a parallel system. - It's an in-memory EH-backed storage backend specialized for - short-retention sub-millisecond queries. Tier 2 is the - precompute + LSM parts store for longer retention. The two share - one schema lifecycle, one control plane API surface, one refresh - path, and one query-engine routing layer. The control plane picks - per-`agg_id` whether to materialize into Tier 1, Tier 2, or both. - -7. **Agent clock skew.** `time_unix_nano` on every sample is - agent-local. Agents with skewed clocks produce windows that don't - align across agents; cross-agent merge silently blends samples - that fall into different "real" time buckets. Worse, skew is - frozen into both live sketches and exact-DB data, so even refresh - doesn't fix it. Open questions: should the backend reject or - correct samples with timestamps too far from wall clock? Should - skew be surfaced as per-agent metadata the control plane can see? - Probably a hard NTP-sync requirement on agents with metrics - exposing skew per agent. - -8. **Control plane state and HA.** The monotonic `aggregation_id` - counter has to live somewhere that survives control plane restarts. - Options: control plane persists its own state (requires a DB for - the control plane); control plane reads `max(agg_id)` from backend's - `/api/v1/db/schemas` at startup (simple but needs CAS for - multi-control-plane-replica HA to avoid two control planes allocating - the same id). Also: who wins if two control planes disagree on - what the current `StreamingConfig` should be? Leader election - or backend-side CAS is needed before HA is viable. This is out - of scope for the sketch DB itself but affects the design - contract. - -9. **Timezone semantics for window boundaries.** All internal code - uses Unix epoch. User PromQL queries like - `quantile_over_time(m[1d]) by (day)` have an implicit "what is - a day?" — strict 24h vs calendar day with DST transitions. The - invariant in this design: **windows are Unix-epoch fixed - intervals**, no calendar alignment. If a UI needs calendar-aware - bucketing, it does so at the query-engine layer by aligning - query start times to local midnight. - -10. **PII / information leak via sketches and metadata.** CMS + - heap implicitly exposes high-frequency label values (e.g. - topk users). HLL cardinality is a sensitive metric in - privacy-regulated contexts. The - `/api/v1/db/debug/entry/*` APIs return raw sketch bytes that - a skilled user might mine. Open: do we need per-metric PII - tags that disable some debug APIs or redact label values in - TopK output? Probably deferred until multi-tenancy is needed. - -11. **Multi-tenancy.** The design assumes a single trust domain - (one operator, one set of metrics). In a shared deployment: - per-tenant quotas, per-tenant access control on - `/api/v1/db/*` endpoints, per-tenant archive buckets, per- - tenant billing. All doable but substantial work. Keeping - single-tenant is a reasonable v1; the design should not - preclude later multi-tenancy (specifically, `AggSchema` - should be extensible with a `tenant_id` field without a - format migration). - -12. **Replication and cross-region HA.** Currently the sketch DB - is single-writer. Read replicas (hot standby that ingests - the WAL of new parts) are straightforward. Multi-writer - (active-active across regions) is not — incremental MV - maintenance with distributed writers requires consensus on - window boundaries, which is a separate design effort. For - now: single-writer, use a DR replica as fallback. diff --git a/docs/design-sketch-db.md b/docs/design-sketch-db.md deleted file mode 100644 index be6d4334..00000000 --- a/docs/design-sketch-db.md +++ /dev/null @@ -1,145 +0,0 @@ -# Sketch DB — Design Index - -Entry point for the sketch DB design. Replaces the earlier monolithic -draft; the content lives in four companion docs so reviewers can load -only the parts relevant to them. - -**Status** (2026-04): Phases 2, 3, 5 of the roadmap have landed; Phase 4 -(semantic compaction), most of Phase 6 (metadata APIs), and Phase 7 -(label postings) are pending. See the [Status table](#status) below. - ---- - -## TL;DR - -The sketch DB is a **materialized-view storage engine over streaming -metrics**. Each view — identified by a monotonic `agg_id` — pins one -aggregation definition (sketch type + parameters + grouping labels + -window size) for its lifetime and is maintained by two complementary -paths: - -- **Incremental** (live ingest): window closes emit one row per - `(agg_id, group_key, window)` with `origin: Native`. -- **Refresh** (backfill): when a reconfigure creates a new `agg_id`, a - backfill job reads raw samples from the exact DB and fills in history - with `origin: Backfilled{job_id}`. - -Queries dispatch **by metric**, not by `agg_id`. The schema timeline -turns a `(metric, time-range)` request into one or more per-segment -sub-queries, each against a single agg. Every result carries an error -bound derived from the sketch type + parameters. - -Six invariants worth remembering: - -1. `agg_id` is immutable — reconfigure always mints a new id. -2. Writes are gated by a lifecycle barrier (`is_writable(agg_id)`). -3. Sealed windows are immutable; flush is async. -4. Typed aux columns (`count/sum/min/max`) stay exact even when the - sketch is approximate. -5. Queries that span reconfigures always go through - `timeline_for_metric`, never raw store scans. -6. Every query result carries accuracy metadata. - ---- - -## Companion docs - -| Doc | What's in it | When to read | -|---|---|---| -| [`design-sketch-db-core.md`](./design-sketch-db-core.md) | §1–8, §10, §14, §15 — the **as-built contract**: motivation, MV framing, principles, architecture, schema, per-agg lifecycle, schema timeline, incremental ingest, backfill, reconfigure workflow, API surface | Reviewing code or writing PRs that touch the sketch DB today | -| [`design-sketch-db-roadmap.md`](./design-sketch-db-roadmap.md) | §9 compaction, §11 admission control, §12 observability, §13 rollout, §16 phase plan, §17 hot-reload relationship, §18 open questions — **not-yet-implemented** subsystems | Planning future work or prioritizing | -| [`design-sketch-db-performance.md`](./design-sketch-db-performance.md) | §19 performance envelope vs Prom/VM + §20 Sketch Profiler library spec | Evaluating adoption or tuning parameters | -| [`design-sketch-db-pluggable.md`](./design-sketch-db-pluggable.md) | How to extract the sketch DB into its own library crate and optional gRPC binary | Planning the library/server split | -| [`design-simple-map-store-persistence.md`](./design-simple-map-store-persistence.md) | LSM-style parts-based persistence layer the sketch DB is built on | Touching the on-disk format | -| [`adding-a-new-sketch.md`](./adding-a-new-sketch.md) | Cross-repo recipe (sketchlib + DataCollector + backend) for extending the supported sketch list | Adding HLL/KLL/CMS/… variants | - ---- - -## Section map (where each § lives) - -Top-level section numbers are **kept stable across all sketch-DB docs** -so cross-references like "see §6.3 for the write barrier" remain valid -regardless of which file a reader is in. - -| § | Title | File | -|---|---|---| -| 1 | Motivation | core | -| 2 | Sketches as materialized views | core | -| 3 | Design principles | core | -| 4 | Architecture (incl. storage tiers) | core | -| 5 | Storage schema | core | -| 6 | Per-`agg_id` schema and its lifetime | core | -| 7 | Schema timeline | core | -| 8 | Incremental view maintenance (live ingest) | core | -| 9 | Semantic compaction | **roadmap** | -| 10 | Refreshable view maintenance (backfill) | core | -| 11 | Admission control and isolation | **roadmap** | -| 12 | Observability and debugging | **roadmap** | -| 13 | Rollout and migration | **roadmap** | -| 14 | Full reconfigure workflow | core | -| 15 | API surface | core | -| 16 | Implementation phases | **roadmap** | -| 17 | Relationship to hot-reload PR #16 | **roadmap** | -| 18 | Open questions | **roadmap** | -| 19 | Performance envelope (incl. §19.9 accuracy bounds, §19.10 merge propagation) | **performance** | -| 20 | Sketch profiler library | **performance** | -| 21 | Related approaches: wavelets and ML models as materialized views | core | - ---- - -## Status - -Summary of which design sections are actually implemented. "✅ done" -means the code matches the spec; "⚠️ partial" means the contract is -partially wired; "❌ not started" means spec only. - -| § | Claim | Status | Evidence | -|---|---|---|---| -| 5.1 | `SketchEntry` with typed aux columns (count/sum/min/max) as first-class fields | ⚠️ partial | `PrecomputedOutput` carries origin but aux scalars live in the accumulator, not as record columns | -| 5.2 | Primary key `(agg_id, window_start, group_key)` | ✅ | `SketchStore` per-agg bucketing + per-key | -| 5.2 | Secondary index `(agg_id, group_key, window_start)` | ❌ | | -| 5.3 | Label posting index (Roaring bitmaps) | ❌ | only label interning exists | -| 6.1–6.2 | `AggSchema`, `AggStatus{Active,Retired,Expired}` | ✅ | `storage_engines/sketch_db/schema.rs` | -| 6.3 | Write-side schema barrier `is_writable(agg_id)` | ✅ | called in `ingest_handler.rs` | -| 6.4 | `AccuracyProfile` on schema | ✅ | `storage_engines/sketch_db/accuracy.rs` | -| 7.2 | `timeline_for_metric(...)` | ✅ | `schema.rs:594` | -| 7.3 | Cross-schema query combiner | ✅ | `storage_engines/sketch_db/query/timeline_dispatch.rs` | -| 8 | Incremental ingest (OTLP / Prometheus / VictoriaMetrics / Kafka drivers) | ✅ | `drivers/ingest/` | -| 8.4 | Watermark + lateness policy | ✅ | `allowed_lateness_ms` + `LateSampleHandlingPolicy` | -| 9 | Semantic compaction (LSM levels) | ❌ | comment-level only | -| 10.2 | Backfill job/source/status types + registry | ✅ | `storage_engines/sketch_db/backfill.rs` | -| 10.3 | Separate backfill worker pool | ✅ | `backfill_service.rs`, `backfill_worker.rs` | -| 10.4 | Coverage tracking | ⚠️ partial | `Coverage` enum exists; not yet wired to query path (Phase 5f) | -| 10.5 | Deterministic rebuild | ⚠️ partial | processor scaffolded; end-to-end determinism pending | -| 10 | HTTP backfill trigger + list | ✅ | `/api/v1/db/backfill`, `/api/v1/db/backfill/jobs` | -| 11 | Per-agg quotas, idle eviction, live vs backfill isolation | ❌ | | -| 12.1 | Prometheus metrics suite | ⚠️ minimal | only `SAMPLES_BLOCKED_BY_SCHEMA_BARRIER` | -| 12.3–12.5 | Audit log, debug APIs, query EXPLAIN | ❌ | | -| 13.3 | Part format versioning | ✅ | `persist_format_versioning_tests.rs` | -| 13.1–13.2 | Feature flags + shadow mode | ⚠️ partial | no framework; noop backfill reader is a de-facto shadow | -| 14 | Reconfigure workflow | ✅ | end-to-end flow works via hot-reload + backfill | -| 15.1 | Store trait + query-engine API | ✅ | `stores/traits.rs` | -| 15.2 | Control plane `/schemas`, `/timeline`, `/backfill`, `/backfill/jobs`, streaming-config swap, retire, expire | ✅ | `drivers/query/servers/http.rs` | -| 15.2 | `/stats`, `/cost_estimate`, `/pressure` | ❌ | planner uses hand-coded estimates | -| 20 | Sketch profiler library | ❌ | not built | - -Biggest gaps, ranked by leverage: semantic compaction (§9), admission -control (§11), observability detail (§12), typed aux columns at the -storage layer (§5.1), label posting index (§5.3), sketch profiler (§20). - ---- - -## Open questions for reviewers - -The most valuable review comments land on these, collected in roadmap -§18: - -1. Where do backfill-source hash seeds live? (determinism contract) -2. Control plane behavior when backfill fails partway through -3. Exact-DB retention vs. maximum backfill horizon -4. Partial-coverage query semantics during in-progress backfill -5. PII / information leakage via debug APIs and TopK sketches -6. Multi-tenancy posture for v1 vs later -7. Single-writer → multi-writer replication - -Pointers are in [`design-sketch-db-roadmap.md`](./design-sketch-db-roadmap.md#18-open-questions). diff --git a/docs/design/holistic-edge-backend-compression.md b/docs/design/holistic-edge-backend-compression.md deleted file mode 100644 index 3a78c69e..00000000 --- a/docs/design/holistic-edge-backend-compression.md +++ /dev/null @@ -1,570 +0,0 @@ -# ASAP holistic edge→backend compression design - -Status: DRAFT for review. Informed by the offline micro-benchmark in this dir -(`./compressbench -dir data_serf` on the Chimp/Serf real datasets) and the -Gorilla / VictoriaMetrics / Serf literature. - -## 0. Goals & principles - -One pass over raw data at the edge feeds BOTH the cold (raw archive) and the -warm (sketch/aggregation) processors (existing parse-once framework). We add a -shared per-series **offset / frame-of-reference** so the numbers each consumer -actually stores are small, then bit-pack — minimizing **bits, bandwidth, -memory, CPU**. - -Unifying principle (the "offset / common-bits" idea, generalized): -> Find the common base in each structure's natural integer representation, -> subtract it, store small residuals bit-packed, exploit sparsity, and -> re-base the frame when it drifts. - -Hard requirements: -- **Cold = lossless** (raw archive). Warm sketches keep their existing - approximation guarantees; the encoding adds no extra error. -- **Backend ingests edge-compressed chunks WITHOUT decode+reinsert.** Decode is - pushed to the (rare) READ path: a custom Thanos StoreAPI decodes chunks to - XOR `AggrChunk`s at query time, so a stock PromQL engine queries them while - S3 holds the compact custom format. Writes are far more frequent than cold - reads, so this is where decode belongs. - -Benchmark headline (real Chimp/Serf datasets, lossless, block-avg 1000/chunk): -- Fixed-decimal series (11/12 datasets — temps, stocks, sensors, pressure, - GPS, dust, wind, grid): integer FOR+delta beats Gorilla. This *is* the offset - idea, on the integer-scaled values. **NOTE on the magnitude:** the - VictoriaMetrics `lib/encoding` number (~4.8× avg) included zstd-wrapping on - some series; since we decided **no zstd** (§5), the realized win is the - **no-zstd best-of-N codec measured in PR #434 (`asap-gorilla-go/intchunk`): - ~2.33× aggregate on fixed-decimal** (up to ~3.6× per series — Wind 3.6×, - Dew-point 2.9×, City-temp 2.8×, Stocks 2.7×). The codec tries fixed-width AND - zigzag-varint residuals (×{delta, delta-of-delta}) plus Gorilla and keeps the - smallest; varint helps skewed blocks but only nudged the aggregate (2.22→2.33×). - **Bottom line: no-zstd cold compression caps ~2.3×; the 4.8× genuinely needs - zstd, which we excluded.** Gorilla-XOR remains the lossless fallback for true - high-precision floats. -- Genuinely high-precision float (float32-derived, 15 sig digits): VM can't - stay decimal-exact → falls back to bit-pattern (worse); **Gorilla-XOR wins**. -- ⇒ The codec must be a per-block **best-of-N including Gorilla-XOR**, not - "VM replaces Gorilla". -- Decode CPU: VM ~41 ns/sample vs Gorilla ~71 — VM decode is *faster*, good - for the decode-on-read path. - ---- - -## 1. Cold raw chunk format - -### 1.1 Part (one S3 object per tenant / 2h-block / shard) -``` -[part header] magic "ASAPCC1" | u8 version | i64 block_start_ms | i64 block_end_ms | uvarint series_count -[chunks] per-series chunks, concatenated -[index] sorted by series: { labels(symbol refs), u64 chunk_off, u32 chunk_len, i64 min_ts, i64 max_ts } -[symbol table] deduped label strings (the index references offsets here) -[footer] u64 index_off | u64 index_len | u64 symtab_off | u32 crc32c -``` -The index + symbol table let the StoreAPI answer `Series(matchers, mint, maxt)` -without scanning chunk bodies. - -### 1.2 Per-series chunk -``` -[chunk header] - u8 codec_tag # see 1.3 - uvarint n_samples - ts: i64 t0_delta(block-relative) then delta-of-delta varints # timestamps - # value codec params (codec-specific): - tag INT_FOR_DELTA / INT_FOR_DOD: - i8 scale_exp # decimal exponent e: int_v = round(v * 10^-e); v = int_v * 10^e - zigzag-varint base # the FOR reference (frame base; see drift, 1.4) - zigzag-varint first_residual - tag GORILLA_XOR: (no extra params; standard XOR stream) -[chunk body] - bit-packed residual stream (INT_*), or XOR stream (GORILLA) -``` - -### 1.3 codec_tag -``` -0 = GORILLA_XOR lossless float64 (fallback for high-precision floats) -1 = INT_FOR_DELTA scale→int64, FOR(base), delta, bit-pack (gauges) -2 = INT_FOR_DOD scale→int64, FOR(base), delta-of-delta, bit-pack (counters / timestamps) -All three are LOSSLESS. No lossy (Serf) and no zstd-wrapping — kept deliberately -simple: the INT_* FOR+delta already captures the win on fixed-decimal data, and -Gorilla-XOR is the lossless fallback for true high-precision floats. -``` - -### 1.4 Encoder: per-series-per-block best-of-N with exactness check -``` -fn encode_chunk(values, opts): - cands = [] - # INT path — ONLY if it round-trips EXACTLY (the lib/decimal precision trap) - (ok, e, ints) = try_scale_to_int64(values) # find decimal exp e s.t. round-trip exact - if ok: - cands += encode_int(ints, FOR_DELTA) # tag 1 - cands += encode_int(ints, FOR_DOD) # tag 2 - # Gorilla is always valid + lossless - cands += encode_gorilla_xor(values) # tag 0 - return argmin(cands, key=byte_len) # smallest — all candidates lossless -``` -`try_scale_to_int64` is the load-bearing guard: never ship INT_* unless decode -reproduces the original float64 bit-exactly (else a naive VM-decimal silently -introduces ~1e-12 error — confirmed on Motor-temp in the benchmark). - -`base` re-bases on **drift** within a block: if a residual would overflow the -chosen bit-width, cut the chunk early and start a new chunk with a fresh base -(the cold analogue of "offset drift → new Full"; see §3). - -### 1.5 Backend write path (NO decode) -On ingest, validate the part header/crc, store the object to S3, and register -its series→part entries in the manifest/index. No decode, no re-encode. - -### 1.6 Decode-on-read StoreAPI (the compat shim) -Implements the Thanos `storepb.StoreServer`: -``` -Series(req {matchers, min_t, max_t}) -> stream of SeriesResponse: - for part in manifest.parts_overlapping(min_t, max_t): - for series in part.index.matching(req.matchers): - for chunk in series.chunks_overlapping(min_t, max_t): - samples = decode(chunk) # by codec_tag; INT_* adds base + scale - emit AggrChunk{ raw: xor_encode(samples) } # hand PromQL a standard XOR chunk -``` -Thanos-query unions this with the >=2h store-gateway path, exactly as the -current gorilla-merger StoreAPI does — we extend that StoreAPI's `Series()`. - -### 1.7 Grouped layout — shared timestamp column (cross-series) - -Measured (`/mydata/xseries-bench`, real cluster groups @2h/15s k=50 + synthetic). -Two cross-series levers; only one pays. - -- **Shared timestamp column — ADOPT.** Same-metric series in a part share ONE - timestamp column instead of every chunk carrying its own. Because INT_FOR - values compress so well, timestamps are **36–52% of per-series bytes** on real - groups; sharing the column across a k-series group removes ≈ ts_frac·(k−1)/k → - **−43% on the real cluster aggregate**, correlation-INdependent, low-risk. - (The Heracles VLDB'21 result, confirmed here.) Part layout becomes: - per-metric group = `{ one shared ts column (delta-of-delta) }` + `{ per-series - value chunks: codec_tag + residuals, NO ts }`. The decode-on-read StoreAPI - zips the shared ts with each series' values. Warm sketch parts get the same - win (same-window sketch series share the window-end column). -- **Cross-series value base — do NOT adopt by default.** Per-timestamp base - `b(t)=min` + per-series value residuals. Measured **net-NEGATIVE (−3.5% vs - shared-ts aggregate)**: it only wins for near-identical-replica series, and - the predictor is the **noise/signal ratio, NOT correlation** (even ρ=0.99 - lost +2.6% when noisy) — per-series delta-of-delta already extracted the - per-series common bits, so a noisy cross-series base just adds entropy. Trap: - must be done in the integer domain (naive float subtraction breaks decimal - representability, ~2× bloat). Optional per-group cost-based opt-in only. - -Takeaway: the remaining cross-series "common bits" worth taking are the -**timestamps** (shared column, −43%), not the values (already extracted -per-series). - ---- - -## 2. Warm sketch encoding - -Same offset idea, applied at each sketch's natural representation. The offset -is a property of the **Full-snapshot epoch** (see §3): a Full carries the -re-based offset/scale in its header; all Deltas until the next Full encode -residuals in that frame; backend reconstructs absolute values at merge/query. - -| family | offset on raw values? | encoding | -|---|---|---| -| SUM / COUNT | yes (linear) | ship `Σresidual` (narrow int, varint) + N + epoch offset; backend `Σresidual + (ΣN)·offset`. Counter sums: ship per-window increment (delta), no fixed offset. | -| KLL (quantile) | yes (shift-equivariant: `q(X−c)=q(X)−c`) | store sampled values as `(v−offset)` fixed-point (i16/i32 vs f64 ⇒ ~½ size); within a level the sorted samples delta-encode; backend adds offset to the quantile result. | -| DDSketch (current family) | NO (log-scale; `v−c`→0 breaks relative error) | FOR+delta on the **bucket-index** array (`min_index` + Δindex varints) + varint counts. | -| HLL (cardinality) | NO (hash) | **sparse mode** (low card: sorted non-zero registers, delta+varint = HLL++) + 6-bit dense packing. | -| CMS / CountSketch (freq/topk) | NO (hash) | narrow counters + per-row FOR + bit-pack; cross-window delta of the matrix; topk heap shipped as k entries. | - -Cross-cutting warm levers: -- **Delta transmission** (already in place: ProtoFull/ProtoDelta + the - delta-stitching carry-in): ship only changed sketch state between Fulls. -- **Offset rides in the Full header only**; Deltas carry residuals only. -- All Deltas in a Full-epoch share one offset frame ⇒ they remain mergeable - (merging `(v−off₁)` and `(v−off₂)` residual-KLLs would be garbage). - -### 2.1 Audit of the current serialization (measured) - -The backend does NOT re-serialize: `asap_sketchlib` and `sketchlib-go` share one -cross-language wire format, and the backend stores the wire bytes **opaquely** -(`SketchSampleState{bytes, encoding}`; flushed parts write `sketch_bytes` -verbatim, no part-level recompression). So **wire format = sketch_db storage = -disk-part bytes** — optimizing `sketchlib-go`'s `Serialize*` wins on bandwidth, -warm memory, AND cold disk at once. - -Measured (N=5000/window; harness `/mydata/sketch-audit`): - -| family | current encoding | bytes | headroom | verdict | -|---|---|---|---|---| -| HLL p=14 | dense 1 byte/register × 16384 (flat, any cardinality) | 16,532 | sparse full-state (delta-idx) → 5–50× for low card; 6-bit dense pack 1.34× | **P1** | -| KLL k=200 | raw f64 items array | 2,157 | `(v−offset)` fixed-point f64→~4 B → ~2× | **P2** | -| DDSketch α=.01 | dense varint counts keyed by FOR offset base, zigzag | 556 | already FOR+varint; sparse would be *larger* (85% occupancy) | skip | -| CMS / CountSketch 3×4096 | sint64 zigzag-varint (~1 B/cell) | ~12.5 KB | per-row FOR ~0 gain | skip | -| SUM/COUNT | OTLP Sum dp, raw f64/group | ~8 B/grp | OTLP framing dominates; residual marginal | skip | - -So the warm scope narrows to **two changes**: -- **P1 — HLL sparse full-state serialize** (HLL++ style: sorted non-zero - registers, delta+varint; fall back to 6-bit-packed dense above the crossover - ~6k nonzero regs). The lib already has a sparse *delta* path (`hll/delta.go`), - just not for full-state. Biggest lever — and since the 16 KB dense state is - stored **uncompressed** per instance, this also cuts **warm SketchStore - memory** 5–50× for low-cardinality series (not just wire, which gzip masks). -- **P2 — KLL value-offset/quantization** (`(v−offset)` fixed-point, ~2×) — the - offset idea, measured. - -DDSketch (already FOR+varint), CMS/CountSketch (already zigzag-varint, off the -legacy float64 matrix), and SUM/COUNT (OTLP-framing-bound) are already -well-encoded — do NOT touch. This **supersedes** the "FOR+delta on DDSketch -indices" / "narrow CMS counters" rows in the table above, which the audit shows -are redundant. - ---- - -## 3. Offset drift → re-base (Full) — unifies warm & cold - -A frame stays valid only while residuals fit it. Re-base the frame (warm: emit -a new **Full sketch**; cold: cut the chunk and start a new **base**) when: -- **Correctness**: a residual would overflow the chosen narrow width → MUST - re-base. -- **Efficiency** (optional): residuals now need K more bits than a re-base - would → re-base to reclaim ratio (K threshold weighed vs Full cost). - -Plus a **max-interval heartbeat Full** even without drift, because a Full is a -self-contained base needed for: (a) durability/recovery — without a recent Full, -a lost Delta makes the chain undecodable (cold S3 + restart-recovery depend on a -recent base); (b) a newly-joining consumer/query window needs a base. - -Rule: **emit Full when (drift) OR (heartbeat elapsed)**. - -Consequences: -- Full cadence becomes **per-series adaptive**: stable gauges almost never - re-base (nearly all Delta); volatile series re-base often (but volatile data - is inherently less compressible — cost lands where it should). -- **Counters are the exception**: use delta-of-value (base = previous sample, - auto-re-bases every sample, never "drifts") — so drift→Full is the gauge/FOR - story; counters just delta. - -Minimal change to today's pipeline: the agent already emits periodic Fulls; -add **drift** as a second Full trigger; backend delta-stitching already treats -a Full as the carry-in base, so a re-base is just a new base. - -### 3.1 v1 policy (concrete, zero-tuning) - -Ship the simplest correct version first; defer the only tunable knob. - -- **Drift trigger = hard/overflow ONLY.** Per frame, pick the residual integer - width from the Full's observed range (i16 if it fits, else i32, else i64); - re-base (warm: emit Full; cold: cut chunk + new base) the moment a residual - would exceed that width. This is a correctness bound, **not a tunable**. -- **Heartbeat = fixed interval.** Emit a Full at least every **N windows** even - without drift. Default `N` = the existing agent Full cadence (today's - ProtoFull period); for cold, the natural chunk bound (a time-block / ≤~120- - sample chunk) already serves as the heartbeat. Bounds crash-loss and the - query base-lookback to ≤ one heartbeat. -- **Counters:** delta-of-value (no drift); the heartbeat Full still applies - (recovery / new-consumer base). - -**DEFERRED — soft/efficiency drift (the tunable `K`):** re-base when residuals -waste `> K` bits vs a fresh frame. Skipped in v1 — it's a second-order -optimization and the only thing that would need per-shape tuning. Add it ONLY -if observation shows long-lived frames whose residuals widen (compression -silently degrading) without ever overflowing. Until then **v1 needs no tuning**. - ---- - -## 4. Shared per-series value stats from parse-once (NOT a shared offset constant) - -What is shared between the cold path and the warm sketches is the **parse-once -computation**, not a single offset constant: - -- **Shared (compute once, both consume):** the per-series **decimal scale - exponent** — which MUST be identical (one series has one natural precision, so - cold INT_FOR and KLL fixed-point land in the same integer domain and stay - mutually consistent) — plus the per-series value stats (running min/range). -- **NOT shared — the actual FOR base differs by cadence.** Cold re-bases the - base **per block** (each chunk takes its own min/first for the tightest - per-block residual width); warm sketches re-base **per Full-epoch** (the base - must stay stable so a run of Deltas remains mergeable, §3). These cadences - conflict — forcing one shared base would hurt whichever tier it's wrong for — - so the actual subtracted constant generally differs even though both derive - from the same parse-once stats. -- **Scope — only the raw-value-offset families:** cold INT_* values ↔ KLL ↔ SUM - (all subtract a reference from the same raw values). DDSketch (FOR on bucket - *indices*), HLL (hash), and CMS/CountSketch (counters) have **no shared - raw-value offset** — their compression uses their own structure (§2.1). -- **Kind matches per shape, not as a constant:** VM uses min-FOR for *gauges* - (same kind as KLL's value-offset) but delta-of-delta for *counters* (base = - first value), which aligns with the warm side's SUM-as-delta — so the - correspondence is gauge→FOR / counter→delta, not one shared number. - -CPU note: the offset does NOT reduce sketch-build cost (hashing/compaction is -fixed) — **sampling (§6) is what closes that CPU gap**. Compression's CPU wins -come from parse-once (done) + decode-on-read (cold decode off the ingest hot -path) + VM's cheaper decode. - ---- - -## 5. Open decisions -1. ~~Drift / heartbeat thresholds~~ **DECIDED (v1 — see §3.1)**: drift = - hard/overflow only (correctness bound, no tunable); heartbeat = fixed `N` - windows (= existing Full cadence). Soft/efficiency-`K` drift DEFERRED as a - later optimization — v1 needs no tuning. -2. ~~Which warm sketch families get the offset/FOR re-encoding first~~ - **RESOLVED by the §2.1 audit**: P1 = HLL sparse full-state (5–50×, + cuts - warm memory), P2 = KLL value-offset (~2×). DDSketch / CMS / CountSketch / - SUM are already well-encoded — skip. - -Explicitly OUT of scope (decided): no zstd-wrapped variants, and no lossy/Serf -option — cold stays purely lossless with the {Gorilla-XOR, INT_FOR_DELTA, -INT_FOR_DOD} best-of-N. - ---- - -## 6. Sampling × compression composition - -Sampling-enhanced sketches (inverse-probability bucket/counter updates, -hash-threshold key sampling, weighted KLL insertion — each with its own derived -error bound) compose with the compression scheme above. They sit on ORTHOGONAL -cost axes and are COMPLEMENTARY: -- **Compression** (FOR / delta / sparse / shared-ts) cuts BYTES — wire, - sketch_db memory, disk parts. -- **Sampling** cuts INGEST CPU + update rate — each item triggers a sketch - update only with probability `p`. This is the axis compression structurally - CANNOT touch: offset/FOR don't reduce the hashing/compaction build cost; - sampling does. (§4's note "offset doesn't reduce sketch-build CPU" — sampling - is what closes that gap.) - -**Scope**: sampling applies to the WARM sketch path only. The cold raw archive -is NOT sampled (it is the lossless backup; sampling would lose data). - -### 6.1 The composition rule -> Store the RAW SAMPLED integer state + one global `p` per frame; apply the -> `×1/p` rescale at QUERY, not at store. - -Storing the `1/p`-rescaled (inflated, often fractional) state would break -varint/FOR and bloat. Storing the raw sampled accumulation — e.g. DDSketch -`m_b = Σ Z_i` (≈ `p·n_b`, a SMALLER integer) + `p`, rescaled `m_b/p` at query — -keeps counts as small integers, so FOR/delta/varint (and the common-bits idea) -keep working. It is also numerically cleaner (integer accumulation, no per-update -fraction). `p` rides in the Full-epoch frame header alongside the offset. - -### 6.2 Per-family interaction -| family | sampling (CPU↓) | compression | interaction | -|---|---|---|---| -| HLL | hash-threshold (also thins registers) | sparse full-state (P1) | **strong synergy**: sampling zeroes more registers → sparser → sparse encoding wins more AND stays sparse up to ~`1/p`× higher true cardinality before the dense crossover. Query `n̂/p`. | -| DDSketch | bucket-update (writes→`p`) | index FOR+varint (done) | orthogonal; bucket SET ≈ unchanged so index encoding unchanged; store sampled counts (smaller int) + `p` → count varint smaller. | -| KLL | weighted insert (inserts→`p`) | value-offset/quantize (P2) | orthogonal; state still `k` items → offset-encode them; weight = global `(1/p)·2^h` (no per-item cost). | -| CMS / CountSketch (Nitro) | sampled counter update (writes→`pd` or fixed `s`) | zigzag-varint (done) | synergy: store sampled small-int counts + `p` → varint smaller; writes cut to `pd`. | - -### 6.3 Cross-cutting -- **Delta transmission**: sampling → fewer updates/window → fewer changed cells - → smaller delta frames. Merging sampled windows is benign — sampling error - `ε_s ∝ 1/√(pN)` shrinks as more windows merge (larger N). -- **Full-epoch frame**: `p` is a frame-level constant in the Full header (like - the offset); deltas inherit it. - -### 6.4 Cautions -- **Error budgets ADD**: `ε_total = sketch error + sampling ε_s (+ KLL value- - offset quantization)`, and must fit the metric's accuracy SLA. Family bounds - (derived separately): DDSketch `ε_s = O(√(log(B/δ)/pN))`; KLL `ε_total ≈ - ε_k + ε_s` with the design balance `pN ≳ k²`; HLL `RSE ≈ √((1−p)/(pn) + - 1.04²/m)`; Nitro adds variance `((1−p)/p)·Σ a_t²`. -- **`p` is a per-metric control-plane knob**: chosen per metric from the - expected N (rate/cardinality) + accuracy SLA. Fits the existing - controller-driven model exactly — the controller already annotates each - metric's tier + sketch type; it adds `p` the same way. - -### 6.5 Early benefits — offline Go benchmark (pre-integration) - -Measured against the REAL sketchlib-go sketches (harness `/mydata/sampling-bench`, -`go run .`, ~16s; UNSAMPLED vs SAMPLED-at-`p` vs EXACT ground truth, with the -§6.1 composition rule applied — raw sampled state stored, `×1/p` at query). **All -§7 bounds held empirically** across the `p`/`N`/`k`/`m` sweeps. - -| family | benefit @ `p=0.1` | accuracy | safe-`p` | -|---|---|---|---| -| DDSketch (α=1%) | 10× fewer bucket writes (1e6→1e5), ~6× wall-clock (239→34 ms) | q99 rank err ~0.005 (within ε_s); value relErr ≈ α | `p`≈0.05–0.1 (q99 suffers first at tiny `p`) | -| KLL | 10× fewer inserts, ~6× wall-clock | rank err tracks ε_s **iff `pN≳k²`**; below it q99 jumps (N=1e6,k=400,p=0.1 → pN Status: active + +## TL;DR + +These documents cover backend-owned storage and identity decisions. Planning +design lives in [control-plane docs](../../control_plane/docs/README.md), and +query serving lives in [data-plane docs](../../data_plane/docs/README.md). + +| Document | Scope | Status | +| --- | --- | --- | +| [Summary storage](summary-storage.md) | Materialization state, windows, ingest/query consistency, and lifecycle. | Active MVP design | +| [Series identity](series-identity.md) | Canonical metric-series identity and recovery behavior. | Active MVP design | +| [Future storage and compression](future-storage-and-compression.md) | Persistence tiers, compaction, pluggable service mode, backfill, compression, and profiling. | Dormant/future | + +Logical planning and summary accuracy algebra are owned by +[ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) and the relevant +summary libraries. Benchmark measurements belong in reproducible run artifacts, +not in these design documents. diff --git a/docs/design_docs/future-storage-and-compression.md b/docs/design_docs/future-storage-and-compression.md new file mode 100644 index 00000000..7ca5a4bc --- /dev/null +++ b/docs/design_docs/future-storage-and-compression.md @@ -0,0 +1,96 @@ +# Future storage and compression + +> Status: dormant +> +> MVP relation: not required for the OTel summary-pipeline MVP. + +## TL;DR + +This document records future backend storage and representation scopes without +turning unimplemented proposals, analytical estimates, or benchmark snapshots +into current architecture. Each scope requires its own acceptance criteria and +implementation proposal before becoming active. + +## Durable summary storage + +A future tier may flush immutable summary parts to disk or object storage and +recover them after restart. It must preserve the active storage contract: +materialization identity, logical windows, completeness, delta/checkpoint +lineage, and plan lifecycle. + +Example query: + +```promql +quantile_over_time(0.99, request_duration_seconds[24h]) +``` + +Serving older persisted panes is valid only when they are compatible and fully +cover the requested interval. + +## Semantic compaction + +Compaction may merge adjacent panes to reduce objects or read work. It is legal +only for a mergeable summary and must not cross incompatible materialization, +accuracy, grouping, or representation boundaries. + +Example: sixty compatible one-minute quantile-summary panes may compact into +one one-hour pane for: + +```promql +quantile_over_time(0.95, request_duration_seconds[1h]) +``` + +The actual PromQL mapping remains Planner-owned; this example describes only +the storage operation after a valid plan exists. + +## Backfill and refresh + +Backfill may build missing summary windows from an exact retained source. It +must be deterministic for the same source snapshot and materialization +contract, isolated from live ingestion, and atomically publish coverage. + +## Pluggable service mode + +Summary storage may eventually run as an in-process library or a separate +service. Both deployment shapes must expose equivalent semantic validation, +readiness, and error behavior. Network transport must not become a second +planning or identity authority. + +## Compression and representation + +Future work may include sparse summary state, delta checkpoints, compressed +raw fallback blocks, shared timestamp columns, or family-specific encodings. +Every representation must declare compatibility, recovery, and accuracy +effects. Lossy compression cannot be presented as exact. + +Example workload: + +```promql +sum by (service) (rate(http_requests_total[1h])) +``` + +Compression is evaluated on the state selected for this workload; it does not +change the logical query or choose a different summary. + +## Sampling and learned summaries + +Sampling, wavelets, anomaly models, or learned summaries require Planner-owned +logical semantics and guarantees before backend support. The backend may store +and execute an accepted family but must not define its query mapping locally. + +## Profiling and cost inputs + +A profiler may measure update cost, merge cost, readout latency, memory, and +encoded size for Planner's cost model. Measurements must identify the summary +implementation, parameters, workload, and hardware. Checked-in estimates or +one-off benchmark results are not substitutes for reproducible artifacts. + +## Activation rule + +A future scope becomes active only when it has: + +- an owning component and stable semantic interface; +- predeclared correctness and performance criteria; +- failure and recovery behavior; +- compatibility with BackendPlan and CollectorPlan; and +- reproducible end-to-end validation. diff --git a/docs/design_docs/series-identity.md b/docs/design_docs/series-identity.md new file mode 100644 index 00000000..a57de291 --- /dev/null +++ b/docs/design_docs/series-identity.md @@ -0,0 +1,81 @@ +# Series identity + +> Status: active +> +> MVP relation: provides stable identity for ingestion, grouping, and result +> labels across collector and backend boundaries. + +## TL;DR + +A series ID (`sid`) names one canonical metric series within a tenant and +identity namespace. The backend registry assigns or validates this mapping; +collectors may cache it, but payload labels remain the recovery evidence needed +to detect stale or unknown IDs. + +## Identity contract + +The canonical series key consists of: + +- tenant or isolation domain; +- metric name; and +- a deterministically ordered set of identifying labels. + +Two observations with the same canonical key resolve to the same `sid` within +one namespace. Different canonical keys must not share a `sid`. Aggregation +group labels and summary parameters are not silently folded into series +identity; they belong to the materialization/group contract. + +For example, these are different series: + +```text +http_requests_total{job="api",region="us-east"} +http_requests_total{job="api",region="eu-west"} +``` + +but reordering the two labels does not create a third identity. + +## Relationship to plan identity + +`sid`, materialization identity, and plan identity serve different purposes. A +series can participate in several materializations and plan versions. Reusing a +`sid` does not authorize reuse of summary state with different family, +grouping, parameters, or windows. + +## Resolution and caching + +The backend registry is authoritative for the namespace. A collector may cache +resolved IDs to reduce coordination, provided it also carries enough canonical +identity evidence for the backend to validate or recover the mapping. + +Resolution is idempotent: retrying the same canonical key returns the same +mapping. Registering an ID without receiving state is allowed and must not make +a query appear complete. + +## Recovery + +When the backend does not recognize a sender-provided `sid`, or finds that it +maps to different labels, it rejects the numeric shortcut and resolves from the +canonical key. A stale cache cannot overwrite an existing authoritative +mapping. + +Backend restart behavior depends on registry durability: + +- with durable identity state, mappings are restored before dependent payloads + become queryable; +- without durable state, collectors re-resolve from canonical labels under a + new namespace/version. + +In both cases ambiguity fails closed. + +## Distributed backend + +Distributed allocation, shard ownership, rebalancing, and high availability are +future deployment concerns. Any scheme must preserve deterministic lookup, +namespace/version evidence, and conflict detection. Numeric partitioning alone +must not weaken the canonical-key contract. + +## Non-goals + +This document does not prescribe RPC messages, integer width, database tables, +cache files, sharding algorithms, or a migration sequence from older `agg_id` +names. diff --git a/docs/design_docs/summary-storage.md b/docs/design_docs/summary-storage.md new file mode 100644 index 00000000..661be3fd --- /dev/null +++ b/docs/design_docs/summary-storage.md @@ -0,0 +1,96 @@ +# Summary storage + +> Status: active +> +> MVP relation: stores the state needed for summary-backed query execution. + +## TL;DR + +Summary storage is a plan-aware materialized-state store. It accepts only state +compatible with an active BackendPlan, indexes it by semantic materialization, +series/group, and logical window, and exposes complete state to planned +readouts. It is not a general raw time-series database and does not choose +summaries. + +## Stored object + +Each stored state belongs to: + +- one tenant and source; +- one active plan version; +- one materialization identity; +- one canonical series or aggregation group; +- one logical window; +- one summary family, parameter set, and representation version; and +- one producer/checkpoint lineage when full or delta state is used. + +The materialization identity includes the query-relevant semantics required for +safe reuse: source matchers, summarized value, grouping, family, parameters, +accuracy contract, and window definition. Storage location and delivery cadence +do not create a different logical materialization. + +## Ingestion + +Ingestion validates payload metadata against BackendPlan before changing +queryable state. Full state replaces a declared checkpoint. Delta state applies +only to its expected base and sequence. Duplicate delivery is idempotent; +missing or conflicting sequences create a visible gap rather than guessed +state. + +A window becomes queryable only after its completeness and freshness conditions +are satisfied. Payload receipt alone is insufficient. + +## Query lookup + +The data plane resolves a BackendPlan route to one or more materializations and +logical windows. Lookup returns either: + +- the complete compatible state required by the readout; or +- an explicit reason it is unavailable, such as missing coverage, stale state, + plan mismatch, delta gap, or unsupported merge. + +For example: + +```promql +quantile_over_time(0.95, request_duration_seconds[5m]) +``` + +may read five compatible one-minute DDSketch panes. The store may compose them +only when they exactly cover the requested interval and share the same +materialization contract. + +## Reconfiguration and lifecycle + +New and old plan versions may coexist during warm-up and drain, but state is +never mixed across incompatible materializations. Activation makes one version +authoritative for its declared interval. Retirement waits until readers, +lateness, and rollback policy no longer require the old version. + +State lifecycle includes staged, active, draining, expired, and rejected +conditions. “Present in storage” is not equivalent to “eligible for query.” + +## Memory and persistence + +The MVP may use bounded in-memory state, but it must report memory use and fail +visibly when limits prevent correct service. Evicting required state without +changing routing/readiness would violate correctness. + +Disk tiers, background flush, compaction, backfill, and standalone storage +service deployment are future extensions described in +[future storage and compression](future-storage-and-compression.md). + +## Guarantees + +- Incompatible summary states never merge. +- Missing series, groups, or windows are not treated as zero. +- Query responses never combine stale-run and current-run state. +- Accuracy metadata follows the selected logical result; storage does not + invent a new bound. +- Every accepted payload and served readout is traceable to plan and + materialization identity. + +## Non-goals + +This document does not define summary algorithms, Planner candidate selection, +Rust storage types, database schemas, file layouts, cache implementation, or +benchmark results. diff --git a/docs/developer_docs/adding-summary-family.md b/docs/developer_docs/adding-summary-family.md new file mode 100644 index 00000000..e4d3600d --- /dev/null +++ b/docs/developer_docs/adding-summary-family.md @@ -0,0 +1,54 @@ +# Adding a summary family to ASAPQuery-backend + +## TL;DR + +ASAPQuery-backend adds runtime support for a summary family only after +ASAPPlanner defines its logical query mapping and guarantee, and the producing +collector/library defines compatible state semantics. The backend must not +invent those contracts locally. + +## Ownership prerequisites + +Before changing this repository, confirm: + +- [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) can represent and + select the family for concrete PromQL examples; +- the summary library defines parameters, update, merge/readout, encoding, and + accuracy behavior; and +- [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector) can advertise, + configure, construct, and transmit the same family/version. + +## Backend work + +Backend support covers four boundaries: + +1. **Capability:** advertise the exact family, algorithm, parameter, readout, + merge, representation, and full/delta support implemented. +2. **Physical compilation:** accept only selected Planner nodes that can be + assigned to compatible collector and backend executors. +3. **BackendPlan and ingestion:** preserve the selected contract and reject + incompatible payloads. +4. **Readout:** execute the declared operation and return aligned + Prometheus-compatible labels, timestamps, values, and errors. + +For example, support for a new quantile family is incomplete until this query +can be planned, produced, ingested, and read end to end: + +```promql +quantile_over_time(0.95, request_duration_seconds[5m]) +``` + +## Validation + +The cross-repository test must cover: + +- supported and deliberately unsupported parameters; +- full-state transmission and delta transmission when claimed; +- duplicate, missing, reordered, stale, and incompatible payloads; +- merge across every claimed grouping/window shape; +- aligned comparison with an identical exact input stream; +- the declared accuracy and freshness SLA; and +- capability downgrade and exact fallback behavior. + +Unit tests for serialization or a local readout alone do not establish pipeline +support. diff --git a/docs/proofs.md b/docs/proofs.md deleted file mode 100644 index ebc8d464..00000000 --- a/docs/proofs.md +++ /dev/null @@ -1,698 +0,0 @@ -# `proofs.md` — Correctness theorems - -Paper-seed document for the ASAPQuery-backend sketch DB. The §theory -chapter of the VLDB / SIGMOD paper draws directly from the three -proofs in §§2–4. This file is reviewer-facing: every claim is bound -to a runtime function in `data_plane/src/`, and every -invariant the proof rests on is anchored to where the code enforces -it. - -§1 collects the per-sketch-family accuracy bounds the proofs cite. -The numerical $\varepsilon$ / $\delta$ formulas come from -`data_plane/src/storage_engines/sketch_db/accuracy.rs` -(`AccuracyProfile::derive`); the proofs treat the bounds as black -boxes (per-segment input). - -Conventions: $\varepsilon$ denotes additive / relative error, -$\delta$ the failure probability, $N$ the stream length (total -multiplicity of updates seen by a given sketch), $w$ and $d$ the -sketch width and depth, and $m = 2^p$ for HyperLogLog precision $p$. - -Cross-links to design docs: - -- §1 / §2 reference the schema timeline of - [`design-sketch-db.md`](./design-sketch-db.md) §7 and the - combinability table of `storage_engines/sketch_db/query/timeline_dispatch.rs`. -- §3 references the §6.3 write barrier of - [`design-sketch-db-core.md`](./design-sketch-db-core.md). -- §4 references the §10.5 deterministic-rebuild contract of - [`design-sketch-db-core.md`](./design-sketch-db-core.md). - -Code anchors below cite **symbols, not line numbers** (line numbers -drift; symbols can be relocated by `grep -n 'fn '`). Each -citation gives the file path + the function or type name as it -appears in the source today. - ---- - -## 1. Accuracy bounds per sketch family - -The following table is reproduced from -`data_plane/src/storage_engines/sketch_db/accuracy.rs` (§6.4 of the -sketch DB design). It lists every sketch family currently -materialisable by `AccuracyProfile::derive`. - -| Sketch | `kind` | $\varepsilon$ formula | $\delta$ formula | -|-------------------------------------|-----------------------|------------------------------|------------------------------| -| Sum / Min / Max / Increase | `Exact` | $0$ | $0$ | -| CountMinSketch $(w, d)$ | `AdditiveFrequency` | $e / w$ | $1 / 2^d$ | -| CountMinSketchWithHeap $(w, d, k)$ | `TopK` | $\max(e/w,\, 1/k)$ | $1 / 2^d$ | -| CountSketch $(w, d)$ | `AdditiveFrequency` | $1 / \sqrt{w}$ | $1 / 2^d$ | -| HLL $(p)$ | `RelativeCardinality` | $1.04 / \sqrt{2^p}$ | — (Gaussian std-dev) | -| KLL $(k)$ | `RankQuantile` | $\approx 2.296 / \sqrt{k}$ | $1/100$ (fixed) | -| DDSketch $(\alpha)$ | `RelativeQuantile` | $\alpha$ | $0$ (deterministic) | - -The `kind` column drives user-facing rendering; it does not alter -the numerical $\varepsilon$. Sources are cited inline in -`accuracy.rs` (Cormode & Muthukrishnan 2005 for CMS; Charikar–Chen– -Farach-Colton for CountSketch; Flajolet et al. 2007 for HLL; Karnin– -Lang–Liberty 2016 for KLL; Masson–Rim–Lee 2019 for DDSketch). - -The proofs in §§2–4 take these per-segment bounds as inputs; they -do not re-derive them. - ---- - -## 2. `combine_statistic` correctness across schema-timeline segments - -When a query's time range crosses one or more reconfigure -boundaries, `timeline_for_metric` partitions the -range into contiguous half-open segments -$[t_0, t_1), [t_1, t_2), \dots, [t_{n-1}, t_n)$, each owned by a -single `SketchInstanceMetadata` with its own sketch parameters. The query -engine evaluates the statistic per segment and feeds the per- -segment scalars to `combine_statistic` -(`data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs`). - -### 2.1 Statement - -Let $\sigma$ be a statistic, let $S = \cup_{i=0}^{n-1} [t_i, t_{i+1})$ -be the query range partitioned into pairwise disjoint covering -segments by `timeline_for_metric`, and let $a_i$ be the per-segment -estimate for $\sigma$ on segment $i$, with per-segment error bound - -$$ -\big| a_i - \sigma_i(S_i) \big| \;\le\; B_i, -$$ - -where $B_i = \varepsilon_i \cdot N_i$ for the -`AdditiveFrequency`/`Relative` sketch families of §1 and $B_i = 0$ -for the `Exact` family. Let -$\hat a = \mathrm{combine\_statistic}(\sigma, \{a_i\}, \emptyset)$. - -**(a) Additive case.** If -$\sigma \in \{\mathrm{Count}, \mathrm{Sum}\}$, then -`combine_statistic` returns `Full(`$\hat a$`)` and - -$$ -\big|\hat a - \sigma(S)\big| \;\le\; \sum_{i=0}^{n-1} B_i. -$$ - -**(b) Idempotent case.** If -$\sigma \in \{\mathrm{Min}, \mathrm{Max}\}$, then -`combine_statistic` returns `Full(`$\hat a$`)` and - -$$ -\big|\hat a - \sigma(S)\big| \;\le\; \max_{0 \le i < n} B_i. -$$ - -**(c) Non-combinable case.** If -$\sigma \in \{\mathrm{Cardinality}, \mathrm{Quantile}, -\mathrm{Topk}, \mathrm{Rate}, \mathrm{Increase}\}$, -`combine_statistic` returns -`Partial { covered, missing }`, where `missing` is the unresolved- -segment list passed in by the caller (possibly empty) and `covered` -is `None` (no scalar combiner is sound). Soundness is preserved by -construction: `Partial` is treated by the caller as "do not surface -as a single answer." - -### 2.2 Setup / Lemmas - -**L2.1 — Disjoint covering by `timeline_for_metric`.** For any -metric $m$ and query range $[t_1, t_2]$, the segments returned by -`timeline_for_metric(m, t1, t2)` -(`data_plane/src/storage_engines/sketch_db/query/timeline.rs:timeline_for_metric`) -are pairwise disjoint, sorted by `start_ms`, and each is owned by -exactly one `agg_id`. The owner's range is -$[\mathtt{created\_at\_ms},\,\mathtt{own\_end})$ with `own_end` -defined as `min(next.first_seen_unix_ms, retired_at_ms)` (open segment -$\to u64::\mathrm{MAX}$ for the currently-Active schema). The -function clips each segment to the query range and skips segments -of zero length, so the returned partition is a refinement of the -query range with no overlaps. (Note: gaps may exist where no -schema was Active; the caller folds those into the `unresolved` -input to `combine_statistic`.) - -**L2.2 — Linearity of additive aggregates over disjoint sets.** For -$\sigma \in \{\mathrm{Count}, \mathrm{Sum}\}$, -$\sigma(\cup_i S_i) = \sum_i \sigma(S_i)$ when the $S_i$ are -pairwise disjoint. This is set-theoretic, not sketch-specific. - -**L2.3 — Idempotence + associativity of `min` / `max` over -multisets.** For $\sigma \in \{\mathrm{Min}, \mathrm{Max}\}$, -$\sigma(\cup_i S_i) = \sigma(\{\sigma(S_i) : 0 \le i < n\})$ when -the $S_i$ are pairwise disjoint and at least one is non-empty. -Idempotence + associativity together justify the pointwise fold. - -**L2.4 — Triangle inequality on real numbers.** Standard. - -**L2.5 — Combiner implementation.** `combine_statistic` -(`data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs:combine_statistic`) -folds segments via `fold(0.0, +)` for `Count` / `Sum`, -`fold(None, |a,v| Some(a.map_or(v, |a| a.min(v))))` for `Min` (mut. -mut. for `Max`), and returns `None` (so a `Partial` wrapper) for -`Cardinality / Quantile / Topk / Rate / Increase`. Empty input -short-circuits to `Full(0.0)` for additive statistics and -`Partial { covered: None, missing: [] }` otherwise. - -**L2.6 — Per-segment error bound.** For each segment $i$ the -sketch family's published bound (Cormode–Muthukrishnan; Karnin– -Lang–Liberty; etc., as reproduced in §1) gives -$|a_i - \sigma_i(S_i)| \le B_i$. The combiner takes $a_i$ as a -black-box scalar; it does not reach into the sketch. - -### 2.3 Proof - -We treat the three cases. - -**Case (a): additive ($\sigma \in \{\mathrm{Count}, \mathrm{Sum}\}$).** -The combiner returns $\hat a = \sum_i a_i$ (L2.5). By L2.2, -$\sigma(S) = \sum_i \sigma_i(S_i)$. Therefore - -$$ -|\hat a - \sigma(S)| = \Big| \sum_i a_i - \sum_i \sigma_i(S_i) \Big| - = \Big| \sum_i (a_i - \sigma_i(S_i)) \Big| - \overset{L2.4}{\le} \sum_i |a_i - \sigma_i(S_i)| - \overset{L2.6}{\le} \sum_i B_i. -$$ - -The combiner's output is `Full(`$\hat a$`)` because the input -unresolved-list is empty (L2.5). - -**Case (b): idempotent ($\sigma \in \{\mathrm{Min}, \mathrm{Max}\}$).** -WLOG $\sigma = \mathrm{Min}$. The combiner returns -$\hat a = \min_i a_i$ (L2.5). By L2.3, -$\sigma(S) = \min_i \sigma_i(S_i)$. Pick any $i^\ast$ achieving the -combiner's argmin, and any $j^\ast$ achieving the true argmin. -Then - -$$ -\hat a - \sigma(S) - = a_{i^\ast} - \sigma_{j^\ast}(S_{j^\ast}) - \le a_{j^\ast} - \sigma_{j^\ast}(S_{j^\ast}) \le B_{j^\ast} - \le \max_i B_i, -$$ - -where the first inequality is by minimality of $a_{i^\ast}$, the -second is L2.6 on segment $j^\ast$, and the third is trivial. -Symmetrically, - -$$ -\sigma(S) - \hat a - = \sigma_{j^\ast}(S_{j^\ast}) - a_{i^\ast} - \le \sigma_{i^\ast}(S_{i^\ast}) - a_{i^\ast} - \le B_{i^\ast} - \le \max_i B_i, -$$ - -so $|\hat a - \sigma(S)| \le \max_i B_i$. - -**Case (c): non-combinable.** The combiner unconditionally returns -`Partial { covered: None, missing: unresolved }` (L2.5, -non-combinable arm of the `match`). The contract is that `Partial` -is **not** a single-schema answer; the caller (the engine) routes -non-combinable cases to fallback per -`ASAPQueryEngine::try_handle_query_promql_via_timeline`. There is -nothing further to prove. - -### 2.4 Caveats - -- **Segment coverage of underlying samples.** The proof assumes - each segment's sketch contains every sample the agg actually - ingested for that range. Sketch parameter changes mid-segment - cannot occur (an agg's `AggregationConfig` is pinned at creation; - `SketchInstanceMetadata::config` is immutable), but a reconfigure that - retires one agg and creates another may leave a brief gap if no - agg is Active at some $t \in [t_i, t_{i+1})$. Such gaps are - surfaced as `unresolved` segments by the caller and are not - silently elided. -- **`Cardinality` is non-combinable at the scalar level only.** - HLL register OR-merge would give a sound combine, but the inputs - to `combine_statistic` are post-`query_statistic` scalars. A - future "merge-then-evaluate" path would need a different proof. -- **`Increase` / `Rate` use endpoint samples.** Stitching two - per-segment rates at a boundary does not give the cross-boundary - rate even when the segments are time-disjoint. The combiner - conservatively returns `Partial`; the caller may delegate to a - fallback (raw-sample re-evaluation against the cold store). -- **Numerical floating-point drift.** Summing $n$ floats accrues - $O(n \cdot \mathrm{ulp})$ rounding error. Pairwise / Kahan - summation would tighten the constant; today's combiner uses naive - accumulation. This is below sketch error for the sketches in §1 - but worth noting for an exact `Sum` over $\sim 10^9$ segments. -- **Concurrent reconfigure during a query.** The proof assumes - `timeline_for_metric` returns a consistent snapshot. The function - acquires a single `RwLock` read before partitioning, so a - concurrent `reconcile` either hasn't started or is fully visible — - no torn read. A reconfigure that retires an agg between segment - evaluations is allowed by the design (the engine fetches each - segment's sketch under the same registry view). -- **Empty range.** An empty `segments` slice (no schema covers the - query range) yields `Full(0.0)` for `Count` / `Sum` and - `Partial { covered: None, missing: [] }` otherwise. The empty - case is meaningful only for additive statistics; the proof above - presupposes $n \ge 1$ for case (b). - -### 2.5 Code anchors - -- `data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs` - - `combine_statistic` — the per-statistic fold (additive arm, - `Min` / `Max` arm, non-combinable arm). - - `CombinedResult::{Full, Partial}` — the `Full` / `Partial` - discrimination relied on in §2.3. - - Module doc-comment — combinability table. -- `data_plane/src/storage_engines/sketch_db/index/mod.rs` - - `timeline_for_metric` — disjoint covering - (L2.1). - - `SketchInstanceMetadata::config` — pinned-at-creation invariant (caveat 1). -- `data_plane/src/query_engines/asap_query_engine/engine.rs` - - `ASAPQueryEngine::try_handle_query_promql_via_timeline` — caller - that wires per-segment evaluation into `combine_statistic`. -- `data_plane/src/storage_engines/sketch_db/accuracy.rs` - - `AccuracyProfile::derive` — source of $B_i$ (L2.6). - ---- - -## 3. Write-barrier safety - -The §6.3 write barrier (`design-sketch-db-core.md` §6.3) guarantees -that no late or replayed sample can leak into a query once its -schema has been forced to expire. This proof formalises that. - -### 3.1 Statement - -Let $a$ be an `agg_id`, let $t^\ast$ be the wall-clock time at which -`SketchStore::force_expire(a)` is invoked and returns -`Some(_)`, and let $s$ be any sample submitted to the ingest path -targeting $a$ at wall-clock time $t > t^\ast$. Then for every -query $Q$ executed at any wall-clock time $t_Q \ge t^\ast$ whose -range $[t_1, t_2]$ includes any $t' \ge t^\ast$, the sample $s$ -does not appear in $Q$'s result. - -### 3.2 Setup / Lemmas - -**L3.1 — `force_expire` is monotonic.** When -`force_expire(a)` returns `Some(_)` at wall-clock time $t^\ast$, -the schema's `retired_at_ms` and `expires_at_ms` are both set to -the value of `now_ms()` captured atomically inside the function -body, under the registry's `RwLock` write guard -(`data_plane/src/storage_engines/sketch_db/index/mod.rs:force_expire`). - -**L3.2 — Status is a pure function of timestamps.** -`SketchInstanceMetadata::status` (`index/mod.rs:status`) is a pure function of -`(retired_at_ms, expires_at_ms, now_ms())`. With both fields set -to $t^\ast$, the match arm -`(Some(_), Some(exp)) if now >= exp` fires for every subsequent -`now >= t^\ast`, returning `AggStatus::Expired`. The arm returning -`AggStatus::Active` requires `retired_at_ms.is_none()`, which is -unreachable after L3.1 (no code path clears `retired_at_ms`; -`SketchInstanceMetadata::retire` is idempotent, `force_expire` only advances). - -**L3.3 — Barrier is in the ingest hot path.** Every sample that -reaches the ingest router passes through -`route_decoded_samples` in -`data_plane/src/precompute_engine/ingest_handler.rs`, which -unconditionally calls `state.schemas.is_writable(config.aggregation_id)` -(see the loop body around the `!state.schemas.is_writable(…)` -guard) before adding the sample to its `by_group` map. Samples -that fail this check are routed to the -`samples_blocked_by_schema_barrier` counter and are not entered -into `by_group`, so they are not forwarded to any worker. - -**L3.4 — `is_writable` rejects non-Active.** -`SketchStore::is_writable(agg_id)` -(`index/mod.rs:is_writable` on the store, delegating to -`SketchInstanceMetadata::is_writable` on the metadata) returns `true` iff -`status() == AggStatus::Active`, by direct match -(`matches!(self.status(), AggStatus::Active)`). - -**L3.5 — Workers only see what the router forwards.** The -`WorkerMessage::GroupSamples` payload carries only the samples -that survived the barrier check (`route_decoded_samples` constructs -the payload from `by_group`; samples blocked at L3.3 are not in -`by_group`). Workers have no other ingest channel. - -**L3.6 — Queries read only what workers wrote.** The store -(`SketchStore` and friends, `data_plane/src/storage_engines/`) -exposes no API that returns samples not previously written through -`insert_precomputed_output_batch`. Queries flow through -`ASAPQueryEngine::query_*`, which read the same store the workers -wrote into. - -### 3.3 Proof - -Direct, by chasing the sample's path. - -1. By assumption, $s$ is submitted at $t > t^\ast$ targeting `a`. -2. By L3.1, at $t^\ast$ the registry stores - `retired_at_ms = expires_at_ms = t^\ast` for `a`. -3. By L3.2, for every wall-clock query of $a$'s status performed - at any time $\ge t^\ast$, `status() = Expired` $\ne$ `Active`. -4. By L3.3, $s$ enters the ingest router and the router calls - `is_writable(a)` to decide whether to admit it. -5. At the moment of step 4, `now >= t > t^\ast`, so by L3.2 the - `is_writable` call observes `Expired`. By L3.4 the call returns - `false`. -6. By L3.3, $s$ is dropped (and the - `samples_blocked_by_schema_barrier` counter is bumped); $s$ is - not added to any `by_group` entry. -7. By L3.5, no worker ever sees $s$. -8. By L3.6, no query ever sees $s$. - -In particular $Q$, which reads through the same store, returns a -result that does not contain $s$. $\blacksquare$ - -### 3.4 Caveats - -- **Crash-window samples.** A sample $s$ that has been deserialised, - routed through `route_decoded_samples`, **and** passed the - `is_writable` check before $t^\ast$ but is still in flight to - the worker at $t^\ast$ is admitted. The proof's $t$ is the time - the barrier check happens, not the sample's wall-clock arrival. - Operationally the gap is microseconds; bounding it formally - would require a fence on the worker channel, which isn't - implemented. -- **Clock skew.** `now_ms()` is the local monotonic-ish wall clock - of the backend process. If the operator forces expiry from an - external endpoint while the process clock is skewed backwards, - some "post-$t^\ast$" samples may carry timestamps that look - pre-$t^\ast$. The barrier checks `now`, not the sample's - embedded `timestamp_ms`, so the safety argument holds against - process-local time; it does not promise consistency with an - external NTP source's view. -- **Persistence-on-restart.** If the process restarts between - `force_expire` and the sample's arrival, the barrier holds iff - the registry was persisted (L3.1's mutation reaches disk via - `save_to_disk_if_persistent` inside `force_expire`). If - persistence is disabled (`persist_path = None`), the registry is - rebuilt from the current `StreamingConfig` only — an - Expired-but-still-listed agg would re-emerge as Active. The - documented production setting uses persistence; this proof - assumes that. -- **Bypass paths.** The proof rests on every ingest entry point - going through `route_decoded_samples`. If a future driver writes - directly to the store, the barrier doesn't fire. Today the only - ingest-side writers are the Prometheus remote-write handler, the - OTLP gRPC handler, and the OTLP HTTP handler, all of which fan - in through `route_decoded_samples`. Backfill writes are exempt - by design (proof §4 covers them) and are blocked by §10.5 - time-disjointness from the barrier's domain. -- **Counter-only path.** A sample dropped by the barrier increments - `queryengine_ingest_samples_blocked_by_schema_barrier_total`. - The proof does not say anything about that counter's correctness - — only that the sample doesn't reach the store. - -### 3.5 Code anchors - -- `data_plane/src/storage_engines/sketch_db/index/mod.rs` - - `SketchStore::force_expire` — sets `retired_at_ms` and - `expires_at_ms` to `now_ms()` (L3.1). - - `SketchInstanceMetadata::status` — pure function of timestamps (L3.2). - - `SketchInstanceMetadata::is_writable` — `status() == Active` check (L3.4). - - `SketchStore::is_writable` — registry-level wrapper (L3.4). - - `AggStatus` — the three-state enum. - - `now_ms` (file-private helper). -- `data_plane/src/precompute_engine/ingest_handler.rs` - - `route_decoded_samples` — calls `is_writable` per sample, - drops on `false`, bumps counter (L3.3). - - `samples_blocked_by_schema_barrier` (atomic on - `IngestState`). -- `data_plane/src/storage_engines/sketch_db/metrics.rs` - - `SAMPLES_BLOCKED_BY_SCHEMA_BARRIER` — Prometheus counter - (`queryengine_ingest_samples_blocked_by_schema_barrier_total`). -- Tests: - `data_plane/src/precompute_engine/ingest_handler.rs::tests::barrier_counter_increments_after_force_expire` - exercises the path end-to-end. - ---- - -## 4. Backfill determinism - -§10.5 of the design doc claims that an offline backfill, fed the -same raw samples in the same order that live ingest would have -seen, builds a sketch byte-identical to the live one. This proof -formalises that claim under the §10.5 invariants enforced at job -creation. - -### 4.1 Statement - -Let `agg_id` $a$ be known to the schema registry with -`SketchInstanceMetadata` $\Sigma$, and let -`BackfillRegistry::create_checked(_, a, (s, e), _, _, retention)` -return `Ok(job_id)`. By construction (see L4.1 below), the job -satisfies: - -1. **Known agg:** $\Sigma \ne \bot$; -2. **Time-disjoint:** $e \le \Sigma.\mathtt{created\_at\_ms}$; -3. **Within retention:** if `retention = Some(R)`, then - $s \ge \mathrm{now}() - R$. - -Let $X = (x_1, x_2, \dots, x_N)$ be the raw-sample sequence the -backfill processor reads in ingest order via `RawSampleReader` for -window $w \subseteq [s, e)$. Let $\sigma_B$ be the sketch produced -by `build_backfilled_accumulator(`$\Sigma$.config, $X$`)` and let -$\sigma_L$ be the sketch a hypothetical live-ingest worker would -have produced from the same $X$ in the same order using -`create_accumulator_updater(`$\Sigma$.config`)` followed by -`update_single` / `update_keyed` per sample. Then - -$$ -\sigma_B.\mathrm{serialize\_to\_bytes}() \;=\; -\sigma_L.\mathrm{serialize\_to\_bytes}(). -$$ - -### 4.2 Setup / Lemmas - -**L4.1 — Invariants enforced by `create_checked`.** -`BackfillRegistry::create_checked` -(`data_plane/src/storage_engines/sketch_db/backfill/mod.rs:create_checked`) -returns `Ok` only after: - -- `schemas.get(agg_id) = Some(_)` (else `CreateError::UnknownAgg`), -- `time_range.1 <= schema.first_seen_unix_ms` (else - `CreateError::Overlap`), -- if `data_retention_ms = Some(R)`, - `time_range.0 >= now_ms().saturating_sub(R)` (else - `CreateError::OutOfRetention`). - -These are the three §10.5 invariants verbatim. - -**L4.2 — Same construction factory.** Both paths build their -accumulator via the same factory: - -- Live: `create_accumulator_updater(config)` in - `data_plane/src/precompute_engine/accumulator_factory.rs`, - then `update_single` / `update_keyed` in ingest order. -- Backfill: `build_backfilled_accumulator(config, samples)` in - `data_plane/src/storage_engines/sketch_db/backfill/window_builder.rs`, - whose body is exactly - `let mut updater = create_accumulator_updater(config); for s in samples { updater.update_*(...) } updater.take_accumulator()`. - -The two paths therefore differ only in (i) where the samples -come from and (ii) the wall-clock time at which each call happens. - -**L4.3 — Pinned config.** `SketchInstanceMetadata::config: AggregationConfig` is -set when the `SketchInstanceMetadata` is registered and never mutated thereafter -(`index/mod.rs`); the comment on the field says verbatim *"Pinned at -schema creation; never mutated."* So the `config` argument both -paths feed into `create_accumulator_updater` is bit-identical for -the same `agg_id`. - -**L4.4 — Sketch construction is a pure function of (config, sample -sequence).** The accumulator types built by -`create_accumulator_updater` (e.g. `SumAccumulator`, -`DDSketchAccumulator`, `KllAccumulator`, -`HllSketchAccumulator`, `CountSketchAccumulator`, -`CountMinSketchAccumulator`) seed any randomness from the `config` -parameters (hash seeds, sketch sizes), not from wall-clock or any -process-global RNG. The §10.5 design-doc constraint *"The hash -function seed for CMS / CountSketch / HLL must be part of -`AggregationConfig.parameters` and stable across Native and -Backfill paths"* is exactly this lemma. - -**L4.5 — `serialize_to_bytes` is deterministic.** All -`AggregateCore` impls produce a serialisation that depends only on -the accumulator's internal state (no embedded wall-clock, -allocator-address, or HashMap-iteration-order fields). The pinning -test -`data_plane/src/storage_engines/sketch_db/backfill/processor.rs::tests::backfill_builds_bit_identical_sum_accumulator_to_live` -locks this invariant for `SumAccumulator` and is the canary for -the rest of the family. - -**L4.6 — Time-disjointness eliminates ordering ambiguity.** L4.1 -gives $e \le \Sigma.\mathtt{created\_at\_ms}$. Live ingest writes -exactly $[\Sigma.\mathtt{created\_at\_ms}, \infty)$ (the §6 schema -lifecycle says `is_writable = false` before `first_seen_unix_ms`, -because the schema doesn't exist yet, and `is_writable = true` -afterwards while `Active`). Backfill writes -$[s, e) \subseteq [0, \Sigma.\mathtt{created\_at\_ms})$. The two -ranges are disjoint by L4.1, so for any sample $x$ at most one -path ever observes it; there is no race, no double-update, and no -ordering ambiguity at the boundary. - -**L4.7 — Retention check makes outputs observable.** L4.1 gives -$s \ge \mathrm{now}() - R$, so every window the backfill writes is -within the store's retention horizon at job-creation time. The -proof itself only needs equality of *internal* sketch state; this -lemma is included so the caveat list in §4.4 doesn't lose track of -why this check is part of "§10.5." - -**L4.8 — Sample-stream alignment.** Both paths see the same -$(\mathrm{label}, \mathrm{timestamp}, \mathrm{value})$ sequence for -window $w$ in the same order. For the **live** path this is the -order in which the underlying ingest channel delivered samples; for -the **backfill** path this is the order returned by -`RawSampleReader::read_samples`, whose contract (top of -`raw_sample_reader.rs`) requires *"samples should be returned in -**ingest order** per-series — §10.5 requires deterministic replay, -and the contract is easiest to satisfy at the reader layer."* The -proof presupposes this contract holds; see caveat (a). - -### 4.3 Proof - -By induction on the length $N$ of the sample stream $X$. - -**Base $N = 0$.** Both paths return the accumulator constructed by -`create_accumulator_updater(config)` with no updates applied. By -L4.3 the config is bit-identical; by L4.4 the empty-state -accumulator is a pure function of the config; therefore -$\sigma_B = \sigma_L$ at the byte level, and L4.5 lifts that to -`serialize_to_bytes`. - -**Step $N \to N + 1$.** Assume the two paths agree after the first -$N$ updates. Both paths now apply the same single update — either -`update_single(value, ts)` or `update_keyed(key, value, ts)` — -where `key` is computed by the same `extract_aggregated_key` -function on the same `(labels, config)` (used by the live worker -in `precompute_engine/worker.rs::extract_aggregated_key_from_series` -and by the backfill builder via the package-private -`extract_aggregated_key` in -`backfill_window_builder.rs`; the doc-comment on -`build_backfilled_accumulator` calls out the parity explicitly). -By L4.4 the update is a pure function of (prior state, value, -timestamp, key); since the prior states agree by IH and all -arguments agree by hypothesis, the post states agree. - -**Termination + serialisation.** After all $N$ samples, both -paths call `take_accumulator()` on their `Box` and then `serialize_to_bytes()` on the -resulting `Box`. By L4.5 the byte output is a -pure function of the accumulator's state, which we have shown to -agree. Hence -$\sigma_B.\mathrm{serialize\_to\_bytes}() = -\sigma_L.\mathrm{serialize\_to\_bytes}()$. $\blacksquare$ - -### 4.4 Caveats - -- **Reader-side ordering (L4.8).** The proof assumes the - `RawSampleReader` returns samples in ingest order. Concrete - reader impls (Prometheus HTTP, S3 Gorilla, ClickHouse) must - honour this; if a reader returns timestamp-sorted samples that - re-order in-second arrivals, the backfilled sketch can still - diverge from live for sketches whose state depends on update - order (KLL sampling decisions; DDSketch buffer eviction order). - The contract is documented; enforcement is per-impl. -- **DataCollector sketch-built deployments.** When live ingest - runs through the DataCollector OTLP path, the sketch is built by - `sketchlib-go` (Go) and the backend only deserialises. Bit- - identical determinism vs. backfill (which builds via the Rust - `asap_sketchlib`) requires Go and Rust sketch builds to agree - byte-for-byte. Cross-language byte parity for DDSketch / KLL / - CountSketch landed in 2026-05-05 (PRs #40/#41/#42 + - #43/#44/#45); HLL / CMS variants are in flight per - `design-asap-precompute-rs.md`. For deployments where - this parity hasn't landed, "bit-identical" in §4.1 weakens to - "agree within sketch error bound $\varepsilon$ of §1." -- **HashMap-iteration order in grouping.** The backfill processor - groups by `group_key` into a `HashMap>` - and iterates groups in HashMap-iteration order. Within each - group the per-sample order is preserved (Vec push-order); the - proof above is per-group. Cross-group order does not affect the - per-(agg_id, group_key, window) sketch, which is the unit of - the §4.1 claim. -- **Timestamp-bound sketches.** Sketches that compute their state - from `(value, timestamp_ms)` (as opposed to ignoring timestamp) - are deterministic in this proof iff the timestamp is the - ingest-side timestamp, not the wall clock at update time. - `update_single(value, ts)` / `update_keyed(key, value, ts)` - pass the sample's `ts`, not `now()`, so the lemma holds. Any - future accumulator that uses `now()` internally would break - this. -- **Schema retired mid-backfill.** L4.6 says the time-ranges are - disjoint at job-creation. If the agg is retired while the job - is running and the backfill processor attempts to look up the - config via `config_for_agg`, it surfaces the error (`agg_id … - not in current StreamingConfig — retired mid-backfill?`). The - in-progress windows that already wrote complete; later windows - fail the job. The proof's claim is per-window: each - successfully-written window is bit-identical; failed windows - are absent. -- **Concurrent retention sweep.** The retention check at job - creation (L4.7) is at $\mathrm{now}()$; if the job is long- - running and retention catches up to $s$ before the job - completes, the early windows may be evicted from the store - *after* the bit-identical write. The proof says nothing about - read-back of evicted windows. -- **Cross-deployment determinism.** The proof is *intra-process*: - same `config`, same backend binary. Different versions of the - same backend with different dependency versions of - `asap_sketchlib` may serialise the same logical state to - different bytes. The serialisation-format-versioning tests - (`tests/persist_format_versioning_tests.rs`) cover the on-disk - format; this proof does not extend across format-version bumps. - -### 4.5 Code anchors - -- `data_plane/src/storage_engines/sketch_db/backfill/mod.rs` - - `BackfillRegistry::create_checked` — enforces the three - §10.5 invariants (L4.1). - - `CreateError::{UnknownAgg, Overlap, OutOfRetention}` — the - three failure modes. -- `data_plane/src/storage_engines/sketch_db/backfill/window_builder.rs` - - `build_backfilled_accumulator` — the backfill-side - construction used in §4.1 and L4.2. - - `extract_aggregated_key` — keyed-grouping function shared - semantically with the live worker. -- `data_plane/src/precompute_engine/accumulator_factory.rs` - - `create_accumulator_updater` — the shared factory (L4.2). -- `data_plane/src/storage_engines/sketch_db/backfill/processor.rs` - - `BackfillWindowProcessor::process_window` — calls - `build_backfilled_accumulator` per group, writes via - `Store::insert_precomputed_output_batch`. - - module doc-comment "Determinism (§10.5)" — explicit - invariant reference. - - test - `tests::backfill_builds_bit_identical_sum_accumulator_to_live` — - runtime canary for L4.4 + L4.5 on `SumAccumulator`. -- `data_plane/src/storage_engines/sketch_db/backfill/raw_sample_reader.rs` - - `RawSampleReader::read_samples` — ingest-order contract - (L4.8). -- `data_plane/src/storage_engines/sketch_db/index/mod.rs` - - `SketchInstanceMetadata::config` — pinned-at-creation invariant (L4.3). - - `SketchInstanceMetadata` registration — `first_seen_unix_ms` capture used by - L4.6. - ---- - -## 5. Cross-reference - -Each theorem anchors to a single module; this section exists so -the paper's §theory chapter can cite both at once. - -- **Accuracy bounds (§1)** → - `data_plane/src/storage_engines/sketch_db/accuracy.rs` - (`AccuracyProfile::derive`). -- **`combine_statistic` correctness (§2)** → - `data_plane/src/storage_engines/sketch_db/query/timeline_dispatch.rs` - (`CombinedResult`, `combine_statistic`). -- **Write-barrier safety (§3)** → - `data_plane/src/storage_engines/sketch_db/index/mod.rs` - (`is_writable`, `status`, `retire`, `force_expire`); barrier - counter `SAMPLES_BLOCKED_BY_SCHEMA_BARRIER` in - `data_plane/src/storage_engines/sketch_db/metrics.rs`. -- **Backfill determinism (§4)** → - `data_plane/src/storage_engines/sketch_db/backfill/mod.rs` - (`BackfillRegistry::create_checked`); construction parity in - `backfill_window_builder.rs::build_backfilled_accumulator`; - pinning test in - `backfill_processor.rs::tests::backfill_builds_bit_identical_sum_accumulator_to_live`.