Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ spin up ASAPCollector + ASAPQuery-backend + Grafana together — live in
[ASAPCollector](https://github.com/ProjectASAP/ASAPCollector). The
full multi-stage MVP demo (10 producers / 2 agents / 1 gateway / 1
backend / Thanos store-gateway / MinIO) is documented in its
[`docs/mvp-demo-runbook.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/mvp-demo-runbook.md).
[`mvp-demo-runbook.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/user_guide/mvp-demo-runbook.md).

To build and run just this backend, see **Building from source** below.

Expand Down Expand Up @@ -192,7 +192,7 @@ backend surfaces in every response's `infos` field.
The wire format is documented in
[`asap_otel_proto`](crates/asap_otel_proto/) and
the cross-language byte-parity gate is described in
[ASAPCollector's edge-framework design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design-asap-edge-framework.md).
[ASAPCollector's system design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/system-overview.md).

## Query response shape

Expand Down Expand Up @@ -231,9 +231,9 @@ that produced the current architecture:
- **Sketch placement planner** — moved into [`ASAPCollector/controller/`](https://github.com/ProjectASAP/ASAPCollector/tree/main/controller)
- **PromQL pattern matchers for the planner** — migrated into the
controller's L3 `intent_algebra` + L4 `sketch_algebra`
- **JSONL cold-fallback path** — deleted; the archive tier replaces
it. See ASAPCollector's
[`docs/design-jsonl-deprecation-and-gorilla-promql-completeness.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design-jsonl-deprecation-and-gorilla-promql-completeness.md)
- **JSONL cold-fallback path** — deleted; the configured exact backend is the
explicit fallback described by
[`query-execution.md`](data_plane/docs/design_docs/query-execution.md).
- **`StorageBackend::ColdJsonlFallback`** enum variant — removed
- **Backend-local cost-model line item for cold-tier scan bytes** —
removed (controller's tier-spanning cost model is the source of
Expand Down
324 changes: 0 additions & 324 deletions TODO.md

This file was deleted.

134 changes: 30 additions & 104 deletions data_plane/docs/README.md
Original file line number Diff line number Diff line change
@@ -1,114 +1,40 @@
# QueryEngineRust Developer Documentation
# ASAPQuery data-plane documentation

Welcome to the QueryEngineRust developer documentation! This directory contains guides for extending the system with new components.
The data plane ingests state produced under an active BackendPlan, stores that
state, answers supported PromQL queries, and uses an explicit exact fallback
for unsupported queries. It executes plans; it does not choose summary families
or re-plan queries.

## Architecture Overview
## Design documents

QueryEngineRust is organized into clear, extensible layers:
- [Plan-aware query execution](design_docs/query-execution.md) — ingestion,
routing, readiness, summary readout, and exact fallback contracts.
- [Repository-wide summary storage](../../docs/design_docs/summary-storage.md) —
materialization state, lifecycle, and query consistency.
- [BackendPlan](../../control_plane/docs/backend-plan.md) — the control-plane
contract installed by the data plane.

```
┌─────────────────────────────────────────────────────────┐
│ Client Applications │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Protocol Servers (HTTP, etc.) │
│ - Parse protocol-specific requests │
│ - Route to appropriate adapter │
│ - Handle protocol-specific endpoints │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Protocol Adapters (Prometheus, etc.) │
│ - Parse query language (PromQL, SQL, etc.) │
│ - Format responses for protocol │
│ - Determine if query is supported │
└─────────────────────────────────────────────────────────┘
┌──────┴──────┐
▼ ▼
┌─────────────────┐ ┌──────────────────┐
│ Query Engine │ │ Fallback Client │
│ - Execute │ │ - Forward │
│ queries │ │ unsupported │
│ - Return │ │ queries │
│ results │ │ │
└────────┬────────┘ └──────────────────┘
┌─────────────────┐
│ Store │
│ - Data storage │
│ - Sketches │
└─────────────────┘
┌────────┴────────┐
│ Ingest Drivers │
│ - Kafka, etc. │
└─────────────────┘
```
## Developer documentation

## Directory Structure
- [Extension boundaries](developer_docs/extension-points.md) — responsibilities
of protocol servers, adapters, and fallback clients.
- [Adding a summary family](../../docs/developer_docs/adding-summary-family.md) —
cross-repository prerequisites and backend validation.

```
src/drivers/
├── ingest/ # Data ingestion (Kafka, etc.)
├── query/
│ ├── adapters/ # Protocol adapters (Prometheus HTTP, etc.)
│ ├── fallback/ # Fallback backends (Prometheus, ClickHouse, etc.)
│ └── servers/ # Protocol servers (HTTP, Flight SQL, etc.)
```
## User guide

## Extension Guides
- [Querying ASAP](user_guide/querying-asap.md) — PromQL behavior, planned
summary execution, exact fallback, freshness, and errors.

- **[Adding a Protocol Adapter](./adding-protocol-adapter.md)** - Add support for new query protocols (e.g., ClickHouse HTTP API)
- **[Adding a Fallback Backend](./adding-fallback-backend.md)** - Add new fallback query backends (e.g., DuckDB, Elasticsearch)
- **[Adding a Protocol Server](./adding-protocol-server.md)** - Add new protocol servers (e.g., Flight SQL, gRPC)
## Ownership

## Key Concepts
- [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) owns logical query
planning, query-to-summary mapping, and accuracy reasoning.
- The ASAPQuery control plane owns physical compilation and BackendPlan.
- The data plane owns ingestion, storage, readout, query execution, and exact
fallback under the installed plan.
- [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector) owns summary
construction and transmission at the edge.

### Protocol Adapter
Handles protocol-specific request/response formatting and query parsing. Examples: Prometheus HTTP API, ClickHouse HTTP API.

### Fallback Backend
External query system to forward unsupported queries to. Examples: Prometheus, ClickHouse, DuckDB.

### Protocol Server
Handles network communication for a specific protocol. Examples: HTTP server, Flight SQL server.

## Quick Reference

### Adding a Protocol Adapter
1. Create `src/drivers/query/adapters/my_adapter.rs`
2. Implement `HttpProtocolAdapter` trait
3. Add to factory in `factory.rs`
4. Update `QueryProtocol` enum

### Adding a Fallback Backend
1. Create `src/drivers/query/fallback/my_backend.rs`
2. Implement `FallbackClient` trait
3. Export from `fallback/mod.rs`

### Adding a Protocol Server
1. Create `src/drivers/query/servers/my_server.rs`
2. Implement server logic with appropriate adapter
3. Export from `servers/mod.rs`

## Testing

Each component should include:
- Unit tests in the same file
- Integration tests in `src/tests/`
- Example usage in documentation

## Contributing

When adding new components:
1. Follow existing naming conventions
2. Add comprehensive documentation
3. Include tests
4. Update this documentation
5. Keep backward compatibility
Historical ingestion paths, file-by-file migrations, and configuration
walkthroughs are not data-plane design contracts.
117 changes: 0 additions & 117 deletions data_plane/docs/adding-fallback-backend.md

This file was deleted.

Loading