feat(capability): Phase 5 — QueryEngine trait + StorageBackend routing for Gorilla-S3 - #86
Merged
Merged
Conversation
…sap-gorilla (Phase 3) Phase 3 of the Gorilla-S3-cold-engine — adds a `ColdStore` adapter that fetches per-hour `index.json` catalogs from an S3-compatible bucket, prunes them by time range, and decodes the selected `GORILLA1` chunks via the freshly-merged `asap-gorilla` crate (ASAPCollector PR #281). Changes ------- * `cold_store/mod.rs` — additive trait extension. New `ChunkRef` descriptor + `list_chunks` / `read_chunk` methods carry default impls returning `ColdStoreError::Unsupported`, so `LocalFsColdStore` and the existing `s3_adapter::ColdFallback` chain compile unchanged. Two new `ColdStoreError` variants (`Backend(String)` for transport failures, `Unsupported(&'static str)` for the default-impl errors). * `cold_store/gorilla_s3.rs` — new module. `GorillaS3ColdStore` + `GorillaS3Config` (with `from_env`) + `ObjectStore` trait + a `rust-s3`-backed production impl (`S3ObjectStore`, with MinIO path-style support) and an in-memory mock used by the tests. LRU cache (default 256 chunks) keyed on the chunk object key, holding pre-decoded `RawSample` lists so repeated reads skip the Gorilla decode pass entirely. * `asap-query-engine/Cargo.toml` — new deps: `asap-gorilla` (path), `rust-s3 = 0.37` (default-features off, `tokio-rustls-tls` to share the rustls backend reqwest already pulls in), `lru = 0.12`. S3 client choice ---------------- Picked `rust-s3` over `aws-sdk-s3` for the lighter dep tree (no full AWS SDK fanout) and first-class MinIO support (`with_path_style()` is the supported config rather than a workaround). The trait-based `ObjectStore` indirection means the choice is replaceable without touching the `ColdStore` impl. Tests (10, all green) --------------------- * `list_chunks_via_indexfile_prunes_by_time` — hand-crafted index with three chunks; window overlaps only the middle one. * `read_chunk_decodes_via_asap_gorilla` — round-trip through a real `GorillaEncoder` block; samples + labels match. * `cache_hit_skips_s3_fetch` — second `read_chunk` on the same key must not trigger another S3 GET (mock counter assertion). * `lru_eviction_under_pressure` — `cache_capacity = 2`, fill three, re-read the first — must trigger a fresh GET. * `index_json_corrupted_returns_error` — clean `Malformed`, no panic. * `s3_unavailable_returns_error` — clean `Backend(...)`, no panic. * `missing_index_is_empty_not_error` — producer hasn't flushed yet; empty result, not error. * `scan_filters_to_requested_range` — chunk overlaps but inner samples filter to zero. * `list_chunks_spans_two_hour_buckets` — request crosses an hour boundary; both hour indexes are fetched. * `from_env_requires_bucket` — config validation surfaces missing env. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s (Phase 4)
Phase 4 of the Gorilla-S3-cold-engine — adds a sibling engine to
`SimpleEngine` that executes PromQL exactly against the Phase-3
`GorillaS3ColdStore`. Returns answers carrying an
`AccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }` and a
`data_source: gorilla_archive` info marker so callers /
dashboards can distinguish cold-tier exact answers from warm-tier
sketch answers.
Architecture
------------
`asap-query-engine/src/engines/gorilla_engine/` (new):
* `mod.rs` — `GorillaQueryEngine` (holds `Arc<dyn ColdStore>` so
tests can inject mocks; production constructor
`with_gorilla_s3` keeps the design.md type signature). Public
surface: `execute(query)` + `execute_at(query, now_ms)`. Wraps
results via `wrap_result` (exact accuracy envelope + window).
`GorillaEngineConfig { max_buffered_samples, query_timeout_secs }`
with sensible defaults (10M / 30s). `EngineError` enum covers
Plan / ColdStore / TooManySamples / Timeout. `ExecutionOutcome`
carries `(value, samples_scanned, chunks_fetched)` + an
`info_lines` builder pinning the on-wire info strings.
* `query_planner.rs` — minimal PromQL → `QueryPlan` translator
(metric, half-open `[start_ms, end_ms)`, `QueryStatistic`).
Supports `sum/count/avg/min/max_over_time`, `rate`, `increase`,
`quantile_over_time(φ, m[range])`, and `topk(k, <vector>)`
(PromQL grammar requires the inner be a vector so `topk(k,
sum_over_time(m[range]))` is the legal spelling). Tests +
caller can use `plan_query_at` to pin `now_ms`.
* `exact_executor.rs` — `ExactExecutor` dispatches per
`QueryStatistic`:
- **Streaming additive** (`Sum/Count/Avg/Min/Max/Rate/Increase`):
`list_chunks` → for each chunk `read_chunk` → fold into a
bounded `AdditiveAccumulator` → drop the decoded chunk
before fetching the next. Memory is O(1) per query. Rate /
Increase track first/last `(ts, value)` and divide by
`range_seconds` at finalisation.
- **Buffered** (`Quantile / TopK`): `collect_buffered_samples`
materialises every in-range sample up to
`max_buffered_samples` (errors with `TooManySamples`
otherwise). Quantile sorts + nearest-rank index; TopK sorts
descending and returns sum of the top-k values.
* `tests.rs` — 20 unit tests via an in-process `MockColdStore`
satisfying `ColdStore` (no S3 / disk dependency). Pinned NOW
via `execute_at` so time math is deterministic. Covers every
`QueryStatistic` happy path, the empty-data sentinel, the
outside-range filter, the buffered-budget guard, the result
wrapping (accuracy envelope + `data_source: gorilla_archive`
marker), and the timeout path.
Test additions (15 spec'd + 5 planner)
--------------------------------------
* `execute_sum_over_time_streaming`
* `execute_count_over_time`
* `execute_avg_over_time`
* `execute_min_over_time` / `execute_max_over_time`
* `execute_rate_basic`
* `execute_increase_basic`
* `execute_quantile_buffered_basic`
* `execute_quantile_too_many_samples_errors`
* `execute_topk_basic`
* `execute_empty_chunks_returns_zero_or_nan`
* `execute_chunks_partially_outside_range_filtered`
* `result_carries_exact_accuracy_envelope`
* `result_includes_data_source_gorilla_archive`
* `engine_respects_config_timeout`
* `query_planner::tests::{plans_sum_over_time, plans_quantile_over_time,
plans_topk, rejects_binary_expression, streaming_classification}`
Touches only the new directory + `engines/mod.rs` (re-exports).
Zero changes to `simple_engine.rs`, the cold store, or the
sketch warm-tier. No new Cargo deps — `asap-gorilla` / `tokio`
were already pulled in by PR #84.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…g for Gorilla-S3
Adds the storage-backend axis to capability matching so queries against
metrics configured for Gorilla-S3 dispatch to GorillaQueryEngine instead
of the warm-tier SimpleEngine. Per docs/design-gorilla-s3-cold-engine.md
§8.
asap_types extensions:
- StorageBackend enum (SketchWarmTier default + GorillaS3Archive +
ColdJsonlFallback + DoubleWrite) with snake_case serde + #[serde(default)]
so pre-Phase-5 configs decode unchanged.
- AccuracyTarget enum (Exact / Approximate, default Approximate).
- compatible_storage_backends(stat, accuracy, metric_storage_config)
returns the ordered failover list the router walks. Exact-on-archive
subsumes approximate-on-warm (Gorilla-only metrics still archive).
- StreamingConfig.storage_backend field with serde-default + builder
constructor (with_storage_backend) + getter.
asap-query-engine extensions (additive):
- engines::EngineError envelope (CapabilityMiss vs Backend) the trait
returns.
- engines::router::QueryEngine async trait + EngineCapabilities struct.
- engines::router::EngineRouter dispatcher: consults
compatible_storage_backends, dispatches to first registered engine,
falls through on Backend / CapabilityMiss to the next compatible
backend (typically ColdJsonlFallback).
- impl QueryEngine for SimpleEngine (adapts handle_query → Result;
None → CapabilityMiss).
- impl QueryEngine for GorillaQueryEngine (Plan errors fold to
CapabilityMiss; ColdStore / Timeout / TooManySamples to Backend).
Tests added (16 new):
- capability_matching: 6 routing tests + 1 source-of-truth agreement
test enumerating (Statistic × AccuracyTarget × StorageBackend).
- streaming_config: 2 serde tests pinning legacy + Phase-5 wire format.
- engines::router: 6 tests (warm-tier dispatch, archive dispatch,
JSONL fallback when archive+warm fail, no-engines clean error,
all-failed surface, hot-swap re-registration).
Test-utility struct-literal sites updated to include the new
StreamingConfig.storage_backend field. No precompute_engine/, stores/, or
precompute_operators/ touched.
Verified clean: cargo build --release -p {asap_types, query_engine_rust};
cargo test -p asap_types; cargo test --release -p query_engine_rust router;
cargo clippy --release -p asap_types --all-targets -- -D warnings.
Pre-existing query_engine_rust clippy/test failures unchanged (verified
via git-stash diff).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
asap_types::capability_matchingso queries against metrics configured for Gorilla-S3 dispatch toGorillaQueryEngineinstead of the warm-tierSimpleEngine. Perdocs/design-gorilla-s3-cold-engine.md§8.QueryEngineasync trait so the newEngineRoutercan holdArc<dyn QueryEngine>rather than case-on-concrete; bothSimpleEngineandGorillaQueryEngineget an additiveimpl QueryEnginethat does NOT change existing methods.StreamingConfig.storage_backend(with#[serde(default)]) so the controller can push per-metric backend pins; pre-Phase-5 configs decode unchanged.Deliverables
QueryEnginetrait +EngineCapabilitiesasap-query-engine/src/engines/router.rsStorageBackendenum (4 variants) +AccuracyTargetenumasap-common/dependencies/rs/asap_types/src/capability_matching.rscompatible_storage_backends(stat, accuracy, metric_storage_config) -> Vec<StorageBackend>EngineRouter(HashMap bydata_source_id, register / execute / failover)asap-query-engine/src/engines/router.rsStreamingConfig.storage_backendwith#[serde(default)]asap-common/dependencies/rs/asap_types/src/streaming_config.rsFile domain
Touched only the files listed under "Touch ONLY" in the brief, plus the test-utility struct-literal sites (
tests/test_utilities/{config_builders,engine_factories}.rs,tests/{capability_matching_tests,sql_pattern_matching_tests}.rs,tests/inference_yaml_pattern_coverage.rs) that needed the new struct field. Noprecompute_engine/,stores/, orprecompute_operators/modified. No submodule pointer drift.Routing matrix
compatible_storage_backends:GorillaS3Archive[GorillaS3Archive](exact subsumes approximate)SketchWarmTier(default)[SketchWarmTier, ColdJsonlFallback]DoubleWriteExact[GorillaS3Archive, SketchWarmTier, ColdJsonlFallback]DoubleWriteApproximate[SketchWarmTier, GorillaS3Archive, ColdJsonlFallback]ColdJsonlFallback[ColdJsonlFallback]EngineRouter::executewalks this list in order, dispatches to first registered engine, falls through onEngineError::BackendorEngineError::CapabilityMiss. ReturnsEngineRouterError::NoEngineRegisteredif nothing in the list has an engine;EngineRouterError::AllFailedif every engine errored.Test plan
cargo build --release -p asap_typescleancargo build --release -p query_engine_rustcleancargo test -p asap_types capability_matching— 36 pass (was 30; +6 routing)cargo test -p asap_types streaming_config— 2 pass (new)cargo test --release -p query_engine_rust router— 6 router tests passcargo test --release -p query_engine_rust gorilla_engine— 20 pass (no regression)cargo test --release -p query_engine_rust capability_matching_tests— 6 pass (no regression)cargo clippy --release -p asap_types --all-targets -- -D warningscleanquery_engine_rustclippy errors and 35 lib-test failures verified unchanged via git-stash baseline diff (failure set identical; +6 new passing tests for the router)Open questions / follow-ups
asap-query-engine/src/drivers/http/...) andmain.rsstill constructSimpleEnginedirectly and callhandle_query— they don't yet consume the newEngineRouter. Wiring the router into the HTTP-driver dispatch (so production traffic routes throughcompatible_storage_backends) is a Phase-6 / follow-up PR, intentionally out of scope here per "Open question" in the brief.compatible_storage_backendscurrently ignoresStatistic(no statistic is special-cased). Once Phase-6 adds statistic-specific routing rules (e.g. archive-only forTopkto avoid sketch error), the test matrix already enumerates every(stat, accuracy, backend)triple and will catch divergence.StorageBackend; this PR ships 4 variants (adds explicitDoubleWrite) — small doc-tightening to land alongside the deploy YAML overlay (§8.4).