Skip to content

feat(capability): Phase 5 — QueryEngine trait + StorageBackend routing for Gorilla-S3 - #86

Merged
zzylol merged 3 commits into
mainfrom
feat/gorilla-s3-capability-routing-phase-5
May 6, 2026
Merged

zzylol merged 3 commits into
mainfrom
feat/gorilla-s3-capability-routing-phase-5

Conversation

@zzylol

@zzylol zzylol commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds the storage-backend axis to asap_types::capability_matching so queries against metrics configured for Gorilla-S3 dispatch to GorillaQueryEngine instead of the warm-tier SimpleEngine. Per docs/design-gorilla-s3-cold-engine.md §8.
  • Introduces a small QueryEngine async trait so the new EngineRouter can hold Arc<dyn QueryEngine> rather than case-on-concrete; both SimpleEngine and GorillaQueryEngine get an additive impl QueryEngine that does NOT change existing methods.
  • Wires StreamingConfig.storage_backend (with #[serde(default)]) so the controller can push per-metric backend pins; pre-Phase-5 configs decode unchanged.

Deliverables

# Item Where
1 QueryEngine trait + EngineCapabilities asap-query-engine/src/engines/router.rs
2 StorageBackend enum (4 variants) + AccuracyTarget enum asap-common/dependencies/rs/asap_types/src/capability_matching.rs
3 compatible_storage_backends(stat, accuracy, metric_storage_config) -> Vec<StorageBackend> same file; ordered failover list
4 EngineRouter (HashMap by data_source_id, register / execute / failover) asap-query-engine/src/engines/router.rs
5 16 new tests (6 capability + 2 streaming_config serde + 6 router + 1 agreement + 1 default-pin) tests in touched files
6 StreamingConfig.storage_backend with #[serde(default)] asap-common/dependencies/rs/asap_types/src/streaming_config.rs

File domain

Touched only the files listed under "Touch ONLY" in the brief, plus the test-utility struct-literal sites (tests/test_utilities/{config_builders,engine_factories}.rs, tests/{capability_matching_tests,sql_pattern_matching_tests}.rs, tests/inference_yaml_pattern_coverage.rs) that needed the new struct field. No precompute_engine/, stores/, or precompute_operators/ modified. No submodule pointer drift.

Routing matrix

compatible_storage_backends:

metric_storage_config accuracy result
GorillaS3Archive any [GorillaS3Archive] (exact subsumes approximate)
SketchWarmTier (default) any [SketchWarmTier, ColdJsonlFallback]
DoubleWrite Exact [GorillaS3Archive, SketchWarmTier, ColdJsonlFallback]
DoubleWrite Approximate [SketchWarmTier, GorillaS3Archive, ColdJsonlFallback]
ColdJsonlFallback any [ColdJsonlFallback]

EngineRouter::execute walks this list in order, dispatches to first registered engine, falls through on EngineError::Backend or EngineError::CapabilityMiss. Returns EngineRouterError::NoEngineRegistered if nothing in the list has an engine; EngineRouterError::AllFailed if every engine errored.

Test plan

  • cargo build --release -p asap_types clean
  • cargo build --release -p query_engine_rust clean
  • cargo test -p asap_types capability_matching — 36 pass (was 30; +6 routing)
  • cargo test -p asap_types streaming_config — 2 pass (new)
  • cargo test --release -p query_engine_rust router — 6 router tests pass
  • cargo test --release -p query_engine_rust gorilla_engine — 20 pass (no regression)
  • cargo test --release -p query_engine_rust capability_matching_tests — 6 pass (no regression)
  • cargo clippy --release -p asap_types --all-targets -- -D warnings clean
  • Pre-existing query_engine_rust clippy errors and 35 lib-test failures verified unchanged via git-stash baseline diff (failure set identical; +6 new passing tests for the router)

Open questions / follow-ups

  • The HTTP driver (asap-query-engine/src/drivers/http/...) and main.rs still construct SimpleEngine directly and call handle_query — they don't yet consume the new EngineRouter. Wiring the router into the HTTP-driver dispatch (so production traffic routes through compatible_storage_backends) is a Phase-6 / follow-up PR, intentionally out of scope here per "Open question" in the brief.
  • compatible_storage_backends currently ignores Statistic (no statistic is special-cased). Once Phase-6 adds statistic-specific routing rules (e.g. archive-only for Topk to avoid sketch error), the test matrix already enumerates every (stat, accuracy, backend) triple and will catch divergence.
  • Design doc §8 currently sketches a 3-variant StorageBackend; this PR ships 4 variants (adds explicit DoubleWrite) — small doc-tightening to land alongside the deploy YAML overlay (§8.4).

zzylol and others added 3 commits May 6, 2026 00:49
…sap-gorilla (Phase 3)

Phase 3 of the Gorilla-S3-cold-engine — adds a `ColdStore` adapter that
fetches per-hour `index.json` catalogs from an S3-compatible bucket,
prunes them by time range, and decodes the selected `GORILLA1` chunks
via the freshly-merged `asap-gorilla` crate (ASAPCollector PR #281).

Changes
-------

* `cold_store/mod.rs` — additive trait extension. New `ChunkRef`
  descriptor + `list_chunks` / `read_chunk` methods carry default
  impls returning `ColdStoreError::Unsupported`, so `LocalFsColdStore`
  and the existing `s3_adapter::ColdFallback` chain compile unchanged.
  Two new `ColdStoreError` variants (`Backend(String)` for transport
  failures, `Unsupported(&'static str)` for the default-impl errors).

* `cold_store/gorilla_s3.rs` — new module. `GorillaS3ColdStore`
  + `GorillaS3Config` (with `from_env`) + `ObjectStore` trait + a
  `rust-s3`-backed production impl (`S3ObjectStore`, with MinIO
  path-style support) and an in-memory mock used by the tests. LRU
  cache (default 256 chunks) keyed on the chunk object key, holding
  pre-decoded `RawSample` lists so repeated reads skip the Gorilla
  decode pass entirely.

* `asap-query-engine/Cargo.toml` — new deps: `asap-gorilla` (path),
  `rust-s3 = 0.37` (default-features off, `tokio-rustls-tls` to share
  the rustls backend reqwest already pulls in), `lru = 0.12`.

S3 client choice
----------------

Picked `rust-s3` over `aws-sdk-s3` for the lighter dep tree (no full
AWS SDK fanout) and first-class MinIO support (`with_path_style()` is
the supported config rather than a workaround). The trait-based
`ObjectStore` indirection means the choice is replaceable without
touching the `ColdStore` impl.

Tests (10, all green)
---------------------

* `list_chunks_via_indexfile_prunes_by_time` — hand-crafted index
  with three chunks; window overlaps only the middle one.
* `read_chunk_decodes_via_asap_gorilla` — round-trip through a real
  `GorillaEncoder` block; samples + labels match.
* `cache_hit_skips_s3_fetch` — second `read_chunk` on the same key
  must not trigger another S3 GET (mock counter assertion).
* `lru_eviction_under_pressure` — `cache_capacity = 2`, fill three,
  re-read the first — must trigger a fresh GET.
* `index_json_corrupted_returns_error` — clean `Malformed`, no panic.
* `s3_unavailable_returns_error` — clean `Backend(...)`, no panic.
* `missing_index_is_empty_not_error` — producer hasn't flushed yet;
  empty result, not error.
* `scan_filters_to_requested_range` — chunk overlaps but inner
  samples filter to zero.
* `list_chunks_spans_two_hour_buckets` — request crosses an hour
  boundary; both hour indexes are fetched.
* `from_env_requires_bucket` — config validation surfaces missing env.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s (Phase 4)

Phase 4 of the Gorilla-S3-cold-engine — adds a sibling engine to
`SimpleEngine` that executes PromQL exactly against the Phase-3
`GorillaS3ColdStore`. Returns answers carrying an
`AccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }` and a
`data_source: gorilla_archive` info marker so callers /
dashboards can distinguish cold-tier exact answers from warm-tier
sketch answers.

Architecture
------------

`asap-query-engine/src/engines/gorilla_engine/` (new):

* `mod.rs` — `GorillaQueryEngine` (holds `Arc<dyn ColdStore>` so
  tests can inject mocks; production constructor
  `with_gorilla_s3` keeps the design.md type signature). Public
  surface: `execute(query)` + `execute_at(query, now_ms)`. Wraps
  results via `wrap_result` (exact accuracy envelope + window).
  `GorillaEngineConfig { max_buffered_samples, query_timeout_secs }`
  with sensible defaults (10M / 30s). `EngineError` enum covers
  Plan / ColdStore / TooManySamples / Timeout. `ExecutionOutcome`
  carries `(value, samples_scanned, chunks_fetched)` + an
  `info_lines` builder pinning the on-wire info strings.

* `query_planner.rs` — minimal PromQL → `QueryPlan` translator
  (metric, half-open `[start_ms, end_ms)`, `QueryStatistic`).
  Supports `sum/count/avg/min/max_over_time`, `rate`, `increase`,
  `quantile_over_time(φ, m[range])`, and `topk(k, <vector>)`
  (PromQL grammar requires the inner be a vector so `topk(k,
  sum_over_time(m[range]))` is the legal spelling). Tests +
  caller can use `plan_query_at` to pin `now_ms`.

* `exact_executor.rs` — `ExactExecutor` dispatches per
  `QueryStatistic`:
  - **Streaming additive** (`Sum/Count/Avg/Min/Max/Rate/Increase`):
    `list_chunks` → for each chunk `read_chunk` → fold into a
    bounded `AdditiveAccumulator` → drop the decoded chunk
    before fetching the next. Memory is O(1) per query. Rate /
    Increase track first/last `(ts, value)` and divide by
    `range_seconds` at finalisation.
  - **Buffered** (`Quantile / TopK`): `collect_buffered_samples`
    materialises every in-range sample up to
    `max_buffered_samples` (errors with `TooManySamples`
    otherwise). Quantile sorts + nearest-rank index; TopK sorts
    descending and returns sum of the top-k values.

* `tests.rs` — 20 unit tests via an in-process `MockColdStore`
  satisfying `ColdStore` (no S3 / disk dependency). Pinned NOW
  via `execute_at` so time math is deterministic. Covers every
  `QueryStatistic` happy path, the empty-data sentinel, the
  outside-range filter, the buffered-budget guard, the result
  wrapping (accuracy envelope + `data_source: gorilla_archive`
  marker), and the timeout path.

Test additions (15 spec'd + 5 planner)
--------------------------------------

* `execute_sum_over_time_streaming`
* `execute_count_over_time`
* `execute_avg_over_time`
* `execute_min_over_time` / `execute_max_over_time`
* `execute_rate_basic`
* `execute_increase_basic`
* `execute_quantile_buffered_basic`
* `execute_quantile_too_many_samples_errors`
* `execute_topk_basic`
* `execute_empty_chunks_returns_zero_or_nan`
* `execute_chunks_partially_outside_range_filtered`
* `result_carries_exact_accuracy_envelope`
* `result_includes_data_source_gorilla_archive`
* `engine_respects_config_timeout`
* `query_planner::tests::{plans_sum_over_time, plans_quantile_over_time,
   plans_topk, rejects_binary_expression, streaming_classification}`

Touches only the new directory + `engines/mod.rs` (re-exports).
Zero changes to `simple_engine.rs`, the cold store, or the
sketch warm-tier. No new Cargo deps — `asap-gorilla` / `tokio`
were already pulled in by PR #84.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…g for Gorilla-S3

Adds the storage-backend axis to capability matching so queries against
metrics configured for Gorilla-S3 dispatch to GorillaQueryEngine instead
of the warm-tier SimpleEngine. Per docs/design-gorilla-s3-cold-engine.md
§8.

asap_types extensions:
  - StorageBackend enum (SketchWarmTier default + GorillaS3Archive +
    ColdJsonlFallback + DoubleWrite) with snake_case serde + #[serde(default)]
    so pre-Phase-5 configs decode unchanged.
  - AccuracyTarget enum (Exact / Approximate, default Approximate).
  - compatible_storage_backends(stat, accuracy, metric_storage_config)
    returns the ordered failover list the router walks. Exact-on-archive
    subsumes approximate-on-warm (Gorilla-only metrics still archive).
  - StreamingConfig.storage_backend field with serde-default + builder
    constructor (with_storage_backend) + getter.

asap-query-engine extensions (additive):
  - engines::EngineError envelope (CapabilityMiss vs Backend) the trait
    returns.
  - engines::router::QueryEngine async trait + EngineCapabilities struct.
  - engines::router::EngineRouter dispatcher: consults
    compatible_storage_backends, dispatches to first registered engine,
    falls through on Backend / CapabilityMiss to the next compatible
    backend (typically ColdJsonlFallback).
  - impl QueryEngine for SimpleEngine (adapts handle_query → Result;
    None → CapabilityMiss).
  - impl QueryEngine for GorillaQueryEngine (Plan errors fold to
    CapabilityMiss; ColdStore / Timeout / TooManySamples to Backend).

Tests added (16 new):
  - capability_matching: 6 routing tests + 1 source-of-truth agreement
    test enumerating (Statistic × AccuracyTarget × StorageBackend).
  - streaming_config: 2 serde tests pinning legacy + Phase-5 wire format.
  - engines::router: 6 tests (warm-tier dispatch, archive dispatch,
    JSONL fallback when archive+warm fail, no-engines clean error,
    all-failed surface, hot-swap re-registration).

Test-utility struct-literal sites updated to include the new
StreamingConfig.storage_backend field. No precompute_engine/, stores/, or
precompute_operators/ touched.

Verified clean: cargo build --release -p {asap_types, query_engine_rust};
cargo test -p asap_types; cargo test --release -p query_engine_rust router;
cargo clippy --release -p asap_types --all-targets -- -D warnings.
Pre-existing query_engine_rust clippy/test failures unchanged (verified
via git-stash diff).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 0a0e607 into main May 6, 2026
@zzylol
zzylol deleted the feat/gorilla-s3-capability-routing-phase-5 branch May 6, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant