Skip to content

feat(engine): GorillaQueryEngine — exact PromQL over Gorilla-S3 chunks (Phase 4) - #85

Merged
zzylol merged 1 commit into
feat/gorilla-s3-cold-store-phase-3from
feat/gorilla-query-engine-phase-4
May 6, 2026
Merged

zzylol merged 1 commit into
feat/gorilla-s3-cold-store-phase-3from
feat/gorilla-query-engine-phase-4

Conversation

@zzylol

@zzylol zzylol commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 4 of the Gorilla-S3-cold-engine: a sibling of SimpleEngine that
executes PromQL exactly against the Phase-3 GorillaS3ColdStore
(merged in PR #84). Same PromQL surface, different backend — instead of
warm-tier sketches with ε/δ bounds, this engine reads decoded
GORILLA1 chunks straight off S3 and folds them into the answer with
AccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }.

Architecture

asap-query-engine/src/engines/gorilla_engine/ (new directory, 4 files):

  • mod.rsGorillaQueryEngine (holds Arc<dyn ColdStore> so
    tests can inject mocks; production constructor with_gorilla_s3
    keeps the design.md type signature). Public surface: execute(query)

    • execute_at(query, now_ms). wrap_result attaches the exact
      accuracy envelope + the request window. GorillaEngineConfig with
      max_buffered_samples (default 10M) + query_timeout_secs (default
      30s). EngineError::{Plan, ColdStore, TooManySamples, Timeout}.
      ExecutionOutcome carries (value, samples_scanned, chunks_fetched)
    • info_lines() builder pinning the on-wire info strings.
  • query_planner.rs — minimal PromQL → QueryPlan translator.
    Supports sum/count/avg/min/max_over_time, rate, increase,
    quantile_over_time(φ, m[range]), and topk(k, <vector>) (PromQL
    grammar requires the inner be a vector so topk(k, sum_over_time(m[range])) is the legal spelling).

  • exact_executor.rsExactExecutor dispatches per
    QueryStatistic:

    • Streaming-additive (Sum / Count / Avg / Min / Max / Rate /
      Increase): walks chunks one at a time, folds into a bounded
      AdditiveAccumulator, drops the decoded chunk before
      fetching the next. Memory cost is O(1) per query.
    • Buffered (Quantile / TopK): materialises every in-range
      sample up to max_buffered_samples; over-budget queries fail
      fast with EngineError::TooManySamples rather than OOM.
  • tests.rs — 20 unit tests via an in-process MockColdStore.

Result wrapping

Every result carries:

  • AccuracyEnvelope::single(AccuracyProfile::exact()) (ε = 0, δ = 0,
    kind = Exact),
  • data_source: gorilla_archive info marker,
  • samples_scanned / chunks_fetched diagnostics.

The data_source: gorilla_archive line is exposed via
ExecutionOutcome::info_lines() so the HTTP driver layer can surface
it on the Prometheus infos array (matching the existing
SimpleEngine convention).

Test plan

  • cargo build --release -p query_engine_rust clean
  • cargo test --release -p query_engine_rust gorilla_engine
    20 / 20 pass
  • cargo clippy --release -p query_engine_rust --all-targets — 0
    warnings on the new module (5 pre-existing warnings in unrelated
    files; not touched per scope guard)
  • All 15 spec'd test names present:
    • execute_sum_over_time_streaming,
      execute_count_over_time, execute_avg_over_time,
      execute_min_over_time, execute_max_over_time,
      execute_rate_basic, execute_increase_basic,
      execute_quantile_buffered_basic,
      execute_quantile_too_many_samples_errors,
      execute_topk_basic,
      execute_empty_chunks_returns_zero_or_nan,
      execute_chunks_partially_outside_range_filtered,
      result_carries_exact_accuracy_envelope,
      result_includes_data_source_gorilla_archive,
      engine_respects_config_timeout
  • Touches only the new engines/gorilla_engine/ dir + the engines
    mod.rs re-export (no simple_engine.rs, no cold-store, no
    sketch warm-tier).
  • Zero docker. Zero new Cargo deps.

Open question for Phase 5

GorillaQueryEngine and SimpleEngine are now sibling engines with
disjoint surface — Phase 5's capability router needs a hook to choose
between them per query. The engine exposes config() for the cost
estimator; a follow-up trait abstraction (pub trait QueryEngine)
would let the router hold an Arc<dyn QueryEngine> and dispatch
without case-on-concrete. Deferred to Phase 5 per scope guard.

Doc tightening for design.md §6

The data_source marker in §6.6 of design-gorilla-s3-cold-engine.md
reads gorilla-s3; this PR ships gorilla_archive per the Phase 4
spec (snake_case + the "archive" tier name preferred by the controller
side). The doc lives in the ASAPCollector repo and is out of scope
for this PR — a follow-up doc PR should align the string.

🤖 Generated with Claude Code

…s (Phase 4)

Phase 4 of the Gorilla-S3-cold-engine — adds a sibling engine to
`SimpleEngine` that executes PromQL exactly against the Phase-3
`GorillaS3ColdStore`. Returns answers carrying an
`AccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }` and a
`data_source: gorilla_archive` info marker so callers /
dashboards can distinguish cold-tier exact answers from warm-tier
sketch answers.

Architecture
------------

`asap-query-engine/src/engines/gorilla_engine/` (new):

* `mod.rs` — `GorillaQueryEngine` (holds `Arc<dyn ColdStore>` so
  tests can inject mocks; production constructor
  `with_gorilla_s3` keeps the design.md type signature). Public
  surface: `execute(query)` + `execute_at(query, now_ms)`. Wraps
  results via `wrap_result` (exact accuracy envelope + window).
  `GorillaEngineConfig { max_buffered_samples, query_timeout_secs }`
  with sensible defaults (10M / 30s). `EngineError` enum covers
  Plan / ColdStore / TooManySamples / Timeout. `ExecutionOutcome`
  carries `(value, samples_scanned, chunks_fetched)` + an
  `info_lines` builder pinning the on-wire info strings.

* `query_planner.rs` — minimal PromQL → `QueryPlan` translator
  (metric, half-open `[start_ms, end_ms)`, `QueryStatistic`).
  Supports `sum/count/avg/min/max_over_time`, `rate`, `increase`,
  `quantile_over_time(φ, m[range])`, and `topk(k, <vector>)`
  (PromQL grammar requires the inner be a vector so `topk(k,
  sum_over_time(m[range]))` is the legal spelling). Tests +
  caller can use `plan_query_at` to pin `now_ms`.

* `exact_executor.rs` — `ExactExecutor` dispatches per
  `QueryStatistic`:
  - **Streaming additive** (`Sum/Count/Avg/Min/Max/Rate/Increase`):
    `list_chunks` → for each chunk `read_chunk` → fold into a
    bounded `AdditiveAccumulator` → drop the decoded chunk
    before fetching the next. Memory is O(1) per query. Rate /
    Increase track first/last `(ts, value)` and divide by
    `range_seconds` at finalisation.
  - **Buffered** (`Quantile / TopK`): `collect_buffered_samples`
    materialises every in-range sample up to
    `max_buffered_samples` (errors with `TooManySamples`
    otherwise). Quantile sorts + nearest-rank index; TopK sorts
    descending and returns sum of the top-k values.

* `tests.rs` — 20 unit tests via an in-process `MockColdStore`
  satisfying `ColdStore` (no S3 / disk dependency). Pinned NOW
  via `execute_at` so time math is deterministic. Covers every
  `QueryStatistic` happy path, the empty-data sentinel, the
  outside-range filter, the buffered-budget guard, the result
  wrapping (accuracy envelope + `data_source: gorilla_archive`
  marker), and the timeout path.

Test additions (15 spec'd + 5 planner)
--------------------------------------

* `execute_sum_over_time_streaming`
* `execute_count_over_time`
* `execute_avg_over_time`
* `execute_min_over_time` / `execute_max_over_time`
* `execute_rate_basic`
* `execute_increase_basic`
* `execute_quantile_buffered_basic`
* `execute_quantile_too_many_samples_errors`
* `execute_topk_basic`
* `execute_empty_chunks_returns_zero_or_nan`
* `execute_chunks_partially_outside_range_filtered`
* `result_carries_exact_accuracy_envelope`
* `result_includes_data_source_gorilla_archive`
* `engine_respects_config_timeout`
* `query_planner::tests::{plans_sum_over_time, plans_quantile_over_time,
   plans_topk, rejects_binary_expression, streaming_classification}`

Touches only the new directory + `engines/mod.rs` (re-exports).
Zero changes to `simple_engine.rs`, the cold store, or the
sketch warm-tier. No new Cargo deps — `asap-gorilla` / `tokio`
were already pulled in by PR #84.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 6ce0fed into feat/gorilla-s3-cold-store-phase-3 May 6, 2026
@zzylol
zzylol deleted the feat/gorilla-query-engine-phase-4 branch May 9, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant