feat(engine): GorillaQueryEngine — exact PromQL over Gorilla-S3 chunks (Phase 4) - #85
Merged
zzylol merged 1 commit intoMay 6, 2026
Conversation
…s (Phase 4)
Phase 4 of the Gorilla-S3-cold-engine — adds a sibling engine to
`SimpleEngine` that executes PromQL exactly against the Phase-3
`GorillaS3ColdStore`. Returns answers carrying an
`AccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }` and a
`data_source: gorilla_archive` info marker so callers /
dashboards can distinguish cold-tier exact answers from warm-tier
sketch answers.
Architecture
------------
`asap-query-engine/src/engines/gorilla_engine/` (new):
* `mod.rs` — `GorillaQueryEngine` (holds `Arc<dyn ColdStore>` so
tests can inject mocks; production constructor
`with_gorilla_s3` keeps the design.md type signature). Public
surface: `execute(query)` + `execute_at(query, now_ms)`. Wraps
results via `wrap_result` (exact accuracy envelope + window).
`GorillaEngineConfig { max_buffered_samples, query_timeout_secs }`
with sensible defaults (10M / 30s). `EngineError` enum covers
Plan / ColdStore / TooManySamples / Timeout. `ExecutionOutcome`
carries `(value, samples_scanned, chunks_fetched)` + an
`info_lines` builder pinning the on-wire info strings.
* `query_planner.rs` — minimal PromQL → `QueryPlan` translator
(metric, half-open `[start_ms, end_ms)`, `QueryStatistic`).
Supports `sum/count/avg/min/max_over_time`, `rate`, `increase`,
`quantile_over_time(φ, m[range])`, and `topk(k, <vector>)`
(PromQL grammar requires the inner be a vector so `topk(k,
sum_over_time(m[range]))` is the legal spelling). Tests +
caller can use `plan_query_at` to pin `now_ms`.
* `exact_executor.rs` — `ExactExecutor` dispatches per
`QueryStatistic`:
- **Streaming additive** (`Sum/Count/Avg/Min/Max/Rate/Increase`):
`list_chunks` → for each chunk `read_chunk` → fold into a
bounded `AdditiveAccumulator` → drop the decoded chunk
before fetching the next. Memory is O(1) per query. Rate /
Increase track first/last `(ts, value)` and divide by
`range_seconds` at finalisation.
- **Buffered** (`Quantile / TopK`): `collect_buffered_samples`
materialises every in-range sample up to
`max_buffered_samples` (errors with `TooManySamples`
otherwise). Quantile sorts + nearest-rank index; TopK sorts
descending and returns sum of the top-k values.
* `tests.rs` — 20 unit tests via an in-process `MockColdStore`
satisfying `ColdStore` (no S3 / disk dependency). Pinned NOW
via `execute_at` so time math is deterministic. Covers every
`QueryStatistic` happy path, the empty-data sentinel, the
outside-range filter, the buffered-budget guard, the result
wrapping (accuracy envelope + `data_source: gorilla_archive`
marker), and the timeout path.
Test additions (15 spec'd + 5 planner)
--------------------------------------
* `execute_sum_over_time_streaming`
* `execute_count_over_time`
* `execute_avg_over_time`
* `execute_min_over_time` / `execute_max_over_time`
* `execute_rate_basic`
* `execute_increase_basic`
* `execute_quantile_buffered_basic`
* `execute_quantile_too_many_samples_errors`
* `execute_topk_basic`
* `execute_empty_chunks_returns_zero_or_nan`
* `execute_chunks_partially_outside_range_filtered`
* `result_carries_exact_accuracy_envelope`
* `result_includes_data_source_gorilla_archive`
* `engine_respects_config_timeout`
* `query_planner::tests::{plans_sum_over_time, plans_quantile_over_time,
plans_topk, rejects_binary_expression, streaming_classification}`
Touches only the new directory + `engines/mod.rs` (re-exports).
Zero changes to `simple_engine.rs`, the cold store, or the
sketch warm-tier. No new Cargo deps — `asap-gorilla` / `tokio`
were already pulled in by PR #84.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 of the Gorilla-S3-cold-engine: a sibling of
SimpleEnginethatexecutes PromQL exactly against the Phase-3
GorillaS3ColdStore(merged in PR #84). Same PromQL surface, different backend — instead of
warm-tier sketches with ε/δ bounds, this engine reads decoded
GORILLA1chunks straight off S3 and folds them into the answer withAccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }.Architecture
asap-query-engine/src/engines/gorilla_engine/(new directory, 4 files):mod.rs—GorillaQueryEngine(holdsArc<dyn ColdStore>sotests can inject mocks; production constructor
with_gorilla_s3keeps the design.md type signature). Public surface:
execute(query)execute_at(query, now_ms).wrap_resultattaches the exactaccuracy envelope + the request window.
GorillaEngineConfigwithmax_buffered_samples(default 10M) +query_timeout_secs(default30s).
EngineError::{Plan, ColdStore, TooManySamples, Timeout}.ExecutionOutcomecarries(value, samples_scanned, chunks_fetched)info_lines()builder pinning the on-wire info strings.query_planner.rs— minimal PromQL →QueryPlantranslator.Supports
sum/count/avg/min/max_over_time,rate,increase,quantile_over_time(φ, m[range]), andtopk(k, <vector>)(PromQLgrammar requires the inner be a vector so
topk(k, sum_over_time(m[range]))is the legal spelling).exact_executor.rs—ExactExecutordispatches perQueryStatistic:Increase): walks chunks one at a time, folds into a bounded
AdditiveAccumulator, drops the decoded chunk beforefetching the next. Memory cost is O(1) per query.
sample up to
max_buffered_samples; over-budget queries failfast with
EngineError::TooManySamplesrather than OOM.tests.rs— 20 unit tests via an in-processMockColdStore.Result wrapping
Every result carries:
AccuracyEnvelope::single(AccuracyProfile::exact())(ε = 0,δ = 0,kind = Exact),data_source: gorilla_archiveinfo marker,samples_scanned/chunks_fetcheddiagnostics.The
data_source: gorilla_archiveline is exposed viaExecutionOutcome::info_lines()so the HTTP driver layer can surfaceit on the Prometheus
infosarray (matching the existingSimpleEngineconvention).Test plan
cargo build --release -p query_engine_rustcleancargo test --release -p query_engine_rust gorilla_engine—20 / 20 pass
cargo clippy --release -p query_engine_rust --all-targets— 0warnings on the new module (5 pre-existing warnings in unrelated
files; not touched per scope guard)
execute_sum_over_time_streaming,execute_count_over_time,execute_avg_over_time,execute_min_over_time,execute_max_over_time,execute_rate_basic,execute_increase_basic,execute_quantile_buffered_basic,execute_quantile_too_many_samples_errors,execute_topk_basic,execute_empty_chunks_returns_zero_or_nan,execute_chunks_partially_outside_range_filtered,result_carries_exact_accuracy_envelope,result_includes_data_source_gorilla_archive,engine_respects_config_timeoutengines/gorilla_engine/dir + the enginesmod.rsre-export (nosimple_engine.rs, no cold-store, nosketch warm-tier).
Open question for Phase 5
GorillaQueryEngineandSimpleEngineare now sibling engines withdisjoint surface — Phase 5's capability router needs a hook to choose
between them per query. The engine exposes
config()for the costestimator; a follow-up trait abstraction (
pub trait QueryEngine)would let the router hold an
Arc<dyn QueryEngine>and dispatchwithout case-on-concrete. Deferred to Phase 5 per scope guard.
Doc tightening for design.md §6
The
data_sourcemarker in §6.6 ofdesign-gorilla-s3-cold-engine.mdreads
gorilla-s3; this PR shipsgorilla_archiveper the Phase 4spec (snake_case + the "archive" tier name preferred by the controller
side). The doc lives in the
ASAPCollectorrepo and is out of scopefor this PR — a follow-up doc PR should align the string.
🤖 Generated with Claude Code