mvp/v5: postings-aware GorillaQueryEngine + S3 cost tracker (issue #46) - #90
Merged
Merged
Conversation
…sue #46 v5) Adds the two backend pieces the v5 MVP needs: 1. **Postings-aware execution path** (`asap-query-engine::engines::gorilla_engine`). The query planner extracts exact-equality label matchers from the PromQL AST (`zone="a"` etc); regex / inequality matchers are surfaced via `has_unsupported_matchers` and fall through to a post-decode filter for correctness. The `ColdStore` trait gains `list_postings_for(metric, range, matchers) -> PostingsHits`; `GorillaS3ColdStore` implements it by per-bucket fetch + LRU cache (256 entries) + intersect-across-matchers / union-across- buckets. The `ExactExecutor` calls it before the chunk-list, then prunes chunks whose `label_hash` isn't in the matched series set. `ExecutionOutcome` grows three new fields (surfaced in `infos`): `chunks_skipped_via_postings`, `postings_filtered_series_count`, `postings_missing` — the last of which drives a `data_source_quirk: postings_missing` line when the engine fell back to scan-all. 2. **S3 cost tracker** (`drivers/query/fallback/cold_store/s3_cost_tracker.rs`). Wraps the `rust-s3`-backed `ObjectStore` in counters for PUT / GET / HEAD / LIST / DELETE + bytes-out per op. A process-wide `OnceLock<Arc<S3CostCounters>>` lets the HTTP server's new `/internal/s3_cost.csv` endpoint dump the CSV the v5 demo script consumes; `/metrics` also appends the same counters in Prometheus exposition. Backward compat: pre-mvp/v5 cold stores keep returning `ColdStoreError::Unsupported("list_postings_for")` from the default trait method; the executor treats that as "postings missing" and stays correct. Test summary: * gorilla_engine: 24 tests pass (21 originals + 3 new for postings-aware happy path / fall-back / no-predicate skip). * s3_cost_tracker: 4 tests pass. * Full suite: 854 pass / 34 fail; the 34 failures pre-date this branch (verified against origin/main). Cross-repo dependency: this PR depends on the matching `mvp/v5-postings-compactor` PR in ASAPCollector — specifically `asap_gorilla::Postings` and the `IndexEntry` extension fields. `asap-query-engine/Cargo.toml` carries a NOTE FOR REVIEWERS explaining the path-dep coupling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v5 MVP — backend side. DRAFT, do not merge. Run agent will execute the demo and post numbers.
engines::gorilla_engine. Planner extracts exact-equality label matchers from PromQL;ColdStore::list_postings_forreturns the matchingseries_ids; theExactExecutorprunes chunks whoselabel_hashisn't in the set. Regex / inequality matchers fall through to a post-decode filter for correctness. NewExecutionOutcomefields (chunks_skipped_via_postings,postings_filtered_series_count,postings_missing) surface in responseinfos.data_source_quirk: postings_missingis emitted when the engine fell back to scan-all.drivers/query/fallback/cold_store/s3_cost_tracker.rs). Wraps the rust-s3ObjectStorein counters; process-wideOnceLocklets the new/internal/s3_cost.csvHTTP endpoint dump the CSV the v5 demo consumes;/metricsalso exposes the same counters in Prometheus exposition.Test plan
cargo test gorilla_engine(24 pass — 21 originals + 3 new for postings-aware happy path / fall-back / no-predicate skip)cargo test s3_cost_tracker(4 pass)cargo buildcleanbash deploy/scripts/run_mvp_demo.shin the collector repo/internal/s3_cost.csvreturns non-zero counters for the ASAP cellCross-repo dependency
Pairs with
mvp/v5-postings-compactorPR in ASAPCollector. Theasap-query-engine/Cargo.tomlcarries a NOTE FOR REVIEWERS explaining the path-dep coupling — both branches must be checked out side-by-side forcargo buildto resolveasap_gorilla::Postings.🤖 Generated with Claude Code