Skip to content

feat(cold-store): GorillaS3ColdStore — list+read chunks from S3 via asap-gorilla (Phase 3) - #84

Merged
zzylol merged 1 commit into
mainfrom
feat/gorilla-s3-cold-store-phase-3
May 6, 2026
Merged

feat(cold-store): GorillaS3ColdStore — list+read chunks from S3 via asap-gorilla (Phase 3)#84
zzylol merged 1 commit into
mainfrom
feat/gorilla-s3-cold-store-phase-3

Conversation

@zzylol

@zzylol zzylol commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 3 of the Gorilla-S3-cold-engine: adds a ColdStore adapter that lists per-hour index.json catalogs out of an S3-compatible bucket, prunes them by time range, and decodes the selected GORILLA1 chunks via the freshly-merged asap-gorilla crate (ASAPCollector#281).

  • cold_store/mod.rs — additive trait extension: new ChunkRef + list_chunks / read_chunk default-impls returning ColdStoreError::Unsupported, plus Backend(String) / Unsupported(&'static str) error variants. Existing LocalFsColdStore + s3_adapter::ColdFallback continue compiling unchanged.
  • cold_store/gorilla_s3.rs (new) — GorillaS3ColdStore + GorillaS3Config (+from_env) + an ObjectStore trait so the impl is generic over the wire client. Production backend uses rust-s3 (S3ObjectStore, with MinIO path-style). LRU cache (default 256, configurable) keyed on chunk object key, holding pre-decoded RawSample lists so repeat reads skip the Gorilla decode pass.
  • asap-query-engine/Cargo.toml — new deps: asap-gorilla (path), rust-s3 = 0.37 (default-features off + tokio-rustls-tls), lru = 0.12.

S3 client choice

Picked rust-s3 over aws-sdk-s3 for the lighter dep tree (no full AWS SDK fanout), first-class MinIO support (with_path_style() is a supported config rather than a workaround), and shared rustls backend with the reqwest already in this crate. The trait-based ObjectStore indirection lets us swap clients without touching the ColdStore impl.

Test plan

Ten new tests, all green; pre-existing cold_store suite (11 tests) unchanged.

  • list_chunks_via_indexfile_prunes_by_time — three-chunk index, window overlaps only the middle entry.
  • read_chunk_decodes_via_asap_gorilla — round-trip through a real GorillaEncoder block.
  • cache_hit_skips_s3_fetch — second read does not increment the mock GET counter.
  • lru_eviction_under_pressure — cache_capacity = 2, fill three, re-read the first triggers a fresh GET.
  • index_json_corrupted_returns_error — clean Malformed, no panic.
  • s3_unavailable_returns_error — clean Backend(..), no panic.
  • missing_index_is_empty_not_error — producer not yet flushed; empty list rather than error.
  • scan_filters_to_requested_range — chunk overlaps but inner samples filter to zero.
  • list_chunks_spans_two_hour_buckets — request crosses an hour boundary; both hour indexes fetched.
  • from_env_requires_bucket — env-config validation surfaces missing required var.

Verification

  • cargo build --release -p query_engine_rust clean.
  • cargo test --release -p query_engine_rust gorilla_s3 — 10/10 pass.
  • cargo test --release -p query_engine_rust cold_store — 21/21 pass (10 new + 11 pre-existing).
  • cargo clippy --release -p query_engine_rust --all-targets -- -D warnings — zero new warnings on the touched files; the pre-existing otel.rs / simple_engine.rs / count_sketch_accumulator.rs lints already fail on main.
  • No docker activity. No submodule pointer drift.

🤖 Generated with Claude Code

@zzylol @claude
feat(cold-store): GorillaS3ColdStore — list+read chunks from S3 via a…
c4f317f 
…sap-gorilla (Phase 3)

Phase 3 of the Gorilla-S3-cold-engine — adds a `ColdStore` adapter that
fetches per-hour `index.json` catalogs from an S3-compatible bucket,
prunes them by time range, and decodes the selected `GORILLA1` chunks
via the freshly-merged `asap-gorilla` crate (ASAPCollector PR #281).

Changes
-------

* `cold_store/mod.rs` — additive trait extension. New `ChunkRef`
  descriptor + `list_chunks` / `read_chunk` methods carry default
  impls returning `ColdStoreError::Unsupported`, so `LocalFsColdStore`
  and the existing `s3_adapter::ColdFallback` chain compile unchanged.
  Two new `ColdStoreError` variants (`Backend(String)` for transport
  failures, `Unsupported(&'static str)` for the default-impl errors).

* `cold_store/gorilla_s3.rs` — new module. `GorillaS3ColdStore`
  + `GorillaS3Config` (with `from_env`) + `ObjectStore` trait + a
  `rust-s3`-backed production impl (`S3ObjectStore`, with MinIO
  path-style support) and an in-memory mock used by the tests. LRU
  cache (default 256 chunks) keyed on the chunk object key, holding
  pre-decoded `RawSample` lists so repeated reads skip the Gorilla
  decode pass entirely.

* `asap-query-engine/Cargo.toml` — new deps: `asap-gorilla` (path),
  `rust-s3 = 0.37` (default-features off, `tokio-rustls-tls` to share
  the rustls backend reqwest already pulls in), `lru = 0.12`.

S3 client choice
----------------

Picked `rust-s3` over `aws-sdk-s3` for the lighter dep tree (no full
AWS SDK fanout) and first-class MinIO support (`with_path_style()` is
the supported config rather than a workaround). The trait-based
`ObjectStore` indirection means the choice is replaceable without
touching the `ColdStore` impl.

Tests (10, all green)
---------------------

* `list_chunks_via_indexfile_prunes_by_time` — hand-crafted index
  with three chunks; window overlaps only the middle one.
* `read_chunk_decodes_via_asap_gorilla` — round-trip through a real
  `GorillaEncoder` block; samples + labels match.
* `cache_hit_skips_s3_fetch` — second `read_chunk` on the same key
  must not trigger another S3 GET (mock counter assertion).
* `lru_eviction_under_pressure` — `cache_capacity = 2`, fill three,
  re-read the first — must trigger a fresh GET.
* `index_json_corrupted_returns_error` — clean `Malformed`, no panic.
* `s3_unavailable_returns_error` — clean `Backend(...)`, no panic.
* `missing_index_is_empty_not_error` — producer hasn't flushed yet;
  empty result, not error.
* `scan_filters_to_requested_range` — chunk overlaps but inner
  samples filter to zero.
* `list_chunks_spans_two_hour_buckets` — request crosses an hour
  boundary; both hour indexes are fetched.
* `from_env_requires_bucket` — config validation surfaces missing env.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 7a0c2eb into main May 6, 2026
@zzylol zzylol mentioned this pull request May 6, 2026
Merged
6 tasks
zzylol added a commit that referenced this pull request May 6, 2026
@zzylol @claude
feat(engine): GorillaQueryEngine — exact PromQL over Gorilla-S3 chunk…
6ce0fed 
…s (Phase 4) (#85)

Phase 4 of the Gorilla-S3-cold-engine — adds a sibling engine to
`SimpleEngine` that executes PromQL exactly against the Phase-3
`GorillaS3ColdStore`. Returns answers carrying an
`AccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }` and a
`data_source: gorilla_archive` info marker so callers /
dashboards can distinguish cold-tier exact answers from warm-tier
sketch answers.

Architecture
------------

`asap-query-engine/src/engines/gorilla_engine/` (new):

* `mod.rs` — `GorillaQueryEngine` (holds `Arc<dyn ColdStore>` so
  tests can inject mocks; production constructor
  `with_gorilla_s3` keeps the design.md type signature). Public
  surface: `execute(query)` + `execute_at(query, now_ms)`. Wraps
  results via `wrap_result` (exact accuracy envelope + window).
  `GorillaEngineConfig { max_buffered_samples, query_timeout_secs }`
  with sensible defaults (10M / 30s). `EngineError` enum covers
  Plan / ColdStore / TooManySamples / Timeout. `ExecutionOutcome`
  carries `(value, samples_scanned, chunks_fetched)` + an
  `info_lines` builder pinning the on-wire info strings.

* `query_planner.rs` — minimal PromQL → `QueryPlan` translator
  (metric, half-open `[start_ms, end_ms)`, `QueryStatistic`).
  Supports `sum/count/avg/min/max_over_time`, `rate`, `increase`,
  `quantile_over_time(φ, m[range])`, and `topk(k, <vector>)`
  (PromQL grammar requires the inner be a vector so `topk(k,
  sum_over_time(m[range]))` is the legal spelling). Tests +
  caller can use `plan_query_at` to pin `now_ms`.

* `exact_executor.rs` — `ExactExecutor` dispatches per
  `QueryStatistic`:
  - **Streaming additive** (`Sum/Count/Avg/Min/Max/Rate/Increase`):
    `list_chunks` → for each chunk `read_chunk` → fold into a
    bounded `AdditiveAccumulator` → drop the decoded chunk
    before fetching the next. Memory is O(1) per query. Rate /
    Increase track first/last `(ts, value)` and divide by
    `range_seconds` at finalisation.
  - **Buffered** (`Quantile / TopK`): `collect_buffered_samples`
    materialises every in-range sample up to
    `max_buffered_samples` (errors with `TooManySamples`
    otherwise). Quantile sorts + nearest-rank index; TopK sorts
    descending and returns sum of the top-k values.

* `tests.rs` — 20 unit tests via an in-process `MockColdStore`
  satisfying `ColdStore` (no S3 / disk dependency). Pinned NOW
  via `execute_at` so time math is deterministic. Covers every
  `QueryStatistic` happy path, the empty-data sentinel, the
  outside-range filter, the buffered-budget guard, the result
  wrapping (accuracy envelope + `data_source: gorilla_archive`
  marker), and the timeout path.

Test additions (15 spec'd + 5 planner)
--------------------------------------

* `execute_sum_over_time_streaming`
* `execute_count_over_time`
* `execute_avg_over_time`
* `execute_min_over_time` / `execute_max_over_time`
* `execute_rate_basic`
* `execute_increase_basic`
* `execute_quantile_buffered_basic`
* `execute_quantile_too_many_samples_errors`
* `execute_topk_basic`
* `execute_empty_chunks_returns_zero_or_nan`
* `execute_chunks_partially_outside_range_filtered`
* `result_carries_exact_accuracy_envelope`
* `result_includes_data_source_gorilla_archive`
* `engine_respects_config_timeout`
* `query_planner::tests::{plans_sum_over_time, plans_quantile_over_time,
   plans_topk, rejects_binary_expression, streaming_classification}`

Touches only the new directory + `engines/mod.rs` (re-exports).
Zero changes to `simple_engine.rs`, the cold store, or the
sketch warm-tier. No new Cargo deps — `asap-gorilla` / `tokio`
were already pulled in by PR #84.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 6, 2026
@zzylol @claude
feat(capability): Phase 5 — QueryEngine trait + StorageBackend routin…
0a0e607 
…g for Gorilla-S3 (#86)

* feat(cold-store): GorillaS3ColdStore — list+read chunks from S3 via asap-gorilla (Phase 3)

Phase 3 of the Gorilla-S3-cold-engine — adds a `ColdStore` adapter that
fetches per-hour `index.json` catalogs from an S3-compatible bucket,
prunes them by time range, and decodes the selected `GORILLA1` chunks
via the freshly-merged `asap-gorilla` crate (ASAPCollector PR #281).

Changes
-------

* `cold_store/mod.rs` — additive trait extension. New `ChunkRef`
  descriptor + `list_chunks` / `read_chunk` methods carry default
  impls returning `ColdStoreError::Unsupported`, so `LocalFsColdStore`
  and the existing `s3_adapter::ColdFallback` chain compile unchanged.
  Two new `ColdStoreError` variants (`Backend(String)` for transport
  failures, `Unsupported(&'static str)` for the default-impl errors).

* `cold_store/gorilla_s3.rs` — new module. `GorillaS3ColdStore`
  + `GorillaS3Config` (with `from_env`) + `ObjectStore` trait + a
  `rust-s3`-backed production impl (`S3ObjectStore`, with MinIO
  path-style support) and an in-memory mock used by the tests. LRU
  cache (default 256 chunks) keyed on the chunk object key, holding
  pre-decoded `RawSample` lists so repeated reads skip the Gorilla
  decode pass entirely.

* `asap-query-engine/Cargo.toml` — new deps: `asap-gorilla` (path),
  `rust-s3 = 0.37` (default-features off, `tokio-rustls-tls` to share
  the rustls backend reqwest already pulls in), `lru = 0.12`.

S3 client choice
----------------

Picked `rust-s3` over `aws-sdk-s3` for the lighter dep tree (no full
AWS SDK fanout) and first-class MinIO support (`with_path_style()` is
the supported config rather than a workaround). The trait-based
`ObjectStore` indirection means the choice is replaceable without
touching the `ColdStore` impl.

Tests (10, all green)
---------------------

* `list_chunks_via_indexfile_prunes_by_time` — hand-crafted index
  with three chunks; window overlaps only the middle one.
* `read_chunk_decodes_via_asap_gorilla` — round-trip through a real
  `GorillaEncoder` block; samples + labels match.
* `cache_hit_skips_s3_fetch` — second `read_chunk` on the same key
  must not trigger another S3 GET (mock counter assertion).
* `lru_eviction_under_pressure` — `cache_capacity = 2`, fill three,
  re-read the first — must trigger a fresh GET.
* `index_json_corrupted_returns_error` — clean `Malformed`, no panic.
* `s3_unavailable_returns_error` — clean `Backend(...)`, no panic.
* `missing_index_is_empty_not_error` — producer hasn't flushed yet;
  empty result, not error.
* `scan_filters_to_requested_range` — chunk overlaps but inner
  samples filter to zero.
* `list_chunks_spans_two_hour_buckets` — request crosses an hour
  boundary; both hour indexes are fetched.
* `from_env_requires_bucket` — config validation surfaces missing env.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(engine): GorillaQueryEngine — exact PromQL over Gorilla-S3 chunks (Phase 4)

Phase 4 of the Gorilla-S3-cold-engine — adds a sibling engine to
`SimpleEngine` that executes PromQL exactly against the Phase-3
`GorillaS3ColdStore`. Returns answers carrying an
`AccuracyEnvelope { kind: Exact, ε: 0, δ: 0 }` and a
`data_source: gorilla_archive` info marker so callers /
dashboards can distinguish cold-tier exact answers from warm-tier
sketch answers.

Architecture
------------

`asap-query-engine/src/engines/gorilla_engine/` (new):

* `mod.rs` — `GorillaQueryEngine` (holds `Arc<dyn ColdStore>` so
  tests can inject mocks; production constructor
  `with_gorilla_s3` keeps the design.md type signature). Public
  surface: `execute(query)` + `execute_at(query, now_ms)`. Wraps
  results via `wrap_result` (exact accuracy envelope + window).
  `GorillaEngineConfig { max_buffered_samples, query_timeout_secs }`
  with sensible defaults (10M / 30s). `EngineError` enum covers
  Plan / ColdStore / TooManySamples / Timeout. `ExecutionOutcome`
  carries `(value, samples_scanned, chunks_fetched)` + an
  `info_lines` builder pinning the on-wire info strings.

* `query_planner.rs` — minimal PromQL → `QueryPlan` translator
  (metric, half-open `[start_ms, end_ms)`, `QueryStatistic`).
  Supports `sum/count/avg/min/max_over_time`, `rate`, `increase`,
  `quantile_over_time(φ, m[range])`, and `topk(k, <vector>)`
  (PromQL grammar requires the inner be a vector so `topk(k,
  sum_over_time(m[range]))` is the legal spelling). Tests +
  caller can use `plan_query_at` to pin `now_ms`.

* `exact_executor.rs` — `ExactExecutor` dispatches per
  `QueryStatistic`:
  - **Streaming additive** (`Sum/Count/Avg/Min/Max/Rate/Increase`):
    `list_chunks` → for each chunk `read_chunk` → fold into a
    bounded `AdditiveAccumulator` → drop the decoded chunk
    before fetching the next. Memory is O(1) per query. Rate /
    Increase track first/last `(ts, value)` and divide by
    `range_seconds` at finalisation.
  - **Buffered** (`Quantile / TopK`): `collect_buffered_samples`
    materialises every in-range sample up to
    `max_buffered_samples` (errors with `TooManySamples`
    otherwise). Quantile sorts + nearest-rank index; TopK sorts
    descending and returns sum of the top-k values.

* `tests.rs` — 20 unit tests via an in-process `MockColdStore`
  satisfying `ColdStore` (no S3 / disk dependency). Pinned NOW
  via `execute_at` so time math is deterministic. Covers every
  `QueryStatistic` happy path, the empty-data sentinel, the
  outside-range filter, the buffered-budget guard, the result
  wrapping (accuracy envelope + `data_source: gorilla_archive`
  marker), and the timeout path.

Test additions (15 spec'd + 5 planner)
--------------------------------------

* `execute_sum_over_time_streaming`
* `execute_count_over_time`
* `execute_avg_over_time`
* `execute_min_over_time` / `execute_max_over_time`
* `execute_rate_basic`
* `execute_increase_basic`
* `execute_quantile_buffered_basic`
* `execute_quantile_too_many_samples_errors`
* `execute_topk_basic`
* `execute_empty_chunks_returns_zero_or_nan`
* `execute_chunks_partially_outside_range_filtered`
* `result_carries_exact_accuracy_envelope`
* `result_includes_data_source_gorilla_archive`
* `engine_respects_config_timeout`
* `query_planner::tests::{plans_sum_over_time, plans_quantile_over_time,
   plans_topk, rejects_binary_expression, streaming_classification}`

Touches only the new directory + `engines/mod.rs` (re-exports).
Zero changes to `simple_engine.rs`, the cold store, or the
sketch warm-tier. No new Cargo deps — `asap-gorilla` / `tokio`
were already pulled in by PR #84.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(capability): Phase 5 — QueryEngine trait + StorageBackend routing for Gorilla-S3

Adds the storage-backend axis to capability matching so queries against
metrics configured for Gorilla-S3 dispatch to GorillaQueryEngine instead
of the warm-tier SimpleEngine. Per docs/design-gorilla-s3-cold-engine.md
§8.

asap_types extensions:
  - StorageBackend enum (SketchWarmTier default + GorillaS3Archive +
    ColdJsonlFallback + DoubleWrite) with snake_case serde + #[serde(default)]
    so pre-Phase-5 configs decode unchanged.
  - AccuracyTarget enum (Exact / Approximate, default Approximate).
  - compatible_storage_backends(stat, accuracy, metric_storage_config)
    returns the ordered failover list the router walks. Exact-on-archive
    subsumes approximate-on-warm (Gorilla-only metrics still archive).
  - StreamingConfig.storage_backend field with serde-default + builder
    constructor (with_storage_backend) + getter.

asap-query-engine extensions (additive):
  - engines::EngineError envelope (CapabilityMiss vs Backend) the trait
    returns.
  - engines::router::QueryEngine async trait + EngineCapabilities struct.
  - engines::router::EngineRouter dispatcher: consults
    compatible_storage_backends, dispatches to first registered engine,
    falls through on Backend / CapabilityMiss to the next compatible
    backend (typically ColdJsonlFallback).
  - impl QueryEngine for SimpleEngine (adapts handle_query → Result;
    None → CapabilityMiss).
  - impl QueryEngine for GorillaQueryEngine (Plan errors fold to
    CapabilityMiss; ColdStore / Timeout / TooManySamples to Backend).

Tests added (16 new):
  - capability_matching: 6 routing tests + 1 source-of-truth agreement
    test enumerating (Statistic × AccuracyTarget × StorageBackend).
  - streaming_config: 2 serde tests pinning legacy + Phase-5 wire format.
  - engines::router: 6 tests (warm-tier dispatch, archive dispatch,
    JSONL fallback when archive+warm fail, no-engines clean error,
    all-failed surface, hot-swap re-registration).

Test-utility struct-literal sites updated to include the new
StreamingConfig.storage_backend field. No precompute_engine/, stores/, or
precompute_operators/ touched.

Verified clean: cargo build --release -p {asap_types, query_engine_rust};
cargo test -p asap_types; cargo test --release -p query_engine_rust router;
cargo clippy --release -p asap_types --all-targets -- -D warnings.
Pre-existing query_engine_rust clippy/test failures unchanged (verified
via git-stash diff).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request May 6, 2026
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zzylol