docs(design): gorilla-s3 cold engine — raw → Gorilla → S3 → exact query path - #280
Merged
Merged
Conversation
…ry path Phase 0 design doc for a new edge → Gorilla-encode → S3 chunk → backend exact-PromQL deployment shape, parallel to today's sketch warm-tier path (not a fallback). Doc-only; no code, no docker, no image rebuild. Covers: end-to-end architecture (current + new branch in one diagram); chunk + index wire format with the hour-bucketed key layout that mirrors the existing cold_store/format.rs; gorillas3processor spec for the OTel runtime (with sketchotap + sketchtelegraf variants); GorillaQueryEngine + GorillaS3ColdStore that extend the existing ColdStore trait additively; capability_matching extension with a new StorageBackend axis; controller 5-layer fit (BindGorillaExact rule at L4, StageId::Storage + Topology::FourStage at L5); paper / product mapping across the 5 evaluation dimensions; 7-phase implementation plan; 10 open questions with pre-defaulted recommendations and the few items flagged for orchestrator decision. Lifts encode + S3-write logic from telegraf-patch's existing gorilla_s3 plugin and the AccuracyProfile::exact() summary contract from the backend's accuracy.rs so the path slots into existing infrastructure without re-inventing either side. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 6, 2026
… (Phase 2) (#282) Phase 2 of the Gorilla-S3-cold-engine: a new OTel collector contrib processor that takes raw metrics, Gorilla-XOR-compresses per-(metric, label_set) per 60s tumbling window, and PutObject's the resulting chunk to S3 / MinIO with a self-describing index.json per hour bucket. With drop_original=true the agent emits zero bytes downstream — paired with the Phase 1 asap-gorilla Rust crate (#281), the backend GorillaQueryEngine (Phase 4, queued) reads + decodes the same chunk for exact PromQL. ## Block format — byte-compatible with asap-gorilla Rust crate (#281) Outer block: `GORILLA1` 8-byte magic + 1-byte version + 4-byte LE series_count + back-to-back per-series chunks. Per-series body: JSON SeriesMeta + uncompressed first-sample seed + Gorilla XOR + delta-of-delta bit-packed body. The encoder shape mirrors sibling `opentelemetry-collector-contrib-patch/processor/gorillaprocessor/` so a chunk written here can be decoded by the existing in-process gorillaprocessor catalog scan AND by `asap-gorilla::GorillaDecoder` on the backend side. (Initial impl used the brief's example "ASGB" 4-byte magic which was not byte-compatible with #281 — that was reconciled to GORILLA1 for cross-runtime chunk interchange.) ## S3 sink aws-sdk-go-v1, S3ForcePathStyle: true for MinIO compatibility, retry/backoff, optional local spool fallback for S3-unavailable failure mode. Per-hour `index.json` is read-modify-write with in-memory mutex + small cache; multi- writer is best-effort and a sidecar compactor is the source-of-truth for multi-agent sharing the same hour bucket. ## Config Window 60s, prefix template `{tenant}/{metric}/{YYYY}/{MM}/{DD}/{HH}/`, drop_original=true canonical, encryption omitted (follow-up). Self-mon emits `gorillas3_chunks_written_total`, `gorillas3_chunk_payload_bytes_total`, `gorillas3_data_points_encoded_total`, `gorillas3_s3_put_failures_total`. ## Tests 16 tests across config / encoder / processor (round-trip, regular-interval compression sanity, multi-series-per-metric grouping, multi-metric chunk splitting at max_object_bytes, drop_original behavior, factory registration, shutdown drain). Build/vet/test verified in main repo (submodules required); worktree submodules unavailable so local repo verification deferred. ## OCB integration `cmd/sketchcollector/builder-config-sketches.yaml` registers gorillas3processor. `deploy/configs/sketchcol-agent-gorillas3-tier.yaml` is an example agent config pointed at the existing MinIO container (`endpoint: http://minio:9000`, `bucket: asap-gorilla`, `drop_original: true`). ## Spec ambiguity surfaced for Phase 0 design doc tightening - Lock GORILLA1 8-byte magic + 13-byte block header in design doc spec - Lock index.json schema (Version/Tenant/Metric/Entries[]) - Lock prefix-template token set: {tenant} {metric} {YYYY} {MM} {DD} {HH} - Multi-writer index ownership: best-effort writer-side, sidecar compactor authoritative Closes part of the Gorilla-S3-cold-engine track per `docs/design-gorilla-s3-cold-engine.md` §5 (PR #280). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/design-gorilla-s3-cold-engine.md, 1,792 lines).index.json, hour-bucketed key layout that mirrorscold_store/format.rs); §5gorillas3processor(sketchcol Phase 2; sketchotap + sketchtelegraf variants Phase 7+); §6GorillaQueryEngine; §7GorillaS3ColdStoreextending the existingColdStoretrait additively; §8 capability_matching extension with a newStorageBackendaxis; §9 controller 5-layer fit (BindGorillaExactrule at L4;StageId::Storage+Topology::FourStageat L5; cost model with S3 PUT/GET/storage terms); §10 paper / product mapping across the 5 evaluation dimensions; §11 7-phase implementation plan; §12 10 open questions with pre-defaulted recommendations.telegraf-patch/plugins/outputs/gorilla_s3/gorilla_s3.go(already shipped) and re-uses theAccuracyProfile::exact()summary contract so the path slots into existing infrastructure without re-inventing either side.Test plan
docs/design-gorilla-s3-cold-engine.mdtouched, no source / config / submodule changes.BindGorillaExactshape) and §9.5 (StageId::Storagevariant) are flagged for orchestrator decision.asap-gorillablock format crate per §11.🤖 Generated with Claude Code