feat(processor): gorillas3processor — Gorilla-encode + S3 chunk write (Phase 2) - #282
Merged
Merged
Conversation
… (Phase 2) Phase 2 of the Gorilla-S3-cold-engine: a new OTel collector contrib processor that takes raw metrics, Gorilla-XOR-compresses per-(metric, label_set) per 60s tumbling window, and PutObject's the resulting chunk to S3 / MinIO with a self-describing index.json per hour bucket. With drop_original=true the agent emits zero bytes downstream — paired with the Phase 1 asap-gorilla Rust crate (#281), the backend GorillaQueryEngine (Phase 4, queued) reads + decodes the same chunk for exact PromQL. ## Block format — byte-compatible with asap-gorilla Rust crate (#281) Outer block: `GORILLA1` 8-byte magic + 1-byte version + 4-byte LE series_count + back-to-back per-series chunks. Per-series body: JSON SeriesMeta + uncompressed first-sample seed + Gorilla XOR + delta-of-delta bit-packed body. The encoder shape mirrors sibling `opentelemetry-collector-contrib-patch/processor/gorillaprocessor/` so a chunk written here can be decoded by the existing in-process gorillaprocessor catalog scan AND by `asap-gorilla::GorillaDecoder` on the backend side. (Initial impl used the brief's example "ASGB" 4-byte magic which was not byte-compatible with #281 — that was reconciled to GORILLA1 for cross-runtime chunk interchange.) ## S3 sink aws-sdk-go-v1, S3ForcePathStyle: true for MinIO compatibility, retry/backoff, optional local spool fallback for S3-unavailable failure mode. Per-hour `index.json` is read-modify-write with in-memory mutex + small cache; multi- writer is best-effort and a sidecar compactor is the source-of-truth for multi-agent sharing the same hour bucket. ## Config Window 60s, prefix template `{tenant}/{metric}/{YYYY}/{MM}/{DD}/{HH}/`, drop_original=true canonical, encryption omitted (follow-up). Self-mon emits `gorillas3_chunks_written_total`, `gorillas3_chunk_payload_bytes_total`, `gorillas3_data_points_encoded_total`, `gorillas3_s3_put_failures_total`. ## Tests 16 tests across config / encoder / processor (round-trip, regular-interval compression sanity, multi-series-per-metric grouping, multi-metric chunk splitting at max_object_bytes, drop_original behavior, factory registration, shutdown drain). Build/vet/test verified in main repo (submodules required); worktree submodules unavailable so local repo verification deferred. ## OCB integration `cmd/sketchcollector/builder-config-sketches.yaml` registers gorillas3processor. `deploy/configs/sketchcol-agent-gorillas3-tier.yaml` is an example agent config pointed at the existing MinIO container (`endpoint: http://minio:9000`, `bucket: asap-gorilla`, `drop_original: true`). ## Spec ambiguity surfaced for Phase 0 design doc tightening - Lock GORILLA1 8-byte magic + 13-byte block header in design doc spec - Lock index.json schema (Version/Tenant/Metric/Entries[]) - Lock prefix-template token set: {tenant} {metric} {YYYY} {MM} {DD} {HH} - Multi-writer index ownership: best-effort writer-side, sidecar compactor authoritative Closes part of the Gorilla-S3-cold-engine track per `docs/design-gorilla-s3-cold-engine.md` §5 (PR #280). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
1 task
zzylol
added a commit
that referenced
this pull request
May 18, 2026
…390) ASAPQuery-backend PR #282 (bundle B3-pop + B4-window) made `WorkloadEntry.grouping_labels` a declarative yaml field — the controller stops trying to infer grouping from the PromQL `by(...)` clause (which doesn't exist for `quantile_over_time(...)`-style queries) and instead reads it from the yaml. Without `grouping_labels:` on each entry, the controller emits `transform/keep_for_<metric>` blocks with `keep_keys(datapoint.attributes, [])` — strips ALL attrs instead of keeping the intended grouping. Result in the smoke test: 1 sid registered (one per metric, no per-zone breakdown). Fix: add `grouping_labels:\n - zone` to all 8 entries. After: - Per-metric sids: 4-5 per metric (one per zone value) - `quantile_over_time(0.99, http_requests_total_latency_ms[5m])` returns per-zone p99 results, e.g.: { "metric": {..., "zone": "z0"}, "value": [..., "93.7"] } - Total: 21 sids across all metrics (was 1 pre-fix) Empirically verified via the single-node MVP smoke test at `/mydata/mvp-smoke-test/run_smoke.sh`. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of Gorilla-S3-cold-engine. Gorilla-encode raw metrics + S3 PutObject + drop_original. GORILLA1 magic byte-compat with #281. 16 tests pass in main repo. Spec ambiguity: ASGB→GORILLA1 reconciled. Closes part of design.md §5.