feat(asap-gorilla): Phase 1 block format encoder + decoder + index file - #281
Merged
Merged
Conversation
Phase 1 of the Gorilla-S3-cold-engine — a new sibling Rust crate that mirrors the canonical GORILLA1 block format defined by the Go gorillaprocessor under opentelemetry-collector-contrib-patch/. Bytes written by either side are interchangeable. The crate ships: - block.rs: on-wire layout (magic, version, series_count, per-series meta_json + first_ts + first_val_bits + ts/val bit streams) - encoder.rs: bit-for-bit mirror of the Go bitWriter + gorillaTimestampEncoder (delta-of-delta) + gorillaValueEncoder (XOR with leading/significant-bit window) - decoder.rs: streaming SampleIter so the backend GorillaQueryEngine can scan cold-store chunks without materializing every sample - index.rs: per-hour index.json catalog with prune_by_time and prune_by_label_hash helpers - 19 round-trip / byte-compat / index / integration tests, plus a smoke bench (currently encode 7.4M pts/s, decode 21M pts/s, ~7.1 bytes/sample on a 100k linear walk) The cross-language byte-parity test against a Go-produced fixture skips when tests/golden/single_series.gor is absent; tests/golden/ regen.sh documents the recipe for producing it once the Go-side fixturegen helper lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 6, 2026
… (Phase 2) (#282) Phase 2 of the Gorilla-S3-cold-engine: a new OTel collector contrib processor that takes raw metrics, Gorilla-XOR-compresses per-(metric, label_set) per 60s tumbling window, and PutObject's the resulting chunk to S3 / MinIO with a self-describing index.json per hour bucket. With drop_original=true the agent emits zero bytes downstream — paired with the Phase 1 asap-gorilla Rust crate (#281), the backend GorillaQueryEngine (Phase 4, queued) reads + decodes the same chunk for exact PromQL. ## Block format — byte-compatible with asap-gorilla Rust crate (#281) Outer block: `GORILLA1` 8-byte magic + 1-byte version + 4-byte LE series_count + back-to-back per-series chunks. Per-series body: JSON SeriesMeta + uncompressed first-sample seed + Gorilla XOR + delta-of-delta bit-packed body. The encoder shape mirrors sibling `opentelemetry-collector-contrib-patch/processor/gorillaprocessor/` so a chunk written here can be decoded by the existing in-process gorillaprocessor catalog scan AND by `asap-gorilla::GorillaDecoder` on the backend side. (Initial impl used the brief's example "ASGB" 4-byte magic which was not byte-compatible with #281 — that was reconciled to GORILLA1 for cross-runtime chunk interchange.) ## S3 sink aws-sdk-go-v1, S3ForcePathStyle: true for MinIO compatibility, retry/backoff, optional local spool fallback for S3-unavailable failure mode. Per-hour `index.json` is read-modify-write with in-memory mutex + small cache; multi- writer is best-effort and a sidecar compactor is the source-of-truth for multi-agent sharing the same hour bucket. ## Config Window 60s, prefix template `{tenant}/{metric}/{YYYY}/{MM}/{DD}/{HH}/`, drop_original=true canonical, encryption omitted (follow-up). Self-mon emits `gorillas3_chunks_written_total`, `gorillas3_chunk_payload_bytes_total`, `gorillas3_data_points_encoded_total`, `gorillas3_s3_put_failures_total`. ## Tests 16 tests across config / encoder / processor (round-trip, regular-interval compression sanity, multi-series-per-metric grouping, multi-metric chunk splitting at max_object_bytes, drop_original behavior, factory registration, shutdown drain). Build/vet/test verified in main repo (submodules required); worktree submodules unavailable so local repo verification deferred. ## OCB integration `cmd/sketchcollector/builder-config-sketches.yaml` registers gorillas3processor. `deploy/configs/sketchcol-agent-gorillas3-tier.yaml` is an example agent config pointed at the existing MinIO container (`endpoint: http://minio:9000`, `bucket: asap-gorilla`, `drop_original: true`). ## Spec ambiguity surfaced for Phase 0 design doc tightening - Lock GORILLA1 8-byte magic + 13-byte block header in design doc spec - Lock index.json schema (Version/Tenant/Metric/Entries[]) - Lock prefix-template token set: {tenant} {metric} {YYYY} {MM} {DD} {HH} - Multi-writer index ownership: best-effort writer-side, sidecar compactor authoritative Closes part of the Gorilla-S3-cold-engine track per `docs/design-gorilla-s3-cold-engine.md` §5 (PR #280). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 18, 2026
…eConfig (ASAPCollector#381 Issue #4) (#389) Two paired fixes to close the apply-loop the smoke test surfaced after ASAPQuery-backend PR #279 landed: (1) No-op short-circuit in processRemoteConfig Before writing the pushed body to disk + exiting, compare to the current on-disk file. If they match byte-for-byte, just report APPLIED and return — don't restart. This handles two scenarios: * Defense-in-depth against a controller resending the same config (which can happen any time the OpAMP server's LastRemoteConfigHash cache is empty or stale — e.g. on reconnect before the agent has reported APPLIED). * The deterministic-emit guarantee from ASAPQuery-backend #281 means same-semantic-content → same bytes, so this check is accurate for the controller's emit. (2) Advertise ReportsRemoteConfig capability In addition to AcceptsRemoteConfig, set `AgentCapabilities_AgentCapabilities_ReportsRemoteConfig` when `remote_config_path` is configured. Without this bit, `opampClient.SetRemoteConfigStatus(APPLIED)` returns `ErrReportsRemoteConfigNotSet` (opamp-go client/internal/clientcommon.go:21), the client lib refuses to record the applied hash, and the server has no way to learn the config was processed — keeping it stuck in the resend loop. Both bits are needed for the full feedback cycle. End-to-end smoke (post-#281 deterministic emit + this PR): * Agent boots from bootstrap YAML * Receives RemoteConfig push, bytes differ, applies + exits * Restarts, loads new YAML * Receives RemoteConfig push, bytes IDENTICAL (deterministic emit) * No-op short-circuit fires → SetRemoteConfigStatus(APPLIED) → keeps running, no more restarts * Pipeline emits sketches normally; smoke test Axis D registers them Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 of the Gorilla-S3-cold-engine — a new sibling Rust crate
asap-gorilla/that mirrors the canonicalGORILLA1block format defined by the Gogorillaprocessorunderopentelemetry-collector-contrib-patch/processor/gorillaprocessor/. Bytes written by either side are interchangeable.meta_json+first_ts+first_val_bits+ ts/val bit streams)bitWriter+gorillaTimestampEncoder(delta-of-delta) +gorillaValueEncoder(XOR with leading/significant-bit window)SampleIterso the backendGorillaQueryEnginecan scan cold-store chunks without materializing every sampleindex.jsoncatalog withprune_by_timeandprune_by_label_hashhelpersThe cross-language byte-parity test against a Go-produced fixture skips when
tests/golden/single_series.goris absent;tests/golden/regen.shdocuments the recipe for producing it once the Go-side fixturegen helper lands.Files
asap-gorilla/Cargo.toml,Cargo.lock,README.md,.gitignoreasap-gorilla/src/{lib,block,encoder,decoder,index,error}.rsasap-gorilla/tests/{round_trip,byte_compat,index_file,integration}.rsasap-gorilla/tests/golden/{regen.sh,.gitkeep}No other paths touched. No workspace
Cargo.tomlexists at repo root yet (sibling cratesasap-precompute-rs/controllerare standalone), so this PR adds the new crate as a third standalone sibling rather than touching workspacemembers.Test plan
cargo build --releaseclean insideasap-gorilla/cargo test --release— 19 tests + 1 doc-test pass (round-trip 8, byte-compat 4, index 4, integration 3)cargo clippy --release --all-targets -- -D warningscleancargo doc --no-depscleantests/golden/regen.shproducessingle_series.gor. The Go-sideinternal/fixturegenhelper does not yet exist; once added, the byte-identical re-encode test activates automatically.Notes for reviewers
Two spec interpretations worth surfacing:
Telegraf vs gorillaprocessor as "spec source": the brief named
telegraf-patch/plugins/outputs/gorilla_s3/gorilla_s3.goas the canonical spec, but that file is a passthrough S3 uploader — it does not encode Gorilla blocks. The actual block encoder lives inopentelemetry-collector-contrib-patch/processor/gorillaprocessor/{bitwriter,encoder,compress}.go. This PR mirrors that file set bit-for-bit. Worth tightening in the Phase 0 design doc.JSON metadata key ordering: the Go side serializes the label set from
map[string]string, whose iteration order is randomized — that makes the Go encoder's bytes non-deterministic for identical inputs.asap-gorilladeliberately strengthens the contract by serializing labels throughBTreeMapso its output is byte-stable. Cross-language byte-parity therefore requires the Go fixturegen sort keys before marshaling (or for us to compare via the parsedSeriesMetastruct rather than raw bytes). README documents this.🤖 Generated with Claude Code