Skip to content

feat(asap-gorilla): Phase 1 block format encoder + decoder + index file - #281

Merged
zzylol merged 1 commit into
mainfrom
feat/asap-gorilla-crate
May 6, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/asap-gorilla-crate

Conversation

@zzylol

@zzylol zzylol commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 1 of the Gorilla-S3-cold-engine — a new sibling Rust crate asap-gorilla/ that mirrors the canonical GORILLA1 block format defined by the Go gorillaprocessor under opentelemetry-collector-contrib-patch/processor/gorillaprocessor/. Bytes written by either side are interchangeable.

  • block.rs — on-wire layout (magic, version, series_count, per-series meta_json + first_ts + first_val_bits + ts/val bit streams)
  • encoder.rs — bit-for-bit mirror of the Go bitWriter + gorillaTimestampEncoder (delta-of-delta) + gorillaValueEncoder (XOR with leading/significant-bit window)
  • decoder.rs — streaming SampleIter so the backend GorillaQueryEngine can scan cold-store chunks without materializing every sample
  • index.rs — per-hour index.json catalog with prune_by_time and prune_by_label_hash helpers
  • 19 round-trip / byte-compat / index / integration tests + a smoke bench: encode 7.4M pts/s, decode 21M pts/s, ~7.1 bytes/sample on a 100k linear walk

The cross-language byte-parity test against a Go-produced fixture skips when tests/golden/single_series.gor is absent; tests/golden/regen.sh documents the recipe for producing it once the Go-side fixturegen helper lands.

Files

  • asap-gorilla/Cargo.toml, Cargo.lock, README.md, .gitignore
  • asap-gorilla/src/{lib,block,encoder,decoder,index,error}.rs
  • asap-gorilla/tests/{round_trip,byte_compat,index_file,integration}.rs
  • asap-gorilla/tests/golden/{regen.sh,.gitkeep}

No other paths touched. No workspace Cargo.toml exists at repo root yet (sibling crates asap-precompute-rs / controller are standalone), so this PR adds the new crate as a third standalone sibling rather than touching workspace members.

Test plan

  • cargo build --release clean inside asap-gorilla/
  • cargo test --release — 19 tests + 1 doc-test pass (round-trip 8, byte-compat 4, index 4, integration 3)
  • cargo clippy --release --all-targets -- -D warnings clean
  • cargo doc --no-deps clean
  • Smoke bench prints encode/decode rates (encode 7.4M pts/s, decode 21M pts/s, ~7.1 B/sample)
  • Cross-language byte-parity test against a real Telegraf-encoded fixture — skipped at PR time; activates once tests/golden/regen.sh produces single_series.gor. The Go-side internal/fixturegen helper does not yet exist; once added, the byte-identical re-encode test activates automatically.

Notes for reviewers

Two spec interpretations worth surfacing:

  1. Telegraf vs gorillaprocessor as "spec source": the brief named telegraf-patch/plugins/outputs/gorilla_s3/gorilla_s3.go as the canonical spec, but that file is a passthrough S3 uploader — it does not encode Gorilla blocks. The actual block encoder lives in opentelemetry-collector-contrib-patch/processor/gorillaprocessor/{bitwriter,encoder,compress}.go. This PR mirrors that file set bit-for-bit. Worth tightening in the Phase 0 design doc.

  2. JSON metadata key ordering: the Go side serializes the label set from map[string]string, whose iteration order is randomized — that makes the Go encoder's bytes non-deterministic for identical inputs. asap-gorilla deliberately strengthens the contract by serializing labels through BTreeMap so its output is byte-stable. Cross-language byte-parity therefore requires the Go fixturegen sort keys before marshaling (or for us to compare via the parsed SeriesMeta struct rather than raw bytes). README documents this.

🤖 Generated with Claude Code

Phase 1 of the Gorilla-S3-cold-engine — a new sibling Rust crate that
mirrors the canonical GORILLA1 block format defined by the Go
gorillaprocessor under opentelemetry-collector-contrib-patch/. Bytes
written by either side are interchangeable.

The crate ships:
- block.rs: on-wire layout (magic, version, series_count, per-series
  meta_json + first_ts + first_val_bits + ts/val bit streams)
- encoder.rs: bit-for-bit mirror of the Go bitWriter +
  gorillaTimestampEncoder (delta-of-delta) + gorillaValueEncoder
  (XOR with leading/significant-bit window)
- decoder.rs: streaming SampleIter so the backend GorillaQueryEngine
  can scan cold-store chunks without materializing every sample
- index.rs: per-hour index.json catalog with prune_by_time and
  prune_by_label_hash helpers
- 19 round-trip / byte-compat / index / integration tests, plus a
  smoke bench (currently encode 7.4M pts/s, decode 21M pts/s,
  ~7.1 bytes/sample on a 100k linear walk)

The cross-language byte-parity test against a Go-produced fixture
skips when tests/golden/single_series.gor is absent; tests/golden/
regen.sh documents the recipe for producing it once the Go-side
fixturegen helper lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit df8f2cc into main May 6, 2026
zzylol added a commit that referenced this pull request May 6, 2026
… (Phase 2) (#282)

Phase 2 of the Gorilla-S3-cold-engine: a new OTel collector contrib processor
that takes raw metrics, Gorilla-XOR-compresses per-(metric, label_set) per
60s tumbling window, and PutObject's the resulting chunk to S3 / MinIO with a
self-describing index.json per hour bucket. With drop_original=true the agent
emits zero bytes downstream — paired with the Phase 1 asap-gorilla Rust crate
(#281), the backend GorillaQueryEngine (Phase 4, queued) reads + decodes the
same chunk for exact PromQL.

## Block format — byte-compatible with asap-gorilla Rust crate (#281)

Outer block: `GORILLA1` 8-byte magic + 1-byte version + 4-byte LE
series_count + back-to-back per-series chunks. Per-series body:
JSON SeriesMeta + uncompressed first-sample seed + Gorilla XOR + delta-of-delta
bit-packed body. The encoder shape mirrors sibling
`opentelemetry-collector-contrib-patch/processor/gorillaprocessor/` so a
chunk written here can be decoded by the existing in-process gorillaprocessor
catalog scan AND by `asap-gorilla::GorillaDecoder` on the backend side.

(Initial impl used the brief's example "ASGB" 4-byte magic which was not
byte-compatible with #281 — that was reconciled to GORILLA1 for cross-runtime
chunk interchange.)

## S3 sink

aws-sdk-go-v1, S3ForcePathStyle: true for MinIO compatibility, retry/backoff,
optional local spool fallback for S3-unavailable failure mode. Per-hour
`index.json` is read-modify-write with in-memory mutex + small cache; multi-
writer is best-effort and a sidecar compactor is the source-of-truth for
multi-agent sharing the same hour bucket.

## Config

Window 60s, prefix template `{tenant}/{metric}/{YYYY}/{MM}/{DD}/{HH}/`,
drop_original=true canonical, encryption omitted (follow-up). Self-mon emits
`gorillas3_chunks_written_total`, `gorillas3_chunk_payload_bytes_total`,
`gorillas3_data_points_encoded_total`, `gorillas3_s3_put_failures_total`.

## Tests

16 tests across config / encoder / processor (round-trip, regular-interval
compression sanity, multi-series-per-metric grouping, multi-metric chunk
splitting at max_object_bytes, drop_original behavior, factory registration,
shutdown drain).

Build/vet/test verified in main repo (submodules required); worktree submodules
unavailable so local repo verification deferred.

## OCB integration

`cmd/sketchcollector/builder-config-sketches.yaml` registers gorillas3processor.
`deploy/configs/sketchcol-agent-gorillas3-tier.yaml` is an example agent config
pointed at the existing MinIO container (`endpoint: http://minio:9000`,
`bucket: asap-gorilla`, `drop_original: true`).

## Spec ambiguity surfaced for Phase 0 design doc tightening

- Lock GORILLA1 8-byte magic + 13-byte block header in design doc spec
- Lock index.json schema (Version/Tenant/Metric/Entries[])
- Lock prefix-template token set: {tenant} {metric} {YYYY} {MM} {DD} {HH}
- Multi-writer index ownership: best-effort writer-side, sidecar compactor authoritative

Closes part of the Gorilla-S3-cold-engine track per
`docs/design-gorilla-s3-cold-engine.md` §5 (PR #280).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol deleted the feat/asap-gorilla-crate branch May 9, 2026 18:00
zzylol added a commit that referenced this pull request May 18, 2026
…eConfig (ASAPCollector#381 Issue #4) (#389)

Two paired fixes to close the apply-loop the smoke test surfaced
after ASAPQuery-backend PR #279 landed:

(1) No-op short-circuit in processRemoteConfig

Before writing the pushed body to disk + exiting, compare to the
current on-disk file. If they match byte-for-byte, just report
APPLIED and return — don't restart. This handles two scenarios:

  * Defense-in-depth against a controller resending the same
    config (which can happen any time the OpAMP server's
    LastRemoteConfigHash cache is empty or stale — e.g. on
    reconnect before the agent has reported APPLIED).
  * The deterministic-emit guarantee from ASAPQuery-backend #281
    means same-semantic-content → same bytes, so this check is
    accurate for the controller's emit.

(2) Advertise ReportsRemoteConfig capability

In addition to AcceptsRemoteConfig, set
`AgentCapabilities_AgentCapabilities_ReportsRemoteConfig` when
`remote_config_path` is configured. Without this bit,
`opampClient.SetRemoteConfigStatus(APPLIED)` returns
`ErrReportsRemoteConfigNotSet` (opamp-go
client/internal/clientcommon.go:21), the client lib refuses to
record the applied hash, and the server has no way to learn the
config was processed — keeping it stuck in the resend loop. Both
bits are needed for the full feedback cycle.

End-to-end smoke (post-#281 deterministic emit + this PR):
  * Agent boots from bootstrap YAML
  * Receives RemoteConfig push, bytes differ, applies + exits
  * Restarts, loads new YAML
  * Receives RemoteConfig push, bytes IDENTICAL (deterministic emit)
  * No-op short-circuit fires → SetRemoteConfigStatus(APPLIED) →
    keeps running, no more restarts
  * Pipeline emits sketches normally; smoke test Axis D registers
    them

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant