Split Gorilla cold path delivery modes - #359
Merged
Merged
Conversation
…ntations # Conflicts: # asap-gorilla-go/go.mod # opentelemetry-collector-contrib-patch/processor/gorillas3processor/README.md # opentelemetry-collector-contrib-patch/processor/gorillas3processor/processor.go
6 tasks
zzylol
added a commit
that referenced
this pull request
May 9, 2026
Adds the three missing per-family gateway merge processors so the controller's typed-stage emit produces YAML the patched collector build can actually load. Without these, the gateway emits names (`kllmerge`, `ddsketchmerge`, `hllmerge`) that have no registered processor — every gateway-merge plan silently failed to load and criterion #2 (controller assigns aggregation tasks to executors) was unverifiable. New processors mirror the existing CMS/CountSketch merge processors but consume typed sketch metric types (`KLLSketch` / `DDSketch` / `HLLSketch`) and key accumulators by serialized attribute set. `processMetrics` is pass-through; the only side effect is updating the in-memory accumulator map (matching existing template). - KLL: Proto/Unspecified replace; Msgpack/MsgpackDelta reserved (warn + drop); no delta variant in sketchlib-go. - DDSketch: Proto via `NewFromStateProtoBytes`; ProtoDelta via `ApplyDelta`; Msgpack via `DeserializeMsgpack`; MsgpackDelta reserved (warn + drop). - HLL: Proto via `DeserializeHyperLogLogFromProtoBytes`; Delta via `DeserializeRegisterDelta` + `ApplyRegisterDelta`; Msgpack supported. Other fixes in same PR: - controller/src/config/stage_config.rs: Cms merge name was `countminmerge` — corrected to `countminsketchmerge`, matching the registered processor type. Doc comments updated to reflect the family-specific naming convention. - deploy/scripts/run_mvp_demo.sh: per-metric capture loop now iterates all 6 distinct metrics in mvp-workload.yaml (was 2, causing 4 sketch families' bindings to be missing from the per-metric snapshot used by the §8 verdict). Replay-class comment count corrected from 6 to 7. - gorillas3processor: removed legacy GORILLA1 wire-format files unused after PR #359's cold-path delivery-modes split: `tsdb_block_writer.go` (test-only shim — runtime uses `gorilla.StreamingTSDBBlockBuilder` directly), `encoder.go`, `window.go`, `postings.go`, and their _test.go siblings. Inlined the 7-line `attributesToMap` helper into `processor.go`; added `readAllSamples` test helper into `processor_test.go`. The 3-mode architecture (best_effort / durable_raw / durable_fragment) is unaffected — runtime path is unchanged. Verification: - `go test ./...` in each new processor dir: 4 tests passing each - `go test ./...` in gorillas3processor: ok - `cargo test --manifest-path controller/Cargo.toml -- --test-threads=1`: 621 passed - `python3 -m pytest deploy/scripts/tests/ -q`: 68 passed Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 9, 2026
…363) Removes scaffolding that was unused after the cold-path delivery-modes split (#359) and the asap-otel unification. The user explicitly authorized breaking external workflows (build scripts, paper sweeps); old benchmark scripts that targeted deleted cmd dirs are removed rather than repointed. Removed: - `opentelemetry-collector-contrib-patch/processor/gorillaprocessor/` (non-S3 sibling — never wired into MVP path; gorillas3processor with `delivery_mode={best_effort,durable_raw,durable_fragment}` covers all cold-path use cases now). - 12 old per-sketch + per-collector cmd dirs: `gorillacol`, `sketchcollector`, `countminsketchcol`, `countsketchcol`, `ddsketchcol`, `hllcol`, `intervalcol`, `nopcol`, `otelcontribcol`, `serfagentcol`, `serfbackendcol`, `serfcol`. Only `asap-otel`, `asap-otel-opamp`, `kll`, and `telemetrygen` remain. - 8 root `backup_*.sh` + 8 root `restore_*.sh` patch-overlay helpers; the patch tree is the source of truth now. - 7 `otel_collector_benchmark/bench_*.sh` scripts targeting deleted col binaries. - `build_ddsketchcol.sh` (root); `build_asap_otel.sh` and `build_asap_telegraf.sh` retained but no longer call the deleted restore scripts. - v0 agent-runtime header aliases (`sketchcollector` / `sketchotap` / `sketchtelegraf`) in `controller/src/config/mod.rs` + their unit test. The `TODO(remove-after-2026-Q3)` marker is obsolete; the new aliases (`asap-otap`/`otap`, `asap-telegraf`/`telegraf`, `asap-otel`) cover all in-tree agents. Updated (not deleted): - `opentelemetry-collector-contrib-patch/cmd/asap-otel*/builder-config.yaml`: dropped the `gorillaprocessor` entry. - `opentelemetry-collector-contrib-patch/cmd/asap-otel-opamp/supervisor-config.yaml`: fixed stale `executable: ./countminsketchcol` → `./asap-otel-opamp`. - `setup.sh`, `README.md`: removed Step 5 patch-overlay paragraph + ddsketchcol build instructions; repointed users to `build_asap_otel.sh`. - `tests/{otel_controller_e2e_test.sh, promql_e2e_collector_test.sh}`: repointed `DDSKETCHCOL` → `ASAP_OTEL`. - `docs/e2e-test-guide.md`: 3 references repointed. - `datasets_eval/debs/benchmark/run.py`: `DEFAULT_COLLECTOR_PATHS` routes all sketch types (except `kll`) at the unified `asap-otel` binary. - `otel_collector_benchmark/README.md` + `cardinality_crossover/` + `matched_accuracy/`: stale `countminsketchcol/builder-config.yaml` comments updated. Verification: - `cargo test --manifest-path controller/Cargo.toml -- --test-threads=1` → 620 passed (was 621; -1 from removed v0-alias unit test). - All 11 patched Go processor test suites pass. - `python3 -m pytest deploy/scripts/tests/ -q` → 68 passed. Known follow-ups (deferred to future PRs): - `cmd/asap-otel/{components.go,go.mod,go.sum}` still hold stale `gorillaprocessor` import lines; OCB regenerates them on next `build_asap_otel.sh`. - `cmd/bench.sh` still has dead case branches for the removed per-sketch cmd dirs (only `kll*` branches remain runnable). - `docs/{pipeline-query-catalog,serf-compression-architecture,opamp-config-push}.md` mention deleted col binaries in explanatory prose; rewrite is out of scope for a code-cleanup PR. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
zzylol
added a commit
that referenced
this pull request
May 9, 2026
…ify (#367) User asked to align the integration tests with the current cold/warm paths and unify the parity gates. Three independent changes in one commit: ## 1. Cold-path test rewritten for Prometheus TSDB blocks `integration/gorilla_s3_e2e/` → `integration/e2e-cold-path/`. The old GORILLA1 XOR-delta decoder no longer applies — after PR #359 `gorillas3processor` runtime emits Prometheus TSDB blocks via `gorilla.NewStreamingTSDBBlockBuilder`. New approach: mc find <bucket> -name 'meta.json' -> per ULID: mc cat the 3 block files into a temp dir -> tsdb.OpenBlock(dir, …) + NewBlockQuerier -> stream (label, ts, value) tuples -> multi-set equality vs golden 100-sample fixture Deleted: `gorilla_decoder.go` + `gorilla_decoder_test.go` (legacy GORILLA1). `runAccuracyExact` / `runDataSourceMarker` / `runCrossLanguageByteCompat` subtests dropped (TSDB is upstream- Prometheus's native format; PromQL accuracy markers and Rust cross- decode no longer apply at this layer). Compose project renamed `asap-gorilla-e2e` → `asap-cold-e2e`; env-var prefix `GORILLA_E2E_*` → `COLD_E2E_*`. Backend service dropped from the overlay (cold-path test no longer probes PromQL; that's the warm-path test's job). `deploy/configs/asap-otel-agent-gorillas3-tier.yaml` updated to current processor schema: `role: gateway_raw`, `delivery_mode: durable_raw`, added `tsdb_bucket`, dropped obsolete `prefix_template` + `tenant`. ## 2. New warm-path e2e gate `integration/e2e-warm-path/` — TestWarmPathE2E gated on `WARM_E2E_LIVE=1`: 100 deterministic latency samples (1..100ms ramp, 100ms spacing) -> OTLP -> asap-otel-warm (DDSketch window mode, _quantile suffix) -> backend-warm (asap/query-backend:dev) curl http://localhost:29191/api/v1/query ?query=quantile_over_time(0.99, http_requests_total_latency_ms[30s]) -> assert |estimate - 99.0| / 99.0 <= ε=0.01 -> assert data_source = "sketch_warm_tier" (canonical) or fuzzy contains("warm") End-to-end runtime budget ~45-55s. No MinIO/Thanos/fake-exporter — focused warm-tier sketch ingest + query gate. ## 3. Parity tests unified ``` integration/parity/ ├── README.md # new index — tabular summary of the 3 gates ├── runtime-impl/ # was integration/parity/ (Phase 2) ├── codec/ # was integration/cross-host-parity/ (Phase 4) └── agent-binary/ # was integration/cross_host_parity/ (Phase 5) ``` Three `git mv` operations preserve history. Each subdir's go.mod module path updated to `…/integration/parity/{runtime-impl|codec| agent-binary}` and all `replace` directives' relative paths bumped one level deeper. Cross-references repointed in `parity_test.go`, `golden_test.go`, `goldenFixtureDir` candidates, `queries-e2e.json` lookup, `run_parity.sh`'s `REPO_ROOT` walk-up + `GOLDEN_PARITY_DIR`, and the 5 volume-mount paths in `deploy/docker-compose/cross-host-parity.yml`. Verification: - `cd integration/parity/runtime-impl && go test ./...` → ok (5 sketch tests + 5 golden generators) - `cd integration/parity/codec && go test ./...` → ok - `cd integration/parity/agent-binary && go test ./...` → ok - `cd integration/e2e-cold-path && go build ./... && go vet ./... && go test ./...` → clean (live test SKIPs without `COLD_E2E_LIVE=1`) - `cd integration/e2e-warm-path && go build ./... && go vet ./... && go test ./...` → clean (live test SKIPs without `WARM_E2E_LIVE=1`) - `gorillas3processor` Go tests still pass. Out of scope (left as historical refs): old paths still mentioned in `PROGRESS.md`, `docs/system-overview.md`, `docs/design-asap-*- integration.md`, `docs/phase-2-perf-bench-go.md`. Will sweep in a follow-up docs PR if the user wants. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
asap-gorilla-gobest_effort,durable_raw, anddurable_fragmentgorillas3processorroles intoagent,gateway_fragment, andgateway_rawVerification
go test -mod=readonly ./...inasap-gorilla-gogo test -mod=readonly ./...inopentelemetry-collector-contrib-patch/processor/gorillas3processor