test(integration): TSDB cold-path rewrite + warm-path e2e + parity unify - #367
Merged
zzylol merged 1 commit intoMay 9, 2026
Merged
Conversation
User asked to align the integration tests with the current cold/warm paths and unify the parity gates. Three independent changes in one commit: ## 1. Cold-path test rewritten for Prometheus TSDB blocks `integration/gorilla_s3_e2e/` → `integration/e2e-cold-path/`. The old GORILLA1 XOR-delta decoder no longer applies — after PR #359 `gorillas3processor` runtime emits Prometheus TSDB blocks via `gorilla.NewStreamingTSDBBlockBuilder`. New approach: mc find <bucket> -name 'meta.json' -> per ULID: mc cat the 3 block files into a temp dir -> tsdb.OpenBlock(dir, …) + NewBlockQuerier -> stream (label, ts, value) tuples -> multi-set equality vs golden 100-sample fixture Deleted: `gorilla_decoder.go` + `gorilla_decoder_test.go` (legacy GORILLA1). `runAccuracyExact` / `runDataSourceMarker` / `runCrossLanguageByteCompat` subtests dropped (TSDB is upstream- Prometheus's native format; PromQL accuracy markers and Rust cross- decode no longer apply at this layer). Compose project renamed `asap-gorilla-e2e` → `asap-cold-e2e`; env-var prefix `GORILLA_E2E_*` → `COLD_E2E_*`. Backend service dropped from the overlay (cold-path test no longer probes PromQL; that's the warm-path test's job). `deploy/configs/asap-otel-agent-gorillas3-tier.yaml` updated to current processor schema: `role: gateway_raw`, `delivery_mode: durable_raw`, added `tsdb_bucket`, dropped obsolete `prefix_template` + `tenant`. ## 2. New warm-path e2e gate `integration/e2e-warm-path/` — TestWarmPathE2E gated on `WARM_E2E_LIVE=1`: 100 deterministic latency samples (1..100ms ramp, 100ms spacing) -> OTLP -> asap-otel-warm (DDSketch window mode, _quantile suffix) -> backend-warm (asap/query-backend:dev) curl http://localhost:29191/api/v1/query ?query=quantile_over_time(0.99, http_requests_total_latency_ms[30s]) -> assert |estimate - 99.0| / 99.0 <= ε=0.01 -> assert data_source = "sketch_warm_tier" (canonical) or fuzzy contains("warm") End-to-end runtime budget ~45-55s. No MinIO/Thanos/fake-exporter — focused warm-tier sketch ingest + query gate. ## 3. Parity tests unified ``` integration/parity/ ├── README.md # new index — tabular summary of the 3 gates ├── runtime-impl/ # was integration/parity/ (Phase 2) ├── codec/ # was integration/cross-host-parity/ (Phase 4) └── agent-binary/ # was integration/cross_host_parity/ (Phase 5) ``` Three `git mv` operations preserve history. Each subdir's go.mod module path updated to `…/integration/parity/{runtime-impl|codec| agent-binary}` and all `replace` directives' relative paths bumped one level deeper. Cross-references repointed in `parity_test.go`, `golden_test.go`, `goldenFixtureDir` candidates, `queries-e2e.json` lookup, `run_parity.sh`'s `REPO_ROOT` walk-up + `GOLDEN_PARITY_DIR`, and the 5 volume-mount paths in `deploy/docker-compose/cross-host-parity.yml`. Verification: - `cd integration/parity/runtime-impl && go test ./...` → ok (5 sketch tests + 5 golden generators) - `cd integration/parity/codec && go test ./...` → ok - `cd integration/parity/agent-binary && go test ./...` → ok - `cd integration/e2e-cold-path && go build ./... && go vet ./... && go test ./...` → clean (live test SKIPs without `COLD_E2E_LIVE=1`) - `cd integration/e2e-warm-path && go build ./... && go vet ./... && go test ./...` → clean (live test SKIPs without `WARM_E2E_LIVE=1`) - `gorillas3processor` Go tests still pass. Out of scope (left as historical refs): old paths still mentioned in `PROGRESS.md`, `docs/system-overview.md`, `docs/design-asap-*- integration.md`, `docs/phase-2-perf-bench-go.md`. Will sweep in a follow-up docs PR if the user wants. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 26, 2026
The integration/ suite (e2e cold/warm/kafka + three parity gates) had rotted into the worst state — half-maintained, protecting nothing: - No CI ran it (the repo has no .github / GitHub Actions). - parity/runtime-impl no longer compiled against current APIs (ddsketchprocessor.Config dropped MetricSuffix; the patched pmetric sketch datapoints dropped Count) — build failed. - The Go<->Rust cross-language gate was silently skipping: after #367 the goldens moved to integration/parity/runtime-impl/golden/, but the Rust test still looks under integration/parity/golden/, so load_golden returned None and every case returned early. Remove the suite plus its only functional dependency: - asap-precompute-rs/tests/cross_language_parity.rs — existed solely to load integration/parity/.../golden/*.bin (already a no-op skip). - .gitignore — drop the now-dead integration/parity/golden/*.bin rule. integration/ subdirs are standalone modules (no go.work, no Go importer), so removal is build-safe. Stale doc/comment references (docs/*.md, asap-precompute-go/sketches, countsketchprocessor, opentelemetry-go-patch) are non-breaking and left for a follow-up. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Aligns integration tests with the current cold/warm paths and unifies the three parity gates. 58 files changed, +2,355 / −2,016.
1. Cold-path test rewritten for Prometheus TSDB blocks
`integration/gorilla_s3_e2e/` → `integration/e2e-cold-path/`. The old GORILLA1 XOR-delta decoder no longer applies — after PR #359 `gorillas3processor` runtime emits Prometheus TSDB blocks. New approach: `mc find -name 'meta.json'` → `mc cat` the 3 block files → `tsdb.OpenBlock` + `NewBlockQuerier` → multi-set equality vs golden fixture. Deleted `gorilla_decoder.go` + `gorilla_decoder_test.go`. Compose project + env-var prefix renamed (`GORILLA_E2E_` → `COLD_E2E_`). Backend service dropped from the overlay; cold-path test no longer probes PromQL.
`deploy/configs/asap-otel-agent-gorillas3-tier.yaml` updated to current processor schema (`role`, `delivery_mode`, `tsdb_bucket`).
2. New warm-path e2e gate
`integration/e2e-warm-path/` — `TestWarmPathE2E` gated on `WARM_E2E_LIVE=1`:
3. Parity tests unified
```
integration/parity/
├── README.md # new index
├── runtime-impl/ # was integration/parity/
├── codec/ # was integration/cross-host-parity/
└── agent-binary/ # was integration/cross_host_parity/
```
Three `git mv` operations preserve history. `go.mod` module paths updated; cross-references in `parity_test.go`, `golden_test.go`, `run_parity.sh`, and 5 compose volume mounts repointed.
Test plan
Known follow-ups (out of scope)
🤖 Generated with Claude Code