refactor: centralized series_id namespace + remove metric-name suffixes + VictoriaMetrics + relocate controller by zzylol · Pull Request #372 · ProjectASAP/ASAPCollector

zzylol · 2026-05-10T17:12:51Z

Summary

Agent / wire-format / OTel-Go side of a multi-phase refactor. Companion PR in ASAPQuery-backend lands the backend half (controller relocation, asap-common deletion, SeriesIdResolver + SketchIndex scaffolding).

Design doc: ASAPQuery-backend/docs/design-controller-into-backend.md covers target architecture, capability model, OTLP metadata layout, centralized series_id namespace with idempotency + ghost-sid handling + fault-tolerance, and per-phase migration order.

What landed (per phase)

Phase 1 — Metric-name suffix removal across 5 sketch processors (DD/KLL/HLL/CMS/CS). Sketch encoding lives in the OTLP pdata variant tag, not in a _quantile/_topk/etc. name suffix. Backend ingests under raw input metric name; PromQL fired against raw name resolves directly against stored sketch state.
Phase 2 — OTLP proto patch:
- metrics.proto: drop precomputed/duplicated DataPoint fields (count/sum/min/max/cardinality/sample_count/dimension/epsilon/delta) via reserved; lift per-instance sketch config to parent container (DDSketch.relative_accuracy, KLLSketch.k, HLLSketch.precision, CountSketch.{rows,cols}, CountMinSketch.{rows,cols}).
- metrics_service.proto: add ExportMetricsServiceResponse.unknown_series_ids — universal sid-cache invalidation primitive.
- Go wire bindings regenerated under opentelemetry-proto-patch/gen/go/.
Phase 3 — VictoriaMetrics swap for B0/B1: agent PRW endpoint → http://victoriametrics:8428/api/v1/write. Smoke verified at 10K series, ~10 samples per series in 10s window, 10 distinct producer_ids.
Phase 4 (Go-side) — applyUnknownSeriesIds + Dictionary::EvictByID in the patched OTel-Go exporter. Recovery primitive: when receiver returns unknown_series_ids, sender evicts and re-emits with attributes on next push.
Phase 6 (config) — backend-storage-routing.yaml removes per-shape allow-list; raw-named metrics fall through to gorilla_s3_archive (Thanos forward) on any warm-tier miss. Runtime SidLookup::{Hit,Ghost,Unknown} classification (in companion PR) drives the actual routing decision.
Phase 9 — controller/ tree deleted (moved to ASAPQuery-backend in the companion PR). Dockerfile.controller removed (no separate image needed). 11 agent YAMLs: ws://controller:4320/v1/opamp → ws://backend:4320/v1/opamp.

Other:

fake-exporter/main.go: producer_id added as a metric label so multi-producer setups don't collapse onto one series.
fake-exporter/traces/ → replay-data/: avoid clash with OTel-trace terminology.
asap-otel-gateway-mvp-placeholder.yaml: compression: none on the OTLP exporter (gateway was dropping every batch under default gzip).

Deferred / out-of-scope (separate PRs)

Phase 2.5 — pdata regen + sketch processor source updates so DataPoints stop writing the now-reserved fields and start writing the lifted parent-container fields. Requires running upstream pdatagen against the patch overlay.
Phase 4 finish — backend ResolveSeriesIDs gRPC service (in companion PR's scaffolding); gateway transparent-forwarder mode.
Phase 5 finish — backend SketchStore migration off legacy aggregation_id keys (companion PR has the new types ready).
Phase 7 — full empirical end-to-end verification once Phase 4/5 wire through.

Test plan

B0/B1 smoke with VictoriaMetrics: 10K series, count_over_time[10s] ≈ FREQ × 10, distinct producer_ids = N_PRODUCERS_PER_NODE × 2
bash build_asap_otel.sh produces a binary with the new bindings (no compile errors from removed fields)
PromQL query against raw metric name resolves through sketch path (after Phase 4/5 final wiring lands)
Apply Phase 2.5 changes; rebuild; verify wire bytes show no count/sum/min/max on DataPoints

🤖 Generated with Claude Code

…es + VictoriaMetrics + relocate controller This commits the ASAPCollector side of a multi-phase refactor that aligns the project with a new design where the asap-query-backend host is the single authoritative point for series_id minting and the controller runs in-process inside it. Companion PR in ASAPQuery-backend lands the backend half (controller crate relocation, asap-common deletion, new SeriesIdResolver + SketchIndex modules). Design doc: ASAPQuery-backend/docs/design-controller-into-backend.md Phase 1 — strip metric-name suffixes from sketch processors Five sketch processors no longer append metric_suffix at emit. The sketch encoding lives in the OTLP pdata variant tag, not in a name suffix. Backend ingests under raw input metric name; PromQL fired against the raw name resolves directly against the stored sketch state. - opentelemetry-collector-contrib-patch/processor/{ddsketch,kll, hll,countsketch,countminsketch}processor/* — encode paths preserve input metric name end-to-end. Phase 2 — OTLP proto patch metrics.proto — drop precomputed/duplicated DataPoint fields (count/sum/min/max/cardinality/sample_count/dimension/epsilon/delta) via `reserved`; lift per-instance sketch config to the parent sketch container (DDSketch.relative_accuracy, KLLSketch.k, HLLSketch.precision, CountSketch.{rows,cols}, CountMinSketch.{rows,cols}). Eliminates per-DataPoint duplication and cache-invalidation drift. metrics_service.proto — add ExportMetricsServiceResponse.unknown_series_ids, the universal sid-cache invalidation primitive (sender evicts, re-emits with attributes, receiver re-resolves). Same code path covers cold bootstrap and every recovery scenario. Go wire bindings regenerated under opentelemetry-proto-patch/gen/go/. Phase 3 — VictoriaMetrics for B0/B1 (config + smoke validated 10K series) - deploy/configs/asap-otel-agent-{b0-prometheus,b1-serf-prometheus}.yaml: PRW endpoint http://victoriametrics:8428/api/v1/write. Phase 4 — Go-side eviction primitive - opentelemetry-go-patch/exporters/otlp/otlpmetric/otlpmetricgrpc/ exporter.go: applyUnknownSeriesIds reads response.UnknownSeriesIds and evicts via dictionary.go::Dictionary::EvictByID. Matches the patched-vs-upstream proto behavior switch via reflective method dispatch. Phase 6 — universal warm-miss → archive fallthrough - deploy/configs/backend-storage-routing.yaml — raw-named metrics (http_requests_total, http_requests_total_latency_ms) fall through from sketch_warm_tier to gorilla_s3_archive (Thanos forward) on any shape miss; the runtime SidLookup classification (Hit/Ghost/ Unknown) drives the actual routing decision rather than a static shape allow-list. Phase 9 — controller relocation (out of this repo) controller/ moved to ASAPQuery-backend/controller/ (companion PR). This commit only carries the deletions on the ASAPCollector side. - deploy/docker/Dockerfile.controller deleted (no separate image). - 11 agent YAMLs: ws://controller:4320/v1/opamp → ws://backend:4320/v1/opamp. Other: - deploy/fake-exporter/main.go: producer_id label added so multi- producer setups don't collapse onto one series. - deploy/fake-exporter/traces/ → replay-data/: renamed to avoid clash with OTel-trace terminology. - asap-otel-gateway-mvp-placeholder.yaml: compression: none on the OTLP exporter (gateway was dropping batches under default gzip). Remaining work (separate PRs): Phase 2.5 — pdata regen + processor source updates so DataPoints stop writing the dropped fields and start writing the lifted parent-container fields. Requires running upstream pdatagen against the patch overlay. Phase 4 — backend ResolveSeriesIDs gRPC service (in companion PR's scaffolding) + gateway transparent-forwarder mode. Phase 5 — backend SketchStore reindex (companion PR's scaffolding). Phase 7 — full empirical end-to-end verification after Phase 4/5 are wired through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…iner fields Carries the Phase 2 OTLP proto changes through to runtime: pdata regen (opentelemetry-collector-patch/pdata/internal/generated_proto_*sketch*.go): Updated pmetric_package.go's structured field descriptors to match the proto changes — drop count/sum/min/max from DD/KLL DP, drop cardinality/precision from HLL DP, drop sample_count/rows/cols from CMS DP, drop dimension/epsilon/delta from CS DP — and add the parent-container fields (DDSketch.RelativeAccuracy, KLLSketch.K, HLLSketch.Precision, CountSketch.Rows/Cols, CountMinSketch.Rows/Cols). Then ran upstream pdatagen against the patch overlay to regenerate the typed accessors. Generated_proto_*_test.go files updated as well. Sketch processor source updates (5 files): - ddsketchprocessor/shim_helpers.go::stampDPMetadata — parent.SetRelativeAccuracy(p.cfg.RelativeAccuracy) on the DDSketch container; per-DP Count stamping removed. - kllprocessor/encode.go::encodeTypedSketch — parent.SetK(uint32(p.cfg.K)); per-DP Count removed. - hllprocessor/encode.go::encodeTypedSketch — parent.SetPrecision(uint32(hll.HLLPrecision)); per-DP Count/Cardinality/Precision stamping removed (cardinality snapshot cache retained for potential future attribute use, but no longer written to the DP). - countsketchprocessor/shim_helpers.go::stampDPMetadata — parent.SetRows(int32(rows)) / SetCols(int32(cols)) from configDimensions; per-DP Dimension/Epsilon/Delta stamping removed. - countminsketchprocessor/shim_helpers.go::encodeTypedSketch — parent.SetRows(int32(p.cfg.Rows)) / SetCols(int32(p.cfg.Columns)); per-DP SampleCount/Rows/Cols stamping removed. Build verification: bash build_asap_otel.sh --skip-patches builds the asap-otel binary clean (Compiled in 8s on warm cache). Closes Phase 2.5 of the refactor; carries the wire-format changes from Phase 2 fully through to the agent's emit path. Backend will pick up the new parent-container fields once the companion ASAPQuery-backend PR's Phase 4 + 5 finishing work wires through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…Phase 9) (#373) Phase 9 of the centralized-sid refactor moved the controller crate from this repo into ASAPQuery-backend, and the original Dockerfile.controller was deleted in #372. But base.yml's controller: service still tries to `build:` from a Dockerfile.controller that doesn't exist, so `docker compose up` fails on the very first phase of run_mvp_demo.sh. Restore the controller container the right way: 1. Dockerfile.backend builds both bins from ASAPQuery-backend workspace (`--bin query_engine_rust --bin controller`) and ships `/usr/local/bin/controller` alongside the existing `/usr/local/bin/asap-query-backend`. EXPOSE bumped to include 8080 / 4320 / 4321 (controller HTTP / OpAMP / gRPC). 2. base.yml's controller: block drops the `build:` block; uses `image: asap/query-backend:dev` with `entrypoint: ["/usr/local/bin/controller"]`. Verified by running run_mvp_demo.sh --mode both end-to-end — baseline arm completes cleanly, ASAP arm reaches Phase 6 (thanos-compact) which is a downstream housekeeping concern. Co-authored-by: zz_y <zz_y@node0.zz-y-304941.softmeasure-pg0.clemson.cloudlab.us> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…for HLL Two Phase-1-era staleness bugs in the replay/e2e query JSONs: 1. Per-sketch metric-name suffixes (_quantile, _kll, _hll) were dropped in #372 — the backend ingests under raw input names and the OTLP pdata variant tag carries the sketch type. The query JSONs still referenced the legacy suffixed names `http_requests_total_latency_ms_{quantile,kll,hll}`. Switched to the raw metric names per `mvp-workload.yaml`: - DDSketch quantile → http_requests_total_latency_ms - KLL quantile → http_requests_total_latency_ms - HLL cardinality → unique_users_per_min 2. The HLL JSON used `count_over_time(metric[r])` which counts SAMPLES, not distinct cardinality. Replaced with MetricsQL's `distinct_over_time(metric[r])` — the natural primitive for the HLL cardinality estimate. See https://docs.victoriametrics.com/victoriametrics/metricsql/#distinct_over_time CMS / CS JSONs already use the raw name `http_requests_total`; left unchanged. NOTE: the backend's warm-tier analyzer (warm_tier_analysis.rs in controller, PR #128) does not yet recognize `distinct_over_time` — it accepts `count_distinct_over_time` / `cardinality_estimate` / `count_distinct`. The analyzer-side alias landing is a follow-up that pairs naturally with the Prometheus → VictoriaMetrics swap; until that lands, the HLL queries here will surface as `UnsupportedFunction("distinct_over_time")` → archive forward, which is the safe fallback. Both `deploy/scripts/` (canonical) and `deploy/mvp-multinode/scripts/` (rsync source) updated to keep them in sync — the multinode orchestrator rsyncs deploy/scripts/ to each node at Phase 0 anyway, but the multinode copy is committed to make the 4-node demo reproducible from a fresh clone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(deploy): 4-node MVP demo orchestrator under version control The 4-node MVP demo driver was built in earlier sessions as local-only scratch under /mydata/mvp-multinode/. Bring it into the repo as deploy/mvp-multinode/ so it's reproducible from a fresh clone. ## Why a separate driver? The single-host driver (deploy/scripts/run_mvp_demo.sh) is fine for PR-time smoke but flatters the bandwidth claims because every "edge" is loopback. The 4-node driver places the producers (node0/3), the gateway (node1), and the backend stack (node2) on separate boxes on a 10 Gbps LAN so wire-bytes per cut edge reflect real NIC traffic. ## Topology - node0 (10.10.1.1) — producers + agent-a (data source) - node1 (10.10.1.2) — gateway (ASAP arm only) - node2 (10.10.1.3) — backend stack (asap-query-backend, minio, thanos-*, prometheus, embedded controller) - node3 (10.10.1.4) — producers + agent-b (data source) ## Image refs updated for Phase 9 topology.env's IMAGES array dropped `asap/controller:dev` per #373: the controller binary now ships from asap/query-backend:dev (multi-bin build in Dockerfile.backend), so there are only 3 images to build+distribute, not 4. run_demo.sh's backend bring-up was already updated for this in an earlier session (no separate controller container; controller is in-process inside the asap-backend container). The 3 images that DO need building+distributing on each node: - asap/asap-otel:dev (build_asap_otel.sh from ASAPCollector root) - asap/fake-exporter:dev (docker build deploy/fake-exporter/) - asap/query-backend:dev (Dockerfile.backend with 4 build contexts) ## What's NOT committed - results/ and logs/ — per-run transient artifacts, gitignored. Same convention as the single-host driver (deploy/eval-results/ is also gitignored). Never commit historical reports — they get stale fast and look like source of truth. - autopilot.log — transient sweep log. - scripts/run_mvp_demo.sh from the source tree — was a stale rsync target; the canonical single-host driver lives at deploy/scripts/run_mvp_demo.sh and run_demo.sh rsyncs deploy/scripts/ to each node at bring-up. ## README deploy/mvp-multinode/README.md documents the topology, image set, bring-up commands (with the correct Dockerfile.backend invocation including the four build contexts), knobs, and the per-node artifact layout. Includes a comparison table single-host vs 4-node so people pick the right driver for the question they're asking. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deploy/mvp-multinode: prune configs to only what run_demo.sh mounts run_demo.sh + topology.env reference exactly these 9 files; the other 40 were sweep / experiment / alt-storage configs from sibling work that crept into the multinode tree's configs/ during earlier sessions. Pruning so the PR is reviewable + the directory shows intent. Kept (mounted by run_demo.sh): - asap-otel-agent-b0-prometheus.yaml (baseline arm — b0) - asap-otel-agent-b1-serf-prometheus.yaml (b1 arm) - asap-otel-agent-b6-asap-single-sketch.yaml (asap arm) - asap-otel-gateway-mvp-placeholder.yaml (gateway, ASAP only) - backend-streaming.yaml (backend ingest) - backend-storage-routing.yaml (backend warm-tier + archive routing) - mvp-workload.yaml (controller workload spec) - prometheus-with-remote-write.yml (B0 Prometheus scrape config) - thanos-objstore.yaml (thanos sidecar/compact MinIO objstore) Removed: 40 unused YAML / JSON / md / test-data files. * cleanup: update single-host configs for Phase-9 single-binary refactor Two stale `asap/controller:dev` image references hung around in single-host topology after #373 retired the standalone controller image: - deploy/docker-compose/e2e-overlay.yml — overrode the controller service's image: + build:. With base.yml now pinning the right image (asap/query-backend:dev) and there being no build stanza to neutralize, both overrides are unnecessary; the overlay keeps its CONTROLLER_WORKLOADS + CONTROLLER_BACKEND_ENDPOINT env tweaks + workloads.yaml mount. - deploy/helm/asap/values.yaml — controller chart key updated to image: asap/query-backend:dev with entrypoint: /usr/local/bin/controller, mirroring the docker-compose pattern. Also: add an "Essential YAML configs the MVP demo touches" section to docs/mvp-demo-runbook.md so the 9-file working set is documented in one place (vs the 49 yaml files in deploy/configs/ that mostly belong to baseline-sweep / alt-storage overlays). Adds the single-binary image table for completeness. No deletions in deploy/configs/ — the env-var-parameterized mounts (e.g. ${AGENT_CONFIG:-...}) make it hard to prove orphan-ness via a static grep, and the baseline-sweep overlays exercise many of those files. The runbook now lists the essentials so readers don't have to guess. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(scripts): demo queries use raw metric names + distinct_over_time for HLL Two Phase-1-era staleness bugs in the replay/e2e query JSONs: 1. Per-sketch metric-name suffixes (_quantile, _kll, _hll) were dropped in #372 — the backend ingests under raw input names and the OTLP pdata variant tag carries the sketch type. The query JSONs still referenced the legacy suffixed names `http_requests_total_latency_ms_{quantile,kll,hll}`. Switched to the raw metric names per `mvp-workload.yaml`: - DDSketch quantile → http_requests_total_latency_ms - KLL quantile → http_requests_total_latency_ms - HLL cardinality → unique_users_per_min 2. The HLL JSON used `count_over_time(metric[r])` which counts SAMPLES, not distinct cardinality. Replaced with MetricsQL's `distinct_over_time(metric[r])` — the natural primitive for the HLL cardinality estimate. See https://docs.victoriametrics.com/victoriametrics/metricsql/#distinct_over_time CMS / CS JSONs already use the raw name `http_requests_total`; left unchanged. NOTE: the backend's warm-tier analyzer (warm_tier_analysis.rs in controller, PR #128) does not yet recognize `distinct_over_time` — it accepts `count_distinct_over_time` / `cardinality_estimate` / `count_distinct`. The analyzer-side alias landing is a follow-up that pairs naturally with the Prometheus → VictoriaMetrics swap; until that lands, the HLL queries here will surface as `UnsupportedFunction("distinct_over_time")` → archive forward, which is the safe fallback. Both `deploy/scripts/` (canonical) and `deploy/mvp-multinode/scripts/` (rsync source) updated to keep them in sync — the multinode orchestrator rsyncs deploy/scripts/ to each node at Phase 0 anyway, but the multinode copy is committed to make the 4-node demo reproducible from a fresh clone. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(scripts): rename 'promql' JSON field to 'metricsql' The replay queries now exercise MetricsQL (VictoriaMetrics superset of PromQL) — yesterday's HLL fix landed `distinct_over_time` which is MetricsQL-only. Keeping the JSON field named `"promql"` was misleading; rename to `"metricsql"` so the schema matches the language. ## Files - 5 query JSONs (canonical at deploy/scripts/ + rsync copies at deploy/mvp-multinode/scripts/): cms, cs, hll, kll, e2e. - promql_replay.py: `load_queries` now reads the `metricsql` key, with `"promql"` accepted as a back-compat fallback (a one-line `q.get("metricsql") or q.get("promql")` guard) so external workload JSONs in user repos don't hard-break mid-migration. Internal record emits use the new key throughout. Docstring example updated. The Python script's filename (`promql_replay.py`) is unchanged for now — that rename is a larger separate concern (touches all the scripts that invoke it). Field-only rename keeps this PR atomic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(deploy/mvp-multinode): group configs by arm, drop duplicate scripts/, finish promql→metricsql rename PR #375 follow-ups: 1. deploy/mvp-multinode/scripts/ was a byte-identical copy of deploy/scripts/ and unused. run_demo.sh rsyncs deploy/scripts/ to each node at Phase 0 and invokes ${ROOT}/deploy/scripts/ for local analysis — the local mvp-multinode/scripts/ tree was dead weight + a drift hazard. Removed. 2. deploy/mvp-multinode/configs/ now groups by arm so each folder is self-contained for one baseline/asap arm: configs/b0/ B0 baseline agent configs/b1/ B1 baseline agent configs/asap/ ASAP agent + gateway + backend stack (streaming, storage-routing) + controller workload configs/shared/ Prometheus + Thanos (cross-arm) run_demo.sh mount paths and the file table in docs/mvp-demo-runbook.md updated. The Phase-0 rsync still pulls the whole configs/ tree recursively, so the subfolder structure propagates to all four nodes unchanged. 3. promql_replay.py → metricsql_replay.py (finishing the rename deferred in e0bbc35). Runtime contract was already MetricsQL- only after the HLL fix (distinct_over_time). All callers + docs updated (run_mvp_demo.sh, run_demo.sh, autopilot.sh, the two sweep scripts, plan_transition.py's import, measure-baseline's docstring, deploy/README.md, four doc pages). The load_queries() back-compat fallback for legacy "promql" JSON keys is preserved, with a regression test added. The inline replay-queries.json heredoc in run_mvp_demo.sh:536-542 that e0bbc35 missed is also migrated to "metricsql". Verification (static — multinode path requires 4 cloudlab nodes): - bash -n deploy/mvp-multinode/run_demo.sh: clean - every hardcoded /mydata/mvp-multinode/configs/... path in run_demo.sh resolves to an existing file under the new layout - python3 deploy/scripts/tests/test_metricsql_replay.py: 10/10 OK Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(deploy): split tree into mvp-singlenode/ + mvp-multinode/ per-demo packages PR #375 follow-up to the duplicate-scripts cleanup. The deploy/ tree mixed two demos in a single namespace: deploy/scripts/ did double-duty as both the single-host driver and the rsync source for the 4-node orchestrator, deploy/README.md described only the single-host path, and deploy/mvp-multinode/'s orchestrator scripts sat at the package root next to configs/. Reorganize so each demo is a self-contained package and the top-level deploy/ becomes an index. ## New layout deploy/ ├── README.md ← NEW: index pointing to both demos ├── docker/ ← universal Dockerfiles (unchanged) ├── fake-exporter/ ← Go source for producer image (unchanged) ├── helm/ ← K8s path (unchanged) ├── mvp-singlenode/ │ ├── README.md ← was deploy/README.md │ ├── scripts/ ← was deploy/scripts/ │ ├── configs/ ← was deploy/configs/ │ └── docker-compose/ ← was deploy/docker-compose/ └── mvp-multinode/ ├── README.md ├── topology.env ├── scripts/ ← *.sh moved here from package root │ ├── run_demo.sh │ ├── autopilot.sh │ ├── validate_arm.sh │ ├── snapshot_resources.sh │ ├── measure_freshness.sh │ └── measure_nic_bw.sh └── configs/ ← (already arm-grouped) ## Key non-obvious fixes - docker-compose YAMLs use `../configs/...` relative paths. Moving docker-compose/ AND configs/ together into singlenode/ keeps these working unchanged — no compose-file edits required. - mvp-multinode/scripts/{run_demo,autopilot}.sh now compute PKG_DIR="$(dirname ${SCRIPT_DIR})" and source topology.env from there (it stays at the package root, not under scripts/). - topology.env's CONFIG_SRC was pointing at deploy/configs/ (single-host) — pre-existing bug that round-2's per-arm subfolder reorg would have surfaced on first run. Repointed to deploy/mvp-multinode/configs/, which is what the per-arm mount paths actually expect. - deploy/mvp-singlenode/configs/tests/test_static_placeholder_5sketch_routing.py walks up to REPO_ROOT; bumped one more os.pardir + updated the embedded path string for the new layout. - 30+ docs/* and config-internal references rewritten via a scripted sed pass; verified zero `deploy/{scripts,configs,docker-compose}/` stragglers outside the singlenode/multinode trees. ## Verification (static) - bash -n: all 13 moved shell scripts clean - python3 test_metricsql_replay.py: 10/10 OK from new location - python3 test_static_placeholder_5sketch_routing.py: 9/9 OK - source mvp-multinode/topology.env: CONFIG_SRC and ROOT both resolve to existing directories - Every `../...` relative path inside the moved mvp-singlenode/docker-compose/*.yml resolves under the new tree (the one apparent miss is a `${GATEWAY_CONFIG:-...}` env-var substitution; resolved value exists) Multi-node runtime path can't be tested without a 4-node cloudlab allocation, but every static check that doesn't require ssh to peers passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * rename(mvp-multinode): autopilot.sh → scripts/run_demo_sweep.sh Matches the singlenode side's run_*_sweep.sh convention (run-baseline-sweep.sh, run_e2e_sweep.sh, etc.) and pairs by prefix with the run_demo.sh it wraps. The PR description's own words for this script were "sweep wrapper" — the file name now agrees. Also updates: - Header docstring and `[autopilot]` log prefixes → `[sweep]` - README.md "Files" table: now shows scripts/ prefix on every driver, replaces the stale "(49 files)" claim on configs/ with the actual `{b0,b1,asap,shared}/` per-arm layout, and clarifies that the per-node Python utilities live under deploy/mvp-singlenode/scripts/ (rsync'd to each node at Phase 0) — the local mvp-multinode/scripts/ tree is only the orchestrator drivers. - `deploy/mvp-multinode/run_demo.sh` ref → `scripts/run_demo.sh` in the diff table - Removes the stale `autopilot.log` line from .gitignore; nothing writes to that path (run logs go under results/${RUN_ID}/run.log which is already covered by results/). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cleanup(deploy/mvp-singlenode): delete 8 unused/superseded YAMLs Per a grep audit, all 8 files had zero load-bearing references — only stale doc comments and (for two of them) v3-compat alias mentions. The mvp-singlenode tree goes from 47 → 42 config YAMLs and 28 → 25 compose files. ## Deleted (no references anywhere) | File | Why dead | |---|---| | `configs/asap-otel-agent-b3-delta-e2e.yaml` | Header itself: "pipeline degenerates to nothing downstream of ddsketch flush" | | `configs/asap-otel-agent-passthrough.yaml` | Smoke-test relic; no compose / script / doc consumes it | | `configs/asap-otel-agent-gorillas3-tier.yaml` | Only mentioned in *comments* of other configs ("see pattern in gorillas3-tier"). Never mounted. | | `docker-compose/cross-host-parity.yml` | Phase 5 binary-mode parity harness — historical, no caller | | `docker-compose/trace-replay.yml` | CSV-trace-replay overlay; no caller (the fake-exporter has its own trace-replay mode, unrelated) | | `docker-compose/mvp-no-resource-limits.yml` | Override designed for one specific debugging session — no caller | ## Deleted (v3-compat aliases — v3 is gone) | File | Replaced by | |---|---| | `configs/asap-otel-agent-b1-serf.yaml` | `asap-otel-agent-b1-serf-prometheus.yaml` (v4 PRW variant, what `baseline-b1-serf.yml` already defaults to) | | `configs/asap-otel-agent-b5-gorilla.yaml` | `asap-otel-agent-b5-gorilla-prometheus.yaml` (v4 PRW variant, what `baseline-b5-gorilla.yml` already defaults to) | `baseline-b{1-serf,5-gorilla}.yml` carried v3-compat comment blocks saying "legacy callers using AGENT_CONFIG=…b1-serf.yaml still work" — those blocks are removed. The only remaining v3 caller was `run-baseline-sweep.sh:84,88` which hard-coded the v3 filenames; bumped to the -prometheus variants. ## Comment housekeeping for surviving files - `b1-serf-prometheus.yaml` and `b5-gorilla-prometheus.yaml` header comments dropped the "variant of X.yaml" cross-ref to the now-deleted v3 file. - `asap-otel-agent-kafka-fragment.yaml` and `asap-otel-gateway-kafka-fragment.yaml` had 4 comment lines pointing at gorillas3-tier as the canonical pattern reference; rephrased to "the older gorillas3-tier config (now removed)". - `docs/design-archive-tier.md:596` bumped to the v4 -prometheus filename. - `deploy/mvp-multinode/configs/b1/asap-otel-agent-b1-serf-prometheus.yaml` (the multinode duplicate) updated for the same dangling cross-ref. ## Verification - bash -n deploy/mvp-singlenode/scripts/run-baseline-sweep.sh: OK - grep audit: zero remaining references to the 8 deleted files (the 3 false-positive hits were unrelated: "trace-replay" the fake-exporter mode name, and "cross-host-parity" the `integration/cross-host-parity/` test directory) Note: the per-family matrix (4× backend-inference / 4× backend- streaming / 4× e2e-overlay / 4× asap-otel-agent-*-direct) was *not* touched — those are intentional per-family configs used by `run_e2e_sweep.sh:115-118` and documented as such in the README. The kafka cold-path demo trio (mvp-kafka-cold-path.yml + two kafka-fragment.yaml) was also kept — it's a manually- invoked feature demo, not dead code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(deploy/mvp-singlenode): collapse 4 per-family e2e overlays into 1 parametrized file The four files `e2e-overlay-{cms,cs,hll,kll}.yml` were byte-identical apart from the `${FAMILY}` slug in the two volume mount paths: services: backend: volumes: - ../configs/backend-streaming-<FAM>.yaml:/etc/asap/streaming.yaml:ro - ../configs/backend-inference-<FAM>.yaml:/etc/asap/inference.yaml:ro Real duplication, unlike the per-family backend-streaming / backend-inference yamls (those encode genuinely different sketch-family schemas + query catalogs and were intentionally left alone). ## What changed - New `e2e-overlay-family.yml` (1 file) uses `${FAMILY}` substitution for both mounts. - Deleted `e2e-overlay-{cms,cs,hll,kll}.yml` (4 files). - `run_e2e_sweep.sh` SKETCHES array: per-family entries now name `e2e-overlay-family.yml` instead of `e2e-overlay-<fam>.yml`. Both `docker compose down` and `up` invocations now export `FAMILY="$FAM"` alongside the existing `AGENT_CONFIG="$AGENT_YAML"`. - README.md inference-overlay table: "Mounted by" column shows `e2e-overlay-family.yml (with FAMILY=cms)` etc. ## Verification - bash -n run_e2e_sweep.sh: OK - `FAMILY=cms docker compose -f base.yml -f e2e-overlay.yml -f e2e-overlay-family.yml config` resolves the volume sources to `backend-{streaming,inference}-cms.yaml` — interpolation works - grep audit: zero references to the 4 deleted family-overlay filenames anywhere docker-compose/ count: 25 → 22. Note on what's NOT collapsed: the 4× backend-{streaming,inference}-* yamls and the 4× asap-otel-agent-*-direct yamls are not byte-near- duplicates the same way. Their differences are per-family schemas (different sketch types, different parameter shapes, different metric naming) — templating those would just move divergence into conditional logic without reducing complexity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy/mvp-singlenode): swap Prometheus → VictoriaMetrics (Step 2g) Replaces every Prometheus container + config in the single-node demo with VictoriaMetrics. The multinode demo was already swapped to VM by an earlier commit on this branch. VictoriaMetrics is a Prometheus-API-compatible TSDB that also implements MetricsQL (a PromQL superset including `distinct_over_time` which yesterday's HLL JSON fix relies on). VM accepts remote_write + OTLP HTTP natively without special flags. ## Service renames | Was | Is | |---|---| | `prometheus` (base.yml, port 9090) | `victoriametrics` (port 8428 inside; 9090 on host) | | `prometheus-b0` (multi-stage, port 19090) | `victoriametrics-b0` (8428 inside; 19090 on host) | | `prometheus-archive` (archive overlay, 29090) | `victoriametrics-archive` (8428 inside; 29090 on host) | ## File renames - `configs/asap-otel-agent-b0-prometheus.yaml` → `…-b0-victoriametrics.yaml` - `configs/asap-otel-agent-b5-gorilla-prometheus.yaml` → `…-b5-gorilla-victoriametrics.yaml` - `docker-compose/mvp-prometheus-archive.yml` → `mvp-victoriametrics-archive.yml` (b1's config kept its filename — was already on VM in an earlier commit.) ## File deletions - `configs/prometheus.yml` - `configs/prometheus-with-remote-write.yml` - `configs/prometheus-otlp-receiver.yml` ## Key flag changes | Was (Prometheus) | Is (VictoriaMetrics) | |---|---| | `--web.enable-remote-write-receiver` | (built-in; no flag needed) | | `--web.enable-otlp-receiver` | (built-in; OTLP HTTP at `/opentelemetry/api/v1/push`) | | `--enable-feature=otlp-deltatocumulative` | (no equivalent needed — VM's OTLP receiver accepts both temporalities natively) | | `--storage.tsdb.retention.time=2h` | `-retentionPeriod=2h` | | `--storage.tsdb.path=/prometheus` | (default storage path; no flag) | | `--config.file=…/prometheus.yml` | (no config file required) | | `--web.listen-address=:9090` | `-httpListenAddr=:8428` | PRW endpoint at `/api/v1/write` is identical between Prometheus and VM — the `prometheusremotewrite` OTel exporter works against VM unchanged. Only the endpoint URL in the agent configs needed updating. ## Script changes - `run_mvp_demo.sh`: `HOST_PROM_B0_PORT` → `HOST_VM_B0_PORT` with back-compat shim: `HOST_VM_B0_PORT="${HOST_VM_B0_PORT:-${HOST_PROM_B0_PORT:-19090}}"` Plus log strings + service name refs. - `measure_freshness.py`, `mvp_report.py`, `measure-baseline.py`, `run-baseline-sweep.sh`, `run_freshness_phase.sh`: comment refresh for renamed configs. ## Grafana `grafana-datasources.yml` kept `type: prometheus` (Grafana's Prometheus datasource is wire-compatible with VM); changed `name:` to `VictoriaMetrics` and `url:` to `http://victoriametrics:8428`. ## Compose validation All `docker compose … config` invocations exit 0: - `base.yml` - `base.yml -f mvp-multi-stage.yml --profile b0` - `base.yml -f mvp-multi-stage.yml -f mvp-victoriametrics-archive.yml` - (bonus) asap-mode + thanos-archive - (bonus) baseline-{b0,b1,b5}.yml + agents-N10.yml ## Diff stat 23 files changed, +417 / -559 (net -142, three Prometheus YAMLs deleted) ## Open caveats - `mvp-freshness-probes.yaml`'s `target_path: prometheus_b0` field was left as-is — it's a semantic routing identifier, not a hostname. - `baseline-{b0,b1,b5}-*.yml` overlays had their `prometheus:` service overrides dropped — those injected Prometheus-only feature flags and remounted now-deleted config files. They are now label-only overlays; VM in base.yml already serves the PRW endpoint these baselines need. - `e2e-overlay.yml`: backend's `--prometheus-server` flag value changed from `http://prometheus:9090` to `http://victoriametrics:8428`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy/mvp-singlenode): keep Prometheus as opt-in storage backend Yesterday's 2g commit made VictoriaMetrics the default storage tier for the single-node demo. Users may still want the Prometheus path (apples-to-apples baseline numbers vs the original architecture, or compatibility with externally-deployed Prometheus instances). Restore the Prometheus stack as an opt-in overlay alongside VM. ## Restored from history - `configs/prometheus.yml` (generic scrape) - `configs/prometheus-with-remote-write.yml` (b0 PRW receiver config) - `configs/prometheus-otlp-receiver.yml` (Mode-3 archive OTLP config) - `configs/asap-otel-agent-b0-prometheus.yaml` (b0 agent PRW → prometheus-b0:9090) - `configs/asap-otel-agent-b5-gorilla-prometheus.yaml` (b5 agent PRW → prometheus:9090) - `docker-compose/mvp-prometheus-archive.yml` (parallel to mvp-victoriametrics-archive.yml) ## New: `docker-compose/mvp-prometheus.yml` overlay When layered on top of `base.yml + mvp-multi-stage.yml`, this overlay: 1. Disables the VM services (`victoriametrics`, `victoriametrics-b0`) by reassigning their `profiles:` to `__disabled_when_prometheus_overlay__` via the `!override` Compose extension (default merge would have merged the arrays and left VM activated under --profile b0). 2. Re-targets grafana's `depends_on: victoriametrics` to point at prometheus instead, and swaps its datasource mount to a new `grafana-datasources-prometheus.yml`. 3. Defines `prometheus` (always-on, port 9090) + `prometheus-b0` (profile-gated under `b0`, port 19090) with command-line flags equivalent to what was there pre-VM-swap. ## New: `configs/grafana-datasources-prometheus.yml` Parallel grafana-datasources file pointing at `http://prometheus:9090` instead of `http://victoriametrics:8428`. Mounted by the overlay. ## Usage ```bash # Default (VictoriaMetrics): docker compose \ -f deploy/mvp-singlenode/docker-compose/base.yml \ -f deploy/mvp-singlenode/docker-compose/mvp-multi-stage.yml \ --profile b0 up -d # Prometheus alternative: AGENT_CONFIG_A=asap-otel-agent-b0-prometheus.yaml \ AGENT_CONFIG_B=asap-otel-agent-b0-prometheus.yaml \ docker compose \ -f deploy/mvp-singlenode/docker-compose/base.yml \ -f deploy/mvp-singlenode/docker-compose/mvp-multi-stage.yml \ -f deploy/mvp-singlenode/docker-compose/mvp-prometheus.yml \ --profile b0 up -d ``` For the asap-arm's archive tier, pair with `mvp-prometheus-archive.yml` instead of `mvp-victoriametrics-archive.yml`. ## Validation - `docker compose -f base.yml -f mvp-multi-stage.yml --profile b0 config` → ✓ valid (VM activated, no Prometheus) - `docker compose -f base.yml -f mvp-multi-stage.yml -f mvp-prometheus.yml --profile b0 config` → ✓ valid (Prometheus activated, VM profile-disabled) - Effective profile check: with overlay, `victoriametrics-b0 profiles=['__disabled_when_prometheus_overlay__']` (no longer in `b0`); `prometheus-b0 profiles=['b0']` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: zz_y <zz_y@node0.zz-y-304941.softmeasure-pg0.clemson.cloudlab.us> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

zzylol mentioned this pull request May 10, 2026

refactor: relocate controller into asap-query-backend + delete asap-common + scaffolding for centralized series_id ProjectASAP/ASAPQuery-backend#120

Merged

6 tasks

zzylol merged commit d91d6da into main May 10, 2026

zzylol deleted the refactor/centralized-sid-and-architecture-cleanup branch May 10, 2026 17:32

This was referenced May 10, 2026

feat: wire SeriesIdResolver + SketchIndex into OTLP receive + query (Phase 4+5+6 finish) ProjectASAP/ASAPQuery-backend#122

Merged

cleanup: ship controller bin from query-backend image (single-binary Phase 9) #373

Merged

zzylol mentioned this pull request Jun 14, 2026

fix(asapedge): value-weighted top-k on the agent-side modified-sketch path (Mode 1) #499

Merged

Navigation Menu

refactor: centralized series_id namespace + remove metric-name suffixes + VictoriaMetrics + relocate controller - #372

refactor: centralized series_id namespace + remove metric-name suffixes + VictoriaMetrics + relocate controller#372
zzylol merged 2 commits into
mainfrom
refactor/centralized-sid-and-architecture-cleanup

zzylol commented May 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zzylol commented May 10, 2026

Summary

What landed (per phase)

Deferred / out-of-scope (separate PRs)

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant