mvp/v4: real-Prom + SERF + stage-split + freshness + accuracy fix - #290
Merged
Merged
Conversation
Adds the realistic stock-OTel baseline the paper compares ASAP
against: agent runs an OTLP receiver + 1s batch + the
`prometheusremotewrite` exporter pointing at a real Prometheus
container's :9090/api/v1/write. No precompute_engine in the loop;
storage + queries are vanilla Prometheus TSDB + PromQL.
* `deploy/configs/sketchcol-agent-b0-prometheus.yaml` — agent
config (PRW exporter, no sketches).
* `deploy/configs/prometheus-with-remote-write.yml` — Prometheus
config with the existing scrape jobs; remote-write-receiver
is enabled via the global flag in the compose overlay.
* `deploy/docker-compose/baseline-b0-prometheus.yml` — overlay
that adds `--web.enable-remote-write-receiver` to Prometheus
and mounts the v4 config + a named TSDB volume.
Note: the v3 `gateway.yaml` comment claimed PRW exporter wasn't in
the patched sketchcol builder; verified against
`opentelemetry-collector-contrib-patch/cmd/sketchcollector/builder-config.yaml`
that PRW IS in fact built in (line ~prometheusremotewriteexporter
v0.141.0). No image rebuild required.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v3 SERF baseline used `drop_original: true` so nothing reached
the gateway / backend / Prometheus — bandwidth was measured purely
against bytes-on-disk. v4 needs every baseline to share the same
Prometheus query surface so accuracy / freshness / latency are
comparable apples-to-apples.
* `deploy/configs/sketchcol-agent-b1-serf-prometheus.yaml` — new
v4 SERF agent config: `drop_original: false` + PRW exporter to
Prometheus :9090/api/v1/write. SERF still writes blobs to
/var/tmp/asap-serf-out (its CPU / RSS / output-bytes self-
telemetry stays meaningful); the raw stream is additionally
forwarded to Prometheus so the query side is identical to B0.
* `deploy/docker-compose/baseline-b1-serf.yml` — audited; v3
overlay was a label-only no-op. v4 layers Prometheus's
remote-write-receiver flag on the Prometheus container so the
PRW exporter has a sink. Legacy
`sketchcol-agent-b1-serf.yaml` callers still work (the flag is
harmless when no client connects).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the v3 all-five-sketches overlay with a single-sketch
agent config + Gorilla-S3 cold archive — the realistic production
shape the controller's plan picks per (metric, query family,
workload). All-five-at-once was paper-figure / §3 architecture
material, NOT §5 evaluation data.
* `deploy/configs/sketchcol-agent-b6-asap-single-sketch.yaml` —
agent: gorillas3 (cold) + ddsketch (warm) + batch + OTLP →
backend. Other four sketch processors are conspicuously absent.
* `deploy/docker-compose/baseline-b6-asap-single-sketch.yml` —
overlay: backend env for GorillaQueryEngine + per-metric
routing, MinIO bucket seed for asap-gorilla, agent-1 OTLP
HTTP port published on host (14328) so the freshness probe
can push synthetic gauges through the agent's gorillas3
processor (going via the gateway would bypass it).
* `deploy/configs/backend-storage-routing.yaml` — v4 fixes the
accuracy NaN failure mode: revert the latency-quantile metric
back to the warm tier (DDSketch / KLL accumulators on
SimpleEngine) so the accuracy reducer gets real numbers.
Criterion ⑤ still verifies cold-archive dispatch via
`http_requests_total`. Adds new freshness-probe routes:
`http_freshness_probe_warm` → sketch_warm_tier;
`http_freshness_probe_archive` → gorilla_s3_archive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pushes two synthetic OTLP gauges every 1s where the gauge value encodes the unix_ts_ms of emission, then polls a PromQL backend every 100 ms with `last_over_time(<metric>[10s])`. On every non-NaN observation, records Δ = obs_wall_clock - emit_ts to `freshness.csv` (columns: baseline, path, sample_ts_ms, observation_ts_ms, delta_ms). Two probe metrics so warm-tier and cold-archive paths can be measured independently: * `http_freshness_probe_warm` → sketch_warm_tier (warm path). * `http_freshness_probe_archive` → gorilla_s3_archive (cold). For B0 / B1 / B5 only the warm column is meaningful (Prometheus has no cold archive); pass `--paths warm`. For the ASAP overlay both paths are exercised (`--paths warm,archive`). Stdlib only. Independent of `run_mvp_demo.sh` — exits cleanly after the configurable measurement window (default 60s). Prints per-(baseline, path) p50 / p99 / count to stdout when done. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Samples per-container CPU cores, RSS MiB, net rx/tx KiB/s, disk
MiB at 1Hz over a measurement window (default 60s) and aggregates
per stage according to a hard-coded mapping:
agent-* → agent
fake-exporter → producer
gateway → gateway
backend → backend-ingest + backend-query (dual-listed; the
report deduplicates RSS before computing
criterion ③)
prometheus → backend-storage (B0 / B1 baselines)
minio → backend-storage (ASAP baselines)
controller → controller
Disk usage probe is stage-aware:
prometheus → /prometheus (TSDB chunk dir)
minio → /data/asap-gorilla
agent-* → /var/tmp/asap-{serf,gorilla}-out (SERF / B5 only)
CSV columns: baseline, stage, container, cpu_cores, rss_mib,
net_in_kibps, net_out_kibps, disk_mib. Each container produces one
row per stage; values are time-averages over the window (CPU mean
cores; RSS mean MiB; net rx/tx window-rate; disk end-of-window).
Stdlib only. Independent of `run_mvp_demo.sh` — exits cleanly
after `--duration` and prints a per-stage summary to stdout.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Driver rewrite for v4. Key changes vs v3:
* Cycles through FOUR baselines back-to-back instead of two:
B0 raw → real Prometheus
B1 SERF → real Prometheus
B5 Gorilla agent-side
B6s ASAP single-sketch + Gorilla-S3 cold archive
* Adds a query-side warm-up step after the agent warm-up:
polls `count_over_time(http_requests_total[1m])` against the
relevant query backend until it goes non-zero (cap 30s).
This is the criterion-④ accuracy NaN fix — without it the
measurement window can open before any sketch has flushed,
so quantile queries return NaN and the accuracy reducer can't
compute relative error.
* Runs `promql_replay.py` + `measure_freshness.py` +
`measure_stages.py` IN PARALLEL during the 60s measurement
window so all three CSVs are time-aligned.
* Per-baseline cell directory layout:
<OUT_BASE>/<baseline>/{measurement.csv, accuracy.csv,
freshness.csv, stages.csv,
replay.jsonl,
ad_hoc_query_response.json}
* Ad-hoc cold-fallback query is gated to the ASAP cell only
(B0 / B1 / B5 have no Gorilla cold archive).
* Output dir defaults to deploy/eval-results/mvp-v4-2026-05-06/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rewrites the report generator for the v4 layout. New sections:
§1 Stage-separated resource breakdown — four baselines × five
stages × {CPU cores, RSS MiB, net in/out KiB/s, disk MiB},
plus per-baseline totals (deduplicated for the
backend-{ingest,query} dual-listing).
§2 Per-criterion verdict (6 criteria) — five originals plus
freshness (sample → first query). Verdicts are PASS / FAIL /
PARTIAL / UNKNOWN, with the underlying numbers inline.
§3 Freshness p50 / p99 — per (baseline, path) sample counts +
p50/p99 deltas in ms. Skips B5 if not measured.
§4 Honest caveats — single-host bench, 60s window vs gorilla
chunk rotation, single-sketch ASAP not all-five, real
Prometheus, freshness-probe encoding details.
Discovers baselines automatically from `--results-dir` (set of
subdirs that carry `measurement.csv`); ordered B0, B1, B5, ASAP
when present.
Resource criterion (③) is computed from `stages.csv` rather than
`measurement.csv` so the table and verdict use the same
provenance. Accuracy (④) is computed from the ASAP cell's
`accuracy.csv` (warm-tier reducer output).
v3 entry points (`--asap-dir / --raw-dir`) preserved as a legacy
fallback so existing v3 callers don't break.
Idempotent: re-running over the same CSVs reproduces the same MD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `sketchcol-agent-b5-gorilla-prometheus.yaml` mirroring the B1 PRW config — runs the gorilla processor with `drop_original: false` and forwards the raw stream via prometheusremotewrite. The b5 docker-compose overlay now also starts Prometheus with the remote-write-receiver flag and a dedicated `prometheus-data-b5` volume, matching the b1-serf overlay. The MVP demo driver is switched from the disk-only `sketchcol-agent-b5-gorilla.yaml` to the PRW variant so criterion ⑥ freshness and criterion ④ accuracy have a Prometheus data source for B5 — without this, B5's column was permanently NaN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
marked this pull request as ready for review
May 6, 2026 19:53
10 tasks
5 tasks
zzylol
added a commit
that referenced
this pull request
May 18, 2026
…ave queries + stale bucket key (#394) Four orthogonal fixes that together make the multinode harness validate the post-#290/#291/#292 sum-by-zone wave end-to-end on the ASAP arm. 1. run_demo.sh::backend_up() now brings up an asap-controller container on node2 alongside asap-backend, mirroring the singlenode docker-compose base.yml. Post-Phase-9 the controller and backend are two binaries in the same asap/query-backend:dev image but distinct processes; without the standalone controller the backend stays on the static DDSketch-only fallback (no Sum/Topk roles) and the wave queries silently return empty. Started after the backend so the startup pre-pop replan_all tick has a live POST target (CONTROLLER_BACKEND_ENDPOINT=http://backend:9091/api/v1/streaming-config, full URL form matching backend_client::BackendClient::new contract). Stale CONTROLLER_* env vars on the backend container removed; they were pre-Phase-9 carry-over the backend never reads. topology.env's "controller in-process" comment updated to reflect separate-container reality. 2. scripts/run_demo.sh:387 had `--endpoint`; the actual metricsql_replay.py arg is `--target`. Replay was silently failing argparse and the exception path returned 0 successes. Fixed. 3. queries-e2e.json extended with the wave's 3 marquee queries: `sum by (zone) (http_requests_total)`, `sum by (zone) (rate(http_requests_total[5m]))`, `topk(5, sum by (zone) (rate(http_requests_total[5m])))`. 4. agent yamls (mvp-multinode + mvp-singlenode) still set `bucket: asap-gorilla`, but PR #388 deleted the Bucket field from gorillas3processor/config.go (only `tsdb_bucket` remains). Result: agent crash-loop with `'gorillas3' ... '' has invalid keys: bucket`. Line removed from both yamls; comment references #388. ## Validation (sync + arm asap, ASAP arm) - replay.jsonl: 384 success lines (was 0 pre-fix); all 3 wave queries show success: `sum by (zone) (http_requests_total)` (128), `sum by (zone) (rate(http_requests_total[5m]))` (128), `topk(5, sum by (zone) (rate(http_requests_total[5m])))` (128). - Controller logs confirm typed-stage-split JSON POST succeeded for each (metric, role) pair: top_endpoint_qps/topk, request_size_bytes/quantile, http_requests_total/sum, etc. — all hitting http://backend:9091/api/v1/streaming-config with 2xx. - Agent logs: clean startup, no crash-loop, all 7 gorillas3 pipeline instances (raw_passthrough, ddsketch_path, hll_path, kll_path, countminsketch_path, countsketch_path) starting with `tsdb_bucket: asap-gorilla-tsdb` (no `bucket:` field). TSDB blocks written. - Manual probe against http://10.10.1.3:9091/api/v1/query, all returning `data_source: asap_query`: * `sum by (zone) (http_requests_total)` → z0:981128625, z1:981127610, z2:981127004, z3:981125787 * `sum by (zone) (rate(http_requests_total[5m]))` → z0:4798703.10, z1:4798695.49, z2:4798690.49, z3:4798680.18 * `topk(5, sum by (zone) (rate(http_requests_total[5m])))` → same 4 zones (only 4 exist; topk(5) returns all) * `quantile_over_time(0.99, http_requests_total_latency_ms[5m])` → empty (the multinode producer only emits http_requests_total, no _latency_ms; pre-existing harness gap, orthogonal to this PR). - Teardown clean. ## Out-of-scope follow-up Controller startup replan_all races the backend's /api/v1/streaming-config handler bind: the first POST burst races the route registration and some pushes get 404'd; subsequent OpAMP-on-connect re-fire makes the plan converge anyway. Lives in ASAPQuery-backend, not addressed here. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
7 tasks
zzylol
added a commit
that referenced
this pull request
May 19, 2026
…10 Mbps double-hop) (#401) The asap-gateway OTel-collector hop was originally the cross-agent fan-in tier for sum-by-zone aggregations. Post ASAPQuery-backend PRs #283 (B2 stage-emit) and #287/#290 (cumulative emit), the wave-tier backend does that fan-in natively via the `evaluate_exact_agg` reducer pulling sids from every agent. The gateway became a pure forwarder that doubled every byte (agent → gateway → backend) for no algorithmic reason, costing ~10 Mbps per arm on the multinode demo (issue #400). Scope: - multinode `run_demo.sh`: remove `gateway_up` / `gateway_down`, drop the gateway out of `arm_up` / `arm_down`. `stop_node node1` stays in the teardown to reap any pre-#400 leftover container; `node1` is otherwise unused. - multinode `topology.env`: drop the `--add-host=gateway:` alias and retire the node1 role note. - multinode + singlenode agent yaml: OTLP `endpoint:` flips from `gateway:4317` to `backend:4317`. The backend already accepts OTLP on 4317 (`--otel-grpc-port=4317` in `run_demo.sh` and base.yml). - multinode + singlenode `mvp-workload.yaml`: the lone `assign_to_role: gateway` entry (sum-by-zone) becomes `assign_to_role: agent`. The controller's typed `split_typed_three_stage` emits only Edge + Backend stages when no Gateway stage is occupied, so `emit_gateway_yaml` simply doesn't fire and `push_to_role(Gateway, ...)` is a no-op — no controller code change needed. - multinode `asap-otel-gateway-mvp-placeholder.yaml`: deleted. - singlenode `base.yml`: gateway service removed; its host ports (14317 / 14318) reassigned to the backend container so external tooling that previously hit the gateway can hit the backend on the same host ports. `fake-exporter` default `EXPORTER_TARGET` flips to `backend:4317`; `depends_on` flips to `backend`. - singlenode `agents-N{1,10,100}.yml` + `gen-agents.sh`: `depends_on` flips from `gateway` → `backend`. Validation (multinode, 60 s NIC sample, asap arm, WARMUP_S=90): - node1 docker ps: empty (no asap-gateway). - node2 docker ps: backend + thanos-{query,store-gateway,compact} + minio + prometheus + controller; no gateway. - All 5 wave queries against node2:9091 return `data_source: asap_query`: quantile_over_time(0.99, http_requests_total_latency_ms[5m]) → 10000 series max by (zone) (quantile_over_time(0.99, ...)) → 4 series sum by (zone) (http_requests_total) → 4 series sum by (zone) (rate(http_requests_total[5m])) → 4 series topk(5, sum by (zone) (rate(...))) → 4 series - Per-node NIC bandwidth (enp130s0f0, 60 s): node0 RX 0.16 Mbps TX 6.80 Mbps (agent-a → backend) node1 RX 0.00 Mbps TX 0.00 Mbps (idle — pre-fix carried the gateway double-hop) node2 RX 13.56 Mbps TX 0.33 Mbps (backend receives both agents) node3 RX 0.16 Mbps TX 6.76 Mbps (agent-b → backend) Pre-fix the node1 link carried ~13 Mbps in + ~13 Mbps out as a pure forwarder. Post-fix that ~26 Mbps of node1 wire traffic is gone; the remaining ~13.6 Mbps lives only on the agent→backend hop. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: DRAFT — author-only
This PR is the author half of issue #46 v4. A follow-up agent will run the demo, fill in
MVP_REPORT_v4.md, and post the comparison comment on issue #46. Do not merge until that follow-up lands.What's authored (code only — demo not yet run)
The user's v4 brief asked for six things in one PR. All six are wired into this branch:
1. Accuracy NaN fix (criterion ④)
run_mvp_demo.shv4: after the 60s agent warm-up, the driver pollscount_over_time(http_requests_total[1m])against the active query backend until non-zero (cap 30s) before opening the measurement window.backend-storage-routing.yamlreverts the latency-quantile metric back tosketch_warm_tierso DDSketch / KLL accumulators answer it (the v3 cold-tier route returned NaN inside a 60s gorilla window).2. Freshness measurement (criterion ⑥)
deploy/scripts/measure_freshness.py: pusheshttp_freshness_probe_{warm,archive}synthetic gauges every 1s where the value encodes the unix_ts_ms of emission, polls PromQL every 100 ms, records Δ tofreshness.csv. Stdlib only,--helpworks, prints p50/p99/count to stdout.http_freshness_probe_warm→sketch_warm_tier,http_freshness_probe_archive→gorilla_s3_archive.3. Real Prometheus baseline (B0)
deploy/configs/sketchcol-agent-b0-prometheus.yaml: stock OTel +prometheusremotewriteexporter.deploy/configs/prometheus-with-remote-write.yml: scrape config (mirrors existingprometheus.yml) for the v4 overlay.deploy/docker-compose/baseline-b0-prometheus.yml: adds--web.enable-remote-write-receiverto Prometheus + named TSDB volume. No precompute_engine in the loop; queries hit Prometheus :9090 directly.4. SERF baseline (B1) wired to the same real Prometheus
deploy/configs/sketchcol-agent-b1-serf-prometheus.yaml: SERF processor (drop_original=false) + PRW exporter to Prometheus. SERF's CPU / RSS / output-bytes self-telemetry is preserved; the raw stream is additionally forwarded so query-side parity with B0 holds.deploy/docker-compose/baseline-b1-serf.ymlaudited: v3 was a label-only no-op. v4 layers Prometheus's remote-write-receiver flag on the Prometheus container so the SERF agent's PRW exporter has a sink. Legacysketchcol-agent-b1-serf.yamlcallers still work.5. Single-sketch ASAP overlay
deploy/configs/sketchcol-agent-b6-asap-single-sketch.yaml: ONE sketch (DDSketch by default; configurable viaASAP_SKETCH_FAMILYenv) + gorillas3processor. Other four sketch processors removed.deploy/docker-compose/baseline-b6-asap-single-sketch.yml: backend env for GorillaQueryEngine + per-metric routing, MinIOasap-gorillabucket seed, agent-1 OTLP HTTP exposed on host.6. Stage-separated resource breakdown
deploy/scripts/measure_stages.py: 1Hzdocker statssampling over the measurement window; aggregates per-stage (agent / gateway / backend-ingest / backend-query / backend-storage) per{cpu_cores, rss_mib, net_in_kibps, net_out_kibps, disk_mib}. Disk probe is stage-aware (Prometheus TSDB / MinIO bucket / SERF blob dir).mvp_report.pyv4 emits §1 stage-split table (4 baselines × 5 stages × 5 metrics + dedup totals), §2 6-criterion verdicts, §3 freshness p50/p99 table, §4 honest caveats. Idempotent.Files
New
deploy/configs/prometheus-with-remote-write.ymldeploy/configs/sketchcol-agent-b0-prometheus.yamldeploy/configs/sketchcol-agent-b1-serf-prometheus.yamldeploy/configs/sketchcol-agent-b6-asap-single-sketch.yamldeploy/docker-compose/baseline-b0-prometheus.ymldeploy/docker-compose/baseline-b6-asap-single-sketch.ymldeploy/scripts/measure_freshness.pydeploy/scripts/measure_stages.pyModified
deploy/configs/backend-storage-routing.yaml— quantile route reverted to warm; freshness probe routes added.deploy/docker-compose/baseline-b1-serf.yml— Prometheus remote-write enable flags + v4 config mount.deploy/scripts/run_mvp_demo.sh— 4-baseline cycle, query-side warm-up, parallel freshness/stage measurement.deploy/scripts/mvp_report.py— stage-split / 6-criterion / freshness / caveats sections.The four postfix-owned files (
run_e2e_sweep.sh,queries-e2e.json,plan_transition.py,measure-baseline.py) are untouched.Test plan (for the run-only follow-up agent)
bash deploy/scripts/run_mvp_demo.shcycles all four baselines without error<baseline>/directory contains all six output files (measurement, accuracy, freshness, stages, replay, ad_hoc — last two ASAP-only)ad_hoc_query_response.jsoncarriesdata_source: gorilla_archive(or PARTIAL signal via gorilla_chunks.txt)MVP_REPORT_v4.mdrenders with §1 stage table + §2 six verdicts + §3 freshness table + §4 caveatsDemo run pending
This branch is DRAFT — the demo has NOT been run. The follow-up agent will:
run_mvp_demo.shend-to-end (estimated wall time: ~15 min, four 60+30+60s cells with 8s settle each)MVP_REPORT_v4.mdwith real numbers🤖 Generated with Claude Code