feat(deploy): per-baseline measurement script + sweep driver - #178
Merged
Merged
Conversation
Adds two small stdlib-only scripts for the paper §6 eval:
deploy/scripts/measure-baseline.py
One-shot Prometheus query → single-row CSV with every
§6.2 / §6.3 signal: per-agent CPU + RSS + input/output
bytes, gateway CPU + RSS + points + out-series, backend
ingest rate + query P99, plus backend CPU/RSS via
`docker stats` (the backend doesn't self-report them).
deploy/scripts/run-baseline-sweep.sh
Iterates (baseline × rate × cardinality) triples,
bringing the stack up/down for each combo, soaking,
calling measure-baseline.py, and concatenating into a
single CSV. Overrides via env: BASELINES / SCALE /
RATES / CARDS / SOAK_S.
Query-shape notes worth the comment blocks in the script:
* Agent side (v0.141): counters have `_total` suffix
(`otelcol_process_cpu_seconds_total`,
`otelcol_receiver_accepted_metric_points_total`). Prefer
these — the patched-processor `otelcol_datacollector_processor_*`
family gives finer per-processor breakdown but only
exists on sketch baselines (B2/B3/B4), so the universal
`otelcol_process_*` covers B0/B1/B5 too.
* Gateway (v0.108): no `_total` suffix on process/receiver
metrics. Duplicated queries aren't accidental.
* Per-baseline NaN columns are semantically correct and
not a script bug: e.g. B5 Gorilla `agent_in_kib_per_s`
is NaN because gorillaprocessor isn't ASAP-patched and
thus doesn't emit `otelcol_datacollector_processor_*`.
Live-validated at N=1 with a 75 s B2 soak, 2 min rate window:
* Agent CPU 0.44 cores, RSS 474 MiB
* Agent in/out 1.46 MiB/s / 73.2 MiB/s (50× sketch-state
expansion vs raw, matching PR #176 B2 numbers)
* Gateway 0.09 cores, 226 MiB, 8617 points/s in
* Backend ingest 8792 samples/s
* All 16 CSV columns populate for the B2 row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 22, 2026
First end-to-end run of all 5 baselines via the PR #178 sweep script. Values are from the 75s soak at (rate=1000/s, cardinality=1000) on a log-normal gauge workload: Baseline CPU RSS Out MiB/s × vs B0 B0 raw OTel 0.77c 214MB 16.34 1.00× B1 Serf XOR 0.59c 271MB 3.44 1/ 4.8× B5 Gorilla XOR 0.58c 297MB 3.84 1/ 4.3× B2 full sketch 1.00c 506MB 163.77 10.03× ⚠ 10× larger than raw B3 delta (60s) 0.45c 225MB 0.053 1/305.8× ✓ paper's bandwidth claim Two takeaways worth flagging: * B2 full-sketch is an order of magnitude WORSE than raw at 1-second batch cadence — per-batch DDSketch+HLL state at cardinality=1000 dwarfs the raw gauge points it summarizes. This motivates why naive sketch-per-batch isn't a viable deployment, and why the paper needs delta + windowing. * B3 delta (60s) hits 305× reduction vs raw, exactly the bandwidth story §6.2 leads with. All four sketch families (DDSketch, HLL, CountSketch, CountMin) are delta-ready end-to-end (ASAPQuery-backend #60-#63 chain) so §6.4 accuracy comparisons can run against reconstituted sketches. * B1 Serf + B5 Gorilla compression is modest (~5% and 2.4%) because log-normal synthetic data has poor temporal correlation; real trace data typically compresses much better. Google-cluster-trace replay (DC TODO #4) is the follow-up that would fix this. Raw CSV archived for reproducibility; regenerate via: BASELINES="b0-raw b1-serf b2-full b3-delta b5-gorilla" \ SCALE=N1 RATES=1000 CARDS=1000 SOAK_S=75 \ ./deploy/scripts/run-baseline-sweep.sh > out.csv B3's measured out-bytes takes a longer soak (≥3 min) because of its 60s window — the sweep default of 75s yields one flush, and the script's 1m rate() window returns NaN on a single sample. The archived number (0.053 MiB/s) is from a separate 180s soak. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two stdlib-only scripts for the paper §6 eval:
measure-baseline.py— one-shot Prometheus query → CSV with per-agent CPU/RSS/bytes, gateway CPU/RSS/points, backend ingest/P99, plus backend CPU/RSS via docker stats.run-baseline-sweep.sh— iterates (baseline × rate × cardinality) triples, bringing stack up/down per combo, calling measure script, concatenating into one CSV.Query-shape notes worth the comments:
_totalsuffix on process/receiver counters; agents (v0.141) keep it — duplicated queries aren't accidental.agent_in_kib_per_sis NaN because gorillaprocessor doesn't emit ASAP self-monitoring).Live-validated against B2 at N=1, 75s soak: Agent 0.44 cores / 474 MiB, Agent in/out 1.46 / 73.2 MiB/s, Gateway 0.09 cores, Backend ingest 8792 samples/s. All 16 columns populate.
🤖 Generated with Claude Code