feat(deploy): paper baseline matrix (B0 raw / B1 Serf / B2 full / B3 delta / B4 tunable window) - #176
Merged
Merged
Conversation
B3 delta / B4 tunable window)
Adds the five baselines paper §6.2 compares head-to-head. Each
baseline is one agent-pipeline YAML + one compose overlay; the
same asap/sketchcol:dev binary serves all of them — only the
mounted config + env differ.
* B0 raw OTel (`sketchcol-agent-b0-raw.yaml`): OTLP →
batch → OTLP. Control — no sketching, no compression.
* B1 Serf (`…-b1-serf.yaml`): Gorilla-style XOR compression
written to local tmpfs. Rebuild sketchcol to include
`serfprocessor` (added to builder-config.yaml).
* B2 full sketch (`…-b2-full.yaml`): DDSketch + HLL,
`transmit_sketch: true, mode: batch`. Today's default.
* B3 delta sketch (`…-b3-delta.yaml`): same sketches,
`mode: window, window_duration: 60s,
delta_transmission: true`. Wire payload = sparse diff
of only-changed buckets since last flush.
* B4 tunable window (`…-b4-tunable.yaml`): same as B3 but
`window_duration: ${env:SKETCH_WINDOW}` — sweep 5s / 30s /
60s / 300s without rebuilding.
Switching baselines:
AGENT_CONFIG=sketchcol-agent-b3-delta.yaml \
docker compose -f base.yml -f agents-N1.yml \
-f baseline-b3-delta.yml up -d
SKETCH_WINDOW=10s \
AGENT_CONFIG=sketchcol-agent-b4-tunable.yaml \
docker compose -f base.yml -f agents-N1.yml \
-f baseline-b4-tunable.yml up -d
The `AGENT_CONFIG` env var threads through agents-N*.yml's
mounted config path. `SKETCH_WINDOW` is plumbed to every agent
by gen-agents.sh (default 60s). Baseline overlays are thin
naming markers that stamp `controller.labels.asap.baseline`;
all five coexist so a single bring-up run can identify its
baseline from `docker inspect`.
sketchcol binary rebuild required (binary bundles serfprocessor
now). Host build chain unchanged — see Dockerfile.sketchcol
header for the submodule + sketchlib-go sibling prereqs.
Live-validated B3 at N=1 after one 60s window flush:
* DDSketch input = 1.31 GiB, output = 3.21 MiB → **408×
bandwidth reduction** on the wire.
* Compared to B2 (same workload, full-sketch batch mode):
output/input ratio was 177% — B3 is ~720× smaller per
window-flush than B2.
Follow-up before the full eval:
* Paper §6.4 accuracy comparison between B2 and B3 needs the
backend's sketch-reconstitution path to handle delta wire-
format — existing `sketchlib-go` delta codec work
(Phase 1 packed arrays) covers this.
* `fake-exporter` could emit a more faithful Google-trace
replay instead of synthetic log-normal — DC TODO #4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 22, 2026
Adds two small stdlib-only scripts for the paper §6 eval:
deploy/scripts/measure-baseline.py
One-shot Prometheus query → single-row CSV with every
§6.2 / §6.3 signal: per-agent CPU + RSS + input/output
bytes, gateway CPU + RSS + points + out-series, backend
ingest rate + query P99, plus backend CPU/RSS via
`docker stats` (the backend doesn't self-report them).
deploy/scripts/run-baseline-sweep.sh
Iterates (baseline × rate × cardinality) triples,
bringing the stack up/down for each combo, soaking,
calling measure-baseline.py, and concatenating into a
single CSV. Overrides via env: BASELINES / SCALE /
RATES / CARDS / SOAK_S.
Query-shape notes worth the comment blocks in the script:
* Agent side (v0.141): counters have `_total` suffix
(`otelcol_process_cpu_seconds_total`,
`otelcol_receiver_accepted_metric_points_total`). Prefer
these — the patched-processor `otelcol_datacollector_processor_*`
family gives finer per-processor breakdown but only
exists on sketch baselines (B2/B3/B4), so the universal
`otelcol_process_*` covers B0/B1/B5 too.
* Gateway (v0.108): no `_total` suffix on process/receiver
metrics. Duplicated queries aren't accidental.
* Per-baseline NaN columns are semantically correct and
not a script bug: e.g. B5 Gorilla `agent_in_kib_per_s`
is NaN because gorillaprocessor isn't ASAP-patched and
thus doesn't emit `otelcol_datacollector_processor_*`.
Live-validated at N=1 with a 75 s B2 soak, 2 min rate window:
* Agent CPU 0.44 cores, RSS 474 MiB
* Agent in/out 1.46 MiB/s / 73.2 MiB/s (50× sketch-state
expansion vs raw, matching PR #176 B2 numbers)
* Gateway 0.09 cores, 226 MiB, 8617 points/s in
* Backend ingest 8792 samples/s
* All 16 CSV columns populate for the B2 row.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the five paper §6.2 baselines as agent-pipeline configs + compose overlays. Same `asap/sketchcol:dev` binary serves all of them; only the mounted config + env differ.
Switching:
```
AGENT_CONFIG=sketchcol-agent-b3-delta.yaml \
docker compose -f base.yml -f agents-N1.yml -f baseline-b3-delta.yml up -d
SKETCH_WINDOW=10s AGENT_CONFIG=sketchcol-agent-b4-tunable.yaml \
docker compose -f base.yml -f agents-N1.yml -f baseline-b4-tunable.yml up -d
```
Test plan
🤖 Generated with Claude Code