Skip to content

eval: 8-node CloudLab real-hardware §6 figures + harness fixes - #503

Merged
zzylol merged 8 commits into
mainfrom
eval/8node-cluster-figures
Jun 17, 2026
Merged

eval: 8-node CloudLab real-hardware §6 figures + harness fixes#503
zzylol merged 8 commits into
mainfrom
eval/8node-cluster-figures

Conversation

@zzylol

@zzylol zzylol commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Wires up the mvp-multinode harness to an 8-node CloudLab cluster and produces real-hardware §6 figures, with the bug fixes needed to get the stack running at scale.

Figures (see deploy/mvp-multinode/eval-8node/RESULTS.md)

  • Fig 1 Pareto — ASAP ~40× cheaper ingest at ~0.97 accuracy.
  • Fig 2 bandwidth — ASAP 39× below raw-none (120× at the warm sketch sink).
  • Fig 3 accuracy — DDSketch p99 median rel-err 0.026 vs KLL 0.084.
  • Fig 6 edge CPU/mem — honest ~3× CPU cost for the wire win; 6b soak: RSS slope −290 MiB/hr → no leak.
  • Fig 7 latency — cold-OFF warm quantile p99 51 ms (vs 167 ms cold-ON archive failover).
  • Fig 8 cross-layer placement — SDK placement drops agent CPU 111%→4.5% (~25×).
  • Fig 9 coordinated sampling — pipeline wired; root-caused that the coordinator passes rates as both rate+freq → uniform p (fix in progress).
  • Fig 10 scaling — per-agent bandwidth flat at N=1–3.
  • Fig 11 storage — gorilla 2.3×–12× vs raw (measured).

Bug fixes

  • Dockerfile.otel-app: COPY the asap-precompute-go sibling its go.mod requires (build was failing).
  • snapshot_resources.sh: auto-detect the 10.10.1.x NIC (was hard-coded enp130s0f0).
  • run_demo.sh/run_demo_sweep.sh: TOPOLOGY_ENV override; run_demo.sh sourceable as a library; DP_MONITOR_FLAGS to enable the CDM coordinator.

New

topology.8node.env, scale_fleet.sh, fig9_coordinated.sh, fig7_latency.sh, fig8_placement.sh, soak_rss.sh, plots.py, fig_costmodel.py, plot_fig8_fig6b.py, and eval-8node/ (figures + raw results + RESULTS.md).

🤖 Generated with Claude Code

zzylol and others added 7 commits June 15, 2026 14:28
@zzylol @claude
eval: 8-node CloudLab harness + real-hardware §6 figures
810e09d 
Wire up the mvp-multinode harness to an 8-node CloudLab cluster (node0 driver,
node1 cold, node2 warm, node3-7 sources) and produce real-hardware §6 figures.

New:
- topology.8node.env: 8-node role split (TOPOLOGY_ENV override)
- scripts/scale_fleet.sh: Fig 10 fleet-size driver (reuses run_demo.sh as a lib)
- scripts/plots.py: Fig 2/6/7/10 figure generators from harness CSV/JSONL
- eval-8node/: rendered figures + raw results + RESULTS.md summary

Results (real hardware): ASAP cuts ingest wire ~39x vs raw-none / ~120x at the
warm sketch sink, at ~3x edge CPU; per-agent bandwidth flat at N=1-3; DDSketch
p99 quantile median rel-err 0.026 vs KLL 0.084; edge gorilla encode ~2.1 us/sample.

Bug fixes surfaced during the eval:
- snapshot_resources.sh: hard-coded NIC enp130s0f0 -> auto-detect 10.10.1.x iface
  (this cluster is eno2); node set configurable via SNAP_NODES.
- Dockerfile.otel-app: COPY the asap-precompute-go sibling its go.mod requires
  (+ build-context); otel-app image build was failing without it.
- run_demo.sh / run_demo_sweep.sh: honor TOPOLOGY_ENV override; run_demo.sh
  sourceable as a library (RUN_DEMO_LIB=1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol @claude
eval: add Fig 1 (Pareto), Fig 11 (storage), Fig 9 scoped gap
2d46fef 
- fig_costmodel.py: Fig 1 cost-vs-accuracy Pareto (measured Fig2/Fig3 anchors +
  cost-model sampling extension) and Fig 11 cold-storage (measured gorilla-XOR
  bytes/sample, 2.3x-12x vs uncompressed raw, data-dependent).
- fig11_storage_bench.txt: real asap-gorilla-go storage benchmark output.
- RESULTS.md: Fig 1 / Fig 11 sections + Fig 9 (coordinated-vs-uniform) documented
  as a scoped gap (coordinator on :4319 is present, but the MVP workload emits an
  empty monitors[] — needs a threshold-monitored metric + coordinated skewed fleet).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol @claude
eval: Fig 9 coordinated-sampling pipeline wired + Fig 10 v3 clean re-run
2f1d12c 
Fig 9 (coordinated sampling on a skewed fleet) — advanced from "empty monitors[]"
gap to an end-to-end-wired pipeline verified on hardware:
- configs/asap/mvp-workload-fig9.yaml: adds a monitor: block to http_requests_total
  so the controller emits a backend monitors[] entry (agg_id + tau + epsilon).
- run_demo.sh: DP_MONITOR_FLAGS plumbs --enable-monitor-coordinator into the
  data-plane (coordinator was off by default).
- scripts/fig9_coordinated.sh: brings up the coordinator + a skewed 3-edge
  trace-replay fleet (synthetic timestamp_ms,series_id,value CSVs, -trace-loop).
Result (fig9_partial.csv): all 3 edges connect to the coordinator and report a
25x skewed rate (80k/16k/3.2k per window), but the coordinator did not issue
differentiated p<1 grants (tried tau=5e6 and 1e5) — grant-trigger in
data_plane/src/monitor/ needs further investigation. Also documented: coordination
only activates in the producer trace-replay path, not the synthetic path.

Fig 10 v3: clean re-run (90s soak, 6-sample averaged CPU) — per-agent bandwidth
flat at N=1-3 (0.047-0.060 MB/s), stable CPU 73-138%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol @claude
eval: Fig 7 cold-OFF latency + Fig 9 grant root-cause + soak driver
e86908b 
- Fig 7 cold-OFF arm (fig7_latency.sh): warm-only quantile p99 = 51ms vs the
  cold-ON 167ms — confirms the cold-ON latency was Thanos archive failover, not
  the warm path. sum resolves warm in ~2.4ms.
- Fig 9 root cause: coordinator.rs::allocate_p passes rates as BOTH rate and freq
  vectors, so √(f_i/rate_i)=1 for every edge → uniform p, the differentiated
  p_i ∝ √(f_i/rate_i) is architecturally not wired (per-key f_i not plumbed to the
  coordinator). Documented the fix path.
- soak_rss.sh: Fig 6b memory-leak soak driver (RSS slope over a soak window).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol @claude
eval: Fig 8 cross-layer placement + Fig 6b soak (no leak)
b32b23a 
- Fig 8 (fig8_placement.sh): same DDSketch agg at SDK vs agent placement. Moving
  the sketch to the SDK drops agent CPU 111%->4.5% (~25x), backend 90%->6.6%, and
  producer 240%->99% (one compact sketch/window vs full raw-buffer) — earlier
  placement is cheaper at every downstream layer; tradeoff is sketch lib in every app.
- Fig 6b (soak_rss.sh): 30-min agent RSS soak, slope -290/-183 MiB/hr (GC, not
  growth) => no leak. In-session proxy for the 24h target.
- plot_fig8_fig6b.py renders both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol @claude
eval: role-based topology names (DRIVER/COLD/WARM/SRC) instead of NOD…
8fcaf43 
…Ek macros

topology.8node.env now names machines by ROLE — DRIVER_HOST/IP, COLD_HOST/IP
(MinIO/Thanos/merger/Prometheus/VM), WARM_HOST/IP (data_plane+control_plane),
SRC_HOSTS/SRC_IPS (sources) — as the source of truth. The team's run_demo.sh
4-slot NODE0..NODE3 vars are kept as DERIVED compat shims (NODE1=COLD, NODE2=WARM,
NODE0/NODE3 = first two SRC). ADD_HOSTS + scale_fleet.sh + soak_rss.sh use the
role names (e.g. WARM_HOST/WARM_IP, COLD_HOST) instead of NODE2_IP/NODE1_HOST.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol @claude
eval(fig9): MEASURED differentiated coordinated sampling (√rate law)
cebd409 
Fig 9 now shows end-to-end differentiated CDM grants on hardware: with the
monitored sid s0 held at constant frequency and total rate skewed 400:80:16,
the coordinator allocates granted p = 0.0010 / 0.0022 / 0.0049 (hot/med/quiet)
— ratio 1:2.2:4.9, matching the √rate law √(400:80:16)=1:2.24:5. Edges auto-learn
the cms_point monitor (key s0) from the controller config.

- mvp-workload-fig9.yaml: monitor functional sum -> cms_point, key: s0.
- fig9_coordinated.sh: seed the monitor into the data-plane BOOT streaming-config
  + restart it (the coordinator only reads monitors() at boot, not on hot-reload),
  tau=7000 / window_ms=15000 so the slack countdown trips each window, pass
  -monitor-key=s0 alongside -monitor-config-url, portable paths, learn+p parsing.
- fig9_cmspoint.csv + figs/fig9_coordinated.png: result + plot (replaces fig9_partial.csv).
- RESULTS.md: rewrite Fig 9 from the prior 'architecturally cannot' note to the
  measured result + the five bugs fixed (needs ASAPQuery-backend#377 coordinator
  per-sid freq + ASAPCollector#504 edge key/flag fixes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Jun 17, 2026
Merged
@zzylol @claude
eval(fig9): drop boot-seed workaround, rely on coordinator hot-reload
625e19a 
With the data-plane coordinator hot-reload landed (ASAPQuery-backend#379), Fig 9
no longer needs the seed-boot-config + restart-data-plane workaround:

- mvp-workload-fig9.yaml: the monitor block now carries the real coordination
  params (tau 7000, window_secs 15) instead of placeholder tau 5e6 / 30 s, so the
  controller-published monitor is directly usable by the coordinator.
- fig9_coordinated.sh: replace the seed+restart of asap-data-plane with a wait for
  the coordinator's 'hot-reloaded monitors' log — the monitor is picked up live.
- RESULTS.md: bug #2 is now a code fix (#379), not a run-side workaround.

Live-validated end-to-end (no seed, no restart): data-plane logs
'hot-reloaded monitors added=1', 0 unconfigured registrations, grants
0.0010/0.0022/0.0049 (hot/med/quiet) — identical to the boot-seed result and
matching the sqrt(rate) law. REQUIRES ASAPQuery-backend#379 in the data-plane image.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zzylol zzylol mentioned this pull request Jun 17, 2026
Merged
@zzylol
zzylol merged commit 646f9da into main Jun 17, 2026
@zzylol
zzylol deleted the eval/8node-cluster-figures branch July 17, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zzylol