feat(eval): accuracy-vs-cost Pareto sweep (Fig 1, real Google trace) - #491
Merged
Merged
Conversation
One combined single-pass sweep on the 2019 Google cluster trace
(data_plane backend + otel-app SDK-sketch producer, fresh backend per
arm) producing (cost, accuracy) operating points for the Fig 1 headline.
y = 1 - p99 rel-err of quantile_over_time(0.99, cpu_rate[30s]) vs the
exact pooled true p99.
x = total cost = 0.5*(edge CPU cores + wire KB/s), each normalized to
raw-forwarding = 1.0 (both sub-axes reported; backend CPU excluded
and stated). wire metered via iptables on OTLP dport 4317; edge CPU
via /proc utime+stime.
Frontier: raw (1.0,1.0) -> DDSketch full p=1 (0.45x, acc 0.997) ->
sampling p down to 0.37x raw (2.7x cheaper) with acc >= 0.96. Wire alone
is cut ~44x. Sampling's measured lever is edge CPU + ingest (10x fewer
admitted points at p=0.1); the ε_s tail (p<=0.25) is kept, not hidden.
Honest negatives recorded in RESULTS.md: delta/ε_cdm does NOT cut wire on
this pooled-tumbling trace (independent windows -> no cross-window
redundancy; first-window-full dominates a short pass), unlike the
synthetic slowly-changing workload in evaluation-plan-figures.md Table 2.
Driver (pareto_sweep.py) + plot (pareto_plot.py) + PNG + raw JSON +
RESULTS.md. Sweep cleans up all procs/ports/iptables on exit (verified 0
left).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fig 1 headline on the real 2019 Google cluster trace. cost =
0.5·(cpu_norm+wire_norm)vs raw=1.0 (both sub-axes reported); fresh backend per arm; accuracy = 1−p99-rel-err vs true quantile.DDSketch warm answers p99 at 0.45× raw / 0.997 acc; sampling → 0.37× (2.7× cheaper) at ≥0.96. Wire egress cut ~44×; then CPU-dominated, p↓ trims it (admitted 4928→486 pts/s).
Honest negative: delta does NOT cut wire on this pooled-tumbling/always-changing trace (independent windows, no redundancy) — the delta/ε-gate regime is DEBS (PR #490). Driver+plot+RESULTS under
datasets_eval/google_cluster/e2e/pareto/.🤖 Generated with Claude Code