Skip to content

test+docs(perf): Phase 2.11 Go benchmarks (path A) - #236

Merged
zzylol merged 1 commit into
mainfrom
phase2/perf-bench-go
May 4, 2026
Merged

zzylol merged 1 commit into
mainfrom
phase2/perf-bench-go

Conversation

@zzylol

@zzylol zzylol commented May 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Methodology

  • Hardware: AMD Ryzen Threadripper PRO 5955WX 16-Cores, Linux 5.15
  • Toolchain: go1.25.3 linux/amd64, golang.org/x/perf/cmd/benchstat
  • count=5, b.ReportAllocs(), deterministic PRNG seed, no -race
  • Bench file is portable across 6b3258d and HEAD: defines tiny inline wrappers around sketchlib-go rather than depending on the post-shim-only sketches/ subpackage. Same source applies to both commits, isolating the measurement to the runtime's Observe path.
  • 1h window so the bench loop never rotates; pure per-observation timings.
cd asap-precompute-go
go test -bench=. -benchmem -count=5 -run=^$ . > /tmp/asap-pre.txt   # at 6b3258d
go test -bench=. -benchmem -count=5 -run=^$ . > /tmp/asap-post.txt  # at HEAD
benchstat /tmp/asap-pre.txt /tmp/asap-post.txt

Results — asap-precompute-go::Observe (the 10% gate)

Median of 5 samples per benchmark.

Sketch Pre (ns/op) Post (ns/op) Δ% Gate
DDSketch 158.10 155.80 −1.45% PASS
KLL 288.40 290.60 +0.76% PASS
HLL 146.60 147.70 +0.75% PASS
CountSketch 241.00 240.50 −0.21% PASS
CountMinSketch 345.20 351.90 +1.94% PASS

benchstat flagged none of the deltas as statistically significant (every p > 0.2 at n=5). Allocations bit-identical pre vs post (24/81/24/32/128 B/op, 2/4/2/3/5 allocs/op).

Per-processor shim baseline (informational)

Median of 5 samples on a 1000-data-point synthetic batch, post-shim only:

Processor Bench ns/op B/op allocs/op
ddsketch ProcessMetrics (window, observe-only) 552,704 355,116 5,002
ddsketch ProcessBatch (batch) 867,072 572,606 10,167
kll ProcessBatch 737,000 450,199 10,113
hll ProcessBatch 783,714 798,082 7,202
countsketch ProcessMetrics (batch) 1,209,997 711,312 10,160
countminsketch ProcessBatch 2,158,571 2,108,229 18,312

Overall verdict

All five sketches PASS the ADR-0002 §"Performance contract" 10% gate. No regression to investigate; nothing to escalate. The shim refactor (PRs #226#230) preserves per-observation latency to within ±2% at single-machine bench scale.

Test plan

  • go test -bench=. -benchmem -count=5 -run=^$ . succeeds on asap-precompute-go at both commits
  • go test -bench=. -benchmem -count=5 -run=^$ . succeeds in each of the 5 processor/<sketch>processor/ packages at HEAD
  • benchstat /tmp/asap-pre.txt /tmp/asap-post.txt shows every Δ inside ±2% with p > 0.2 (n=5)
  • No source code modified outside the new bench / doc files
  • No submodule pointer drift in the commit
  • Pre-existing unit tests still pass (one prior TestCSDelta_MultipleWindowsConvergence failure on countsketchprocessor reproduces with our bench file removed — not introduced here)

🤖 Generated with Claude Code

…-shim

Adds testing.B benchmarks for Precompute.Observe across the 5 sketch
types in asap-precompute-go, plus shim-level benchmarks per processor.
Documents results comparing commit 6b3258d (pre-shim) vs HEAD
(post-shim) per ADR-0002 §"Performance contract" 10% p99 gate.

asap-precompute-go::Observe results (median of 5):

  Sketch          Pre (ns/op)  Post (ns/op)   Δ%       Gate
  DDSketch          158.10        155.80    -1.45%   PASS
  KLL               288.40        290.60    +0.76%   PASS
  HLL               146.60        147.70    +0.75%   PASS
  CountSketch       241.00        240.50    -0.21%   PASS
  CountMinSketch    345.20        351.90    +1.94%   PASS

All five sketch types stay well within the ADR-0002 10% gate; benchstat
flags none of the deltas as statistically significant (p > 0.2 with
n=5). Allocations are bit-identical pre vs post.

precompute_bench_test.go is portable: it uses tiny inline wrappers
around sketchlib-go (not the post-shim sketches/ subpackage) so the
same source compiles on both 6b3258d and HEAD, pinning the measurement
to the runtime's Observe path itself.

Per-processor shim benchmarks (post-shim only — no pre-shim equivalent
since the legacy code wasn't a shim) are recorded as informational
baselines for future regression detection. See
docs/phase-2-perf-bench-go.md for the full methodology, raw benchstat
output, and absolute shim-level numbers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit c86a62c into main May 4, 2026
@zzylol
zzylol deleted the phase2/perf-bench-go branch May 4, 2026 15:16
zzylol added a commit that referenced this pull request May 4, 2026
Runs the existing docker-compose b3-delta harness on commit 6b3258d
(pre-shim) and HEAD c86a62c (post-shim). Documents whatever throughput
/ CPU / memory / bandwidth numbers the existing infrastructure
produces.

Companion to Phase 2.11A (Go bench micro-results, PR #236 merged).
Together close ADR-0002 §"Performance contract" with both micro and
deployment-level confirmation.

Headline results (b3-delta, N=1, c=1000, 10 Hz, mean of 2 samples):

  agent_cpu_cores      0.0025 → 0.0025   (Δ  0.0%)
  agent_rss_mib        291.1  → 302.6    (Δ +4.0%)
  agent_in_kib_per_s   25.74  → 25.74    (Δ  0.0%)
  agent_out_kib_per_s  3.52   → 3.52     (Δ  0.0%)
  agent_points_per_s   133.3  → 133.3    (Δ  0.0%)

Throughput / IO / CPU within run-to-run noise; RSS +4% explained
by the explicit Precompute runtime structure replacing inlined
processor state. Within ADR-0002 tolerance.

Doc also enumerates 6 gaps in the existing deployment-perf harness
(gateway v0.108→v0.141 metric-name skew, ingest-only NaN'd backend
columns, no per-Observe latency emission, etc.) with no proposed
fixes — separating measurement from action.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant