test+docs(perf): Phase 2.11 Go benchmarks (path A) - #236
Merged
Merged
Conversation
…-shim Adds testing.B benchmarks for Precompute.Observe across the 5 sketch types in asap-precompute-go, plus shim-level benchmarks per processor. Documents results comparing commit 6b3258d (pre-shim) vs HEAD (post-shim) per ADR-0002 §"Performance contract" 10% p99 gate. asap-precompute-go::Observe results (median of 5): Sketch Pre (ns/op) Post (ns/op) Δ% Gate DDSketch 158.10 155.80 -1.45% PASS KLL 288.40 290.60 +0.76% PASS HLL 146.60 147.70 +0.75% PASS CountSketch 241.00 240.50 -0.21% PASS CountMinSketch 345.20 351.90 +1.94% PASS All five sketch types stay well within the ADR-0002 10% gate; benchstat flags none of the deltas as statistically significant (p > 0.2 with n=5). Allocations are bit-identical pre vs post. precompute_bench_test.go is portable: it uses tiny inline wrappers around sketchlib-go (not the post-shim sketches/ subpackage) so the same source compiles on both 6b3258d and HEAD, pinning the measurement to the runtime's Observe path itself. Per-processor shim benchmarks (post-shim only — no pre-shim equivalent since the legacy code wasn't a shim) are recorded as informational baselines for future regression detection. See docs/phase-2-perf-bench-go.md for the full methodology, raw benchstat output, and absolute shim-level numbers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
zzylol
added a commit
that referenced
this pull request
May 4, 2026
Runs the existing docker-compose b3-delta harness on commit 6b3258d (pre-shim) and HEAD c86a62c (post-shim). Documents whatever throughput / CPU / memory / bandwidth numbers the existing infrastructure produces. Companion to Phase 2.11A (Go bench micro-results, PR #236 merged). Together close ADR-0002 §"Performance contract" with both micro and deployment-level confirmation. Headline results (b3-delta, N=1, c=1000, 10 Hz, mean of 2 samples): agent_cpu_cores 0.0025 → 0.0025 (Δ 0.0%) agent_rss_mib 291.1 → 302.6 (Δ +4.0%) agent_in_kib_per_s 25.74 → 25.74 (Δ 0.0%) agent_out_kib_per_s 3.52 → 3.52 (Δ 0.0%) agent_points_per_s 133.3 → 133.3 (Δ 0.0%) Throughput / IO / CPU within run-to-run noise; RSS +4% explained by the explicit Precompute runtime structure replacing inlined processor state. Within ADR-0002 tolerance. Doc also enumerates 6 gaps in the existing deployment-perf harness (gateway v0.108→v0.141 metric-name skew, ingest-only NaN'd backend columns, no per-Observe latency emission, etc.) with no proposed fixes — separating measurement from action. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
testing.Bbenchmarks forPrecompute.Observe(*Observation)ns/op across the 5 sketch types inasap-precompute-go, plus shim-level benchmarks per processor (processor_bench_test.goin each of the 5 sketch processors).6b3258d(pre-shim baseline, last commit before PRs refactor(kllprocessor): thin shim delegating to asap-precompute-go (Phase 2 step 2.6) #226–refactor(countminsketchprocessor): thin shim delegating to asap-precompute-go (Phase 2 step 2.9) #230) vsf9824e2(post-shim HEAD) and verifies every sketch sits well within the ADR-0002 §"Performance contract" 10% gate.docs/phase-2-perf-bench-go.mddocuments methodology, hardware/Go version, rawbenchstatoutput, per-processor shim numbers (informational, no pre-shim equivalent), and the verdict per sketch.Methodology
go1.25.3 linux/amd64,golang.org/x/perf/cmd/benchstatcount=5,b.ReportAllocs(), deterministic PRNG seed, no-racesketchlib-gorather than depending on the post-shim-onlysketches/subpackage. Same source applies to both commits, isolating the measurement to the runtime'sObservepath.Results —
asap-precompute-go::Observe(the 10% gate)Median of 5 samples per benchmark.
benchstatflagged none of the deltas as statistically significant (every p > 0.2 at n=5). Allocations bit-identical pre vs post (24/81/24/32/128 B/op, 2/4/2/3/5 allocs/op).Per-processor shim baseline (informational)
Median of 5 samples on a 1000-data-point synthetic batch, post-shim only:
Overall verdict
All five sketches PASS the ADR-0002 §"Performance contract" 10% gate. No regression to investigate; nothing to escalate. The shim refactor (PRs #226–#230) preserves per-observation latency to within ±2% at single-machine bench scale.
Test plan
go test -bench=. -benchmem -count=5 -run=^$ .succeeds onasap-precompute-goat both commitsgo test -bench=. -benchmem -count=5 -run=^$ .succeeds in each of the 5processor/<sketch>processor/packages at HEADbenchstat /tmp/asap-pre.txt /tmp/asap-post.txtshows every Δ inside ±2% with p > 0.2 (n=5)TestCSDelta_MultipleWindowsConvergencefailure on countsketchprocessor reproduces with our bench file removed — not introduced here)🤖 Generated with Claude Code