Skip to content

Instruction-count benchmark gates - #35

Merged
Kinflou merged 6 commits into
mainfrom
feat/benchmark-gates
Oct 25, 2026
Merged

Kinflou merged 6 commits into
mainfrom
feat/benchmark-gates

Conversation

@Kinflou

@Kinflou Kinflou commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Stage 1 of runtime#29: benchmark gates. Was to be stacked on #32, which has merged, so this targets main.

  • Shared definitions, runtime/conformance/benches/: five format shapes (small scalar, string-heavy, nested, 1000-element array, enum-heavy) in the vectors' value and type encoding, framing payloads of 16 B to 64 KB, the handshake frame and one connection step, with a README of the format and the bench ids. A conformance test checks each shape round-trips through Wire and decode_typed in all three formats.
  • iai-callgrind benches, runtime/core/benches/runtime.rs (58 benchmarks, iai-callgrind pinned to 0.16.1): encode and decode through serde-derive mirrors of each shape, framing encode and decode for datagram@1, datagram@2 and jsonrpc@1, handshake encode, decode and check, and State::encode_request then receive. Inputs come from setup functions. A test asserts each mirror encodes to exactly its definition's bytes in every format.
  • Gate, .github/workflows/bench.yml: benches the base in a git worktree with --save-baseline=base, then the head with --baseline=base --callgrind-limits=ir=1%. A base with no benches (this PR) records only. The comparison goes to the job summary, and rust.bench.json is uploaded as an artifact (not committed).
  • Noise and threshold:
    • The same commit, run three times locally, differed by 0 instructions on all 58 benches. In CI, a no-op change compared against this branch was also 58 of 58 unchanged.
    • With the default codegen units, slowing only the handshake encode also moved jsonrpc@1 encode by up to +2.6%, with no change to its code. [profile.bench] codegen-units = 1 removes that.
    • The limit is ir=1%.
  • Gate demo (throwaway [demo, do not merge] bench gate check #36, closed): an extra 20-iteration loop in HandshakeFrame::encode failed the job with handshake/encode 336 → 357 (+6.25%); the other 57 benches were unchanged.
  • Base and head build into separate target directories. In a shared one, Cargo reused the base's build for the head: the path packages hash the same from either checkout, and the head's sources look older. The first demo run passed because of this. They still share iai-callgrind's output directory for the baseline.
  • No --regression-fail-fast: it stops at the first regression and prints no summary. Without it, every bench is compared and the run still exits non-zero.

Kinflou added 4 commits October 25, 2026 14:45
Five format shapes in the vectors' value encoding, the framing payload
sizes, the handshake frame and the connection step, vendored by every
language so each measures the same work. The Rust mirrors of the shapes,
which the benches will use, are checked to encode exactly as their
definitions do in every format.
Encode and decode per format per shape through serde-derive mirrors,
framing encode and decode per framing per payload size, the handshake
frame's encode, decode and check, and one connection step. Inputs are
built by setup functions and handed back, so neither building nor
dropping them is measured.

iai-callgrind is pinned to 0.16.1; iai-callgrind-runner must match.
Runs the base's benches in a worktree, saves them as a baseline, then
runs the head's against it with an ir limit of 1%. A base with no benches
is skipped and the head only recorded. The comparison goes to the job
summary and the head's counts to a rust.bench.json artifact.

The bench profile builds one codegen unit: with the default partitioning,
slowing the handshake encode also moved the JSON-RPC framing benches by
up to 2.6%, with no change to their code.
Kinflou added 2 commits October 25, 2026 14:55
In one shared target directory Cargo names a workspace's path packages the
same from either checkout, and the head's sources, checked out before the
base was built, look older than the base's build, so the head reused it and
was never measured. iai-callgrind's output moves to its own directory, which
both runs share for the baseline.
@Kinflou
Kinflou merged commit 17802c2 into main Oct 25, 2026
12 checks passed
@Kinflou
Kinflou deleted the feat/benchmark-gates branch October 25, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant