Skip to content

ci: indexing bench on pull requests, base vs head on the same runner - #13

Merged
Chofito merged 1 commit into
mainfrom
feat/bench-ci
Oct 6, 2026
Merged

Chofito merged 1 commit into
mainfrom
feat/bench-ci

Conversation

@Chofito

@Chofito Chofito commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Closes #8

What

  • bun run bench accepts reference repository names as well as paths. A name resolves through scripts/repos.ts (from feat(eval): A/B eval runner (Claude Code with vs without Astrograph) #12) to a fresh checkout at its pinned commit, so no one has to point it at a clone by hand.
  • Two options: --baseline <file> (default .bench-baseline.json, still gitignored) and --bin <bin.ts> (the astrograph entry point to measure; default this checkout).
  • New bench job in CI, on pull requests only: indexes fastify, umami, trpc, koel (JS/CJS, Next.js app, TS monorepo, PHP/Laravel) with the base branch (git worktree at $GITHUB_BASE_REF), then with the PR, and fails on >20% slower or bigger. Clones are cached by the hash of eval/repos/*.json.
  • AGENTS.md: the budget rule now points at the CI job.

Why not a committed baseline (as the issue suggested)

GitHub's shared runners change hardware between runs, so a committed number would drift with the machine, not the code. Measuring base and head on the same runner, minutes apart, compares code. No baseline file to keep up to date.

t3code is a reference repository for the eval but is left out here: it indexes 23k files (it vendors two upstream repos under .repos/) in ~38 s / ~1 GB, too heavy for every PR.

Verified

  • bun test, bun run typecheck, bun run check green.
  • Locally: bench --save --baseline … --bin <other checkout> then bench --baseline … on the four names → all ok (fastify 673 ms / 344 MB, umami 1045 ms / 416 MB, trpc 980 ms / 426 MB, koel 1303 ms / 400 MB on an M-series Mac).
  • This PR's own bench job is the end-to-end test of the workflow.

🤖 Generated with Claude Code

`bun run bench` now accepts reference repository names (eval/repos/, checked
out at their pinned commit through scripts/repos.ts), `--baseline <file>` and
`--bin <bin.ts>`. The new `bench` CI job indexes fastify, umami, trpc and koel
with the base branch, then with the pull request, on the same runner, and
fails on a >20% regression in time or peak memory.

Closes #8

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Chofito
Chofito merged commit 638d330 into main Oct 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run bench in CI on pinned public repos

1 participant