Loosen the vs-scipy timing ceilings past CI runner noise - #59
Merged
Merged
Conversation
`matvec_vs_scipy` and `matmat_vs_scipy` were gated at a ratio of 1.0 -- at least as fast as scipy -- from a measurement of ~0.25 taken on a workstation. Shared GitHub runners measure 1.09 to 1.35 for the same commit, so the gate failed on changes that cannot affect throughput, including a docstring-only one. Re-running an unchanged commit turned a failure into a pass. Ceilings to 2.5 and the margin to 10.0, which clears the worst observed runner figure by ~85% while still catching a 2x regression. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rvu3cf8ZL7F5EPX22eo6Je
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
matvec_vs_scipyandmatmat_vs_scipyare gated at a ratio of 1.0 — at least as fast as scipy — recorded from a ~0.25 measurement on a workstation. Shared GitHub runners measure 1.09 to 1.35 for the same commit, so the gate fails on changes that cannot affect throughput, including a docstring-only one.Re-running three unchanged commits turned every failure into a pass.
Fix
Ceilings to 2.5 and the margin to 10.0. The margin alone would not have helped: it is applied at
--recordtime, so the recorded ceiling has to move too.Measurements
2.5 clears the worst observed runner figure (1.347) by ~85% and still catches a 2x regression.
Merge order
First, ahead of everything else — until it lands, unrelated PRs fail this gate intermittently.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Rvu3cf8ZL7F5EPX22eo6Je