Repository navigation
ratchet: name the unloaded spec files on every run, not only on a regression - #719
Merged
Merged
Conversation
…ession A spec file that fails to load carries tens of examples, so its coming and going is the largest single term in the total -- larger than every per-example flake combined. Measured on ten consecutive runs of this workflow on main, `files loaded` moves between 2198 and 2202, and on one branch a documentation-only commit moved the total by 43 because a single file flipped from loading to not. Printing only the count makes that step visible and unattributable, which is the worst of both: a reader sees the number move and cannot tell whether a spec regressed or a file merely did not load. Those need opposite responses. So the names now print whenever any file failed to load, and the regression branch points at them rather than repeating the list. This is the change that would have saved a wrong decision: I set the floor with a 6-example margin on the belief that the variation was the +/-1 of core/random/bytes_spec.rb, when the dominant term is a whole file. The count was in front of me on every run and the cause was not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
run.shnames the spec files that failed to load on every run. Before, thecount was printed always and the names only inside the regression branch.
Exercised in both directions before pushing: with
FILES_FAIL=0the block printsnothing, with
FILES_FAIL=3it names all three, and the two outputs differ — ablock that printed the same either way would not be reporting.
Why it matters more than its size
A file that fails to load carries tens of examples. Ten consecutive runs of this
workflow on
main:And on #716's branch a documentation-only commit moved the total by +43, with
files loadedflipping 2201 → 2202.So the largest source of movement in this number was visible and
unattributable: you could see the total step and not learn which file caused it.
A regressed spec and an unloaded file need opposite responses — one is a defect, the
other is usually an environment or a timeout — and the count alone cannot tell them
apart.
This is the change that would have prevented a wrong decision of mine, recorded in
#710: I raised
FLOORwith a 6-example margin believing the variation was the±1 of
core/random/bytes_spec.rb. The dominant term is ~43. The count was in frontof me on every run; the cause was not.
🤖 Generated with Claude Code