Repository navigation
docs: the README's ruby/spec count now comes from BASELINE, checked by a test - #765
Merged
Merged
Conversation
…y a test
The README carried three figures for the same quantity and they disagreed with
each other and with the repository:
badge 22,488
prose (3 places) 23,471
scripts/.../BASELINE sums to 23,481
The badge was 1,015 low and carries no date, so it read as current. The prose
at the top ("It runs 23,471 of ruby/spec's examples") and in the architecture
section read as current too. All three now come from BASELINE, the per-file
record CI actually enforces.
The TABLE row keeps 23,471, and that is deliberate: the table declares itself
"Measured 2026-09-28 on 862a9f3, darwin/arm64" against a pinned corpus, and a
dated measurement going stale is not an error -- an undated claim going stale
is. Its other rows (failing, erroring, skipped, ran, share) are not derivable
from BASELINE, which records passing counts and nothing else, so refreshing one
cell would have made the table internally inconsistent. Re-running the full
sweep to refresh all of them would have made the docs LESS true right now: the
machine is at load 10, and the FILEFAIL/timeout rows are exactly what load
moves. A clause by the table says which number tracks what, so a reader is not
left with two figures and no explanation.
A number a human retypes is a number that rots, so this is machine-checked from
here: TestReadmeConformanceCountMatchesTheBaseline reads BASELINE, sums the OK
rows, and fails with the figure to write. Verified by its known-bad control --
putting 22,488 and 23,471 back makes it name both sites and the correct value.
The rule it applies is the table's own: every bold thousands-separated number
outside a table row must equal the baseline total. There are exactly three
today and all three are this count; a future bold number about something else
should be made unambiguous rather than have the rule widened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Oct 4, 2026
… is shared (#770) The embedding section promises its contract "as measured rather than intended". Two of its statements were not what the code does, and a third capability was not mentioned at all. Each correction below is a measurement against origin/main through the public `Run` and nothing else. "Nothing is shared between calls" is true of the Ruby heap and false of the process. A script that assigns to `ENV` and calls `Dir.chdir` changes them for the host Go program and they stay changed after `Run` returns: a host whose PATH was 860 bytes and whose cwd was its build directory came back with a 13-byte PATH and a cwd of /private/tmp. The bullet immediately below it already documented that `require_relative` resolves against the *process* working directory, so the README contained the fact that contradicted it; one `Run` can therefore plant a file a later `Run` loads. The bullet is now scoped to the heap, which is the part that holds -- a global, a constant and a String monkey-patch really are gone in the next call. `exit`, `exit!` and `abort` make `Run` return nil, so an embedder cannot tell "finished" from "stopped": `abort "fatal"` writes its message to `out` and reports success, and `exit 3` loses the 3. Only a Ruby exception becomes a Go error. Said explicitly, with the reason it cannot be worked around today -- the CLI recovers the status through `internal/vm`, which is unreachable. `Kernel#fork` exists and runs the block in the same process, returning a synthetic pid while `Process.pid` is unchanged, so the "child" mutates the parent's globals, ENV and cwd. The README said only that `Process.fork` does not exist, which reads as "fork is unavailable" -- but `fork { }` is the spelling almost every program uses, and code that forks to isolate or to drop privileges gets neither. Added as a divergence row and cross-referenced from the Process section. Finally a "Run is not a sandbox" subsection. This is not a defect report: MRI is not sandboxed either and conformance is the goal. It is worth stating because MRI is normally a CLI a person runs on their own code whereas `Run` is a function someone links into a server. It also records the two things that are deliberately out of reach -- no path from Ruby to the host's Go state, and the host's standard descriptors are not exposed, with IO.for_fd(1) raising EBADF where MRI writes to the real descriptor -- because those are properties worth not losing by accident. The new test file pins all of it, in the spirit of #765: the prose cannot drift from the code silently, and each failure is the signal to rewrite the bullet. The raise-becomes-an-error twin is there on purpose -- without it the exit-returns-nil test would also pass if `Run` swallowed every error. Refs #768. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The README carried three figures for the same quantity, and they disagreed
with each other and with the repository:
scripts/conformance/rubyspec/BASELINE, summedThe badge was 1,015 low and carries no date, so it read as current. The
prose at the top — "It runs 23,471 of ruby/spec's examples" — and in the
architecture section read as current too. All three now come from
BASELINE,the per-file record CI actually enforces.
The table row is deliberately left alone
It keeps
23,471, because the table declares itself "Measured 2026-09-28 on862a9f3, darwin/arm64" against a pinned corpus. A dated measurement goingstale is not an error; an undated claim going stale is.
Two further reasons not to touch it:
from
BASELINE, which records passing counts and nothing else — sorefreshing one cell would have made the table internally inconsistent
(
pass + fail + errorwould stop equallingran);less true right now: the machine is at load ~10, and the FILEFAIL/timeout
rows are exactly what load moves.
A clause beside the table now says which number tracks what, so a reader is not
left with two figures and no explanation.
Machine-checked from here
A number a human retypes is a number that rots.
TestReadmeConformanceCountMatchesTheBaselinereadsBASELINE, sums theOKrows, and fails with the figure to write:
That output is its known-bad control — putting the old numbers back makes it
name both sites and the correct value, and restoring them makes it pass.
The rule it applies is the table's own: every bold thousands-separated number
outside a table row must equal the baseline total. There are exactly three
today and all three are this count; a future bold number about something else
should be made unambiguous rather than have the rule widened.
Also audited, and found correct
go-1.26.4+) matchesgo.mod'sgo 1.26.4; CI'sgo-version: stableis above that floor, so there is no contradiction;scripts/conformance/rubyspec/README.mdalready says the gate "used to be"a
FLOORfile — past tense, correct since conformance: switch the ruby/spec gate to the per-file baseline (#710) #739;"have not been re-measured since" note;
2,202 of 2,206 produced a result; 4 produced nonematchesBASELINEexactly today (2202OKrows, 4FILEFAIL).Green
go test ./... -count=1 -timeout=90mexit 0 (10 packages),gofmtclean,go vetclean.🤖 Generated with Claude Code