Skip to content

docs: the README's ruby/spec count now comes from BASELINE, checked by a test - #765

Merged
tannevaled merged 1 commit into
mainfrom
wave60-doc-count
Oct 4, 2026
Merged

tannevaled merged 1 commit into
mainfrom
wave60-doc-count

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

What

The README carried three figures for the same quantity, and they disagreed
with each other and with the repository:

where claimed
the badge (line 8) 22,488
prose, 3 places 23,471
scripts/conformance/rubyspec/BASELINE, summed 23,481

The badge was 1,015 low and carries no date, so it read as current. The
prose at the top — "It runs 23,471 of ruby/spec's examples" — and in the
architecture section read as current too. All three now come from BASELINE,
the per-file record CI actually enforces.

The table row is deliberately left alone

It keeps 23,471, because the table declares itself "Measured 2026-09-28 on
862a9f3, darwin/arm64"
against a pinned corpus. A dated measurement going
stale is not an error; an undated claim going stale is.

Two further reasons not to touch it:

  • its other rows (failing, erroring, skipped, ran, share) are not derivable
    from BASELINE, which records passing counts and nothing else — so
    refreshing one cell would have made the table internally inconsistent
    (pass + fail + error would stop equalling ran);
  • re-running the full sweep to refresh all of them would have made the docs
    less true right now: the machine is at load ~10, and the FILEFAIL/timeout
    rows are exactly what load moves.

A clause beside the table now says which number tracks what, so a reader is not
left with two figures and no explanation.

Machine-checked from here

A number a human retypes is a number that rots.
TestReadmeConformanceCountMatchesTheBaseline reads BASELINE, sums the OK
rows, and fails with the figure to write:

README badge claims 22,488 ruby/spec examples, BASELINE sums to 23481
  update the badge to 23,481
README.md:15 claims 23,471 examples, BASELINE sums to 23481

That output is its known-bad control — putting the old numbers back makes it
name both sites and the correct value, and restoring them makes it pass.

The rule it applies is the table's own: every bold thousands-separated number
outside a table row must equal the baseline total. There are exactly three
today and all three are this count; a future bold number about something else
should be made unambiguous rather than have the rule widened.

Also audited, and found correct

  • the Go badge (go-1.26.4+) matches go.mod's go 1.26.4; CI's
    go-version: stable is above that floor, so there is no contradiction;
  • scripts/conformance/rubyspec/README.md already says the gate "used to be"
    a FLOOR file — past tense, correct since conformance: switch the ruby/spec gate to the per-file baseline (#710) #739;
  • the front-end campaign section already carries its own date and an explicit
    "have not been re-measured since" note;
  • the table's 2,202 of 2,206 produced a result; 4 produced none matches
    BASELINE exactly today (2202 OK rows, 4 FILEFAIL).

Green

go test ./... -count=1 -timeout=90m exit 0 (10 packages), gofmt clean,
go vet clean.

🤖 Generated with Claude Code

…y a test

The README carried three figures for the same quantity and they disagreed with
each other and with the repository:

    badge                       22,488
    prose (3 places)            23,471
    scripts/.../BASELINE sums to 23,481

The badge was 1,015 low and carries no date, so it read as current. The prose
at the top ("It runs 23,471 of ruby/spec's examples") and in the architecture
section read as current too. All three now come from BASELINE, the per-file
record CI actually enforces.

The TABLE row keeps 23,471, and that is deliberate: the table declares itself
"Measured 2026-09-28 on 862a9f3, darwin/arm64" against a pinned corpus, and a
dated measurement going stale is not an error -- an undated claim going stale
is. Its other rows (failing, erroring, skipped, ran, share) are not derivable
from BASELINE, which records passing counts and nothing else, so refreshing one
cell would have made the table internally inconsistent. Re-running the full
sweep to refresh all of them would have made the docs LESS true right now: the
machine is at load 10, and the FILEFAIL/timeout rows are exactly what load
moves. A clause by the table says which number tracks what, so a reader is not
left with two figures and no explanation.

A number a human retypes is a number that rots, so this is machine-checked from
here: TestReadmeConformanceCountMatchesTheBaseline reads BASELINE, sums the OK
rows, and fails with the figure to write. Verified by its known-bad control --
putting 22,488 and 23,471 back makes it name both sites and the correct value.

The rule it applies is the table's own: every bold thousands-separated number
outside a table row must equal the baseline total. There are exactly three
today and all three are this count; a future bold number about something else
should be made unambiguous rather than have the rule widened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit a2d6d40 into main Oct 4, 2026
9 checks passed
@tannevaled
tannevaled deleted the wave60-doc-count branch October 4, 2026 15:45
tannevaled added a commit that referenced this pull request Oct 4, 2026
… is shared (#770)

The embedding section promises its contract "as measured rather than intended".
Two of its statements were not what the code does, and a third capability was
not mentioned at all. Each correction below is a measurement against
origin/main through the public `Run` and nothing else.

"Nothing is shared between calls" is true of the Ruby heap and false of the
process. A script that assigns to `ENV` and calls `Dir.chdir` changes them for
the host Go program and they stay changed after `Run` returns: a host whose
PATH was 860 bytes and whose cwd was its build directory came back with a
13-byte PATH and a cwd of /private/tmp. The bullet immediately below it already
documented that `require_relative` resolves against the *process* working
directory, so the README contained the fact that contradicted it; one `Run` can
therefore plant a file a later `Run` loads. The bullet is now scoped to the
heap, which is the part that holds -- a global, a constant and a String
monkey-patch really are gone in the next call.

`exit`, `exit!` and `abort` make `Run` return nil, so an embedder cannot tell
"finished" from "stopped": `abort "fatal"` writes its message to `out` and
reports success, and `exit 3` loses the 3. Only a Ruby exception becomes a Go
error. Said explicitly, with the reason it cannot be worked around today --
the CLI recovers the status through `internal/vm`, which is unreachable.

`Kernel#fork` exists and runs the block in the same process, returning a
synthetic pid while `Process.pid` is unchanged, so the "child" mutates the
parent's globals, ENV and cwd. The README said only that `Process.fork` does
not exist, which reads as "fork is unavailable" -- but `fork { }` is the
spelling almost every program uses, and code that forks to isolate or to drop
privileges gets neither. Added as a divergence row and cross-referenced from
the Process section.

Finally a "Run is not a sandbox" subsection. This is not a defect report: MRI
is not sandboxed either and conformance is the goal. It is worth stating
because MRI is normally a CLI a person runs on their own code whereas `Run` is
a function someone links into a server. It also records the two things that are
deliberately out of reach -- no path from Ruby to the host's Go state, and the
host's standard descriptors are not exposed, with IO.for_fd(1) raising EBADF
where MRI writes to the real descriptor -- because those are properties worth
not losing by accident.

The new test file pins all of it, in the spirit of #765: the prose cannot drift
from the code silently, and each failure is the signal to rewrite the bullet.
The raise-becomes-an-error twin is there on purpose -- without it the
exit-returns-nil test would also pass if `Run` swallowed every error.

Refs #768.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant