Skip to content

Raise ruby/spec floor 21740 -> 21970 after wave 27 - #638

Merged
tannevaled merged 1 commit into
mainfrom
wave27/floor
Sep 23, 2026
Merged

Raise ruby/spec floor 21740 -> 21970 after wave 27#638
tannevaled merged 1 commit into
mainfrom
wave27/floor

Conversation

@tannevaled

@tannevaled tannevaled commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Four units landed in this wave.

PR area gain
#628 language (compiler) — defined?(super) (18 examples behind one opcode), the defined? tags frozen, array/hash literal probing, the two method probes MRI distinguishes, block auto-splat through #to_ary, the pattern-matching deconstruct contract language 2158 → 2222
#634 predefined globals, trace_var/untrace_var, rb_feature_p, Kernel#Complex, Module#module_function, Kernel#warn and #open predefined_spec 73 → 124 passing, core/kernel +26, core/module +6
#629 FileTest re-exported from File's own records, the Errno table from 28 names to MRI's 158, SystemCallError#initialize/#errno/.===, the real realpath/realdirpath walk, File::Stat#atime/#ctime (both had returned mtime), #inspect, File#stat/#lstat core/file 618 → 629, core/filetest 45 → 88, core/exception 149 → 174
#537 ci: actions/setup-go v7 — unblocked by rebasing; it had been red for three weeks on a coverage gate main no longer enforces on that lane —

Measurement

scripts/conformance/rubyspec/run.sh against the pinned corpus at 87b1631, 16-way, TIMEOUT=25, three consecutive runs:

passing examples: 21983   (files loaded: 2191, failed to load: 15)
passing examples: 21982
passing examples: 21981
frozen floor:     21740

The floor is set eleven below the lowest run — the same margin rule as the last two bumps: enough for the run-to-run jitter, not enough to hide the loss of a whole spec file, which is what the ratchet exists to catch. #615 remains open and can still cost a file when it fires.

This one is conformance throughout

Unlike the previous bump, none of this comes from fixing our own harness or from a dependency making previously-unparseable files visible. All 241 examples are behaviour the VM did not have before.

Three Windows-only failures were fixed along the way, and the split is worth recording because two were ours and one was not:

  • realpath_rec built /C: — it rooted the walk at / and turned the volume letter into a path component. A real defect that would have broken File.realpath for every Windows user.
  • File::Stat#inspect printed blocks=0 where #blocks answers nil — the member values had been written twice and drifted at the one member that can be nil.
  • featureProvided's test fixture used /lib, which is not absolute on Windows. The matching was right; the fixture was POSIX-shaped. Five of its eight cases had been passing on Windows for the wrong reason — the prefix never matched, so the rule each named was never reached.

🤖 Generated with Claude Code

@tannevaled @claude
Raise ruby/spec floor 21740 -> 21970 after wave 27
edad6ab 
Four units:

  #628  language (compiler): defined?(super) — 18 examples behind one opcode —
        the defined? tags frozen, array/hash literal probing, the two method
        probes MRI distinguishes, block auto-splat through #to_ary, and the
        pattern-matching deconstruct contract   (language 2158 -> 2222)
  #634  predefined globals, trace_var/untrace_var, rb_feature_p, Kernel#Complex,
        Module#module_function, Kernel#warn and #open
                                   (predefined_spec 73 -> 124 passing,
                                    core/kernel +26, core/module +6)
  #629  FileTest re-exported from File's own records, the Errno table from 28
        names to MRI's 158, SystemCallError#initialize/#errno/.===, the real
        realpath/realdirpath walk, File::Stat#atime/#ctime (both had returned
        mtime), #inspect, File#stat/#lstat
                                   (core/file 618 -> 629, core/filetest 45 -> 88,
                                    core/exception 149 -> 174)
  #537  ci: actions/setup-go v7 (unblocked by rebasing; it had been red for
        three weeks on a coverage gate main no longer enforces on that lane)

Measured with scripts/conformance/rubyspec/run.sh against the pinned corpus at
87b1631, 16-way, TIMEOUT=25, three consecutive runs: 21983, 21982, 21981. The
floor is set eleven below the lowest, the same margin rule as the last two
bumps: enough for the run-to-run jitter, not enough to hide the loss of a whole
spec file. #615 remains open and can still cost a file when it fires.

This bump is VM conformance throughout — unlike the previous one, none of it
comes from the harness or from a dependency making new files visible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Sep 23, 2026
@tannevaled @claude
README: re-measure against d498ddc — #629 merged after the first base…
4825391 
…line

#629 (FileTest, the Errno subsystem, realpath_rec, the stat surface) merged at
17:10Z, ten minutes after the 68cb53a baseline this branch was written against.
Two statements were left wrong in the conservative direction. Rebased onto
d498ddc, rebuilt, re-measured everything.

ruby/spec, four consecutive runs on d498ddc: 21982, all four identical (the
68cb53a runs spread 21845-21903 on #615; it did not fire this time, but the
issue is open, so the text still says to run more than once and take the low
run). Detail sweep: 21983 pass, 1321 fail, 740 error, 473 skip, 2191 of 2206
files producing a result, 91.4% of the 24044 examples that ran.

The floor on main is still 21740; #638 is open to take it to 21970. Every place
the floor appears now says which one it is and names #638, rather than carrying
a number that goes stale when that PR merges.

Errno: was "28 constants, not MRI's 158". Measured on d498ddc, Errno.constants
is 158 under rbgo AND under MRI 4.0.5. Dumping every name/value pair from both
and joining them: all 158 names present in both, 32 disagree, and all 32 are
the same shape -- rbgo reports 0 where MRI on darwin has a real number (EAUTH
0 vs 80, EBADRPC 0 vs 72, EDEVERR 0 vs 83). The other 126 match exactly. That
is #633, and 32 is the number that matters rather than the 81 constants that
read 0 in rbgo, because MRI reads 0 for 49 of them too.

FileTest: was "11 of MRI's 26 predicates", with 15 listed as not implemented.
Measured: FileTest.methods(false) is 26 under both, name for name identical.
Differentially tested all 26 against MRI 4.0.5 over a temp tree (regular file,
empty file, directory, symlink, executable, FIFO, missing path, /dev/null,
/dev/disk0): 252 assertions, 0 disagreements.

Everything else in the limitations table re-probed on d498ddc and still holds:
Numeric#to_int NoMethodError, File::Stat#== identity ([false, true] vs MRI's
[true, true]), __LINE__ 0, const_source_location nil, Thread#backtrace of
another thread NotImplementedError, no Process.fork. All nine issues still open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 23, 2026
Merged
Merged
Merged
@tannevaled
tannevaled merged commit 2200e17 into main Sep 23, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave27/floor branch September 23, 2026 18:02
tannevaled added a commit that referenced this pull request Sep 23, 2026
@tannevaled @claude
README: re-measure against d498ddc — #629 merged after the first base…
22d4f9a 
…line

#629 (FileTest, the Errno subsystem, realpath_rec, the stat surface) merged at
17:10Z, ten minutes after the 68cb53a baseline this branch was written against.
Two statements were left wrong in the conservative direction. Rebased onto
d498ddc, rebuilt, re-measured everything.

ruby/spec, four consecutive runs on d498ddc: 21982, all four identical (the
68cb53a runs spread 21845-21903 on #615; it did not fire this time, but the
issue is open, so the text still says to run more than once and take the low
run). Detail sweep: 21983 pass, 1321 fail, 740 error, 473 skip, 2191 of 2206
files producing a result, 91.4% of the 24044 examples that ran.

The floor on main is still 21740; #638 is open to take it to 21970. Every place
the floor appears now says which one it is and names #638, rather than carrying
a number that goes stale when that PR merges.

Errno: was "28 constants, not MRI's 158". Measured on d498ddc, Errno.constants
is 158 under rbgo AND under MRI 4.0.5. Dumping every name/value pair from both
and joining them: all 158 names present in both, 32 disagree, and all 32 are
the same shape -- rbgo reports 0 where MRI on darwin has a real number (EAUTH
0 vs 80, EBADRPC 0 vs 72, EDEVERR 0 vs 83). The other 126 match exactly. That
is #633, and 32 is the number that matters rather than the 81 constants that
read 0 in rbgo, because MRI reads 0 for 49 of them too.

FileTest: was "11 of MRI's 26 predicates", with 15 listed as not implemented.
Measured: FileTest.methods(false) is 26 under both, name for name identical.
Differentially tested all 26 against MRI 4.0.5 over a temp tree (regular file,
empty file, directory, symlink, executable, FIFO, missing path, /dev/null,
/dev/disk0): 252 assertions, 0 disagreements.

Everything else in the limitations table re-probed on d498ddc and still holds:
Numeric#to_int NoMethodError, File::Stat#== identity ([false, true] vs MRI's
[true, true]), __LINE__ 0, const_source_location nil, Thread#backtrace of
another thread NotImplementedError, no Process.fork. All nine issues still open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Sep 23, 2026
@tannevaled @claude
README: the floor IS 21,970 — #638 merged
324176b 
#638 merged at 18:02Z. FLOOR on origin/main is 21970, which I re-read from the
tree. Three mentions phrased it as a pending proposal and are now simply wrong.

The arithmetic mattered more than the link. Every sentence built on a floor of
21,740 has been recomputed: the margin is 12, not ~240, and it appeared as
"~240 above the floor" in two places and "the gap is ~240 examples" in a third.

Rewrote the floor-vs-score explanation rather than swapping the number into it.
The old wording said the measured total "normally sits above" the floor and the
floor is "raised deliberately after a win", which reads as the floor lagging out
of laziness and leaves a reader wondering why 21,982 and 21,970 are nearly
equal. #638 states the actual rule, and its own arithmetic confirms it -- its
lowest run was 21981 against a floor of 21970, eleven below:

  the floor is set a handful of examples below the LOWEST observed run --
  enough to absorb the #615 jitter, and no more, because a floor slack enough
  to hide the loss of a whole spec file would defeat what the ratchet exists
  to catch.

So a small gap is the ratchet working; a large one would mean it had gone slack.
My own four runs give a margin of 12 against the same rule.

Provenance: the measurement was taken on d498ddc. main is now 2200e17, whose
only change is the FLOOR file, so I rebuilt and compared -- the binaries are
byte-identical (sha256 32cd4f9e...), and the figures carry over rather than
being assumed to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Sep 23, 2026
@tannevaled @claude
README: measured ruby/spec conformance (21,982 vs the 15,400+ claimed…
e71f4fa 
…), and the claims that had drifted (#637)

* README: measured ruby/spec conformance, and the claims that had drifted

The ruby/spec figure said 15,400+, which was the FLOOR bumped on 2026-08-25
(#451). Measured on main (68cb53a) against the pinned corpus today: 21,901
passing, four runs 21845/21903/21901/21902, floor 21,740.

Also corrected against the code and .github/workflows/ci.yml:

* the 100% coverage claim ("across all six 64-bit targets and three OSes"):
  the gate runs on ubuntu + macOS only, per-PR arch lanes are amd64/arm64
  without coverage, the exotic four are nightly QEMU + scheduled hardware
* the bound-module count, 181 -> 178 modules across 176 orgs, with the seven
  bound outside the go-ruby-* convention named
* Set (in-tree over object.Hash, not go-composites/set), BigDecimal
  (go-ruby-bigdecimal, not go-composites/bigfloat) and Date (go-ruby-date,
  not go-composites/date)
* the phase badge: plan-rbgo.md marks Phase 6 "substantially landed", not done

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* README: the subsystems that landed since the feature list was written, and a plain limitations section

Verified item by item against main (68cb53a) with a probe run through the
built binary, MRI 4.0.5 on the same host as the oracle:

* Process (spawn/exec, the wait family, process groups, priorities, rlimits) —
  and that Process.fork does NOT exist
* Fiber#raise / #kill / storage / blocking?
* IO.popen / fcntl / select / read_nonblock
* File.realpath / realdirpath, and FileTest — 11 of MRI's 26 predicates, with
  the 15 that are missing named
* autoload, defined?(super), module_function, the predefined globals with MRI's
  per-name setters, trace_var
* Thread#backtrace: current thread only; another thread raises
  NotImplementedError

New "Not implemented" section. __LINE__ is always 0 — rbgo records no line map —
which costs const_source_location, warn(uplevel:) and every backtrace line
number; the section shows the rbgo/MRI output side by side. The other entries
each carry their open issue (#610 #615 #621 #630 #631 #632 #633 #635 #636),
with the two that are harness rather than VM marked as such.

Errno was NOT claimed at 158 constants: main has 28. The full table is in
#629, still open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* README: use one thousands separator throughout the new figures

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* README: re-measure against d498ddc — #629 merged after the first baseline

#629 (FileTest, the Errno subsystem, realpath_rec, the stat surface) merged at
17:10Z, ten minutes after the 68cb53a baseline this branch was written against.
Two statements were left wrong in the conservative direction. Rebased onto
d498ddc, rebuilt, re-measured everything.

ruby/spec, four consecutive runs on d498ddc: 21982, all four identical (the
68cb53a runs spread 21845-21903 on #615; it did not fire this time, but the
issue is open, so the text still says to run more than once and take the low
run). Detail sweep: 21983 pass, 1321 fail, 740 error, 473 skip, 2191 of 2206
files producing a result, 91.4% of the 24044 examples that ran.

The floor on main is still 21740; #638 is open to take it to 21970. Every place
the floor appears now says which one it is and names #638, rather than carrying
a number that goes stale when that PR merges.

Errno: was "28 constants, not MRI's 158". Measured on d498ddc, Errno.constants
is 158 under rbgo AND under MRI 4.0.5. Dumping every name/value pair from both
and joining them: all 158 names present in both, 32 disagree, and all 32 are
the same shape -- rbgo reports 0 where MRI on darwin has a real number (EAUTH
0 vs 80, EBADRPC 0 vs 72, EDEVERR 0 vs 83). The other 126 match exactly. That
is #633, and 32 is the number that matters rather than the 81 constants that
read 0 in rbgo, because MRI reads 0 for 49 of them too.

FileTest: was "11 of MRI's 26 predicates", with 15 listed as not implemented.
Measured: FileTest.methods(false) is 26 under both, name for name identical.
Differentially tested all 26 against MRI 4.0.5 over a temp tree (regular file,
empty file, directory, symlink, executable, FIFO, missing path, /dev/null,
/dev/disk0): 252 assertions, 0 disagreements.

Everything else in the limitations table re-probed on d498ddc and still holds:
Numeric#to_int NoMethodError, File::Stat#== identity ([false, true] vs MRI's
[true, true]), __LINE__ 0, const_source_location nil, Thread#backtrace of
another thread NotImplementedError, no Process.fork. All nine issues still open.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* README: the limitations section names d498ddc too

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* README: the floor IS 21,970 — #638 merged

#638 merged at 18:02Z. FLOOR on origin/main is 21970, which I re-read from the
tree. Three mentions phrased it as a pending proposal and are now simply wrong.

The arithmetic mattered more than the link. Every sentence built on a floor of
21,740 has been recomputed: the margin is 12, not ~240, and it appeared as
"~240 above the floor" in two places and "the gap is ~240 examples" in a third.

Rewrote the floor-vs-score explanation rather than swapping the number into it.
The old wording said the measured total "normally sits above" the floor and the
floor is "raised deliberately after a win", which reads as the floor lagging out
of laziness and leaves a reader wondering why 21,982 and 21,970 are nearly
equal. #638 states the actual rule, and its own arithmetic confirms it -- its
lowest run was 21981 against a floor of 21970, eleven below:

  the floor is set a handful of examples below the LOWEST observed run --
  enough to absorb the #615 jitter, and no more, because a floor slack enough
  to hide the loss of a whole spec file would defeat what the ratchet exists
  to catch.

So a small gap is the ratchet working; a large one would mean it had gone slack.
My own four runs give a margin of 12 against the same rule.

Provenance: the measurement was taken on d498ddc. main is now 2200e17, whose
only change is the FLOOR file, so I rebuilt and compared -- the binaries are
byte-identical (sha256 32cd4f9e...), and the figures carry over rather than
being assumed to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tannevaled