Repository navigation
docs: refresh ruby/spec conformance state; bump ratchet floor to 15400 - #451
Merged
Merged
Conversation
The ruby/spec language + core ratchet floor was 14850, measured when the passing
total was ~15183; the suites now pass 15459-15461 examples (three consecutive
local runs), so the floor was stale by ~600. Bump it to 15400 — comfortably
below the measured total (leaving margin for the known random/bytes and require
flakes and for cross-runner variance) — to lock in the accumulated conformance
wins as a hard floor.
Docs:
- README gains a "ruby/spec ratchet" subsection documenting the shrink-only
language + core lane and how to run/update it.
- docs/plan-rbgo.md Phase 8 records the recent core-method batches (dump/undump
/hex/append_as_bytes, the pack/unpack u and w directives, numerator/
denominator) and re-frames "still ahead" around the real remaining mass: the
deep subsystems (IO/File/Dir/ARGF/Time/Process/ENV/Marshal), the cross-repo
engines, and front-end source positions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Sep 23, 2026
The ruby/spec figure said 15,400+, which was the FLOOR bumped on 2026-08-25 (#451). Measured on main (68cb53a) against the pinned corpus today: 21,901 passing, four runs 21845/21903/21901/21902, floor 21,740. Also corrected against the code and .github/workflows/ci.yml: * the 100% coverage claim ("across all six 64-bit targets and three OSes"): the gate runs on ubuntu + macOS only, per-PR arch lanes are amd64/arm64 without coverage, the exotic four are nightly QEMU + scheduled hardware * the bound-module count, 181 -> 178 modules across 176 orgs, with the seven bound outside the go-ruby-* convention named * Set (in-tree over object.Hash, not go-composites/set), BigDecimal (go-ruby-bigdecimal, not go-composites/bigfloat) and Date (go-ruby-date, not go-composites/date) * the phase badge: plan-rbgo.md marks Phase 6 "substantially landed", not done Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Sep 23, 2026
The ruby/spec figure said 15,400+, which was the FLOOR bumped on 2026-08-25 (#451). Measured on main (68cb53a) against the pinned corpus today: 21,901 passing, four runs 21845/21903/21901/21902, floor 21,740. Also corrected against the code and .github/workflows/ci.yml: * the 100% coverage claim ("across all six 64-bit targets and three OSes"): the gate runs on ubuntu + macOS only, per-PR arch lanes are amd64/arm64 without coverage, the exotic four are nightly QEMU + scheduled hardware * the bound-module count, 181 -> 178 modules across 176 orgs, with the seven bound outside the go-ruby-* convention named * Set (in-tree over object.Hash, not go-composites/set), BigDecimal (go-ruby-bigdecimal, not go-composites/bigfloat) and Date (go-ruby-date, not go-composites/date) * the phase badge: plan-rbgo.md marks Phase 6 "substantially landed", not done Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Sep 23, 2026
…), and the claims that had drifted (#637) * README: measured ruby/spec conformance, and the claims that had drifted The ruby/spec figure said 15,400+, which was the FLOOR bumped on 2026-08-25 (#451). Measured on main (68cb53a) against the pinned corpus today: 21,901 passing, four runs 21845/21903/21901/21902, floor 21,740. Also corrected against the code and .github/workflows/ci.yml: * the 100% coverage claim ("across all six 64-bit targets and three OSes"): the gate runs on ubuntu + macOS only, per-PR arch lanes are amd64/arm64 without coverage, the exotic four are nightly QEMU + scheduled hardware * the bound-module count, 181 -> 178 modules across 176 orgs, with the seven bound outside the go-ruby-* convention named * Set (in-tree over object.Hash, not go-composites/set), BigDecimal (go-ruby-bigdecimal, not go-composites/bigfloat) and Date (go-ruby-date, not go-composites/date) * the phase badge: plan-rbgo.md marks Phase 6 "substantially landed", not done Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * README: the subsystems that landed since the feature list was written, and a plain limitations section Verified item by item against main (68cb53a) with a probe run through the built binary, MRI 4.0.5 on the same host as the oracle: * Process (spawn/exec, the wait family, process groups, priorities, rlimits) — and that Process.fork does NOT exist * Fiber#raise / #kill / storage / blocking? * IO.popen / fcntl / select / read_nonblock * File.realpath / realdirpath, and FileTest — 11 of MRI's 26 predicates, with the 15 that are missing named * autoload, defined?(super), module_function, the predefined globals with MRI's per-name setters, trace_var * Thread#backtrace: current thread only; another thread raises NotImplementedError New "Not implemented" section. __LINE__ is always 0 — rbgo records no line map — which costs const_source_location, warn(uplevel:) and every backtrace line number; the section shows the rbgo/MRI output side by side. The other entries each carry their open issue (#610 #615 #621 #630 #631 #632 #633 #635 #636), with the two that are harness rather than VM marked as such. Errno was NOT claimed at 158 constants: main has 28. The full table is in #629, still open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * README: use one thousands separator throughout the new figures Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * README: re-measure against d498ddc — #629 merged after the first baseline #629 (FileTest, the Errno subsystem, realpath_rec, the stat surface) merged at 17:10Z, ten minutes after the 68cb53a baseline this branch was written against. Two statements were left wrong in the conservative direction. Rebased onto d498ddc, rebuilt, re-measured everything. ruby/spec, four consecutive runs on d498ddc: 21982, all four identical (the 68cb53a runs spread 21845-21903 on #615; it did not fire this time, but the issue is open, so the text still says to run more than once and take the low run). Detail sweep: 21983 pass, 1321 fail, 740 error, 473 skip, 2191 of 2206 files producing a result, 91.4% of the 24044 examples that ran. The floor on main is still 21740; #638 is open to take it to 21970. Every place the floor appears now says which one it is and names #638, rather than carrying a number that goes stale when that PR merges. Errno: was "28 constants, not MRI's 158". Measured on d498ddc, Errno.constants is 158 under rbgo AND under MRI 4.0.5. Dumping every name/value pair from both and joining them: all 158 names present in both, 32 disagree, and all 32 are the same shape -- rbgo reports 0 where MRI on darwin has a real number (EAUTH 0 vs 80, EBADRPC 0 vs 72, EDEVERR 0 vs 83). The other 126 match exactly. That is #633, and 32 is the number that matters rather than the 81 constants that read 0 in rbgo, because MRI reads 0 for 49 of them too. FileTest: was "11 of MRI's 26 predicates", with 15 listed as not implemented. Measured: FileTest.methods(false) is 26 under both, name for name identical. Differentially tested all 26 against MRI 4.0.5 over a temp tree (regular file, empty file, directory, symlink, executable, FIFO, missing path, /dev/null, /dev/disk0): 252 assertions, 0 disagreements. Everything else in the limitations table re-probed on d498ddc and still holds: Numeric#to_int NoMethodError, File::Stat#== identity ([false, true] vs MRI's [true, true]), __LINE__ 0, const_source_location nil, Thread#backtrace of another thread NotImplementedError, no Process.fork. All nine issues still open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * README: the limitations section names d498ddc too Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * README: the floor IS 21,970 — #638 merged #638 merged at 18:02Z. FLOOR on origin/main is 21970, which I re-read from the tree. Three mentions phrased it as a pending proposal and are now simply wrong. The arithmetic mattered more than the link. Every sentence built on a floor of 21,740 has been recomputed: the margin is 12, not ~240, and it appeared as "~240 above the floor" in two places and "the gap is ~240 examples" in a third. Rewrote the floor-vs-score explanation rather than swapping the number into it. The old wording said the measured total "normally sits above" the floor and the floor is "raised deliberately after a win", which reads as the floor lagging out of laziness and leaves a reader wondering why 21,982 and 21,970 are nearly equal. #638 states the actual rule, and its own arithmetic confirms it -- its lowest run was 21981 against a floor of 21970, eleven below: the floor is set a handful of examples below the LOWEST observed run -- enough to absorb the #615 jitter, and no more, because a floor slack enough to hide the loss of a whole spec file would defeat what the ratchet exists to catch. So a small gap is the ratchet working; a large one would mean it had gone slack. My own four runs give a margin of 12 against the same rule. Provenance: the measurement was taken on d498ddc. main is now 2200e17, whose only change is the FLOOR file, so I rebuilt and compared -- the binaries are byte-identical (sha256 32cd4f9e...), and the figures carry over rather than being assumed to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ruby/spec language + core ratchet floor was
14850, measured when thepassing total was ~15183; the suites now pass 15459–15461 examples (three
consecutive local runs), so the floor was stale by ~600. This bumps it to
15400 — comfortably below the measured total (margin for the known
random/bytes/requireflakes and cross-runner variance) — to lock in theaccumulated conformance wins as a hard floor.
Docs
core lane, and how to run/update it.
docs/plan-rbgo.mdPhase 8 records the recent core-method batches(
dump/undump/hex/append_as_bytes, thepack/unpackuandwdirectives,
numerator/denominator) and re-frames still ahead around thereal remaining mass — deep subsystems (IO/File/Dir/ARGF/Time/Process/ENV/
Marshal), cross-repo engines, and front-end source positions.
No code change; the ratchet lane validates the new floor (local run: 15461 ≥
15400, OK).
🤖 Generated with Claude Code