[tannevaled]

The ruby/spec section described the ratchet mechanism carefully and carried no number at all — no pass count, no floor value, nothing a reader could compare a run against. Meanwhile the CI-enforcement claim overstated what CI enforces, and the Rails/Puppet figures appeared twice, undated, disagreeing with themselves about Puppet's file count.

Every figure below was measured in this session against rbgo main (68cb53a).

What I measured

scripts/conformance/rubyspec/run.sh, darwin/arm64, pinned corpus SPEC_SHA=87b1631992bd00cf0c4934474766d54dad088191, four consecutive runs:

21845 · 21903 · 21901 · 21902      FLOOR: 21740

A second sweep captured the whole RBGO_RESULT line rather than just pass=, for a real denominator: 21 901 pass, 1 349 fail, 794 error, 473 skip; 2 191 of 2 206 files produce a result.

docs/conformance.md

New — "Where it stands (measured 2026-09-23)". The table above, plus an admonition that says which number is which:

FLOOR says "no run may come in below this". It is raised deliberately, in its own PR, after a wave lands — so the measured total normally sits above it. Today the gap is ~160 examples. Quoting the floor as the conformance figure understates rbgo; quoting the measured total as a guarantee overstates it.

The 58-example spread across those four runs is one file — core/module/autoload_spec.rb crashing while popping a frame, #615, roughly one run in four. The page says to read the low run as the guaranteed figure.

It also says, plainly, that there is no honest denominator for "percent of Ruby": 91.1 % is the share of the examples this shim actually ran; 473 skips and 15 unreadable files sit outside the ratio.

New — "Not every gain is VM conformance". Two recent jumps were gains in what the measurement can see:

• the mspec shim (#624, refs #621) — core/string/valid_encoding/utf_8_spec.rb scored 0/28, and 0/28 under MRI 4.0.5 through the same shim; corpus 20 905 → 20 936, 28 attributable;
• parser v0.2.0 (#625) — 13 language/*_spec.rb files that had never parsed at all; language/ 1 443 → 1 776 passing, with fail+error rising 283 → 436 at the same time.

The rest of the climb is the VM: floor 6 000 on 2026-08-03 (#263) → 21 740 today, across 26 waves.

New — "Known limitations". Led by the widest one, shown as measured output: __LINE__ is always 0 — rbgo records no line map — which costs const_source_location (nil vs MRI's []), warn(uplevel:) (no file:line: prefix) and every backtrace line number. Then a table carrying #610, #615, #631, #632, #633, #635, #636, plus Errno at 28 constants (not 158), FileTest at 11 of MRI's 26 predicates, Thread#backtrace current-thread-only, no Process.fork, and BEGIN {}/END {} not parsing.

#630 and #621 are listed apart and marked as harness, not VM — #630's 11 realpath failures reproduce under MRI 4.0.5 through the same shim.

Corrected — "What CI enforces". It claimed 100 % coverage "across all six 64-bit architectures" and "three OSes". Per ci.yml: the gate runs on ubuntu + macOS only (windows runs the full -race suite but is not gated — POSIX-only paths unreachable); per-PR arch cover is the native amd64/arm64 lanes running go test ./..., no coverage; the exotic four are nightly QEMU + scheduled real hardware; and there is a wasip1/wasm lane the page did not mention.

Corrected — Rails & Puppet. Now dated: measured 2026-06-27 against clones taken 2026-06-25, not re-measured since, naming the script that reproduces it. Puppet was stated as both 2 156 and 2 154 / 2 154 on the same page; reconciled. And a duplicate section restating the same table was removed.

"Run it yourself" now shows how to run the ratchet against an existing corpus, and says to run it more than once because of #615.

docs/index.md

• the same 100 %/six-arch claim, corrected the same way;
• "binds 181 of them" → 178 modules across 176 orgs (grep -oE 'github\.com/go-ruby-[a-z0-9-]*/[a-z0-9-]*' go.mod | sort -u | wc -l);
• seven links pointed at orgs that do not exist. commonmark, kramdown, liquid, mustache, nokogiri, rouge and xslt are bound from github.com/go-<name>/<name>, not go-ruby-<name>/<name>;
• set is not bound at all — Set is implemented in-tree over rbgo's own object.Hash (internal/vm/set.go). The entry was removed and the fact stated;
• activeldap, ldap and fast-gettext-locale were missing from the list.

docs/roadmap.md

Phases 2 and 5 read "In progress" where the interpreter's own docs/plan-rbgo.md marks both ✅ DONE (core); phase 6 is "🚧 substantially landed" there. Phase 8's entry carried only the perf work and the front-end sweep — it now carries the runtime-conformance campaign with the measured numbers.

docs/phases/phase0.md

"~92% test coverage" sat unqualified against a site-wide 100 % gate; it is a Phase 0 exit criterion and is now scoped to when the phase closed. "No blocks (Phase 3)" → Phase 1, which is where the roadmap puts them.

Left alone deliberately

• docs/benchmarks.md entirely. Re-running it needs MRI+YJIT, JRuby and TruffleRuby and a measurement not taken under load. Two things there are worth a separate look by someone who can re-measure: the AOT speedup is stated as both "~4× faster than MRI + YJIT" and "18–24×" in different places, and TruffleRuby appears as 34.0.1 in the table headers but 3.4.9 in the prose at line 97.
• The Rails/Puppet percentages themselves. Reproducing them needs a Rails + Puppet clone and an MRI oracle sweep; rather than restate them as current I dated them.
• docs/phases/ holds only phase0.md while the roadmap describes nine phases. That is a gap, not a wrong claim, and filling it is more than a correction pass.

No merge. No screenshots.

---

Update — re-measured against d498ddc (2026-09-23 19:2x CEST)

main moved under this branch. #629 — FileTest, the Errno subsystem, realpath_rec and the stat surface — merged at 17:10Z, ten minutes after the 68cb53a baseline the text above was written against. I was explicit in this PR that I was documenting main rather than an unmerged PR, so the branch has been rebased onto d498ddc, rebuilt, and everything re-measured. Two statements were wrong, both in the conservative direction.

Errno — was "28 constants, not MRI's 158"

Errno.constants.size is now 158 under rbgo, and 158 under MRI 4.0.5 on the same host. I dumped every name/value pair from both and joined them:

names present in both      158
disagree                    32
  rbgo 0, MRI non-zero      32      EAUTH 0 vs 80 · EBADRPC 0 vs 72 · EDEVERR 0 vs 83 · …
  any other kind of         0

So the table is complete and the class-per-number identity holds; 32 constants report errno 0 on darwin where MRI has a real number, because Go's portable syscall package does not expose the platform-only names. That is #633, still open.

One measurement worth keeping: 81 of rbgo's constants read 0 — but 49 of MRI's do too, on the same darwin host, because its libc does not define those names either. So the honest figure is the 32 that disagree, not the 81 that read zero. The issue title's "82" counts the former shape; I have published the disagreement count instead.

FileTest — was "11 of MRI's 26 predicates", with 15 listed as missing

FileTest.methods(false) is 26 under both, and the two lists are identical name for name. I then differential-tested all 26 against MRI 4.0.5 over a temp tree — regular file, empty file, directory, symlink, executable, FIFO, missing path, /dev/null, /dev/disk0:

252 assertions, 0 disagreements

The "not implemented" sentence is gone; the limitations row is gone.

ruby/spec — re-measured

Four consecutive runs on d498ddc, pinned corpus, darwin/arm64:

21982 · 21982 · 21982 · 21982      FLOOR on main: 21740

All four identical. The detail sweep reads 21 983 pass / 1 321 fail / 740 error / 473 skip — 91.4 % of the 24 044 examples that ran; 2 191 of 2 206 files produce a result. (The one-example gap between 21 982 and 21 983 is the #615 noise.)

Note on the spread: the earlier four runs on 68cb53a ranged 21 845–21 903, which I attributed to #615. These four did not hit it. The issue is open and the rate is roughly one in four, so the text still says to run more than once and read the low run as guaranteed — I have not claimed it is fixed.

The floor

FLOOR on main is 21 740. #638 is open to raise it to 21 970 and has not merged as of this writing. Rather than write a number that goes stale within the hour, every place the floor appears now says which floor it means and links #638.

Re-probed on d498ddc and still true

Numeric#to_int → NoMethodError (#631) · File::Stat#== → [false, true] against MRI's [true, true] (#632) · __LINE__ → 0 · const_source_location → nil · Thread#backtrace of another thread → NotImplementedError · no Process.fork · SPEC_TMP_BASE still /tmp (#630). All nine issues remain open.

---

Update 2 — the floor is 21,970 (#638 merged 18:02Z)

I re-read FLOOR from origin/main: 21970. The text above described it as a pending proposal, which is now simply wrong. Every mention states it as current; #638 survives in one place per document, as provenance ("raised there after wave 27"), and nowhere as an open PR.

The arithmetic mattered more than the link

Three sentences were built on a floor of 21,740 and would have quietly kept the old margin:

where	was	now
the floor-vs-score paragraph	"today the gap is ~240 examples"	margin of 12
the ## Testing & conformance ratchet section	"~240 above the floor"	margin of 12
the floor-climb sentences (×2)	"6,000 → 21,740 across 27 waves"	6,000 → 21,970

21,982 − 21,970 = 12, not 242.

The new gap needed an explanation, not just a number

A reader meeting "21,982 passing, floor 21,970" will reasonably ask why they are nearly equal — and my old wording made it worse, saying the measured total "normally sits above" the floor and the floor is "raised deliberately after a win". That reads as the floor lagging out of laziness, which is not what is happening.

#638 states the actual rule, and its own numbers confirm it — its lowest run was 21981 against a floor of 21970, eleven below:

The floor is set eleven below the lowest run — the same margin rule as the last two bumps: enough for the run-to-run jitter, not enough to hide the loss of a whole spec file, which is what the ratchet exists to catch.

So the explanation is now:

The floor is set a handful of examples below the lowest observed run — enough to absorb the run-to-run jitter of #615, and no more, because a floor slack enough to hide the loss of a whole spec file would defeat the thing the ratchet exists to catch. A small gap is the ratchet working; a large one would mean it had gone slack.

They are still different claims, and the text says so: the floor is the guarantee CI enforces, the measured total is what rbgo does on a given run — quoting either as the other is wrong by exactly the margin. My own four runs give a margin of 12 under the same rule.

Provenance — and why the figures did not need re-measuring

The measurement was taken on d498ddc. main is now 2200e17. Rather than assume that was harmless, I checked: #638 touched only scripts/conformance/rubyspec/FLOOR, which is a data file the runner reads, not something compiled in. I rebased, rebuilt, and compared:

d498ddc binary  sha256 32cd4f9ec68cc8bb…
2200e17 binary  sha256 32cd4f9ec68cc8bb…   identical

So 21,982 carries over unchanged, and each page now says so rather than quietly citing an older commit.

🤖 Generated with Claude Code