Skip to content

conformance: the measured ruby/spec numbers (21,982 vs floor 21,740), and the claims that had drifted #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tannevaled merged 3 commits into from
Sep 23, 2026
Merged

conformance: the measured ruby/spec numbers (21,982 vs floor 21,740), and the claims that had drifted #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
662aa0c
conformance: the measured ruby/spec numbers, and the claims that had …
tannevaled Sep 23, 2026
8cf4990
conformance: re-measure against d498ddc — #629 merged after the first…
tannevaled Sep 23, 2026
21f0a6b
conformance: the floor IS 21,970 — #638 merged
tannevaled Sep 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 139 additions & 23 deletions docs/conformance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,21 @@ scripts/oracle.sh -b scripts/conformance/core_ext.txt

On top of the oracle, the interpreter holds a hard CI gate:

- **100% statement coverage**, enforced as a CI gate;
- across **all six 64-bit architectures** — amd64, arm64, riscv64, loong64,
ppc64le, s390x;
- across **three OSes**;
- **100% statement coverage**, gated on the two **POSIX** lanes — ubuntu and
macOS. The full `-race` suite also runs on **windows**, but the gate is not
applied there: a handful of POSIX-only paths (opening `/dev/null` as a
character device) are unreachable, so its measured coverage is structurally
below 100 %;
- **per-PR architecture cover is the two native lanes**, `amd64` and `arm64`,
which run `go test ./...` — not the coverage gate. The four exotic 64-bit
targets (riscv64, loong64, ppc64le, s390x) are validated **off** the per-PR
critical path: nightly under QEMU and on scheduled real hardware (below);
- a **`wasip1/wasm`** lane builds the interpreter and runs it under wazero;
- with **every feature differential-tested against MRI 4.0.5**.

[`ci.yml`](https://github.com/go-embedded-ruby/ruby/blob/main/.github/workflows/ci.yml)
is the authority on which lane runs what.

A feature does not count as landed until it agrees with the reference and the
lines that implement it are covered.

Expand Down Expand Up @@ -98,6 +107,114 @@ floor is a deliberately **conservative lower bound**. The
[`rubyspec-ratchet`](https://github.com/go-embedded-ruby/ruby/blob/main/.github/workflows/rubyspec-ratchet.yml)
workflow enforces it on every push and pull request.

### Where it stands — measured 2026-09-23

On darwin/arm64 against the pinned corpus
(`SPEC_SHA=87b1631992bd00cf0c4934474766d54dad088191`), on `d498ddc` — the tip of
`main` at the time. `main` has since advanced to `2200e17`, whose only change is
the `FLOOR` file ([#638](https://github.com/go-embedded-ruby/ruby/pull/638)); the
interpreter built from it is byte-identical, so these figures stand unchanged.

| | |
| --- | --- |
| **passing examples** | **21 982** — four consecutive runs, all four identical |
| fail / error | 1 321 / 740 |
| skipped | 473 |
| pass rate of the examples that ran | **91.4 %** (21 983 of 24 044) |
| spec files | 2 191 of 2 206 produce a result; 15 produce none |
| **`FLOOR`** | **21 970** |

The ratchet reports 21 982; a separate sweep that also captures
`fail`/`error`/`skip` read 21 983. The one-example difference is the `#615` noise
described below.

!!! warning "The floor is a gate, not a score — and they are close on purpose"
`FLOOR` says *"no run may come in below this"*. It is raised in its own PR
after a wave lands. The measured total sits **above** it, but only just:
21 982 against 21 970 is a margin of **12**, and that is deliberate.

The floor is set a handful of examples below the **lowest observed run** —
enough to absorb the run-to-run jitter of
[#615](https://github.com/go-embedded-ruby/ruby/issues/615), and no more,
because a floor slack enough to hide the loss of a whole spec file would
defeat the thing the ratchet exists to catch. A small gap is therefore the
ratchet working; a large one would mean it had gone slack.

They remain different claims. The floor is the **guarantee CI enforces**; the
measured total is **what rbgo does** on a given run. Quoting the floor as the
conformance figure understates rbgo by the margin; quoting the measured total
as a guarantee overstates it by the same amount.

Run-to-run spread is possible, and it is a **known bug rather than noise in the
method**. `core/module/autoload_spec.rb` crashes intermittently while popping a
frame ([#615](https://github.com/go-embedded-ruby/ruby/issues/615)), and the whole
file's examples are lost when it does — an earlier set of four runs, on the commit
before this one, spread 21 845–21 903 for exactly that reason. The four runs above
happened not to hit it. Run it more than once and read the **low** run as the
guaranteed figure.

There is no honest denominator for "percent of Ruby". The 91.4 % above is the
share of the examples *this shim actually ran*: the shim is not mspec, 473 skips
sit outside the ratio entirely, and 15 files produce no result at all.

### Not every gain is VM conformance

Two of the recent jumps came from fixing the **measurement**, not the
interpreter, and blurring the two would be misleading:

- **The mspec shim** ([#624](https://github.com/go-embedded-ruby/ruby/pull/624),
refs [#621](https://github.com/go-embedded-ruby/ruby/issues/621)) ran each
example under `instance_eval`, so a helper defined with `def` in an outer
`describe` was unreachable from a nested one.
`core/string/valid_encoding/utf_8_spec.rb` scored **0 of 28 — and 0 of 28 under
MRI 4.0.5 as well**, run through the same shim. Fixing the shim moved the
corpus 20 905 → 20 936, of which **28 are attributable**.
- **The parser upgrade to v0.2.0**
([#625](https://github.com/go-embedded-ruby/ruby/pull/625)) made **13
`language/*_spec.rb` files parseable that had never parsed at all**. They had
been contributing zero to *both* columns — not failing, invisible. `language/`
went 1 443 → 1 776 passing (**+333**), while `fail+error` rose 283 → 436 at the
same time, because those files brought their own failures with them.

Both are gains in what the measurement can **see**, not in what the VM can
**do**. The rest of the climb is the VM: the floor went from **6 000** when the
ratchet landed on 2026-08-03
([#263](https://github.com/go-embedded-ruby/ruby/pull/263)) to **21 970** today,
across 27 conformance waves.

### Known limitations

Named plainly, each with an open issue. **`__LINE__` is always `0`** — rbgo
records no line map — which costs `const_source_location`, `warn(uplevel:)` and
every backtrace line number entirely:

```ruby
puts __LINE__ # rbgo: 0 MRI 4.0.5: 1
Object.const_source_location(:Comparable) # rbgo: nil MRI 4.0.5: []
# e.backtrace.first -> "probe.rb:0:in '<main>'" against MRI's "probe.rb:4:…"
```

| | |
| --- | --- |
| `Errno` carries all **158** of MRI's constants, but **32** of them report errno `0` on darwin where MRI has a real number (`EAUTH` 0 vs 80, `EBADRPC` 0 vs 72, …); the other 126 match exactly | [#633](https://github.com/go-embedded-ruby/ruby/issues/633) |
| `Thread#backtrace` answers for the **current** thread only; another thread raises `NotImplementedError` | — |
| `Process.fork` **does not exist** — Go's runtime cannot be forked safely | — |
| `Numeric#to_int` is not defined | [#631](https://github.com/go-embedded-ruby/ruby/issues/631) |
| `File::Stat#==` answers identity, not `Comparable#==` | [#632](https://github.com/go-embedded-ruby/ruby/issues/632) |
| `File#stat` cannot answer for a file unlinked while open | [#635](https://github.com/go-embedded-ruby/ruby/issues/635) |
| `BEGIN { }` / `END { }` do not parse | front-end |
| **Windows:** no text-mode newline translation | [#610](https://github.com/go-embedded-ruby/ruby/issues/610) |
| **Windows:** `File::Stat#atime`/`#ctime` fall back to mtime | [#635](https://github.com/go-embedded-ruby/ruby/issues/635) |
| **Windows:** `File.realpath` does not expand 8.3 short names | [#636](https://github.com/go-embedded-ruby/ruby/issues/636) |

Two more are **harness**, not interpreter, and are listed apart so they are not
read as VM gaps: the shim's `SPEC_TMP_BASE` is `/tmp`, a symlink on darwin, which
fails 11 `realpath`/`realdirpath` examples on paths that are correct
([#630](https://github.com/go-embedded-ruby/ruby/issues/630)) — **MRI 4.0.5 fails
them the same way through the same shim** — and the `instance_eval` rebinding
above ([#621](https://github.com/go-embedded-ruby/ruby/issues/621)), largely fixed
but not closed.

## Real-world corpora

Synthetic tests prove a feature works in isolation; they do not prove the idioms
Expand All @@ -123,12 +240,15 @@ considers valid? A released gem / framework file is valid Ruby (`ruby -c` clean)
so a rejection is a genuine front-end gap.

After the 2026-06 conformance campaign (5 go-ruby-parser rounds interleaved with
5 rbgo activation rounds), the final figures against the MRI 4.0.5 oracle:
5 rbgo activation rounds), the final figures against the MRI 4.0.5 oracle.
**Measured 2026-06-27 against clones of `rails/rails` and `puppetlabs/puppet`
taken on 2026-06-25; not re-measured since** — reproduce with
`scripts/conformance/heavyweight/sweep.sh`:

| Repo | `.rb` files | rbgo **parses** | rbgo **parses + compiles** |
|--------|------------:|----------------:|---------------------------:|
| Rails | 3 423 | **100.00 %** | **99.82 %** |
| Puppet | 2 156 | **100.00 %** (of valid) | **100.00 %** |
| Puppet | 2 156 | **100.00 %** (of valid: 2 154) | **100.00 %** (2 154 / 2 154) |
| **Total** | 5 579 | **99.96 %** | **99.89 %** |

- **Parse: 99.96 %** of the combined corpus (5577 / 5579). The only 2 misses are
Expand Down Expand Up @@ -173,23 +293,6 @@ suite:
whole applications on the runtime stdlib + C-extension surface is the next,
ongoing rung.

## Heavyweight front-end sweep: Rails & Puppet

The two largest reference Ruby codebases — **Rails** and **Puppet** — are run
through the pure-Go front-end (`parser.Parse` → `compiler.Compile`, no execution)
as a conformance stress test, with **MRI `ruby -c`** as the oracle for "is this
valid Ruby?".

| Repo | Parse | Parse + compile |
| ------ | ----: | --------------: |
| Rails | 100.00 % (3 423 / 3 423) | **99.82 %** (3 417 / 3 423) |
| Puppet | 100.00 % (2 154 / 2 154) | **100.00 %** (2 154 / 2 154) |

Across the corpus rbgo parses **100 % of all valid Ruby** (the only 2 misses are
intentional syntax-error fixtures MRI itself rejects), with **0 over-permissive**
(rbgo never accepts Ruby MRI rejects). See
[`CONFORMANCE-RAILS-PUPPET.md`](https://github.com/go-embedded-ruby/ruby/blob/main/CONFORMANCE-RAILS-PUPPET.md)
in the interpreter repo for the full report and the reproducible sweep.

These are **front-end (parse + compile) acceptance** figures, not whole-app
execution. Running a real application additionally needs the runtime stdlib
Expand Down Expand Up @@ -256,3 +359,16 @@ scripts/oracle.sh -b scripts/conformance/core_ext.txt
```

Either invocation exits non-zero if any implementation disagrees.

The ruby/spec ratchet, against a corpus you already have (it clones one at
`SPEC_SHA` otherwise):

```sh
CGO_ENABLED=0 SPECDIR=/path/to/ruby-spec CACHE=/path/to/ruby-spec \
scripts/conformance/rubyspec/run.sh
```

It prints the measured total and the frozen floor. Run it **more than once**:
`core/module/autoload_spec.rb` crashes intermittently
([#615](https://github.com/go-embedded-ruby/ruby/issues/615)) and a single run can
read ~57 examples low.
Loading
Loading