From 1e34e85d9aa2e7030f34b4e96a49fdda24fb4964 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sat, 26 Sep 2026 14:42:14 +0200 Subject: [PATCH] landing: the measured conformance and AOT figures, and an honest Go API card Re-measured on 3e8e3cc (darwin/arm64) against MRI 4.0.5 on the same host. - "rbgo+AOT beats MRI+YJIT 18-24x on loop/fib" does not hold, by two independent readings. The repository's own BENCHMARKS.md table gives AOT 10 ms vs YJIT 360 ms on loop (36x) and AOT 20 ms vs YJIT 100 ms on fib (5x) -- neither is 18-24x, so the figure does not follow from the numbers it cites. Re-running the project's own bench/loop.rb and bench/fib.rb here with hyperfine: AOT 43 ms vs YJIT 373 ms on loop (8.7x) and AOT 56 ms vs YJIT 110 ms on fib(35) (2.0x). My MRI and MRI+YJIT timings match the published table almost exactly (495/110 ms against its 480-500/ 100-110 ms), so the harnesses are comparable; the AOT row is where they diverge. The site now names both measured ratios and says recursion keeps the call overhead the loop case removes. - Conformance: 21,982 / floor 21,970 -> 22,488 / floor 22,475, and the share of examples that ran rose from a stated 91.4% to a measured 93.3% (22,488 of 24,116). Added the scope that was missing everywhere: only language/ and core/ are swept, so library/, optional/ and security/ are not run and this is not a percentage of Ruby. - Dropped the run-to-run jitter narrative: three full sweeps here gave identical totals, and the spec file blamed for it returned identical counts in five runs. - "A complete Ruby, written in Go" overstates what the measurement supports; the headline is now "A Ruby you can embed". - The "Truly embeddable" card promised an object model "callable straight from Go". The public API is one function, ruby.Run(src, out); calling a Ruby method from Go is not exposed. The card now says so. - The hero terminal showed invented output for rbgo build --closed. The command does work with that argument order, so it is kept, now with the real three lines it prints -- including the 172.5 MiB binary size it had omitted. - Six-arch claim qualified: amd64 and arm64 run per pull request; riscv64, loong64, ppc64le and s390x are validated nightly, not on the pull-request path. Co-Authored-By: Claude Opus 5 --- hugo.toml | 2 +- layouts/_default/list.html | 13 +++++++------ layouts/index.html | 20 +++++++++++--------- 3 files changed, 19 insertions(+), 16 deletions(-) diff --git a/hugo.toml b/hugo.toml index a4c0927..2bd4280 100644 --- a/hugo.toml +++ b/hugo.toml @@ -40,4 +40,4 @@ disableKinds = ["taxonomy", "term", "RSS", "sitemap", "404"] [[params.phases]] name = "Phase 8 — Conformance & performance" status = "active" - blurb = "Since July the work has been **runtime** conformance, measured against [ruby/spec](https://github.com/ruby/spec)'s `language/` + `core/` suites: **21,982 examples pass** (measured 2026-09-23 on `main`, `d498ddc`, four consecutive runs all identical), gated in CI by a **shrink-only floor of 21,970** — the floor is the level CI refuses to fall below, *not* the score. It sits just under the measured total by design: a margin wide enough for the run-to-run jitter, narrow enough that losing a whole spec file still trips it. It has climbed from 6,000 when the ratchet landed on 2026-08-03, across 27 waves. Two of the recent jumps were fixes to the **measurement** rather than the interpreter — the mspec shim, and a parser upgrade that made 13 previously-unparseable spec files visible — and those are gains in what the measurement can *see*, not in what the VM can *do*. Earlier, a conformance campaign brought the pure-Go **front-end** to ~**100% of all valid Ruby** parsed across Rails + Puppet (99.96% of files; 2 misses are syntax-error fixtures MRI rejects) and **parse + compile** of **Rails 99.82% / Puppet 100%**, with **0 over-permissive** — and RuboCop 99.7%, Sinatra/Jekyll/Thor/Kramdown/dry-struct 100%, RSpec DSL 10/10. On the runtime side, that road is now underway: **the real `puppet apply` CLI runs end-to-end** on a pure-Go CGO=0 `rbgo` — `require \"puppet\"` boots Puppet 8.11.0 (all pure-Ruby gem deps on the path), and the genuine `Puppet::Util::CommandLine` → `Puppet::Application::Apply` path (real `OptionParser`, all types + providers, settings catalog applied to disk, then the user catalog via the transaction/RAL) converges `notify`, `file` and `exec` resources (the `exec` provider runs commands via pure-Go process execution with `onlyif`/`unless`/`creates`/`path` guards), emits real Puppet output (`Notice: hi from rbgo cli`) and exits 0 with the YAML run report round-tripped. Broader resource **providers** are the active next milestone (`user`/`group` provider-ready; `package`/`service` need a host package manager/systemd + root). rbgo is also increasingly composed from the standalone pure-Go **go-ruby-* family** — **178 modules across 176 orgs are now bound into rbgo as native modules** (parser, regexp, erb, marshal, yaml, format, optparse, strscan, json, date, … a selection is on the landing page), each its own 100%-coverage 6-arch org. (`Set` is not among them: it is implemented in-tree, over rbgo's own Hash.) Every one of the six arches is validated on **real hardware**, not just qemu — amd64/arm64 natively, riscv64/ppc64le/loong64 on the GCC Compile Farm (RVV, POWER8E/POWER9, LoongArch), s390x on the IBM LinuxONE Community Cloud — which is what makes the SIMD-accelerated modules and perf numbers measured rather than estimated. On performance, a 6-runtime suite (rbgo/rbgo+AOT vs MRI/YJIT/JRuby/TruffleRuby): **rbgo+AOT beats MRI+YJIT 18–24× on loop/fib**; plus inline method caches, small-integer interning (a small-int loop drops from ~245k allocations to 1) and capture-tracked frame recycling (~14% faster on fib)." + blurb = "Since July the work has been **runtime** conformance, measured against [ruby/spec](https://github.com/ruby/spec)'s `language/` + `core/` suites: **22,488 examples pass** (measured 2026-09-26 on `main`, `3e8e3cc`; three independent runs, identical), gated in CI by a **shrink-only floor of 22,475** — the floor is the level CI refuses to fall below, *not* the score. It has climbed from 6,000 when the ratchet landed on 2026-08-03. **Only `language/` and `core/` are swept** — ruby/spec's `library/`, `optional/` and `security/` trees are not run at all, so this is not a percentage of Ruby. Two of the recent jumps were fixes to the **measurement** rather than the interpreter — the mspec shim, and a parser upgrade that made 13 previously-unparseable spec files visible — and those are gains in what the measurement can *see*, not in what the VM can *do*. Earlier, a conformance campaign brought the pure-Go **front-end** to ~**100% of all valid Ruby** parsed across Rails + Puppet (99.96% of files; 2 misses are syntax-error fixtures MRI rejects) and **parse + compile** of **Rails 99.82% / Puppet 100%**, with **0 over-permissive** — and RuboCop 99.7%, Sinatra/Jekyll/Thor/Kramdown/dry-struct 100%, RSpec DSL 10/10. On the runtime side, that road is now underway: **the real `puppet apply` CLI runs end-to-end** on a pure-Go CGO=0 `rbgo` — `require \"puppet\"` boots Puppet 8.11.0 (all pure-Ruby gem deps on the path), and the genuine `Puppet::Util::CommandLine` → `Puppet::Application::Apply` path (real `OptionParser`, all types + providers, settings catalog applied to disk, then the user catalog via the transaction/RAL) converges `notify`, `file` and `exec` resources (the `exec` provider runs commands via pure-Go process execution with `onlyif`/`unless`/`creates`/`path` guards), emits real Puppet output (`Notice: hi from rbgo cli`) and exits 0 with the YAML run report round-tripped. Broader resource **providers** are the active next milestone (`user`/`group` provider-ready; `package`/`service` need a host package manager/systemd + root). rbgo is also increasingly composed from the standalone pure-Go **go-ruby-* family** — **178 modules across 176 orgs are now bound into rbgo as native modules** (parser, regexp, erb, marshal, yaml, format, optparse, strscan, json, date, … a selection is on the landing page), each its own 100%-coverage 6-arch org. (`Set` is not among them: it is implemented in-tree, over rbgo's own Hash.) Of the six arches, amd64 and arm64 are tested on every pull request; riscv64, loong64, ppc64le and s390x are validated on a **nightly schedule** — under qemu, and natively on the GCC Compile Farm plus an IBM LinuxONE s390x host — not on the pull-request path. On performance, a 6-runtime suite (rbgo/rbgo+AOT vs MRI/YJIT/JRuby/TruffleRuby): **rbgo+AOT beats MRI+YJIT on integer-bound kernels — measured here at 8.7× on the tight-loop benchmark and 2.0× on recursive `fib(35)`** (darwin/arm64, whole-process wall time); plus inline method caches, small-integer interning (a small-int loop drops from ~245k allocations to 1) and capture-tracked frame recycling (~14% faster on fib)." diff --git a/layouts/_default/list.html b/layouts/_default/list.html index c5473bc..5d039f4 100644 --- a/layouts/_default/list.html +++ b/layouts/_default/list.html @@ -54,9 +54,9 @@

A · Per-module benchmarks

B · Runtime micro-benchmarks

-

rbgo's ahead-of-time backend compiles hot methods to unboxed Go kernels. Against MRI with YJIT enabled, on tight loop / fib kernels:

+

rbgo's ahead-of-time backend compiles integer-bound methods to unboxed Go kernels. Measured on this project's own micro-benchmarks (bench/loop.rb, bench/fib.rb) on darwin/arm64, best-of-N whole-process wall time, against MRI 4.0.5 with YJIT enabled:

-
18–24×

AOT vs MRI+YJIT

rbgo+AOT beats MRI with YJIT enabled by 18–24× on loop / fib — unboxed int64 kernels, no boxing on the hot path.

+
8.7×

AOT vs MRI+YJIT

On the tight-loop benchmark, rbgo+AOT ran 8.7× faster than MRI+YJIT (43 ms vs 373 ms). On recursive fib(35) the margin is 2.0× (56 ms vs 110 ms) — recursion keeps the call overhead the loop case removes. Only methods the AOT backend can lower benefit at all.

245k → 1

Small-integer interning

Interned small integers drop a small-int loop from ~245,000 allocations to 1 — the hot integers are shared, immutable singletons.

~14%

Frame recycling

Capture-tracked frame recycling is ≈14% faster on fib; frames are reused unless a closure captures them. Plus inline method caches on every call site.

@@ -68,11 +68,12 @@

C · Application conformance & performance

Runtime conformance — ruby/spec

-

21,982 examples pass in ruby/spec's language/ + core/ suites, measured 2026-09-23 on main (d498ddc), four consecutive runs all identical.

+

22,488 examples pass in ruby/spec's language/ + core/ suites, measured 2026-09-26 on main (3e8e3cc). Three independent runs — two through the ratchet at different timeouts, one through a separate sweep — reported the same total.

    -
  • CI gates on a shrink-only floor of 21,970, raised there after wave 27. The floor is the level a run may not fall below — it is not the score. It sits just under the measured total on purpose: a margin of 12, wide enough to absorb the run-to-run jitter of #615 and no wider, because a floor slack enough to hide the loss of a whole spec file would defeat what the ratchet exists to catch.
    • -
  • Of the examples that ran, 91.4% pass (21,983 of 24,044); 473 more are skipped and 15 of 2,206 spec files produce no result.
    • -
  • The floor has climbed from 6,000 on 2026-08-03 to 21,970 across 27 waves. Two recent jumps were fixes to the measurement — the mspec shim, and a parser upgrade that made 13 previously-unparseable spec files visible — which are gains in what the measurement can see, not in what the VM can do.
    • +
  • CI gates on a shrink-only floor of 22,475. The floor is the level a run may not fall below — it is not the score: it is what CI guarantees, while the measured total is what one run did.
    • +
  • Only language/ and core/ are swept. ruby/spec's library/, optional/ and security/ trees are not run at all, so this figure is not a percentage of Ruby, and most of the standard library sits outside it.
    • +
  • Of the examples that ran, 93.3% pass (22,488 of 24,116); 477 more are skipped because our shim stubs the matcher they need, and 10 of 2,206 spec files produce no result. Skips and unreadable files sit outside the ratio entirely.
    • +
  • The floor has climbed from 6,000 on 2026-08-03 to 22,475. Two earlier jumps were fixes to the measurement — the mspec shim, and a parser upgrade that made 13 previously-unparseable spec files visible — which are gains in what the measurement can see, not in what the VM can do.
diff --git a/layouts/index.html b/layouts/index.html index 2bfb272..112b07d 100644 --- a/layouts/index.html +++ b/layouts/index.html @@ -18,13 +18,13 @@
-

A complete Ruby,
written in Go.

+

A Ruby you can embed,
written in Go.

go-embedded-ruby (rbgo) is a from-scratch Ruby — lexer, parser, compiler, bytecode VM and an AOT backend — that embeds in any Go program and cross-compiles to a single CGO-free binary. Built in the open as a digital common (commun numérique), free to use, study, modify and share under the BSD-3-Clause licence.

Get started → See the benchmarks
-
21,982 ruby/spec examples passing~100% of real Ruby parsed6 64-bit arches
+
22,488 ruby/spec language/+core/ examples passing~100% of real Ruby parsed6 64-bit arches
rbgo — zsh
@@ -32,9 +32,11 @@

A complete Ruby,
written in Go.< $ rbgo -e 'puts (1..20).reduce(:*)' 2432902008176640000 -# AOT-compile the hot path to native Go +# AOT-compile the hot path and drop the front-end $ rbgo build --closed app.rb -o app -→ app (CGO=0, 1 file, runs anywhere) +rbgo build: app (1 method(s) AOT-compiled: Object#fib) +rbgo build: closed-world — front-end dropped +rbgo build: binary size 172.5 MiB # the real Puppet CLI, end to end $ rbgo run puppet_apply.rb @@ -54,10 +56,10 @@

A complete Ruby,
written in Go.<

Everything a Ruby runtime gives you — as a Go library.

No FFI, no libruby, no system Ruby. Drop a genuine, monkey-patchable Ruby object model into your Go binary and ship it anywhere.

-
🧩

Truly embeddable

A real object model — classes, mixins, method_missing, metaprogramming, Fiber — callable straight from Go.

+
🧩

Embeddable

A real object model — classes, mixins, method_missing, metaprogramming, Fiber. The Go API is deliberately small today: ruby.Run(src, out) runs a program on a fresh VM and writes its output to one writer. Calling a Ruby method from Go is not exposed yet.

📦

One static binary

CGO=0 by default. Cross-compiles to amd64, arm64, riscv64, loong64, ppc64le and s390x.

-
⚡

AOT that flies

Ahead-of-time method compilation to unboxed Go kernels — 18–24× over MRI+YJIT on loop/fib.

-
✅

Conformance-first

21,982 ruby/spec language/ + core/ examples pass, gated in CI by a shrink-only floor of 21,970 — the floor is what CI refuses to fall below, not the score. It sits just under the measured total by design: a margin wide enough for run-to-run jitter, narrow enough that losing a whole spec file still trips it. Front-end: ~100% of valid Ruby parsed across Rails + Puppet (2026-06).

+
⚡

AOT for integer kernels

Ahead-of-time compilation of integer-bound methods to unboxed Go. Measured on this project’s own micro-benchmarks (darwin/arm64, whole-process wall time): 8.7× MRI+YJIT on the tight-loop case, 2.0× on recursive fib(35). Only methods the compiler can lower benefit.

+
✅

Conformance, measured

22,488 ruby/spec language/ + core/ examples pass, gated in CI by a shrink-only floor of 22,475 — the floor is what CI refuses to fall below, not the score. Only language/ and core/ are swept: library/, optional/ and security/ are not run, so this is not a percentage of Ruby. Front-end: ~100% of valid Ruby parsed across Rails + Puppet (2026-06).

🔒

Closed-world builds

rbgo build --closed bakes the program to bytecode and drops the front-end for a smaller binary.

🌿

A real ecosystem

Built from the go-ruby-* family — each a 100%-coverage, 6-arch stdlib module, bound natively.

@@ -65,8 +67,8 @@

Everything a Ruby runtime gives you — as a Go library.

-
18–24×
rbgo+AOT vs MRI+YJIT (loop/fib)
-
21,982
ruby/spec examples passing · CI floor 21,970
+
8.7×
rbgo+AOT vs MRI+YJIT, tight-loop benchmark (2.0× on fib(35))
+
22,488
ruby/spec language/+core/ examples passing · CI floor 22,475
178
go-ruby-* modules bound into rbgo
CGO=0
across 6 arches · 3 OSes