[tannevaled]

What this fixes

A Ruby program with runaway recursion took the whole host process down with it.

def f(n); f(n + 1); end
begin
  f(0)
rescue SystemStackError => e
  puts "SystemStackError rescued: #{e.class}"
end
puts "still alive"

	stdout	stderr	exit
MRI 4.0.5	SystemStackError rescued: SystemStackError / still alive	0 bytes	0
rbgo, main	0 bytes	51224 bytes, opening runtime: goroutine stack exceeds 1000000000-byte limit	2
rbgo, this branch	SystemStackError rescued: SystemStackError / still alive	0 bytes	0

stdout is byte-identical to MRI's (cmp clean), stderr is empty on both sides, exit status matches. The run takes 1.46 s.

This is a security finding and not only a conformance gap: a Go stack overflow is a fatal runtime error, not a panic, so a deferred recover() never runs and an embedding host cannot contain it. ruby.Run(src, out) is the documented embedding API, so three lines of untrusted Ruby could end the host process. See #768, where the recover() and debug.SetMaxStack controls are written up.

Before this change, grep -rn 'SystemStackError|stack level too deep|maxDepth|callDepth' internal/vm had no matches, p defined?(SystemStackError) printed nil, and SystemStackError.ancestors raised NameError.

SystemStackError.superclass is Exception

Set by MRI in Init_Proc:

rb_eSysStackError = rb_define_class("SystemStackError", rb_eException);          /* proc.c:4424 */
rb_vm_register_special_exception(ruby_error_sysstack, rb_eSysStackError,
                                 "stack level too deep");                        /* proc.c:4425 */

proc.c:4424–4425 at tag v3_4_1 — note: v4_0_5 and v4_0_0 do not exist in github.com/ruby/ruby (both 404), so the citation is from the newest tag that is published upstream. The facts it asserts are confirmed first-hand against the MRI 4.0.5 installed here:

$ ruby -e 'p SystemStackError.superclass, SystemStackError.ancestors'
Exception
[SystemStackError, Exception, Object, Kernel, BasicObject]

No StandardError anywhere in it, so a bare rescue must not catch it, and TestBareRescueDoesNotCatchSystemStackError asserts that. The message is copied from proc.c:4425, not invented.

The limit: measured, not chosen by taste

28 recursion paths were bisected in subprocesses — binary-searching the largest limit each path can be given before the Go runtime kills the process, under a reduced debug.SetMaxStack — which yields bytes of Go stack per Ruby frame, the portable quantity. On darwin/arm64, Go 1.27.1:

path	B/frame	overflow depth
send(:public_send, :send, :f, n + 1)	8132	66016 ← governs the limit
catch(:x) { f(n + 1) }	6531	82208
1.times { f(n + 1) } / [1].each { }	6228	86255
[1].sort_by { } / {1=>2}.each { }	6348	84576
1.instance_exec { }	6267	85664
[1].inject(0) { } / flat_map / zip	5998	89504
define_method(:f) { }	5779	92857
f(n + 1) (plain send)	5619	95552

Measuring the cheapest path would have overstated the margin by 45%. The 20 paths not listed (Struct, method_missing, super chains, Method#call, symbol-to-proc, operator dispatch, to_s through interpolation, select/min_by/sum/each_slice/each_with_object/group_by, nested blocks, catch) all fall between 5619 and 6531.

Bytes-per-frame is stable across stack sizes — 6452.8, 6450.3, 6448.4, 6448.1 B/frame at SetMaxStack of 8, 16, 64 and 256 MiB — so it is a property of rbgo's call path, not of the measurement.

The depth is not 1 GB divided by it

Getting this wrong would have been a factor of two, and my first extrapolation did get it wrong. Go's default maximum is 1,000,000,000 bytes (decimal) and goroutine stacks are powers of two, so the largest stack actually permitted is 2^29 = 536,870,912 bytes — 512 MiB, because 2^30 = 1,073,741,824 exceeds the default.

• predicted from 512 MiB / 8132 B/frame: 66016
• bisected directly at Go's real default, no SetMaxStack at all: 66051

0.05% apart. A second path cross-checks it: catch_block predicted 82208, measured 81442 (0.9%). A naive 1<<30 / bpf would have said 166521 — and a limit chosen under that belief would not have been safe.

Chosen: 16384

• 4.03x below the measured overflow depth of 66051
• 1.50x above MRI's deepest reachable depth
• holds about 133 MB of Go stack at the limit

MRI's budget is in bytes, so its depth is not one number. Against the MRI 4.0.5 here: 10919 frames of def f(n); $d = n; f(n + 1); end, but only 3853 levels of the same recursion through a block. 10919 is the figure that binds: a limit below MRI's deepest would make rbgo raise where MRI does not, turning a security fix into a conformance regression. TestDefaultCallDepthLimitLeavesRoomForMRIProgrammes guards from that side; TestStackDepthLimitSitsBelowTheGoOverflow re-derives the whole measurement on every run and fails under a 2x margin.

The margin is deliberately asymmetric: overshooting is fatal and unrecoverable, undershooting raises an exception the program already had to handle. It also has to absorb a figure measured on one platform and toolchain.

Why len(vm.frameNames) and not a counter

exec is the one place a Ruby-level frame is created — methods, blocks, procs, lambdas, define_method bodies, class and module bodies, eval and instance_eval all arrive there — so one check covers every way a Ruby program can recurse. TestEveryKindOfRubyFrameIsCounted drives 14 different dispatch paths into it.

The depth is len(vm.frameNames) rather than a counter of its own, and that is the whole reason it cannot leak. A counter would have to be decremented on each of exec's exits, and the ones that unwind by panic never run exec's tail. frameNames already solves that, and solves it by truncating to a saved depth rather than counting pops: whoever catches an unwind resets the stack to its own depth, discarding every abandoned frame's entry in one assignment. That makes the depth self-healing — it cannot be left inflated by a rescued runaway recursion, so it cannot poison a later one.

raiseDeep needs no Ruby frame of its own, which is what lets it be called from the place that has just refused to create one: exceptionObject builds the exception as a Go *RObject and never dispatches #initialize, and rbgo runs rescue/ensure bodies as instructions inside the frame that owns them rather than as new frames. MRI needs a preallocated instance (special_exceptions[ruby_error_sysstack], vm_insnhelper.c:61) for exactly the same reason.

What the depth does on each non-local exit

exit	what restores the depth
normal return	exec's tail truncates to frameNamesDepth-1 (vm.go:3299)
rescued exception	the handler-path recover truncates to frameNamesDepth (vm.go:3185)
non-local return / next	the returnSignal recover truncates to frameNamesDepth-1 (vm.go:2124)
block break	was leaking — fixed here, see below
throw / catch	Kernel#catch keeps the same six depths; findYield likewise (find.go:146)
ensure crossed by a raise	runs as instructions in its own frame; the eventual catcher truncates
escaping Run	frameNames[:0] at all four boundaries

A real pre-existing leak, found by the test this design owes

sendCatchBreak recovers a breakSignal and returns its value but never restored the per-frame tracking stacks, so

[1, 2, 3].each { |x| break if x == 2 }

left one frameNames entry behind, permanently. That was invisible while those stacks only fed backtraces and Kernel#caller: the next unwind caught anywhere truncates to its own saved depth and swallows the stale entry with it, so the leak only survives when nothing deeper is caught afterwards — and no backtrace test ever stood in that position.

It stopped being invisible the moment len(frameNames) became the limit: leaked depth is budget that never comes back, so a long-running program breaking out of a loop a few tens of thousands of times would raise SystemStackError with no recursion in sight.

TestDepthDoesNotLeakOnAnyNonLocalExit is what found it. It drives a runaway recursion to the limit, rescues it, performs a non-local exit, then does it again and compares the depth reached: 197 then 196 is the leak, 197 then 197 is the fix. It covers break, next, a block return, throw/catch, an ensure crossed by a raise, and a deep variant of each. TestASecondRunawayRecursionStillRaises states the consequence a user would notice, three rounds in a row.

frameCrefs and frameMethods are restored with frameNames, or index i stops meaning the same frame in all three (#649).

Known incomplete, and measured rather than assumed

A frame count cannot bound the Go stack on its own. A program can put an unbounded number of native dispatch layers between two Ruby frames, and a native layer costs Go stack while pushing no frame. Each extra Kernel#send layer in send(:send, …, :f, n + 1) costs 833 B:

sends per Ruby frame	1	2	4	8	16	32
bytes per frame	7282	8117	9777	13107	19761	33091
overflow depth	73728	66144	54912	40960	27168	16224

So about 25 nested sends per frame bring the overflow depth below 16384, and no limit above MRI's 10919 can cover an unbounded chain — this is a property of the approach, not of the constant.

Closing it needs a count of native dispatch nesting as well. That is a change to the hot dispatch path (vm.send is reached from the OpSend handler on an inline-cache miss, vm.go:2617) and therefore a scope decision with a benchmark attached, not a line added here. It is recorded in stackdepth.go and on #768 rather than left for the next person to rediscover.

I also could not use the obvious byte-budget approach MRI uses: Go has no portable way to read the stack pointer, and a stored stack address cannot serve as a baseline because the Go runtime copies a goroutine's stack when it grows, leaving the saved address stale by the relocation delta.

One more shape, checked rather than reasoned about

rbgo keeps one frame stack for the whole VM, shared by every Ruby thread under the GVL, and Run resets it to [:0] at its boundaries — so a thread body could have been handed a fresh depth budget while its parent's Go frames were still live, letting a program amplify its reachable depth by one limit per thread it spawns. Measured with the real constant: plain recursion reaches 16382, and recursion that spawns and joins a Thread at every level reaches 8191 — exactly half, because each level costs two entries (the method frame and the block frame). The shared stack keeps accumulating across the boundary instead of restarting.

That is conservative: each Ruby thread is a goroutine with its own Go stack, so the budget is stricter than the stacks require. Conservative is the safe direction for a limit whose job is to stay below a fatal error. TestTheLimitHoldsAcrossAThreadBoundary pins it.

A second gap this makes reachable, not asked for and not fixed here

The probe in this PR rescues the error, and there rbgo is byte-identical to MRI. Unrescued, it is not — and only because the limit now lets a 16384-deep SystemStackError backtrace exist at all:

	exit	stderr
MRI 4.0.5	1	448 bytes, 17 lines
rbgo, this branch	1	458814 bytes, 16387 lines

The first line is identical (t3.rb:1:in 'Object#f': stack level too deep (SystemStackError)) and so is the exit status. MRI truncates, and the rule is specific to this one class — Thread::Backtrace.limit defaults to -1, unlimited, so nothing general is at work (vm_backtrace.c:911). From eval_error.c:231–240:

if (eclass == rb_eSysStackError) {
    long trace_head = 9;
    long trace_tail = 4;
    long trace_max = trace_head + trace_tail + 5;
    if (len > trace_max) {
        skip_start = trace_head;
        skip_len = len - trace_max + 5;
    }
}

giving a head, \t ... N levels..., and a tail.

This is a pre-existing shape, not a new behaviour in kind. At depth 3000 raising a RuntimeError, main and MRI both print all 3003 lines — rbgo already printed full backtraces, and MRI does too below its threshold. What changed is that a backtrace this deep is now reachable instead of the process dying. Fixing it means teaching the uncaught-exception printer a class-specific truncation rule with its own tests, which is a separable conformance change rather than a line to add here, so it is left for a follow-up and recorded on #768.

One useful by-product: MRI's own marker read ... 10908 levels..., and 10908 + the 13 printed frames corroborates the 10919 figure measured separately — two independent routes to the same number.

The regression this branch made, and what caught it

Worth recording, because it is the failure mode of the design and not an accident. maxCallDepth was first set in the constructor — but a VM is not always built through one: internal tests assemble a &VM{} literal and bootstrap it by hand (bareVM in prelude_test.go). Those VMs got maxCallDepth 0, and since exec's check is >=, a zero limit refuses every frame rather than none:

--- FAIL: TestLoadSourceRuntimeError
    prelude_test.go:32: got SystemStackError: stack level too deep, want NoMethodError
--- FAIL: TestLoadSourceSuccess
    prelude_test.go:38: unexpected error: SystemStackError: stack level too deep

x = 1 + 1 died on its first call. The same two tests failed on the arch arm64 check run of the first push, and ./internal/vm was run against origin/main in a separate checkout to confirm it was this branch's regression and not a pre-existing failure — main passes.

The assignment now lives in bootstrap, unconditionally, with the constructor's copy removed so there is one source of truth. bootstrap is the one gate every usable VM passes through: a VM that has not been bootstrapped has no classes and cannot run anything. TestEveryBootstrappedVMHasALimit covers the literal-plus-bootstrap path and not only the constructor, and names why zero is dangerous in both directions — >= reads it as "refuse everything", and a check written the other way round would read it as "no limit at all", which is silently the original defect.

Verification

ruby/spec conformance ratchet — OK, nothing went backwards

The authoritative figure, from this PR's own ruby/spec ratchet check run on ubuntu (corpus pinned at 87b1631992bd00cf0c4934474766d54dad088191):

files: 2206 measured, 2206 in baseline
passing examples: 23501 (baseline 23481, +20)
12 file(s) improved
ruby/spec ratchet OK -- no file went backwards

0 files regressed, 12 improved, +20 examples.

core/float/inspect_spec.rb                    1 passing, baseline 0  (+1)
core/kernel/require_spec.rb                  70 passing, baseline 69 (+1)
core/numeric/imag_spec.rb                      1 passing, baseline 0  (+1)
core/numeric/rect_spec.rb                      1 passing, baseline 0  (+1)
core/random/bytes_spec.rb                      9 passing, baseline 8  (+1)
core/string/dedup_spec.rb                      1 passing, baseline 0  (+1)
core/thread/each_caller_location_spec.rb       6 passing, baseline 5  (+1)
core/unboundmethod/equal_value_spec.rb        17 passing, baseline 15 (+2)
language/keyword_arguments_spec.rb            20 passing, baseline 14 (+6)
language/method_spec.rb                      160 passing, baseline 157 (+3)
language/rescue_spec.rb                       53 passing, baseline 52 (+1)
language/send_spec.rb                         72 passing, baseline 71 (+1)

The language/ movers are the shape a depth limit produces: a spec file that previously killed the rbgo process contributed 0 for everything after the crash, so the whole tail of the file comes back at once — keyword_arguments_spec.rb +6 is the clearest case.

BASELINE is deliberately not regenerated here: locking in a win is a separate decision, and the sweep that would write it must be the ubuntu one that recorded the current file.

A local run disagreed, and the disagreement is instructive. The same sweep on macOS reported +24 across 16 files — the extra four being core/file/stat/inspect_spec.rb, core/kernel/backtick_spec.rb, core/marshal/load_spec.rb and core/process/setrlimit_spec.rb, all plausibly platform rather than this change. BASELINE was recorded on the ubuntu lane, so only the ubuntu measurement is comparable to it; the macOS number is quoted here only to show that the two agree on direction and on every language/ file.

Also worth knowing: the cached corpus at /tmp/rubyspec-corpus on the development machine contained zero *_spec.rb files while still having the language/ and core/ directories run.sh probes for, so a local sweep against it would have measured nothing. The judge does refuse an empty sweep with exit 2, so the guard held — but the local run used a freshly cloned corpus.

Per-function coverage ratchet — OK on both records

$ covratchet -record scripts/coverage/BELOW100.macos-latest -lane macos-latest -profile cover.txt
functions below 100%: 16 measured, 16 recorded
coverage ratchet OK -- no function newly below 100%          exit 0

The ubuntu record passes against the same (macOS) profile too, exit 0, reporting 3 functions that reached 100%. So none of the new code is below 100%.

The rest

• GOOS=wasip1 GOARCH=wasm go build ./... — green
• GOOS=js GOARCH=wasm go build ./... — green, and the wasm (js, browser) check run passes
• gofmt -l . — empty
• go build -tags rbgo_closed ./internal/... . — green. go build -tags rbgo_closed ./... fails at cmd/rbgo/closed_main.go: undefined: embeddedProgram on main too (verified against origin/main in a separate checkout): embeddedProgram is injected by rbgo build --closed via -overlay, so a bare build of that package can never link. Not a regression from this branch.
• CGO_ENABLED=0 go test ./... -timeout=90m — see the three test (…) check runs

🤖 Generated with Claude Code

[tannevaled]

Independent verification, from the session that filed #768

I rebuilt this branch (d5925e7) myself and re-ran the checks against the installed MRI 4.0.5 rather than taking the PR's own measurements. All of it agrees, and three of these probes are ones the PR does not claim, so they are evidence rather than confirmation.

1. The issue's probe, byte for byte. Identical output and identical exit status:

MRI          SystemStackError rescued: SystemStackError / still alive   exit=0
this branch  SystemStackError rescued: SystemStackError / still alive   exit=0

2. SystemStackError.superclass is Exception, so a bare rescue must not catch it. Both engines:

NOT caught by bare rescue: SystemStackError / stack level too deep
Exception

3. The counter does not leak on any non-local exit. This is the part I was most worried about, so each unwind path gets 300 frames and is then followed by a bounded recursion that must still reach 400:

after raise:  deep(400)=400
after return: deep(400)=400
after throw:  deep(400)=400
after break:  deep(400)=400
still raises

Byte-identical to MRI, and the trailing still raises means a runaway after all four unwinds is not pre-poisoned. Two consecutive runaways in one program also both raise, with g(500) working in between.

4. The limit is above MRI's, which is the safe direction.

	depth reached
MRI	11,913
this branch	16,383

So no program that recurses legitimately on MRI can hit rbgo's ceiling first. Worth stating explicitly because the opposite choice — a limit tuned tightly to the Go stack — would have turned an availability fix into a conformance regression, and that trade is invisible unless someone measures the depth.

5. Every call shape raises, not just the cheap one. This is the check that matters most, since Go stack bytes per Ruby frame differ by path and the limit has to hold on the worst one. Eight shapes, all SystemStackError, all identical to MRI, exit 0:

plain, block/yield, send, lambda, define_method, method_missing, mutual, proc-in-block
-> SystemStackError on every one, under BOTH engines
ALL SHAPES SURVIVED, host alive

No shape I could construct produced a fatal error: stack overflow.

Still out of scope here, deliberately, so it is not mistaken for a gap in this PR:

• the same class of defect in the parser (Deeply nested source overflows the Go stack fatally (140 KB is enough): no nesting limit, and recover() cannot contain it go-ruby-parser/parser#48) — 140 KB of nested source is still fatal, and run.go calls parser.Parse before any of this code runs, so a depth limit in the VM cannot help it;
• Marshal.load has no depth limit (the recursion budget in marshal_codec.go guards dump only), fatal at ~1.2M nesting levels.

Both are tracked on #768.

🤖 Generated with Claude Code