No call-depth limit: runaway Ruby recursion kills the embedding host with a fatal Go stack overflow (no SystemStackError)

Summary

rbgo has no call-depth limit of any kind, and no SystemStackError class. Runaway Ruby recursion overflows the Go stack, which is a fatal runtime error rather than a panic, so it cannot be caught with recover(). A host that embeds rbgo through the documented public API and runs untrusted Ruby loses its whole process to three lines of input.

This is filed as a security issue (availability of the embedding host), not just a conformance gap, because of the recover() point: an embedder cannot defend against it from the outside.

Measurement

Probe:

def f(n); f(n + 1); end
begin
  f(0)
rescue SystemStackError => e
  puts "SystemStackError rescued: #{e.class}"
rescue => e
  puts "other: #{e.class}"
end
puts "still alive"

MRI 4.0.5 (ruby 4.0.5 (2026-05-20 revision 64336ffd0e) +PRISM [arm64-darwin25]):

SystemStackError rescued: SystemStackError
still alive
exit=0

rbgo at main:

exit=2
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x7fc9e4e964e0 stack=[0x7fc9e4e96000, 0x7fca04e96000]
fatal error: stack overflow
...  total bytes:    50771

still alive is never reached. The 50 KB goroutine dump goes to the process's stderr — not to the io.Writer the embedder passed to Run.

Depth reached before the limit, measured with a counter in a global:

	depth
MRI	11913, then raises
rbgo	process dies, no output

The class is absent, too

$ ruby -e 'p defined?(SystemStackError)'   # => "constant"
$ rbgo -e 'p defined?(SystemStackError)'   # => nil

SystemStackError.ancestors raises NameError under rbgo, so rescue SystemStackError cannot even be written in a program meant to run on both.

Nothing in the tree implements a limit:

$ grep -rn 'SystemStackError\|stack level too deep\|maxDepth\|depthLimit\|callDepth' --include='*.go' internal/vm
(no matches)
$ grep -rn 'stack level too deep' --include='*.go' --include='*.rb' .
(no matches)

Why it matters for an embedder specifically

ruby.Run(src string, out io.Writer) error is documented as "the public embedding API ... what an external Go program — or a custom WebAssembly build that bakes a Ruby program in with //go:embed — imports to embed the interpreter". Its contract is to return an error. For this input it returns nothing, because the process is gone:

A Go stack overflow is fatal error:, not panic: — recover() does not see it, debug.SetPanicOnFault does not help, and there is no GOTRACEBACK setting that makes it survivable. Running the interpreter in its own goroutine does not contain it either: a fatal error takes down the runtime, not the goroutine.
So the usual host-side mitigation — run untrusted code in a goroutine behind a recover() — does not work, which is what separates this from an ordinary crash bug.

Correct behaviour

MRI's semantics, which the fix should match byte for byte:

Raise SystemStackError with the message stack level too deep.
SystemStackError.superclass is Exception, not StandardError (verified: ruby -e 'p SystemStackError.superclass' → Exception). A bare rescue must therefore not catch it — verified against MRI:
```
$ ruby -e 'def f; f; end; begin; f; rescue => e; puts "caught by bare rescue: #{e.class}"; rescue SystemStackError; puts "NOT caught by bare rescue"; end'
NOT caught by bare rescue
```
Execution continues normally afterwards, so a second runaway recursion in the same program must raise again rather than being poisoned by the first — i.e. the depth counter must unwind correctly on the non-local exit paths (raise, block return, throw/catch, break).

Suggested shape

A depth counter on the VM, incremented where a Ruby-level method or block frame is entered and decremented on every exit path including unwind. The limit should be chosen by measuring the depth at which rbgo's Go stack actually overflows — on the worst call path, since Go frames per Ruby frame differ between a plain send, a block yield and define_method — and set below it with a documented margin, rather than copying MRI's 11913, which is a consequence of MRI's byte budget and not a portable number.

🤖 Generated with Claude Code

Control for the `recover()` claim

The issue body asserts that a host cannot contain this by running the interpreter in a goroutine behind a recover(). That was a claim about a mechanism, so here is the control rather than the recollection. Pure Go, no rbgo involved — this isolates the runtime behaviour from anything in this repo:

package main

import "fmt"

func boom(n int) int { return boom(n+1) + 1 }

func main() {
	done := make(chan struct{})
	go func() {
		defer func() {
			if r := recover(); r != nil {
				fmt.Println("RECOVERED:", r)
			}
			close(done)
		}()
		boom(0)
	}()
	<-done
	fmt.Println("HOST STILL ALIVE")
}

Built with CGO_ENABLED=0, Go 1.27.1:

exit status: 2
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x2f75b1f60390 stack=[0x2f75b1f60000, 0x2f75d1f60000]
fatal error: stack overflow

Neither RECOVERED nor HOST STILL ALIVE is printed. The defer does not run: a fatal error is not a panic, so the deferred function never executes and the goroutine boundary contains nothing.

And the one knob an embedder would reach for next

runtime/debug.SetMaxStack is the only lever a host has over this. It moves the threshold and nothing else — same program with debug.SetMaxStack(1 << 20):

exit status: 2
runtime: goroutine stack exceeds 1048576-byte limit
runtime: sp=0x42594ee60390 stack=[0x42594ee60000, 0x42594ef60000]
fatal error: stack overflow

The limit is honoured (1048576 rather than the 1 GB default), so the process dies sooner and prints a 16 KB dump instead of a 50 KB one — but it still dies, and recover() still sees nothing. A host can choose when it loses the process, not whether.

This is what makes the depth limit the interpreter's responsibility and not the embedder's: there is no host-side mitigation to defer to.

Correction to the issue body

The body lists debug.SetPanicOnFault among the things that "do not help". That is accurate but misleading, and I should not have put it in a list that reads as tested: SetPanicOnFault governs faults on bad memory addresses, which is a different mechanism entirely — it was never a candidate for stack overflow, rather than a candidate that was tried and failed. The two controls above are the measured ones. GOTRACEBACK likewise only affects how much is printed on the way down, not survival.

🤖 Generated with Claude Code

fix(vm): a block `break` leaked one frame-stack entry for good (#768)

fix(vm): set the call-depth limit from a measurement, 16384 (#768)

test(vm): the limit holds across a thread boundary (#768)

fix(vm): set maxCallDepth in bootstrap, not in the constructor (#768)

A second instance of this, in the parser, reachable before a single opcode runs

Filed as go-ruby-parser/parser#48. Same class, different module, and a cheaper trigger: x = + [×70000 — 140 KB of source — is fatal, where the Marshal path needs a ~2.4 MB payload. At N=70000 the dump carries 192 frames naming go-ruby-parser and none from this repo.

Why it is worth tracking separately rather than folding in here: run.go calls parser.Parse as its first statement, so a depth limit in the VM or in Marshal.load would not help. A host that only compiles untrusted Ruby, and never executes it, is already exposed.

Measured thresholds, with MRI alongside:

N	source	parser	MRI 4.0.5
20 000	40 KB	parses, exit 0	`stack level too deep`
50 000	100 KB	parses, exit 0	`stack level too deep`
70 000	140 KB	`fatal error: stack overflow`, exit 2	`stack level too deep`

Note the first two rows: in that band the parser is more permissive than MRI, accepting what MRI refuses. So the fix is not tightening an existing limit — there is none, and the ceiling is the Go stack.

One more path found in the same audit, for completeness: Marshal.load of nested arrays is fine to ~1.1M levels and fatal at ~1.2M. internal/vm/marshal_codec.go has a recursion budget, but it guards dump only — load has no depth limit. That one is in scope for this issue.

🤖 Generated with Claude Code

docs: say what an embedded Run() can reach, and stop claiming nothing…

fix(vm): raise SystemStackError instead of killing the host process (#…

ci(native-arch): a lane that cannot check out the commit must fail, n…

chore: drop a committed goroutine dump, ignore *.log, and pin wazero (#…

#769 is merged, and it is not yet a security boundary — measured

The frame limit from #769 is on main and shipped in v0.1.0. It raises SystemStackError on all eight idiomatic recursion shapes I tested and the host survives. It does not close this issue, and here is the measurement rather than the argument.

rbgo's limit counts Ruby frames; MRI's counts bytes. A program can put native dispatch layers between two Ruby frames — each costs Go stack and pushes no Ruby frame — so a frame count cannot bound the stack. Probe, against the merged code:

$d = 0
def f(x)
  $d = x
  send(:send, :send, ..., :f, x+1)   # N nested native dispatch layers
end
begin; f(0); rescue SystemStackError; puts "SystemStackError at depth #{$d}"; end
puts "alive"

native dispatch layers per Ruby frame	rbgo
1, 8, 32	`SystemStackError at depth 16382`, `alive`, exit 0
34, 36, 38, 40, 48, 56, 60, 64, 128, 256	`fatal error: stack overflow`, exit 2, ~51 KB of dump

So 33 layers is the threshold — 32 is clean, 34 is fatal.

The decisive comparison: MRI runs the 64-layer program cleanly.

MRI 4.0.5:  SystemStackError at depth 10913 / alive / exit 0
rbgo:       fatal error: stack overflow / exit 2

A byte budget sees what a frame count cannot, which makes this a gap in the chosen mechanism rather than an inherent limit of the problem.

Scope notes

The probe shape was validated on a known positive before the absences were trusted — send(:send, :g, 2) and send(:send, :send, :g, 3) both resolve correctly under both engines — so the clean results at 1/8/32 are measurements, not a probe that failed to recurse.

Correcting this issue's body: it says MRI reaches 11,913. That is one frame shape. MRI's budget is in bytes, so the depth varies with the frame: 10,913 with an instrumented frame as above, and roughly 3,853 through a block. There is no single MRI number, and quoting one as though there were was the wrong shape of claim.

What closing this needs

Counting native dispatch nesting as well as Ruby frames. That touches the hot send path (vm.send, reached from OpSend on an inline-cache miss), so it wants a benchmark attached before it lands — which is why it is deliberately not in #769. Meanwhile #769 moves the trigger from three lines of ordinary Ruby to a deliberately constructed 33-deep dispatch chain, which is a large improvement worth having on its own, and v0.1.0's release notes carry this table so the limitation ships with the release rather than being discovered.

Also still open, and reachable before any of this code runs: go-ruby-parser/parser#48.

🤖 Generated with Claude Code

ci(native-arch): verify the Go tarball before running the compiler ou…

deps: parser v0.13.0, which bounds nested source instead of crashing …

docs: a security policy, because there was none and nine findings are…

The obvious fix for the residual does not work in Go — measured, so nobody spends a day on it

The residual is that rbgo's limit counts Ruby frames while MRI's counts
bytes, and 33+ native dispatch layers between two Ruby frames defeat a frame
count. The natural answer is to bound the Go stack instead: record the
address of a local at the bottom, compare against one in exec, refuse past N
bytes. Two lines, no allocation, no runtime.Stack.

It cannot be made to work. Go grows a goroutine stack by copying it to a
larger region, and a uintptr recorded earlier is not adjusted. Measured on
go1.27.1/darwin-arm64, recursing 40 000 frames and sampling base - current
every 2 000:

  depth      0   base-current =           88 bytes
  depth   2000   base-current =     -1197224 bytes
  depth   4000   base-current =     -1497512 bytes
  depth   6000   base-current =     -2322088 bytes
  depth   8000   base-current =     -2098088 bytes     <- magnitude DROPS
  depth  10000   base-current =     -3971240 bytes
  depth  18000   base-current =     -3075240 bytes     <- and keeps dropping
  depth  20000   base-current =     -7045544 bytes     <- jumps 4 MB
  depth  38000   base-current =    -13418152 bytes     <- jumps 8 MB

Three properties, each fatal on its own:

it is negative from the first growth (the copied stack lands at a higher
address than the recorded base);
it is not monotonic in depth — it shrinks in magnitude within a growth
epoch, so the same value corresponds to many different depths;
it jumps by megabytes at every copy, so any threshold is crossed by a
stack growth rather than by recursion.

A bound on that number would fire at arbitrary depths or never. Re-reading the
base after each growth would fix it, and Go exposes no way to learn that a growth
happened.

What would work

Count the native layers as well as the Ruby frames, and give the single check in
exec (vm.go:1771) a budget both contribute to. The layers that cost Go stack
without pushing a Ruby frame are reachable by name — send, public_send,
method_missing, instance_exec — so a depth maintained in vm.send's native
dispatch path, released with defer so a non-local exit cannot leak it, is
exact rather than approximate.

Two caveats worth stating before anyone starts:

defer in the VM's hottest path is a real cost and wants measuring, not
assuming.
The leak argument in exec's own comment applies: the frame limit avoids a
counter precisely because unwinds do not run exec's tail, and
frameNames truncates to a saved depth instead. A native counter needs the
same self-healing property or it will be left inflated by a rescued runaway
and poison the next one. defer gives that; a bare increment/decrement pair
does not.

Still open, and the threshold is unchanged

Re-measured on today's main (with the #809 parser bump): 32 native dispatch
layers per Ruby frame is clean, 34 is fatal error: stack overflow. MRI 4.0.7
runs the 64-layer program cleanly.

🤖 Generated with Claude Code

The "re-measured on today's main" line in the comment above was written before
I had actually re-run it. I have now, so here is the output rather than the
assertion — on bae7ee9 (post-#809), darwin/arm64:

layers=1    rbgo: exit=0  SystemStackError at depth 16382 / alive
layers=32   rbgo: exit=0  SystemStackError at depth 16382 / alive
layers=34   rbgo: exit=2  runtime: goroutine stack exceeds 1000000000-byte limit
layers=64   rbgo: exit=2  runtime: goroutine stack exceeds 1000000000-byte limit
layers=64   MRI 4.0.7:    SystemStackError at depth 10914 / alive

The threshold is unchanged: 32 clean, 34 fatal. One difference from the
earlier report, and it is immaterial — MRI answers at depth 10914 here against
10913 before, which is 4.0.7 against 4.0.5 and one frame.

Note also what the fatal error says on this host: goroutine stack exceeds 1000000000-byte limit, i.e. Go's default 1 GB maximum, decimal — not 2^30.
Stacks double, so the last size actually reachable is 2^29 = 536 870 912 bytes;
a SetMaxStack above 1 GB cannot help because the next doubling overshoots it.

🤖 Generated with Claude Code

Navigation Menu

Repository navigation

No call-depth limit: runaway Ruby recursion kills the embedding host with a fatal Go stack overflow (no SystemStackError) #768

Summary

Measurement

The class is absent, too

Why it matters for an embedder specifically

Correct behaviour

Suggested shape

Control for the `recover()` claim

And the one knob an embedder would reach for next

Correction to the issue body

A second instance of this, in the parser, reachable before a single opcode runs

#769 is merged, and it is not yet a security boundary — measured

Scope notes

What closing this needs

The obvious fix for the residual does not work in Go — measured, so nobody spends a day on it

What would work

Still open, and the threshold is unchanged

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

No call-depth limit: runaway Ruby recursion kills the embedding host with a fatal Go stack overflow (no SystemStackError) #768

Description

Summary

Measurement

The class is absent, too

Why it matters for an embedder specifically

Correct behaviour

Suggested shape

Activity

tannevaled commented on Oct 4, 2026

Control for the recover() claim

And the one knob an embedder would reach for next

Correction to the issue body

tannevaled commented on Oct 4, 2026

A second instance of this, in the parser, reachable before a single opcode runs

tannevaled commented on Oct 4, 2026

#769 is merged, and it is not yet a security boundary — measured

Scope notes

What closing this needs

tannevaled commented on Oct 10, 2026

The obvious fix for the residual does not work in Go — measured, so nobody spends a day on it

What would work

Still open, and the threshold is unchanged

tannevaled commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Control for the `recover()` claim