Embedding API: an Interp that keeps state, exchanges values with Go, and lets Go define Ruby

README.md

@@ -17,7 +17,9 @@ no libruby.
 examples ([what that counts](#runtime-conformance--rubyspec)) — a large and growing
 subset of the language, and **not** a drop-in replacement for CRuby. Read
 [What does not work yet](#what-does-not-work-yet) before you depend on it; the
-embedding API is currently **one function**.
+embedding API is small: `Run` for a one-shot program, and an `Interp` that keeps
+its state between evaluations, exchanges values with Go and lets Go define Ruby
+modules, classes and methods ([Embedding](#embedding-in-a-go-program)).
 
 This repository is the interpreter: a compiler that lowers Ruby to bytecode, and
 a stack VM (mruby/YARV lineage) that runs it. The Ruby **front-end** (lexer,
@@ -842,17 +844,109 @@ $ ./fib
 
 ## Embedding in a Go program
 
-The public API is deliberately tiny — **one function**:
+The public API has two halves: `Run`, which runs one program on a fresh VM, and
+`Interp`, an interpreter that stays alive between evaluations.
 
 ```go
 package ruby // github.com/go-embedded-ruby/ruby
 
 func Run(src string, out io.Writer) error
+
+func New(opts ...Option) *Interp            // WithStdout(w), WithStderr(w)
+func (in *Interp) Eval(src, file string, line int) (Value, error)
+func (in *Interp) Set(name string, v any) error
+func (in *Interp) Get(name string) (any, error)
+func (in *Interp) DefineModule(name string) (*Module, error)
+func (in *Interp) DefineClass(name string, super *Module) (*Module, error)
+func (m *Module) DefineMethod(name string, fn Func)
+func (m *Module) DefineSingletonMethod(name string, fn Func)
+type Func func(c *Call) (any, error)        // c.Self, c.Args, c.Kwargs, c.Block
+```
+
+Everything else in this repository is under `internal/`, so this is the whole
+embedding surface. The reference is the package documentation (`go doc
+github.com/go-embedded-ruby/ruby`); what follows is what it is for.
+
+### `Interp`: one program, evaluated piece by piece
+
+`Eval` runs source as top-level code and returns the value of its last
+expression. The **top-level local variables persist** from one `Eval` to the
+next, as one binding — so do methods, classes, constants and globals, which live
+in the VM anyway:
+
+```go
+in := ruby.New(ruby.WithStdout(&out))
+in.Eval("x = 40", "cell1.rb", 1)
+v, _ := in.Eval("x + 2", "cell2.rb", 1)
+v.Inspect() // "42"
+```
+
+`file` and `line` are what `__FILE__`, `__LINE__`, backtraces and errors report.
+A Ruby exception — a `SyntaxError` included — comes back as an `*ruby.Error`
+with its `Class`, `Message`, `Backtrace`, and the `File`/`Line` of the innermost
+frame in the evaluated file, so an exception raised deep in a library is placed
+at the line of your source that led to it. `exit` is an `*Error` too
+(`ExitStatus()` reads its status), and the interpreter stays usable after any of
+them. `$stdout` and `$stderr` are separate streams, and `SetStdout`/`SetStderr`
+redirect them between evaluations. An `Interp` is not safe for concurrent use.
+
+### Values
+
+`Set` assigns a top-level local and `Get` reads one, converting:
+
+| Go | Ruby | back to Go |
+| --- | --- | --- |
+| `nil`, `bool` | `nil`, `true`/`false` | the same |
+| `int`…`int64`, `uint`…`uint64` | `Integer` | `int64` |
+| `*big.Int` | `Integer` | `int64`, or `*big.Int` when it does not fit |
+| `float32`, `float64` | `Float` | `float64` |
+| `complex64`, `complex128` | `Complex` | `complex128` |
+| `*big.Rat` | `Rational` | `*big.Rat` |
+| `string`, `[]byte` | `String` | `string` |
+| `ruby.Symbol` | `Symbol` | `ruby.Symbol` |
+| `*ndarray.Array` | `NDArray` — **the same array, no copy** | the same `*ndarray.Array` |
+| slices, arrays | `Array` | `[]any` |
+| maps | `Hash` | `map[any]any` |
+| any other Go value | an opaque object (`GoValue`, or the class `Bind` names) | the very same Go value |
+| `ruby.Value` | itself | — |
+
+A Ruby object with no Go counterpart comes back as a `ruby.Value`, which can be
+inspected, called (`v.Call("method", args...)`) and passed back in.
+
+### Defining Ruby from Go
+
+```go
+stats, _ := in.DefineModule("Stats")
+normal, _ := stats.DefineClass("Normal", nil)
+normal.Bind(&Normal{})                     // a *Normal crossing into Ruby is a Stats::Normal
+normal.DefineSingletonMethod("new", func(c *ruby.Call) (any, error) {
+	loc := c.Kwargs["loc"].Go()              // keyword arguments by name
+	...
+	return &Normal{...}, nil
+})
+normal.DefineMethod("pdf", func(c *ruby.Call) (any, error) {
+	n := c.Self.Go().(*Normal)               // the Go value back
+	...
+})
+stats.DefineSingletonMethod("minimize", func(c *ruby.Call) (any, error) {
+	y, err := c.Block.Call(x)                // call the block from Go
+	if err != nil {
+		return nil, err                        // re-raises the block's exception
+	}
+	...
+})
 ```
 
+A `Func` returns its result (converted as above) or an error, which raises:
+`ruby.NewError("ArgumentError", "...")` raises that class, an error that came out
+of Ruby raises the same exception again, anything else is a `RuntimeError`. A
+`break`, `throw` or `return` in a block unwinds through the Go code that called
+it to where Ruby expects it.
+
+### `Run`
+
 `Run` parses, compiles and executes `src` on a **fresh VM**, writing the program's
-output to `out`. Everything else in this repository is under `internal/`, so this
-is the whole embedding surface today.
+output to `out`.
 
 ```console
 $ go get github.com/go-embedded-ruby/ruby@v0.1.0
@@ -902,14 +996,15 @@ The contract, as measured rather than intended:
   whose cwd was its build directory came back with a 13-byte `PATH` and a cwd of
   `/private/tmp`. Because `require_relative` resolves against that same working
   directory (next bullet), one `Run` can also write a file that a later `Run`
-  loads. There is no exported way to hold a VM open, pre-load code into it, call
-  a Ruby method from Go, or pass Go values in.
+  loads. (Holding a VM open, pre-loading code, calling Ruby from Go and passing
+  Go values in is what `Interp` is for.)
 - **A program that exits is reported as success.** `exit`, `exit!` and `abort`
   stop the program and `Run` returns **`nil`**, so an embedder cannot tell
   “finished” from “stopped”: `abort "fatal"` writes `fatal` to `out` and reports
   no error, and `exit 3` loses the 3. Only a Ruby *exception* becomes a Go error.
   The `rbgo` CLI recovers the status through `internal/vm`, which is unreachable
-  from outside, so today there is no way for an embedder to obtain it.
+  from outside, so `Run` cannot report it; `Interp.Eval` returns it as an
+  `*Error` whose `ExitStatus()` holds it.
 - **`require_relative` resolves against the process working directory**, not
   against a script, because `Run` has no file to anchor to. From a different
   working directory the same program fails with

define.go

@@ -0,0 +1,214 @@
+package ruby
+
+import (
+	"fmt"
+	"reflect"
+	"strings"
+
+	"github.com/go-embedded-ruby/ruby/internal/object"
+	"github.com/go-embedded-ruby/ruby/internal/vm"
+)
+
+// Func is a method written in Go. It returns the method's result, converted to
+// Ruby as Value describes, or an error, which raises: an *Error raises its
+// class (NewError("ArgumentError", "...")), an *Error that came from Ruby —
+// out of Block.Call, say — raises that same exception again, and any other
+// error raises a RuntimeError with its message.
+type Func func(c *Call) (any, error)
+
+// Call is one call of a Func.
+type Call struct {
+	Interp *Interp
+	// Self is the receiver: the module or class for a singleton method, the
+	// instance for an instance method.
+	Self Value
+	// Args are the positional arguments.
+	Args []Value
+	// Kwargs are the keyword arguments, by name; nil when there are none.
+	// Ruby passes keywords to a method written in Go as a trailing Hash: a
+	// final argument that is a non-empty Hash whose keys are all Symbols is
+	// read as the keywords, unless the call site said it passed none.
+	Kwargs map[string]Value
+	// Block is the block the method was called with, or nil.
+	Block *Block
+}
+
+// Block is a block passed to a Func.
+type Block struct {
+	in *Interp
+	p  *vm.Proc
+}
+
+// Call calls the block with args converted to Ruby, and returns its value. A
+// Ruby exception raised in the block comes back as an *Error; returning that
+// error from the Func re-raises the same exception. A break, a throw or a
+// non-local return in the block unwinds through the Go code that called it, so
+// run nothing after Call that must not be skipped other than with defer.
+func (b *Block) Call(args ...any) (Value, error) {
+	rargs, err := b.in.toRubyAll(args)
+	if err != nil {
+		return Value{}, err
+	}
+	r, rerr := b.in.vm.Protect(func() object.Value { return b.in.vm.CallBlock(b.p, rargs) })
+	if rerr != nil {
+		return Value{}, b.in.newError(rerr, "")
+	}
+	return b.in.wrap(r), nil
+}
+
+// Arity is the block's Proc#arity.
+func (b *Block) Arity() int { return b.in.vm.ProcArity(b.p) }
+
+// Module is a Ruby module or class defined, or found, by the host.
+type Module struct {
+	in *Interp
+	c  *vm.RClass
+}
+
+// Name is the module's qualified name, e.g. "Stats::Normal".
+func (m *Module) Name() string { return m.c.Name() }
+
+// Value is the module itself as a Ruby value.
+func (m *Module) Value() Value { return m.in.wrap(m.c) }
+
+// DefineModule returns the top-level module name, creating it if needed. name
+// may be qualified ("Stats::Tests"); every part but the last must exist or is
+// created as a module.
+func (in *Interp) DefineModule(name string) (*Module, error) {
+	return in.define(nil, name, nil, true)
+}
+
+// DefineClass returns the top-level class name, creating it as a subclass of
+// super (nil: Object) if needed. name may be qualified as for DefineModule.
+func (in *Interp) DefineClass(name string, super *Module) (*Module, error) {
+	return in.define(nil, name, super, false)
+}
+
+// DefineModule returns the module name inside m, creating it if needed.
+func (m *Module) DefineModule(name string) (*Module, error) {
+	return m.in.define(m.c, name, nil, true)
+}
+
+// DefineClass returns the class name inside m, creating it as a subclass of
+// super (nil: Object) if needed.
+func (m *Module) DefineClass(name string, super *Module) (*Module, error) {
+	return m.in.define(m.c, name, super, false)
+}
+
+func (in *Interp) define(parent *vm.RClass, name string, super *Module, module bool) (*Module, error) {
+	parts := strings.Split(name, "::")
+	for i, p := range parts {
+		if !isConstName(p) {
+			return nil, fmt.Errorf("ruby: %q is not a constant name", name)
+		}
+		last := i == len(parts)-1
+		var (
+			c  *vm.RClass
+			ok bool
+		)
+		switch {
+		case !last || module:
+			c, ok = in.vm.DefineModule(parent, p)
+		default:
+			var sc *vm.RClass
+			if super != nil {
+				sc = super.c
+			}
+			c, ok = in.vm.DefineClass(parent, p, sc)
+		}
+		if !ok {
+			what := "a module"
+			if last && !module {
+				what = "a class"
+			}
+			return nil, fmt.Errorf("ruby: %s is already defined and is not %s", strings.Join(parts[:i+1], "::"), what)
+		}
+		parent = c
+	}
+	return &Module{in: in, c: parent}, nil
+}
+
+func isConstName(s string) bool {
+	if s == "" || s[0] < 'A' || s[0] > 'Z' {
+		return false
+	}
+	for _, r := range s {
+		if r != '_' && !(r >= 'a' && r <= 'z') && !(r >= 'A' && r <= 'Z') && !(r >= '0' && r <= '9') {
+			return false
+		}
+	}
+	return true
+}
+
+// DefineMethod defines the instance method name, written in Go.
+func (m *Module) DefineMethod(name string, fn Func) { m.c.DefineNative(name, m.in.native(fn)) }
+
+// DefineSingletonMethod defines the singleton method name — Stats.mean, or a
+// class method such as Normal.new — written in Go.
+func (m *Module) DefineSingletonMethod(name string, fn Func) {
+	m.c.DefineSingletonNative(name, m.in.native(fn))
+}
+
+// SetConst defines the constant name inside m, converted to Ruby.
+func (m *Module) SetConst(name string, v any) error {
+	if !isConstName(name) {
+		return fmt.Errorf("ruby: %q is not a constant name", name)
+	}
+	rv, err := m.in.toRuby(v)
+	if err != nil {
+		return err
+	}
+	m.in.vm.SetConstIn(m.c, name, rv)
+	return nil
+}
+
+// Bind makes every Go value of sample's dynamic type an instance of m when it
+// crosses into Ruby, so the methods defined on m apply to it, and Value.Go
+// gives the Go value back. Bind(&Normal{}) binds *Normal.
+func (m *Module) Bind(sample any) { m.in.types[reflect.TypeOf(sample)] = m.c }
+
+// Wrap makes v an instance of m, whatever its type.
+func (m *Module) Wrap(v any) Value { return m.in.wrap(m.in.vm.WrapGo(v, m.c)) }
+
+// native adapts a Func to the VM's calling convention.
+func (in *Interp) native(fn Func) vm.NativeFn {
+	return func(v *vm.VM, self object.Value, args []object.Value, blk *vm.Proc) object.Value {
+		noKW := v.NativeNoKW()
+		c := &Call{Interp: in, Self: in.wrap(self)}
+		if n := len(args); n > 0 && !noKW {
+			if h, ok := args[n-1].(*object.Hash); ok && h.Len() > 0 && symbolKeys(h) {
+				c.Kwargs = make(map[string]Value, h.Len())
+				for _, k := range h.Keys {
+					e, _ := h.Get(k)
+					c.Kwargs[string(k.(object.Symbol))] = in.wrap(e)
+				}
+				args = args[:n-1]
+			}
+		}
+		c.Args = make([]Value, len(args))
+		for i, a := range args {
+			c.Args[i] = in.wrap(a)
+		}
+		if blk != nil {
+			c.Block = &Block{in: in, p: blk}
+		}
+		res, err := fn(c)
+		if err != nil {
+			in.raise(err)
+		}
+		r, err := in.toRuby(res)
+		if err != nil {
+			vm.Raise("TypeError", err.Error())
+		}
+		return r
+	}
+}
+
+func symbolKeys(h *object.Hash) bool {
+	for _, k := range h.Keys {
+		if _, ok := k.(object.Symbol); !ok {
+			return false
+		}
+	}
+	return true
+}

example_interp_test.go

@@ -0,0 +1,40 @@
+package ruby_test
+
+import (
+	"fmt"
+	"os"
+
+	ruby "github.com/go-embedded-ruby/ruby"
+)
+
+// An interpreter keeps its top-level locals between evaluations, takes Go
+// values in, and calls Go methods that call Ruby blocks back.
+func ExampleInterp() {
+	in := ruby.New(ruby.WithStdout(os.Stdout))
+	in.Set("xs", []float64{1, 2, 3})
+
+	m, _ := in.DefineModule("Calc")
+	m.DefineSingletonMethod("sum_by", func(c *ruby.Call) (any, error) {
+		total := 0.0
+		for _, x := range c.Args[0].Go().([]any) {
+			y, err := c.Block.Call(x)
+			if err != nil {
+				return nil, err
+			}
+			total += y.Go().(float64)
+		}
+		return total, nil
+	})
+
+	in.Eval("squares = Calc.sum_by(xs) { |x| x * x }", "example.rb", 1)
+	v, _ := in.Eval(`puts "sum of squares: #{squares}"; squares / 2`, "example.rb", 2)
+	got, _ := in.Get("squares")
+	fmt.Println(v.Inspect(), got)
+
+	_, err := in.Eval("Calc.nope", "example.rb", 3)
+	fmt.Println(err)
+	// Output:
+	// sum of squares: 14.0
+	// 7.0 14
+	// example.rb:3: undefined method 'nope' for module Calc (NoMethodError)
+}