Skip to content

compiler: for-loop targets, block-local variables, and what defined? declares (+382 in language/) - #626

Merged
tannevaled merged 8 commits into
mainfrom
wave26/compiler
Sep 23, 2026
Merged

tannevaled merged 8 commits into
mainfrom
wave26/compiler

Conversation

@tannevaled

@tannevaled tannevaled commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Four compiler-side gaps the v0.2.0 parser (#625) exposed, and a fifth found while
chasing them. Three of the four spec files this unblocks had been contributing
zero to both columns of the conformance measurement — not failing, invisible.

language/ over the whole tree (find language -name '*_spec.rb', 80 files),
main @ 097d86a vs this branch, same binary lineage and the same harness:

pass fail error
main 1776 310 126
this branch 2158 382 130

+382 passing, and no file loses an example. The only four files that move at
all are the three unblocked ones and rescue_spec, which gains one:

language/block_spec.rb      (0, 0, 0)  -> (139, 27, 0)   +139
language/defined_spec.rb    (0, 0, 0)  -> (213, 45, 0)   +213
language/for_spec.rb        (0, 0, 0)  -> (29,  0,  5)   +29
language/rescue_spec.rb     (49, 7, 4) -> (50,  7,  3)   +1

The rise in fail+error (436 → 512) is entirely the examples those three files can
now reach at all; every one of them was previously unmeasured.

rescue_spec's +1 is not incidental: its "using a safely navigated setter method
on a nil target" example (rescue => o&.captured_error) reaches the same store arm
the for safe-navigation fix below repairs, and stopped raising NoMethodError.

1. Block-local variables — { |a; b| … }

They are fresh, nil-valued locals of the block that shadow any enclosing binding.
The parser carries them on Block.Locals; the compiler ignored the field, so
block_spec.rb stopped at "undefined local variable" before one example ran.

MRI read: prism_compile.c v3_4_0:6325-6334 walks the block-parameters node's
locals list and calls pm_insert_local_index for each, after the
requireds/optionals/rest/posts and before the "fill in any locals we missed" pass
— table entries, not parameters, so they start nil. parse.y v3_4_0:5128-5148
gives the grammar (opt_bv_decl/bv_decls/bvar).

One example in that group still fails, and deliberately: { |foo; foo| } is a
SyntaxError in MRI, raised by the PARSER (parse.y v3_4_0:13621, new_bv →
shadowing_lvar_0), and go-ruby-parser does not raise it yet. Nothing is forced
here on its behalf — localLookup answers with the first slot of a name, so the
repeated declaration is dead and the name keeps meaning the parameter. Raising
belongs where MRI raises it.

2. for — For.Target, the block's parameter shape, and the scope

Four things, all in the lowering of for.

For.Target. v0.2.0 puts every loop variable Vars cannot spell into
For.Target and leaves Vars empty — a lone non-local (for @v in …,
for a[0] in …), or any list with a *rest, a nested group, a non-local, or a
trailing comma. The compiler read only Vars, so for i, in [[1,2]] compiled a
loop over nothing.

The each-block's parameter shape. MRI picks it by the kind of loop variable
(prism_compile.c v3_4_0:6293-6301): a lone local target bumps lead_num, so the
block takes one required argument; anything else sets has_rest, so the block
collects every yielded value into an Array. rbgo always took one required
argument, so a for x, y in o over an each that does yield 1, 2 bound y to
nil where MRI binds 2. The single-non-local arm is MRI's expandarray 1, 0 on
that rest Array (prism_compile.c v3_4_0:4862-4880), which is why the loop
variable takes the FIRST yielded value and drops the others.

For the multi-target arm MRI emits (args.length == 1 && Array.try_convert(args[0])) || args
then expandarray (compile.c compile_for_masgn v3_4_0:8302-8329, and the
identical prism arm at prism_compile.c v3_4_0:4881-4919). The try_convert half
is redundant in front of an expandarray that already converts: when args holds
one element, expanding args[0] yields its Array conversion if it has one and the
one-element list [args[0]] otherwise — which is what expanding args gives. So
the same lowering is args.length == 1 ? args[0] : args, emitted with the same
idLength and idAREF sends MRI uses, and a redefined Array#length or
Array#[] is honoured here exactly as it is there.

for o&.target in m. storeMultiTarget ignored Call.Safe, so a nil receiver
raised NoMethodError instead of skipping the write. A multiple assignment can never
reach that branch (a&.x, b = … is a SyntaxError), but a for-var can:
prism_compile.c v3_4_0:4862 routes PM_CALL_TARGET_NODE through
pm_compile_target_node, which honours PM_CALL_NODE_FLAGS_SAFE_NAVIGATION.

for opens no scope. Its each-block is a block ISeq but not a Ruby scope, so a
variable FIRST ASSIGNED in the body belongs to the scope around the loop: after
for a in [6]; for b in [7]; c = a * b; end; end, c is a local of the enclosing
method. It hops out of enclosing for bodies but not out of an enclosing
block, which is a scope — the for_spec example "can be nested with blocks in
between" pins both halves. builder.declOwner names that scope, and every site
that declares a named local now goes through it (assignment, op-assignment, named
regexp captures, rescue => e), which also removes four copies of the
resolve-or-allocate dance.

3. defined? — the locals its operand declares, and two tags

defined? inspects its operand's syntactic kind instead of compiling it, so the
assignments inside emit nothing — but they still DECLARE. MRI builds its local
table in the PARSER, which walks the operand like any other code, so
defined?(a += 1) leaves a a nil local of the enclosing scope and a later bare
a is a local-variable read. rbgo declares as it compiles, so a got no slot and
the next mention of it — defined?(a.b += 1) six lines further on, whose receiver
the compiler DOES evaluate — failed to compile. Verified against MRI:
defined?(a += 1); p a prints nil.

Two tags were also wrong:

  • a.b += 1 and a[:b] ||= 1 reach the compiler as the parser's textual
    desugaring into a setter Call and were tagged "method". MRI tags them
    "assignment": PM_CALL_OPERATOR_WRITE_NODE, PM_CALL_OR_WRITE_NODE,
    PM_CALL_AND_WRITE_NODE and the PM_INDEX_* trio are all in the DEFINED_ASGN
    group (prism_compile.c v3_4_0:4052-4093; compile.c's
    NODE_OP_ASGN1/NODE_OP_ASGN2 at v3_4_0:6151-6164 says the same). A plain
    a[0] = 1 is NOT in that group — it is an ordinary attrasgn call and stays
    "method", which ruby/spec pins.
  • A ||= v and A::B ||= v desugar to (defined?(A) && A) || A = v and so
    arrived as a ||, tagged "expression". MRI tags them "assignment"
    (PM_CONSTANT_OR_WRITE_NODE / PM_CONSTANT_PATH_OR_WRITE_NODE,
    prism_compile.c v3_4_0:4059 and 4063; compile.c's NODE_OP_CDECL at
    v3_4_0:6162).

Both desugarings are recognised by AST pointer identity between the shared
read and the write, the test opassign.go already uses — a hand-written
(defined?(A) && A) || A = 1 has distinct reads and stays "expression", which MRI
confirms.

4. variables_spec — already fixed, and I can show it

The brief listed cannot compile *ast.SplatArg in language/variables_spec.rb as
the largest of the four. It does not reproduce, on either parser version.

main @ ddf7195 (parser v0.1.9):  RBGO_RESULT pass=113 fail=5 error=1
main @ 097d86a (parser v0.2.0):  RBGO_RESULT pass=113 fail=5 error=1
this branch:                     RBGO_RESULT pass=113 fail=5 error=1

Compiling every *_spec.rb under language/ turns up no cannot compile *ast.SplatArg anywhere. The wave-25 work had already closed it; the 113 examples
were the precedent the brief cites, not a pending item. Nothing to do, so nothing
was done — an honest three.

5. Found on the way: ... forwarding lost its block depth

mustResolve returned a slot and discarded the depth, so g(...) written inside
ANY block — def f(...); [1].each { g(...) }; end as much as a for body —
indexed the block's own frame and died with a Go index-out-of-range surfaced as
ArgumentError. Pre-existing and independent of for; for_spec is simply the file
that exposes it.

Gates

  • CGO_ENABLED=0 go build ./..., CGO_ENABLED=0 GOOS=wasip1 GOARCH=wasm go build ./..., gofmt -l internal/ empty.
  • CGO_ENABLED=0 GOWORK=off go test ./... — full suite green, TestEmbeddedPreludeMatchesSource included (nothing prelude.rb uses changed shape, so prelude_frozen_gen.go is untouched and still matches).
  • Coverage: every new function at 100% in the package profile; compared per
    function against main, nothing that was at 100% dropped and no function
    decreased at all. Total 98.4% → 98.7%.
  • Byte-for-byte against MRI 4.0.5 for every construct touched: the for shapes
    (including the hash, multi-yield, attribute, safe-navigation, index and
    leak-past-the-loop cases), block-locals, and the defined? tags and
    declarations — diff of raw output, identical.

Known-remaining, deliberately not in this PR

In defined_spec's 45 remaining failures, two groups dominate and neither is a
regression: defined?(super) answers "expression" instead of "super"/nil (18
examples — it needs a new opcode and VM support), and the tag Strings are not
frozen (about 12; half of that is one line in internal/vm/defined.go, outside
this branch's file set). for_spec's 5 errors are Kernel#local_variables and
#redo, neither implemented. assignments_spec wants a rescuable SyntaxError
where the compiler currently reports cannot compile *ast.BlockPass.

🤖 Generated with Claude Code

tannevaled and others added 8 commits September 22, 2026 13:51
The v0.2.0 parser makes 13 more language/*_spec.rb files of the pinned
corpus parseable, and adds For.Target and Block.Locals to the AST. None
of the compiler work in this branch is visible on v0.1.9.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`{ |a; b, c| … }` declares b and c as fresh, nil-valued locals of the
block that shadow any enclosing binding of the same name. The v0.2.0
parser carries them on Block.Locals; the compiler ignored the field, so
every mention of one resolved to the outer binding — or to nothing, and
language/block_spec.rb stopped at "undefined local variable" before a
single example ran.

MRI read: prism_compile.c v3_4_0:6325-6334, which walks the block
parameters node's `locals` list and calls pm_insert_local_index for each
right after the requireds/optionals/rest/posts and before the "fill in
any locals we missed" pass — so they are table entries, not parameters,
and start nil. parse.y v3_4_0:5128-5148 (opt_bv_decl/bv_decls/bvar) and
new_bv at 13621 show the parser already rejects a block-local that
shadows a parameter, so no dedup is needed at this end.

language/block_spec.rb: 0 -> 139 passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four things, all in the lowering of `for`:

1. For.Target. The v0.2.0 parser puts every loop variable Vars cannot
   spell into For.Target and leaves Vars empty — a lone non-local
   (`for @v in …`, `for a[0] in …`), or any list with a `*rest`, a
   nested group, a non-local, or a trailing comma. The compiler read
   only Vars, so `for i, in [[1,2]]` compiled a loop over nothing and
   language/for_spec.rb stopped at "undefined local variable".

2. The each-block's parameter shape. MRI picks it by the kind of loop
   variable (prism_compile.c v3_4_0:6293-6301): a lone local target
   bumps lead_num, so the block takes one required argument; anything
   else sets has_rest, so the block collects every yielded value into an
   Array. rbgo always took one required argument, so a `for x, y in o`
   over an `each` that does `yield 1, 2` bound y to nil where MRI binds
   2. The single-non-local arm is MRI's `expandarray 1, 0` on that rest
   Array (prism_compile.c v3_4_0:4862-4880), which is why the loop
   variable takes the FIRST yielded value and drops the others.

   For the multi-target arm MRI emits `(args.length == 1 &&
   Array.try_convert(args[0])) || args` then expandarray (compile.c
   compile_for_masgn v3_4_0:8302-8329, prism_compile.c v3_4_0:4881-4919).
   The try_convert half is redundant in front of an expandarray that
   already converts: when args holds one element, expanding args[0]
   yields its Array conversion if it has one and the one-element list
   [args[0]] otherwise — which is what expanding args gives. So the same
   lowering is `args.length == 1 ? args[0] : args`, emitted with the same
   idLength and idAREF sends MRI uses.

3. `for o&.target in m`. storeMultiTarget ignored Call.Safe, so a nil
   receiver raised NoMethodError instead of skipping the write. A
   multiple assignment can never reach that branch (`a&.x, b = …` is a
   SyntaxError), but a for-var can: prism_compile.c v3_4_0:4862 routes
   PM_CALL_TARGET_NODE through pm_compile_target_node, which honours
   PM_CALL_NODE_FLAGS_SAFE_NAVIGATION.

4. `for` opens no scope. The each-block is a block ISeq but not a Ruby
   scope, so a variable FIRST ASSIGNED in the body belongs to the scope
   around the loop: after `for a in [6]; for b in [7]; c = a * b; end;
   end`, c is a local of the enclosing method. It hops out of enclosing
   `for` bodies but not out of an enclosing block, which is a scope — the
   for_spec "can be nested with blocks in between" example pins both
   halves. builder.declOwner names that scope; every site that declares a
   named local now goes through it (assignment, op-assignment, named
   regexp captures, `rescue => e`), which also removes four copies of the
   resolve-or-allocate dance.

Also fixed on the way, because for_spec is the file that exposes it:
mustResolve dropped the block depth, so `g(...)` argument forwarding
inside ANY block — `def f(...); [1].each { g(...) }; end` as much as a
`for` body — indexed the block's own frame and crashed with a Go
index-out-of-range surfaced as ArgumentError.

language/for_spec.rb: 0 -> 29 passing, 0 failing. The 5 remaining errors
are Kernel#local_variables and #redo, neither implemented, neither in
this branch's files. language/rescue_spec.rb: 49 -> 50.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… desugared compound assignment

`defined?` inspects its operand's syntactic kind instead of compiling it,
so the assignments inside it emit nothing — but they still DECLARE. MRI
builds its local table in the PARSER, which walks the operand like any
other code, so `defined?(a += 1)` leaves `a` a nil local of the enclosing
scope and a later bare `a` is a local-variable read. rbgo declares as it
compiles, so `a` got no slot and the next mention of it — in
language/defined_spec.rb, `defined?(a.b += 1)` six lines further on,
whose receiver the compiler DOES evaluate — failed with "undefined local
variable". Verified against MRI: `defined?(a += 1); p a` prints nil, and
`p defined?(a = 1), a.inspect` prints "assignment" then "nil".

Second, `a.b += 1` and `a[:b] ||= 1` reach the compiler as the parser's
textual desugaring into a setter Call, and were tagged "method". MRI tags
them "assignment": PM_CALL_OPERATOR_WRITE_NODE, PM_CALL_OR_WRITE_NODE,
PM_CALL_AND_WRITE_NODE and the PM_INDEX_* trio are all in the
DEFINED_ASGN group (prism_compile.c v3_4_0:4052-4093; compile.c's
NODE_OP_ASGN1/NODE_OP_ASGN2 at v3_4_0:6151-6164 say the same). A plain
`a[0] = 1` is NOT in that group — it is an ordinary attrasgn call and
stays "method", which ruby/spec pins. opassign.go's shared-node
recognisers tell the two apart.

language/defined_spec.rb: 0 -> 211 passing.

Of the 47 that remain, two groups are worth naming: `defined?(super)`
answers "expression" instead of "super"/nil (18 examples), and the tag
Strings are not frozen (about 12). Half of the latter is a one-line
change in internal/vm/defined.go, outside this branch's files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nches

Covers every new branch in the package profile: the For.Target shapes and
where each declares its locals, the each-block parameter shape and the two
target lowerings, the safe-navigation guard, block-local slots, the
defined? declaration walk, and the depth mustResolve had been dropping.

Also drops an unreachable guard: compileFor's declare() skipped an empty
name, but every For.Target shape that carries one (`for i, * in …`) also
carries a parallel Targets list with a typed nil in that position, so the
all-locals arm — the only caller that passes Names straight through — never
sees it. An unreachable branch cannot be covered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`A ||= v` and `A::B ||= v` are not assignment nodes: the parser desugars
them to `(defined?(A) && A) || A = v`, so they reached compileDefined as a
`||` and were tagged "expression". MRI tags them "assignment" —
PM_CONSTANT_OR_WRITE_NODE and PM_CONSTANT_PATH_OR_WRITE_NODE are in the
DEFINED_ASGN group (prism_compile.c v3_4_0:4059 and 4063; compile.c's
NODE_OP_CDECL at v3_4_0:6162 agrees) — and ruby/spec pins both. The `&&=`
and `+=` forms already arrived as ConstAssign/ScopedConstAssign and were
right.

Recognised by AST pointer identity between the `defined?` probe's
argument, the `&&`'s right operand and the scoped assignment's target,
the same test opassign.go uses: a HAND-WRITTEN `(defined?(A) && A) || A =
1` has three distinct reads and stays "expression", which MRI confirms.

language/defined_spec.rb: 211 -> 213 passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The note said the parser rejects a block-local that repeats a parameter
name. MRI's does (parse.y v3_4_0:13621, new_bv -> shadowing_lvar_0);
go-ruby-parser does not yet, which is the one ruby/spec example in
"Block-local variables" still failing. Nothing is forced here on its
behalf: localLookup answers with the first slot of a name, so the
repeated declaration is dead and the name keeps meaning the parameter.
Comment only; no behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit b8cd4df into main Sep 23, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave26/compiler branch September 23, 2026 07:38
tannevaled added a commit that referenced this pull request Sep 23, 2026
Seven units since the last bump:

  #617  core/process: spawn, exec, the rlimits, priorities, process groups,
        kill, the wait family                     (pass 20 -> 165)
  #618  core/kernel + core/string: require_relative with an absolute path,
        the require/load argument contract, LoadError#path, byteindex/
        byterindex, split, chop/chars/each_char   (kernel 233 -> 184,
                                                   string 179 -> 153 failing)
  #620  language: compound assignment evaluation order, the flip-flop
        operator, magic-comment encodings, loop break values
                                                  (pass 1308 -> 1443)
  #622  object: Float#to_s laid out by MRI's rule rather than Go's %g  (+11)
  #624  the mspec shim: a nested describe can see the helpers of the one
        around it                                                     (+28)
  #625  deps: parser v0.2.0                        (language +333)
  #626  compiler: for-loop targets, block-local variables, what defined?
        declares                                   (language +382)

Measured with scripts/conformance/rubyspec/run.sh against the pinned corpus at
87b1631, 16-way, TIMEOUT=25, three consecutive runs: 21749, 21750, 21750. The
floor is set nine below the lowest, the same margin rule as the last bump: it
absorbs the run-to-run jitter without being wide enough to hide the loss of a
whole spec file, which is what the ratchet exists to catch. #615 (an
intermittent frame-pop crash) is still open and can cost a file when it fires.

TWO OF THESE ARE NOT VM CONFORMANCE, and the number should not be read as if
they were. #624 fixed our own mspec shim: core/string/valid_encoding/utf_8_spec
scored 0/28 under MRI 4.0.5 too, so those 28 were harness noise counted as VM
failure. #625 is a dependency bump; the VM did not change, but thirteen
language spec files became parseable for the first time and brought 333 already-
passing examples into view. Both are real gains in what the measurement can
see, not in what the VM can do.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant