Skip to content

Close the nine syntax gaps of #27, the mis-parse, and the two unreduced files - #28

Merged
tannevaled merged 14 commits into
mainfrom
wave26/syntax
Sep 21, 2026
Merged

tannevaled merged 14 commits into
mainfrom
wave26/syntax

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Closes #27 — all nine parse errors and the mis-parse, plus the two files the
issue deliberately left unclaimed, plus four further gaps each of them uncovered.

Every one of the 68 files in the pinned ruby/spec language/ corpus now
parses (13 did not before), and so do 4405 of the 4413 .rb files in the
whole corpus, up from 4391 — the 8 that remain include two deliberate
syntax-error fixtures. No file that parsed before stopped parsing.

Each fix reads the MRI rule first; the commit messages cite it by name and line
in parse.y v3_4_0, and every behaviour was checked against the installed MRI
4.0.5, comparing values through rbgo rather than merely "it parses".

What the issue filed

# source rule read in parse.y v3_4_0 commit
1, 2 alias :"#{:a}" v / undef :"#{:m}" `fitem: fname symbol, symbol: ssym
3 (o).a, (o).a = 1, 2 mlhs_node: primary_value call_op tIDENTIFIER (3639-3652) — the receiver is any primary_value ac7e3a1
4 defined?($zz and true) keyword_defined '\n'? '(' begin_defined expr rparen (4458), a primary; and keyword_defined '\n'? begin_defined arg (4039) c1df337
5 for i, in [[1,2]] `for_var: lhs mlhs, with mlhs_head: mlhs_item ','` a complete mlhs
6 p({:a==>1}) parse_ident: `c=='=' && … (!peek('=')
7 in Array(0,1,2) p_expr_basic: p_const p_lparen p_args rparen — the same quartet as p_lbracket afab703
8 p $= parse_gvar's tGVAR arm lists _ ~ * $ ? ! @ / \ ; , . = : < > " 05f6a4c
9 -> a=a() { a } lambda_beginning_p() = lpar_beg == paren_nest, set at the -> 9b7c51b
mis-parse o.s (:a){ 1 } tLPAREN_ARG … {SET_LEX_STATE(EXPR_ENDARG);} ')' (4405) then command: … command_args cmd_brace_block (3464) 1ba2c43

The mis-parse is the one that mattered most: it was silent. o.s (:a){ 1 }
bound the block to :a, so rbgo raised LocalJumpError where MRI prints 1. The
negative case is pinned too — f g(1) { 2 } uses a HUGGING paren, so its )
leaves EXPR_END and MRI gives the block to g(1); that still happens.

The two files the issue could not reduce

Both reduce, and neither error was at the line it pointed to in the way it
looked:

  • class_spec.rb (and metaclass_spec.rb, same cause): class C; end.should == nil.
    The parser let a definition take a .method and a modifier but not be an
    OPERAND. In MRI k_class cpath superclass bodystmt k_end, the class << expr
    form, k_module cpath bodystmt k_end and defn_head f_arglist bodystmt k_end
    are all alternatives of primary (4599-4657). Removing the statement-level
    interception was the whole fix (9ce5aec).
  • string_spec.rb: %@hey #{@ip}@. The %-literal terminator scan looked
    INSIDE the interpolation and closed on the @ of @ip. MRI's tokadd_string
    leaves the scan at #{ and hands the embedded expression to the main lexer.
    Two more %-delimiter faults were behind it: %=hey= (MRI tests IS_BEG()
    before the op-assign =, so = delimits a literal where a value is expected
    and stays op-assign after one) and %\hey\ (the terminator test precedes the
    backslash test in tokadd_string) (92c7bd2).

Found on the way, each blocking the next line of a spec file

  • defined?(break), defined?(return): a bare break/next/retry is a
    primary (4682-4697) and so is a bare return (k_return, 4438), with the
    call_args forms alternatives of command (c1df337).
  • in [0, 1, ] parsed but silently became the EXACT pattern [0, 1], so a
    case [0,1,2,3] fell to else where MRI matches. p_top_expr_body: p_expr ','
    makes a trailing comma an ANONYMOUS REST (afab703).
  • in {"a": 0}: p_kw_label: tLABEL | tSTRING_BEG string_contents tLABEL_END.
    in {"a" => 1} stays refused, as ruby/spec requires (afab703).
  • in ^@a: p_var_ref: '^' tIDENTIFIER | '^' nonlocal_var (afab703).
  • |a, (b, (c, d))|: f_marg: f_norm_arg | tLPAREN f_margs rparen nests, but
    parseBlockParams held a flat copy of logic the def list already had
    recursively (1ba2c43).

AST changes, all additive

ast.Alias gains NewNameExpr/OldNameExpr, ast.Undef gains Exprs,
ast.Block gains Locals, ast.For gains Target. Each is nil/empty for the
shapes that worked before, so existing field reads keep their meaning and
existing consumers keep building. ast.For.Target is the one a consumer must
learn: Vars is EMPTY whenever it is set, so there is only ever one place to
read the loop variable from, and for i in … / for a, b in … still arrive in
Vars unchanged.

Measurement

rbgo built against this branch, over the pinned corpus, before and after.
Ten files went from "no result at all" to a measured result, revealing
311 passing examples that no measurement had ever seen:

file now
pattern_matching_spec.rb pass=95 fail=16 error=2
predefined_spec.rb pass=68 fail=89 error=4
class_spec.rb pass=32 fail=8 error=3
hash_spec.rb pass=29 fail=11 error=0
string_spec.rb pass=23 fail=9 error=2
alias_spec.rb pass=22 fail=2 error=2
metaclass_spec.rb pass=18 fail=0 error=3
assignments_spec.rb pass=18 fail=20 error=0
undef_spec.rb pass=6 fail=0 error=2
lambda_spec.rb pass=0 fail=0 error=2 (its SpecEvaluate helper is missing from the corpus harness)

Corpus totals: pass 1031 → 1342, files with no result 14 → 4. Nothing regressed:
the before/after diff over all 68 files contains only these additions.

The four files still reporting nothing all PARSE now and stop in rbgo's
compiler, not here:

  • for_spec.rb — undefined local variable "j": rbgo reads For.Vars, which is
    empty for for i, *j in …. It needs to read For.Target.
  • block_spec.rb — undefined local variable "bl": block-locals need a slot.
    Block.Locals now carries the names (74ed65f); rbgo has to allocate them.
  • defined_spec.rb — undefined local variable "a": defined?(a += 1) declares
    the local at parse time, as MRI does, but rbgo's compiler allocates no slot for
    it. Reproduces identically on main's parser, so it is pre-existing.
  • variables_spec.rb — cannot compile *ast.SplatArg.

One more consumer-side finding, from assignments_spec.rb: rbgo evaluates the
right-hand side of a multiple assignment before the target receivers, where MRI
evaluates the receivers first. The AST records the two sides, not an order, so
that is a compiler matter.

Gates

go build ./..., go vet ./..., go test -race ./... all green; the 100%
coverage gate holds (and peek3's end-of-input branch, which the ROUNDED total
was hiding at 100.0%, is now covered — 60303c5). gofmt -l . lists only
paren_singleton_def_test.go, which is untouched and already listed on main.

🤖 Generated with Claude Code

tannevaled and others added 14 commits September 21, 2026 21:38
Three related faults in the %-literal scan, all found by `language/string_spec.rb`,
which now parses (it stopped at line 71 before).

1. The terminator scan looked inside `#{…}`, so `%@Hey #{@ip}@` closed on the
   `@` of `@ip`. MRI's `tokadd_string` (parse.y v3_4_0) leaves the literal scan
   at a `#` that introduces an interpolation and hands the embedded expression
   to the main lexer, so the terminator is never looked for inside one. The
   brace-balanced interpolation is now copied verbatim, in %Q/%() strings and in
   the interpolating %W/%I word lists (%r/%x/%s already did this).

2. `%=hey=` was lexed as the `%=` op-assign. MRI's `parse_percent` tests
   `IS_BEG()` first and only falls through to `if ((c = nextc(p)) == '=') …
   return tOP_ASGN;` away from expression-begin, so `=` delimits a literal where
   a value is expected and stays op-assign after one. `percentDelimOK` now makes
   that distinction; `isPercentDelim` keeps only the state-free rules.

3. `%\hey\` treated the closing `\` as an escape. In `tokadd_string` the
   terminator test precedes the backslash test, so a backslash delimiter closes
   the literal; an escaped delimiter inside a differently-delimited literal
   (`%(a\)b)`) still escapes, because the escape consumes its byte before the
   loop top is reached again.

Verified against MRI 4.0.5: `%@Hey #{@ip}@`, `%=hey=`, `%\hey\`, `%'hey'`,
`%$hey$`, `%w=a b=`, `%q(a\)b)`, `a %= 2`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ent form

`class C; end.should == nil` and `class << true; self; end.should == TrueClass`
failed with `unexpected "=="`: parseStatement intercepted CLASS/MODULE/DEF and
ran only the postfix and modifier layers over the result, so the definition
could take a `.method` but could not be an operand.

In MRI these are alternatives of `primary` (parse.y v3_4_0, lines 4599-4657:
`k_class cpath superclass bodystmt k_end`, `k_class tLSHFT expr_value term
bodystmt k_end`, `k_module cpath bodystmt k_end`, `defn_head f_arglist bodystmt
k_end`), so they compose with everything a primary composes with. parsePrimary
already handled all three; the statement-level interception was what kept them
from being reached through the expression path. Removing it routes them through
parseKeywordLogical → parseExprOrAssign → … → parsePrimary, which still applies
the postfix tail and the modifiers, and additionally the operator layers.

Unblocks language/class_spec.rb (32 passing examples) and
language/metaclass_spec.rb (18), both of which parse for the first time.

Verified against MRI 4.0.5.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Routing class/module/def through the expression path moved their `rescue`
modifier to withRescueModifier. `alias`/`undef` are the remaining statements
that never enter the expression layers (MRI parse.y v3_4_0: `stmt: keyword_alias
fitem fitem | keyword_undef undef_list`), so they are what still reaches
applyModifiers' RESCUE arm. Pin it; the 100% coverage gate had slipped to 99.9%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`alias :"#{'a'}" value` and `undef :"#{'meth'}"` failed with `expected a method
name`: parseFitem accepted only a SYMBOL, a GVAR or a bare fname.

MRI's `fitem: fname | symbol` with `symbol: ssym | dsym` and
`dsym: tSYMBEG string_contents tSTRING_END` (parse.y v3_4_0) makes an
interpolated symbol an ordinary alias/undef name, whose value is only known at
run time. The lexer already desugars `:"…#{…}…"` into `"…".to_sym`, so parseFitem
now takes exactly that — the string concatenation plus the `.to_sym` call, and
no further token, since the name after an alias's first item (`alias :"#{x}" v`)
must not be read as a paren-less argument of to_sym.

ast.Alias gains NewNameExpr/OldNameExpr and ast.Undef gains Exprs, all nil for
the static forms, so the existing string fields keep their meaning and existing
consumers keep building.

Unblocks language/alias_spec.rb and language/undef_spec.rb, which now parse.

Verified against MRI 4.0.5: `class C; def v; 5; end; alias :"#{:a}" v; end`
answers 5 to `C.new.a`, and the undef form removes the method.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`p $=` failed with `unexpected token "$"`. MRI's parse_gvar (parse.y v3_4_0)
lists `=` in the switch arm that lexes a single punctuation character as tGVAR —
`_ ~ * $ ? ! @ / \ ; , . = : < > "` — with a second arm for the back-references
`& ` ' +`. Every one of those was already accepted here except `=`, the obsolete
ignore-case flag, which `language/predefined_spec.rb` reads.

Unblocks language/predefined_spec.rb, which now parses.

Verified against MRI 4.0.5: `p $=` prints false.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rimaries

`defined?($zz and true)` failed with `expected ), got "and"`: `defined?` was
parsed as an ordinary method call, so its argument was a call argument.

MRI gives `defined?` two dedicated productions (parse.y v3_4_0):
`keyword_defined '\n'? '(' begin_defined expr rparen` at 4458, an alternative of
`primary`, and `keyword_defined '\n'? begin_defined arg` at 4039, an alternative
of `arg`. The parenthesised form takes a full `expr` — which includes
`expr keyword_and expr`, `expr keyword_or expr` and `keyword_not '\n'? expr` —
but one expr only, not a statement sequence (`defined?($zz; $x)` is a syntax
error in MRI), and only a hugging parenthesis selects it: MRI answers
"expression" to `defined? (1) && nil` and nil to `defined?(1) && nil`, because a
space leaves the parenthesis to open the `arg`. All four properties are pinned
by tests.

Reaching further into language/defined_spec.rb then needed the jump keywords in
expression position. A bare `break`/`next`/`redo`/`retry` is an alternative of
`primary` (4682-4697) and so is a bare `return` (`k_return`, 4438), while
`k_return call_args` / `keyword_break call_args` / `keyword_next call_args` are
alternatives of `command` (3518-3534); the parser had them only as statement
forms. `defined?(break)`, `defined?(return)` and `defined?(break 1)` are the
shapes ruby/spec uses. `redo` has no token here and still lexes as an
identifier, which parses.

Unblocks language/defined_spec.rb, which now parses.

Verified against MRI 4.0.5.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`{:a==>1}` failed with `unexpected token ">"`: the lexer refused the trailing
`=` of a setter symbol whenever another `=` followed, so `:a==>1` lexed as
`:a`, `==`, `>`.

MRI's parse_ident (parse.y v3_4_0) is finer:

	c == '=' && IS_lex_state(EXPR_FNAME) &&
	 (!peek(p,'~') && !peek(p,'>') && (!peek(p,'=') || peek_n(p,'>',1)))

The `=` joins the name unless it opens `=~` or `=>`, or unless it opens `==`
that is NOT itself followed by `>`. That last clause is the one that was
missing, and it is what makes `{:a==>1}` the pair `:a= => 1` while
`{:a ==> 1}` — where the symbol has already ended at the space — stays a syntax
error, as language/hash_spec.rb requires of both.

Unblocks language/hash_spec.rb, which now parses.

Verified against MRI 4.0.5: `p({:a==>1})` prints {"a=": 1}.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 100% gate reads the ROUNDED total, which stayed at 100.0% while peek3's
out-of-range return had never run. `:a==` at end of input is the source that
reaches it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…block

`-> a=a() { a }` failed with `expected {, got "\n"`: the brace was attached to
the `a()` call in the parameter default, leaving nothing to open the lambda
body.

MRI decides this in the lexer, by bracket nesting. At a `->` it records
`p->lex.lpar_beg = p->lex.paren_nest`, and a later `{` returns tLAMBEG — the
lambda body — exactly when `lambda_beginning_p()`
(`p->lex.lpar_beg == p->lex.paren_nest`) holds (parse.y v3_4_0). paren_nest
counts `(`, `[` and `{`, so the rule is: while the unparenthesised parameter
list is open and no bracket is, the next `{` is the body.

The parser now carries noBraceBlock, set over an unparenthesised stabby-lambda
parameter list and cleared by enterBracket wherever a bracket is opened (a
parenthesised group, a `(…)`/`[…]` argument list, a parenthesised parameter
list). That reproduces both sides: `-> x = ([1].map { |v| v }) { x }` keeps its
inner block, while the same expression without the parentheses is a syntax
error in MRI too, which is why the flag is not cleared for a bracket that has
already closed.

Unblocks language/lambda_spec.rb, which now parses.

Verified against MRI 4.0.5: `def a; 1; end; p(-> a=a() { a }.call)` prints 1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…block

Two faults, both found by language/block_spec.rb, which now parses.

1. `o.s (:a){ 1 }` bound the `{ … }` to the argument instead of to the call,
   silently — rbgo then raised LocalJumpError where MRI prints 1. In MRI a
   SPACED `(` in argument position lexes as tLPAREN_ARG and its `)` sets
   EXPR_ENDARG (parse.y v3_4_0: `primary: tLPAREN_ARG compstmt
   {SET_LEX_STATE(EXPR_ENDARG);} ')'`), so the `{` right after it lexes as
   tLBRACE_ARG and the grammar hands it to the command through
   `command: primary_value call_op operation2 command_args cmd_brace_block`
   (3464-3480). The parser now records where that `)` is (argParenEnd) and
   attaches the brace block to the command instead of letting the postfix chain
   take it. A HUGGING paren is unaffected — its `)` leaves EXPR_END, which is
   why MRI gives the block in `f g(1) { 2 }` to `g(1)` and raises from `f`;
   that negative case is pinned too.

2. `|a, (b, (c, d))|` failed with `expected IDENT, got "("`. A destructuring
   block parameter nests in MRI — `f_marg: f_norm_arg | tLPAREN f_margs rparen`
   puts f_marg back inside f_margs — but parseBlockParams carried a flat copy of
   the logic that could only read identifiers one level down. It now calls
   parseDestructureParam, the recursive implementation the `def` parameter list
   already used, which also makes `|((a, b), c)|` and `|(a, *b)| ` parse.

Verified against MRI 4.0.5, values compared through rbgo for each case.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`{ |one; bl| bl }` parsed, but the names after the `;` were declared in the
parser's scope and then dropped, so `bl` in the body became an ast.VarRef the
consumer had no slot for — rbgo answers `compile error: undefined local
variable "bl"` where MRI prints nil. The parser's half of that is the loss of
the list.

MRI's `opt_bv_decl: '\n'? ';' bv_decls '\n'?` with
`bvar: tIDENTIFIER { new_bv(p, $1); }` (parse.y v3_4_0) adds each name to the
BLOCK's own local table, where it starts nil and shadows any enclosing binding.
ast.Block gains Locals, nil when the list has no `;` part, so a consumer can
allocate them.

This is what language/block_spec.rb line 782 needs; the file now parses (see the
preceding commit) and reaches its compiler.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All four are MRI grammar alternatives the parser did not have. They are
sequential: each one was the next thing language/pattern_matching_spec.rb
stopped at. The file now parses.

1. `in Array(0, 1, 2)` failed with `expected LABEL, got "0"` — a `Const(…)`
   pattern was always read as a hash pattern. In MRI the parenthesised and the
   bracketed const pattern have the SAME four alternatives (parse.y v3_4_0,
   p_expr_basic): `p_const p_lparen p_args rparen`, `… p_find rparen`,
   `… p_kwargs rparen`, `p_const '(' rparen`, mirroring p_lbracket/rbracket. The
   body decides, not the bracket, so parseArrayPattern is now parameterised by
   its bracket pair and serves both.

2. A trailing comma was dropped. `in [0, 1, ]` parsed but became the EXACT
   pattern `[0, 1]`, so `case [0,1,2,3]` took the else branch where MRI matches
   — a wrong answer with no diagnostic. MRI's `p_top_expr_body: p_expr ','`
   makes it an array pattern with an ANONYMOUS REST, matching any array that
   starts with the listed elements. The un-bracketed spelling `in 0, 1,;` is the
   same rule and was a parse error; both now build the rest.

3. `in {"a": 0}` failed. p_kw_label has two spellings — `tLABEL` and
   `tSTRING_BEG string_contents tLABEL_END` — so a quoted string followed by `:`
   is the same symbol key. `in {"a" => 1}` stays a parse error, which is what
   ruby/spec asserts.

4. `in ^@a` failed. `p_var_ref: '^' tIDENTIFIER | '^' nonlocal_var` with
   `nonlocal_var: tIVAR | tGVAR | tCVAR` (5883-5897) pins a non-local too.

Verified against MRI 4.0.5, values compared through rbgo for each case
including the negative ones (`in [0, 1]` stays exact, `in {"a" => 1}` refused).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`(o).a, (o).a = 1, 2` failed with `unexpected "," after statement`: a `(` at the
start of a target was always taken for a nested target group, so `(o).a` ended
the scan.

MRI's mlhs_node takes any primary_value as the receiver —
`primary_value call_op tIDENTIFIER`, `primary_value '[' opt_call_args rbracket`,
`primary_value tCOLON2 tIDENTIFIER` (parse.y v3_4_0, 3639-3652) — and a
parenthesised expression is a primary. looksLikeMlhs and parseMlhsTarget now
look past the balanced `(…)`: a postfix chain after it makes the group a
RECEIVER, anything else leaves it the nested target list it was. So
`(a, b), c = [1, 2], 3` is unchanged and `(o).a = 1` stays a plain attribute
assignment rather than becoming a one-target masgn.

Unblocks language/assignments_spec.rb, which now parses. ruby/spec uses this
shape to pin evaluation order — `(ScratchPad << :a; obj).a, … = …` — and rbgo
still evaluates the right-hand side before the target receivers, which is a
compiler matter: the AST records the two sides, not an order.

Verified against MRI 4.0.5 through rbgo: same targets, same values.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`for i, in [[1,2]]` failed with `expected IDENT, got "in"`, and so did every
other loop-variable shape beyond a comma-separated list of bare names:
`for @v in`, `for $v in`, `for C in`, `for o.a in`, `for o&.a in`,
`for a[0] in`, `for i, *j in`, `for (i, j, k) in`, `for i, (j, k) in`,
`for (i, j), k in`.

MRI's loop variable is `for_var: lhs | mlhs` (parse.y v3_4_0) — any assignment
target, or the same target list a multiple assignment takes, `mlhs_basic` and
all. parseFor now parses it with the mlhs machinery instead of a loop over
IDENTs.

Two distinctions the grammar makes and the AST has to keep:

  - `mlhs_head: mlhs_item ','` is a COMPLETE mlhs, so a trailing comma means
    destructuring: `for i, in [[1,2]]` binds i to 1 where `for i in [[1,2]]`
    binds it to [1,2]. MRI 4.0.5 prints 1 and [1, 2] respectively.
  - a SINGLE target is `for_var: lhs`, not a one-element `mlhs`, so it binds the
    whole element: `for @v in [[1,2]]` sets @v to [1, 2].

ast.For gains Target for everything Vars cannot spell. Vars is EMPTY whenever
Target is set, so a consumer reads the loop variable from exactly one place, and
the shapes that already worked — `for i in …`, `for a, b in …` — still arrive in
Vars unchanged.

Unblocks language/for_spec.rb. With this every one of the 68 files in the pinned
ruby/spec language corpus parses.

Verified against MRI 4.0.5.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 097d69c into main Sep 21, 2026
11 checks passed
@tannevaled
tannevaled deleted the wave26/syntax branch September 21, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nine syntax gaps that make 12 ruby/spec language files unparseable

1 participant