Repository navigation
Close the nine syntax gaps of #27, the mis-parse, and the two unreduced files - #28
Merged
Merged
Conversation
Three related faults in the %-literal scan, all found by `language/string_spec.rb`,
which now parses (it stopped at line 71 before).
1. The terminator scan looked inside `#{…}`, so `%@Hey #{@ip}@` closed on the
`@` of `@ip`. MRI's `tokadd_string` (parse.y v3_4_0) leaves the literal scan
at a `#` that introduces an interpolation and hands the embedded expression
to the main lexer, so the terminator is never looked for inside one. The
brace-balanced interpolation is now copied verbatim, in %Q/%() strings and in
the interpolating %W/%I word lists (%r/%x/%s already did this).
2. `%=hey=` was lexed as the `%=` op-assign. MRI's `parse_percent` tests
`IS_BEG()` first and only falls through to `if ((c = nextc(p)) == '=') …
return tOP_ASGN;` away from expression-begin, so `=` delimits a literal where
a value is expected and stays op-assign after one. `percentDelimOK` now makes
that distinction; `isPercentDelim` keeps only the state-free rules.
3. `%\hey\` treated the closing `\` as an escape. In `tokadd_string` the
terminator test precedes the backslash test, so a backslash delimiter closes
the literal; an escaped delimiter inside a differently-delimited literal
(`%(a\)b)`) still escapes, because the escape consumes its byte before the
loop top is reached again.
Verified against MRI 4.0.5: `%@Hey #{@ip}@`, `%=hey=`, `%\hey\`, `%'hey'`,
`%$hey$`, `%w=a b=`, `%q(a\)b)`, `a %= 2`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ent form `class C; end.should == nil` and `class << true; self; end.should == TrueClass` failed with `unexpected "=="`: parseStatement intercepted CLASS/MODULE/DEF and ran only the postfix and modifier layers over the result, so the definition could take a `.method` but could not be an operand. In MRI these are alternatives of `primary` (parse.y v3_4_0, lines 4599-4657: `k_class cpath superclass bodystmt k_end`, `k_class tLSHFT expr_value term bodystmt k_end`, `k_module cpath bodystmt k_end`, `defn_head f_arglist bodystmt k_end`), so they compose with everything a primary composes with. parsePrimary already handled all three; the statement-level interception was what kept them from being reached through the expression path. Removing it routes them through parseKeywordLogical → parseExprOrAssign → … → parsePrimary, which still applies the postfix tail and the modifiers, and additionally the operator layers. Unblocks language/class_spec.rb (32 passing examples) and language/metaclass_spec.rb (18), both of which parse for the first time. Verified against MRI 4.0.5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Routing class/module/def through the expression path moved their `rescue` modifier to withRescueModifier. `alias`/`undef` are the remaining statements that never enter the expression layers (MRI parse.y v3_4_0: `stmt: keyword_alias fitem fitem | keyword_undef undef_list`), so they are what still reaches applyModifiers' RESCUE arm. Pin it; the 100% coverage gate had slipped to 99.9%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`alias :"#{'a'}" value` and `undef :"#{'meth'}"` failed with `expected a method
name`: parseFitem accepted only a SYMBOL, a GVAR or a bare fname.
MRI's `fitem: fname | symbol` with `symbol: ssym | dsym` and
`dsym: tSYMBEG string_contents tSTRING_END` (parse.y v3_4_0) makes an
interpolated symbol an ordinary alias/undef name, whose value is only known at
run time. The lexer already desugars `:"…#{…}…"` into `"…".to_sym`, so parseFitem
now takes exactly that — the string concatenation plus the `.to_sym` call, and
no further token, since the name after an alias's first item (`alias :"#{x}" v`)
must not be read as a paren-less argument of to_sym.
ast.Alias gains NewNameExpr/OldNameExpr and ast.Undef gains Exprs, all nil for
the static forms, so the existing string fields keep their meaning and existing
consumers keep building.
Unblocks language/alias_spec.rb and language/undef_spec.rb, which now parse.
Verified against MRI 4.0.5: `class C; def v; 5; end; alias :"#{:a}" v; end`
answers 5 to `C.new.a`, and the undef form removes the method.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`p $=` failed with `unexpected token "$"`. MRI's parse_gvar (parse.y v3_4_0) lists `=` in the switch arm that lexes a single punctuation character as tGVAR — `_ ~ * $ ? ! @ / \ ; , . = : < > "` — with a second arm for the back-references `& ` ' +`. Every one of those was already accepted here except `=`, the obsolete ignore-case flag, which `language/predefined_spec.rb` reads. Unblocks language/predefined_spec.rb, which now parses. Verified against MRI 4.0.5: `p $=` prints false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rimaries
`defined?($zz and true)` failed with `expected ), got "and"`: `defined?` was
parsed as an ordinary method call, so its argument was a call argument.
MRI gives `defined?` two dedicated productions (parse.y v3_4_0):
`keyword_defined '\n'? '(' begin_defined expr rparen` at 4458, an alternative of
`primary`, and `keyword_defined '\n'? begin_defined arg` at 4039, an alternative
of `arg`. The parenthesised form takes a full `expr` — which includes
`expr keyword_and expr`, `expr keyword_or expr` and `keyword_not '\n'? expr` —
but one expr only, not a statement sequence (`defined?($zz; $x)` is a syntax
error in MRI), and only a hugging parenthesis selects it: MRI answers
"expression" to `defined? (1) && nil` and nil to `defined?(1) && nil`, because a
space leaves the parenthesis to open the `arg`. All four properties are pinned
by tests.
Reaching further into language/defined_spec.rb then needed the jump keywords in
expression position. A bare `break`/`next`/`redo`/`retry` is an alternative of
`primary` (4682-4697) and so is a bare `return` (`k_return`, 4438), while
`k_return call_args` / `keyword_break call_args` / `keyword_next call_args` are
alternatives of `command` (3518-3534); the parser had them only as statement
forms. `defined?(break)`, `defined?(return)` and `defined?(break 1)` are the
shapes ruby/spec uses. `redo` has no token here and still lexes as an
identifier, which parses.
Unblocks language/defined_spec.rb, which now parses.
Verified against MRI 4.0.5.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`{:a==>1}` failed with `unexpected token ">"`: the lexer refused the trailing
`=` of a setter symbol whenever another `=` followed, so `:a==>1` lexed as
`:a`, `==`, `>`.
MRI's parse_ident (parse.y v3_4_0) is finer:
c == '=' && IS_lex_state(EXPR_FNAME) &&
(!peek(p,'~') && !peek(p,'>') && (!peek(p,'=') || peek_n(p,'>',1)))
The `=` joins the name unless it opens `=~` or `=>`, or unless it opens `==`
that is NOT itself followed by `>`. That last clause is the one that was
missing, and it is what makes `{:a==>1}` the pair `:a= => 1` while
`{:a ==> 1}` — where the symbol has already ended at the space — stays a syntax
error, as language/hash_spec.rb requires of both.
Unblocks language/hash_spec.rb, which now parses.
Verified against MRI 4.0.5: `p({:a==>1})` prints {"a=": 1}.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 100% gate reads the ROUNDED total, which stayed at 100.0% while peek3's out-of-range return had never run. `:a==` at end of input is the source that reaches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…block
`-> a=a() { a }` failed with `expected {, got "\n"`: the brace was attached to
the `a()` call in the parameter default, leaving nothing to open the lambda
body.
MRI decides this in the lexer, by bracket nesting. At a `->` it records
`p->lex.lpar_beg = p->lex.paren_nest`, and a later `{` returns tLAMBEG — the
lambda body — exactly when `lambda_beginning_p()`
(`p->lex.lpar_beg == p->lex.paren_nest`) holds (parse.y v3_4_0). paren_nest
counts `(`, `[` and `{`, so the rule is: while the unparenthesised parameter
list is open and no bracket is, the next `{` is the body.
The parser now carries noBraceBlock, set over an unparenthesised stabby-lambda
parameter list and cleared by enterBracket wherever a bracket is opened (a
parenthesised group, a `(…)`/`[…]` argument list, a parenthesised parameter
list). That reproduces both sides: `-> x = ([1].map { |v| v }) { x }` keeps its
inner block, while the same expression without the parentheses is a syntax
error in MRI too, which is why the flag is not cleared for a bracket that has
already closed.
Unblocks language/lambda_spec.rb, which now parses.
Verified against MRI 4.0.5: `def a; 1; end; p(-> a=a() { a }.call)` prints 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…block
Two faults, both found by language/block_spec.rb, which now parses.
1. `o.s (:a){ 1 }` bound the `{ … }` to the argument instead of to the call,
silently — rbgo then raised LocalJumpError where MRI prints 1. In MRI a
SPACED `(` in argument position lexes as tLPAREN_ARG and its `)` sets
EXPR_ENDARG (parse.y v3_4_0: `primary: tLPAREN_ARG compstmt
{SET_LEX_STATE(EXPR_ENDARG);} ')'`), so the `{` right after it lexes as
tLBRACE_ARG and the grammar hands it to the command through
`command: primary_value call_op operation2 command_args cmd_brace_block`
(3464-3480). The parser now records where that `)` is (argParenEnd) and
attaches the brace block to the command instead of letting the postfix chain
take it. A HUGGING paren is unaffected — its `)` leaves EXPR_END, which is
why MRI gives the block in `f g(1) { 2 }` to `g(1)` and raises from `f`;
that negative case is pinned too.
2. `|a, (b, (c, d))|` failed with `expected IDENT, got "("`. A destructuring
block parameter nests in MRI — `f_marg: f_norm_arg | tLPAREN f_margs rparen`
puts f_marg back inside f_margs — but parseBlockParams carried a flat copy of
the logic that could only read identifiers one level down. It now calls
parseDestructureParam, the recursive implementation the `def` parameter list
already used, which also makes `|((a, b), c)|` and `|(a, *b)| ` parse.
Verified against MRI 4.0.5, values compared through rbgo for each case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`{ |one; bl| bl }` parsed, but the names after the `;` were declared in the
parser's scope and then dropped, so `bl` in the body became an ast.VarRef the
consumer had no slot for — rbgo answers `compile error: undefined local
variable "bl"` where MRI prints nil. The parser's half of that is the loss of
the list.
MRI's `opt_bv_decl: '\n'? ';' bv_decls '\n'?` with
`bvar: tIDENTIFIER { new_bv(p, $1); }` (parse.y v3_4_0) adds each name to the
BLOCK's own local table, where it starts nil and shadows any enclosing binding.
ast.Block gains Locals, nil when the list has no `;` part, so a consumer can
allocate them.
This is what language/block_spec.rb line 782 needs; the file now parses (see the
preceding commit) and reaches its compiler.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All four are MRI grammar alternatives the parser did not have. They are
sequential: each one was the next thing language/pattern_matching_spec.rb
stopped at. The file now parses.
1. `in Array(0, 1, 2)` failed with `expected LABEL, got "0"` — a `Const(…)`
pattern was always read as a hash pattern. In MRI the parenthesised and the
bracketed const pattern have the SAME four alternatives (parse.y v3_4_0,
p_expr_basic): `p_const p_lparen p_args rparen`, `… p_find rparen`,
`… p_kwargs rparen`, `p_const '(' rparen`, mirroring p_lbracket/rbracket. The
body decides, not the bracket, so parseArrayPattern is now parameterised by
its bracket pair and serves both.
2. A trailing comma was dropped. `in [0, 1, ]` parsed but became the EXACT
pattern `[0, 1]`, so `case [0,1,2,3]` took the else branch where MRI matches
— a wrong answer with no diagnostic. MRI's `p_top_expr_body: p_expr ','`
makes it an array pattern with an ANONYMOUS REST, matching any array that
starts with the listed elements. The un-bracketed spelling `in 0, 1,;` is the
same rule and was a parse error; both now build the rest.
3. `in {"a": 0}` failed. p_kw_label has two spellings — `tLABEL` and
`tSTRING_BEG string_contents tLABEL_END` — so a quoted string followed by `:`
is the same symbol key. `in {"a" => 1}` stays a parse error, which is what
ruby/spec asserts.
4. `in ^@a` failed. `p_var_ref: '^' tIDENTIFIER | '^' nonlocal_var` with
`nonlocal_var: tIVAR | tGVAR | tCVAR` (5883-5897) pins a non-local too.
Verified against MRI 4.0.5, values compared through rbgo for each case
including the negative ones (`in [0, 1]` stays exact, `in {"a" => 1}` refused).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`(o).a, (o).a = 1, 2` failed with `unexpected "," after statement`: a `(` at the start of a target was always taken for a nested target group, so `(o).a` ended the scan. MRI's mlhs_node takes any primary_value as the receiver — `primary_value call_op tIDENTIFIER`, `primary_value '[' opt_call_args rbracket`, `primary_value tCOLON2 tIDENTIFIER` (parse.y v3_4_0, 3639-3652) — and a parenthesised expression is a primary. looksLikeMlhs and parseMlhsTarget now look past the balanced `(…)`: a postfix chain after it makes the group a RECEIVER, anything else leaves it the nested target list it was. So `(a, b), c = [1, 2], 3` is unchanged and `(o).a = 1` stays a plain attribute assignment rather than becoming a one-target masgn. Unblocks language/assignments_spec.rb, which now parses. ruby/spec uses this shape to pin evaluation order — `(ScratchPad << :a; obj).a, … = …` — and rbgo still evaluates the right-hand side before the target receivers, which is a compiler matter: the AST records the two sides, not an order. Verified against MRI 4.0.5 through rbgo: same targets, same values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`for i, in [[1,2]]` failed with `expected IDENT, got "in"`, and so did every other loop-variable shape beyond a comma-separated list of bare names: `for @v in`, `for $v in`, `for C in`, `for o.a in`, `for o&.a in`, `for a[0] in`, `for i, *j in`, `for (i, j, k) in`, `for i, (j, k) in`, `for (i, j), k in`. MRI's loop variable is `for_var: lhs | mlhs` (parse.y v3_4_0) — any assignment target, or the same target list a multiple assignment takes, `mlhs_basic` and all. parseFor now parses it with the mlhs machinery instead of a loop over IDENTs. Two distinctions the grammar makes and the AST has to keep: - `mlhs_head: mlhs_item ','` is a COMPLETE mlhs, so a trailing comma means destructuring: `for i, in [[1,2]]` binds i to 1 where `for i in [[1,2]]` binds it to [1,2]. MRI 4.0.5 prints 1 and [1, 2] respectively. - a SINGLE target is `for_var: lhs`, not a one-element `mlhs`, so it binds the whole element: `for @v in [[1,2]]` sets @v to [1, 2]. ast.For gains Target for everything Vars cannot spell. Vars is EMPTY whenever Target is set, so a consumer reads the loop variable from exactly one place, and the shapes that already worked — `for i in …`, `for a, b in …` — still arrive in Vars unchanged. Unblocks language/for_spec.rb. With this every one of the 68 files in the pinned ruby/spec language corpus parses. Verified against MRI 4.0.5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #27 — all nine parse errors and the mis-parse, plus the two files the
issue deliberately left unclaimed, plus four further gaps each of them uncovered.
Every one of the 68 files in the pinned
ruby/speclanguage/corpus nowparses (13 did not before), and so do 4405 of the 4413
.rbfiles in thewhole corpus, up from 4391 — the 8 that remain include two deliberate
syntax-error fixtures. No file that parsed before stopped parsing.
Each fix reads the MRI rule first; the commit messages cite it by name and line
in
parse.yv3_4_0, and every behaviour was checked against the installed MRI4.0.5, comparing values through rbgo rather than merely "it parses".
What the issue filed
alias :"#{:a}" v/undef :"#{:m}",symbol: ssym(o).a, (o).a = 1, 2mlhs_node: primary_value call_op tIDENTIFIER(3639-3652) — the receiver is anyprimary_valueac7e3a1defined?($zz and true)keyword_defined '\n'? '(' begin_defined expr rparen(4458), aprimary; andkeyword_defined '\n'? begin_defined arg(4039)c1df337for i, in [[1,2]], withmlhs_head: mlhs_item ','` a complete mlhsp({:a==>1})parse_ident: `c=='=' && … (!peek('=')in Array(0,1,2)p_expr_basic: p_const p_lparen p_args rparen— the same quartet as p_lbracketafab703p $=parse_gvar's tGVAR arm lists_ ~ * $ ? ! @ / \ ; , . = : < > "05f6a4c-> a=a() { a }lambda_beginning_p()=lpar_beg == paren_nest, set at the->9b7c51bo.s (:a){ 1 }tLPAREN_ARG … {SET_LEX_STATE(EXPR_ENDARG);} ')'(4405) thencommand: … command_args cmd_brace_block(3464)1ba2c43The mis-parse is the one that mattered most: it was silent.
o.s (:a){ 1 }bound the block to
:a, so rbgo raisedLocalJumpErrorwhere MRI prints 1. Thenegative case is pinned too —
f g(1) { 2 }uses a HUGGING paren, so its)leaves
EXPR_ENDand MRI gives the block tog(1); that still happens.The two files the issue could not reduce
Both reduce, and neither error was at the line it pointed to in the way it
looked:
class_spec.rb(andmetaclass_spec.rb, same cause):class C; end.should == nil.The parser let a definition take a
.methodand a modifier but not be anOPERAND. In MRI
k_class cpath superclass bodystmt k_end, theclass << exprform,
k_module cpath bodystmt k_endanddefn_head f_arglist bodystmt k_endare all alternatives of
primary(4599-4657). Removing the statement-levelinterception was the whole fix (
9ce5aec).string_spec.rb:%@hey #{@ip}@. The%-literal terminator scan lookedINSIDE the interpolation and closed on the
@of@ip. MRI'stokadd_stringleaves the scan at
#{and hands the embedded expression to the main lexer.Two more
%-delimiter faults were behind it:%=hey=(MRI testsIS_BEG()before the op-assign
=, so=delimits a literal where a value is expectedand stays op-assign after one) and
%\hey\(the terminator test precedes thebackslash test in
tokadd_string) (92c7bd2).Found on the way, each blocking the next line of a spec file
defined?(break),defined?(return): a barebreak/next/retryis aprimary(4682-4697) and so is a barereturn(k_return, 4438), with thecall_argsforms alternatives ofcommand(c1df337).in [0, 1, ]parsed but silently became the EXACT pattern[0, 1], so acase [0,1,2,3]fell toelsewhere MRI matches.p_top_expr_body: p_expr ','makes a trailing comma an ANONYMOUS REST (
afab703).in {"a": 0}:p_kw_label: tLABEL | tSTRING_BEG string_contents tLABEL_END.in {"a" => 1}stays refused, as ruby/spec requires (afab703).in ^@a:p_var_ref: '^' tIDENTIFIER | '^' nonlocal_var(afab703).|a, (b, (c, d))|:f_marg: f_norm_arg | tLPAREN f_margs rparennests, butparseBlockParamsheld a flat copy of logic thedeflist already hadrecursively (
1ba2c43).AST changes, all additive
ast.AliasgainsNewNameExpr/OldNameExpr,ast.UndefgainsExprs,ast.BlockgainsLocals,ast.ForgainsTarget. Each is nil/empty for theshapes that worked before, so existing field reads keep their meaning and
existing consumers keep building.
ast.For.Targetis the one a consumer mustlearn:
Varsis EMPTY whenever it is set, so there is only ever one place toread the loop variable from, and
for i in …/for a, b in …still arrive inVarsunchanged.Measurement
rbgobuilt against this branch, over the pinned corpus, before and after.Ten files went from "no result at all" to a measured result, revealing
311 passing examples that no measurement had ever seen:
pattern_matching_spec.rbpredefined_spec.rbclass_spec.rbhash_spec.rbstring_spec.rbalias_spec.rbmetaclass_spec.rbassignments_spec.rbundef_spec.rblambda_spec.rbSpecEvaluatehelper is missing from the corpus harness)Corpus totals: pass 1031 → 1342, files with no result 14 → 4. Nothing regressed:
the before/after diff over all 68 files contains only these additions.
The four files still reporting nothing all PARSE now and stop in rbgo's
compiler, not here:
for_spec.rb—undefined local variable "j": rbgo readsFor.Vars, which isempty for
for i, *j in …. It needs to readFor.Target.block_spec.rb—undefined local variable "bl": block-locals need a slot.Block.Localsnow carries the names (74ed65f); rbgo has to allocate them.defined_spec.rb—undefined local variable "a":defined?(a += 1)declaresthe local at parse time, as MRI does, but rbgo's compiler allocates no slot for
it. Reproduces identically on
main's parser, so it is pre-existing.variables_spec.rb—cannot compile *ast.SplatArg.One more consumer-side finding, from
assignments_spec.rb: rbgo evaluates theright-hand side of a multiple assignment before the target receivers, where MRI
evaluates the receivers first. The AST records the two sides, not an order, so
that is a compiler matter.
Gates
go build ./...,go vet ./...,go test -race ./...all green; the 100%coverage gate holds (and
peek3's end-of-input branch, which the ROUNDED totalwas hiding at 100.0%, is now covered —
60303c5).gofmt -l .lists onlyparen_singleton_def_test.go, which is untouched and already listed onmain.🤖 Generated with Claude Code