feat: ast.Program.Lines — the source line of every statement by tannevaled · Pull Request #30 · go-ruby-parser/parser

tannevaled · 2026-09-23T20:39:21Z

The gap

The AST carried no position information at all — 55 node types, zero line or
column fields. Any consumer that has to say where something is had nothing to
read.

That is not hypothetical. go-embedded-ruby/ruby, which this parser was
extracted from, reports line 0 in every backtrace and __LINE__ evaluates to
0 everywhere, because there is no line to thread through its compiler. Across
its pinned ruby/spec corpus that costs 101 examples in five directories.

The change

ast.Program gains Lines map[Node]int: the 1-based source line each
statement node began on, keyed by the node itself (every Node is a distinct
pointer, so the map is exact). Parse populates it.

It is on Program, filled by the existing entry point, rather than behind a
second ParseWithLines — deliberately. Consumers thread a *ast.Program through
several layers before anything looks at a line, so an opt-in entry point would
force every one of those layers to change signature. On Program they get
positions for free.

Why this shape

Statement granularity. MRI attaches a line to every node and compiles a
per-instruction table from it (compile.c v3_4_0: ADD_INSN records
nd_line(node); iseq.c:673 rb_iseq_insns_info_encode_positions builds
insns_info + positions). But that table is compressed deliberately — one
entry per line change, not per instruction — and every reader goes through
rb_iseq_line_no (iseq.c v3_4_0:2314), which resolves a pc to the nearest
preceding entry. Statement boundaries are where the line changes in all but
pathological source, so this map carries the same information at the granularity
a consumer can actually observe, without a position field on 55 structs.

First-write-wins, which is what makes the stamp precise rather than merely
cheap: an inner statement is recorded when its own parseStatement returns,
before the enclosing one finishes, so a construct that hands its body's node
straight back (begin; foo; end) keeps foo's line. MRI reaches the same answer
from the other side — the node it compiles is foo's node, and nd_line(foo)
is foo's line.

Verified against MRI 4.0.5 (+PRISM)

puts(__LINE__ +      # line 5
  1)

ruby prints 6 — so __LINE__ is the line the statement started on, not
the line the operand sits on. That is the rule this table encodes.

Cost

Measured on a 2570-line Ruby source, 200 iterations x 5, Apple M4 Max (medians):

	ns/op	B/op	allocs/op
v0.2.0	1 631 000	5 288 396	13 870
this change	1 728 000	5 397 636	13 880
	+6.0%	+2.1%	+10

The map is sized up front from the lexer's NEWLINE count — a tight upper bound
on the number of statements — which is what keeps it to +10 allocations rather
than the rehash chain (1.782 ms, +22 allocs when unsized).

Tests

Nine new tests: top-level statements (with blank lines, comments and
;-separated statements on one line), nested def / class / block bodies, the
first-write-wins pass-through, a multi-line call, a statement rewritten by
applyModifiers, every non-default arm of parseStatement, the parse-error and
internal-error contracts, and an empty program (non-nil, empty map).

go vet, go build, the full suite and the 100% coverage gate (2944/2944
statements) are green locally.

🤖 Generated with Claude Code

The AST carried no position information at all: 55 node types, zero line/column fields. A consumer that needs to say WHERE a frame is executing — a backtrace, Ruby's __LINE__, Module#const_source_location — had nothing to read, and go-embedded-ruby/ruby, extracted from this parser, has been reporting line 0 everywhere for exactly that reason (101 ruby/spec examples across five directories). Program.Lines maps each STATEMENT node to the 1-based line it began on. It is populated by Parse itself, on Program, rather than behind a second entry point: every existing consumer then gets positions without changing a call site, which matters because the consumers thread a *ast.Program through several layers before anything looks at a line. Why statement granularity: MRI attaches a line to every node and compiles a per-instruction table from it (compile.c v3_4_0: ADD_INSN records nd_line(node); iseq.c:673 rb_iseq_insns_info_encode_positions builds insns_info + positions). But that table is COMPRESSED deliberately — one entry per line CHANGE, not per instruction — and every reader goes through rb_iseq_line_no (iseq.c v3_4_0:2314), which resolves a pc to the nearest preceding entry. Statement boundaries are where the line changes in all but pathological source, so this map holds the same information at the granularity a consumer can observe, without a field on 55 structs. The stamp is first-write-wins, which is what makes it precise rather than merely cheap: an inner statement is recorded when its own parseStatement returns, before the enclosing one finishes, so a construct that hands its body's node straight back (`begin; foo; end`) keeps foo's line. MRI reaches the same answer from the other side — the node it compiles IS foo's node, and nd_line(foo) is foo's line. Verified against MRI 4.0.5 (+PRISM): for `puts(__LINE__ +\n 1)` opening on line 5, ruby prints 6 — __LINE__ is the line the STATEMENT started on, not the line the operand sits on. Cost, measured on a 2570-line Ruby source, 200 iterations x 5, Apple M4 Max (medians): v0.2.0 1.631 ms/op 5 288 396 B/op 13 870 allocs/op this change 1.728 ms/op 5 397 636 B/op 13 880 allocs/op +6.0% +2.1% +10 The map is sized up front from the lexer's NEWLINE count — a tight upper bound on the number of statements — which is what keeps it to +10 allocations instead of the rehash chain (1.782 ms, +22 allocs unsized). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…and block/class frame labels (#645) * feat(bytecode,compiler): a compressed source map on every ISeq __LINE__ evaluates to 0 everywhere and every backtrace reads `:0:` because the ISeq carries no line map. This is the map. REPRESENTATION — MRI's, not a line per instruction. MRI records a line for every instruction while compiling and then encodes the result as insns_info[], a table of DISTINCT entries, beside positions[], the instruction index each entry starts at (iseq.c v3_4_0: 673, rb_iseq_insns_info_encode_positions). It compresses because a line spans a run of instructions; storing one per instruction would multiply every ISeq for nothing. Reading it back is a search for the last entry at or before the pc — get_insn_info_binary_search (iseq.c v3_4_0:2160), the implementation MRI ships as VM_INSN_INFO_TABLE_IMPL == 1. bytecode.LineEntry{PC, Line} is one (positions[i], insns_info[i]). ISeq.Lines is the pair, sorted by PC. ISeq.LineAt(pc) is that binary search, answering 0 for a pc it cannot place — which is what rb_iseq_line_no returns when get_insn_info finds nothing (iseq.c v3_4_0:2314). ISeq.FirstLine is body->location.first_lineno, the fallback rb_vm_get_sourceline uses when the per-pc lookup yields 0 (vm_backtrace.c v3_4_0:105). WHERE THE LINES COME FROM. The parser's AST carried no position information at all — 55 node types, zero line fields — so there was nothing to thread. ast.Program.Lines (go-ruby-parser/parser#30) now gives the line each STATEMENT node began on, and compileNode moves the current line to a node's own when the map has one. An expression inside a statement inherits the statement's line, so receiver, arguments and the send all land on one line and the table gets one entry for the lot — the same shape MRI's insns_info has, because MRI's nd_line for those inner nodes is that line too. The entry is written in emit(), not where the line is SET. That is what keeps the table compressed: a statement that emits nothing (a discarded `defined?`) contributes no entry, a statement that emits twenty contributes one. A line is written down only because an instruction exists to carry it. A Program with no Lines map — a hand-assembled AST in a test — compiles to an empty source map and every lookup answers 0, exactly the behaviour this VM had before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(compiler): __LINE__ answers truly, and refuses to be assigned language/line_spec.rb: 0/6 -> 6/6. __LINE__ was a Kernel method returning 0, standing in for the line map that did not exist. It is not a method in MRI at all — it is a KEYWORD the grammar replaces with an Integer literal (parse.y v3_4_0: `keyword__LINE__` yields NEW_INTEGER of the current line). compileCall now does that substitution one layer down, from the source map, for the receiverless argumentless blockless bareword only. Confining it to that one form is what reproduces MRI's three refusals, all verified against ruby 4.0.5 (+PRISM): self.__LINE__ NoMethodError (rbgo used to answer 0) 1.send(:__LINE__) NoMethodError (rbgo used to answer 0) __LINE__() SyntaxError so the stand-in method is deleted rather than kept as a fallback: while it existed, the two NoMethodError cases silently answered 0. Assignment is refused too. ruby 4.0.5 answers `__LINE__ = 1` with `Can't assign to __LINE__`, a SyntaxError, and line_spec.rb pins `eval("__LINE__ = 1")` raising it. The refusal is not only conformance: an accepted assignment would declare a LOCAL of that name, and every later `__LINE__` in the scope would parse as a read of it — the keyword would quietly stop being one. __FILE__ and __ENCODING__ are refused the same way; ruby gives the same message for both. Byte-for-byte against MRI on a file exercising top level, a paren expression spanning two lines, a def, a class body and a block: `cmp` of the two outputs is clean. ALSO, and not optional: aot/freeze.go now freezes Lines and FirstLine. An AOT-compiled program has no source to recompile from, so a source map dropped by the emitter is a backtrace reporting line 0 for every frozen frame — the state this VM was in before the map existed, reintroduced silently and only on that path. TestEmbeddedPreludeMatchesSource is what caught it: the frozen prelude no longer round-tripped. internal/vm/puppet_boot_batch3_test.go pinned __LINE__ at 0, 0 and 1. Each of the three is re-measured against ruby 4.0.5 by running the same source from a file: 1, 2 and 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(vm): backtraces read the line map — `:0:` becomes a real line Every frame now reports where it actually is. Verified byte-for-byte against ruby 4.0.5 on a raise chain and a Kernel#caller chain: the line of all five frames matches. HOW A FRAME GETS A LINE. Not by storing one. frameISeqs/framePCs join frameNames/frameFiles as per-frame stacks, holding which ISeq the frame runs and how far it has got; frameLine COMPUTES the line from the pair when somebody asks. That is MRI's arrangement exactly — cfp->iseq and cfp->pc, resolved by calc_lineno(loc->iseq, loc->pc) (vm_backtrace.c v3_4_0:87) — and it is why the cost is one integer store per instruction rather than a search. frameLine is rb_vm_get_sourceline (v3_4_0:105), fallback included: a pc the map cannot place falls back to the ISeq's first_lineno. formatBacktraceEntry is location_format (vm_backtrace.c v3_4_0:446), including the part that is easy to miss — the ":%d" is appended only `if (lineno != 0)`. An unplaceable frame reports "path:in 'label'", and so says nothing about its line rather than claiming line zero. FirstLine is now actually set, in Compiler.push, so every scope is stamped with the line of the `def`/`class`/`{` that opened it — one place rather than the eight sites that open a scope. Proc#source_location reads it, which is the right source for that method and not the pc: MRI reports where the block was WRITTEN (proc.c rb_proc_location via iseq_location), so a proc halfway through its body still answers the line of its `do`. A HANG, AND WHAT IT TAUGHT. The pc publish was first written unconditional. TestConditionVariable then hung for nine minutes. The write was indexing past the end: Run resets frameNames to :0 at its boundary, and a thread body runs through such a boundary while the frame that started it is still live and will resume the loop afterwards. The panic never surfaced as a panic — a recover on the way out swallowed it, and the thread simply never signalled. The bounds check is load-bearing, and skipping the write is also the CORRECT answer: a frame absent from frameNames is a frame no backtrace renders. PINNED TESTS. Eight asserted `:0:` or a consequence of it. Each new value is MRI's, obtained by writing the test's own source to a file and running ruby 4.0.5: backtrace_test.go five tests, lines 3/6/9, 3, 3+6, 3+5, 3, 6 backtrace_internal_test.go a hand-built frame stack now builds the code stacks too — it was reporting line 109 from the PRELUDE's ISeq, left over from VM boot. A wrong line is worse than none. wave24_module_internal_test.go two `true`s become `false`. They compared two backtraces taken on DIFFERENT lines, which stringified identically while every line was 0 — so the comparison compared nothing. ruby 4.0.5 answers false to both. puppet_apply_primitives_test.go Proc#source_location 0 -> 1; ruby prints ["<file>", 1] for a proc on line 1. go.mod takes go-ruby-parser/parser v0.3.0, which carries ast.Program.Lines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(vm,compiler): frame labels for blocks, class bodies and module bodies base_label_spec 0/5 -> 4/5, label_spec 5/36 -> 7/36, and absolute_path/ensure/rescue one each: +9 examples. Every frame that was not a method reported "<main>" — a class body, a module body, a singleton-class body and a block all claimed to be the top level. Two halves fix that, and they are on opposite sides of the compiler/VM line because MRI's are too. BLOCK LABELS, AT COMPILE TIME. blockLabel is calculate_iseq_label's ISEQ_TYPE_BLOCK arm (vm_backtrace.c v3_4_0:229): walk out through the enclosing scopes counting the block ones, then "block in NAME" for one level and "block (N levels) in NAME" beyond. MRI walks parent_iseq at backtrace time; the answer is a purely lexical fact, so the walk happens once, at compile time, and the block ISeq is NAMED for the frame it will be. `for` scopes are skipped in the count rather than followed: `for` opens no Ruby scope, so a block inside a `for` body is one level deep. Verified against ruby 4.0.5 — a method, a block in it and a block in that print "block in …" and "block (2 levels) in …" with the same nesting counts, and #base_label prints the undecorated name. SCOPE LABELS, AT BACKTRACE TIME. frameLabel consults the frame's ISeq when frameNames has no method name. The recognised names are a WHITELIST, and that is the care this needed: an ISeq's Name is not always a label — a required file's top-level ISeq is named by its PATH, so taking any name turned "prog.rb:9:in '<main>'" into "prog.rb:9:in 'prog.rb'", a frame labelled with its own file. The full suite caught it. Anything not on the list is the top level by another name and reports "<main>". #base_label stops being an alias of #label. MRI reads the two from different fields (vm_backtrace.c v3_4_0:331) and the difference is exactly the block decoration, so stripBlockQualifier reconstructs base_label by removing it. WHAT IS STILL MISSING, deliberately: the OWNER prefix. ruby prints "Object#foo" where this prints "foo", because rb_gen_method_name (vm_backtrace.c v3_4_0:~200) qualifies a method label with its owner. That is a different mechanism — it needs the frame's defining module, not its ISeq — and it changes every method label in every backtrace, so it is not being bolted onto this change. It is the bulk of label_spec's remaining 29. blockISeqs in wave26_for_defined_test.go matched the literal "<block>"; it now matches the prefix, so it keeps finding block ISeqs whatever the enclosing scope is called. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(vm): one frame-code stack, and the honest number for what it costs frameISeqs and framePCs become one []frameCode. A frame push touches both halves, so one bounds check and one store now serve for both. It did not buy the speed it was reached for, and the number is in the code so nobody re-derives it. bench/blocks.rb — millions of tiny block calls, the worst case for anything paid per frame — is about +7% either way. A best-of-5 pass appeared to show the merge taking it from +8.7% to -0.2%, which is why the merge was made; 15 interleaved runs of each binary put it at +7.5% best-of-15 and +6.8% by median. The -0.2% was noise in a best-of-5 on a benchmark whose own spread is 870-1037 ms. The cost is the per-instruction pc store and the per-frame push, not the slice count. The rest of the standard bench set, best-of-9, branch vs merged main: fib +2.1% dispatch -0.4% loop +2.4% mandelbrot +1.8% blocks +7.5% Binary: 181,646,482 -> 181,796,146 bytes, +0.08%. The frozen prelude grows by ~1070 lines of source map, which is the AOT half of the same table. Whitebox tests for the three new compiler functions, so the coverage gate's below-100 list matches main's exactly: blockLabel over one, two and three levels, `for` scopes skipped, no parent, and end-to-end through a compiled ISeq; refuseKeywordAssign over all three keywords plus two names that merely resemble one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

tannevaled force-pushed the lines/statement-map branch from fa64f6f to 0ccf760 Compare September 23, 2026 20:41

tannevaled changed the title ~~feat: ParseWithLines — a statement-line side table for backtraces and __LINE__~~ feat: ast.Program.Lines — the source line of every statement Sep 23, 2026

tannevaled merged commit b31277c into main Sep 24, 2026
11 checks passed

tannevaled deleted the lines/statement-map branch September 24, 2026 06:27

tannevaled mentioned this pull request Sep 24, 2026

conformance: a source map on every ISeq — __LINE__, backtrace lines, and block/class frame labels go-embedded-ruby/ruby#645

Merged

Navigation Menu

Repository navigation

feat: ast.Program.Lines — the source line of every statement - #30

feat: ast.Program.Lines — the source line of every statement#30
tannevaled merged 1 commit into
mainfrom
lines/statement-map

tannevaled commented Sep 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tannevaled commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The gap

The change

Why this shape

Verified against MRI 4.0.5 (+PRISM)

Cost

Tests

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tannevaled commented Sep 23, 2026 •

edited

Loading