Repository navigation
perf(strscan): re-pin @main for O(n) anchored scans (fix O(n²) tokenize) - #89
Merged
Merged
Conversation
go-ruby-strscan's anchored scan primitives (scan/skip/match?/check) were O(remaining length) per call: they matched by re.Match(str[pos..]) and then rejected a result that did not start at the cursor, so re.Match first scanned forward through the tail. A lexer that tries several alternative patterns per position paid that tail scan on every miss, making a full tokenize O(n^2) — the largest perf gap in the ecosystem (~80x slower than MRI, super-linear). The library now uses go-ruby-regexp's Regexp.MatchAt(s, pos), a cursor-anchored match (\G bound to pos, whole string visible for ^/\A/lookbehind) that tries a single match at the cursor: O(match length), not O(tail). Semantics stay byte-identical to MRI StringScanner (the differential-vs-MRI oracle passes). This binding routes through the fast path automatically: it delegates directly to sc.Scan/sc.Skip/sc.Match/sc.Check, so only the dependency pin changes. Re-pinned github.com/go-ruby-strscan/strscan @main (d0cc926); go-ruby-regexp bumps to fa3d6604 transitively for MatchAt. Full internal suite green, all 6 arches build. Measured rbgo-vs-MRI 4.0.5 tokenize (identifier lexer, alternatives-miss-scan tail), input x1/x2/x4/x8: before: 7.4x / 12.4x / 20.0x / 35.1x (ratio grows -> O(n^2)) after: 4.4x / 4.8x / 4.9x / 4.8x (ratio flat -> O(n)) At x8 the fix is 7.4x faster than before and the gap widens with input size; the residual ~4.8x is flat interpreter overhead, not the algorithmic bug. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the largest perf gap in the go-ruby-* ecosystem:
StringScannertokenize was O(n²) (~80× slower than MRI, super-linear). This PR re-pins the fixedgo-ruby-strscanand proves the O(n²)→O(n) linearization end-to-end in rbgo.Root cause (in the library, now fixed)
The anchored scan primitives (
scan/skip/match?/check) matched by callingre.Match(str[pos:])and then rejecting a result that didn't start at the cursor.re.Matchscans forward to the first match, so every anchored scan was O(remaining length). A lexer that tries several alternative patterns per position pays that tail scan on each miss, making a full tokenize O(n²).The fix
go-ruby-strscannow usesgo-ruby-regexp'sRegexp.MatchAt(s, pos)— a cursor-anchored match (\Gbound topos, whole string visible so^/\A/lookbehind see the real prefix) that tries a single match at the cursor: O(match length), not O(tail). Semantics stay byte-identical to MRI StringScanner (the differential-vs-MRI oracle still passes).go-ruby-strscan/strscand0cc926(splitmatchAtintomatchAnchoredviaMatchAtandmatchForwardfor the inherently-forward*_untilsearch); 100% coverage under-race, 6-arch, gofmt/vet clean, MRI oracle byte-identical.sc.Scan/sc.Skip/sc.Match/sc.Check, which now take the fast path. Only the dependency pin changes (go-ruby-regexpbumps tofa3d6604transitively forMatchAt).Measured (rbgo vs MRI 4.0.5, identifier-lexer tokenize; alternatives miss at cursor and scan the tail)
Before: the ratio grows with input size (7.4→35.1×), per-doubling ~2.9–3.6× — quadratic. After: the ratio is flat (~4.8×), per-doubling ~1.99× — linear. At ×8 the fix is already 7.4× faster than before and the gap widens with every doubling. The residual ~4.8× is flat interpreter overhead (dispatch/boxing), not the algorithmic bug.
Notes
origin/main(post-json: single-pass parse/generate (close the JSON perf gap vs MRI) #88).🤖 Generated with Claude Code