Skip to content

kernel/string: conformance wave 25 — require path resolution and coercion, byteindex reverse search, encoding-aware chars/chop/split (core/kernel 233→184, core/string 179→153) - #618

Merged
tannevaled merged 9 commits into
mainfrom
wave25/kernstr
Sep 21, 2026

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Conformance wave 25, core/kernel + core/string. Every change was made after
reading the MRI C source for the method, and each one was witnessed
byte-for-byte against MRI 4.0.5 (ruby -e vs the built rbgo -e, raw bytes)
before and after.

Result (pinned rubyspec corpus, fail+error)

suite before after
core/kernel 233 184
core/string 179 153

No spec file anywhere in core/kernel or core/string regressed; the three
pre-existing crashes (kernel/exit_spec, kernel/sleep_spec,
string/unpack/carret_spec) are unchanged.

Biggest movers: require_relative_spec 26 → 2, require_spec 29 → 14,
load_spec 17 → 7, byterindex_spec 10 → 2, byteindex_spec 7 → 1,
split_spec 6 → 1, to_sym_spec 6 → 3, chars_spec / each_char_spec
3 → 1 each.

What was fixed, and what was read for it

load.c — rb_f_require_relative, search_required, load_lock

  • rb_f_require_relative is
    rb_require_string_internal(rb_file_absolute_path(fname, dirname(base))), and
    rb_file_absolute_path ignores the base when fname is already absolute.
    rbgo joined it onto the requiring file's directory, so every absolute
    require_relative failed
    .
  • The same expansion happens before require_internal runs, so the failing
    name load_failed reports — and the @path it stamps — is the expanded,
    cleaned absolute path with no .rb appended.
  • load_lock issues the circular-require warning through rb_warning, so only
    in verbose mode, and only when rb_thread_shield_owned says the current
    thread already holds the feature's shield. A second thread arriving mid-load
    is a concurrent require, not a circular one — the loading marker is keyed by
    thread so the concurrent case stays silent.
  • search_required asks rb_feature_p before rb_find_file_ext; for a name
    with no extension, a $LOADED_FEATURES entry equal to that name answers
    'u', and require reports false instead of raising.

file.c — rb_get_path, rb_find_file, is_explicit_relative

  • rb_get_path = rb_get_path_check_to_string + rb_get_path_check_convert.
    The first takes a String as-is, else calls #to_path through
    rb_check_funcall_default(obj, to_path, 0, 0, obj) and runs StringValue
    over the result — so a #to_path returning a non-String has #to_str
    called on that value. requireName accepted a String only.
  • rb_find_file expands a leading ~ through HOME and returns directly; an
    absolute path and an explicitly-relative one (./x, ../x) likewise return
    before the $LOAD_PATH walk. rbgo searched the load path for all three.
  • Every $LOAD_PATH entry goes through rb_get_path as the list is walked, so
    an entry answering #to_path is a usable directory.
  • isdirsep accepts a backslash only on Windows, so is_explicit_relative
    answers differently per platform; os.IsPathSeparator draws the same line.

error.c — raise_loaderror, rb_builtin_class_name

  • rb_attr(rb_eLoadError, path, TRUE, FALSE, FALSE) exposes the path
    raise_loaderror stamps from the fname as passed. LoadError#path was
    missing.
  • rb_builtin_class_name spells nil/true/false as those words; rbgo's
    classNameOf collapsed every object to "Object", so
    "no implicit conversion of true into String" read "... of Object ...".

string.c — rb_str_byteindex_m, rb_str_byterindex_m, rb_str_split_m,
chopped_length, rb_str_chars, rb_str_intern

  • Both byte-index methods read the offset with NUM2LONG and the pattern with
    StringValue, so #to_int and #to_str apply; rbgo hard-checked both types.
  • rb_str_byterindex_m searches with rb_reg_search(sub, str, pos, 1) — the
    trailing 1 is onig_search's reverse flag, so the search runs backwards
    over the whole subject. rbgo matched against the tail slice s[p:], which
    makes every slice start look like the start of the string: /\A/ reported 9
    instead of 0. Replaced with an anchored probe downwards from the offset, the
    byte-indexed twin of the existing strRindexRegexp.
  • rb_str_split_m calls mustnot_broken on a String pattern and on the
    receiver; the awk and regexp paths reach the same refusal through
    rb_enc_check and the regexp engine. Its limit is read with NUM2INT, so a
    value outside the 32-bit range is a RangeError — and #to_int runs exactly
    once.
  • chopped_length walks back with rb_enc_prev_char, and rb_str_chars steps
    with rb_enc_mbclen: "the last character" and "a character" are questions
    the encoding answers.
    Decoding as UTF-8 regardless took two bytes off a
    binary string, and — worse than a length error — the []rune round trip
    rewrote every invalid byte in the prefix as U+FFFD, corrupting bytes the
    operation never touched. That was the real cause of the split spec that builds
    its broken string with chop!.
  • rb_str_intern re-tags an ASCII-only string as US-ASCII whatever encoding it
    arrived in; sym_encoding is rb_obj_encoding(rb_sym2str(sym)).
    Symbol#encoding was missing entirely.

One finding worth flagging: core/string/valid_encoding/utf_8_spec.rb

This file was the wave's headline target at pass=0 / 28 errors, described as
"probably one missing behaviour". It is not a VM gap at all. MRI 4.0.5 scores
exactly the same 0 pass / 28 error on it
with this harness.

The file defines a utf8 helper with def inside a describe block and calls
it from it blocks in nested describes. The shim's SpecContext#run
instance_evals each example against the innermost context object, so self
is rebound and the helper — which landed on the outer context's singleton — is
unreachable. mspec calls example blocks plainly, leaving self as the
lexically enclosing describe's context, which is why it works there. rbgo
already matches MRI byte-for-byte on valid_encoding? for every case the file
exercises. The fix belongs in scripts/conformance/rubyspec/spec_helper.rb,
which this branch does not own.

Deliberately left red

Named here rather than accommodated, so none of them is sealed in by a weakened
assertion:

  • "helloYOU.".byterindex(/YOU\G/, 8) needs the match's \G anchor pinned to
    the search start rather than to the probed start; the pinned
    go-ruby-regexp engine exposes no API for a separate global position.
  • The EUC-JP Encoding::CompatibilityError cases in byte_index_common need
    regexp-side encoding negotiation.
  • Three to_sym specs need the symbol to carry its source encoding
    (UTF-16LE, binary non-ASCII, and re-interning under a different encoding).
    rbgo's object.Symbol is an untagged Go string; Symbol#encoding derives
    US-ASCII / UTF-8 / ASCII-8BIT from the bytes, which matches MRI for every
    case this VM can represent, and answers by shape rather than provenance for
    the rest.
  • require still resolves a bare name against the requiring file's directory
    and the CWD before $LOAD_PATH; MRI searches $LOAD_PATH only. That is a
    deliberate, long-standing rbgo convenience, and changing it reaches well
    beyond this suite — left for a wave that can measure the app and library
    suites alongside.
  • gsub has the same slice-anchoring bug that byterindex had
    ("Text\n".gsub(/^/, " ") gives " T e x t \n ", and /\A/ re-matches every
    iteration), worth ~2 of gsub_spec's 12. The scan loop lives in
    internal/vm/regexp.go, outside this branch's file ownership for the wave, so
    it is reported rather than touched.

One pinned test updated

TestLoadPathRequire asserted that a non-String $LOAD_PATH entry is silently
skipped. MRI raises TypeError instead, and because
rb_get_expanded_load_path coerces the whole list before the search begins, the
position of the bad entry makes no difference — witnessed with 123 both
unshifted before and pushed after a directory that does hold the feature. The
test now pins the MRI-verified behaviour and gains the #to_path entry case.

Gates

  • CGO_ENABLED=0 go build ./... and the full GOWORK=off go test ./... green
    (the only noise is the pre-existing tests dialing 127.0.0.1:1).
  • internal/vm coverage 99.9%, with require.go and string_byteops.go at
    100% and the set of functions below 100% identical to main's, compared
    against a coverage profile taken from a worktree at the merge base. Twelve new
    whitebox tests cover every added branch.
  • gofmt -l internal/vm/ empty; GOOS=wasip1 GOARCH=wasm and
    GOOS=windows GOARCH=amd64 both build; the one platform-dependent assertion
    is guarded on runtime.GOOS.

🤖 Generated with Claude Code

tannevaled and others added 8 commits September 20, 2026 18:13
…e/load

Read ruby/ruby v3_4_0 load.c and file.c before each change:

- rb_f_require_relative (load.c) is
  rb_require_string_internal(rb_file_absolute_path(fname, dirname(base))).
  rb_file_absolute_path ignores the base when fname is already absolute, so an
  absolute require_relative argument must be used verbatim. rbgo joined it onto
  the requiring file's directory, so every absolute require_relative failed.

- rb_get_path (file.c) = rb_get_path_check_to_string + rb_get_path_check_convert.
  The first takes a String as-is, else calls #to_path through
  rb_check_funcall_default(obj, to_path, 0, 0, obj) and runs StringValue over the
  RESULT -- so a #to_path returning a non-String gets #to_str called on that
  value. requireName accepted a String only.

- rb_find_file (file.c): a leading ~ is expanded through HOME and returned
  directly; an absolute path and an is_explicit_relative one ("./x", "../x")
  likewise return before the $LOAD_PATH walk. rbgo searched $LOAD_PATH for all
  three.

- error.c raise_loaderror sets the exception's path ivar to the fname as passed;
  rb_attr(rb_eLoadError, path, TRUE, FALSE, FALSE) exposes it. LoadError#path
  was missing.

core/kernel require_spec 29->18, require_relative_spec 26->4, load_spec 17->9
(fail+error), i.e. 72->31 across the trio.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…Error name

Read ruby/ruby v3_4_0 load.c and file.c:

- load_lock (load.c) warns "loading in progress, circular require considered
  harmful - %s" through rb_warning, so only in verbose mode; require_internal
  passes RTEST(ruby_verbose) as its warn argument. The warning fires only when
  rb_thread_shield_owned reports the CURRENT thread already holds the feature's
  shield -- another thread reaching a feature mid-load is a concurrent require,
  not a circular one, and waits without warning. The loading marker is therefore
  keyed by thread.

- rb_find_file (file.c) runs each $LOAD_PATH entry through rb_get_path as it
  walks the list, so a non-String entry answering #to_path is a usable load-path
  directory; loadPathDirs skipped those.

- rb_f_require_relative (load.c) expands the name before require_internal sees
  it, so load_failed names the expanded, cleaned absolute path with no ".rb"
  appended. Witnessed against MRI 4.0.5.

core/kernel require_spec 18->16, require_relative_spec 4->2, load_spec 9->7.
Trio total for the wave so far: 72 -> 25 (fail+error).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…earch

Read ruby/ruby v3_4_0 string.c rb_str_byteindex_m / rb_str_byterindex_m and
error.c rb_builtin_class_name:

- Both read the offset with NUM2LONG (so #to_int applies) and, once the Regexp
  branch is ruled out, take the pattern through StringValue (so #to_str
  applies). rbgo hard-checked both types, so every mock with #to_str/#to_int
  raised TypeError.

- rb_builtin_class_name spells nil/true/false as those words and otherwise names
  the object's class; the package-level classNameOf collapses every object to
  "Object", which made "no implicit conversion of true into String" read
  "... of Object into String".

- rb_str_byterindex_m searches with rb_reg_search(sub, str, pos, 1) -- the
  trailing 1 is onig_search's reverse flag, so the search runs backwards over the
  WHOLE subject. rbgo matched against the tail slice s[p:], which makes every
  slice start look like the start of the string: /\A/ reported 9 instead of 0.
  Replaced with an anchored probe from off downwards, the byte-indexed twin of
  the existing strRindexRegexp.

core/string byteindex_spec 7->1, byterindex_spec 10->2 (fail+error). Every
change witnessed byte-for-byte against MRI 4.0.5.

Remaining there and NOT fixed: /YOU\G/ with a start offset needs the match's \G
anchor pinned to the search start rather than to the probed start, which the
pinned go-ruby-regexp engine exposes no API for; and the EUC-JP
Encoding::CompatibilityError cases need regexp-side encoding negotiation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ncoding-aware

Read ruby/ruby v3_4_0 string.c:

- rb_str_split_m calls mustnot_broken on a String pattern (9237) and on the
  receiver in the string and chars paths (9317, 9337); the awk and regexp paths
  reach the same refusal through rb_enc_check and the regexp engine. Witnessed
  against MRI 4.0.5: the awk, " ", "", Regexp, String and nil forms all raise
  ArgumentError "invalid byte sequence in UTF-8" on a broken receiver, while a
  binary receiver never does. rbgo split them happily.

- The limit is read with NUM2INT, an int conversion, so 2147483649 is a
  RangeError ("too big to convert to 'int'", and "too small" at the other end).
  NUM2INT also runs exactly once, so the coerced value replaces the argument
  rather than letting stringSplit call #to_int a second time.

- chopped_length walks back with rb_enc_prev_char, so "the last character" is a
  question the encoding answers: in ASCII-8BIT chop removes exactly one byte.
  chopStr always decoded as UTF-8, which took two bytes off a binary string --
  and on a UTF-8 string holding invalid bytes the []rune round-trip rewrote
  every invalid byte in the PREFIX as U+FFFD, corrupting bytes chop never
  touched. That was the real cause of the split spec that builds its broken
  string with chop!.

core/string split_spec 6->1 (fail+error); chop/chomp unchanged at their
pre-existing UTF-32BE failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…whitebox tests

Read ruby/ruby v3_4_0 string.c and symbol.c:

- rb_str_each_char / rb_str_chars step with rb_enc_mbclen, so "what is a
  character" is the encoding's question: one byte in ASCII-8BIT, and 1 for a
  byte that begins no valid UTF-8 sequence. Ranging over a Go string decodes as
  UTF-8 unconditionally and yields U+FFFD per invalid byte, so "\xA4" in a UTF-8
  string came back as a three-byte replacement character and a binary string was
  split on boundaries that mean nothing in it. strCharPieces returns slices of
  the original, so no byte is rewritten.

- symbol.c rb_str_intern re-tags an ASCII-only string as US-ASCII whatever
  encoding it arrived in and otherwise keeps the string's own; sym_encoding is
  rb_obj_encoding(rb_sym2str(sym)). Symbol#encoding was missing entirely.
  rbgo's Symbol carries no encoding tag, so the non-ASCII answer is derived from
  the bytes -- UTF-8 when valid, ASCII-8BIT when not, which matches MRI for
  every case this VM can represent. A symbol interned from a UTF-16LE String
  still answers by shape rather than provenance; the three to_sym specs that
  need the tag stay red on purpose rather than being accommodated.

- require nil now says "no implicit conversion of nil into String", MRI's
  rb_builtin_class_name wording, instead of naming NilClass.

TestLoadPathRequire pinned rbgo's older behaviour of silently skipping a
non-String $LOAD_PATH entry. MRI raises TypeError instead, and
rb_get_expanded_load_path coerces the whole list before the search, so the
position of the bad entry does not matter -- witnessed against MRI 4.0.5 with
123 both unshifted before and pushed after a directory that holds the feature.
The test now pins the MRI-verified behaviour and adds the #to_path entry case.

wave25_kernstr_test.go covers every new branch: isExplicitRelative,
builtinClassName, chopStr per encoding, strCharPieces, the require #to_path /
#to_str chain, LoadError#path, the resolution rules, requireRelativeBase, the
circular-require warning in all three $VERBOSE states, splitCheckArgs and the
byteindex coercions. internal/vm coverage 99.9%, with require.go and
string_byteops.go at 100% and the below-100% set identical to main's.

core/string chars_spec and each_char_spec 3->1 each, to_sym_spec 6->3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…dError

search_required (ruby/ruby v3_4_0 load.c) asks rb_feature_p BEFORE
rb_find_file_ext. For a name carrying no extension, an entry in
$LOADED_FEATURES equal to that name answers 'u':

    if (!*(e = f + len)) { if (ext) continue; return 'u'; }

which reaches `case 0: if (ft) goto feature_present;`, so require_internal
returns with *path == 0 and reports false rather than failing to load.

Only the verbatim form is matched. MRI also accepts an entry spelled
"<$LOAD_PATH entry>/<name>", and its loaded-features index carries the
"distractor" rule for entries with an unusable extension; neither is reproduced
here. The check can only turn a LoadError into false, never the other way
round, so it cannot hide a require that would otherwise work.

Witnessed against MRI 4.0.5, including the three cases that must still raise: an
extensioned name is not matched by a bare entry, an unlisted name still fails,
and require_relative does not take this path at all.

core/kernel require_spec 16->14 (fail+error).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e too

MRI's isdirsep is `(x) == '/'` on POSIX and additionally accepts a backslash
only on Windows, so is_explicit_relative (file.c v3_4_0) gives a different
answer for ".\x" on each platform: explicitly relative there, an ordinary
$LOAD_PATH name here. The first version counted a backslash on both.

os.IsPathSeparator draws exactly that line. The test asserts
runtime.GOOS == "windows" for those two cases rather than pinning one
platform's answer everywhere -- the windows lane runs the same tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ith "/"

The windows lane failed this assertion. It checked that the path
require_relative reports is absolute by testing `start_with?("/")`, which is
only true on POSIX: featurePath names a file the way MRI does, so on Windows
the value is "C:/Users/.../nope_xyz" — absolute, and correctly so, but not
rooted at "/".

Ask the question directly instead: `e.path == File.expand_path(e.path)` is true
exactly when the path is already expanded, on either platform. Verified against
MRI 4.0.5 and against this branch's binary, both printing "true true".

The VM is right here and the assertion was written POSIX-shaped; this states
what the test means rather than what one platform happens to look like.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Sep 21, 2026
The windows lane of #618 died with `panic: test timed out after 50m0s`. The
dump says the budget ran out rather than something hanging: the test listed as
running had been running for 0s, and the binary's own total was 3007.445s —
50.1 minutes, a few seconds past the alarm.

The three OS lanes run the full suite under -race with coverage, and windows is
roughly twice the wall-clock of macOS and ubuntu. The vm suite grows with every
conformance wave, so the margin that 50m left has been consumed. 90m restores
it for several more.

This is deliberately generous rather than exact: a timeout is a backstop against
a hang, and the job-level budget (6h) is the real ceiling. Sizing it to just
above today's runtime would mean revisiting it every wave, and each revisit is
a red lane first.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit ddf7195 into main Sep 21, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave25/kernstr branch September 21, 2026 20:21
tannevaled added a commit that referenced this pull request Sep 23, 2026
Seven units since the last bump:

  #617  core/process: spawn, exec, the rlimits, priorities, process groups,
        kill, the wait family                     (pass 20 -> 165)
  #618  core/kernel + core/string: require_relative with an absolute path,
        the require/load argument contract, LoadError#path, byteindex/
        byterindex, split, chop/chars/each_char   (kernel 233 -> 184,
                                                   string 179 -> 153 failing)
  #620  language: compound assignment evaluation order, the flip-flop
        operator, magic-comment encodings, loop break values
                                                  (pass 1308 -> 1443)
  #622  object: Float#to_s laid out by MRI's rule rather than Go's %g  (+11)
  #624  the mspec shim: a nested describe can see the helpers of the one
        around it                                                     (+28)
  #625  deps: parser v0.2.0                        (language +333)
  #626  compiler: for-loop targets, block-local variables, what defined?
        declares                                   (language +382)

Measured with scripts/conformance/rubyspec/run.sh against the pinned corpus at
87b1631, 16-way, TIMEOUT=25, three consecutive runs: 21749, 21750, 21750. The
floor is set nine below the lowest, the same margin rule as the last bump: it
absorbs the run-to-run jitter without being wide enough to hide the loss of a
whole spec file, which is what the ratchet exists to catch. #615 (an
intermittent frame-pop crash) is still open and can cost a file when it fires.

TWO OF THESE ARE NOT VM CONFORMANCE, and the number should not be read as if
they were. #624 fixed our own mspec shim: core/string/valid_encoding/utf_8_spec
scored 0/28 under MRI 4.0.5 too, so those 28 were harness noise counted as VM
failure. #625 is a dependency bump; the VM did not change, but thirteen
language spec files became parseable for the first time and brought 333 already-
passing examples into view. Both are real gains in what the measurement can
see, not in what the VM can do.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant