Skip to content

wave 34 (encoding): transcoding errors carry their attributes, an encoding tag is resolved before it keys a codec (+37) - #679

Merged
tannevaled merged 4 commits into
mainfrom
wave34-string
Sep 26, 2026
Merged

tannevaled merged 4 commits into
mainfrom
wave34-string

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Wave 34 (core/string, core/symbol, core/encoding). Measured +37 ruby/spec
examples: 22489 → 22526 (core + language, serial 2-way, TIMEOUT=25,
corpus 87b1631). No file moved down. FLOOR is untouched.

Per file: before → after → measured MRI ceiling

Every ceiling below is MRI 4.0.5 run through our own mspec shim, not the file's
example count.

file before after MRI ceiling
core/encoding/undefined_conversion_error/source_encoding_spec.rb 0 3 3
core/encoding/undefined_conversion_error/source_encoding_name_spec.rb 0 3 3
core/encoding/undefined_conversion_error/error_char_spec.rb 0 3 3
core/encoding/undefined_conversion_error/destination_encoding_spec.rb 0 2 2
core/encoding/undefined_conversion_error/destination_encoding_name_spec.rb 0 2 2
core/encoding/invalid_byte_sequence_error/source_encoding_spec.rb 0 3 3
core/encoding/invalid_byte_sequence_error/source_encoding_name_spec.rb 0 3 3
core/encoding/invalid_byte_sequence_error/error_bytes_spec.rb 0 3 3
core/encoding/invalid_byte_sequence_error/readagain_bytes_spec.rb 0 3 3
core/encoding/invalid_byte_sequence_error/destination_encoding_spec.rb 0 2 2
core/encoding/invalid_byte_sequence_error/destination_encoding_name_spec.rb 0 2 2
core/encoding/invalid_byte_sequence_error/incomplete_input_spec.rb 1 3 3
core/string/encode_spec.rb 150 152 152
core/string/encoding_spec.rb 24 25 25
core/string/unicode_normalize_spec.rb 12 14 14
core/string/unicode_normalized_spec.rb 10 11 11

All sixteen files are now at their ceiling.

What was wrong, and the C functions read for each decision

1. The transcoding exceptions carried no attributes (+33).
Encoding::UndefinedConversionError and Encoding::InvalidByteSequenceError had
none of #source_encoding, #source_encoding_name, #destination_encoding,
#destination_encoding_name, #error_char, #error_bytes, #readagain_bytes,
#incomplete_input?. The data was already right — #primitive_errinfo reported it
byte for byte — but two things kept it off the object.

Read: make_econv_exception, rb_econv_check_error, ecerr_source_encoding,
ecerr_source_encoding_name, ecerr_destination_encoding,
ecerr_destination_encoding_name, ecerr_error_char, ecerr_error_bytes,
ecerr_readagain_bytes, ecerr_incomplete_input (transcode.c).

make_econv_exception picks the class, formats the message and attaches the
attributes in one breath, and rb_econv_check_error hands that very object to
rb_exc_raise. rbgo recorded one exception in #last_error and then raised a
second one built from the message alone, so what the user rescued had nothing to
read. setEconvErrorAttrs now mirrors make_econv_exception (including
@readagain_bytes being nil, not "", when readagain_len is 0, and the
set_encs rule that a name is always stored but an Encoding object only when the
name resolves), and #convert / #finish raise the recorded object itself. Each
accessor is rb_attr_get of one ivar, so Encoding::InvalidByteSequenceError.new .incomplete_input? is nil rather than a NoMethodError.

2. An encoding TAG was used raw where a canonical name was needed (+1).
MagicSourceEncoding's comment claimed the vm's case-insensitive lookup made
# encoding: iso-8859-9 resolve to the canonical object. That is true of
String#encoding, which goes through the registry — and not true of the
transcoding path, which keyed xtextEncodings on the literal tag. So such a file's
literals reported ISO-8859-9 yet raised
ConverterNotFoundError (iso-8859-9 to UTF-8) on #encode.

Read: rb_enc_associate_index, rb_enc_find_index (encoding.c). MRI cannot have
this defect: a String carries an encoding index, so the spelling is resolved
once and every later comparison is index equality. canonicalEncName is that
resolution, in one place, and encodeEncName's private copy of it is gone.

3. String#encode looked for a converter MRI never needs (+2).
Read: str_transcode0 (transcode.c). It short-circuits before any converter
is looked up when source and destination encodings match and no newline/xml
decorator is in play: with an explicit invalid: the result is
rb_enc_str_scrub in that one encoding, otherwise the bytes pass through. rbgo
checked converter availability first, so
"\x80".force_encoding("Emacs-Mule").encode(invalid: :replace) raised instead of
returning "?".

That case also needed an Emacs-Mule character automaton, transcribed from the
trans table of enc/emacs_mule.c (the grammar comment there is not enough: the
three secondary leads take different extension ranges, and EncLen_EmacsMule is
only an upper bound — MRI judges validity with the automaton). validInEncoding now
asks that same scanner instead of keeping a second opinion, so #scrub,
#valid_encoding? and this #encode path cannot disagree about what a well-formed
character is.

4. #unicode_normalize accepted any encoding (+3).
Read: UnicodeNormalize.normalize / .normalized? in
lib/unicode_normalize/normalize.rb — this one is Ruby, not C. It dispatches on the
encoding first: UTF-8 direct, US-ASCII already normalized in every form, the
UNICODE_ENCODINGS list through UTF-8 and back, everything else
Encoding::CompatibilityError. rbgo normalized the bytes of any encoding as if they
were UTF-8, so ISO-8859-1 reported "invalid byte sequence in UTF-8" where MRI
reports the incompatibility. The order matters and is now preserved: that
ArgumentError can only come from the UTF-8 branch.

Verified in both directions

  • Witnesses, not guards. TestEconvErrorAttributes, TestEncodeSameEncoding,
    TestValidInEncodingUsesTheScanner and
    TestUnicodeNormalizeEncodingDispatch were each run against the unfixed product
    — the behaviour reverted while the new symbols stayed, so the mutation still
    compiled — and each failed. The remaining tests in the file cover helpers
    that only exist with the fix; they are labelled GUARDS in the file header.
  • Coverage. The below-100% function list is byte-identical to
    origin/main's, measured with the same -coverpkg=./internal/... profile on a
    detached worktree at origin/main. Nothing was orphaned. (Totals were not
    compared: both read 99.9%.) One default: branch in setConvError that no
    caller can reach was removed rather than left untestable — setEconvErrorAttrs
    already holds that judgement, reachably and tested.
  • Full go test ./..., go build ./..., GOOS=wasip1 GOARCH=wasm go build ./...,
    gofmt -l internal/ cmd/ all clean.

Left undone, with reasons

  • core/string/valid_encoding_spec.rb (1 example short of its ceiling of 8) needs
    CP949, Emacs-Mule, EUC-TW and ISO-2022-JP to agree with MRI in one
    assertion. Emacs-Mule and ISO-2022-JP now would; CP949 and EUC-TW need their
    Onigmo DFAs (enc/cp949.c, enc/euc_tw.c) transcribed — several hundred lines
    of table for one example. Not worth this wave.
  • core/encoding/converter/last_error_spec.rb and finish_spec.rb (1 each):
    ISO-8859-1 → Big5 does not raise at all in rbgo, and the stateful ISO-2022-JP
    converter does not emit its closing escape from #finish. Both are
    converter_codec.go codec coverage, not error plumbing.
  • String#crypt (6 examples) is a genuine gap and needs real DES crypt(3).
    Deliberately out of scope here.

🤖 Generated with Claude Code

tannevaled and others added 4 commits September 26, 2026 14:14
Encoding::UndefinedConversionError and
Encoding::InvalidByteSequenceError had no #source_encoding,
#destination_encoding, #error_char, #error_bytes, #readagain_bytes or
#incomplete_input?, so 28 ruby/spec examples raised NoMethodError. The
data was already computed correctly — Encoding::Converter#primitive_errinfo
reported it byte for byte — but two things kept it off the exception.

MRI's make_econv_exception (transcode.c) picks the class, formats the
message and attaches the attributes in one breath, and
rb_econv_check_error passes that very object to rb_exc_raise. rbgo
recorded an exception in #last_error and then raised a *second* one built
from the message alone, so the object the user rescued had no attributes
to read.

setEconvErrorAttrs now mirrors make_econv_exception, and #convert /
#finish raise the recorded object itself, so the rescued exception and
#last_error can never disagree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ere MRI does

Three defects that all live on the String#encode path.

An encoding TAG was used raw where a canonical name was needed.
MagicSourceEncoding's comment says the vm's case-insensitive lookup makes
`# encoding: iso-8859-9` resolve to the canonical object — true of
String#encoding, which goes through the registry, but NOT of the transcoding
path, which keyed xtextEncodings on the literal tag. So a file with that
magic comment produced strings reporting ISO-8859-9 that no converter could
be found for. MRI cannot have this bug: rb_enc_associate_index (encoding.c)
resolves the spelling once and a String then carries an encoding INDEX.
canonicalEncName is that resolution, in one place.

String#encode raised ConverterNotFoundError where MRI needs no converter.
str_transcode0 (transcode.c) short-circuits before any converter is looked
up when source and destination encodings match and no decorator is in play:
with an explicit `invalid:` it is rb_enc_str_scrub in that one encoding,
otherwise the bytes pass through. rbgo checked converter availability first.

#unicode_normalize accepted any encoding. UnicodeNormalize.normalize
(lib/unicode_normalize/normalize.rb) dispatches on the encoding first: UTF-8
direct, US-ASCII already normalized, the other Unicode encodings through
UTF-8, everything else Encoding::CompatibilityError. rbgo normalized the
bytes of any encoding as if they were UTF-8, so ISO-8859-1 reported
"invalid byte sequence in UTF-8" where MRI reports the incompatibility.

Also adds the Emacs-Mule character automaton, transcribed from the `trans`
table of enc/emacs_mule.c, so #scrub, #valid_encoding? and the new
same-encoding #encode path agree about what a well-formed Emacs-Mule
character is. validInEncoding now asks that same scanner rather than keeping
a second opinion.

ruby/spec: string/encode 150->152, string/encoding 24->25,
string/unicode_normalize 12->14, string/unicode_normalized? 10->11 (each now
at its measured MRI ceiling).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
setConvError had grown a `default: return` that no caller can reach — every
call site passes one of the three failure statuses — so it was an untestable
branch. setEconvErrorAttrs already returns without touching the exception for
any other status, and that guard IS reachable and tested, so the duplicate
one comes out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit ff047b8 into main Sep 26, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave34-string branch September 26, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant