Skip to content

conformance: the should-proxy answered its own predicates -- 2806 assertion sites asserted nothing (#655) - #660

Merged
tannevaled merged 3 commits into
mainfrom
wave31/shim
Sep 25, 2026
Merged

tannevaled merged 3 commits into
mainfrom
wave31/shim

Conversation

@tannevaled

@tannevaled tannevaled commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Closes #655.

The defect, and how big it was

ShouldProxy inherited Object. Every predicate Object already answers was
therefore dispatched on the proxy and its result discarded, so the
assertion was never made. x.should.equal?(y) compared the proxy with y,
got false, threw it away, and the example passed whatever the two values were.

Eleven spellings were affected — nine predicates and two operators. Counted in
language/ + core/ of the pinned corpus (87b16319):

spelling should. should_not. files what it did instead
equal? 765 147 376 Object#equal? on the proxy → always false → discarded
instance_of? 738 2 341 Object#instance_of? on the proxy
eql? 542 26 68 Object#eql? on the proxy
is_a? 418 2 181 Object#is_a? on the proxy
frozen? 68 35 35 Object#frozen? on the proxy
respond_to? 21 30 32 Object#respond_to? → always true (respond_to_missing? returned true)
nil? 2 6 3 Object#nil? on the proxy
kind_of? 2 0 2 Object#kind_of? on the proxy
instance_variable_defined? 0 2 1 Object#instance_variable_defined? on the proxy
should === y 4 1 3 Object#=== → the proxy's ==, comparing the wrong pair — a false FAILURE
should !~ re 11 0 1 Object#!~ = !(x =~ y) → ran the proxy's positive =~ → the exact INVERSE

2 806 assertion sites across 811 files asserted nothing (or, for the last
two rows, asserted something else).

The fix: inherit less

Upstream mspec is immune by construction, and this reproduces it. Its operator
matchers subclass BasicObject
(mspec/lib/mspec/matchers/base.rb — class SpecPositiveOperatorMatcher < BasicObject), which answers only ==, !=, !, equal?, __send__,
__id__, instance_eval, instance_exec and method_missing. mspec spells
out the three it must and lets every other predicate fall through
#method_missing onto the real receiver, where its truthiness is asserted
(SpecExpectation.fail_predicate,
mspec/lib/mspec/expectations/expectations.rb).

So: class ShouldProxy < BasicObject, with equal? spelled out beside the
existing ==/!=, !~ spelled out, the deprecated matcher spellings
equal/eql kept (mspec/lib/mspec/matchers/equal.rb — the receiver has no
such method to forward to), and respond_to_missing? dropped. This also covers
spellings the corpus has not used yet: a predicate is broken exactly when the
proxy answers it
, so the proxy must answer as little as possible.

Every constant inside the class is now ::-qualified. From inside a
BasicObject subclass MRI does not reach the top-level constants, so a bare
Exception raised NameError: uninitialized constant ShouldProxy::Exception
under MRI 4.0.5 — while rbgo resolved it happily. Since MRI through this shim
is the judge that separates a real rbgo defect from an inexpressible assertion,
that difference would have broken the judge and not us.

The measurement

One host, serially (an 8-way sweep of this corpus is not reproducible:
core/kernel/require_spec.rb reads 69 alone and 68 under load), three runs
each side, same corpus, same rbgo binary built from this branch.

run 1 run 2 run 3 median
before 22340 22338 22339 22339
after 22136 22137 22136 22136

Decomposed per example: 205 examples newly FAIL, 2 newly PASS, and
core/random/bytes_spec.rb drifts ±1 between any two runs (it and
core/module/autoload_spec.rb are the only files that move between two
identical runs). 2 − 205 = −203; the −202 median-to-median difference is that
flake.

Counting note, because it bit me: matching failures by (kind, description)
first read 207 and 4. Two examples changed kind from error to fail —
a NoMethodError raised downstream of a vacuous assertion became the
assertion's own failure — and a kind change looks like one loss plus one
gain. Keyed on description alone the pair is 205 and 2.

106 files moved: 103 down, 3 up (one of the three up is the flake).

Classification of the new failures — MRI 4.0.5 as the per-file ceiling

Every one of the 205 newly-failing examples was matched by description against
the same file run under MRI 4.0.5 through the same fixed shim:

  • (a) real rbgo defect the vacuous assertion was hiding: 205 / 205
  • (b) assertion the shim still cannot express: 0 / 205

A sample, with the ceiling beside it:

file rbgo before rbgo after MRI 4.0.5 what the hidden defect is
core/io/internal_encoding_spec.rb 79 67 79 IO#internal_encoding returns nil where MRI returns IBM866
core/io/external_encoding_spec.rb 59 48 59 IO#external_encoding returns the wrong Encoding object
core/rational/exponent_spec.rb 24 17 24 Rational#eql? is false for (1/1).eql?((1/1))
language/predefined_spec.rb 124 117 151 $& carries the wrong encoding; $& assignment raises NameError, not SyntaxError
language/singleton_class_spec.rb 42 35 53 singleton classes of true/false/nil
core/builtin_constants/builtin_constants_spec.rb 17 11 17 RUBY_VERSION, RUBY_COPYRIGHT … are not frozen
core/exception/receiver_spec.rb 7 1 8 NameError#receiver returns nil
core/numeric/step_spec.rb 208 202 208 float step yields 12.700000000000001 for 12.7
core/time/subsec_spec.rb 6 1 6 Time#subsec Rational identity (eql?)
core/dir/children_spec.rb 13 9 13 Dir.children ignores the requested encoding
core/hash/constructor_spec.rb 20 16 20 Hash.[] returns Hash where a subclass is required
core/integer/to_s_spec.rb 10 6 10 Integer#to_s returns UTF-8, not US-ASCII

Both honest gains are examples the shadowing made fail, not pass:
x.should === y reached Object#=== → the proxy's == and compared a class
with an instance (core/exception/case_compare_spec.rb,
core/range/case_compare_spec.rb).

One divergence this could not fix, stated rather than papered over

x.should !~ /re/ is now spelled out on the proxy and is correct under MRI.
rbgo never dispatches !~ as a method — it compiles a !~ b into
!(a =~ b) (and inlines ! the same way), where MRI sends :!~. So under
rbgo the definition is dead code and those 11 sites in
core/exception/full_message_spec.rb still run the positive =~ assertion,
i.e. the inverse of what the spec wrote. That is an rbgo/MRI divergence of its
own and worth its own issue; the shim cannot reach it from here. It changes no
number in this PR.

FLOOR: 22345 → 22170

The number fell because the measurement got stricter, not because the VM
regressed.
No Go code changed on this branch; CGO_ENABLED=0 GOWORK=off go test ./... is green.

The floor is taken from the lane that enforces it, which is also where the
22345 came from:

before after
this host, serially, ×3 22340 / 22338 / 22339 22136 / 22137 / 22136
ruby/spec ratchet CI 22345 (frozen) 22170 (measured on this branch)

The 34-example gap between the two environments is the environment, not the
change: on this host the unchanged shim read 22339, six below the frozen
22345, so part of that number was never reproducible here either. A floor 34
below what the enforcing lane reads would be slack rather than safety, so the
gate moves 22345 → 22170, a drop of 175, all of it measurement.

A floor that no longer reflects a truthful measurement is worse than a lower
one. The 205 examples that now fail were never passing on their merits, and
each one is a real rbgo defect with a named ceiling.

Per-file movement, all 106 files (before → after, median of three runs each)
delta file before after note
-12 core/io/internal_encoding_spec.rb 79 67
-11 core/io/external_encoding_spec.rb 59 48
-7 core/rational/exponent_spec.rb 24 17
-7 language/predefined_spec.rb 124 117
-7 language/singleton_class_spec.rb 42 35
-6 core/builtin_constants/builtin_constants_spec.rb 17 11
-6 core/exception/receiver_spec.rb 7 1
-6 core/numeric/step_spec.rb 208 202
-5 core/time/subsec_spec.rb 6 1
-4 core/dir/children_spec.rb 13 9
-4 core/hash/constructor_spec.rb 20 16
-4 core/integer/to_s_spec.rb 10 6
-3 core/marshal/load_spec.rb 229 226
-3 core/range/step_spec.rb 76 73
-3 core/string/uminus_spec.rb 6 3
-3 core/time/new_spec.rb 115 112
-2 core/dir/entries_spec.rb 7 5
-2 core/env/shift_spec.rb 4 2
-2 core/exception/name_spec.rb 4 2
-2 core/false/to_s_spec.rb 3 1
-2 core/float/ceil_spec.rb 8 6
-2 core/float/floor_spec.rb 7 5
-2 core/float/to_s_spec.rb 27 25
-2 core/kernel/Float_spec.rb 99 97
-2 core/matchdata/post_match_spec.rb 4 2
-2 core/matchdata/pre_match_spec.rb 4 2
-2 core/module/name_spec.rb 18 16
-2 core/nil/to_s_spec.rb 3 1
-2 core/proc/dup_spec.rb 2 0
-2 core/range/frozen_spec.rb 4 2
-2 core/rational/ceil_spec.rb 9 7
-2 core/rational/divide_spec.rb 9 7
-2 core/rational/divmod_spec.rb 7 5
-2 core/rational/floor_spec.rb 8 6
-2 core/rational/minus_spec.rb 6 4
-2 core/rational/multiply_spec.rb 6 4
-2 core/rational/plus_spec.rb 6 4
-2 core/string/each_grapheme_cluster_spec.rb 15 13
-2 core/string/grapheme_clusters_spec.rb 13 11
-2 core/time/at_spec.rb 39 37
-2 core/time/dup_spec.rb 6 4
-2 core/time/to_r_spec.rb 2 0
-2 core/true/to_s_spec.rb 3 1
-1 core/argf/argf_spec.rb 2 1
-1 core/array/max_spec.rb 13 12
-1 core/array/min_spec.rb 13 12
-1 core/array/to_a_spec.rb 3 2
-1 core/array/to_ary_spec.rb 2 1
-1 core/basicobject/basicobject_spec.rb 7 6
-1 core/complex/divide_spec.rb 9 8
-1 core/dir/exist_spec.rb 12 11
-1 core/dir/to_path_spec.rb 3 2
-1 core/enumerable/to_set_spec.rb 3 2
-1 core/enumerator/arithmetic_sequence/each_spec.rb 2 1
-1 core/env/each_pair_spec.rb 5 4
-1 core/env/element_reference_spec.rb 5 4
-1 core/exception/backtrace_spec.rb 13 12
-1 core/exception/frozen_error_spec.rb 1 0
-1 core/fiber/new_spec.rb 5 4
-1 core/file/expand_path_spec.rb 26 25
-1 core/float/arg_spec.rb 6 5
-1 core/float/to_i_spec.rb 2 1
-1 core/float/truncate_spec.rb 3 2
-1 core/hash/element_set_spec.rb 11 10
-1 core/hash/merge_spec.rb 25 24
-1 core/hash/to_hash_spec.rb 2 1
-1 core/integer/ceil_spec.rb 10 9
-1 core/integer/floor_spec.rb 9 8
-1 core/io/read_spec.rb 105 104
-1 core/kernel/Rational_spec.rb 31 30
-1 core/kernel/backtick_spec.rb 3 2
-1 core/kernel/singleton_class_spec.rb 5 4
-1 core/kernel/sprintf_spec.rb 167 166
-1 core/matchdata/string_spec.rb 4 3
-1 core/module/define_method_spec.rb 85 84
-1 core/module/module_eval_spec.rb 16 15
-1 core/numeric/clone_spec.rb 4 3
-1 core/numeric/conjugate_spec.rb 1 0
-1 core/numeric/dup_spec.rb 2 1
-1 core/numeric/quo_spec.rb 7 6
-1 core/objectspace/_id2ref_spec.rb 11 10
-1 core/proc/clone_spec.rb 2 1
-1 core/proc/new_spec.rb 11 10
-1 core/proc/source_location_spec.rb 5 4
-1 core/range/clone_spec.rb 2 1
-1 core/range/dup_spec.rb 2 1
-1 core/range/new_spec.rb 10 9
-1 core/range/to_set_spec.rb 4 3
-1 core/string/chars_spec.rb 11 10
-1 core/string/chomp_spec.rb 65 64
-1 core/string/each_char_spec.rb 11 10
-1 core/string/freeze_spec.rb 3 2
-1 core/string/modulo_spec.rb 265 264
-1 core/thread/start_spec.rb 4 3
-1 core/time/ceil_spec.rb 7 6
-1 core/time/floor_spec.rb 5 4
-1 core/time/inspect_spec.rb 8 7
-1 core/time/local_spec.rb 47 46
-1 core/time/now_spec.rb 17 16
-1 core/time/round_spec.rb 5 4
-1 core/time/to_s_spec.rb 3 2
-1 core/tracepoint/raised_exception_spec.rb 2 1
-1 language/range_spec.rb 5 4
+1 core/exception/case_compare_spec.rb 5 6
+1 core/random/bytes_spec.rb 8 9 ±1 run-to-run flake
+1 core/range/case_compare_spec.rb 41 42

🤖 Generated with Claude Code

tannevaled and others added 3 commits September 25, 2026 21:33
…655)

`ShouldProxy` inherited Object, so every predicate Object already answers
was dispatched on the PROXY instead of the receiver and its result thrown
away. `x.should.equal?(y)` therefore asserted nothing at all: it compared
the proxy with y, got false, discarded it, and the example passed whatever
the values were. 765 `.should.equal?` uses across 328 files of language/ +
core/ could not fail. Same for `.should.eql?`, `.should.is_a?`,
`.should.kind_of?`, `.should.instance_of?`, `.should.nil?`,
`.should.frozen?`, `.should.respond_to?` and
`.should.instance_variable_defined?` -- nine spellings in all.

Upstream mspec is immune by construction: its operator matchers subclass
BasicObject (mspec/lib/mspec/matchers/base.rb,
`class SpecPositiveOperatorMatcher < BasicObject`), which answers only ==,
!=, !, equal?, __send__, __id__, instance_eval, instance_exec and
method_missing. mspec spells out those it must and lets every other
predicate fall through #method_missing onto the real receiver, where its
truthiness is asserted.

Do the same: `class ShouldProxy < BasicObject`, with `equal?` spelled out
beside the existing `==`/`!=`, and the deprecated matcher spellings
`equal`/`eql` (mspec/lib/mspec/matchers/equal.rb) kept because the receiver
has no such method to forward to. This also covers spellings the corpus has
not used yet: a predicate is broken exactly when the proxy answers it.

Every constant inside the class is now ::-qualified. From a BasicObject
subclass MRI does not reach top-level constants, so a bare `Exception`
raised `NameError: uninitialized constant ShouldProxy::Exception` under
MRI 4.0.5 -- while rbgo resolved it. The shim has to run under MRI too (it
is the judge that separates a real rbgo defect from an inexpressible
assertion), so the difference would have broken the judge and not us.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…regressed

Two parts.

1. `x.should !~ /re/` (11 sites in core/exception/full_message_spec.rb).
   Object#!~ is defined as `!(x =~ y)`, so the spelling ran the proxy's
   POSITIVE `=~` assertion and threw the result away: it asserted the exact
   INVERSE of what the spec wrote. Spelling `!~` out on the proxy asserts the
   right thing under MRI. Measured caveat, in the file: rbgo does not dispatch
   !~ as a method at all -- it compiles `a !~ b` into `!(a =~ b)`, and inlines
   `!` the same way -- so under rbgo the definition is dead code and those 11
   sites stay inverted until the VM sends :!~. That divergence is rbgo's, not
   the shim's, and it moves no number here.

2. FLOOR 22345 -> 22136.

   THE NUMBER FELL BECAUSE THE MEASUREMENT GOT STRICTER, NOT BECAUSE THE VM
   REGRESSED. No Go code changed in this branch.

   Measured on one host, serially (an 8-way sweep of this corpus is not
   reproducible: core/kernel/require_spec.rb reads 69 alone and 68 under load),
   three runs each, same corpus 87b16319, same rbgo binary:

     before the shim fix : 22340 / 22338 / 22339   (median 22339)
     after               : 22136 / 22137 / 22136   (median 22136)

   Note the frozen floor of 22345 was already 6 above what this host measures
   serially for the UNCHANGED shim, so 22136 is the lowest honest reading of
   the four post-fix runs rather than their median.

   The movement, per example: 205 examples newly FAIL, 2 newly PASS, and
   core/random/bytes_spec.rb drifts +-1 between any two runs: 2 - 205 = -203,
   and the -202 median-to-median difference is that flake. 106 files moved, 103
   down and 3 up (one of the three is the flake).

   (Counting note: matching failures by (kind, description) first read 207 and
   4, because two examples changed KIND from error to fail -- a NoMethodError
   downstream of a vacuous assertion became the assertion's own failure -- and
   a kind change looks like one loss plus one gain. Keyed on description alone
   the pair is 205 and 2.)

   All 205 newly-failing examples were classified with MRI 4.0.5 run through
   the SAME fixed shim as the per-file ceiling: 205 of 205 are real rbgo
   defects that the vacuous assertion was hiding, and 0 are assertions the shim
   cannot express. Examples:

     core/io/internal_encoding_spec.rb   79 -> 67, MRI 79: IO#internal_encoding
                                         returns nil where MRI returns IBM866
     core/rational/exponent_spec.rb      24 -> 17, MRI 24: Rational#eql? says
                                         false for (1/1).eql?((1/1))
     core/builtin_constants/...          17 -> 11, MRI 17: RUBY_VERSION and
                                         friends are not frozen
     core/hash/constructor_spec.rb       20 -> 16, MRI 20: Hash.[] returns Hash
                                         where a subclass is required
     core/numeric/step_spec.rb          208 -> 202, MRI 208: float step yields
                                         12.700000000000001 for 12.7

   Both honest gains are examples the shadowing made FAIL, not pass:
   `x.should === y` reached Object#=== -> the proxy's `==`, comparing a class
   with an instance (core/exception/case_compare_spec.rb,
   core/range/case_compare_spec.rb).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…2136

The first reading came from one developer host, serially. The ratchet lane
then measured 22170 on the same commit -- 34 above it. That gap is the
environment, not the change: the previous floor of 22345 was a CI number too,
and on this host the UNCHANGED shim read 22339, six below it.

Comparing like with like, the honest statement of what this branch did to the
gate is 22345 -> 22170 on CI, a drop of 175, all of it measurement:

  this host, serially   before 22340/22338/22339   after 22136/22137/22136
  ruby/spec ratchet CI  before 22345 (frozen)      after 22170 (measured)

A floor 34 below what the enforcing lane reads is slack, not safety, so take
the lane's own number -- which is how README.md says to re-freeze it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 9ec6a52 into main Sep 25, 2026
7 checks passed
@tannevaled
tannevaled deleted the wave31/shim branch September 25, 2026 21:22
tannevaled added a commit that referenced this pull request Sep 26, 2026
Read this number against the previous one with care: the floor FELL from 22345
to 22170 earlier in this same wave, because #660 fixed the harness. The
comparison that means something is 22170 -> 22205, both measured with the
corrected shim.

  #660  The shim's ShouldProxy inherited Object, so every predicate Object
        answers was dispatched on the PROXY and discarded. Eleven spellings,
        2806 assertion sites across 811 files. Two were not merely vacuous:
        `should === y` compared the wrong pair (a false FAILURE) and
        `should !~ re` ran the POSITIVE match (the exact inverse). Fixed by
        inheriting BasicObject, as upstream mspec does. 205 examples newly
        fail; classified against MRI through the same shim, 205/205 are real
        rbgo defects the vacuous assertion had hidden and 0/205 are limits of
        the harness.
  #662  const_source_location 13 -> 39, const_added 4 -> 13, classpath by
        permanence, `class Foo::Bar` firing a pending autoload — and the #615
        crash, FIXED. That flake had distorted every core/module measurement
        for three weeks.
  #661  Refinements: the Refinement class surface, a refinement's superclass
        being the class it refines, `using` replacing the frame cref.

Measured with scripts/conformance/rubyspec/run.sh against the pinned corpus at
87b1631, 16-way, TIMEOUT=25, three consecutive runs: 22221, 22221, 22220. The
floor is set fifteen below the lowest.

Two agents had their briefing premise overturned by measurement this wave, both
mine: a gsub defect I routed had been fixed three waves earlier, and a
refinement activation lead was fixed between its measurement and the brief.
Running MRI through our own shim to get a per-file ceiling is now the standing
method, and it is what caught both.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Sep 26, 2026
Three units, +283 measured, and the first wave whose numbers are trustworthy
throughout: #660 repaired the harness at the end of wave 31, so this is the
first measurement in the campaign with no known vacuous assertions under it.

  #666  core/kernel, numerics, symbols (+162). 45 of those were INVISIBLE
        rather than failing: core/integer/{pow,exponent}_spec.rb each scored 0
        because the corpus contains `100000000 ** 1000000000` and rbgo computed
        it unbounded until the harness killed the process. bignum.c's
        BIGLEN_LIMIT refuses it. core/rational and core/builtin_constants are
        now AT the ceiling.
  #668  IO/Dir encodings resolved once at open, #syswrite reaching the
        descriptor, IO::Buffer's base/size distinction (+56). THIRTEEN of the
        sixteen files it touched now sit exactly at the ceiling the harness can
        express.
  #665  singleton classes of nil/true/false, post parameters in Method#arity
        AND #parameters together, and `!~`/`!`/`!=` DISPATCHED rather than
        synthesised — one cause, three defects, found by asking whether #663's
        neighbours shared its shape.

Measured with scripts/conformance/rubyspec/run.sh against the pinned corpus at
87b1631, 16-way, TIMEOUT=25, three consecutive runs: 22489, 22489, 22488. The
floor is set thirteen below the lowest.

Three of my briefing targets dissolved under measurement again — the third wave
running. core/io/write_spec's "9 failures" is a real gap of 1 (MRI fails the
other eight through this shim); core/array/fill_spec's "9" is a gap of ZERO
(ceiling 38, rbgo 38); core/numeric/step_spec's 12.700000000000001 is what MRI
prints. Running MRI through our own shim for a per-file ceiling is what catches
these, and it is now done before planning rather than after.

The largest lever left is not in any of these lanes: core/file/printf_spec.rb
scores 0 against a ceiling of 160 because rbgo cannot parse a do...end block
inside an arrow-lambda that is a paren-less command argument. One parser fix,
160 examples — issue #669.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mspec shim: should.equal? asserts nothing — 765 uses across 328 corpus files

1 participant