Repository navigation
conformance: the should-proxy answered its own predicates -- 2806 assertion sites asserted nothing (#655) - #660
Merged
Merged
Conversation
…655) `ShouldProxy` inherited Object, so every predicate Object already answers was dispatched on the PROXY instead of the receiver and its result thrown away. `x.should.equal?(y)` therefore asserted nothing at all: it compared the proxy with y, got false, discarded it, and the example passed whatever the values were. 765 `.should.equal?` uses across 328 files of language/ + core/ could not fail. Same for `.should.eql?`, `.should.is_a?`, `.should.kind_of?`, `.should.instance_of?`, `.should.nil?`, `.should.frozen?`, `.should.respond_to?` and `.should.instance_variable_defined?` -- nine spellings in all. Upstream mspec is immune by construction: its operator matchers subclass BasicObject (mspec/lib/mspec/matchers/base.rb, `class SpecPositiveOperatorMatcher < BasicObject`), which answers only ==, !=, !, equal?, __send__, __id__, instance_eval, instance_exec and method_missing. mspec spells out those it must and lets every other predicate fall through #method_missing onto the real receiver, where its truthiness is asserted. Do the same: `class ShouldProxy < BasicObject`, with `equal?` spelled out beside the existing `==`/`!=`, and the deprecated matcher spellings `equal`/`eql` (mspec/lib/mspec/matchers/equal.rb) kept because the receiver has no such method to forward to. This also covers spellings the corpus has not used yet: a predicate is broken exactly when the proxy answers it. Every constant inside the class is now ::-qualified. From a BasicObject subclass MRI does not reach top-level constants, so a bare `Exception` raised `NameError: uninitialized constant ShouldProxy::Exception` under MRI 4.0.5 -- while rbgo resolved it. The shim has to run under MRI too (it is the judge that separates a real rbgo defect from an inexpressible assertion), so the difference would have broken the judge and not us. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…regressed
Two parts.
1. `x.should !~ /re/` (11 sites in core/exception/full_message_spec.rb).
Object#!~ is defined as `!(x =~ y)`, so the spelling ran the proxy's
POSITIVE `=~` assertion and threw the result away: it asserted the exact
INVERSE of what the spec wrote. Spelling `!~` out on the proxy asserts the
right thing under MRI. Measured caveat, in the file: rbgo does not dispatch
!~ as a method at all -- it compiles `a !~ b` into `!(a =~ b)`, and inlines
`!` the same way -- so under rbgo the definition is dead code and those 11
sites stay inverted until the VM sends :!~. That divergence is rbgo's, not
the shim's, and it moves no number here.
2. FLOOR 22345 -> 22136.
THE NUMBER FELL BECAUSE THE MEASUREMENT GOT STRICTER, NOT BECAUSE THE VM
REGRESSED. No Go code changed in this branch.
Measured on one host, serially (an 8-way sweep of this corpus is not
reproducible: core/kernel/require_spec.rb reads 69 alone and 68 under load),
three runs each, same corpus 87b16319, same rbgo binary:
before the shim fix : 22340 / 22338 / 22339 (median 22339)
after : 22136 / 22137 / 22136 (median 22136)
Note the frozen floor of 22345 was already 6 above what this host measures
serially for the UNCHANGED shim, so 22136 is the lowest honest reading of
the four post-fix runs rather than their median.
The movement, per example: 205 examples newly FAIL, 2 newly PASS, and
core/random/bytes_spec.rb drifts +-1 between any two runs: 2 - 205 = -203,
and the -202 median-to-median difference is that flake. 106 files moved, 103
down and 3 up (one of the three is the flake).
(Counting note: matching failures by (kind, description) first read 207 and
4, because two examples changed KIND from error to fail -- a NoMethodError
downstream of a vacuous assertion became the assertion's own failure -- and
a kind change looks like one loss plus one gain. Keyed on description alone
the pair is 205 and 2.)
All 205 newly-failing examples were classified with MRI 4.0.5 run through
the SAME fixed shim as the per-file ceiling: 205 of 205 are real rbgo
defects that the vacuous assertion was hiding, and 0 are assertions the shim
cannot express. Examples:
core/io/internal_encoding_spec.rb 79 -> 67, MRI 79: IO#internal_encoding
returns nil where MRI returns IBM866
core/rational/exponent_spec.rb 24 -> 17, MRI 24: Rational#eql? says
false for (1/1).eql?((1/1))
core/builtin_constants/... 17 -> 11, MRI 17: RUBY_VERSION and
friends are not frozen
core/hash/constructor_spec.rb 20 -> 16, MRI 20: Hash.[] returns Hash
where a subclass is required
core/numeric/step_spec.rb 208 -> 202, MRI 208: float step yields
12.700000000000001 for 12.7
Both honest gains are examples the shadowing made FAIL, not pass:
`x.should === y` reached Object#=== -> the proxy's `==`, comparing a class
with an instance (core/exception/case_compare_spec.rb,
core/range/case_compare_spec.rb).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…2136 The first reading came from one developer host, serially. The ratchet lane then measured 22170 on the same commit -- 34 above it. That gap is the environment, not the change: the previous floor of 22345 was a CI number too, and on this host the UNCHANGED shim read 22339, six below it. Comparing like with like, the honest statement of what this branch did to the gate is 22345 -> 22170 on CI, a drop of 175, all of it measurement: this host, serially before 22340/22338/22339 after 22136/22137/22136 ruby/spec ratchet CI before 22345 (frozen) after 22170 (measured) A floor 34 below what the enforcing lane reads is slack, not safety, so take the lane's own number -- which is how README.md says to re-freeze it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Sep 26, 2026
Read this number against the previous one with care: the floor FELL from 22345 to 22170 earlier in this same wave, because #660 fixed the harness. The comparison that means something is 22170 -> 22205, both measured with the corrected shim. #660 The shim's ShouldProxy inherited Object, so every predicate Object answers was dispatched on the PROXY and discarded. Eleven spellings, 2806 assertion sites across 811 files. Two were not merely vacuous: `should === y` compared the wrong pair (a false FAILURE) and `should !~ re` ran the POSITIVE match (the exact inverse). Fixed by inheriting BasicObject, as upstream mspec does. 205 examples newly fail; classified against MRI through the same shim, 205/205 are real rbgo defects the vacuous assertion had hidden and 0/205 are limits of the harness. #662 const_source_location 13 -> 39, const_added 4 -> 13, classpath by permanence, `class Foo::Bar` firing a pending autoload — and the #615 crash, FIXED. That flake had distorted every core/module measurement for three weeks. #661 Refinements: the Refinement class surface, a refinement's superclass being the class it refines, `using` replacing the frame cref. Measured with scripts/conformance/rubyspec/run.sh against the pinned corpus at 87b1631, 16-way, TIMEOUT=25, three consecutive runs: 22221, 22221, 22220. The floor is set fifteen below the lowest. Two agents had their briefing premise overturned by measurement this wave, both mine: a gsub defect I routed had been fixed three waves earlier, and a refinement activation lead was fixed between its measurement and the brief. Running MRI through our own shim to get a per-file ceiling is now the standing method, and it is what caught both. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Sep 26, 2026
Three units, +283 measured, and the first wave whose numbers are trustworthy throughout: #660 repaired the harness at the end of wave 31, so this is the first measurement in the campaign with no known vacuous assertions under it. #666 core/kernel, numerics, symbols (+162). 45 of those were INVISIBLE rather than failing: core/integer/{pow,exponent}_spec.rb each scored 0 because the corpus contains `100000000 ** 1000000000` and rbgo computed it unbounded until the harness killed the process. bignum.c's BIGLEN_LIMIT refuses it. core/rational and core/builtin_constants are now AT the ceiling. #668 IO/Dir encodings resolved once at open, #syswrite reaching the descriptor, IO::Buffer's base/size distinction (+56). THIRTEEN of the sixteen files it touched now sit exactly at the ceiling the harness can express. #665 singleton classes of nil/true/false, post parameters in Method#arity AND #parameters together, and `!~`/`!`/`!=` DISPATCHED rather than synthesised — one cause, three defects, found by asking whether #663's neighbours shared its shape. Measured with scripts/conformance/rubyspec/run.sh against the pinned corpus at 87b1631, 16-way, TIMEOUT=25, three consecutive runs: 22489, 22489, 22488. The floor is set thirteen below the lowest. Three of my briefing targets dissolved under measurement again — the third wave running. core/io/write_spec's "9 failures" is a real gap of 1 (MRI fails the other eight through this shim); core/array/fill_spec's "9" is a gap of ZERO (ceiling 38, rbgo 38); core/numeric/step_spec's 12.700000000000001 is what MRI prints. Running MRI through our own shim for a per-file ceiling is what catches these, and it is now done before planning rather than after. The largest lever left is not in any of these lanes: core/file/printf_spec.rb scores 0 against a ceiling of 160 because rbgo cannot parse a do...end block inside an arrow-lambda that is a paren-less command argument. One parser fix, 160 examples — issue #669. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #655.
The defect, and how big it was
ShouldProxyinheritedObject. Every predicateObjectalready answers wastherefore dispatched on the proxy and its result discarded, so the
assertion was never made.
x.should.equal?(y)compared the proxy withy,got false, threw it away, and the example passed whatever the two values were.
Eleven spellings were affected — nine predicates and two operators. Counted in
language/+core/of the pinned corpus (87b16319):should.should_not.equal?Object#equal?on the proxy → always false → discardedinstance_of?Object#instance_of?on the proxyeql?Object#eql?on the proxyis_a?Object#is_a?on the proxyfrozen?Object#frozen?on the proxyrespond_to?Object#respond_to?→ always true (respond_to_missing?returned true)nil?Object#nil?on the proxykind_of?Object#kind_of?on the proxyinstance_variable_defined?Object#instance_variable_defined?on the proxyshould === yObject#===→ the proxy's==, comparing the wrong pair — a false FAILUREshould !~ reObject#!~=!(x =~ y)→ ran the proxy's positive=~→ the exact INVERSE2 806 assertion sites across 811 files asserted nothing (or, for the last
two rows, asserted something else).
The fix: inherit less
Upstream mspec is immune by construction, and this reproduces it. Its operator
matchers subclass
BasicObject(
mspec/lib/mspec/matchers/base.rb—class SpecPositiveOperatorMatcher < BasicObject), which answers only==,!=,!,equal?,__send__,__id__,instance_eval,instance_execandmethod_missing. mspec spellsout the three it must and lets every other predicate fall through
#method_missingonto the real receiver, where its truthiness is asserted(
SpecExpectation.fail_predicate,mspec/lib/mspec/expectations/expectations.rb).So:
class ShouldProxy < BasicObject, withequal?spelled out beside theexisting
==/!=,!~spelled out, the deprecated matcher spellingsequal/eqlkept (mspec/lib/mspec/matchers/equal.rb— the receiver has nosuch method to forward to), and
respond_to_missing?dropped. This also coversspellings the corpus has not used yet: a predicate is broken exactly when the
proxy answers it, so the proxy must answer as little as possible.
Every constant inside the class is now
::-qualified. From inside aBasicObjectsubclass MRI does not reach the top-level constants, so a bareExceptionraisedNameError: uninitialized constant ShouldProxy::Exceptionunder MRI 4.0.5 — while rbgo resolved it happily. Since MRI through this shim
is the judge that separates a real rbgo defect from an inexpressible assertion,
that difference would have broken the judge and not us.
The measurement
One host, serially (an 8-way sweep of this corpus is not reproducible:
core/kernel/require_spec.rbreads 69 alone and 68 under load), three runseach side, same corpus, same rbgo binary built from this branch.
Decomposed per example: 205 examples newly FAIL, 2 newly PASS, and
core/random/bytes_spec.rbdrifts ±1 between any two runs (it andcore/module/autoload_spec.rbare the only files that move between twoidentical runs). 2 − 205 = −203; the −202 median-to-median difference is that
flake.
106 files moved: 103 down, 3 up (one of the three up is the flake).
Classification of the new failures — MRI 4.0.5 as the per-file ceiling
Every one of the 205 newly-failing examples was matched by description against
the same file run under MRI 4.0.5 through the same fixed shim:
A sample, with the ceiling beside it:
core/io/internal_encoding_spec.rbIO#internal_encodingreturnsnilwhere MRI returnsIBM866core/io/external_encoding_spec.rbIO#external_encodingreturns the wrongEncodingobjectcore/rational/exponent_spec.rbRational#eql?is false for(1/1).eql?((1/1))language/predefined_spec.rb$&carries the wrong encoding;$&assignment raisesNameError, notSyntaxErrorlanguage/singleton_class_spec.rbtrue/false/nilcore/builtin_constants/builtin_constants_spec.rbRUBY_VERSION,RUBY_COPYRIGHT… are not frozencore/exception/receiver_spec.rbNameError#receiverreturnsnilcore/numeric/step_spec.rbstepyields12.700000000000001for12.7core/time/subsec_spec.rbTime#subsecRational identity (eql?)core/dir/children_spec.rbDir.childrenignores the requested encodingcore/hash/constructor_spec.rbHash.[]returnsHashwhere a subclass is requiredcore/integer/to_s_spec.rbInteger#to_sreturns UTF-8, not US-ASCIIBoth honest gains are examples the shadowing made fail, not pass:
x.should === yreachedObject#===→ the proxy's==and compared a classwith an instance (
core/exception/case_compare_spec.rb,core/range/case_compare_spec.rb).One divergence this could not fix, stated rather than papered over
x.should !~ /re/is now spelled out on the proxy and is correct under MRI.rbgo never dispatches
!~as a method — it compilesa !~ binto!(a =~ b)(and inlines!the same way), where MRI sends:!~. So underrbgo the definition is dead code and those 11 sites in
core/exception/full_message_spec.rbstill run the positive=~assertion,i.e. the inverse of what the spec wrote. That is an rbgo/MRI divergence of its
own and worth its own issue; the shim cannot reach it from here. It changes no
number in this PR.
FLOOR: 22345 → 22170
The number fell because the measurement got stricter, not because the VM
regressed. No Go code changed on this branch;
CGO_ENABLED=0 GOWORK=off go test ./...is green.The floor is taken from the lane that enforces it, which is also where the
22345 came from:
ruby/spec ratchetCIThe 34-example gap between the two environments is the environment, not the
change: on this host the unchanged shim read 22339, six below the frozen
22345, so part of that number was never reproducible here either. A floor 34
below what the enforcing lane reads would be slack rather than safety, so the
gate moves 22345 → 22170, a drop of 175, all of it measurement.
A floor that no longer reflects a truthful measurement is worse than a lower
one. The 205 examples that now fail were never passing on their merits, and
each one is a real rbgo defect with a named ceiling.
Per-file movement, all 106 files (before → after, median of three runs each)
🤖 Generated with Claude Code