test: gate the canonical/legacy test split - #239
Merged
Conversation
`rake test:canonical` is defined as "exactly what must still pass once test/legacy/ and the v1 trees are gone". Three times in one day a test asserting LEGACY behaviour was written into test/unit/, i.e. into that suite: a canonical surface table demanding the shim-only SnapDiff.start (#236), three legacy-constant probes in a canonical file (#237), and a pre-existing umbrella guard #236 had to relocate. Each would have failed the day the deletion landed, long after its author moved on. Reviews caught all three; the fourth would ship. The test-tree twin of core_tree_has_no_legacy_deps_test.rb: no file under test/unit/ or test/integration/ may require a doomed path, name a v1 namespace constant, or use a shim-only name (SnapDiff.start, .silence_deprecations, SnapDiff::Deprecation, suppress_migration_notice!). test/legacy/ is deliberately not policed -- exercising the legacy surface is its job. Same conventions as the twin: file:line: reason -- `code`, whole-line comments ignored, a vacuity guard, and a sub-test that fails on stale allowlist entries. The allowlist holds two entries, both gates rather than tests of behaviour (deletion_3_0_test.rb, which names the deletion set by construction, and the twin gate's own pattern literal). A third entry means canonical tests are still entangled and needs a decision, not a green build. This file cannot scan itself: a line-level allowlist has to quote the lines it blesses, and every quote is itself an offence -- no fixed point exists.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
Warning Review limit reached
Next review available in: 31 minutes Limit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Reviewer's GuideAdds a test-based gate that scans canonical test suites to ensure they do not reference legacy namespaces, shim-only APIs, or doomed require paths, with a precise allowlist and self-consistency checks to prevent stale entries and self-scanning issues. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Screenshot diffs detected
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rake test:canonicalis defined as "exactly what must still pass oncetest/legacy/and the v1 trees are gone". Three times in one day a testasserting legacy behaviour was written into
test/unit/, i.e. into thatsuite:
CANONICAL_SURFACEdemandedSnapDiff.start— a method defined inlegacy_shims.rb, deleted with it.Each was green on the day it was written and red the day the deletion lands,
long after its author moved on. Reviews caught all three. The fourth ships.
The gate
test/unit/canonical_suite_has_no_legacy_refs_test.rb— the test-tree twin oftest/unit/core_tree_has_no_legacy_deps_test.rb, which guards the same edgefor
lib/. No file undertest/unit/ortest/integration/may:LEGACY_REQUIRErequire/require_relativeofcapybara/screenshot…,capybara_screenshot_diff…, thecapybara-screenshot-diffgem-name entry, orsnap_diff/{legacy_shims,deprecation}. Plainrequire "capybara"must not match. Unanchored, unlike the lib-side twin: tests drive subprocesses, so a legacy require is as likely to sit inside a heredoc or a-estring, and both really load it.LEGACY_CONSTANTCapybara::Screenshot/CapybaraScreenshotDiff.LEGACY_SHIM_SURFACESnapDiff.start,SnapDiff.silence_deprecations,SnapDiff::Deprecation,suppress_migration_notice!. This is incident (a)'s shape.test/legacy/is deliberately not policed — exercising the legacy surfaceis its whole job, and it is deleted with what it tests.
Conventions match the twin: reports
file:line: reason -- \code``, ignoreswhole-line comments (a comment about where a forwarder used to live is history,
not a dependency), carries a vacuity guard, and has a sub-test that fails on
stale allowlist entries.
The allowlist
Two entries, and neither is a test of legacy behaviour:
unit/deletion_3_0_test.rb(5 lines) — it simulates the deletion for real,so it names the deletion set, the edits, and its own gate line by
construction. It cannot assert that its gate rejects an intact tree without
spelling the doomed names.
unit/core_tree_has_no_legacy_deps_test.rb(1 line) — the twin gate'sLEGACY_CONSTANTliteral. A detector has to spell what it detects.A third entry means canonical tests are still entangled with the legacy
surface: that needs a decision, not a green build.
Self-exclusion. The gate does not scan its own file, and says why in a
comment: a line-level allowlist has to quote the exact lines it blesses, every
such quote is itself a legacy reference, and blessing them would need an
allowlist entry whose text is again an offence. No fixed point exists.
Mutation evidence
test/unit/vcs_test.rb(a require, a constant,SnapDiff.start)unit/vcs_test.rb:4: requires a doomed path,:5: references a v1 namespace constant,:6: uses a shim-only name that the deletion removesMUTATION_LEGACY_DIR = CapybaraScreenshotDiffadded totest/legacy/errors_alias_test.rbdeletion_3_0_test.rballowlistallowlisted line no longer presentAll probes reverted;
git diffagainst the probe baseline is empty.Suites
rake test:unit— 572 runs, 0 failures (baseline 570; +2 is this gate)rake test:canonical— 469 runs, 0 failures, 1 skip (baseline 467/0/1)standardrbcleanKnown gap, stated plainly
Incident (a) is caught only when the shim-only name is written as
SnapDiff.start. In #236 it was a barestartinside a%w[]surface table —textually indistinguishable from any other word, so no scan can catch that
without unbearable false positives. The other two incident shapes are caught
outright.
Out of scope by design:
test/test_helper.rbandtest/support/, which areshared with
test/legacy/.test_helper.rbstill callsSnapDiff::Deprecation.suppress_migration_notice!— that is deletion fallout,handled where the deletion happens, not by this gate.
Summary by Sourcery
Prevent canonical tests from becoming coupled to the legacy surface ahead of its removal.
New Features:
Enhancements:
Tests: