Repository navigation
Hold module-level skips and missing modules in the Spec test gate - #205
Merged
Merged
Conversation
A test_spec_*.py module skipped while it is collected (module-level skip or importorskip) is now a failed collection, and a run over the whole tests folder fails when no test was collected from a module of EXPECTED_MODULES (a deleted or emptied Spec module). Fixes #203 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm # Conflicts: # CHANGELOG.md
#196 added the module on main; the gate refuses an unlisted Spec module, as designed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #203
What and why
tests/spec_gate.pyonly turned skips of already-collected items into failures. Now:pytest_collectreporthook (tryfirst, so pytest's own bookkeeping sees the change) turns the skipped collect report of atest_spec_*.pymodule (pytest.skip(..., allow_module_level=True)or a module-levelimportorskip) into a failed collection.pytest_itemcollected, before any-k), andpytest_collection_finishrefuses (usage error, non-zero exit) a run over the whole tests folder that collected no item from a module ofEXPECTED_MODULES(the 14 current modules). Runs of chosen files, test ids or another folder check only what they select, so they do not fail for modules they did not ask for. The list asks for at least one item per module, not exact counts, because parametrized counts follow the vendored fixtures;test_every_spec_module_of_the_repository_is_listedkeeps the list equal to the files on disk.spec_gate_plugin.pyreadsSPEC_GATE_EXPECTEDfor the gate's own tests;docs/development.mdand CHANGELOG updated.Regression tests (
tests/test_spec_gate.py)Old gate code (tests run with the old
spec_gate.py,conftest.pyand plugin): 5 FAILED, 4 passed.test_a_module_level_skip_in_a_spec_module_fails_the_run: fails on old, passes now.test_a_module_level_importorskip_in_a_spec_module_fails_the_run: fails on old, passes now.test_a_missing_spec_module_fails_a_whole_folder_run: fails on old, passes now.test_a_spec_module_without_a_collected_test_fails_a_whole_folder_runandtest_every_spec_module_of_the_repository_is_listed: fail on old, pass now.Also checked by hand on the real tree: moving
tests/test_spec_rules_table.pyaway makespytest testsstop with "no test was collected from the Spec conformance module(s)"; restored, 4049 tests collect.Checks
python tools/check.py(ruff check, ruff format --check, pytest, code rules): 4045 passed, 4 skipped, 0 failed, 2 warnings. The 4 skips are existing platform skips (case-insensitive file system, file-mode permissions as root), not Spec tests. CHANGELOG entry added; expect conflicts with other open PRs, keep both entries.🤖 Generated with Claude Code
https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm