Skip to content

Spec test gate misses module-level skips and missing test_spec modules #203

Closed

Description

@guijacquemet

Release blocker found by the external review of #183 (ChatGPT batch 25, C2).

tests/spec_gate.py:18-40 only marks test items that have already been collected (pytest_collection_modifyitems), and only turns their skipped pytest_runtest_makereport results into failures. Two cases get through:

  • Module-level skips: pytest.skip(..., allow_module_level=True) or pytest.importorskip(...) in a test_spec_*.py module runs during collection, before any item exists. The whole module is reported as skipped and the run still succeeds.
  • Deleted modules: a deleted test_spec_*.py module is not detected.

The self-test (tests/test_spec_gate.py:12-57) covers only skips that happen after a test item exists.

Fix: add a collection-report hook (pytest_collectreport) that fails on a skipped test_spec_* module. Add a session-level assertion that the expected Spec modules and items were actually collected. Add regression tests for a module-level skip, a module-level importorskip, and a missing module.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions