Pin CLAUDE.md's pytest module count to len(testpaths), red-first #171

This is a decision to record, not a defect to fix. Nothing is broken today.
An offer was made in a PR body, nobody answered it, and it will be lost when
that PR merges. This issue exists so the answer is "no, and here is why" or
"yes, do it" rather than silence.

The offer

While auditing documentation drift, PR #168 was asked to add a pin to
tools/test_pytest_boundary.py asserting that the module count stated in
CLAUDE.md equals len(testpaths). The author declined, and closed the
argument with:

Consequently there is no red-first test to show. If you would rather have the
count-agreement pin anyway, say so and I will write it red first — I just do
not think it earns its brittleness.

— PR #168, "The pin: I did not add one, and here is why".

No reply was posted. The offer is still open and is currently buried in a PR
description.

What the pin would be

tools/test_pytest_boundary.py (222 lines, 5 assertions) already proves — by
AST, in both directions — that pytest.ini's testpaths is exactly the set of
tools/test_*.py modules pytest can run cleanly:

• test_every_testpath_exists
• test_testpaths_lists_every_runnable_tools_module
• test_testpaths_lists_nothing_pytest_cannot_run
• test_rig_dirs_hold_no_pytest_named_files
• test_norecursedirs_covers_every_rig_dir

The proposal adds a sixth: locate the count stated in CLAUDE.md's prose and
assert it equals len(testpaths) read from pytest.ini.

The case FOR

Issue #161's comment thread sets the repo's standing rule for exactly this
class. Its framing: two copies of a fact, only one of them real — an assertion
that "passes because something coincides with the truth, and nothing
distinguishes coincidence from cause." The prescription:

check whether the second copy would actually change if the first one did

and, where it would not:

Where a fact can be mechanically pinned, pin it instead of restating it.

The thread cites two in-tree instances as the pattern to follow — the proposed
.assert LIB_NISTCURVES_REU_BANK_COMB = $02 (#120) and build/flags.stamp
(#159) — both of which "replace a sentence someone must remember with a build
error."

A prose module count fails the triage question outright: it does not change when
testpaths changes. And the rot is not hypothetical. At master right now:

• pytest.ini lists 6 entries in testpaths.
• CLAUDE.md:606 says "pins testpaths to the three pure-logic modules" and
  CLAUDE.md:608 says "Bare pytest at root: 31 passed."
• README.md:442-445 repeats "the three modules" and "31 passed", and
  README.md:452 adds "31 passed, 74 fixture errors".

Three stale statements of one fact, in two files. That is the defect the pin
would have caught.

The rot is also fast. #168's own body records the count as "five modules and
41 tests" at the time it was written, and notes six/47 was then the state of
branch fix/unresolvable-test-hostname. That branch merged (#164, 0b55c30),
so the correct figure moved from five to six during the PR's own review.
Any number written into prose here has a shelf life measured in days.

The case AGAINST

Taken from #168's argument, which is a real one and should not be waved off.

1. Pinning a number inside prose has no non-brittle failure mode. The pin
must locate the count with a pattern. On a reword such as "pins testpaths to
the pure-logic modules" there are exactly two behaviours available:

• fail when the pattern is absent — then every terseness edit to CLAUDE.md
  reds the suite with no defect behind it, in a repo that rewrites that file
  constantly (the CLAUDE.md / engineering-notes split was days ago);
• pass when the pattern is absent — a test that no-ops on a reword. That is the
  vacuous-green shape test: six suites could exit 0 having tested nothing #158 spent a PR hunting, and the same shape Three VICE suites pass or fail for reasons unrelated to what they measure (test_finished_verify's 16 negatives are vacuous on UCI) #161's own
  thread leads with.

2. #168 arguably discharged the risk already, and did it better. Rather than
pin the duplicate, that PR deletes it. On branch docs/drift-audit-161,
CLAUDE.md:634-641 now reads:

pytest.ini pins testpaths to the pure-logic modules — that list is the
enumeration; read it there, not here — and tools/test_pytest_boundary.py
proves it is exactly the set pytest can run, in both directions. […] A bare
pytest at the root is green; the total is not quotable, because it tracks
testpaths and the build state (test_uci_data_acc.py skips its cases
without a UCI PRG in build/). Run it rather than citing a number.

README.md gets the same treatment. #161's rule is "pin rather than restate";
not restating at all is the stronger form of not restating, and it leaves
nothing for a pin to pin. If #168 merges as written, this issue is moot until
someone re-adds a count — which is itself the thing to decide: whether to
guard against that reintroduction, or accept that a reviewer catches it.

3. The pass-count half cannot be pinned at all. 41 is build-state
dependent: tools/test_uci_data_acc.py skips its cases when build/c64-https.prg
or build/labels.txt is absent or is not a BACKEND=uci image
(tools/test_uci_data_acc.py:583-584, :591, :714-720). No single number is
correct even at a fixed commit. Only the module count is pinnable; the more
frequently-quoted number is not.

4. The negative form does not work either. #168 considered forbidding any
doc from stating a bare pytest pass count, and rejected it: docs/engineering-notes.md
is the measurement record, and its dated 25 passed, 75 errors line is a
legitimate historical fact. No regex separates "dated historical measurement"
from "present-tense claim".

What is actually being decided

1. Does the repo want a mechanical guard against a count being reintroduced
   into CLAUDE.md / README.md prose, or is deletion plus review sufficient?
2. If a guard is wanted, is a count-agreement pin the right shape, or is a
   narrower "no bare pytest pass count in present tense" rule (with its
   acknowledged regex problem) closer to the real invariant?
3. Whichever way, docs/engineering-notes.md's dated historical counts must
   survive it.

If accepted, the terms

The author offered to write it red-first, and that offer should be taken
literally:

• Red first. Land the assertion against a CLAUDE.md that still carries the
  stale "three modules", watch it fail, then fix the prose and watch it pass.
  A pin added green proves nothing about whether it can fire.
• Fail loudly when the prose pattern is absent. Missing-pattern must be a
  hard failure with a message naming the file and the expected shape — never a
  silent skip or a match against the empty set. Three VICE suites pass or fail for reasons unrelated to what they measure (test_finished_verify's 16 negatives are vacuous on UCI) #161's second qualification is
  explicit that a pin which passes by coincidence "recreates the same problem
  one level down", and a regex matching nothing is the canonical instance.
• Mutation-test it, and do not over-read the result. Per Three VICE suites pass or fail for reasons unrelated to what they measure (test_finished_verify's 16 negatives are vacuous on UCI) #161: mutation
  testing "proves an assertion can fail. It does not prove it fails for the
  right reason." At minimum, mutate the count in the prose and delete the
  sentence entirely, and check both go red.
• Pin the module count only. Not the pass total (point 3 above).

References

• PR docs: correct seven documentation-drift defects, and stop restating rot-prone counts #168 — docs/drift-audit-161, where the offer was made and declined
• Issue Three VICE suites pass or fail for reasons unrelated to what they measure (test_finished_verify's 16 negatives are vacuous on UCI) #161 — the "pin rather than restate" rule and its two qualifications
• Issue Load the Lim-Lee comb table from disk instead of computing it at boot #120, PR build: make the flag set a tracked dependency (build/flags.stamp) #159 — the two in-tree pins Three VICE suites pass or fail for reasons unrelated to what they measure (test_finished_verify's 16 negatives are vacuous on UCI) #161 cites as the pattern
• pytest.ini, tools/test_pytest_boundary.py — the mechanism a pin would
  extend