Agent-friendly testing, phase 1: rules, guards, one command, goldens, Swift 6 census

[r3dbars]

Why

Last week (Sep 24–28, 200 Swift CI runs) 10 runs went red and none of them was a bug a user would have hit: 6 were a release check tripping on half-finished release PRs, 2 were a wall-clock test on a busy runner, 1 was a hand-kept source list missing a new file, and 1 was a runner glitch. Meanwhile about 1 in 5 fast-test files reads app code as text, and agents spent roughly one test edit for every two app edits keeping tests in sync.

This PR is phase 1 of making the suite agent-friendly: tests check promises through inputs and outputs, and the kinds of checks that go red for no reason stop growing. Plan and before/after: the "Transcripted Testing Plan" doc.

What's in it

Rules (Tests/README.md "Test rules", mirrored in AGENTS.md, AGENT_START.md, CLAUDE.md, the agent contract's tests area, the /tests command, and a new test-writer agent that writes tests from the promise, not the code).

Guards

• scripts/dev/check-test-shape.py: blocks new tests that read Sources/ as text or assert on wall-clock time. Existing ones are grandfathered in .agents/test-shape-baseline.json, which can only shrink. Runs in linux-checks.sh (so repo-hygiene CI) and the test matrix. It caught Skip the engine warmup when the pinned recorder will record #1874 adding two new source-text reads an hour after it was written (grandfathered here since that's on main).
• Tests/quarantine.txt: bench a flaky fast-test suite the same day; run-tests.sh skips and lists it; the guard validates the file.
• scripts/dev/explain-missing-sources.py: "cannot find 'X' in scope" in run-tests.sh or the Parakeet lifecycle smoke now prints the file that declares X and the list to add it to. It replays the Sep 27 integration-smoke failure correctly.
• scripts/dev/check-known-traps.py: every Tools package runs in CI and has a matrix rule; every root command is documented.
• run-tests.sh defaults TZ to America/Chicago when unset.

One command: bash check.sh runs the mapped checks for your diff (via agent-check.py); quick, full and hardware tiers; plain PASS/FAIL with the rerun command.

Output and real-world layers

• MeetingMarkdownGoldenTests + Tests/Fixtures/golden-meetings/: whole-file approved copies of saved meeting Markdown for three canned meetings. Time-zone proof (passes in Chicago, UTC, Kolkata). Regenerate with TRANSCRIPTED_UPDATE_GOLDENS=1.
• scripts/ops/nightly-hardware-smokes.sh: runs check.sh hardware plus the synthetic audio pass with dated logs and a notification on failure. Not installed: it needs a first manual run for permissions and plays a test tone.
• scripts/dev/concurrency-census.sh: typecheck-only pass with -strict-concurrency=complete (no binary). 121 concurrency warnings across 8 Sources/ folders; baseline in .agents/concurrency-baseline.json can only shrink. Mapped to Sources/**/*.swift changes and check.sh full, not added to hosted CI (about 90 s).

Tests converted

• 5 wall-clock assertions became outcome checks (detached timeout, model-load waiter, default-input submit, llama child shutdown, recovery-owner deadline counts joins instead of timing).
• 8 source-text files converted fully or partly (12 source reads out, 19 behavior assertions in); docs/testing-source-text-inventory.md lists all 51 files with the promise each guards and the seam each needs.

Mutation probe (scripts/dev/mutation-probe.py, docs/mutation-testing.md): breaks one operator at a time in a Swift file, runs the tests, and lists the breaks nobody catches. It always restores the file (refuses to start on a dirty file, restores on Ctrl-C) and keeps one probe per checkout. First real runs, 20 mutants each:

• DictationReadinessWaitPolicy.swift: 17 caught, 3 missed (85%).
• DictationInputDeviceSelectionPolicy.swift: 11 caught, 9 missed (55%). 8 of the 9 also survive the full fast suite. The worst: flipping id != 0 in requireSelection makes every real mic selection throw and all fast tests stay green. Also uncaught: a Bluetooth headset with a plain product name, and a user-chosen mic with the lid closed. The new test-writer agent then wrote tests for all 5 gaps from the promises alone (not the code), and showed each new test goes red under the exact mutation that had slipped through: a real mic passes the selection check, a Bluetooth transport counts even with a plain product name, a user-chosen external mic wins with the lid closed, hard recoveries stop exactly at the budget, and the refresh gives up at exactly its timeout.

First seam and Swift 6 fixes (the only production changes)

• SentryEventPolicy.allowedDiagnosticTagKeys is internal instead of private, so its test iterates the real set instead of parsing the file.
• The census flagged a new warning from Today: day sentence, writing lane, and a preview of each capture #1873 (Today). Fixed all three Today warnings: two Int limits on TodayViewModel are nonisolated static let, and the preview's arrowButton helper is @MainActor. 121 to 119.

Not in this PR (phase 2 and decisions)

• 38 source-text files need a production seam first (top three: the dictation stop/start pipeline, a protocol around AVAudioEngine calls in ParakeetEngine, moving FailedMeetingItem out of MeetingSessionController.swift).
• Fakes for clock, mic, clipboard and STT; state enums for flag-heavy types; Swift 6 per-folder flips.
• Decisions: install the nightly runner and at what time; adopt swift-format for new files (it would restyle about 5% of existing lines, mostly guard ... else wrapping); make release-watch.py "worse" block the next release; keep SpeakerEvalHarness out of CI.

Verification

bash check.sh on the branch after merging main at #1788 (40 mapped checks): deterministic proof PASS, every check green.

• build.sh --no-open, run-tests.sh (19,650 passed), build-deps.sh --force, run-integration-smoke.sh, swift test (588, 1 skipped), Tools package tests, linux-checks.sh (55/55), check-source-pins.py --changed-only, check-test-shape.py, concurrency-census.sh --check (119/119), check-known-traps.py, every new self-test, telemetry-key and analytics-emitter checks.
• Golden tests pass under TZ=UTC, America/Chicago and Asia/Kolkata; a one-word drift fails with the exact line.
• Quarantine demo: a benched suite is skipped and listed. Missing-source demo: dropping a file from APP_SOURCES makes the runner name it.
• Every merge of main so far needed the new guards' baselines re-grandfathered: Skip the engine warmup when the pinned recorder will record #1874, the Writing tab and Make system-audio meeting warnings actionable #1761 added 18 source-text reads between them before the guard existed. After this merges, new ones fail in CI instead.
• Not run: hardware smokes (need mic and System Audio permission), the nightly installer.

🤖 Generated with Claude Code