Lay out the test layers: what runs at commit, at merge, and at self-test #749

Problem

Nothing in the repository says which test belongs to which gate. So every suite runs everywhere, or a suite quietly runs nowhere, and nobody can answer "is this covered before it merges?" from a document — only by reading the YAML.

Where things run today (measured 2026-10-22):

The gaps this leaves: a test that needs a real model or network can only be written as a SelfTest case, but nothing marks the others as safe-and-cheap, so the core job keeps growing; and a SelfTest case that finds something has no path back to an automated test, so the same defect is rediscovered by hand.

What to build

1. docs/testing-layers.md — the three layers, each with: what it must be able to catch, what it must not depend on, a time budget, and the exact command that runs it.

   • L1 commit / push — deterministic, hermetic, no network, no model, no clock. Runs in test.yml. Budget: the whole layer under 10 minutes on a cold runner.
   • L2 merge / queue — everything a merge must not break: the L1 suites plus the packaging, patch-pin, type, changelog and workflow-lint checks, plus the verdict/hold judgement at the entry SHA. This is what main requires.
   • L3 self-test — the product as a user meets it: two real homes, real Bot Social, a real model, a person watching. Not deterministic, so it produces issues rather than failing a merge.

2. A single machine-readable manifest (docs/testing-layers.json or an exported table) mapping every suite to its layer: core, each plugins/*, .github/scripts/*.test.mjs, test:interop, docker, compat matrix, live e2e. The doc's table is generated from it, so the two cannot drift.

3. .github/scripts/testing-layers.test.mjs — every test file in the repository either appears in the manifest or is explicitly listed as none with a reason; a new *.test.mjs / *.test.js that nobody classified fails the check.

4. The rule that keeps it honest: a case may only enter a layer that can run it. Anything needing a real model, the network or wall-clock time becomes an L3 case and is named in docs/self-test.md (see Self-test case catalog with stable ids — to land as docs/self-test.md (current cases, gaps, planned) #742) with a stable id; when an L3 case turns out to be reproducible offline, it moves down to L1 and the SelfTest entry keeps a pointer to it.

Acceptance

• docs/testing-layers.md answers, for any file in the repository, which layer runs it and what that layer guarantees.
• adding an unclassified test file fails package with a message naming the file.
• the manifest has two kinds of row and both are enforced: a test-file suite (core, each plugins/*, .github/scripts/*.test.mjs, test:interop) and a workflow-only layer that carries no test file of its own (docker, the nightly compatibility matrix, live-e2e, label-hygiene). Every file in .github/workflows/ is either named in the manifest or explicitly listed as not-a-layer with a reason, so a new workflow cannot slip in unclassified.
• red evidence, two of them, both required:
  1. removing plugins/timeline from the manifest fails the check;
  2. removing the docker row fails the check and names docker.yml — a row for a workflow-only layer must be checked against the workflow file, not against a test path that does not exist.
• no new required status check is added to the ruleset without saying so here; L2 stays exactly the checks main already requires.

Boundary

docs/self-test.md belongs to #742, which carries the self-test case catalog. This issue references it and must not create or rewrite it: the L3 section of docs/testing-layers.md says where the cases live and how they map to a layer, and #742 owns their content and ids.

label-hygiene.yml (#738, merged) is an existing workflow and therefore appears in the manifest as a workflow-only layer; its absence is exactly the kind of omission the check must catch.

Out of scope

Changing the ruleset, moving the existing suites' contents, writing new test cases, and any change to the release workflow.