feat(evaluation): namespace-aware, manifest-driven fixture-area registration

[obvious-autobuild]

Acceptance criteria

1. Manifest file listing fixture areas with {path, fixtureClass, runnerBinding} — evaluation/fixtures/manifest.json (schema v1).
2. Seeded with the existing extraction-accuracy corpus; harness output unchanged — bun src/run.ts is byte-identical to the pre-change run and the --expect-failure negative control still fails the broken adapter (diffs against pre-change baselines at the tested SHA, see Test Evidence).
3. scenario-and-authorization fixture class defined (bun-test-loadable data; validator/runner semantics distinct from extraction accuracy) without adding the catch-up manifest entry — the catch-up PR registers its own area.
4. Docs in evaluation/README.md on how a new area registers.
5. No changes to the six existing fixtures — verified by diff (fixtures/*.json 01–06 absent from the diff).

Why

The merged candidate-agnostic harness (PR #6) discovers fixtures by scanning flat evaluation/fixtures/*.json. Fixture areas in sub-namespaces — agent-experience/catchup/ (since-last-seen catch-up, 14 scenario-and-authorization fixtures per art_6qhBut41) — stay invisible to every run, so those lanes cannot land data that the harness even acknowledges. Registration needs to become explicit and namespace-aware before the catch-up PR can land its fixtures.

What

• fixtures/manifest.json — the registration surface. Each area: path (relative to the manifest's directory; . = the root corpus), fixtureClass (closed set), runnerBinding (one legal binding per class). Seeded with exactly one entry: the existing flat corpus as extraction-accuracy / adapter-corpus.
• src/manifest.ts — FIXTURE_CLASSES registry stating each class's validator + runner semantics:
  • extraction-accuracy → adapter-corpus: executed by the corpus runner against a CandidateAdapter; results count toward pass/fail; every file must parse and carry a known kind + non-empty id (enforced at registration via the now-exported assertFixture).
  • scenario-and-authorization → bun-test-data: pure data (e.g. history/grants/cases) consumed directly by bun test suites (packages/domain/test/catchup.test.ts); the harness validates JSON loadability only, never executes it, and it never affects the exit code.
  • Fail-fast discovery: missing/empty area, unknown class, illegal binding, path escape (.., absolute), duplicate path, or a manifest with no extraction-accuracy area all fail the run before any fixture executes. Nothing is silently skipped — matching the existing loader's contract.
• src/run.ts — manifest-driven: partitions registered areas by class, loads extraction areas through the unchanged loadFixtures (now with excludeAbsolutePaths so the manifest never loads as a fixture), and lists scenario areas at the end of the output (only when any are registered — so today's output is byte-identical).
• src/manifest-selfcheck.ts — registration self-check: the seed manifest resolves to the six-fixture corpus with the manifest excluded, and every fail-fast rejection rule is exercised (classes, bindings, both cross-pairings, path escapes, duplicates, bad version, missing/empty areas, kind discipline).
• evaluation/README.md — "Fixture areas & the manifest" section: schema, class table, invariants, and a three-step how-to for registering a new area (data-only; only a new class touches src/).

Deliberately excluded: no agent-experience/catchup manifest entry — that registration belongs to the catch-up PR (which rebases on this one; ordering constraint honored: this merges first). No fixture data changes.

How to Review

Key files: evaluation/src/manifest.ts (the whole contract), evaluation/src/run.ts (dispatch + output), evaluation/fixtures/manifest.json (the seed). Backward-compat surface: loadFixtures gained an optional options param; assertFixture went from module-private to exported; no other src/ behavior changed.

Test Evidence — local verification at the exact tested SHA

This PR's verification surface is local, not CI: CI's pnpm turbo run test does not cover evaluation/ (it is not a pnpm workspace member). Local bun runs at the exact head 1f1287bc9ea8af072734b956b3c96af0f95734e6 are the gate for this area (per .obvious/obvious.md). Observed at that SHA:

Command	Result
bun src/run.ts	exit 0 — byte-identical (diff) to the pre-change baseline run: 6/6 fixtures PASS
bun src/run.ts --adapter=./src/example/broken-adapter.ts --expect-failure	exit 0 — byte-identical (diff) to the pre-change negative control: faults caught (5/6 FAIL)
bun src/manifest-selfcheck.ts	exit 0 — seed manifest resolves (6 corpus files, manifest excluded); all rejection rules proven
bun run typecheck (tsc strict, noUncheckedIndexedAccess, verbatimModuleSyntax)	exit 0
Simulated catch-up-style registration (temp scenario area + entry, reverted)	6/6 corpus PASS unchanged + block: agent-experience/catchup (3 file(s)) listed, not executed; negative control still exit 0

CI on this PR covers the workspace packages (untouched by this diff — git diff names only evaluation/ files); green CI there plus the local suite above is the complete verification story.

Merge plan

Serialized merge owner: this thread. Merges before the catch-up implementation PR; the catch-up lane then adds its own manifest entry. Evidence receipt to follow per repo contract.

Human author: Gilbert Polanco (gilbertpolanco42@gmail.com)

🔗 Obvious Project · 🧵 Obvious Thread

[obvious-autobuild]

Evidence receipt (merged PR)

Field	Value
PR	#22
Tested head SHA	1f1287bc9ea8af072734b956b3c96af0f95734e6 — all local verification + CI ran against this exact head
Review result	pass (own-lane review per repo policy — no human approvals required; diff inspected against all five acceptance criteria) — harness-owner lane, todo_l5QeXI2q, 2026-09-17
Checks	All green: CI (GitHub Actions) success on 1f1287b, verified via live snapshot — 1/1 passed, 0 pending
Merge commit	e74c7be88c64097e76b98612a607aa37a38a641c (squash on master)
Post-merge smoke	On e74c7be: pnpm typecheck 4/4 exit 0 · pnpm test 4/4 exit 0 · pnpm build 4/4 exit 0 · bun test ./security 17 pass / 0 fail · evaluation: bun src/run.ts 6/6 PASS exit 0, broken-adapter --expect-failure negative control exit 0 (faults caught), bun src/manifest-selfcheck.ts all registration checks pass
Unlocked tasks	Catch-up lane (todo_RwfcpCGe / th_J9U4j8d5) may now register its agent-experience/catchup manifest entry — rebase on e74c7be or later; consolidation item §5 (art_gfwpdmWF) fulfilled for this slice

Verification caveat (stated plainly, per PR body): CI's pnpm turbo run test does not cover evaluation/ (standalone bun project). The gate for this area is local bun verification at the exact head — all runs listed above were executed at the stated SHAs, with bun src/run.ts and the negative control byte-identical to pre-change baselines (diff-verified) before merge.

Human author: Gilbert Polanco (gilbertpolanco42@gmail.com)

[obvious-autobuild]

Post-merge verification — independent reviewer verdict

Verdict: pass (two non-blocking notes). Completes the evidence receipt above — the "Review result" reviewer field is this thread's independent verification, run after merge.

Merge integrity — squash e74c7be, parent 0c65862:

• Files introduced by the merge commit are exactly the PR's 8-file diff, all under evaluation/ — scope confined to the harness + manifest loader, no domain/backend changes.
• git diff 1f1287b e74c7be -- evaluation/ is empty: the merged evaluation/ content is byte-identical to the tested head.

Verified at merge commit e74c7be (fresh clean worktree):

• Local gates uncached: pnpm turbo run typecheck --force 8/8, test --force 4/4, build --force 4/4 — 0 cached.
• bun test ./security — 17 pass, 0 fail.
• Harness: 6/6 fixtures passed; captured output byte-identical to the pre-PR harness run at the merge parent 0c65862 (diff of outputs: empty).
• Negative control (--adapter=./src/example/broken-adapter.ts --expect-failure): broken adapter caught loudly — 1/6 with explicit per-fixture failure lines; exit 0 = the harness correctly detected the failure.
• Manifest self-check (bun src/manifest-selfcheck.ts): all checks pass — seed manifest resolves 1 extraction-accuracy area (6 corpus files, manifest excluded), and the fail-fast rejection matrix holds (unknown class, illegal bindings both directions, path escape/absolute/trailing slash, duplicate path, bad version, no-corpus-area, missing directory, unknown kind at registration, scenario JSON-loadability).
• CI: run 35260504732 — success on the exact tested head 1f1287b ("Typecheck, test, build").

Ordering contract — the manifest registers exactly one area (. → extraction-accuracy / adapter-corpus). No catchup area is pre-registered; catchup/agent-experience appears only in README/comment prose as examples of how a future area registers. No coupling in either direction — the catch-up PR adds its own registration and lands after this merge, as required.

Namespace discipline — the only evaluation/fixtures/ change is the added manifest.json; the six corpus fixtures are sha256-identical to the pre-PR parent; no family-history paths exist in or were re-homed by the merged tree.

Notes (non-blocking)

1. loadManifest requires ≥1 extraction-accuracy area per manifest (fail-fast: "the corpus run would be empty"). Deliberate and documented, and no coupling to the catch-up PR — but a scenario-only manifest is illegal by design.
2. The whole-tree diff 1f1287b..e74c7be is non-empty (31 files — PR feat(domain): family knowledge records — schema, invariants, Convex adapter, synthetic fixtures #16/feat(handoff): since-last-seen digest + grounded follow-up prototypes #17 content that merged onto master between this PR's base and its merge). Stated explicitly because "merge tree matches tested head" holds for the PR's own file set (the property that matters), not for the whole tree.