Skip to content

feat(evaluation): parent-home rubric pack — fixtures, journeys, harness protocols (P1–P6) - #20

Open
obvious-autobuild[bot] wants to merge 1 commit into
masterfrom
eval/home-journeys
Open

obvious-autobuild[bot] wants to merge 1 commit into
masterfrom
eval/home-journeys

Conversation

@obvious-autobuild

Copy link
Copy Markdown
Contributor

Acceptance criteria

  1. Candidate-agnostic. The pack is data-only fixtures + a six-method HomeCandidateAdapter seam + a deterministic runner. No candidate code is imported; scoring runs bun src/home/run.ts --adapter=<path> against any conforming implementation.
  2. Pre-registered rubric fidelity. Implements gates G1–G7 of the frozen rubric (art_uWsp80Fg v1.0, authored before any candidate output was read): fail-closed authorization (G1), idempotent publish (G2), raw SHA-256 fidelity (G3), read-only Q&A (G4), fiction-free extraction (G5), append-only corrections (G6), gap/window disclosure (G7).
  3. Deterministic evidence. No network, no live LLM. Worked example passes 7/7 protocols; the negative control (--expect-failure) fails on exactly its encoded violations.
  4. DST-safe pinned epochs. Every fixture epoch is machine-derived via the fixed-offset method (derive-epochs.ts), for the Nov 1 2026 America/New_York fall-back day (Nov instants EST, Oct seed days EDT), and re-asserted at import — timestamps.ts aborts every entry point on drift.
  5. Structural validation. validate.ts re-asserts fixture↔timestamp equality and fixture structure before any run.
  6. Merged corpus untouched and green. The existing src/run.ts harness and its negative control still pass unmodified.
  7. Strict typecheck. tsc --noEmit clean for the pack.

What's in the pack

  • evaluation/fixtures/home/ — utterances.json (voice u1, text-parity twin u2, interrupted-and-resumed u1b, dad's u3 with three-heads fiction; correction/rejection/bogus-target cases; Ruth's questions; authorization probes), mini-history.json (5-day published seed, deliberate Oct 29 gap, private-audience entry, pre-seeded correction chain), journeys.json (five journeys).
  • evaluation/src/home/ — adapter seam, matcher, P1–P6 runner, structural validator, self-asserting timestamps, provenance derivation script, README.
  • evaluation/src/home/example/ — worked example adapter (7/7) and a negative-control adapter failing only on encoded violations.
  • evaluation/package.json — test chains validate + example + negative control; adds eval:home, eval:home:broken, validate:home.

Verification (tested HEAD: b41171b)

  • bunx tsc --noEmit — clean
  • bun src/home/validate.ts — PASS (utterances, mini-history, journeys)
  • bun src/home/run.ts — 7/7 protocols PASS
  • bun src/home/run.ts --adapter=./src/home/example/home-broken-adapter.ts --expect-failure — fails as encoded, exit 0
  • bun src/run.ts + broken control — green (merged corpus untouched)
  • Branch rebased onto origin/master (652ace5); all suites re-run and green on the exact rebased HEAD b41171b.

Non-goals

  • No candidate scoring and no base/graft recommendation here — that begins only after this pack merges and candidate handoff artifacts exist.
  • This is evaluator infrastructure, not an arena/candidate-* branch: it follows the normal review→merge workflow.

…journeys, harness protocols

Independent evaluator pack for the parent-home candidate arena, authored
and frozen before reading any candidate output (rubric art_uWsp80Fg v1.0,
pre-registered).

Fixtures (evaluation/fixtures/home/, data-only):
- utterances.json — four captures on the Nov 1 2026 DST fall-back day:
  voice u1, text-parity twin u2, interrupted-and-resumed u1b, dad's u3
  with the three-heads fiction; correction cases (valid patch u4,
  cross-actor rejection, bogus-target statement); Ruth's caregiver
  questions with a fail-closed write attempt; authorization probes.
- mini-history.json — five-day published seed (Oct 28-31 EDT) with a
  deliberate Oct 29 gap, a private-audience entry, and a pre-seeded
  correction chain; Nov 1 left empty for the capture journeys to fill.
- journeys.json — five journeys with per-step expected outcomes.

Infrastructure (evaluation/src/home/):
- timestamps.ts — single source of truth for every pinned epoch,
  fixed-offset derived (derive-epochs.ts prints the values), with
  import-time assertions: zone offsets, wall-clock mappings, durations,
  the U3 'an hour ago' convention chain, narrative ordering, month
  window, and the merged-corpus cross-anchor.
- adapter.ts — six-method HomeCandidateAdapter seam.
- match.ts — expectation matcher reusing the merged harness helpers.
- run.ts — P1-P6 protocols: raw SHA-256 fidelity, stale-attempt
  resolution, idempotent publish, no-write Q&A with audience checks,
  correction lineage, gap/window disclosure, fail-closed authorization.
- validate.ts — structural fixture validation.
- example/home-example-adapter.ts — worked example, 7/7 protocols.
- example/home-broken-adapter.ts — negative control failing only on
  its encoded violations.
- README.md, derive-epochs.ts — evidence boundary, commands, provenance.

package.json: test chains validate + example + negative control; adds
eval:home, eval:home:broken, validate:home.

Verification: tsc --noEmit clean; fixture validation PASS; worked
example 7/7; negative control fails as encoded; merged corpus harness
unchanged and green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants