feat(evaluation): parent-home rubric pack — fixtures, journeys, harness protocols (P1–P6) - #20
Open
obvious-autobuild[bot] wants to merge 1 commit into
Open
obvious-autobuild[bot] wants to merge 1 commit into
obvious-autobuild[bot] wants to merge 1 commit into
Conversation
…journeys, harness protocols Independent evaluator pack for the parent-home candidate arena, authored and frozen before reading any candidate output (rubric art_uWsp80Fg v1.0, pre-registered). Fixtures (evaluation/fixtures/home/, data-only): - utterances.json — four captures on the Nov 1 2026 DST fall-back day: voice u1, text-parity twin u2, interrupted-and-resumed u1b, dad's u3 with the three-heads fiction; correction cases (valid patch u4, cross-actor rejection, bogus-target statement); Ruth's caregiver questions with a fail-closed write attempt; authorization probes. - mini-history.json — five-day published seed (Oct 28-31 EDT) with a deliberate Oct 29 gap, a private-audience entry, and a pre-seeded correction chain; Nov 1 left empty for the capture journeys to fill. - journeys.json — five journeys with per-step expected outcomes. Infrastructure (evaluation/src/home/): - timestamps.ts — single source of truth for every pinned epoch, fixed-offset derived (derive-epochs.ts prints the values), with import-time assertions: zone offsets, wall-clock mappings, durations, the U3 'an hour ago' convention chain, narrative ordering, month window, and the merged-corpus cross-anchor. - adapter.ts — six-method HomeCandidateAdapter seam. - match.ts — expectation matcher reusing the merged harness helpers. - run.ts — P1-P6 protocols: raw SHA-256 fidelity, stale-attempt resolution, idempotent publish, no-write Q&A with audience checks, correction lineage, gap/window disclosure, fail-closed authorization. - validate.ts — structural fixture validation. - example/home-example-adapter.ts — worked example, 7/7 protocols. - example/home-broken-adapter.ts — negative control failing only on its encoded violations. - README.md, derive-epochs.ts — evidence boundary, commands, provenance. package.json: test chains validate + example + negative control; adds eval:home, eval:home:broken, validate:home. Verification: tsc --noEmit clean; fixture validation PASS; worked example 7/7; negative control fails as encoded; merged corpus harness unchanged and green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Acceptance criteria
HomeCandidateAdapterseam + a deterministic runner. No candidate code is imported; scoring runsbun src/home/run.ts --adapter=<path>against any conforming implementation.--expect-failure) fails on exactly its encoded violations.derive-epochs.ts), for the Nov 1 2026 America/New_York fall-back day (Nov instants EST, Oct seed days EDT), and re-asserted at import —timestamps.tsaborts every entry point on drift.validate.tsre-asserts fixture↔timestamp equality and fixture structure before any run.src/run.tsharness and its negative control still pass unmodified.tsc --noEmitclean for the pack.What's in the pack
evaluation/fixtures/home/— utterances.json (voice u1, text-parity twin u2, interrupted-and-resumed u1b, dad's u3 with three-heads fiction; correction/rejection/bogus-target cases; Ruth's questions; authorization probes), mini-history.json (5-day published seed, deliberate Oct 29 gap, private-audience entry, pre-seeded correction chain), journeys.json (five journeys).evaluation/src/home/— adapter seam, matcher, P1–P6 runner, structural validator, self-asserting timestamps, provenance derivation script, README.evaluation/src/home/example/— worked example adapter (7/7) and a negative-control adapter failing only on encoded violations.evaluation/package.json—testchains validate + example + negative control; addseval:home,eval:home:broken,validate:home.Verification (tested HEAD: b41171b)
bunx tsc --noEmit— cleanbun src/home/validate.ts— PASS (utterances, mini-history, journeys)bun src/home/run.ts— 7/7 protocols PASSbun src/home/run.ts --adapter=./src/home/example/home-broken-adapter.ts --expect-failure— fails as encoded, exit 0bun src/run.ts+ broken control — green (merged corpus untouched)Non-goals
arena/candidate-*branch: it follows the normal review→merge workflow.