Skip to content

feat(evaluation): conversational correction challenge corpus (CC-01..CC-15) - #19

Merged
obvious-autobuild[bot] merged 1 commit into
masterfrom
eval/corrections-corpus
Sep 17, 2026
Merged

obvious-autobuild[bot] merged 1 commit into
masterfrom
eval/corrections-corpus

Conversation

@obvious-autobuild

Copy link
Copy Markdown
Contributor

Acceptance criteria

  1. 15 independent conversational-correction cases (CC-01..CC-15) as data-only fixtures — six case-group files plus a shared synthetic household env (3 children, 4 members, explicit read/read-write grants, America/New_York timezone). Families: time corrections, wrong child, wrong prior event, multiple authors / stale suppression, negation (including zero-care capture), repeated corrections / undo.
  2. Candidate-agnostic adapter contract with an environment-dependent factory (createCorrectionAdapter(env)); a deterministic runner drives every candidate identically and resolves each turn to applied / rejected / clarification.
  3. Reference adapter passes 15/15 (mechanical rule interpreter — regex dispatch, lexical child/entry resolution, snapshot-stack undo, timezone-safe retargeting via Intl; no I/O, no LLM, no clock).
  4. Negative control fails the corpus: the reference with five named faults (duplicate-on-retarget, lineage truncation, stale clobber, skipped grant check, suppressed zero-care write) is flagged on 7/15 cases; --expect-failure inverts the exit code.
  5. Mechanical enforcement per turn and at final state: raw transcript immutability, original-author stability, append-only revision lineage (kind + author per entry), entry-count deltas, publication as the only status transition and runner-driven, replayed correctionId = applied no-op, fail-closed write grants, undo = restore state before last mutation with an appended undo revision, zero-care = captured reviewable record (never a 0-duration quantity), ambiguous referents clarify with zero writes.
  6. packages/domain untouched — proposal-level lineage shapes; scope documented in the README section and package scripts (eval:corrections, eval:corrections:broken).

Verification evidence (exact head under review)

Head SHA: c32e79a — all results below ran against this commit in a clean worktree off origin/master (652ace5):

Check Result
bun x tsc --noEmit exit 0 (strict, noUncheckedIndexedAccess, verbatimModuleSyntax)
bun src/corrections/run.ts 15/15 cases PASS, exit 0
bun src/corrections/run.ts --adapter=./src/corrections/example/broken-adapter.ts 7/15 (one flag per fault), exit 1
same + --expect-failure faults caught, exit 0
bun src/run.ts (capture corpus regression) 6/6 fixtures PASS, exit 0

Scope

  • Adds only evaluation/corrections/**, evaluation/src/corrections/**, and edits evaluation/README.md + evaluation/package.json (two scripts + README section).
  • evaluation/ is not a pnpm workspace member, so no lockfile change and no product/backend surface touched.
  • Companion to the merged capture corpus (PR feat(evaluation): shared acceptance corpus + candidate-agnostic cross-review harness #6); the correction problem's product implementation stays out of scope until the contract lane lands lineage schemas.

…CC-15)

Independent, candidate-agnostic harness for the conversational correction
problem: spoken follow-ups resolve to applied / rejected / clarification
per turn, with mechanical enforcement of append-only revision lineage,
raw-transcript immutability, original-author stability, publication
separation, stale-suppression idempotency, fail-closed write grants,
and missing-vs-zero-care representation.

- evaluation/corrections/: household env + 6 case-group fixtures,
  15 cases (data only), forbidden-op vocabulary per case
- evaluation/src/corrections/: contract, fixture types, fail-fast loader,
  deterministic runner, CLI
- example adapter: mechanical rule interpreter, 15/15 pass
- broken control: five named faults, 7/15 cases flagged, --expect-failure
  inverts exit code
- README section + eval:corrections scripts; packages/domain untouched
  (proposal-level lineage shapes)
@obvious-autobuild
obvious-autobuild Bot force-pushed the eval/corrections-corpus branch from c32e79a to 8d386de Compare September 17, 2026 18:47
@obvious-autobuild
obvious-autobuild Bot merged commit 69e7bd3 into master Sep 17, 2026
1 check passed
@obvious-autobuild
obvious-autobuild Bot deleted the eval/corrections-corpus branch September 17, 2026 18:51
@obvious-autobuild

Copy link
Copy Markdown
Contributor Author

Evidence receipt

PR:               https://github.com/OCPdev25/obv-hackaton/pull/19
Tested head SHA:  8d386deb37a8cd4dc7a9b59a0aa3e7201dc97ca4
                  (rebased head — original c32e79a invalidated by the PR #22/#24
                  base movement per workflow rule 5; full battery re-run at 8d386de)
Review result:    pass — author review per repo policy (no required human
                  approvals); full diff read against the six acceptance criteria
                  in the PR body
Checks:           all green — "Typecheck, test, build" pass on 8d386de
                  (run 35260948598, 42s); PR CLEAN/MERGEABLE pre-merge
Merge commit:     69e7bd3 (squash onto master)
Post-merge smoke: on 69e7bd3 — pnpm install --frozen-lockfile ok;
                  pnpm typecheck ok; pnpm test ok; pnpm build ok;
                  bun test ./security 17 pass / 0 fail;
                  bun src/run.ts 6/6; capture broken control --expect-failure ok;
                  bun src/corrections/run.ts 15/15 (reference-rule-adapter);
                  corrections broken control 7/15 flagged, --expect-failure ok;
                  bun src/manifest-selfcheck.ts ok (no fixture-area interference)
Unlocked tasks:   slot 24 (append-only corrections / stale-result) can consume
                  this corpus as its candidate-agnostic gate; cross-review can
                  now score spoken-correction adapters on both corpora

Conflict resolution note (rule 5 rebase onto 44856c6): the two conflicts with
PR #22 were resolved as unions — package.json keeps both selfcheck:manifest
and the eval:corrections* scripts; the README keeps the manifest docs plus the
corrections section. The corrections corpus is deliberately not a
manifest-registered area (own runner + turn-protocol adapter contract); the
future correction-protocol fixture class is the documented seam if one CLI is
wanted later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants