feat(grader): a CHECKER breaks ties against the simulated user (0.8.9)

[that-guy-wade]

Summary

When the reader or the judge is sure the simulated user misread an assistant turn, the run waits for a person. Those confident disputes are usually wrong. On the published records, the published adjudications sided with the simulated user in 32 of 45 such disputes. With this change a CHECKER breaks the tie: it answers the same question, and if it is sure the user read the turn right, the user's reading stands, two readings to one. If it agrees with the reader or judge, or is unsure, the dispute goes to a person as before. Without a CHECKER nothing changes. Grader version 0.8.9.

Changes

• Question.user_answer: the answer a simulated-user decision implies. It is set by _read_checks and is not part of the key, so saved answers still match. Question.recorded() leaves it out of every recorded question (the causality context, which is part of that question's key, and the annotation queue), so saved causality answers still match and verdict JSON keeps its format.
• Ladder._upholds_user: a sure reader or judge answer against user_answer goes to the checker. The checker's answer is saved under its own rung, like any other, so --offline reproduces it.
• GRADER_VERSION 0.8.9; README role table, endpoints.py and the grader README describe the tie-break.

Testing

Measured on the 918 published trial 3–5 records, re-graded offline from their published saved answers, with the checker (Gemini 3.1 Pro) live:

• Disputes: with the published adjudications removed, there are 64 confident model disputes of a simulated-user reading.
• Against the 45 adjudicated ones:
  • the checker upholds the user 30 times, 25 in agreement with the adjudication;
  • it sides with the model 15 times, and those still go to a person.
  • The 5 upholds that disagree are borderline readings. In one, the rule's own text ("saying so counts as asking") supports the checker.
• Outcomes, adjudications removed: pending runs fall from 17 to 11 (0.8.8 with the checker, against this branch). Every resolved run lands on its published outcome, and no run is newly decided against its published outcome.
• Outcomes, adjudications kept: no outcome changes from the tie-break.
• Tests: test_a_checker_breaks_a_tie_against_the_simulated_users_reading covers the reader- and judge-decided cases, a checker that agrees with the model or is unsure, and no checker. test_a_tie_break_keeps_question_keys is also new. Both fail on main.
• Offline parity: with no checker, the branch reproduces all 918 published trial 3–5 outcomes from the published answers, including the two whose human causality answers are saved. test_a_causality_answer_saved_before_the_tie_break_still_replays pins a 0.8.8 causality key.
• Suite: uv run pytest -q gives 671 passed, and the parity goldens are unchanged. ruff format --check is clean.

Size

+98 / −9. CC: Ladder.decide_one 10 → 12, the direct cost of the tie-break; the tie-break itself is _upholds_user, 5.

Risks

• Comparability: with a CHECKER, verdicts can differ from 0.8.8 wherever a reader or judge confidently disputed the simulated user. The version is recorded in every verdict.
• The checker is a model. Repeat asks of borderline questions can flip; 2 of about 35 did in these runs. Saved answers make a re-grade reproducible.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; the follow-up fixes the saved causality-answer mismatch without changing the annotation queue's format.

Summary

Adds a CHECKER tie-break for confident disagreements with the simulated user in grader 0.8.9.

• The CHECKER can uphold a simulated user's reading of a turn.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Reader or judge gives a confident answer] --> B{Agrees with simulated user?}
  B -->|Yes| C[Keep the reading]
  B -->|No| D[Ask CHECKER or use its saved answer]
  D --> E{Confidently supports the user?}
  E -->|Yes| F[Keep the user's reading]
  E -->|No or unavailable| G[Send disagreement to a person]

Loading

_{Reviews (2) · Last reviewed commit: "fix(grader): keep user_answer out of rec..." · Reviewed by Greptile}