Repository navigation
fix(annotate): replay a checker-decided reading with its answer - #5
Merged
Merged
Conversation
A recorded question has no user_answer, so a reading the checker upheld for the simulated user replayed as the reader's answer and looked stale: annotation asked for a regrade before the causal question forever. A reading the checker decided now replays with its recorded value as the user's answer, which decides it the same way from unchanged answers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Assuming a checker-decided reading matched the simulated user missed a re-check the checker decided against the user: a reader answer that later became sure replayed as a tie-break and looked unchanged. The causal question's context records the simulated user's decisions, so each reading now replays with the answer its decision implies, as _read_checks sets it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Since 0.8.9 a CHECKER can uphold the simulated user's reading against a sure reader or judge. Recorded questions leave out
user_answer, which steers that tie-break, so that saved causality answers keep their keys.annotate._stalereplays the readings in the causal question's context to check the answers have not changed since the verdict. It rebuilt each question withoutuser_answer, so a tie-break reading replayed as the reader's own answer and always looked stale.annotatethen printed "answers changed; regrade before causal adjudication" forever. Found in review of the DonnaBench pin of 0.8.9.The causal question's context records the simulated user's decisions, and a reading's id names its decision.
_user_answerderives the answer that decision implies, as_read_checkssets it, and each reading replays with it. Question keys and verdicts do not change.Changes
annotate._user_answer(question_id, decisions), and_stale(..., user_answer)for the causal question's readings.Testing
test_a_checker_decided_reading_replays_with_the_users_answer:main);test_a_readings_user_answer_is_the_one_the_grader_sets:_user_answerequals_read_checks' answer for approve, refuse, reveal with a yes or a no, "i don't know", decline, finished and stopped-waiting decisions.uv run pytest -q: 679 passed.ruff format --checkis clean.Size
+106 / −2.
Risks
None to grading: only the annotation tool's staleness check changes.
🤖 Generated with Claude Code
The PR appears safe to merge; the replay now preserves the user's original answer.Summary
annotatenow rebuilds each reading'suser_answerfrom the saved simulated-user decisions before checking whether its answer changed.Reviews (2) · Last reviewed commit: "fix(annotate): replay readings with the ..." · Reviewed by Greptile