Repository navigation
fix(grader): use a checker's saved answers offline (0.8.8) - #2
Merged
Merged
Conversation
The checker rung was skipped whenever no CHECKER was configured, before its saved answers were read, so `--offline` turned a verdict the checker settled (such as the causality question) back into one waiting for a person. Saved checker answers now decide as the reader's and judge's do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--offlinegrading skipped the checker rung because no CHECKER is configured offline, and it did so before reading the checker's saved answers. A verdict the checker settled, such as whether a result holds despite the simulated user's misreadings, re-graded offline as waiting for a person. The checker rung now uses saved answers as the reader and judge rungs do. Grader version 0.8.8.Changes
Ladder.decide_one: drop the live-checker gate; an unset rung already answers from saved answers only.GRADER_VERSION0.8.8, with a version note indocs/grader/README.md.Testing
test_a_checker_answers_the_causality_question_the_judges_never_seenow also re-grades offline from the same saved answers and expects the same verdict. It fails onmain.uv run pytest -q: 664 passed, including the parity goldens.ruff format --check: clean.Nonechecker answer is the field's default, and the "no saved answer" note is the same.Size
+11 / −7. CC:
Ladder.decide_onedrops one condition.Risks
Grading without a CHECKER now uses checker answers already saved in the answers file, as it does for the other rungs.
🤖 Generated with Claude Code
The PR appears safe to merge; saved checker answers now follow the existing offline rules.Summary
The grader now reads saved checker answers even when no live checker is configured.
Reviews (1) · Last reviewed commit: "fix(grader): use a checker's saved answe..." · Reviewed by Greptile