Repository navigation
fix: stop regenerating answers over quality guesses - #167
Merged
Merged
Conversation
Validation now blocks an answer only when the application cannot render or persist it: output that fails the transmitted schema, an answer cut off with no complete block, or a research answer with no resolvable citation. Every other check repairs the answer or attaches a note under Source notes and limitations, and never triggers regeneration. - Unmatched or ambiguous citations are removed with a CITATIONS_REMOVED note instead of failing the save. - Inline markers for undeclared references are stripped. - Answers that reach the output limit keep their complete blocks with an ANSWER_TRUNCATED note. - Mass and percentage figures missing from a block's own citations add an UNVERIFIED_FIGURES note. - Uncited comparison subjects add a PARTIAL_EVIDENCE note. - First-message requests quote the stored text directly. - Numbered-item, filler, sentence-split and repeated-warning heuristics no longer reject answers; the first two are logged for review. - The transcript analyst keeps findings with unverified figures instead of discarding them, so grounding problems never fail an analysis. - Accepted answers with notes record a private answer_review trace. - Answer acceptance hashes the saved text synchronously, which removes the async gap before commit.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…nswers Truncation recovery dropped the last block of every cut-off answer, even when that block had closed and only the outer JSON was missing. A complete two-item comparison lost its second item, and a complete single-block answer was regenerated. Recovery now keeps every block whose object closed in the raw text and drops only a block cut off mid-way. JSON that closes cleanly despite the limit still loses its last block, which may have been closed early. A first-message answer without the exact stored text was prefixed with the correct quote, leaving the model's wrong statement underneath. The application knows the answer, so it now replaces the whole answer.
Notes alone left unsupported claims confidently worded in the answer. Following review, specific problems are now marked in place: - A figure whose cited source has the same value with a different unit, a likely unit error, is marked "(unverified)" next to the figure. - A citation that matches no saved evidence shows "[source unavailable]" where it appeared, as deleted sources already show "[source deleted]". - An inline reference to real evidence that its block did not declare stays where the model placed it instead of being dropped. - A filler-only answer is not an answer, so it gets one repair again.
devhims
changed the title
The renderer keeps inline citations, but the figure check read only a block's declared evidenceIds. A correct figure cited inline from source B was marked unverified because only source A was declared, and an answer citing only inline skipped the check entirely. Rendering, the figure check and comparison coverage now share one reference set per block: declared plus inline references after alias mapping. Detection and inline marking also disagreed on Hindi digits: the checker normalized ५४.२ ग्राम to 54.2g, but the marker searched the original text with an ASCII-only pattern and never placed "(unverified)". Both now use one matcher on the original text that reads ASCII, Devanagari and fullwidth digits, so the mark lands on the exact figure that was flagged.
An answer that reached the output limit but whose JSON closed cleanly lost its last block, on the theory that the provider might have closed it early. That hid the case instead of showing whether it happens. Such answers are now kept whole with the ANSWER_TRUNCATED note, and the answer-review trace records how each length-limited answer ended: dropped_block, complete_blocks or closed_cleanly.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem / Motivation
Final answers fail in production because validation treats quality guesses as hard failures. Session
ba0c904a(run3809e44c) produced two complete answers that were both rejected withUNGROUNDED_ANSWER, then fell back to a partial summary and an error.The most likely cause is the cross-video name rule. Four Manali videos had been analyzed. Findings about Rohtang Pass came from Hindi-script transcripts, while only one other video recorded the English name, so any paragraph that mentioned "Rohtang Pass" and cited the video those facts came from would be rejected. The numerical rule could not plausibly fire, because the evidence held no mass or percentage figures. This is an inference from the logged error code, the saved analyses and the validator code: the rejected drafts were not saved, so the exact failing paragraph cannot be confirmed. The rejected-answer traces included here make the next case directly readable.
Other rules had the same shape: a regex guess that, when wrong, discards a complete answer and restarts generation, adding 10 to 20 seconds and resetting the streamed draft.
Rule
An answer is regenerated (once) only when it is unusable:
When the application knows the correct result, it corrects the answer. When a specific claim is likely unsupported, the claim is qualified where it appears, not only in a note. Everything else is recorded for review.
What changed
54.2 g (unverified)in the answer, plusUNVERIFIED_FIGURESnoteUNVERIFIED_FIGURESnote listing the figures[source unavailable]where it appeared (as deleted sources show[source deleted]),CITATIONS_UNAVAILABLEnote[source unavailable]ANSWER_TRUNCATEDnote. JSON that closes cleanly despite the limit is kept whole, and the review trace records which case occurred. Regenerate only if no complete block remainsPARTIAL_EVIDENCEnote naming the subjectneedsEvidencefieldThe transcript analyst keeps a finding whose figures cannot be matched, marked unverified, instead of discarding it. When some figures in a finding do match, the finding is still rewritten from those figures only. Grounding problems no longer trigger an analyst retry or fail an analysis.
Diagnostics: this PR includes the two trace commits from #165. Rejected answers are saved to private traces as
final_answer_rejection. Accepted answers that received notes, corrections or review signals record ananswer_reviewtrace and anagent_answer_reviewlog event, without answer text, so each check can be measured in production.The Durable Object now hashes the accepted answer synchronously after building it, because marking citations unavailable can change the saved text. This also removes the only
awaitbefore commit, so cancellation, deletion or a concurrent finalize can no longer interleave with acceptance.Tests
All 1,592 unit tests and 348 runtime/session integration tests pass. Type checking passes. Tests that encoded the old rejections now assert the new outcome. Tests that used a quality rejection to exercise the repair path now use a schema-invalid or filler answer, so the repair path keeps its coverage. New tests cover:
ba0c904a(Hindi transcripts, Rohtang Pass cited from another video)[source unavailable]at save time and for unusable inline markersScope
This PR changes validation only. It does not implement synthesis-only finalization; the existing context-gathering reads remain. The research-handoff commit from #165 is not included. Failover timers that restart long streaming answers are a separate issue.
Compatibility and deployment
No migration. Warning codes are free-form strings, so the new note codes need no contract change. Existing traces and answers are unaffected.