Repository navigation
feat: the defect lens asks the scan for the trigger up front - #341
Merged
Merged
Conversation
The verify pass (#331) confirmed 2 of the 12 top findings a scan produced; the other ten were casts every caller satisfied, guards one call away, or environments the project does not target. The rubric the verifier applies is now in the defect lens's own prompt: a finding is a reachable failure with its trigger named, and type hygiene goes at severity low under the pattern name `type-hygiene`. Measured on this repository, two runs per arm on the same model, findings judged by hand: half as many findings (24, 26 → 10, 13), medium-severity precision roughly doubled (2/12, 3/13 → 1/4, 3/6), the same number of real defects per run on average (2, 3 → 1, 4), same cost. The reading guide says to read `type-hygiene` rows as notes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 14, 2026
Merged
TheAmericanMaker
added a commit
that referenced
this pull request
Sep 14, 2026
The closeout for the session that turned verify on this repository and shipped 0.24.1: what shipped, the two-runs-per-arm measurement that decided the defect-lens rubric, the decisions (a prompt change is gated on a number; one release per batch; rename-then-rm does not fix a stale-break race), and the Pi-on-your-own-provider recipe. ROADMAP marks #143 shipped as the hybrid and adds the #341/#338/#339 rows; CONTRIBUTING gains the provider-extension note on the Pi row and the prompt-measurement recipe. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Item 3a of the 2026-09-14 recommendation, with the number the release was held for.
The
verifypass (#331) took the 2026-09-13 sweep's top twelve from 17% precision to 12/12 agreement with a reviewer by applying one rubric: a finding is a reachable failure — name the trigger; a cast every caller satisfies is not a defect. This PR puts that rubric into the defect lens's own batch prompt, after the per-language pattern list, so the noise is cut where it is produced rather than paid for twice. Type hygiene that is worth a note goes at severity low under the pattern nametype-hygiene, and the reading guide says to read those rows as notes.Measurement
Same repository (this one, at
bcb34ba), same model (google/gemini-3.7-flash:batch, low effort), defect lens only, two runs per arm. Every finding at medium was judged by hand against the source before reading the verifier's verdicts; lows were judged on the rubric runs and on the second baseline run.2026-09-13T20-00-19-293Z--namespace2026-09-14T04-05-15-830Z--namespace, linger timer2026-09-14T03-53-20-895Z2026-09-14T04-05-15-533Z--namespace, stale-lock break racerenderSafeLinkat low)verifyand a second run are for.verifyon the two second-sample runs: 11/12 agreement with the hand judgment on each (each "confirmed" one hypothetical the code does not reach: a pre-aborted signal no caller passes; a./-prefixedprimary_output). $0.14 and $0.16.--namespace(fix: /codecarto-library-init refuses an incomplete or invalid --namespace #336, fixed), theclearPhaselinger timer on re-entry, and the stale-lock break race inacquireLock— the last two are filed as issues.Recommendation: merge — it is free, it halves what a reader has to dismiss, and it does not cost recall on this evidence.
Test plan
tests/broadside.test.mjs: the rubric is pinned in the defect prompt for every language profile, after the pattern list.npm run build+npm test— 907 pass.🤖 Generated with Claude Code