Skip to content

feat: the defect lens asks the scan for the trigger up front - #341

Merged
TheAmericanMaker merged 1 commit into
mainfrom
feat/defect-lens-trigger-rubric
Sep 14, 2026
Merged

TheAmericanMaker merged 1 commit into
mainfrom
feat/defect-lens-trigger-rubric

Conversation

@TheAmericanMaker

Copy link
Copy Markdown
Member

Summary

Item 3a of the 2026-09-14 recommendation, with the number the release was held for.

The verify pass (#331) took the 2026-09-13 sweep's top twelve from 17% precision to 12/12 agreement with a reviewer by applying one rubric: a finding is a reachable failure — name the trigger; a cast every caller satisfies is not a defect. This PR puts that rubric into the defect lens's own batch prompt, after the per-language pattern list, so the noise is cut where it is produced rather than paid for twice. Type hygiene that is worth a note goes at severity low under the pattern name type-hygiene, and the reading guide says to read those rows as notes.

Measurement

Same repository (this one, at bcb34ba), same model (google/gemini-3.7-flash:batch, low effort), defect lens only, two runs per arm. Every finding at medium was judged by hand against the source before reading the verifier's verdicts; lows were judged on the rubric runs and on the second baseline run.

arm run findings (med / low) true at medium medium precision true overall cost
baseline (0.24.0 prompt) 2026-09-13T20-00-19-293Z 24 (12 / 12) amendment lock, --namespace 2/12 = 17% 2 $0.102
baseline 2026-09-14T04-05-15-830Z 26 (13 / 13) amendment lock, --namespace, linger timer 3/13 = 23% 3 $0.104
rubric (this PR) 2026-09-14T03-53-20-895Z 10 (4 / 6) amendment lock 1/4 = 25% 1 $0.097
rubric 2026-09-14T04-05-15-533Z 13 (6 / 7) amendment lock, --namespace, stale-lock break race 3/6 = 50% 4 (+ renderSafeLink at low) $0.096
  • Volume halves (mean 25 → 11.5); medium precision roughly doubles (mean 20% → 38%); real defects per run are the same on average (2.5 → 2.5), and either prompt misses a real one on some runs — the batch pass is a sample, which is what verify and a second run are for.
  • verify on the two second-sample runs: 11/12 agreement with the hand judgment on each (each "confirmed" one hypothetical the code does not reach: a pre-aborted signal no caller passes; a ./-prefixed primary_output). $0.14 and $0.16.
  • Four real defects surfaced across the four runs: amendment lock (fix: amendments judge pipeline completeness on the state read under the lock #337, fixed), --namespace (fix: /codecarto-library-init refuses an incomplete or invalid --namespace #336, fixed), the clearPhase linger timer on re-entry, and the stale-lock break race in acquireLock — the last two are filed as issues.
  • n = 2 per arm on one repository. The direction is consistent in both pairs; the magnitude is not a measurement.

Recommendation: merge — it is free, it halves what a reader has to dismiss, and it does not cost recall on this evidence.

Test plan

  • tests/broadside.test.mjs: the rubric is pinned in the defect prompt for every language profile, after the pattern list.
  • npm run build + npm test — 907 pass.
  • Live: the two rubric runs above.

🤖 Generated with Claude Code

The verify pass (#331) confirmed 2 of the 12 top findings a scan produced;
the other ten were casts every caller satisfied, guards one call away, or
environments the project does not target. The rubric the verifier applies
is now in the defect lens's own prompt: a finding is a reachable failure
with its trigger named, and type hygiene goes at severity low under the
pattern name `type-hygiene`.

Measured on this repository, two runs per arm on the same model, findings
judged by hand: half as many findings (24, 26 → 10, 13), medium-severity
precision roughly doubled (2/12, 3/13 → 1/4, 3/6), the same number of real
defects per run on average (2, 3 → 1, 4), same cost. The reading guide says
to read `type-hygiene` rows as notes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TheAmericanMaker
TheAmericanMaker merged commit 3a93260 into main Sep 14, 2026
5 checks passed
@TheAmericanMaker
TheAmericanMaker deleted the feat/defect-lens-trigger-rubric branch September 14, 2026 04:18
TheAmericanMaker added a commit that referenced this pull request Sep 14, 2026
The closeout for the session that turned verify on this repository and
shipped 0.24.1: what shipped, the two-runs-per-arm measurement that decided
the defect-lens rubric, the decisions (a prompt change is gated on a number;
one release per batch; rename-then-rm does not fix a stale-break race), and
the Pi-on-your-own-provider recipe. ROADMAP marks #143 shipped as the hybrid
and adds the #341/#338/#339 rows; CONTRIBUTING gains the provider-extension
note on the Pi row and the prompt-measurement recipe.



Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant