Skip to content

Harden agent compatibility scoring - #230

Open
garethpaul wants to merge 3 commits into
cursor:mainfrom
garethpaul:improve-agent-compatibility
Open

Harden agent compatibility scoring#230
garethpaul wants to merge 3 commits into
cursor:mainfrom
garethpaul:improve-agent-compatibility

Conversation

@garethpaul

@garethpaul garethpaul commented Aug 19, 2026

Copy link
Copy Markdown

Summary

  • pin and wrap the deterministic scanner with executable version, path, output-shape, and repository-classification checks
  • make a fail-closed synthesizer the sole authority for score arithmetic, degraded states, target identity, evidence, and isolated execution provenance
  • tighten specialist safety boundaries and structured handoffs, including separate temporary copies for stateful reviews
  • add contract tests, Cloudflare Worker and genuine CLI fixtures, symlink-install regressions, release metadata, and CI coverage

Validation

  • npx -y node@20 --test agent-compatibility/test/*.test.mjs (36 passed; one Windows-only hosted-runner assertion skipped on Linux)
  • node scripts/validate-plugins.mjs
  • Prettier 3.6.2 check across every changed file
  • git diff --check
  • reproduced scanner 0.1.7 misclassifying a real Cloudflare Worker as a CLI; the new guard correctly returned unreliable and excluded that deterministic score
  • added shell-free Windows scanner invocation plus Ubuntu and Windows CI matrix coverage after review feedback
  • three independent critic rounds; final verdict 9.4/10 with no blocking findings

Limitation

Full end-to-end orchestration inside an installed Cursor runtime was not available in this environment. The documented symlink-install boundary is covered by executable regression tests.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.

Reviewed by Cursor Bugbot for commit 0a36372. Configure here.

Comment thread agent-compatibility/test/deterministic-scan.test.mjs
Comment thread agent-compatibility/test/synthesize-results.test.mjs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant