feat(governance): show what a retired-rule drop actually changes, not the rule text (v0.437.0) - #810
Merged
Merged
Conversation
… the rule text (v0.437.0)
v0.428.0 surfaced rules the product retired that a tenant still enforces and
left the drop to an owner's click. The click was still half-blind: it showed
the rule. A rule's effect is not readable from the rule.
On expresstech the retired shell.exec+risky rule sat at index 0, AHEAD of all
three `never` guardrails, so first-match shadowed every one of them:
destructive rm -rf approve:owner -> deny (any action: destructive)
spend over the cap approve:owner -> deny (any action: amountUsd > cap)
bulk delete over cap approve:owner -> deny (any action: deleteCount > 25)
plain risky command approve:owner -> allow (default policy)
Dropping it made that tenant STRICTER - the opposite of what the rule text
suggested, and invisible without classifying the whole ordered document both
ways. Any tenant carrying that rule at the top had the same three holes.
classificationDiff() reuses the monotonicity sweep's own arg space
(sampleArgDomains / sampleCapabilities) to collect every DISTINCT verdict that
moves between two rulesets - both directions, deduped by capability + verdict
pair + reason, each with a MINIMAL example greedily shrunk from its sweep point
so a row reads {risky:true, destructive:true} rather than every branch arg the
ruleset can key on. retiredRuleImpact() applies it to one drop; GET /api/policy
carries it per retired hit; Settings -> Policy renders direction, before ->
after, the rule that applies afterwards, and an explicit note when a rule is
shadowing a stricter one below it.
~6ms per rule on a real 8-rule document, 9ms end to end over HTTP, and only
tenants that actually carry a retired rule pay it (a clean tenant has no hits).
Also corrects the justification comment in policy-baseline.ts, which stated
something false about the fleet: that the same rule was "a guardrail somebody is
actually USING on expresstech (5 rejected of 26)". Four of those rejections were
the heredoc false-positive class fixed in v0.425.1; the fifth was an rm -rf
carrying destructive:true that should never have been approvable. The click
stays, for the stronger reason now recorded in its place.
scripts/policy-baseline-test.cjs 32 -> 44 cases, pinning the shadowing case,
the minimality of examples, the below-the-guardrails case where only the
loosening remains, and the not-retired / out-of-range / clean no-ops.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gwz3ZqU9iid84Sc6EzUt9M
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
v0.428.0 surfaced rules the product retired that a tenant still enforces, and left the drop to an owner's click. The click was still half-blind — it showed the rule. A rule's effect is not readable from the rule.
On expresstech the retired
shell.exec+riskyrule sat at index 0, ahead of all threeneverguardrails. First-match meant it shadowed every one of them:Dropping it made that tenant stricter — the opposite of what the rule text suggested, and invisible without classifying the whole ordered document both ways. Any tenant carrying that rule at the top had the same three holes.
What ships
classificationDiff()(src/governance/policy.ts) reuses the monotonicity sweep's own arg space (sampleArgDomains/sampleCapabilities) to collect every distinct verdict that moves between two rulesets:firstLooseninganswers "is this safe?" with one counter-example and stops; this answers "what does this do?"),{risky:true, destructive:true}, not every branch arg the ruleset can key on,retiredRuleImpact()applies it to a single drop.GET /api/policycarries it per retired hit; Settings → Policy renders direction, before → after, the rule that applies afterwards, and an explicit note when a rule is shadowing a stricter one below it.Cost: ~6ms per rule on a real 8-rule document, 9ms end-to-end over HTTP. Only tenants that actually carry a retired rule pay it — a clean tenant has no hits.
Also: a correction
The justification comment in
policy-baseline.tsstated something false about the fleet — that the same rule was "a guardrail somebody is actually USING on expresstech (5 rejected of 26)". Checking those rejections: four were the heredoc false-positive class fixed in v0.425.1, and the fifth was anrm -rfcarryingdestructive: truethat should never have been approvable at all. The click stays, for the stronger reason now recorded in its place.Verification
scripts/policy-baseline-test.cjs32 → 44 cases: the shadowing case, minimality of examples, the below-the-guardrails case where only the loosening remains, and the not-retired / out-of-range / clean no-ops.impactpresent in the payload with live thresholds resolved from settings, stricter rows first.npm run typecheck,cd web && npm run build, fullnpm run test:governancegreen.🤖 Generated with Claude Code
https://claude.ai/code/session_01Gwz3ZqU9iid84Sc6EzUt9M