Skip to content

feat(governance): show what a retired-rule drop actually changes, not the rule text (v0.437.0) - #810

Merged
vikasprogrammer merged 1 commit into
mainfrom
feat/drift-impact
Sep 10, 2026
Merged

feat(governance): show what a retired-rule drop actually changes, not the rule text (v0.437.0)#810
vikasprogrammer merged 1 commit into
mainfrom
feat/drift-impact

Conversation

@vikasprogrammer

Copy link
Copy Markdown
Owner

Why

v0.428.0 surfaced rules the product retired that a tenant still enforces, and left the drop to an owner's click. The click was still half-blind — it showed the rule. A rule's effect is not readable from the rule.

On expresstech the retired shell.exec+risky rule sat at index 0, ahead of all three never guardrails. First-match meant it shadowed every one of them:

stricter  shell.exec {"risky":true,"destructive":true}   approve:owner → deny   (any action: destructive)
stricter  shell.exec {"risky":true,"amountUsd":501}      approve:owner → deny   (any action: amountUsd > 500)
stricter  shell.exec {"risky":true,"deleteCount":26}     approve:owner → deny   (any action: deleteCount > 25)
looser    shell.exec {"risky":true}                      approve:owner → allow  (default policy)

Dropping it made that tenant stricter — the opposite of what the rule text suggested, and invisible without classifying the whole ordered document both ways. Any tenant carrying that rule at the top had the same three holes.

What ships

classificationDiff() (src/governance/policy.ts) reuses the monotonicity sweep's own arg space (sampleArgDomains / sampleCapabilities) to collect every distinct verdict that moves between two rulesets:

  • both directions (firstLoosening answers "is this safe?" with one counter-example and stops; this answers "what does this do?"),
  • deduped by capability + verdict pair + reason,
  • each with a minimal example, greedily shrunk from its sweep point — a row reads {risky:true, destructive:true}, not every branch arg the ruleset can key on,
  • stricter rows sorted first, since a tightening you're about to lose is the point.

retiredRuleImpact() applies it to a single drop. GET /api/policy carries it per retired hit; Settings → Policy renders direction, before → after, the rule that applies afterwards, and an explicit note when a rule is shadowing a stricter one below it.

Cost: ~6ms per rule on a real 8-rule document, 9ms end-to-end over HTTP. Only tenants that actually carry a retired rule pay it — a clean tenant has no hits.

Also: a correction

The justification comment in policy-baseline.ts stated something false about the fleet — that the same rule was "a guardrail somebody is actually USING on expresstech (5 rejected of 26)". Checking those rejections: four were the heredoc false-positive class fixed in v0.425.1, and the fifth was an rm -rf carrying destructive: true that should never have been approvable at all. The click stays, for the stronger reason now recorded in its place.

Verification

  • scripts/policy-baseline-test.cjs 32 → 44 cases: the shadowing case, minimality of examples, the below-the-guardrails case where only the loosening remains, and the not-retired / out-of-range / clean no-ops.
  • End-to-end over HTTP on a scratch tenant: impact present in the payload with live thresholds resolved from settings, stricter rows first.
  • npm run typecheck, cd web && npm run build, full npm run test:governance green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gwz3ZqU9iid84Sc6EzUt9M

… the rule text (v0.437.0)

v0.428.0 surfaced rules the product retired that a tenant still enforces and
left the drop to an owner's click. The click was still half-blind: it showed
the rule. A rule's effect is not readable from the rule.

On expresstech the retired shell.exec+risky rule sat at index 0, AHEAD of all
three `never` guardrails, so first-match shadowed every one of them:

  destructive rm -rf   approve:owner -> deny  (any action: destructive)
  spend over the cap   approve:owner -> deny  (any action: amountUsd > cap)
  bulk delete over cap approve:owner -> deny  (any action: deleteCount > 25)
  plain risky command  approve:owner -> allow (default policy)

Dropping it made that tenant STRICTER - the opposite of what the rule text
suggested, and invisible without classifying the whole ordered document both
ways. Any tenant carrying that rule at the top had the same three holes.

classificationDiff() reuses the monotonicity sweep's own arg space
(sampleArgDomains / sampleCapabilities) to collect every DISTINCT verdict that
moves between two rulesets - both directions, deduped by capability + verdict
pair + reason, each with a MINIMAL example greedily shrunk from its sweep point
so a row reads {risky:true, destructive:true} rather than every branch arg the
ruleset can key on. retiredRuleImpact() applies it to one drop; GET /api/policy
carries it per retired hit; Settings -> Policy renders direction, before ->
after, the rule that applies afterwards, and an explicit note when a rule is
shadowing a stricter one below it.

~6ms per rule on a real 8-rule document, 9ms end to end over HTTP, and only
tenants that actually carry a retired rule pay it (a clean tenant has no hits).

Also corrects the justification comment in policy-baseline.ts, which stated
something false about the fleet: that the same rule was "a guardrail somebody is
actually USING on expresstech (5 rejected of 26)". Four of those rejections were
the heredoc false-positive class fixed in v0.425.1; the fifth was an rm -rf
carrying destructive:true that should never have been approvable. The click
stays, for the stronger reason now recorded in its place.

scripts/policy-baseline-test.cjs 32 -> 44 cases, pinning the shadowing case,
the minimality of examples, the below-the-guardrails case where only the
loosening remains, and the not-retired / out-of-range / clean no-ops.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Gwz3ZqU9iid84Sc6EzUt9M
@vikasprogrammer
vikasprogrammer merged commit 56f0c56 into main Sep 10, 2026
1 check passed
@vikasprogrammer
vikasprogrammer deleted the feat/drift-impact branch September 10, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant