Release 0.14.13: ess:hardening, a specification-hardening catalogue (#35)

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## [0.14.13] — 2026-09-26
+
+Hardening a specification after its suite is green.
+
+- `ess:hardening`: eight specification-hardening techniques for after a conformance suite is green —
+  mutation, random sequences against a reference model, caller replay, determinism, metamorphic
+  relations, guard analysis, spec diff in the gate and design review — in cheapest-first order with
+  the design review early, and the rule that a check nobody has seen fail is not evidence. Its
+  `references/` carry a procedure per technique, the reference-model pattern (to ship as `ess`
+  features, beyond10x/ess#114), a design-review brief and spec-diff classification. (#35)
+- `ess:conformance` offers the catalogue when a coverage task ends on a green suite, and
+  `ess:testing-conformance` links it from its mutation section.
+
 ## [0.14.12] — 2026-09-26
 
 Skills that start from what a repository already has, and plans that say what the other host needs.

Cargo.toml

@@ -3,7 +3,7 @@ resolver = "2"
 members = ["crates/agentplugins-check", "crates/b10x"]
 
 [workspace.package]
-version = "0.14.12"
+version = "0.14.13"
 edition = "2021"
 rust-version = "1.85"
 license = "Apache-2.0"

README.md

@@ -17,7 +17,7 @@ What each plugin ships:
 
 <!-- plugin-tree:start -->
 - [`ess`](plugins/ess/) · [docs](website/docs/plugins/ess.md)
-  - skills: [`init`](plugins/ess/skills/init/SKILL.md) · [`upgrade`](plugins/ess/skills/upgrade/SKILL.md) · [`retrofitting`](plugins/ess/skills/retrofitting/SKILL.md) · [`specifying`](plugins/ess/skills/specifying/SKILL.md) · [`testing-conformance`](plugins/ess/skills/testing-conformance/SKILL.md)
+  - skills: [`init`](plugins/ess/skills/init/SKILL.md) · [`upgrade`](plugins/ess/skills/upgrade/SKILL.md) · [`hardening`](plugins/ess/skills/hardening/SKILL.md) · [`retrofitting`](plugins/ess/skills/retrofitting/SKILL.md) · [`specifying`](plugins/ess/skills/specifying/SKILL.md) · [`testing-conformance`](plugins/ess/skills/testing-conformance/SKILL.md)
   - agents: [`author`](plugins/ess/agents/author.md) · [`conformance`](plugins/ess/agents/conformance.md) · [`retrofitter`](plugins/ess/agents/retrofitter.md)
 - [`aep`](plugins/aep/) · [docs](website/docs/plugins/aep.md)
   - skills: [`init`](plugins/aep/skills/init/SKILL.md) · [`upgrade`](plugins/aep/skills/upgrade/SKILL.md) · [`implementing`](plugins/aep/skills/implementing/SKILL.md) · [`migrating`](plugins/aep/skills/migrating/SKILL.md) · [`planning`](plugins/aep/skills/planning/SKILL.md)

crates/agentplugins-check/src/main.rs

@@ -61,6 +61,11 @@ const PLUGINS: &[(&str, &[&str])] = &[
             "skills/specifying/references/syntax.md",
             "skills/retrofitting/SKILL.md",
             "skills/testing-conformance/SKILL.md",
+            "skills/hardening/SKILL.md",
+            "skills/hardening/references/techniques.md",
+            "skills/hardening/references/reference-model.md",
+            "skills/hardening/references/design-review.md",
+            "skills/hardening/references/spec-diff.md",
             "agents/author.md",
             "agents/retrofitter.md",
             "agents/conformance.md",
@@ -1376,6 +1381,34 @@ one product only: `b10x upgrade ess`
         std::fs::remove_dir_all(&sandbox).expect("the sandbox is removable");
     }
 
+    /// A green suite is where hardening starts, so the two places an agent stands when it gets
+    /// there — the conformance agent's charter and the skill's mutation section — hand it the
+    /// catalogue. Without the link the catalogue is a file nobody finishing a coverage task reads.
+    #[test]
+    fn a_green_suite_is_handed_the_hardening_catalogue() {
+        let root = Path::new(env!("CARGO_MANIFEST_DIR"))
+            .parent()
+            .and_then(Path::parent)
+            .expect("checker is under repository root");
+        let agent = std::fs::read_to_string(root.join("plugins/ess/agents/conformance.md"))
+            .expect("the conformance agent is readable");
+        assert!(
+            agent.contains("`ess:hardening`"),
+            "the conformance agent does not offer `ess:hardening`"
+        );
+        let skill =
+            std::fs::read_to_string(root.join("plugins/ess/skills/testing-conformance/SKILL.md"))
+                .expect("the testing-conformance skill is readable");
+        let mutation = skill
+            .split_once("## Before anything: does a green run mean anything?")
+            .and_then(|(_, rest)| rest.split_once("\n## ").map(|(section, _)| section))
+            .expect("the mutation section is present");
+        assert!(
+            mutation.contains("`ess:hardening`"),
+            "the mutation section does not link `ess:hardening`"
+        );
+    }
+
     /// The committed critics carry the pin. This is the half of the check that would catch a fifth
     /// critic added without one.
     #[test]

plugins/aep/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "aep",
   "displayName": "AEP",
   "description": "Plan governed work in the AEP artifact store and deliver it in reviewed waves: decomposition, plan critique, reverse engineering, story scoping, implementation and adversarial review.",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "author": {
     "name": "Beyond10x"
   },

plugins/aep/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "aep",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "description": "Plan governed work in the AEP artifact store and deliver it in reviewed waves.",
   "author": {
     "name": "Beyond10x"

plugins/b10x/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "b10x",
   "displayName": "Beyond10x",
   "description": "Set up, upgrade and check the Beyond10x plugins and binaries, route work to them, and create portable plugins.",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "author": {
     "name": "Beyond10x"
   },

plugins/b10x/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "b10x",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "description": "Set up, upgrade and check the Beyond10x plugins and binaries, route work to them, and create portable plugins.",
   "author": {
     "name": "Beyond10x"

plugins/b10x/skills/routing/SKILL.md

@@ -30,6 +30,7 @@ Route the request; do not reproduce a specialist plugin's full workflow.
 | Specify a system or API | `ess:specifying` |
 | Derive a specification for an existing system | `ess:retrofitting` |
 | Run or raise a conformance suite | `ess:testing-conformance` |
+| Harden a specification once its suite is green | `ess:hardening` |
 | Create, inspect, finish, or safely clean Git worktrees | `worktree:managing-worktrees` |
 | Set up providers, inspect Connector readiness, or invoke configured integrations through the CLI | `connectors:integrating` |
 

plugins/connectors/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "connectors",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "description": "Set up, inspect, and invoke governed integrations through the connectors CLI.",
   "author": { "name": "Beyond10x" },
   "license": "Apache-2.0",

plugins/connectors/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "connectors",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "description": "Set up, inspect, and invoke governed integrations through the connectors CLI.",
   "author": { "name": "Beyond10x" },
   "license": "Apache-2.0",

plugins/ess/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "name": "ess",
   "displayName": "ESS",
   "description": "Write, retrofit, validate and project Executable System Specifications, and hold implementations to them with conformance suites.",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "author": {
     "name": "Beyond10x"
   },

plugins/ess/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "ess",
-  "version": "0.14.12",
+  "version": "0.14.13",
   "description": "Write, retrofit, validate and project Executable System Specifications, and hold implementations to them with conformance suites.",
   "author": {
     "name": "Beyond10x"

plugins/ess/agents/conformance.md

@@ -16,3 +16,6 @@ Charter:
 - Never weaken a scenario or a target to turn a failure green.
 - Report: `passed`/`skipped`/`failed` before and after, the mutation you ran and the scenario it
   failed, and the exact commands.
+- When a coverage task finishes on a green suite, offer the `ess:hardening` catalogue as the next
+  step, not only mutation: name the techniques it would run first (the design review, the spec
+  diff in the gate) and what each needs. Offer it; do not start it unasked.

plugins/ess/skills/hardening/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: hardening
+description: >-
+  Harden an ESS specification and the implementation held to it once the conformance suite is green — ask the questions a passing suite cannot, with a catalogue of eight techniques from mutation to a design review. Use when the user asks how to harden a specification, whether a green suite can be trusted, what to do after coverage is raised, or asks for mutation testing, random command sequences or a model-based check, replay of a real caller, determinism checks, metamorphic relations, guard analysis, classifying a specification diff as breaking, or reviewing a design document against a specification. Not for raising the number of scenarios that execute, which is the `ess:testing-conformance` skill; not for writing the specification, which is `ess:specifying`.
+---
+
+# Hardening a specification
+
+A green conformance suite answers one question: do the scenarios somebody wrote pass. Each
+technique here asks a different one, and each runs only after the suite is green — a red suite is
+the work, and `ess:testing-conformance` is where it is done.
+
+## The rule that makes any of this count
+
+**A check nobody has seen fail is not evidence. Plant a defect first.** Before a technique's green
+result is reported, break the implementation (or the specification) in the way the technique claims
+to catch, run it, and record the named failure. Then restore and run it green. A technique that
+cannot be made to fail on a planted defect has measured nothing, whatever it prints.
+
+Record both runs: the planted defect, the command, the failure it produced, and the green run after
+restoring.
+
+## The catalogue
+
+| # | technique | the question | what it catches | needs from the IR |
+|---|---|---|---|---|
+| 1 | mutation audit | would the suite notice if a declared rule broke? | declared behaviour no scenario pins down: a dropped `from` state, a moved guard boundary, a dropped `sets` entry | lifecycles, guards, `sets`, payloads |
+| 2 | random command sequences against a reference model | does the implementation agree with the spec along paths nobody wrote? | engine defects that only show later in a sequence: a view that caps its rows, a refusal that still writes, an identity reused, a state not re-enterable; invariants that hold only through undeclared defaults | everything the [reference model](references/reference-model.md) reads: lifecycles, guards, `sets`, payloads, views, invariants, `wrong_state` |
+| 3 | replay of the real caller | does the code that *uses* the implementation send only what the spec accepts? | a caller relying on behaviour the spec does not declare; refusal outcomes the caller never triggers | the reference model, plus actors (who may send which command) |
+| 4 | determinism | is the implementation deterministic where the spec assumes it? | clock reads, randomness, iteration order leaking into outcomes | none beyond technique 2's runner |
+| 5 | metamorphic relations | do promises that span two runs hold? | a command allowed where the design forbids it ("a pause changes nothing but the pause commands") | the reference model; the relation comes from the design |
+| 6 | exhaustive guard analysis | are guards dead, overlapping, gap-leaving, or weak enough to admit an invariant break? | a `when:` that can never match, two that match together, an input no outcome answers | guards, field types, invariants |
+| 7 | spec diff in the gate | is a change breaking, and was that acknowledged? | a narrowing released as if it were additive | two compiled revisions |
+| 8 | design review by an agent | what does the design say that the spec omits or contradicts? | rules the design states and the spec never declares ("one entry per player", "the count always goes up") | the spec files and the design document; no IR |
+
+Procedures, with the defect to plant for each: [references/techniques.md](references/techniques.md).
+
+Formal model checking (a TLA+ or Alloy export) is not in the catalogue. On a spec with 31 commands,
+random sequences reached every declared outcome, 59 of 59; reach for a model checker only when
+technique 2 reports declared outcomes it never reaches.
+
+## The order
+
+Cheapest first, with one exception: **the design review runs early, because it finds the most.** On
+one adopter's spec (3 domains, 7 entities, 31 commands, a 121-scenario suite that passed) it
+produced 26 findings, the most of any technique. What it finds changes the spec, and
+every later technique is then run against the spec that will stay.
+
+1. **Design review** (8) — no tooling, one agent, the [brief](references/design-review.md).
+2. **Spec diff in the gate** (7) — one command and a [classification](references/spec-diff.md); it
+   protects everything after it.
+3. **Mutation audit** (1) — reuses the suite you already have.
+4. **Guard analysis** (6) — reads the IR only; no implementation runs.
+5. **Reference model and random sequences** (2) — the one piece of code the catalogue needs; build it
+   once.
+6. **Determinism** (4), **metamorphic relations** (5) and **caller replay** (3) — each reuses the
+   runner from step 5.
+
+Stop where the cost exceeds what the spec is worth, and say which techniques were not run.
+
+## What the IR is for
+
+Techniques 1–6 read the canonical IR, not the YAML:
+
+```console
+ess specify compile --path <specification> --format json --out <ir.json>
+```
+
+The IR is the resolved model: every name qualified, every guard in structured form, every default
+applied. A technique that reads the YAML re-implements resolution and disagrees with the compiler
+somewhere nobody looked.
+
+## The reference model
+
+Techniques 2, 3 and 5 rest on one small interpreter over the IR — about 150 lines — that answers,
+for any command in any state, what the spec says happens. `ess` does not ship it yet;
+[beyond10x/ess#114](https://github.com/beyond10x/ess/issues/114) proposes shipping the mutation audit
+and the sequence runner as `ess` features, and carries a draft for the TypeScript target. Until it
+ships, build it from the pattern: [references/reference-model.md](references/reference-model.md).
+
+## Reporting
+
+Per technique run: the planted defect and the failure it produced, the green result after restoring,
+the command, and the findings — each one a spec gap, an implementation defect or a false positive of
+the technique, said which. Name the techniques not run. A finding that belongs to `ess` itself (a
+generator or validate gap) is an issue on beyond10x/ess, not a workaround here.
+
+## Next
+
+- A finding changes the specification: `ess:specifying`.
+- A finding is a missing scenario or a skipped one: `ess:testing-conformance`.

plugins/ess/skills/hardening/references/design-review.md

@@ -0,0 +1,71 @@
+# Design review: a brief for an agent
+
+Technique 8. Dispatch one agent with the brief below, filled in. It found the most of any technique
+on the one adopter where all eight were run — 26 findings — so run it early.
+
+The agent needs read access to three things and write access to none of them.
+
+---
+
+## Brief
+
+You are reviewing a specification against the design it claims to implement. You do not edit
+either. You produce a list of findings.
+
+**Inputs**
+
+- the design document: `<path>`
+- the mapping, if one exists — the document or table that says which design statement became which
+  spec declaration: `<path>`, or `none`
+- the specification: `<directory>`; validate it first with `ess specify validate --path <directory>`
+  and stop if it refuses
+
+**Procedure**
+
+1. Read the design end to end. List every statement that promises behaviour: a rule, a limit, an
+   ordering, a uniqueness, a "never", an "always", a "only when".
+2. For each statement, find where the spec declares it — an entity field or invariant, a lifecycle
+   transition, a command outcome and its `when:` guard, an actor's grant, a view.
+3. Then go the other way: for every spec declaration, find the design statement that justifies it.
+4. Classify each mismatch with exactly one of the classifications below.
+
+**Classifications**
+
+| classification | meaning |
+|---|---|
+| `missing` | the design states it; the spec does not declare it |
+| `contradicts` | both state it, and they disagree |
+| `stale mapping` | the mapping points a design statement at a declaration that no longer says it, or at one that does not exist |
+| `spec-only` | the spec declares it; no design statement justifies it |
+| `unclear` | the design statement is too vague to decide which of the above applies |
+
+**Every finding cites both sides.** The design by its line (`design.md:42`, quoted), the spec by
+`file:line` (quoted). A `missing` finding cites the spec file and line where the declaration would
+belong; a `spec-only` finding cites the nearest design section and says it has nothing. A finding
+without both citations is not a finding; drop it.
+
+**Quote a vague statement and mark it `unclear`; never invent a meaning for it.** "Players should
+not be able to cheat" is `unclear` with that sentence quoted. Supplying a plausible rule and then
+reporting the spec as missing it is the one failure this review exists to avoid: downstream, the
+invented rule reads as the designer's.
+
+**Output**, one row per finding:
+
+```text
+| # | classification | design (file:line, quoted) | spec (file:line, quoted) | note |
+```
+
+Then the counts per classification, and the design sections you did not reach, if any.
+
+---
+
+## After the review
+
+- Each `missing` and `contradicts` finding goes to the design's owner or becomes a spec change
+  (`ess:specifying`); the review does not decide which.
+- `spec-only` findings are candidates for deletion — "a declared refusal nobody executes is a
+  fiction" (`ess:testing-conformance`).
+- `unclear` findings go back to the design's author as questions, with the quote.
+
+**Plant a defect before trusting a clean review:** on a copy of the spec, delete one declaration the
+design plainly states, and run the brief on the copy. It must come back `missing`, citing both sides.