Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,37 @@
# Changelog

## [0.15.0] — 2026-09-28

Eight methods from two public MIT skill collections, rewritten into the `aep`, `ess` and `b10x`
plugins, each with an eval case beside it.

- `aep:diagnosing` is a new activity skill. It builds one command that goes red on the reported
symptom, and has already been run, before any hypothesis. Then it minimises, ranks falsifiable
hypotheses, probes one variable at a time and writes the regression test at a seam that
reproduces the real call pattern. The red and green runs are `test_result` evidence; a missing
seam is filed as a draft story. `b10x:routing` reaches it.
- `ess:specifying` interviews before drafting a domain nobody has described
(`references/interview.md`): the open decisions in numbered rounds, each with the answer the
agent would take, facts looked up rather than asked. Headless, the questions become
`approval-record`s. `aep:planning` guardrail 7 points at it.
- The adversary and `ess:testing-conformance` reject a test that would still pass if every
function it calls returned a default value, and name the five shapes that do.
- A wave verifies a green unit's claim against the base before merging: one command on both
branches, one verdict of `VERIFIED`, `NOT VERIFIED` or `INCONCLUSIVE`, recorded as
`verification` evidence. Only `VERIFIED` merges.
- The story scoper returns a `Safety fact` line with how far it was proved on a five-step ladder;
the security reviewer proves that fact with a test before calling a change safe.
- The decomposer cuts stories as vertical slices and drafts a codebase-wide mechanical change as
expand, migrate batches and contract. The design critic's fifth defect is the horizontal slice.
- On request, the adversary step runs as a cross-family panel (`references/adversary-panel.md`),
one `review-result` per seat; with one model family it is one reviewer and says so.
- `b10x:authoring-plugins` carries five rules for writing what an agent reads
(`references/writing.md`).
- `evals/` holds 17 cases; its README recounts coverage from the `subject:` fields.

Sources: `github.com/mattpocock/skills` at `c55ee46` and `github.com/cursor/plugins` at `ecc249f`,
both MIT. Each method is adapted in this repository's words; no text is copied verbatim.

## [0.14.17] — 2026-09-27

Codex parity for the AEP agents, two planning commands, and routing that reaches every skill.
Expand Down
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/agentplugins-check", "crates/b10x"]

[workspace.package]
version = "0.14.17"
version = "0.15.0"
edition = "2021"
rust-version = "1.85"
license = "Apache-2.0"
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ What each plugin ships:
- skills: [`init`](plugins/ess/skills/init/SKILL.md) · [`upgrade`](plugins/ess/skills/upgrade/SKILL.md) · [`hardening`](plugins/ess/skills/hardening/SKILL.md) · [`retrofitting`](plugins/ess/skills/retrofitting/SKILL.md) · [`specifying`](plugins/ess/skills/specifying/SKILL.md) · [`testing-conformance`](plugins/ess/skills/testing-conformance/SKILL.md)
- agents: [`author`](plugins/ess/agents/author.md) · [`conformance`](plugins/ess/agents/conformance.md) · [`retrofitter`](plugins/ess/agents/retrofitter.md)
- [`aep`](plugins/aep/) · [docs](website/docs/plugins/aep.md)
- skills: [`init`](plugins/aep/skills/init/SKILL.md) · [`upgrade`](plugins/aep/skills/upgrade/SKILL.md) · [`decompose`](plugins/aep/skills/decompose/SKILL.md) · [`drive`](plugins/aep/skills/drive/SKILL.md) · [`implementing`](plugins/aep/skills/implementing/SKILL.md) · [`migrating`](plugins/aep/skills/migrating/SKILL.md) · [`planning`](plugins/aep/skills/planning/SKILL.md) · [`review-plan`](plugins/aep/skills/review-plan/SKILL.md) · [`wave`](plugins/aep/skills/wave/SKILL.md)
- skills: [`init`](plugins/aep/skills/init/SKILL.md) · [`upgrade`](plugins/aep/skills/upgrade/SKILL.md) · [`decompose`](plugins/aep/skills/decompose/SKILL.md) · [`diagnosing`](plugins/aep/skills/diagnosing/SKILL.md) · [`drive`](plugins/aep/skills/drive/SKILL.md) · [`implementing`](plugins/aep/skills/implementing/SKILL.md) · [`migrating`](plugins/aep/skills/migrating/SKILL.md) · [`planning`](plugins/aep/skills/planning/SKILL.md) · [`review-plan`](plugins/aep/skills/review-plan/SKILL.md) · [`wave`](plugins/aep/skills/wave/SKILL.md)
- agents: [`adversary`](plugins/aep/agents/adversary.md) · [`decomposer`](plugins/aep/agents/decomposer.md) · [`implementor`](plugins/aep/agents/implementor.md) · [`plan-critic-acceptance`](plugins/aep/agents/plan-critic-acceptance.md) · [`plan-critic-design`](plugins/aep/agents/plan-critic-design.md) · [`plan-critic-parallel-safety`](plugins/aep/agents/plan-critic-parallel-safety.md) · [`plan-critic-scope`](plugins/aep/agents/plan-critic-scope.md) · [`plan-reviewer`](plugins/aep/agents/plan-reviewer.md) · [`reverse-engineer`](plugins/aep/agents/reverse-engineer.md) · [`security-reviewer`](plugins/aep/agents/security-reviewer.md) · [`story-scoper`](plugins/aep/agents/story-scoper.md)
- [`worktree`](plugins/worktree/) · [docs](website/docs/plugins/worktree.md)
- skills: [`init`](plugins/worktree/skills/init/SKILL.md) · [`upgrade`](plugins/worktree/skills/upgrade/SKILL.md) · [`cleanup`](plugins/worktree/skills/cleanup/SKILL.md) · [`managing-worktrees`](plugins/worktree/skills/managing-worktrees/SKILL.md)
Expand Down
29 changes: 15 additions & 14 deletions crates/agentplugins-check/src/evals.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1398,20 +1398,21 @@ mod tests {
&["plugins/aep/skills/implementing/references/unit-brief.md".to_owned()],
)
.expect("the corpus scopes");
// The adversary's case, and the golden path — whose step 6 is the wave. Both are right.
assert_eq!(
matched,
vec![
(
"evals/adversary-tests-only".to_owned(),
"plugins/aep".to_owned()
),
(
"evals/golden-path-end-to-end".to_owned(),
"plugins/aep".to_owned()
),
]
);
// Every case that names `aep:implementing` — the adversary's, the wave's, the scoper's and
// the security reviewer's — and the golden path, whose step 6 is the wave. All are right.
let expected: Vec<(String, String)> = [
"evals/adversary-panel-one-family",
"evals/adversary-tautological-test",
"evals/adversary-tests-only",
"evals/golden-path-end-to-end",
"evals/security-reviewer-safety-fact",
"evals/story-scoper-safety-fact",
"evals/wave-claim-verdict",
]
.iter()
.map(|case| ((*case).to_owned(), "plugins/aep".to_owned()))
.collect();
assert_eq!(matched, expected);
}

/// An agent is a thin adapter over a role procedure its owning skill carries (R4), so a change
Expand Down
1 change: 1 addition & 0 deletions crates/agentplugins-check/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ const PLUGINS: &[(&str, &[&str])] = &[
"agents/plan-critic-parallel-safety.md",
"skills/implementing/SKILL.md",
"skills/implementing/references/drive.md",
"skills/diagnosing/SKILL.md",
"skills/wave/SKILL.md",
"skills/drive/SKILL.md",
"skills/review-plan/SKILL.md",
Expand Down
28 changes: 19 additions & 9 deletions evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ A case is four things and no others:
| the expectations | `expectations.trace.yaml` | a `trace-spec/1` document: what the run must have looked like |
| the transcript | `recorded/` | a recorded run, replayed through the checker for nothing |

## The eight cases
## The seventeen cases

Every cell in the middle column is that case's `subject:` in full, read from its `case.yaml`, and
those fields — not this table and not any prose elsewhere — are the source of truth for what the
corpus covers: counted from them the eight cases name **7 of this repository's 11 agents** and
**4 of its 8 skills**.
corpus covers: counted from them the seventeen cases name **8 of this repository's 14 agents** and
**6 of its 21 skills** (the 6 command skills are not counted).

| Case | `subject:` agents and skills | The claim it holds the subject to |
|---|---|---|
Expand All @@ -29,12 +29,22 @@ corpus covers: counted from them the eight cases name **7 of this repository's 1
| `golden-path-end-to-end` | `aep:decomposer`, `aep:plan-critic-acceptance`, `aep:story-scoper`, `aep:planning`, `aep:implementing`, `aep:implementing` (and the path `website/docs/golden-path.md`) | the eight published steps in the published order, with the CLIs as the stores' only writers |
| `adversary-tests-only` | `aep:adversary`, `aep:implementing` | tests were written, `src/` was not touched, and no `aep plan artifact` command ran |
| `connectors-readiness` | `connectors:integrating` | diagnosis uses the CLI; help is allowed and Connector mutations are rejected |

No case names the agents `aep:implementor`, `aep:plan-reviewer`, `aep:reverse-engineer` or
`aep:security-reviewer`, nor the skills `aep:migrating`, `b10x:init`, `b10x:routing` or
`b10x:authoring-plugins`; those eight are
the remaining scope of
`story:plugin-eval-cases` in `.engineering/planning`, and a change to one of them turns no row red.
| `specifying-interview-headless` | `ess:specifying` | with no operator, no question was asked, the questions became `approval-record`s, and the drafted domain was validated |
| `diagnosing-red-loop-first` | `aep:diagnosing` | a test command ran before the first write under `src/`, and the red and green runs became `test_result` evidence |
| `adversary-tautological-test` | `aep:adversary`, `aep:implementing` | the seeded test that could not fail was rewritten, and `src/` was not touched |
| `wave-claim-verdict` | `aep:implementing` | a green unit whose claim does not hold got a `verification` record and was not merged |
| `story-scoper-safety-fact` | `aep:story-scoper`, `aep:implementing` | the returned Scope section carries a `Safety fact` line with its ladder step, and nothing was written |
| `security-reviewer-safety-fact` | `aep:security-reviewer`, `aep:implementing` | a test for the safety fact was written and run, and `src/` was not touched |
| `decomposer-expand-migrate-contract` | `aep:decomposer` | a codebase-wide rename became at least four stories linked by at least three `depends_on` edges |
| `adversary-panel-one-family` | `aep:implementing`, `aep:adversary` | a panel asked for under one model family recorded one seat and said the panel was one reviewer |
| `authoring-prohibition-only-rule` | `b10x:authoring-plugins` | a wording review read the seeded skill, edited nothing, and named the prohibition with a positive target |

No case names the agents `aep:implementor`, `aep:plan-reviewer`, `aep:reverse-engineer`,
`ess:author`, `ess:conformance` or `ess:retrofitter`, nor the activity skills `aep:migrating`,
`b10x:routing`, `ess:retrofitting`, `ess:testing-conformance`, `ess:hardening` or
`worktree:managing-worktrees`, nor any `init` or `upgrade` skill; a change to one of them turns no
row red. The `aep` ones are the remaining scope of `story:plugin-eval-cases` in
`.engineering/planning`.

### Connector command checks

Expand Down
26 changes: 26 additions & 0 deletions evals/adversary-panel-one-family/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
format: eval-case/1
id: adversary-panel-one-family
title: Asked for a cross-family panel where only one family is available, the wave said so and recorded one seat
workflow: adp/default
states: [adversarial_verify]
arm: plugin
verdict: held

subject:
skills: [aep:implementing]
agents: [aep:adversary]
paths:
- plugins/aep/skills/implementing/references/adversary-panel.md

# The recorded harness serves one model family, so the decidable half of the panel is the one that
# matters most in practice: it does not pretend a second family ran. One `review-result` is written
# through the CLI, and the report states that the panel was one reviewer.

task: |
The unit on branch `unit/invoice-total` is green. The operator asks: attack it with a
cross-family adversary panel before it merges. Use the `aep:implementing` skill, in wave mode,
and record each seat's report as the skill says. No operator is present; record each stop and
continue. Do not merge, push or tag.

expectations: expectations.trace.yaml
recorded: recorded/
67 changes: 67 additions & 0 deletions evals/adversary-panel-one-family/expectations.trace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
format: trace-spec/1
id: eval-case/adversary-panel-one-family
title: One seat ran, one review-result was recorded, and the report says the panel was one reviewer

expectations:

- id: the-skill-was-offered
statement: the plugin loaded and offered the implementing skill, so a miss is not "it was not there"
expect:
env.skill_available:
skill: aep:implementing

- id: a-seat-ran
statement: the adversary ran as a subagent
expect:
subagent.spawned:
count: {at_least: 1}

- id: the-seat-was-recorded
statement: the seat's report became a `review-result` through `new --from`
expect:
tool.called:
tool: Bash
args:
command: {regex: 'aep +plan +artifact +new +review-result .*--from'}
count: {at_least: 1}

- id: the-report-says-one-family
statement: the final report states that only one model family was available, so the panel was one reviewer
expect:
text.matches:
regex: '(?i)(one (model )?family|single (model )?family|panel (of|was) one)'

- id: nothing-was-published
statement: no merge of the unit, no push and no tag
expect:
tool.absent:
tool: Bash
args:
command: {regex: 'git +(push|tag|merge)'}

- id: nothing-was-refused
statement: nothing the run tried had to be refused
expect:
permission.denied:
count: {at_most: 0}

- id: terminal-record-clean
statement: the session ended the way a finished session ends
expect:
result:
is_error: false
terminal_reason: completed

- id: turns-within-reason
statement: the panel did not loop
severity: advisory
expect:
turns:
count: {at_most: 60}

- id: cost-within-the-per-case-budget
statement: the run stayed inside the per-case cap the corpus is run under
severity: advisory
expect:
cost.total:
at_most_usd: 5.0
33 changes: 33 additions & 0 deletions evals/adversary-panel-one-family/recorded/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# `recorded/` — the adversary panel with one family

**Empty on purpose, and this file says what would fill it.** No transcript of this case has been
recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own
rows were fitted to, which measures the document and not the plugin.

`task check` skips an empty `recorded/` with a printed notice and does not fail.

## The run that would produce it

Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap:

```console
$ METAHARNESS_LIVE=1 aep drive eval run \
--case evals/adversary-panel-one-family \
--arm plugin \
--harness claude \
--plugin-dir plugins/aep \
--cwd <a checkout with the wave page, the integration branch and unit/invoice-total> \
--budget-usd 5 \
--observed-at <the date it was observed> \
--redact \
--out <a directory outside this repository>
```

Copy `<out>/<name>.events.jsonl` in with the manifest's `observed_at`, harness version and model.

## What it needs in the working tree

The `adversary-tautological-test` tree on a branch `unit/invoice-total`, an integration branch, an
AEP store holding its story in `active`, and a wave page naming the unit. The harness must serve one
model family only; a run under a harness with two would make `the-report-says-one-family` wrong for
a reason that is not about the plugin.
26 changes: 26 additions & 0 deletions evals/adversary-tautological-test/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
format: eval-case/1
id: adversary-tautological-test
title: The adversary rewrote the test that could not fail, and left the implementation alone
workflow: adp/default
states: [adversarial_verify]
arm: plugin
verdict: held

subject:
agents: [aep:adversary]
skills: [aep:implementing]
paths:
- plugins/aep/skills/implementing/references/adversary.md

# The working tree carries one seeded test, `tests/total.rs`, that asserts only
# `assert!(total(&lines).is_ok())`. It passes whatever `total` computes. The rows decide that the
# adversary edited that file and not `src/`, and that its report names the test as one that cannot
# fail. They do not read the adversary's markdown.

task: |
`story:invoice-total` was implemented in this worktree and the suite is green. Attack it with the
`aep:adversary` agent. You may add and change test files. You may not change an implementation
file. Run the suite and report its output verbatim. Record nothing in the planning store.

expectations: expectations.trace.yaml
recorded: recorded/
Loading
Loading