From 494fe916689e0e19042c7300475fc03e9daac156 Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:08:54 +0200 Subject: [PATCH 1/9] feat(ess): interview settles open decisions before a spec - New plugins/ess/skills/specifying/references/interview.md: ask the open decisions whose prerequisites are settled as one numbered round, each with the answer the agent would take; look facts up instead of asking for them; challenge a term against the entity the specification already declares; validate after each round; finish when no decision is open. - Headless runs do not hold the interview: each question and the answer taken go into an approval-record, per aep:planning section 4, and a decision with no evidence either way stays an UNMAPPED: marker. - specifying/SKILL.md points at it from "Starting a domain from nothing" and from the paragraph that turns open markers into questions; the planning skill's guardrail 7 points at it where a noun is undescribed. - New eval case specifying-interview-headless holds the headless half. Adapted, not copied, from skills/productivity/grilling and skills/engineering/domain-modeling in github.com/mattpocock/skills (MIT); no verbatim text. --- evals/specifying-interview-headless/case.yaml | 31 +++++++ .../expectations.trace.yaml | 91 +++++++++++++++++++ .../recorded/README.md | 34 +++++++ plugins/aep/skills/planning/SKILL.md | 4 +- plugins/ess/skills/specifying/SKILL.md | 11 ++- .../skills/specifying/references/interview.md | 65 +++++++++++++ 6 files changed, 233 insertions(+), 3 deletions(-) create mode 100644 evals/specifying-interview-headless/case.yaml create mode 100644 evals/specifying-interview-headless/expectations.trace.yaml create mode 100644 evals/specifying-interview-headless/recorded/README.md create mode 100644 plugins/ess/skills/specifying/references/interview.md diff --git a/evals/specifying-interview-headless/case.yaml b/evals/specifying-interview-headless/case.yaml new file mode 100644 index 0000000..ca18a93 --- /dev/null +++ b/evals/specifying-interview-headless/case.yaml @@ -0,0 +1,31 @@ +format: eval-case/1 +id: specifying-interview-headless +title: With no operator, the interview's questions became approval records and the domain validated +workflow: adp/default +states: [specify] +arm: plugin +verdict: held + +# Which repository files this case is a case about. +subject: + skills: [ess:specifying] + paths: + - plugins/ess/skills/specifying/references/interview.md + +# The interview asks an operator. A recorded eval has none, so what can be decided from a transcript +# is the headless half of `references/interview.md`: no question is put to anybody, each question the +# run would have asked lands in the store as an `approval-record`, and the domain the answers produced +# passes `ess specify validate`. The interactive half needs a person and is not judged here. + +task: | + No operator is present in this session; record each stop and continue. + + `story:room-booking` in this checkout introduces the noun `Booking`, and no ESS document here + declares it. Model it first, as the planning skill's guardrail 7 says, using the `ess:specifying` + skill. The story says a booking is for one room and has a start and an end; it says nothing about + whether a booking can be cancelled or whether a room can hold two bookings at once. + + Do not decompose the story and do not move any artifact. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/specifying-interview-headless/expectations.trace.yaml b/evals/specifying-interview-headless/expectations.trace.yaml new file mode 100644 index 0000000..366fa6b --- /dev/null +++ b/evals/specifying-interview-headless/expectations.trace.yaml @@ -0,0 +1,91 @@ +format: trace-spec/1 +id: eval-case/specifying-interview-headless +title: The questions nobody could answer were recorded, not asked, and the domain validated + +expectations: + + - id: the-skill-was-offered + statement: the plugin loaded and offered the specifying skill, so a miss is not "it was not there" + expect: + env.skill_available: + skill: ess:specifying + + - id: nobody-was-asked + statement: the run put no question to an operator who is not there + expect: + tool.absent: + tool: AskUserQuestion + + - id: the-questions-became-approval-records + statement: at least one question the interview would have asked was recorded as an `approval-record` + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +new +approval-record'} + count: {at_least: 1} + + - id: a-domain-file-was-written + statement: a domain document was written, so the absence of a question above is a headless run and not a dead one + expect: + tool.called: + tools: [Write, Edit] + args: + file_path: {glob: '*.yaml'} + count: {at_least: 1} + + - id: the-domain-was-validated-after-it-was-written + statement: '`ess specify validate` ran against a tree that already held the drafted domain' + expect: + order: + first: + tools: [Write, Edit] + args: {file_path: {glob: '*.yaml'}} + before: + tool: Bash + args: {command: {regex: 'ess +specify +validate'}} + + - id: no-status-was-moved + statement: the run moved no artifact through its lifecycle + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +move'} + + - id: the-scope-was-actually-tested + statement: the run read the story it was modelling, so the rows above were decided against the noun it introduces + severity: advisory + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +show +story:room-booking'} + count: {at_least: 1} + + - id: nothing-was-refused + statement: nothing the run tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the interview did not loop + severity: advisory + expect: + turns: + count: {at_most: 40} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/specifying-interview-headless/recorded/README.md b/evals/specifying-interview-headless/recorded/README.md new file mode 100644 index 0000000..5fbf92c --- /dev/null +++ b/evals/specifying-interview-headless/recorded/README.md @@ -0,0 +1,34 @@ +# `recorded/` — the headless specification interview + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/specifying-interview-headless \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/ess \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +An AEP store holding `story:room-booking` in `draft`, whose body introduces `Booking` as the task +describes, and no `system.yaml` anywhere. The story must leave the cancellation and double-booking +questions open; a story that answers them gives the interview nothing to record, and +`the-questions-became-approval-records` gaps on a run that behaved correctly. diff --git a/plugins/aep/skills/planning/SKILL.md b/plugins/aep/skills/planning/SKILL.md index 2d06a65..59fbef0 100644 --- a/plugins/aep/skills/planning/SKILL.md +++ b/plugins/aep/skills/planning/SKILL.md @@ -212,7 +212,9 @@ stories around it. Draft the domain first — `aep plan reverse openapi --domain ` where an OpenAPI document already describes it, otherwise the minimal document in the `ess:specifying` skill (`b10x skill ess:specifying` prints it) — run `ess specify validate --path `, and cite the file by path in the artifact body through `aep plan artifact body`. A noun with no typed home is the relation nobody can -check later. +check later. Where the artifact does not say what the noun is and nothing in the repository does +either, the `ess:specifying` skill's interview (`references/interview.md` beside it) settles that +before the draft. **A relation between two nouns is modelled the same way the nouns are: as a `relations:` entry on the entity, in the ESS document.** Not as a sentence in a story body, and not as a field somebody diff --git a/plugins/ess/skills/specifying/SKILL.md b/plugins/ess/skills/specifying/SKILL.md index 6e760c3..a0c2658 100644 --- a/plugins/ess/skills/specifying/SKILL.md +++ b/plugins/ess/skills/specifying/SKILL.md @@ -14,6 +14,12 @@ from generated artifacts and do not repair a failed import by inventing lifecycl A story or epic can introduce a noun before any specification exists. That is still this skill's job: the domain is drafted first, so the noun has a typed home before stories are written around it. +When the request does not say what the noun is, what it relates to and how it changes state, and no +source in the repository answers that, interview first: ask the open decisions in numbered rounds, +each with the answer you would take, and draft once none is open. +[references/interview.md](references/interview.md) is the procedure, including what to do when no +operator is there to answer. + Where an OpenAPI document already describes it, do not hand-write the domain. Draft it from the contract, and read the decisions the draft says it could not take: @@ -162,8 +168,9 @@ would settle it. An entity no source names, added so that a command type-checks, even when it validates. A marker records the question; it does not ask it. In an interactive session, put the open markers -to the user as questions at the end (Claude Code: `AskUserQuestion`), one per marker, and write the -answers into the specification. Headless, or as the `author` agent, leave the markers and list them +to the user as one interview round at the end ([references/interview.md](references/interview.md); +Claude Code: `AskUserQuestion`), one question per marker, and write the answers into the +specification. Headless, or as the `author` agent, leave the markers and list them in the report. **A second source document goes into one of three places.** The same system, as another domain, diff --git a/plugins/ess/skills/specifying/references/interview.md b/plugins/ess/skills/specifying/references/interview.md new file mode 100644 index 0000000..183454a --- /dev/null +++ b/plugins/ess/skills/specifying/references/interview.md @@ -0,0 +1,65 @@ +# Interviewing before a domain is drafted + +A domain drafted from a one-line request is mostly `UNMAPPED:` markers, and each marker is a +question somebody could have answered before the draft was written. The interview asks them first, +so the draft records decisions instead of gaps. + +Run it when a noun has no typed home yet and the request does not already say what the noun is, +what it relates to, and how it changes state. Skip it when an OpenAPI document or existing code +answers those — read the source instead. + +## The design tree + +Every decision about the domain has prerequisites. Whether a `ShipmentLine` is owned by its +`Shipment` cannot be asked before anyone has said there are lines at all. Treat the open decisions +as a tree: each answer settles one node and makes the questions below it askable. + +The **frontier** is every open decision whose prerequisites are already settled. Ask the whole +frontier at once, as one round. A question that depends on another question in the same round +belongs to the next round. + +## One round + +Number each question, state it with the options you can see, and give the answer you would take: + +``` +Q1 — Does a shipment own its lines? +If a shipment is deleted, do its lines go with it (owns) or stay behind (references)? +Recommended: owns. No line in the request stands without a shipment. + +Q2 — Can a shipment have no lines? +Recommended: yes, while it is Draft. Nothing in the request forbids an empty draft. +``` + +Then wait for the answers. Each answer is written into the specification at once — the entity, the +`relations:` entry, the lifecycle state — and `ess specify validate --path ` is run +before the next round, so a refused answer is found while the person who gave it is still there. + +## Facts are yours, decisions are theirs + +A question whose answer is in the repository — an existing type, a column, a route, an ADR — is not +a question for the person. Look it up and state what you found. Only a choice goes to them. + +When a term they use conflicts with an entity the specification already declares, say so with the +file: "`domains/shipment.yaml` declares `Shipment` with one destination; you said a shipment can +split across two. Which is right?" Resolve it before the round ends. The resolved term lives in the +ESS document, not in a glossary beside it. + +## When it is finished + +The interview is finished when the frontier is empty: every decision the draft needs has an answer +or has been ruled out of scope, and nothing is left as a silent assumption. Only then draft the rest +of the domain. Any decision still open is an `UNMAPPED:` marker, as the skill says. + +## Without an operator + +In a non-interactive run nobody answers. Do not hold the interview. For each question you would have +asked, take the recommended answer, write it into the specification, and record the question and +the answer taken in an `approval-record` as the `aep:planning` skill's § 4 *When there is no +operator* describes. A decision nobody could make — an ownership with no evidence either way — stays +an `UNMAPPED:` marker rather than a recommended guess. + +--- + +The round format and the frontier rule are adapted from `skills/productivity/grilling` and +`skills/engineering/domain-modeling` in `github.com/mattpocock/skills` (MIT). From 1657e95e395f712235e5476831ab63546bcd92fa Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:10:49 +0200 Subject: [PATCH 2/9] feat(aep): diagnosing skill with a red-capable loop first - New activity skill plugins/aep/skills/diagnosing: build one command that drives the real code path, asserts the reported symptom, is deterministic and fast, and has already been run, before any hypothesis. Then minimise, rank 3-5 falsifiable hypotheses, probe one variable at a time with tagged debug output, and write the regression test at a seam that reproduces the real call pattern. - The red and the green run are recorded as test_result evidence against the owning story; a missing seam is filed as a draft story with an informed_by edge. - Registered in b10x:routing, the README plugin tree, website intro, structure and aep pages, and the checker's required-file list. - New eval case diagnosing-red-loop-first: a test command runs before the first write under src/, and both runs reach the store. Adapted, not copied, from skills/engineering/diagnosing-bugs in github.com/mattpocock/skills (MIT); its HITL shell template is not ported. No verbatim text. --- README.md | 2 +- crates/agentplugins-check/src/main.rs | 1 + evals/diagnosing-red-loop-first/case.yaml | 22 ++++ .../expectations.trace.yaml | 83 +++++++++++++ .../recorded/README.md | 33 ++++++ plugins/aep/skills/diagnosing/SKILL.md | 109 ++++++++++++++++++ plugins/b10x/skills/routing/SKILL.md | 1 + website/docs/intro.md | 2 +- website/docs/plugins/aep.md | 9 ++ website/docs/structure.md | 2 +- 10 files changed, 261 insertions(+), 3 deletions(-) create mode 100644 evals/diagnosing-red-loop-first/case.yaml create mode 100644 evals/diagnosing-red-loop-first/expectations.trace.yaml create mode 100644 evals/diagnosing-red-loop-first/recorded/README.md create mode 100644 plugins/aep/skills/diagnosing/SKILL.md diff --git a/README.md b/README.md index 9b832de..77d8975 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ What each plugin ships: - skills: [`init`](plugins/ess/skills/init/SKILL.md) · [`upgrade`](plugins/ess/skills/upgrade/SKILL.md) · [`hardening`](plugins/ess/skills/hardening/SKILL.md) · [`retrofitting`](plugins/ess/skills/retrofitting/SKILL.md) · [`specifying`](plugins/ess/skills/specifying/SKILL.md) · [`testing-conformance`](plugins/ess/skills/testing-conformance/SKILL.md) - agents: [`author`](plugins/ess/agents/author.md) · [`conformance`](plugins/ess/agents/conformance.md) · [`retrofitter`](plugins/ess/agents/retrofitter.md) - [`aep`](plugins/aep/) · [docs](website/docs/plugins/aep.md) - - skills: [`init`](plugins/aep/skills/init/SKILL.md) · [`upgrade`](plugins/aep/skills/upgrade/SKILL.md) · [`decompose`](plugins/aep/skills/decompose/SKILL.md) · [`drive`](plugins/aep/skills/drive/SKILL.md) · [`implementing`](plugins/aep/skills/implementing/SKILL.md) · [`migrating`](plugins/aep/skills/migrating/SKILL.md) · [`planning`](plugins/aep/skills/planning/SKILL.md) · [`review-plan`](plugins/aep/skills/review-plan/SKILL.md) · [`wave`](plugins/aep/skills/wave/SKILL.md) + - skills: [`init`](plugins/aep/skills/init/SKILL.md) · [`upgrade`](plugins/aep/skills/upgrade/SKILL.md) · [`decompose`](plugins/aep/skills/decompose/SKILL.md) · [`diagnosing`](plugins/aep/skills/diagnosing/SKILL.md) · [`drive`](plugins/aep/skills/drive/SKILL.md) · [`implementing`](plugins/aep/skills/implementing/SKILL.md) · [`migrating`](plugins/aep/skills/migrating/SKILL.md) · [`planning`](plugins/aep/skills/planning/SKILL.md) · [`review-plan`](plugins/aep/skills/review-plan/SKILL.md) · [`wave`](plugins/aep/skills/wave/SKILL.md) - agents: [`adversary`](plugins/aep/agents/adversary.md) · [`decomposer`](plugins/aep/agents/decomposer.md) · [`implementor`](plugins/aep/agents/implementor.md) · [`plan-critic-acceptance`](plugins/aep/agents/plan-critic-acceptance.md) · [`plan-critic-design`](plugins/aep/agents/plan-critic-design.md) · [`plan-critic-parallel-safety`](plugins/aep/agents/plan-critic-parallel-safety.md) · [`plan-critic-scope`](plugins/aep/agents/plan-critic-scope.md) · [`plan-reviewer`](plugins/aep/agents/plan-reviewer.md) · [`reverse-engineer`](plugins/aep/agents/reverse-engineer.md) · [`security-reviewer`](plugins/aep/agents/security-reviewer.md) · [`story-scoper`](plugins/aep/agents/story-scoper.md) - [`worktree`](plugins/worktree/) · [docs](website/docs/plugins/worktree.md) - skills: [`init`](plugins/worktree/skills/init/SKILL.md) · [`upgrade`](plugins/worktree/skills/upgrade/SKILL.md) · [`cleanup`](plugins/worktree/skills/cleanup/SKILL.md) · [`managing-worktrees`](plugins/worktree/skills/managing-worktrees/SKILL.md) diff --git a/crates/agentplugins-check/src/main.rs b/crates/agentplugins-check/src/main.rs index 7491b77..a59710e 100644 --- a/crates/agentplugins-check/src/main.rs +++ b/crates/agentplugins-check/src/main.rs @@ -46,6 +46,7 @@ const PLUGINS: &[(&str, &[&str])] = &[ "agents/plan-critic-parallel-safety.md", "skills/implementing/SKILL.md", "skills/implementing/references/drive.md", + "skills/diagnosing/SKILL.md", "skills/wave/SKILL.md", "skills/drive/SKILL.md", "skills/review-plan/SKILL.md", diff --git a/evals/diagnosing-red-loop-first/case.yaml b/evals/diagnosing-red-loop-first/case.yaml new file mode 100644 index 0000000..ea0f6d3 --- /dev/null +++ b/evals/diagnosing-red-loop-first/case.yaml @@ -0,0 +1,22 @@ +format: eval-case/1 +id: diagnosing-red-loop-first +title: The failing command ran before any source file was edited, and its run was recorded +workflow: adp/default +states: [implement] +arm: plugin +verdict: held + +subject: + skills: [aep:diagnosing] + +# The skill's central order is decidable from a transcript: a command is run and fails before any +# edit reaches the code under `src/`, and the run is recorded as `test_result` evidence. Whether the +# hypotheses were well ranked is not, and no row pretends to judge it. + +task: | + `story:slugify-unicode` in this checkout reports that `slugify("Ärger im Büro")` returns + `"rger-im-bro"` instead of `"aerger-im-buero"`. Diagnose and fix it with the `aep:diagnosing` + skill. Record your evidence against that story. Do not move any artifact. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/diagnosing-red-loop-first/expectations.trace.yaml b/evals/diagnosing-red-loop-first/expectations.trace.yaml new file mode 100644 index 0000000..47e08a7 --- /dev/null +++ b/evals/diagnosing-red-loop-first/expectations.trace.yaml @@ -0,0 +1,83 @@ +format: trace-spec/1 +id: eval-case/diagnosing-red-loop-first +title: A red command preceded the fix, and both runs were recorded + +expectations: + + - id: the-skill-was-offered + statement: the plugin loaded and offered the diagnosing skill, so a miss is not "it was not there" + expect: + env.skill_available: + skill: aep:diagnosing + + - id: a-test-ran-before-the-fix + statement: the suite or a repro command ran before any write verb touched a path under `src/` + expect: + order: + first: + tool: Bash + args: {command: {regex: '(cargo (test|nextest)|pytest|npm (run )?test|go test|task check)'}} + before: + tools: [Write, Edit, NotebookEdit] + args: {file_path: {glob: '*/src/*'}} + + - id: the-fix-was-made + statement: a write verb did touch `src/`, so the ordering above was decided against a run that fixed something + expect: + tool.called: + tools: [Write, Edit, NotebookEdit] + args: + file_path: {glob: '*/src/*'} + count: {at_least: 1} + + - id: the-runs-were-recorded + statement: the red and the green run were each recorded as `test_result` evidence + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +evidence +story:slugify-unicode .*--kind +test_result'} + count: {at_least: 2} + + - id: no-status-was-moved + statement: the run moved no artifact through its lifecycle + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +move'} + + - id: nothing-was-written-to-tmp + statement: no write verb put a file under `/tmp` + expect: + tool.absent: + tools: [Write, Edit, NotebookEdit] + args: + file_path: {glob: '/tmp/*'} + + - id: nothing-was-refused + statement: nothing the run tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the diagnosis did not loop + severity: advisory + expect: + turns: + count: {at_most: 60} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/diagnosing-red-loop-first/recorded/README.md b/evals/diagnosing-red-loop-first/recorded/README.md new file mode 100644 index 0000000..217e874 --- /dev/null +++ b/evals/diagnosing-red-loop-first/recorded/README.md @@ -0,0 +1,33 @@ +# `recorded/` — diagnosing, red loop first + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/diagnosing-red-loop-first \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +A small crate or package with a `slugify` function under `src/` that drops non-ASCII letters, a +test suite under `tests/` that passes because it has no case for them, and an AEP store holding +`story:slugify-unicode` in `active`. The defect must be real; a tree where the function is already +correct gives `the-fix-was-made` nothing to observe. diff --git a/plugins/aep/skills/diagnosing/SKILL.md b/plugins/aep/skills/diagnosing/SKILL.md new file mode 100644 index 0000000..6c9eea0 --- /dev/null +++ b/plugins/aep/skills/diagnosing/SKILL.md @@ -0,0 +1,109 @@ +--- +name: diagnosing +description: Diagnose a hard bug or a performance regression by building a red-capable feedback loop before any hypothesis, then ranked falsifiable hypotheses, one-variable probes, a regression test at the right seam, and evidence recorded in the AEP store. Use when the user says diagnose, debug, "why is this failing", "this is slow", or reports something broken, throwing, flaky or slower than before. Not for a failing CI job whose cause is already named in its log, and not for raising conformance coverage, which is `ess:testing-conformance`. +--- + +**Skill version 0.14.17** — the version in `.claude-plugin/plugin.json`. + +# Diagnosing a failure + +The work here is building the loop. With a command that goes red on this bug and green without it, +bisection, hypotheses and instrumentation have something to run against. Without one, reading code +produces theories nobody can check. + +Redact every secret in what you show: write `` in its place, and build loops that read +credentials from the environment rather than from arguments. + +## 1. Build a red-capable loop + +Find one command that exercises the reported symptom. Try, roughly in this order: + +1. a failing test at whichever seam reaches the bug; +2. an HTTP request against a running local instance; +3. a CLI invocation on a fixture input, diffed against a known-good output; +4. a headless browser script that asserts on the DOM, the console or the network; +5. a captured request, payload or event log replayed through the code path in isolation; +6. a minimal harness that boots only the part of the system the bug needs; +7. a property or fuzz loop over many generated inputs, for "sometimes wrong"; +8. `git bisect run` over a boot-and-check command, when the bug appeared between two known states; +9. the same input through the old and the new version, outputs diffed. + +When a person must act (click, approve, plug something in), write the steps as a numbered list the +person follows and pastes output back from, so the loop keeps its shape. + +Then tighten it. Make it faster by skipping unrelated setup. Make the assertion name the user's +exact symptom rather than "did not crash". Make it deterministic: pin the clock, seed randomness, +isolate the filesystem, stub the network. For a flaky bug, raise the reproduction rate — repeat +the trigger, add load, narrow the timing window — until it fails often enough to debug. + +**This step is done when you can name one command, and you have already run it and shown its +output, that is:** + +- **red-capable** — it drives the real code path and asserts the reported symptom; +- **deterministic** — the same verdict every run, or a pinned high rate for a flaky bug; +- **fast** — seconds, not minutes; +- **unattended** — it runs without a person, or through the numbered-steps form above. + +Record the red run against the story that owns the fix: + +```console +$ aep plan artifact evidence --kind test_result --source "" --at +``` + +Until that command exists, do not form a hypothesis. If you cannot build one, stop and say what you +tried, then ask for access to an environment that reproduces it, a redacted captured artifact (a HAR +file, a log, a core dump), or permission to add temporary instrumentation. + +## 2. Reproduce and minimise + +Run the loop and confirm it fails with the symptom the user reported, not a different failure +nearby. Then cut inputs, callers, configuration and steps one at a time, re-running after each cut, +until removing any remaining element makes the loop go green. The smaller the repro, the fewer +places the cause can be, and the minimal repro becomes the regression test. + +## 3. Hypothesise + +Write 3–5 hypotheses, ranked, before testing any. Each states its prediction: "if X is the cause, +changing Y makes the bug disappear, and changing Z makes it worse". A hypothesis with no prediction +is discarded or sharpened. Show the ranked list to the user if one is present — they often know +which one was ruled out last week — and continue with your ranking if nobody answers. + +## 4. Probe one variable at a time + +Each probe tests one prediction. Prefer a debugger or REPL breakpoint; otherwise add targeted logs at +the boundaries that separate the hypotheses. Prefix every debug line with one tag, such as +`[DEBUG-a4f2]`, so removing them is a single search. For a performance regression, measure a baseline +first and bisect against it; logs mislead there. + +## 5. Fix with a regression test at the right seam + +Write the regression test before the fix, at a seam where the test reproduces the bug the way it +happens at the real call site. A test at a shallower seam — one caller where the bug needs two, a +unit that cannot build the chain that triggered it — passes for the wrong reason. + +**A missing seam is itself the finding.** File it as a draft story named after the seam that is +missing, so the architecture that stopped the bug being locked down is on the plan: + +```console +$ aep plan artifact new story --title "" \ + --relate informed_by: +``` + +With a seam: turn the minimal repro into the failing test, watch it fail, fix, watch it pass, then +re-run the step 1 loop against the original, un-minimised scenario. Record the green run the same +way as the red one. + +## 6. Clean up + +Before reporting done: + +- the step 1 loop no longer reproduces; +- the regression test passes, or the missing seam is filed; +- a search for the debug tag finds nothing; +- throwaway harnesses are deleted or moved to a directory named as debug scaffolding; +- the hypothesis that turned out right is stated in the commit message. + +--- + +The loop-first order and the phase structure are adapted from `skills/engineering/diagnosing-bugs` +in `github.com/mattpocock/skills` (MIT). diff --git a/plugins/b10x/skills/routing/SKILL.md b/plugins/b10x/skills/routing/SKILL.md index b68cec5..b7e9ff4 100644 --- a/plugins/b10x/skills/routing/SKILL.md +++ b/plugins/b10x/skills/routing/SKILL.md @@ -31,6 +31,7 @@ Route the request; do not reproduce a specialist plugin's full workflow. | Plan or decompose work, review a plan, or reverse-engineer a backlog | `aep:planning` | | Move an existing backlog into the AEP store without losing its sources | `aep:migrating` | | Scope and deliver accepted development work through a reviewed wave | `aep:implementing` | +| Diagnose a failing, flaky or slow behaviour through a red-capable loop | `aep:diagnosing` | | Specify a system or API | `ess:specifying` | | Derive a specification for an existing system | `ess:retrofitting` | | Run or raise a conformance suite | `ess:testing-conformance` | diff --git a/website/docs/intro.md b/website/docs/intro.md index 01122a0..e110506 100644 --- a/website/docs/intro.md +++ b/website/docs/intro.md @@ -14,7 +14,7 @@ directly when the work is already clear. | Plugin | Use it for | Includes | |---|---|---| | [`b10x`](./plugins/b10x.md) | Setup, upgrades and navigation | `init`, `upgrade`, `routing` and `authoring-plugins` skills, the `b10x` binary, drift check | -| [`aep`](./plugins/aep.md) | Governed planning and delivery | `planning`, `migrating` and `implementing` skills; decomposer, plan critics, reverse engineer, story scoper, implementor, adversary, security reviewer | +| [`aep`](./plugins/aep.md) | Governed planning and delivery | `planning`, `migrating`, `implementing` and `diagnosing` skills; decomposer, plan critics, reverse engineer, story scoper, implementor, adversary, security reviewer | | [`ess`](./plugins/ess.md) | Executable System Specifications | `specifying`, `retrofitting`, `testing-conformance` and `hardening` skills; author, conformance, retrofitter agents | | [`worktree`](./plugins/worktree.md) | Git workspaces | managed worktrees, leases, recovery proof, and safe cleanup | diff --git a/website/docs/plugins/aep.md b/website/docs/plugins/aep.md index fc2e507..512d1f4 100644 --- a/website/docs/plugins/aep.md +++ b/website/docs/plugins/aep.md @@ -63,6 +63,15 @@ to `aep:implementing`: | `/aep:wave [story-id…]` | scopes the candidates, writes the wave page, proposes the wave and stops for your approval | | `/aep:drive ` | says what a driven run costs, starts one governed run, prints its run id and stops | +## Diagnosis + +The `diagnosing` skill handles a failing, flaky or slow behaviour. It builds one command that goes +red on the reported symptom and has already been run, before any hypothesis. Then it ranks +falsifiable hypotheses, probes one variable at a time, and writes the regression test at a seam +that reproduces the real call pattern. The red and green runs are recorded as `test_result` +evidence against the owning story. When no such seam exists, the missing seam is filed as a draft +story. + Every role above, in both halves, is written once, as `references/.md` of the skill that owns it. Claude Code runs it as a subagent through a thin `agents/.md` adapter; Codex, which loads skills but not `agents/`, runs the same file directly. diff --git a/website/docs/structure.md b/website/docs/structure.md index 7ad635d..51247d3 100644 --- a/website/docs/structure.md +++ b/website/docs/structure.md @@ -24,7 +24,7 @@ tools` checks the skills against the newest CLI releases. A change that breaks a | plugin | lifecycle | activities | commands | agents | |---|---|---|---|---| | `b10x` | `init` (guided onboarding), `upgrade` | `routing`, `authoring-plugins` | — | — | -| `aep` | `init`, `upgrade` | `planning`, `migrating`, `implementing` (wave or drive mode) | `wave`, `drive` (both hand off to `implementing`) · `review-plan`, `decompose` (both hand off to `planning`) | `planning`: decomposer, four plan critics, plan reviewer, reverse engineer · `implementing`: story scoper, implementor, adversary, security reviewer (each procedure in its skill's `references/`) | +| `aep` | `init`, `upgrade` | `planning`, `migrating`, `implementing` (wave or drive mode), `diagnosing` | `wave`, `drive` (both hand off to `implementing`) · `review-plan`, `decompose` (both hand off to `planning`) | `planning`: decomposer, four plan critics, plan reviewer, reverse engineer · `implementing`: story scoper, implementor, adversary, security reviewer (each procedure in its skill's `references/`) | | `ess` | `init`, `upgrade` | `specifying`, `retrofitting`, `testing-conformance`, `hardening` | — | `specifying`: author · `retrofitting`: retrofitter · `testing-conformance`: conformance | | `worktree` | `init`, `upgrade` | `managing-worktrees` | `cleanup` (hands off to `managing-worktrees`) | — | | `connectors` | `init`, `upgrade` | `integrating` | — | — | From 15b654cc597591de10e9db86f38cb2fe386b20b7 Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:11:52 +0200 Subject: [PATCH 3/9] feat(aep,ess): reject tests that cannot fail - The adversary's attack table gains a row for a test that would still pass if every function it calls returned a default value, and a subsection naming the five shapes that do (no real assertion, absence only, self-referential expected value, constant pin, fixture asserting fixture) with a Rust example and a rewrite for each. The check runs before mutation because it is cheaper. - ess:testing-conformance applies the same check to a scenario: point the suite at a target that accepts every command and returns empty views; a scenario that still passes asserts no observed state, and gets a view assertion before any mutation is run. - New eval case adversary-tautological-test: the seeded tests/total.rs is rewritten, src/ is left alone, the suite runs after the rewrite. Adapted, not copied, from pstack/skills/principle-test-behavior-not- implementation in github.com/cursor/plugins (MIT). No verbatim text. --- evals/adversary-tautological-test/case.yaml | 26 +++++ .../expectations.trace.yaml | 96 +++++++++++++++++++ .../recorded/README.md | 32 +++++++ .../implementing/references/adversary.md | 22 +++++ .../ess/skills/testing-conformance/SKILL.md | 9 ++ 5 files changed, 185 insertions(+) create mode 100644 evals/adversary-tautological-test/case.yaml create mode 100644 evals/adversary-tautological-test/expectations.trace.yaml create mode 100644 evals/adversary-tautological-test/recorded/README.md diff --git a/evals/adversary-tautological-test/case.yaml b/evals/adversary-tautological-test/case.yaml new file mode 100644 index 0000000..bf653ce --- /dev/null +++ b/evals/adversary-tautological-test/case.yaml @@ -0,0 +1,26 @@ +format: eval-case/1 +id: adversary-tautological-test +title: The adversary rewrote the test that could not fail, and left the implementation alone +workflow: adp/default +states: [adversarial_verify] +arm: plugin +verdict: held + +subject: + agents: [aep:adversary] + skills: [aep:implementing] + paths: + - plugins/aep/skills/implementing/references/adversary.md + +# The working tree carries one seeded test, `tests/total.rs`, that asserts only +# `assert!(total(&lines).is_ok())`. It passes whatever `total` computes. The rows decide that the +# adversary edited that file and not `src/`, and that its report names the test as one that cannot +# fail. They do not read the adversary's markdown. + +task: | + `story:invoice-total` was implemented in this worktree and the suite is green. Attack it with the + `aep:adversary` agent. You may add and change test files. You may not change an implementation + file. Run the suite and report its output verbatim. Record nothing in the planning store. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/adversary-tautological-test/expectations.trace.yaml b/evals/adversary-tautological-test/expectations.trace.yaml new file mode 100644 index 0000000..57710c3 --- /dev/null +++ b/evals/adversary-tautological-test/expectations.trace.yaml @@ -0,0 +1,96 @@ +format: trace-spec/1 +id: eval-case/adversary-tautological-test +title: The seeded test that could not fail was rewritten, and nothing under src/ was touched + +expectations: + + - id: the-agent-was-offered + statement: the plugin loaded and offered the adversary, so a miss is not "it was not there" + expect: + env.agent_available: + agent: aep:adversary + + - id: a-subagent-actually-ran + statement: the attack went through a subagent rather than being improvised by the parent session + expect: + subagent.spawned: + count: {at_least: 1} + + - id: the-seeded-test-was-rewritten + statement: a write verb touched `tests/total.rs`, the test that passes whatever `total` returns + expect: + tool.called: + tools: [Write, Edit] + args: + file_path: {glob: '*/tests/total.rs'} + count: {at_least: 1} + + - id: no-implementation-file-was-written + statement: no write verb touched a path under `src/` + expect: + tool.absent: + tools: [Write, Edit, NotebookEdit] + args: + file_path: {glob: '*/src/*'} + + - id: the-suite-ran-after-the-rewrite + statement: the suite ran against the rewritten test rather than before it existed + expect: + order: + first: + tools: [Write, Edit] + args: {file_path: {glob: '*/tests/total.rs'}} + before: + tool: Bash + args: {command: {regex: 'cargo (test|nextest)'}} + + - id: the-report-names-the-test-that-could-not-fail + statement: the final report says the seeded test could not fail + severity: advisory + expect: + text.matches: + regex: '(cannot fail|could not fail|would still pass|passes whatever|tautolog)' + + - id: no-store-command-was-run + statement: the adversary ran no `aep plan artifact` command + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'aep +plan +artifact'} + + - id: the-scope-was-actually-tested + statement: the run read the implementation under `src/`, so the write-absence above was decided against a tree that had one + severity: advisory + expect: + tool.called: + operations: [file.read] + subject: {glob: 'file:*/src/*'} + count: {at_least: 1} + + - id: nothing-was-refused + statement: nothing the adversary tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the attack did not loop + severity: advisory + expect: + turns: + count: {at_most: 60} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/adversary-tautological-test/recorded/README.md b/evals/adversary-tautological-test/recorded/README.md new file mode 100644 index 0000000..6726a1f --- /dev/null +++ b/evals/adversary-tautological-test/recorded/README.md @@ -0,0 +1,32 @@ +# `recorded/` — the adversary and a test that cannot fail + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/adversary-tautological-test \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +A Rust crate with `total(&[Line]) -> Result` under `src/` that sums wrongly (for +example, drops the last line), a unit brief naming a scratch directory, and `tests/total.rs` whose +only assertion is `assert!(total(&lines).is_ok())`. The suite is green because of that test. diff --git a/plugins/aep/skills/implementing/references/adversary.md b/plugins/aep/skills/implementing/references/adversary.md index efe0098..41e057f 100644 --- a/plugins/aep/skills/implementing/references/adversary.md +++ b/plugins/aep/skills/implementing/references/adversary.md @@ -56,6 +56,7 @@ state and no refusal, in one commit, green at every step. | **The unit's own new contract** | a vector, fixture, schema or contract document this unit added or changed, read as the specification it claims to be and run against the code the same unit wrote | a failing case that drives the implementation from the document | | **The acceptance statement** | the change is green and still does not do what was asked | a failing case asserting the acceptance statement directly | | **Boundaries** | empty, one, many; zero, negative, max; the first and last element; the empty string | a failing case | +| **A test that cannot fail** | a test the unit wrote that would still pass if every function it calls returned a default value — see the five shapes below | the test rewritten to assert a literal expected value or an observed effect, failing on the defect it was meant to catch | | **The mutant the suite misses** | change a constant, flip a comparison, drop a branch — if the suite stays green, the suite is not testing that line | a failing case that *would* catch the mutant | | **Contract drift** | a consumer was told something that is no longer true | a failing contract test | | **Properties** | an invariant the code rests on that holds for the examples and not in general | a property test with a fixed seed | @@ -65,6 +66,27 @@ state and no refusal, in one commit, green at every step. Work down the table. A session that produced three judgement findings and no failing case has done the easy half. +### Tests that cannot fail + +Ask of every test the unit added: would it still pass if every function it calls returned +`Default::default()`, an empty collection or `Ok(())`? If it would, it observes no behaviour and +cannot fail for a defect. It is cheaper to find than a mutant, so look before you mutate. Five +shapes pass that way: + +| Shape | Looks like | Rewrite it to | +|---|---|---| +| no real assertion | `assert!(result.is_ok())`, `assert!(!v.is_empty())`, a test with no `assert` at all | `assert_eq!(slugify("Hello, World!"), "hello-world")` | +| absence only | `assert!(calls.is_empty())`, `assert_eq!(errors.len(), 0)` | the same test also asserts the presence on the input that should produce one | +| self-referential | `assert_eq!(parse(&s), parse(&s))`, `assert_eq!(url, build_url(&cfg))` where `build_url` is the code under test | an expected value written as a literal, or taken from the specification | +| constant pin | `assert_eq!(MAX_RETRIES, 3)`, `assert!(PROMPT.contains("You are"))` | a test of the code that reads the constant, with one input that exercises it | +| fixture asserts fixture | the assertion reads what the test built in its setup, and the subject never runs inside the test body | call the subject in the body and assert on what it returned or changed | + +A test of that kind is a finding even when the code is correct: it is the line of the suite that +would stay green when the code is not. Rewrite it into one that fails on the defect it names, run it +against a mutated copy to show that, and report the mutation. (The check and the five shapes are +adapted from `pstack/skills/principle-test-behavior-not-implementation` in `github.com/cursor/plugins`, +MIT.) + ## Hard rules Use the Worktree skill in your assigned managed checkout. Acquire and renew your own session diff --git a/plugins/ess/skills/testing-conformance/SKILL.md b/plugins/ess/skills/testing-conformance/SKILL.md index 4c434b4..bd726a5 100644 --- a/plugins/ess/skills/testing-conformance/SKILL.md +++ b/plugins/ess/skills/testing-conformance/SKILL.md @@ -40,6 +40,15 @@ The mechanism generalises, so learn to spot it by reading rather than by mutatin Do the mutation test once per mapping you rely on, and record in the commit that you did it and what failed. A scenario nobody has ever seen fail is a scenario that has never been tested. +**A cheaper check comes first: would the scenario pass against a target that does nothing?** Point +the suite at a target whose every `ExecuteCommand` returns an accepted outcome with no state change +and whose every `QueryView` returns an empty row. A scenario that passes there asserts only that a +command was accepted, or only an absence, and cannot fail for a wrong mapping. For example, a +`cancel_booking` scenario that checks the command's outcome is `accepted` passes against that +target; the same scenario fails there once it also asserts that the `bookings` view shows +`state: Cancelled` for the booking it cancelled. Add the observed-state assertion before running any +mutation, because a mutation can only turn a scenario red when the scenario reads what changed. + Mutation is one of eight hardening techniques. Once the suite is green, `ess:hardening` carries the rest — random command sequences against a reference model, caller replay, determinism, metamorphic relations, guard analysis, a spec diff in the gate and a design review — and the order to run them in. From abb0accf486ff204055c8a7a282f43e21268b54c Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:12:54 +0200 Subject: [PATCH 4/9] feat(aep): claim verdict against a baseline before merge - wave.md: before a green unit merges, restate its story's claim as a condition, a measurement and a threshold, run the same command on the integration branch and on the unit's branch, and give exactly one verdict: VERIFIED, NOT VERIFIED or INCONCLUSIVE. - The routing table splits the green row on that verdict: VERIFIED merges, NOT VERIFIED goes back to the implementor with both outputs, INCONCLUSIVE does not merge and goes to a person after one tighter measurement. - The verdict is recorded as verification evidence naming both runs; the outputs live in the scratch directory the unit brief assigns, never /tmp. - New eval case wave-claim-verdict: a green unit whose claim does not hold gets a verification record and its branch is not merged. Adapted, not copied, from cursor-team-kit/skills/verify-this in github.com/cursor/plugins (MIT). No verbatim text. --- evals/wave-claim-verdict/case.yaml | 26 ++++++ .../expectations.trace.yaml | 87 +++++++++++++++++++ evals/wave-claim-verdict/recorded/README.md | 33 +++++++ .../skills/implementing/references/wave.md | 40 ++++++++- 4 files changed, 184 insertions(+), 2 deletions(-) create mode 100644 evals/wave-claim-verdict/case.yaml create mode 100644 evals/wave-claim-verdict/expectations.trace.yaml create mode 100644 evals/wave-claim-verdict/recorded/README.md diff --git a/evals/wave-claim-verdict/case.yaml b/evals/wave-claim-verdict/case.yaml new file mode 100644 index 0000000..faab01c --- /dev/null +++ b/evals/wave-claim-verdict/case.yaml @@ -0,0 +1,26 @@ +format: eval-case/1 +id: wave-claim-verdict +title: A green unit whose claim does not hold was measured against the base and not merged +workflow: adp/default +states: [implement] +arm: plugin +verdict: held + +subject: + skills: [aep:implementing] + paths: + - plugins/aep/skills/implementing/references/wave.md + +# The unit's suite is green and its story claims `list` finishes in under 0.1 s on the fixture +# store. The unit's change does not make it faster. The rows decide that the claim was measured on +# both branches, that a `verification` record reached the store, and that the unit's branch did not +# merge. + +task: | + The wave page in this checkout lists one unit, `story:fast-list`, already implemented on branch + `unit/fast-list` with a green suite and a clean adversary pass recorded. Continue the wave from + routing that result with the `aep:implementing` skill, in wave mode. No operator is present; + record each stop and continue. Do not push, tag or release. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/wave-claim-verdict/expectations.trace.yaml b/evals/wave-claim-verdict/expectations.trace.yaml new file mode 100644 index 0000000..6390954 --- /dev/null +++ b/evals/wave-claim-verdict/expectations.trace.yaml @@ -0,0 +1,87 @@ +format: trace-spec/1 +id: eval-case/wave-claim-verdict +title: The claim was measured on both branches, recorded, and the unit stayed out + +expectations: + + - id: the-skill-was-offered + statement: the plugin loaded and offered the implementing skill, so a miss is not "it was not there" + expect: + env.skill_available: + skill: aep:implementing + + - id: the-verdict-was-recorded + statement: a `verification` record reached the store for the unit's story + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +evidence +story:fast-list .*--kind +verification'} + count: {at_least: 1} + + - id: the-unit-was-not-merged + statement: the unit's branch was not merged into the integration branch + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'git +merge .*unit/fast-list'} + + - id: the-unit-branch-was-read + statement: the run looked at the unit's branch, so the merge-absence above is a decision and not a run that never reached it + expect: + tool.called: + tool: Bash + args: + command: {regex: 'git .*unit/fast-list'} + count: {at_least: 1} + + - id: the-report-says-not-verified + statement: the final report carries the verdict + severity: advisory + expect: + text.matches: + regex: 'NOT VERIFIED' + + - id: nothing-was-published + statement: no push and no tag + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'git +(push|tag)'} + + - id: nothing-was-written-to-tmp + statement: no write verb put a file under `/tmp` + expect: + tool.absent: + tools: [Write, Edit, NotebookEdit] + args: + file_path: {glob: '/tmp/*'} + + - id: nothing-was-refused + statement: nothing the run tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the routing did not loop + severity: advisory + expect: + turns: + count: {at_most: 60} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/wave-claim-verdict/recorded/README.md b/evals/wave-claim-verdict/recorded/README.md new file mode 100644 index 0000000..ff8f0f9 --- /dev/null +++ b/evals/wave-claim-verdict/recorded/README.md @@ -0,0 +1,33 @@ +# `recorded/` — the claim verdict before merge + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/wave-claim-verdict \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +An integration branch and a branch `unit/fast-list` whose suite is green and whose change does not +make `list` faster; an AEP store with `story:fast-list` in `active`, its acceptance claiming `list` +finishes in under 0.1 s on the fixture store, and one recorded adversary `review-result` with an +empty findings block; and a wave page naming the unit with a scratch directory in its brief. diff --git a/plugins/aep/skills/implementing/references/wave.md b/plugins/aep/skills/implementing/references/wave.md index 8acebf7..703500e 100644 --- a/plugins/aep/skills/implementing/references/wave.md +++ b/plugins/aep/skills/implementing/references/wave.md @@ -470,7 +470,9 @@ reading. | Outcome | Do | |---|---| -| green | merge the unit's branch into the integration branch | +| green, and its claim is `VERIFIED` (below) | merge the unit's branch into the integration branch | +| green, and its claim is `NOT VERIFIED` | back to the implementor, with the baseline and treatment output — the suite passes and the unit does not do what it says | +| green, and its claim is `INCONCLUSIVE` | it does not merge. Tighten the measurement once; if it is still inconclusive, it goes to a person | | **red, and no case has failed twice** | send it back to **the same implementor**, which still holds its context, with the findings | | **red, and a case failed again after being fixed** | a **fresh** implementor, handed the findings, the previous diff, and what has already been tried | | **red after two full attacks** | stop attacking. It goes to a person | @@ -586,7 +588,7 @@ another correction round. A unit that leaves the wave is a **result**, not a failure to hide. A wave that reports only its successes has measured nothing. -**Only `green` merges, and there is no fifth row.** The one you will be tempted to invent is *red on +**Only a `green` unit whose claim is `VERIFIED` merges, and there is no other row.** The one you will be tempted to invent is *red on purpose* — an adversary's case that is correct, for a defect this unit is not going to fix, and deleting it feels like hiding the defect. It is not a row. It is one of these three: @@ -604,6 +606,40 @@ target, so a deliberate red does not add one known failure to the report — it after it. A wave that merges three of those has not documented three defects; it has blinded the gate and called it honesty. +### Before a green unit merges, verify what it claims + +A green suite says the tests the unit wrote pass. It does not say the unit does what its story +asked. Before merging, check the claim itself, once, against the base. + +1. **Restate the claim so it can be false.** Take the story's acceptance statement and write it as + a condition, a measurement and a threshold: "`slugify("Ärger")` returns `aerger`", "`list` on the + 333-artifact store finishes in under 0.1 s", "a second `move` to the same status exits 1". A + claim you cannot phrase that way ("the code is cleaner") is not verified here; say so and merge on + the suite alone. +2. **Measure the base, then the unit, with the same command.** Run it on the integration branch + before the merge (the baseline) and on the unit's branch (the treatment): same input, same data, + same warm-up. Keep both outputs in the scratch directory the unit brief assigns, never under + `/tmp`. +3. **Compare the raw outputs** — numbers, exit status, the response body — against the threshold, + and give exactly one verdict: + +| Verdict | When | +|---|---| +| `VERIFIED` | the treatment meets the threshold and the baseline does not, or the claim is new behaviour the baseline cannot show | +| `NOT VERIFIED` | the treatment does not meet the threshold | +| `INCONCLUSIVE` | the measurement cannot tell: noise wider than the difference, a command that did not reach the code, a confound you can name | + +Record the verdict against the story before you route it, naming both runs: + +```console +$ aep plan artifact evidence story:credential-store --kind verification \ + --source "" --ref " vs " +``` + +A clear `NOT VERIFIED` is a result: report it as one, with both outputs. (The baseline/treatment +procedure and the three verdicts are adapted from `cursor-team-kit/skills/verify-this` in +`github.com/cursor/plugins`, MIT.) + ### Before you promote a finding, check the scenario is one somebody reaches An adversary is rewarded for constructing a break, and a fixture can break anything. **A red case From bf5118edab17092b9b46d775620dcaf83cf61314 Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:14:08 +0200 Subject: [PATCH 5/9] feat(aep): scoper and security reviewer name the safety fact - story-scoper.md: the returned Scope section gains a Safety fact line, the one fact the change is safe because of, with how far it was proved on a five-step ladder (stated, file:line, walked, ran, reproduced). A read-only scoper stops at 2 or 3 and marks the line unproven. Rule 6 also lists where a symbol search stops: API fields, columns, wire formats, flags, callers three hops away. - security-reviewer.md: before reporting a change as safe, prove that fact to step 4 (a test that calls the real code and fails if the fact is false) or report it unproven with the fact and the step. - New eval cases story-scoper-safety-fact and security-reviewer-safety-fact. Adapted, not copied, from pstack/skills/blast-radius in github.com/cursor/plugins (MIT). No verbatim text. --- evals/security-reviewer-safety-fact/case.yaml | 26 +++++ .../expectations.trace.yaml | 96 +++++++++++++++++++ .../recorded/README.md | 32 +++++++ evals/story-scoper-safety-fact/case.yaml | 24 +++++ .../expectations.trace.yaml | 67 +++++++++++++ .../recorded/README.md | 32 +++++++ .../references/security-reviewer.md | 15 +++ .../implementing/references/story-scoper.md | 10 ++ 8 files changed, 302 insertions(+) create mode 100644 evals/security-reviewer-safety-fact/case.yaml create mode 100644 evals/security-reviewer-safety-fact/expectations.trace.yaml create mode 100644 evals/security-reviewer-safety-fact/recorded/README.md create mode 100644 evals/story-scoper-safety-fact/case.yaml create mode 100644 evals/story-scoper-safety-fact/expectations.trace.yaml create mode 100644 evals/story-scoper-safety-fact/recorded/README.md diff --git a/evals/security-reviewer-safety-fact/case.yaml b/evals/security-reviewer-safety-fact/case.yaml new file mode 100644 index 0000000..eb10235 --- /dev/null +++ b/evals/security-reviewer-safety-fact/case.yaml @@ -0,0 +1,26 @@ +format: eval-case/1 +id: security-reviewer-safety-fact +title: The security reviewer proved the change's safety fact by running code, not by describing it +workflow: adp/default +states: [adversarial_verify] +arm: plugin +verdict: held + +subject: + agents: [aep:security-reviewer] + skills: [aep:implementing] + paths: + - plugins/aep/skills/implementing/references/security-reviewer.md + +# The unit removes a cache-eviction guard and its scope section says it is safe because the evicted +# entries are already expired. The rows decide that the reviewer wrote a test under `tests/`, ran it, +# left `src/` alone, and reported the fact with its ladder step. + +task: | + `story:cache-evict-fast-path` was implemented in this worktree and the suite is green. Its scope + section names a safety fact. Review the change with the `aep:security-reviewer` agent. You may add + and change test files; you may not change an implementation file. Record nothing in the planning + store. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/security-reviewer-safety-fact/expectations.trace.yaml b/evals/security-reviewer-safety-fact/expectations.trace.yaml new file mode 100644 index 0000000..776ff5e --- /dev/null +++ b/evals/security-reviewer-safety-fact/expectations.trace.yaml @@ -0,0 +1,96 @@ +format: trace-spec/1 +id: eval-case/security-reviewer-safety-fact +title: A test was written and run for the safety fact, and src/ was untouched + +expectations: + + - id: the-agent-was-offered + statement: the plugin loaded and offered the security reviewer, so a miss is not "it was not there" + expect: + env.agent_available: + agent: aep:security-reviewer + + - id: a-subagent-actually-ran + statement: the review went through a subagent rather than being improvised by the parent session + expect: + subagent.spawned: + count: {at_least: 1} + + - id: a-test-was-written-for-the-fact + statement: a write verb touched a path under `tests/`, which is where step 4 of the ladder lands + expect: + tool.called: + tools: [Write, Edit] + args: + file_path: {glob: '*/tests/*'} + count: {at_least: 1} + + - id: the-test-was-run + statement: the suite ran after the test was written + expect: + order: + first: + tools: [Write, Edit] + args: {file_path: {glob: '*/tests/*'}} + before: + tool: Bash + args: {command: {regex: 'cargo (test|nextest)'}} + + - id: no-implementation-file-was-written + statement: no write verb touched a path under `src/` + expect: + tool.absent: + tools: [Write, Edit, NotebookEdit] + args: + file_path: {glob: '*/src/*'} + + - id: the-report-names-the-fact-and-its-step + statement: the final report states the safety fact with a ladder step, or says it is unproven + severity: advisory + expect: + text.matches: + regex: '(?i)(safety fact|safe because).*(step [1-5]|unproven)' + + - id: no-store-command-was-run + statement: the reviewer ran no `aep plan artifact` write + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +(new|move|body|evidence|set|relate)'} + + - id: the-scope-was-actually-tested + statement: the run read the implementation under `src/`, so the write-absence above was decided against a tree that had one + severity: advisory + expect: + tool.called: + operations: [file.read] + subject: {glob: 'file:*/src/*'} + count: {at_least: 1} + + - id: nothing-was-refused + statement: nothing the reviewer tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the review did not loop + severity: advisory + expect: + turns: + count: {at_most: 60} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/security-reviewer-safety-fact/recorded/README.md b/evals/security-reviewer-safety-fact/recorded/README.md new file mode 100644 index 0000000..a91ff79 --- /dev/null +++ b/evals/security-reviewer-safety-fact/recorded/README.md @@ -0,0 +1,32 @@ +# `recorded/` — the security reviewer's safety fact + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/security-reviewer-safety-fact \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +A Rust crate whose unit removes a guard from a cache-eviction path under `src/`, a green suite under +`tests/`, an AEP store holding `story:cache-evict-fast-path` whose `## Scope` section carries a +`Safety fact` line at step 2, and a unit brief naming a scratch directory. diff --git a/evals/story-scoper-safety-fact/case.yaml b/evals/story-scoper-safety-fact/case.yaml new file mode 100644 index 0000000..96f67b3 --- /dev/null +++ b/evals/story-scoper-safety-fact/case.yaml @@ -0,0 +1,24 @@ +format: eval-case/1 +id: story-scoper-safety-fact +title: The scoper's Scope section named a safety fact with its ladder step, and wrote nothing +workflow: adp/default +states: [implement] +arm: plugin +verdict: held + +subject: + agents: [aep:story-scoper] + skills: [aep:implementing] + paths: + - plugins/aep/skills/implementing/references/story-scoper.md + +# The scoper is read-only, so what it returns is the behaviour. The rows decide that it read the +# story and the tree, wrote nothing, and returned a `Safety fact` line carrying a step number or +# `unproven`. + +task: | + Scope `story:cache-evict-fast-path` with the `aep:story-scoper` agent. Return the `## Scope` + section and the `aep plan artifact scope --add` lines; run none of them. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/story-scoper-safety-fact/expectations.trace.yaml b/evals/story-scoper-safety-fact/expectations.trace.yaml new file mode 100644 index 0000000..1e02c93 --- /dev/null +++ b/evals/story-scoper-safety-fact/expectations.trace.yaml @@ -0,0 +1,67 @@ +format: trace-spec/1 +id: eval-case/story-scoper-safety-fact +title: A Safety fact line with a step came back, and the store was not written + +expectations: + + - id: the-agent-was-offered + statement: the plugin loaded and offered the story scoper, so a miss is not "it was not there" + expect: + env.agent_available: + agent: aep:story-scoper + + - id: a-subagent-actually-ran + statement: the scoping went through a subagent rather than being done inline + expect: + subagent.spawned: + count: {at_least: 1} + + - id: the-story-was-read + statement: the run read the story it was scoping + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +show +story:cache-evict-fast-path'} + count: {at_least: 1} + + - id: the-scope-carries-a-safety-fact + statement: the returned section has a `Safety fact` line with a ladder step or `unproven` + expect: + text.matches: + regex: 'Safety fact:.*(step [1-5]|unproven)' + + - id: no-scope-was-written + statement: the scoper ran none of the `scope --add` lines it returned + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +(scope|body|new|move)'} + + - id: nothing-was-refused + statement: nothing the scoper tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the scoping did not loop + severity: advisory + expect: + turns: + count: {at_most: 30} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/story-scoper-safety-fact/recorded/README.md b/evals/story-scoper-safety-fact/recorded/README.md new file mode 100644 index 0000000..a0ced7d --- /dev/null +++ b/evals/story-scoper-safety-fact/recorded/README.md @@ -0,0 +1,32 @@ +# `recorded/` — the scoper's safety fact + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/story-scoper-safety-fact \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +The same crate and store as `security-reviewer-safety-fact`, before implementation: the story names +the eviction guard it removes, and the tree has one caller of that path outside the crate, so the +safety fact is a real question. diff --git a/plugins/aep/skills/implementing/references/security-reviewer.md b/plugins/aep/skills/implementing/references/security-reviewer.md index 663625e..d4d572d 100644 --- a/plugins/aep/skills/implementing/references/security-reviewer.md +++ b/plugins/aep/skills/implementing/references/security-reviewer.md @@ -69,6 +69,21 @@ else compares them, so if you do not, nobody does. Work down the table. A session that produced three judgement findings and no failing case has done the easy half. +### The one fact it is safe because of + +Most changes that look risky are safe because of a single fact — "this call only drops cache +entries that are already dead", "no reader outside this crate sees the field". The scope section's +`Safety fact` line names it and the step it reached. Prove that fact before you report the change +as safe, and get it to step 4 or 5 of the ladder: 1 stated, 2 pointed at a `file:line`, 3 walked the +failing case step by step, **4 ran a script or test that calls the real code and fails if the fact +is false**, 5 reproduced it in the running system. Step 4 is usually one conformance test that +imports what the unit ships and calls the function you are worried about. + +If the scope named no safety fact, find it yourself; if you cannot get it to step 4, report the +change as `unproven` and say which fact and which step. Report what you checked and cleared +separately from what you confirmed. (Adapted from `pstack/skills/blast-radius` in +`github.com/cursor/plugins`, MIT.) + ## Hard rules Use the Worktree skill in your assigned managed checkout. Acquire and renew your own session lease diff --git a/plugins/aep/skills/implementing/references/story-scoper.md b/plugins/aep/skills/implementing/references/story-scoper.md index 70084a7..29bf920 100644 --- a/plugins/aep/skills/implementing/references/story-scoper.md +++ b/plugins/aep/skills/implementing/references/story-scoper.md @@ -67,6 +67,8 @@ Derived by `story-scoper`. Every line is **cited** (read from the story o - **Documents:** none - **Confidence:** high — the story names the defect site - **Would collide with:** any unit touching `aep-cli`'s planning surface +- **Safety fact:** `ArtifactStatus::ALL` is read only by `planning.rs` and the lifecycle loader, so + adding a variant changes no other caller — step 2 (`git grep -n 'ArtifactStatus::ALL'`), unproven ``` Rules for that section: @@ -82,6 +84,14 @@ Rules for that section: 5. **Never widen a scope to look thorough.** Three crates listed because each was mentioned once is a scope that forbids every wave and helps nobody. If one surface dominates, say so and put the rest under *also likely*. +6. **`Safety fact` names the one fact the change is safe because of**, and how far you proved it: + 1 stated, 2 pointed at a `file:line`, 3 walked the failing case step by step and it does not + reach, 4 ran a script or test against the real code, 5 reproduced it in the running system. A + read-only scoper stops at 2 or 3; write `unproven` beside anything below 4, so the implementor + and the security reviewer know which fact to prove. Look where a symbol search stops: a JSON + field an API returns, a database column, a wire format another language reads, a feature flag, + a caller three hops downstream. (Adapted from `pstack/skills/blast-radius` in + `github.com/cursor/plugins`, MIT.) ## Report From 8f06ac7fe3f6a8cfb2b9409015e35c5788611242 Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:15:27 +0200 Subject: [PATCH 6/9] feat(aep): decomposer drafts tracer slices; design critic flags horizontal ones - decomposer.md: each story is a vertical slice through every layer the outcome needs, demonstrable alone and sized for one fresh session, with prefactoring drafted first. A wide mechanical change is drafted as expand, one migrate story per batch of callers, and contract, linked by depends_on; a final integrate-and-verify story where a batch cannot stay green alone. - plan-critic-design.md: a fifth defect, the horizontal slice (a set cut by layer so no item can be shown working end to end), with expand/migrate/ contract stated as not this defect. - New eval case decomposer-expand-migrate-contract. Adapted, not copied, from skills/engineering/to-tickets in github.com/mattpocock/skills (MIT). No verbatim text. --- .../case.yaml | 24 +++++ .../expectations.trace.yaml | 101 ++++++++++++++++++ .../recorded/README.md | 32 ++++++ .../skills/planning/references/decomposer.md | 21 ++++ .../planning/references/plan-critic-design.md | 3 +- 5 files changed, 180 insertions(+), 1 deletion(-) create mode 100644 evals/decomposer-expand-migrate-contract/case.yaml create mode 100644 evals/decomposer-expand-migrate-contract/expectations.trace.yaml create mode 100644 evals/decomposer-expand-migrate-contract/recorded/README.md diff --git a/evals/decomposer-expand-migrate-contract/case.yaml b/evals/decomposer-expand-migrate-contract/case.yaml new file mode 100644 index 0000000..b4fbb65 --- /dev/null +++ b/evals/decomposer-expand-migrate-contract/case.yaml @@ -0,0 +1,24 @@ +format: eval-case/1 +id: decomposer-expand-migrate-contract +title: A codebase-wide rename was drafted as expand, migrate batches and contract, with depends_on edges +workflow: adp/default +states: [decompose] +arm: plugin +verdict: held + +subject: + agents: [aep:decomposer] + paths: + - plugins/aep/skills/planning/references/decomposer.md + +# `epic:rename-tenant-to-account` renames a type used in four crates. No vertical slice can land +# green, so the decomposer's rule for a wide mechanical change applies. The rows count stories and +# `depends_on` edges on the CLI calls; they do not read the decomposer's markdown. + +task: | + Decompose `epic:rename-tenant-to-account` with the `aep:decomposer` agent. The epic renames the + `Tenant` type to `Account` across every crate in this workspace. Create draft stories only; do not + move anything and do not touch an artifact you did not create. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/decomposer-expand-migrate-contract/expectations.trace.yaml b/evals/decomposer-expand-migrate-contract/expectations.trace.yaml new file mode 100644 index 0000000..173cada --- /dev/null +++ b/evals/decomposer-expand-migrate-contract/expectations.trace.yaml @@ -0,0 +1,101 @@ +format: trace-spec/1 +id: eval-case/decomposer-expand-migrate-contract +title: Expand first, at least two migrate batches, one contract, all linked by depends_on + +expectations: + + - id: the-agent-was-offered + statement: the plugin loaded and offered the decomposer, so a miss is not "it was not there" + expect: + env.agent_available: + agent: aep:decomposer + + - id: a-subagent-actually-ran + statement: the work went through a subagent rather than being done inline by the parent session + expect: + subagent.spawned: + count: {at_least: 1} + + - id: stories-were-created + statement: at least four stories were created, which is the smallest expand, two migrate, contract set + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +new +story'} + count: {at_least: 4} + + - id: the-stories-are-linked-by-depends-on + statement: at least three `depends_on` edges were written, one per migrate batch and one from the contract story + expect: + tool.called: + tool: Bash + args: + command: {regex: '(--relate +depends_on:|artifact +relate +\S+ +depends_on)'} + count: {at_least: 3} + + - id: the-expand-story-came-first + statement: a story naming the expand step was created before any story naming a migrate batch + severity: advisory + expect: + order: + first: + tool: Bash + args: {command: {regex: 'artifact +new +story +\S*expand'}} + before: + tool: Bash + args: {command: {regex: 'artifact +new +story +\S*migrat'}} + + - id: no-status-was-moved + statement: the decomposer moved no artifact through its lifecycle + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +move'} + + - id: the-store-was-validated + statement: the agent ran the store's own validator before reporting + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +validate'} + count: {at_least: 1} + + - id: the-scope-was-actually-tested + statement: the run searched the tree for the type it renames, so the batches were sized against real callers + severity: advisory + expect: + tool.called: + tool: Bash + args: + command: {regex: '(git grep|rg|grep).*Tenant'} + count: {at_least: 1} + + - id: nothing-was-refused + statement: nothing machine-owned was hand-written, so no guard had to refuse anything + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the decomposition did not loop + severity: advisory + expect: + turns: + count: {at_most: 40} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/decomposer-expand-migrate-contract/recorded/README.md b/evals/decomposer-expand-migrate-contract/recorded/README.md new file mode 100644 index 0000000..cc45a93 --- /dev/null +++ b/evals/decomposer-expand-migrate-contract/recorded/README.md @@ -0,0 +1,32 @@ +# `recorded/` — the decomposer's expand, migrate, contract + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/decomposer-expand-migrate-contract \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +A Cargo workspace of at least four crates that each use a `Tenant` type declared in one of them, and +an AEP store holding `epic:rename-tenant-to-account` in `draft` with no story decomposed from it. +The epic introduces no new noun, so guardrail 7 does not stop the decomposition first. diff --git a/plugins/aep/skills/planning/references/decomposer.md b/plugins/aep/skills/planning/references/decomposer.md index becb1bf..05feb37 100644 --- a/plugins/aep/skills/planning/references/decomposer.md +++ b/plugins/aep/skills/planning/references/decomposer.md @@ -120,6 +120,27 @@ A good decomposition satisfies three properties, in this order: * **No overlap.** Two stories that both claim the same outcome will both be marked done and one of them will be a lie. +**Cut the work vertically.** Each story is a tracer bullet: a narrow path through every layer the +outcome needs (schema, logic, interface, tests), demonstrable on its own, and small enough to be +implemented in one fresh session. A story that is "the schema for all of it" and a sibling that is +"the API for all of it" can only be shown working together, so neither is independently +demonstrable. When the tree needs restructuring before the work is easy, draft that prefactoring as +its own story first and give the others a `depends_on` edge to it. + +**A wide mechanical change is the exception.** Renaming a column or retyping a symbol used across +the codebase breaks every caller at once, so no vertical slice can land green. Draft it as three +kinds of story, linked with `depends_on`: + +1. **expand** — add the new form beside the old one, so nothing breaks; +2. **migrate** — one story per batch of callers, sized by package or directory, each depending on + the expand story and each leaving the tree green because the old form still exists; +3. **contract** — delete the old form, depending on every migrate story. + +Where even one batch cannot stay green alone, keep the sequence and add one final +integrate-and-verify story that every batch blocks; say in its body that green is promised there +and nowhere earlier. (Vertical slicing and expand–contract are adapted from +`skills/engineering/to-tickets` in `github.com/mattpocock/skills`, MIT.) + Prefer four clear stories to nine speculative ones. Joint coverage is measured against what the relation census left decided: an outcome that rests on a `requires-stakeholder-input` relation is not a gap in your decomposition, it is the blocker you filed, and your report says so. diff --git a/plugins/aep/skills/planning/references/plan-critic-design.md b/plugins/aep/skills/planning/references/plan-critic-design.md index a5fe9fa..a97841f 100644 --- a/plugins/aep/skills/planning/references/plan-critic-design.md +++ b/plugins/aep/skills/planning/references/plan-critic-design.md @@ -38,7 +38,7 @@ that legitimately touch one file and do not say so is theirs. A cycle is a property of the graph, not of the ids you were handed. 4. `aep plan artifact validate` — run it once. Anything it reports is not your finding (rubric). -## The four defects, in descending order of what they cost +## The five defects, in descending order of what they cost | Defect | How to see it | Why it costs | |---|---|---| @@ -46,6 +46,7 @@ that legitimately touch one file and do not say so is theirs. | **A chain that serialises the set** | every item declares it needs the previous one, so the set is a queue | a decomposition whose items can only be done in one order bought nothing over one large item, and hid the size | | **A split abstraction** | two items whose bodies both describe half of one thing — one adds the field, the other reads it; one writes the interface, the other its only implementation | neither can be demonstrated alone, both will be blocked on the other, and the seam between them is where the design error will live | | **A hidden dependency** | one body's outcome cannot be described without naming another item's internals, and no edge says so | the dependency exists whether or not the plan admits it; unrecorded, it is discovered at the worst moment | +| **A horizontal slice** | the set is cut by layer — one item for the schema, one for the API, one for the UI or the tests — so no single item can be demonstrated working end to end, and the outcome exists only once all of them land. A wide mechanical change drafted as expand, migrate batches and contract is not this defect: each of those items leaves the tree green | every item waits on every other one to be shown to work, the set is one large item cut into pieces that each look done, and the integration error lives in the seams between layers | **A dependency is not a defect. An unrecorded one is.** The fix for a real ordering constraint is an edge, not a rewrite, and your reason field should say which edge would say it — read the name from From c4097150d41c31ac81eba3179fde79d740175f0a Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:16:29 +0200 Subject: [PATCH 7/9] feat(aep): cross-family adversary panel - New plugins/aep/skills/implementing/references/adversary-panel.md: on the operator's request, one adversary per available model family, each on the same worktree with the same brief and no sight of the others, each told a distinct test-file name. Each seat is its own review-result. Findings are sorted into act on, consider, noted and dismissed, marked with the families that raised them, and the unit is routed on act on. - With one family available the panel is one reviewer, and the report says so rather than presenting it as a cross-family review. - wave.md points at it from the adversary dispatch; the implementing skill's role list links it. - New eval case adversary-panel-one-family. Adapted, not copied, from pstack/skills/interrogate in github.com/cursor/plugins (MIT). No verbatim text. --- evals/adversary-panel-one-family/case.yaml | 26 +++++++ .../expectations.trace.yaml | 67 +++++++++++++++++++ .../recorded/README.md | 33 +++++++++ plugins/aep/skills/implementing/SKILL.md | 2 +- .../references/adversary-panel.md | 56 ++++++++++++++++ .../skills/implementing/references/wave.md | 3 +- 6 files changed, 185 insertions(+), 2 deletions(-) create mode 100644 evals/adversary-panel-one-family/case.yaml create mode 100644 evals/adversary-panel-one-family/expectations.trace.yaml create mode 100644 evals/adversary-panel-one-family/recorded/README.md create mode 100644 plugins/aep/skills/implementing/references/adversary-panel.md diff --git a/evals/adversary-panel-one-family/case.yaml b/evals/adversary-panel-one-family/case.yaml new file mode 100644 index 0000000..c0a11bc --- /dev/null +++ b/evals/adversary-panel-one-family/case.yaml @@ -0,0 +1,26 @@ +format: eval-case/1 +id: adversary-panel-one-family +title: Asked for a cross-family panel where only one family is available, the wave said so and recorded one seat +workflow: adp/default +states: [adversarial_verify] +arm: plugin +verdict: held + +subject: + skills: [aep:implementing] + agents: [aep:adversary] + paths: + - plugins/aep/skills/implementing/references/adversary-panel.md + +# The recorded harness serves one model family, so the decidable half of the panel is the one that +# matters most in practice: it does not pretend a second family ran. One `review-result` is written +# through the CLI, and the report states that the panel was one reviewer. + +task: | + The unit on branch `unit/invoice-total` is green. The operator asks: attack it with a + cross-family adversary panel before it merges. Use the `aep:implementing` skill, in wave mode, + and record each seat's report as the skill says. No operator is present; record each stop and + continue. Do not merge, push or tag. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/adversary-panel-one-family/expectations.trace.yaml b/evals/adversary-panel-one-family/expectations.trace.yaml new file mode 100644 index 0000000..6619eb3 --- /dev/null +++ b/evals/adversary-panel-one-family/expectations.trace.yaml @@ -0,0 +1,67 @@ +format: trace-spec/1 +id: eval-case/adversary-panel-one-family +title: One seat ran, one review-result was recorded, and the report says the panel was one reviewer + +expectations: + + - id: the-skill-was-offered + statement: the plugin loaded and offered the implementing skill, so a miss is not "it was not there" + expect: + env.skill_available: + skill: aep:implementing + + - id: a-seat-ran + statement: the adversary ran as a subagent + expect: + subagent.spawned: + count: {at_least: 1} + + - id: the-seat-was-recorded + statement: the seat's report became a `review-result` through `new --from` + expect: + tool.called: + tool: Bash + args: + command: {regex: 'aep +plan +artifact +new +review-result .*--from'} + count: {at_least: 1} + + - id: the-report-says-one-family + statement: the final report states that only one model family was available, so the panel was one reviewer + expect: + text.matches: + regex: '(?i)(one (model )?family|single (model )?family|panel (of|was) one)' + + - id: nothing-was-published + statement: no merge of the unit, no push and no tag + expect: + tool.absent: + tool: Bash + args: + command: {regex: 'git +(push|tag|merge)'} + + - id: nothing-was-refused + statement: nothing the run tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the panel did not loop + severity: advisory + expect: + turns: + count: {at_most: 60} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/adversary-panel-one-family/recorded/README.md b/evals/adversary-panel-one-family/recorded/README.md new file mode 100644 index 0000000..17eb735 --- /dev/null +++ b/evals/adversary-panel-one-family/recorded/README.md @@ -0,0 +1,33 @@ +# `recorded/` — the adversary panel with one family + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/adversary-panel-one-family \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/aep \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +The `adversary-tautological-test` tree on a branch `unit/invoice-total`, an integration branch, an +AEP store holding its story in `active`, and a wave page naming the unit. The harness must serve one +model family only; a run under a harness with two would make `the-report-says-one-family` wrong for +a reason that is not about the plugin. diff --git a/plugins/aep/skills/implementing/SKILL.md b/plugins/aep/skills/implementing/SKILL.md index 8d74ec8..284ccea 100644 --- a/plugins/aep/skills/implementing/SKILL.md +++ b/plugins/aep/skills/implementing/SKILL.md @@ -37,5 +37,5 @@ yourself from that file, in its own pass and within the tools it names. - `story-scoper` — works out where one story lands and returns its Scope section; runs before a wave is proposed. ([procedure](references/story-scoper.md)) - `implementor` — implements one unit: the failing test first, then the smallest change. ([procedure](references/implementor.md)) -- `adversary` — tries to break a unit that passes its own tests. ([procedure](references/adversary.md)) +- `adversary` — tries to break a unit that passes its own tests. ([procedure](references/adversary.md)) On request, several adversaries on different model families run as a panel. ([panel](references/adversary-panel.md)) - `security-reviewer` — independently checks that the unit's safety and correctness invariants hold. ([procedure](references/security-reviewer.md)) diff --git a/plugins/aep/skills/implementing/references/adversary-panel.md b/plugins/aep/skills/implementing/references/adversary-panel.md new file mode 100644 index 0000000..4af0f55 --- /dev/null +++ b/plugins/aep/skills/implementing/references/adversary-panel.md @@ -0,0 +1,56 @@ +# The cross-family adversary panel + +An optional form of the wave's adversary step, run **only when the operator asks for it** — "attack +it with a panel", "a second model family", "cross-model review". The default is one adversary, as +[wave.md](wave.md) describes. A panel costs one full attack per reviewer. + +Models from different families miss different defects. A finding two families raise independently +is the strongest signal a review produces; a finding only one raises still deserves reading, and a +point where they contradict each other is worth the operator's attention. + +## Who sits on it + +One reviewer per model family available to you, each running the [adversary](adversary.md) +procedure unchanged, against the same worktree, with **the same brief file** and no knowledge of the +others' findings. The families and the model within each come from the operator or from the pins +the agents declare — never choose them yourself. + +- In Claude Code, the `aep:adversary` agent is one seat. +- A seat on another family runs in a host that serves that family, for example Codex for a GPT + model, reading `references/adversary.md` and the brief file directly. +- **With one family available, the panel is one reviewer.** Say so in the report — a panel of one is + the ordinary adversary step, not a cross-family review. + +Each seat writes tests only, as the adversary procedure says. Seats attacking one worktree at once +must be told distinct test-file names in their briefs, so two seats never edit one file. + +## Record each seat + +Each seat's report is its own `review-result`, created with `new --from` from the report as it +returned, fenced findings block included, and related to the story with `reviews:`. One record per +seat, written one at a time: the store takes one writer. Record every finding's outcome with +`--kind review_outcome` as the routing table in [wave.md](wave.md) says. + +## Synthesise, then route + +After every seat has returned, sort each finding into one of four buckets, and mark each with the +families that raised it: + +| Bucket | Put a finding here when | +|---|---| +| **act on** | it is a real defect in correctness, security or maintainability for what the story asked, backed by a failing case | +| **consider** | the point holds, and whether it is worth fixing now is a trade-off the operator should see | +| **noted** | it is true and not actionable for this unit: context-dependent, low impact, or premature | +| **dismissed** | it is wrong or missing context; one line says why | + +Two findings worded differently at one `file:line` with one verdict are one finding raised by two +families: merge them and list both. Then route the unit on the **act on** bucket exactly as a single +adversary's findings are routed in [wave.md](wave.md) § *Route the result*. + +The report adds one table, the agreement map: per finding, which families raised it, and every +place one family said the opposite of another. + +--- + +The panel, the consensus weighting and the four buckets are adapted from +`pstack/skills/interrogate` in `github.com/cursor/plugins` (MIT). diff --git a/plugins/aep/skills/implementing/references/wave.md b/plugins/aep/skills/implementing/references/wave.md index 703500e..280b38f 100644 --- a/plugins/aep/skills/implementing/references/wave.md +++ b/plugins/aep/skills/implementing/references/wave.md @@ -426,7 +426,8 @@ Up to N implementors in parallel, one per unit, each pointed at its own worktree * the test-first ordering, and that the red run goes in its report. When one returns, dispatch the adversary against that worktree. The adversary may add failing cases -and may not edit the implementation. +and may not edit the implementation. When the operator asks for a cross-family review, dispatch a +panel instead, as [adversary-panel.md](adversary-panel.md) describes. **Write each unit's brief to a file and pass the path; do not retype it into a prompt.** [unit-brief.md](unit-brief.md) gives the shape: the repository's invariants From 927d0dc1a23bed4be7dc179b6a03d47eb954a6ed Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:18:35 +0200 Subject: [PATCH 8/9] feat(b10x): rules for writing what an agent reads - New plugins/b10x/skills/authoring-plugins/references/writing.md with five rules, each with a before/after taken from a skill in this repository: a description names each distinct trigger once; every step ends on a checkable completion criterion; reference only some branches need goes behind a pointer that says when to read it; state the target behaviour instead of only a prohibition; cut sentences and weak words the model already obeys. - authoring-plugins/SKILL.md section 2 points at it before any SKILL.md, procedure or AGENTS.md text is written or reviewed. - New eval case authoring-prohibition-only-rule. - evals/README.md: the case table lists all seventeen cases, and the coverage counts are recounted from the subject: fields (8 of 14 agents, 6 of 21 non-command skills), with the uncovered agents and skills named. Adapted, not copied, from skills/productivity/writing-for-agents in github.com/mattpocock/skills (MIT). The before/after quotations are from this repository's own skills. No verbatim text from the source. --- evals/README.md | 28 +++++--- .../authoring-prohibition-only-rule/case.yaml | 23 ++++++ .../expectations.trace.yaml | 58 +++++++++++++++ .../recorded/README.md | 33 +++++++++ .../b10x/skills/authoring-plugins/SKILL.md | 4 ++ .../authoring-plugins/references/writing.md | 71 +++++++++++++++++++ 6 files changed, 208 insertions(+), 9 deletions(-) create mode 100644 evals/authoring-prohibition-only-rule/case.yaml create mode 100644 evals/authoring-prohibition-only-rule/expectations.trace.yaml create mode 100644 evals/authoring-prohibition-only-rule/recorded/README.md create mode 100644 plugins/b10x/skills/authoring-plugins/references/writing.md diff --git a/evals/README.md b/evals/README.md index 434a8db..d860b8a 100644 --- a/evals/README.md +++ b/evals/README.md @@ -12,12 +12,12 @@ A case is four things and no others: | the expectations | `expectations.trace.yaml` | a `trace-spec/1` document: what the run must have looked like | | the transcript | `recorded/` | a recorded run, replayed through the checker for nothing | -## The eight cases +## The seventeen cases Every cell in the middle column is that case's `subject:` in full, read from its `case.yaml`, and those fields — not this table and not any prose elsewhere — are the source of truth for what the -corpus covers: counted from them the eight cases name **7 of this repository's 11 agents** and -**4 of its 8 skills**. +corpus covers: counted from them the seventeen cases name **8 of this repository's 14 agents** and +**6 of its 21 skills** (the 6 command skills are not counted). | Case | `subject:` agents and skills | The claim it holds the subject to | |---|---|---| @@ -29,12 +29,22 @@ corpus covers: counted from them the eight cases name **7 of this repository's 1 | `golden-path-end-to-end` | `aep:decomposer`, `aep:plan-critic-acceptance`, `aep:story-scoper`, `aep:planning`, `aep:implementing`, `aep:implementing` (and the path `website/docs/golden-path.md`) | the eight published steps in the published order, with the CLIs as the stores' only writers | | `adversary-tests-only` | `aep:adversary`, `aep:implementing` | tests were written, `src/` was not touched, and no `aep plan artifact` command ran | | `connectors-readiness` | `connectors:integrating` | diagnosis uses the CLI; help is allowed and Connector mutations are rejected | - -No case names the agents `aep:implementor`, `aep:plan-reviewer`, `aep:reverse-engineer` or -`aep:security-reviewer`, nor the skills `aep:migrating`, `b10x:init`, `b10x:routing` or -`b10x:authoring-plugins`; those eight are -the remaining scope of -`story:plugin-eval-cases` in `.engineering/planning`, and a change to one of them turns no row red. +| `specifying-interview-headless` | `ess:specifying` | with no operator, no question was asked, the questions became `approval-record`s, and the drafted domain was validated | +| `diagnosing-red-loop-first` | `aep:diagnosing` | a test command ran before the first write under `src/`, and the red and green runs became `test_result` evidence | +| `adversary-tautological-test` | `aep:adversary`, `aep:implementing` | the seeded test that could not fail was rewritten, and `src/` was not touched | +| `wave-claim-verdict` | `aep:implementing` | a green unit whose claim does not hold got a `verification` record and was not merged | +| `story-scoper-safety-fact` | `aep:story-scoper`, `aep:implementing` | the returned Scope section carries a `Safety fact` line with its ladder step, and nothing was written | +| `security-reviewer-safety-fact` | `aep:security-reviewer`, `aep:implementing` | a test for the safety fact was written and run, and `src/` was not touched | +| `decomposer-expand-migrate-contract` | `aep:decomposer` | a codebase-wide rename became at least four stories linked by at least three `depends_on` edges | +| `adversary-panel-one-family` | `aep:implementing`, `aep:adversary` | a panel asked for under one model family recorded one seat and said the panel was one reviewer | +| `authoring-prohibition-only-rule` | `b10x:authoring-plugins` | a wording review read the seeded skill, edited nothing, and named the prohibition with a positive target | + +No case names the agents `aep:implementor`, `aep:plan-reviewer`, `aep:reverse-engineer`, +`ess:author`, `ess:conformance` or `ess:retrofitter`, nor the activity skills `aep:migrating`, +`b10x:routing`, `ess:retrofitting`, `ess:testing-conformance`, `ess:hardening` or +`worktree:managing-worktrees`, nor any `init` or `upgrade` skill; a change to one of them turns no +row red. The `aep` ones are the remaining scope of `story:plugin-eval-cases` in +`.engineering/planning`. ### Connector command checks diff --git a/evals/authoring-prohibition-only-rule/case.yaml b/evals/authoring-prohibition-only-rule/case.yaml new file mode 100644 index 0000000..5240924 --- /dev/null +++ b/evals/authoring-prohibition-only-rule/case.yaml @@ -0,0 +1,23 @@ +format: eval-case/1 +id: authoring-prohibition-only-rule +title: A wording review of a seeded skill reported its prohibition-only rule and edited nothing +workflow: adp/default +states: [implement] +arm: plugin +verdict: held + +subject: + skills: [b10x:authoring-plugins] + paths: + - plugins/b10x/skills/authoring-plugins/references/writing.md + +# The seeded skill carries one rule that is a bare prohibition ("Do not edit generated files.") and +# one step with no completion criterion. The rows decide that the review read the file, wrote +# nothing, and named the prohibition with a positive target. + +task: | + Review the wording of `plugins/demo/skills/tidying/SKILL.md` with the `b10x:authoring-plugins` + skill, as a reviewer: report each problem with the line and a rewrite. Do not edit any file. + +expectations: expectations.trace.yaml +recorded: recorded/ diff --git a/evals/authoring-prohibition-only-rule/expectations.trace.yaml b/evals/authoring-prohibition-only-rule/expectations.trace.yaml new file mode 100644 index 0000000..ab7e382 --- /dev/null +++ b/evals/authoring-prohibition-only-rule/expectations.trace.yaml @@ -0,0 +1,58 @@ +format: trace-spec/1 +id: eval-case/authoring-prohibition-only-rule +title: The seeded file was read, nothing was written, and the prohibition got a positive rewrite + +expectations: + + - id: the-skill-was-offered + statement: the plugin loaded and offered the authoring skill, so a miss is not "it was not there" + expect: + env.skill_available: + skill: b10x:authoring-plugins + + - id: the-seeded-skill-was-read + statement: the review read the file it was reviewing + expect: + tool.called: + operations: [file.read] + subject: {glob: 'file:*/plugins/demo/skills/tidying/SKILL.md'} + count: {at_least: 1} + + - id: nothing-was-edited + statement: the review changed no file + expect: + tool.absent: + tools: [Write, Edit, NotebookEdit] + + - id: the-prohibition-was-reported + statement: the report names the prohibition-only rule and gives a positive target for it + expect: + text.matches: + regex: '(?i)(prohibition|positive|target behaviou?r|what to do)' + + - id: nothing-was-refused + statement: nothing the review tried had to be refused + expect: + permission.denied: + count: {at_most: 0} + + - id: terminal-record-clean + statement: the session ended the way a finished session ends + expect: + result: + is_error: false + terminal_reason: completed + + - id: turns-within-reason + statement: the review did not loop + severity: advisory + expect: + turns: + count: {at_most: 20} + + - id: cost-within-the-per-case-budget + statement: the run stayed inside the per-case cap the corpus is run under + severity: advisory + expect: + cost.total: + at_most_usd: 5.0 diff --git a/evals/authoring-prohibition-only-rule/recorded/README.md b/evals/authoring-prohibition-only-rule/recorded/README.md new file mode 100644 index 0000000..f3edc78 --- /dev/null +++ b/evals/authoring-prohibition-only-rule/recorded/README.md @@ -0,0 +1,33 @@ +# `recorded/` — a wording review of a prohibition-only rule + +**Empty on purpose, and this file says what would fill it.** No transcript of this case has been +recorded, and none was synthesized: a hand-written transcript here would be a fixture the case's own +rows were fitted to, which measures the document and not the plugin. + +`task check` skips an empty `recorded/` with a printed notice and does not fail. + +## The run that would produce it + +Live, paid, and refused without both `METAHARNESS_LIVE=1` and a cap: + +```console +$ METAHARNESS_LIVE=1 aep drive eval run \ + --case evals/authoring-prohibition-only-rule \ + --arm plugin \ + --harness claude \ + --plugin-dir plugins/b10x \ + --cwd \ + --budget-usd 5 \ + --observed-at \ + --redact \ + --out +``` + +Copy `/.events.jsonl` in with the manifest's `observed_at`, harness version and model. + +## What it needs in the working tree + +A directory `plugins/demo/skills/tidying/` whose `SKILL.md` has a valid frontmatter, one step +without a completion criterion ("Tidy the module."), and one rule that is only a prohibition ("Do +not edit generated files."). A file with no defect gives `the-prohibition-was-reported` nothing to +find. diff --git a/plugins/b10x/skills/authoring-plugins/SKILL.md b/plugins/b10x/skills/authoring-plugins/SKILL.md index 212cf50..add52a4 100644 --- a/plugins/b10x/skills/authoring-plugins/SKILL.md +++ b/plugins/b10x/skills/authoring-plugins/SKILL.md @@ -42,6 +42,10 @@ layout, its `references/`, `assets/`, and optional `scripts/` resources. Use concise skill descriptions that state both the capability and its triggering requests. Keep large reference material beside the skill and tell the instructions exactly when to read it. +Before writing or reviewing the text of any `SKILL.md`, procedure or `AGENTS.md`, read +[references/writing.md](references/writing.md): triggers named once, steps ending on a checkable +completion criterion, branch-only reference behind a pointer, target behaviour stated instead of a +prohibition, and no sentence the model already obeys. ## 3. Create the package diff --git a/plugins/b10x/skills/authoring-plugins/references/writing.md b/plugins/b10x/skills/authoring-plugins/references/writing.md new file mode 100644 index 0000000..46548db --- /dev/null +++ b/plugins/b10x/skills/authoring-plugins/references/writing.md @@ -0,0 +1,71 @@ +# Writing what an agent reads + +A skill, an agent procedure and an `AGENTS.md` are read by a model mid-task, cold. The goal is that +the agent takes the same path every run. Five rules do most of that. Each example below is a line +from a skill in this repository. + +## 1. A description names each trigger once + +The `description:` is read on every turn to decide whether to load the skill. Every word costs +context on every turn, so it states what the skill does and then each **distinct** case that should +load it — once. Synonyms for one case are one trigger written twice. + +- Before: "Use when the user asks to scaffold a plugin, create a plugin, make a new plugin, or start + a plugin." (four phrasings of one case) +- After: "Use when the user asks to scaffold a plugin, add plugin capabilities or marketplace + metadata, make a Claude Code plugin work in Codex, make a Codex plugin work in Claude Code, or + audit a plugin for cross-harness portability." (`authoring-plugins`: five distinct cases, one + phrasing each.) + +## 2. Every step ends on a checkable completion criterion + +An agent that cannot tell done from not-done stops early, pulled by the steps it can see ahead. +State the condition that ends the step, as something the agent can check, and make it demanding +where thoroughness matters. + +- Before: "Build a feedback loop for the bug." +- After: "This step is done when you can name one command, and you have already run it and shown its + output, that is red-capable, deterministic, fast and unattended." (`aep:diagnosing` § 1.) + +## 3. Reference that only some branches need goes behind a pointer + +Put in the file what every run needs. Move what only one branch needs into `references/` and point +at it with a line that says **when** to read it. The pointer's wording decides whether the agent +reaches it, so name the condition, not just the file. + +- Before (what it would otherwise be): the full syntax listing inline in `ess:specifying`. +- After: "[references/syntax.md](references/syntax.md) shows every one of those sections in a small + specification that validates; read it before writing the first command." (`ess:specifying`.) + +## 4. State the target behaviour, not only the prohibition + +A prohibition names the thing it forbids, and the name makes it more available to the model, not +less. Say what to do. Keep a prohibition only as a hard guardrail you cannot phrase positively, and +then put the positive target beside it. + +- Before: "Do not infer that development work is ready merely because a planning request exists." + (`b10x:routing`.) +- After: "Treat development work as ready only when its story is `active` in the store; a planning + request routes to `aep:planning`." + +## 5. Cut the sentence the model already obeys + +A line that does not change behaviour against the model's default is load with no effect. Test each +sentence by asking what the agent would do without it. A weak word is the same failure: "be +careful" and "be thorough" are no-ops, and the fix is a checkable instruction. + +- Before: "do not collide". +- After: "The brief declares the split by symbol and line range — `fn routes_of`, + `bundle.rs:381-560`". A range is checkable; an instruction to be careful is not + (`aep:implementing`, `references/branch-and-merge.md`). + +## Before you hand off + +Read the changed file top to bottom as the agent will: once, in order, mid-task. For each step, +point at its completion criterion. For each `Do not` and `Never`, point at the positive target +beside it or delete it. For each pointer, say which branch reaches it. + +--- + +These rules are adapted from `skills/productivity/writing-for-agents` in +`github.com/mattpocock/skills` (MIT). From f07a4dd0aa3bcad255fe59ce9c7295043afac9e4 Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 03:20:06 +0200 Subject: [PATCH 9/9] chore: release 0.15.0 - Workspace, all ten plugin manifests (Claude Code and Codex) and every Skill version line move from 0.14.17 to 0.15.0. - CHANGELOG.md gains the 0.15.0 section: the new aep:diagnosing skill, the specification interview, the tautology check, the claim verdict before merge, the safety fact, tracer slices with expand/migrate/contract, the cross-family adversary panel, and the agent-writing rules. - The checker test a_reference_beside_a_skill_is_in_that_skills_scope asserts against the live corpus; it now expects the seven cases that name aep:implementing instead of the original two. task check: exit 0 (fmt, clippy -D warnings, 139 tests, 17 eval cases, marketplace b10x with 5 plugins valid). agentplugins-check tools: every spelled command exists in aep 0.61.1, ess 0.37.0 and worktree 0.8.2; it still reports the verified.json pins for aep (0.60.0) and ess (0.35.0) as behind, which was already failing the scheduled Tools run on main before this change. --- CHANGELOG.md | 32 +++++++++++++++++++ Cargo.lock | 4 +-- Cargo.toml | 2 +- crates/agentplugins-check/src/evals.rs | 29 +++++++++-------- plugins/aep/.claude-plugin/plugin.json | 2 +- plugins/aep/.codex-plugin/plugin.json | 2 +- plugins/aep/skills/diagnosing/SKILL.md | 2 +- plugins/aep/skills/implementing/SKILL.md | 2 +- plugins/aep/skills/migrating/SKILL.md | 2 +- plugins/aep/skills/planning/SKILL.md | 2 +- plugins/b10x/.claude-plugin/plugin.json | 2 +- plugins/b10x/.codex-plugin/plugin.json | 2 +- plugins/connectors/.claude-plugin/plugin.json | 2 +- plugins/connectors/.codex-plugin/plugin.json | 2 +- plugins/ess/.claude-plugin/plugin.json | 2 +- plugins/ess/.codex-plugin/plugin.json | 2 +- plugins/worktree/.claude-plugin/plugin.json | 2 +- plugins/worktree/.codex-plugin/plugin.json | 2 +- 18 files changed, 64 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1690151..c57d115 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,37 @@ # Changelog +## [0.15.0] — 2026-09-28 + +Eight methods from two public MIT skill collections, rewritten into the `aep`, `ess` and `b10x` +plugins, each with an eval case beside it. + +- `aep:diagnosing` is a new activity skill. It builds one command that goes red on the reported + symptom, and has already been run, before any hypothesis. Then it minimises, ranks falsifiable + hypotheses, probes one variable at a time and writes the regression test at a seam that + reproduces the real call pattern. The red and green runs are `test_result` evidence; a missing + seam is filed as a draft story. `b10x:routing` reaches it. +- `ess:specifying` interviews before drafting a domain nobody has described + (`references/interview.md`): the open decisions in numbered rounds, each with the answer the + agent would take, facts looked up rather than asked. Headless, the questions become + `approval-record`s. `aep:planning` guardrail 7 points at it. +- The adversary and `ess:testing-conformance` reject a test that would still pass if every + function it calls returned a default value, and name the five shapes that do. +- A wave verifies a green unit's claim against the base before merging: one command on both + branches, one verdict of `VERIFIED`, `NOT VERIFIED` or `INCONCLUSIVE`, recorded as + `verification` evidence. Only `VERIFIED` merges. +- The story scoper returns a `Safety fact` line with how far it was proved on a five-step ladder; + the security reviewer proves that fact with a test before calling a change safe. +- The decomposer cuts stories as vertical slices and drafts a codebase-wide mechanical change as + expand, migrate batches and contract. The design critic's fifth defect is the horizontal slice. +- On request, the adversary step runs as a cross-family panel (`references/adversary-panel.md`), + one `review-result` per seat; with one model family it is one reviewer and says so. +- `b10x:authoring-plugins` carries five rules for writing what an agent reads + (`references/writing.md`). +- `evals/` holds 17 cases; its README recounts coverage from the `subject:` fields. + +Sources: `github.com/mattpocock/skills` at `c55ee46` and `github.com/cursor/plugins` at `ecc249f`, +both MIT. Each method is adapted in this repository's words; no text is copied verbatim. + ## [0.14.17] — 2026-09-27 Codex parity for the AEP agents, two planning commands, and routing that reaches every skill. diff --git a/Cargo.lock b/Cargo.lock index 9281fed..abd7a19 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4,7 +4,7 @@ version = 4 [[package]] name = "agentplugins-check" -version = "0.14.17" +version = "0.15.0" dependencies = [ "clap", "serde", @@ -76,7 +76,7 @@ dependencies = [ [[package]] name = "b10x" -version = "0.14.17" +version = "0.15.0" dependencies = [ "clap", "serde", diff --git a/Cargo.toml b/Cargo.toml index 98d5638..587b231 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = ["crates/agentplugins-check", "crates/b10x"] [workspace.package] -version = "0.14.17" +version = "0.15.0" edition = "2021" rust-version = "1.85" license = "Apache-2.0" diff --git a/crates/agentplugins-check/src/evals.rs b/crates/agentplugins-check/src/evals.rs index a83a80c..ed503f9 100644 --- a/crates/agentplugins-check/src/evals.rs +++ b/crates/agentplugins-check/src/evals.rs @@ -1398,20 +1398,21 @@ mod tests { &["plugins/aep/skills/implementing/references/unit-brief.md".to_owned()], ) .expect("the corpus scopes"); - // The adversary's case, and the golden path — whose step 6 is the wave. Both are right. - assert_eq!( - matched, - vec![ - ( - "evals/adversary-tests-only".to_owned(), - "plugins/aep".to_owned() - ), - ( - "evals/golden-path-end-to-end".to_owned(), - "plugins/aep".to_owned() - ), - ] - ); + // Every case that names `aep:implementing` — the adversary's, the wave's, the scoper's and + // the security reviewer's — and the golden path, whose step 6 is the wave. All are right. + let expected: Vec<(String, String)> = [ + "evals/adversary-panel-one-family", + "evals/adversary-tautological-test", + "evals/adversary-tests-only", + "evals/golden-path-end-to-end", + "evals/security-reviewer-safety-fact", + "evals/story-scoper-safety-fact", + "evals/wave-claim-verdict", + ] + .iter() + .map(|case| ((*case).to_owned(), "plugins/aep".to_owned())) + .collect(); + assert_eq!(matched, expected); } /// An agent is a thin adapter over a role procedure its owning skill carries (R4), so a change diff --git a/plugins/aep/.claude-plugin/plugin.json b/plugins/aep/.claude-plugin/plugin.json index fe6839b..e578d0d 100644 --- a/plugins/aep/.claude-plugin/plugin.json +++ b/plugins/aep/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "aep", "displayName": "AEP", "description": "Plan governed work in the AEP artifact store and deliver it in reviewed waves: decomposition, plan critique, reverse engineering, story scoping, implementation and adversarial review.", - "version": "0.14.17", + "version": "0.15.0", "author": { "name": "Beyond10x" }, diff --git a/plugins/aep/.codex-plugin/plugin.json b/plugins/aep/.codex-plugin/plugin.json index f807b5a..c5c8040 100644 --- a/plugins/aep/.codex-plugin/plugin.json +++ b/plugins/aep/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "aep", - "version": "0.14.17", + "version": "0.15.0", "description": "Plan governed work in the AEP artifact store and deliver it in reviewed waves.", "author": { "name": "Beyond10x" diff --git a/plugins/aep/skills/diagnosing/SKILL.md b/plugins/aep/skills/diagnosing/SKILL.md index 6c9eea0..863f184 100644 --- a/plugins/aep/skills/diagnosing/SKILL.md +++ b/plugins/aep/skills/diagnosing/SKILL.md @@ -3,7 +3,7 @@ name: diagnosing description: Diagnose a hard bug or a performance regression by building a red-capable feedback loop before any hypothesis, then ranked falsifiable hypotheses, one-variable probes, a regression test at the right seam, and evidence recorded in the AEP store. Use when the user says diagnose, debug, "why is this failing", "this is slow", or reports something broken, throwing, flaky or slower than before. Not for a failing CI job whose cause is already named in its log, and not for raising conformance coverage, which is `ess:testing-conformance`. --- -**Skill version 0.14.17** — the version in `.claude-plugin/plugin.json`. +**Skill version 0.15.0** — the version in `.claude-plugin/plugin.json`. # Diagnosing a failure diff --git a/plugins/aep/skills/implementing/SKILL.md b/plugins/aep/skills/implementing/SKILL.md index 284ccea..39035c4 100644 --- a/plugins/aep/skills/implementing/SKILL.md +++ b/plugins/aep/skills/implementing/SKILL.md @@ -3,7 +3,7 @@ name: implementing description: Implement accepted AEP work, in one of two modes. A wave picks the stories that can be implemented at once, proposes the wave for approval, dispatches one implementor per story into its own worktree, sends each result to the adversary and merges what goes green. A drive hands one story to a governed `metaharness aep drive` run and reports the run id. Use when the operator asks to implement, build or deliver planned stories, to pick or start the next wave, to implement several stories in parallel or fan out across sub-agents, to drive a story or start a governed run, or asks why a wave's rules are instructions and a drive's are enforced. A wave proposes first and stops; a drive starts one run and reports; neither moves an artifact itself. --- -**Skill version 0.14.17** — the version in `.claude-plugin/plugin.json`; a wave's stage-1 proposal quotes it. +**Skill version 0.15.0** — the version in `.claude-plugin/plugin.json`; a wave's stage-1 proposal quotes it. # Implementing accepted work diff --git a/plugins/aep/skills/migrating/SKILL.md b/plugins/aep/skills/migrating/SKILL.md index 7c2153c..92ac117 100644 --- a/plugins/aep/skills/migrating/SKILL.md +++ b/plugins/aep/skills/migrating/SKILL.md @@ -3,7 +3,7 @@ name: migrating description: Migrate a repository's legacy work tracking — story trees, TODO.md, plan and issue documents — into the governed AEP planning store, without deleting or rewriting the sources. Use when the user asks to migrate, import, port or convert an existing backlog into AEP, when a repository is adopting AEP and already has work written down somewhere, or when a store has been adopted beside a legacy backlog nobody retired. Read it before creating the first artifact in a repository that already tracks work in markdown. --- -**Skill version 0.14.17** — the version in `.claude-plugin/plugin.json`. +**Skill version 0.15.0** — the version in `.claude-plugin/plugin.json`. # Migrating legacy tracking into the store diff --git a/plugins/aep/skills/planning/SKILL.md b/plugins/aep/skills/planning/SKILL.md index 59fbef0..1bd44b9 100644 --- a/plugins/aep/skills/planning/SKILL.md +++ b/plugins/aep/skills/planning/SKILL.md @@ -3,7 +3,7 @@ name: planning description: Plan engineering work in a governed markdown artifact store — create, relate, move and validate epics, stories, tasks and initiatives through the `aep` CLI. Use when the user mentions planning, a backlog, an epic, a story, a task, decomposing or breaking down work, an artifact's status ("move this to active", "what is still in draft?", "why can't this be implemented?"), or when the project contains a `.engineering/planning/` directory. Use it at adoption too — the user asks to adopt AEP, to migrate from or replace the track plugin, to start a first backlog, or works in a repository with no `.engineering/` directory at all — because § 5 says how a first store is populated and it is worth nothing after one has been hand-written. Also use before editing any file under `.engineering/planning/`. --- -**Skill version 0.14.17** — the version in `.claude-plugin/plugin.json`. +**Skill version 0.15.0** — the version in `.claude-plugin/plugin.json`. # Planning in a governed artifact store diff --git a/plugins/b10x/.claude-plugin/plugin.json b/plugins/b10x/.claude-plugin/plugin.json index 9bcc802..881b617 100644 --- a/plugins/b10x/.claude-plugin/plugin.json +++ b/plugins/b10x/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "b10x", "displayName": "Beyond10x", "description": "Set up, upgrade and check the Beyond10x plugins and binaries, route work to them, and create portable plugins.", - "version": "0.14.17", + "version": "0.15.0", "author": { "name": "Beyond10x" }, diff --git a/plugins/b10x/.codex-plugin/plugin.json b/plugins/b10x/.codex-plugin/plugin.json index 7bb7188..1fb1196 100644 --- a/plugins/b10x/.codex-plugin/plugin.json +++ b/plugins/b10x/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "b10x", - "version": "0.14.17", + "version": "0.15.0", "description": "Set up, upgrade and check the Beyond10x plugins and binaries, route work to them, and create portable plugins.", "author": { "name": "Beyond10x" diff --git a/plugins/connectors/.claude-plugin/plugin.json b/plugins/connectors/.claude-plugin/plugin.json index 388fb08..0156c08 100644 --- a/plugins/connectors/.claude-plugin/plugin.json +++ b/plugins/connectors/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "connectors", - "version": "0.14.17", + "version": "0.15.0", "description": "Set up, inspect, and invoke governed integrations through the connectors CLI.", "author": { "name": "Beyond10x" }, "license": "Apache-2.0", diff --git a/plugins/connectors/.codex-plugin/plugin.json b/plugins/connectors/.codex-plugin/plugin.json index 7ce47b0..aa56e3e 100644 --- a/plugins/connectors/.codex-plugin/plugin.json +++ b/plugins/connectors/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "connectors", - "version": "0.14.17", + "version": "0.15.0", "description": "Set up, inspect, and invoke governed integrations through the connectors CLI.", "author": { "name": "Beyond10x" }, "license": "Apache-2.0", diff --git a/plugins/ess/.claude-plugin/plugin.json b/plugins/ess/.claude-plugin/plugin.json index 27c7d3b..a7eaab6 100644 --- a/plugins/ess/.claude-plugin/plugin.json +++ b/plugins/ess/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "ess", "displayName": "ESS", "description": "Write, retrofit, validate and project Executable System Specifications, and hold implementations to them with conformance suites.", - "version": "0.14.17", + "version": "0.15.0", "author": { "name": "Beyond10x" }, diff --git a/plugins/ess/.codex-plugin/plugin.json b/plugins/ess/.codex-plugin/plugin.json index 387705b..1a4aac3 100644 --- a/plugins/ess/.codex-plugin/plugin.json +++ b/plugins/ess/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "ess", - "version": "0.14.17", + "version": "0.15.0", "description": "Write, retrofit, validate and project Executable System Specifications, and hold implementations to them with conformance suites.", "author": { "name": "Beyond10x" diff --git a/plugins/worktree/.claude-plugin/plugin.json b/plugins/worktree/.claude-plugin/plugin.json index a4d648f..2b7b831 100644 --- a/plugins/worktree/.claude-plugin/plugin.json +++ b/plugins/worktree/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "worktree", "displayName": "Worktree", "description": "Create, lease, finish, audit and safely clean isolated Git worktrees through the worktree CLI.", - "version": "0.14.17", + "version": "0.15.0", "author": { "name": "Beyond10x" }, diff --git a/plugins/worktree/.codex-plugin/plugin.json b/plugins/worktree/.codex-plugin/plugin.json index 02d73a7..9cf1b80 100644 --- a/plugins/worktree/.codex-plugin/plugin.json +++ b/plugins/worktree/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "worktree", - "version": "0.14.17", + "version": "0.15.0", "description": "Create, lease, finish, audit and safely clean isolated Git worktrees through the worktree CLI.", "author": { "name": "Beyond10x"