From a4bf1d2abf31efa35667d109c4f89ee835888e8e Mon Sep 17 00:00:00 2001 From: "b10x-bot[bot]" <316511680+b10x-bot[bot]@users.noreply.github.com> Date: Fri, 25 Sep 2026 23:26:51 +0200 Subject: [PATCH] feat: trial definitions, full-package ESS trial and measured baseline; release 0.14.9 - trials//trial.yaml: prompt, fixture, setup, measures and expected outputs for seven trials; `task trial:run TRIAL=` prepares the sandbox and runs one; the gate checks the definitions (agentplugins-check trials.rs, trial-prepare). - ess-full-package: every ESS output, an in-memory Go implementation and `go test` against the generated suite; sandboxes get go. - agentplugins-check trial-report (report.rs): tool calls, validate, synthesis, UNMAPPED markers from disk, outputs present, go test counts; exits 1 against trials/baseline.json when a measure gets worse. trial:run calls it after trial-isolation. - improving-by-trial and AGENTS.md describe the definitions, TRIAL= and the baseline. task check: 101 tests passed. --- .agents/skills/improving-by-trial/SKILL.md | 71 +- AGENTS.md | 3 +- CHANGELOG.md | 18 + Cargo.lock | 4 +- Cargo.toml | 2 +- Taskfile.yml | 34 +- crates/agentplugins-check/src/main.rs | 115 +++ crates/agentplugins-check/src/report.rs | 862 ++++++++++++++++++ crates/agentplugins-check/src/trials.rs | 486 ++++++++++ plugins/aep/.claude-plugin/plugin.json | 2 +- plugins/aep/.codex-plugin/plugin.json | 2 +- plugins/b10x/.claude-plugin/plugin.json | 2 +- plugins/b10x/.codex-plugin/plugin.json | 2 +- plugins/connectors/.claude-plugin/plugin.json | 2 +- plugins/connectors/.codex-plugin/plugin.json | 2 +- plugins/ess/.claude-plugin/plugin.json | 2 +- plugins/ess/.codex-plugin/plugin.json | 2 +- plugins/worktree/.claude-plugin/plugin.json | 2 +- plugins/worktree/.codex-plugin/plugin.json | 2 +- trials/aep-backlog/fixture/README.md | 4 + trials/aep-backlog/fixture/TODO.md | 20 + trials/aep-backlog/trial.yaml | 15 + trials/baseline.json | 1 + trials/ess-full-package/trial.yaml | 34 + trials/ess-new/trial.yaml | 15 + .../fixture/spec/domains/booking.yaml | 177 ++++ trials/ess-pipeline/fixture/spec/system.yaml | 10 + trials/ess-pipeline/trial.yaml | 21 + trials/ess-retrofit/fixture/README.md | 16 + trials/ess-retrofit/fixture/go.mod | 3 + trials/ess-retrofit/fixture/main.go | 76 ++ trials/ess-retrofit/fixture/shed.go | 165 ++++ trials/ess-retrofit/trial.yaml | 15 + trials/upgrade-seeded/trial.yaml | 17 + trials/worktree-onboarding/fixture/README.md | 8 + .../worktree-onboarding/fixture/pancakes.txt | 5 + trials/worktree-onboarding/trial.yaml | 16 + 37 files changed, 2197 insertions(+), 36 deletions(-) create mode 100644 crates/agentplugins-check/src/report.rs create mode 100644 crates/agentplugins-check/src/trials.rs create mode 100644 trials/aep-backlog/fixture/README.md create mode 100644 trials/aep-backlog/fixture/TODO.md create mode 100644 trials/aep-backlog/trial.yaml create mode 100644 trials/baseline.json create mode 100644 trials/ess-full-package/trial.yaml create mode 100644 trials/ess-new/trial.yaml create mode 100644 trials/ess-pipeline/fixture/spec/domains/booking.yaml create mode 100644 trials/ess-pipeline/fixture/spec/system.yaml create mode 100644 trials/ess-pipeline/trial.yaml create mode 100644 trials/ess-retrofit/fixture/README.md create mode 100644 trials/ess-retrofit/fixture/go.mod create mode 100644 trials/ess-retrofit/fixture/main.go create mode 100644 trials/ess-retrofit/fixture/shed.go create mode 100644 trials/ess-retrofit/trial.yaml create mode 100644 trials/upgrade-seeded/trial.yaml create mode 100644 trials/worktree-onboarding/fixture/README.md create mode 100644 trials/worktree-onboarding/fixture/pancakes.txt create mode 100644 trials/worktree-onboarding/trial.yaml diff --git a/.agents/skills/improving-by-trial/SKILL.md b/.agents/skills/improving-by-trial/SKILL.md index 1ae3132..35d1e18 100644 --- a/.agents/skills/improving-by-trial/SKILL.md +++ b/.agents/skills/improving-by-trial/SKILL.md @@ -17,22 +17,44 @@ task trial:sandbox NAME= This creates `/var/tmp/b10x-trials-$USER//` (the Taskfile's `TRIALS`) with its own `home/`, a `b10x` built from this checkout, and an `env` file. The `env` file sets `HOME` to the sandbox and `B10X_MARKETPLACE` to this checkout, so `b10x init` installs the plugins as they are in the working tree, before any -release. It also unsets `ANTHROPIC_API_KEY`, `TMPDIR` and `TMPPREFIX`. Put the fixture the trial -needs (a small service, a `TODO.md`, an OpenAPI document) under `work/`, and commit it there with -`git` when the trial needs history or a remote. +release. It also unsets `ANTHROPIC_API_KEY`, `TMPDIR` and `TMPPREFIX`. A defined trial +(`task trial:run TRIAL=`, § 2) builds its own sandbox and fixture; for an ad-hoc one, put the +fixture it needs (a small service, a `TODO.md`, an OpenAPI document) under `work/`, and commit it +there with `git` when the trial needs history or a remote. To trial the released version instead, remove `home/.local/bin/b10x` and unset `B10X_MARKETPLACE` in `env`; the agent then follows `SETUP.md` from the release. ## 2. Run it headless, never as a sub-agent +The round's trials are defined in `trials/`, one directory each: + +| file | holds | +|---|---| +| `trials//trial.yaml` | `name`, `kind`, `prompt`, optional `dir` (the run's subdirectory of `work/`), `fixture` (a directory beside it), `remote` (a bare `origin` in the sandbox), `seeded`, `setup` (shell lines run from the sandbox root with its `env` before the run), `measures`, and `outputs` (name → path below the run's directory) | +| `trials//fixture/` | the small service or backlog the trial starts from | +| `trials/baseline.json` | the measures of the last accepted run of each trial | + +`task check` validates every definition. Run one by name: + ```console -task trial:run NAME= PROMPT='' [DIR=] +task trial:run TRIAL= +``` + +This builds a fresh sandbox (`trial:sandbox`), copies the fixture into `work/` and commits it +there (`agentplugins-check trial-prepare`), runs the definition's `setup` (installing the plugins +under test with `b10x init … --out plan.json` and `b10x setup apply --plan plan.json --yes`, or +seeding an older release), then runs the prompt. An ad-hoc trial still runs in an existing sandbox: + +```console +task trial:run NAME= PROMPT='' [DIR=] [SEEDED=true] ``` The task copies the operator's credentials in (mode 600), runs `claude -p` with the sandbox's `env`, `--strict-mcp-config` (no MCP server, including the account's claude.ai connectors) and -stream-json output into `run.jsonl`, and deletes the credentials when it finishes. +stream-json output into `run.jsonl`, and deletes the credentials when it finishes. The sandbox +`PATH` has `cargo` and `go`, and `go` is an allowed tool, so a trial can build and test an +implementation. **Never run a trial as a sub-agent of the working session.** A sub-agent gets the session's own agents and plugins whatever `HOME` its shell uses. In trial 3 the aep critics that ran were this @@ -44,7 +66,7 @@ confused it, and pasting the final command output verbatim. Give it no other hin ## 3. Only an isolated run counts -`trial:run` ends with: +`trial:run` checks isolation before it measures: ```console agentplugins-check trial-isolation /run.jsonl --sandbox --version @@ -62,15 +84,37 @@ trial 3, sandboxes under `~/.cache` loaded the operator's `~/.claude/CLAUDE.md` its commit rules) and a `.claude/settings.local.json` an earlier trial left in `~/.cache`, which enabled `aep@b10x`. `trial:sandbox` refuses to build below any of those files; that is why `TRIALS` is outside `$HOME`. Neither leak shows in the `init` event, so the placement is the only guard. An upgrade trial seeds older plugins on -purpose: run it with `SEEDED=true`, so plugins must match what the sandbox was seeded with. +purpose (`seeded: true` in its definition, or `SEEDED=true`), so plugins must match what the sandbox +was seeded with; a registry that did not exist before the run is read after it. ## 4. Read the run -From `run.jsonl`: +`trial:run` ends with the numbers, one line each: + +```console +agentplugins-check trial-report /run.jsonl --trial [--baseline trials/baseline.json] [--write-baseline] +``` + +| measure | read from | +|---|---| +| `tool_calls` | `tool_use` blocks in the run | +| `validate` | the last `ess specify validate` the run ran: `valid`, not valid, or not run | +| `synthesis` | `N scenario(s) … M refusal(s)` in the last `ess verify conform synthesize` output | +| `unmapped` | `UNMAPPED:` markers in the YAML files the run wrote, read from disk, not from its prose | +| `outputs` | which of the definition's `outputs` exist (a directory counts when it is not empty) | +| `go_test` | passed, failed and skipped tests of the last `go test` (`-v` or `-json`); a package that does not build counts as a failure | + +A trial reports the measures its definition lists; without `--trial`, an ad-hoc run gets every +measure but `outputs`. With `--baseline` it exits 1 when a measure got worse than the trial's entry: +validate stops passing, refusals or `go test` failures go up, a synthesis or `go test` that ran no +longer runs, an output the baseline had is missing, or tool calls rise by more than 50%. Other +changes print and pass. `--write-baseline` records the run as the trial's entry; do that for an +isolated run the round accepts, and commit `trials/baseline.json` with the fixes it led to. + +The numbers say what happened, not why. From `run.jsonl`, also collect: | collect | how | |---|---| -| tool calls | count of `tool_use` blocks | | stuck points | every refusal and error in `tool_result`s, verbatim | | guesses | what the final report says it inferred or could not find | | waste | calls repeated, files read twice, commands that failed and were retried unchanged | @@ -108,9 +152,12 @@ and the fix is in a release, not when the issue closes. passes when the finding does not recur. New findings go back to step 5. 4. Release when every trial of the round passes. -Each round has 3 ESS trials (a new specification; a retrofit of an existing service; generate plus -a synthesized suite), and 1 trial each for aep planning, worktree, and onboarding or upgrade. -Change the domain each round so the agents cannot copy the previous answer from the skills. +Each round runs every trial in `trials/`: 4 ESS trials (`ess-new`, a new specification; +`ess-retrofit`, an existing service; `ess-pipeline`, generation plus a synthesized suite; +`ess-full-package`, every output plus a Go implementation held to the synthesized suite), and +`aep-backlog`, `worktree-onboarding` and `upgrade-seeded`. Change the domains and fixtures each +round so the agents cannot copy the previous answer from the skills; a changed trial starts a new +baseline entry. ## 7. Clean up diff --git a/AGENTS.md b/AGENTS.md index 5fb6710..d36b61f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,6 +16,7 @@ CLIs. Serves O2 (decisions as data) and O3 (any harness). | `website/` | public docs; must pass `task site-build` | | `SETUP.md` | agent bootstrap, published as a release asset | | `.agents/skills/improving-by-trial/` | how plugins are improved: isolated headless trials (`task trial:sandbox`, `task trial:run`), triage, fix, re-run | +| `trials/` | the round's trial definitions (`/trial.yaml`, fixtures) and `baseline.json`; `task trial:run TRIAL=` runs one, `agentplugins-check trial-report` measures it against the baseline | ## Rules @@ -34,7 +35,7 @@ CLIs. Serves O2 (decisions as data) and O3 (any harness). ## Gate ```console -task check # fmt, clippy, tests, agentplugins-check +task check # fmt, clippy, tests, agentplugins-check (includes the trial definitions) task site-build # when website/ changes cargo run --locked --bin agentplugins-check -- tools # network: every spelled command exists in the newest aep, ess, worktree ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index c9ebf13..f36e352 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ # Changelog +## [0.14.9] — 2026-09-25 + +Trials become measurable: their prompts live in the repository, and each run is scored against a +committed baseline, so a drop in quality fails instead of waiting to be noticed. + +- `trials//trial.yaml` defines each trial (prompt, fixture, setup, measures, expected + outputs); `task trial:run TRIAL=` builds the sandbox, prepares the fixture and runs it. Seven + trials: ESS new spec, retrofit, pipeline and full package; aep backlog planning; worktree + onboarding; upgrade from a seeded install. The gate checks the definitions. +- New trial `ess-full-package`: every ESS output (JSON Schema, OpenAPI, AsyncAPI, docs and site, + Rust and Go type libraries, the Go conformance suite), then an in-memory Go implementation run + with `go test` against the generated suite. Sandboxes get `go`. +- `agentplugins-check trial-report` measures a run: tool calls, the last `validate` verdict, + synthesis scenarios and refusals, `UNMAPPED:` markers read from the files written, which outputs + exist, and `go test` passed/failed/skipped. With `--baseline trials/baseline.json` it exits 1 when + a measure gets worse; `--write-baseline` records an accepted run. `trial:run` calls it after the + isolation check. + ## [0.14.8] — 2026-09-25 `b10x-harness` joins the catalog, and `b10x` chooses a prebuilt archive only where one exists for diff --git a/Cargo.lock b/Cargo.lock index 49671d9..208b1c7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4,7 +4,7 @@ version = 4 [[package]] name = "agentplugins-check" -version = "0.14.8" +version = "0.14.9" dependencies = [ "clap", "serde", @@ -76,7 +76,7 @@ dependencies = [ [[package]] name = "b10x" -version = "0.14.8" +version = "0.14.9" dependencies = [ "clap", "serde", diff --git a/Cargo.toml b/Cargo.toml index 064f917..33f2baf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = ["crates/agentplugins-check", "crates/b10x"] [workspace.package] -version = "0.14.8" +version = "0.14.9" edition = "2021" rust-version = "1.85" license = "Apache-2.0" diff --git a/Taskfile.yml b/Taskfile.yml index f0305fb..41b91e8 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -45,7 +45,7 @@ tasks: - rm -rf '{{.T}}' - mkdir -p -m 700 '{{.TRIALS}}' - mkdir -p '{{.T}}/home/.local/bin' '{{.T}}/work' '{{.T}}/tools' - - for t in claude codex; do p="$(command -v $t)" && ln -sf "$(readlink -f "$p")" '{{.T}}/tools/'$t; done; true + - for t in claude codex go gofmt; do p="$(command -v $t)" && ln -sf "$(readlink -f "$p")" '{{.T}}/tools/'$t; done; true - for t in cargo rustc rustup; do ln -sf "$HOME/.cargo/bin/$t" '{{.T}}/tools/'$t; done - echo '{}' > '{{.T}}/home/.claude.json' - nice -n 19 env CARGO_BUILD_JOBS=4 cargo build --quiet --locked -p b10x @@ -62,28 +62,42 @@ tasks: - echo '{{.T}}' trial:run: - desc: "Run one headless trial in a sandbox and check its isolation: task trial:run NAME= PROMPT='…' [DIR=] [SEEDED=true for an upgrade trial]." - requires: - vars: [NAME, PROMPT] + desc: "Run one headless trial, check its isolation and measure it: task trial:run TRIAL= (in a fresh sandbox), or task trial:run NAME= PROMPT='…' [DIR=] [SEEDED=true for an upgrade trial] in an existing one." vars: + NAME: '{{.NAME | default .TRIAL}}' T: '{{.TRIALS}}/{{.NAME}}' DIR: '{{.DIR | default ""}}' - SEEDED: '{{if eq (.SEEDED | default "") "true"}}--seeded{{end}}' VERSION: sh: sed -n 's/^version = "\(.*\)"$/\1/p' Cargo.toml | head -1 + preconditions: + - sh: test -n '{{.TRIAL}}' || { test -n '{{.NAME}}' && test -n "$PROMPT"; } + msg: "set TRIAL= (a definition under trials/), or NAME= and PROMPT='…'" cmds: - - cp '{{.T}}/home/.claude/plugins/installed_plugins.json' '{{.T}}/registry-at-start.json' + - task: trial:sandbox + vars: { NAME: '{{.NAME}}' } + if: test -n '{{.TRIAL}}' + - cmd: cargo run --quiet --locked --bin agentplugins-check -- trial-prepare '{{.TRIAL}}' --sandbox '{{.T}}' + if: test -n '{{.TRIAL}}' + - cmd: | + printf '%s' "$PROMPT" > '{{.T}}/prompt.txt' + printf '%s' '{{.DIR}}' > '{{.T}}/workdir' + rm -f '{{.T}}/setup.sh' '{{.T}}/seeded' + if [ '{{.SEEDED}}' = true ]; then : > '{{.T}}/seeded'; fi + if: test -z '{{.TRIAL}}' + - if [ -f '{{.T}}/setup.sh' ]; then (. '{{.T}}/env' && cd '{{.T}}' && . ./setup.sh); fi + - rm -f '{{.T}}/registry-at-start.json'; if [ -f '{{.T}}/home/.claude/plugins/installed_plugins.json' ]; then cp '{{.T}}/home/.claude/plugins/installed_plugins.json' '{{.T}}/registry-at-start.json'; fi - install -m 600 "$HOME/.claude/.credentials.json" '{{.T}}/home/.claude/.credentials.json' - defer: rm -f '{{.T}}/home/.claude/.credentials.json' - | - (. '{{.T}}/env' && mkdir -p '{{.T}}/work/{{.DIR}}' && cd '{{.T}}/work/{{.DIR}}' && - timeout 2400 claude -p "$PROMPT" --permission-mode acceptEdits \ - --allowedTools 'Bash(b10x:*)' 'Bash(ess:*)' 'Bash(aep:*)' 'Bash(worktree:*)' 'Bash(git:*)' \ + (. '{{.T}}/env' && dir="$(cat '{{.T}}/workdir')" && mkdir -p "{{.T}}/work/$dir" && cd "{{.T}}/work/$dir" && + timeout 2400 claude -p "$(cat '{{.T}}/prompt.txt')" --permission-mode acceptEdits \ + --allowedTools 'Bash(b10x:*)' 'Bash(ess:*)' 'Bash(aep:*)' 'Bash(worktree:*)' 'Bash(git:*)' 'Bash(go:*)' \ 'Bash(mkdir:*)' 'Bash(ls:*)' 'Bash(cat:*)' 'Bash(find:*)' 'Bash(curl:*)' 'Bash(tar:*)' \ 'Bash(sha256sum:*)' 'Bash(mv:*)' 'Bash(chmod:*)' Read Write Edit Glob Grep Skill Task Agent \ --strict-mcp-config --output-format stream-json --verbose > '{{.T}}/run.jsonl' 2> '{{.T}}/run.err' echo "EXIT=$?" >> '{{.T}}/run.err') - - cargo run --quiet --locked --bin agentplugins-check -- trial-isolation '{{.T}}/run.jsonl' --sandbox '{{.T}}' --marketplace '{{.ROOT_DIR}}' --version '{{.VERSION}}' {{.SEEDED}} + - cargo run --quiet --locked --bin agentplugins-check -- trial-isolation '{{.T}}/run.jsonl' --sandbox '{{.T}}' --marketplace '{{.ROOT_DIR}}' --version '{{.VERSION}}' $([ -f '{{.T}}/seeded' ] && echo --seeded) + - cargo run --quiet --locked --bin agentplugins-check -- trial-report '{{.T}}/run.jsonl' --sandbox '{{.T}}' {{if .TRIAL}}--trial '{{.TRIAL}}' --baseline '{{.ROOT_DIR}}/trials/baseline.json'{{end}} env: PROMPT: '{{.PROMPT}}' diff --git a/crates/agentplugins-check/src/main.rs b/crates/agentplugins-check/src/main.rs index 5d3fa62..77bba12 100644 --- a/crates/agentplugins-check/src/main.rs +++ b/crates/agentplugins-check/src/main.rs @@ -8,8 +8,10 @@ use clap::{Parser, Subcommand}; mod concept; mod evals; mod readiness; +mod report; mod tools; mod trial; +mod trials; /// The marketplace identity in every marketplace format. const MARKETPLACE: &str = "b10x"; @@ -928,6 +930,7 @@ fn check(root: &Path) -> Result<(), String> { remote: std::collections::BTreeSet::new(), }, )?; + trials::check(root)?; evals::evals(root) } @@ -997,6 +1000,83 @@ enum Top { #[arg(long)] seeded: bool, }, + /// Prepare a fresh sandbox for a trial defined under `trials/`: copy and commit its fixture, + /// and write the prompt, working directory and setup `task trial:run` reads. + TrialPrepare { + /// The trial's name (its directory under `trials/`). + trial: String, + /// The sandbox directory made by `task trial:sandbox`. + #[arg(long)] + sandbox: PathBuf, + }, + /// Measure a trial run: tool calls, validation, synthesis, `UNMAPPED:` markers, outputs and + /// `go test` counts, one line each; with `--baseline`, exit 1 when a measure got worse. + TrialReport { + /// The run's stream-json output. + run: PathBuf, + /// The trial under `trials/` the run executed; without it, an ad-hoc `PROMPT=` run gets + /// every measure that needs no definition. + #[arg(long)] + trial: Option, + /// The sandbox directory; default: the directory holding the run. + #[arg(long)] + sandbox: Option, + /// Compare with this baseline (`trials/baseline.json`) and exit 1 when a measure got worse. + #[arg(long, requires = "trial")] + baseline: Option, + /// Record this run as the trial's baseline, in `--baseline` or `trials/baseline.json`. + #[arg(long, requires = "trial")] + write_baseline: bool, + }, +} + +/// `trial-report`: print the measures, compare them with the baseline, and record them. +fn trial_report( + root: &Path, + run: &Path, + trial: Option<&str>, + sandbox: Option, + baseline: Option, + write_baseline: bool, +) -> Result { + let definition = trial.map(|name| trials::load(root, name)).transpose()?; + let text = std::fs::read_to_string(run) + .map_err(|error| format!("reading {}: {error}", run.display()))?; + let sandbox = sandbox + .or_else(|| run.parent().map(Path::to_path_buf)) + .ok_or("no sandbox: pass --sandbox")?; + let report = report::measure(&text, &sandbox, definition.as_ref())?; + println!("trial {}: {}", trial.unwrap_or("(ad hoc)"), run.display()); + for line in &report.lines { + println!(" {line}"); + } + let mut held = true; + if let (Some(trial), Some(path)) = (trial, baseline.as_deref()) { + match report::read_baseline(path)?.get(trial) { + None => println!( + "baseline: none recorded for `{trial}` in {}", + path.display() + ), + Some(then) => { + let worse = report::worse(then, &report.measures); + if worse.is_empty() { + println!("baseline: no measure got worse than {}", path.display()); + } else { + held = write_baseline; + eprintln!("worse than the baseline in {}:", path.display()); + for line in worse { + eprintln!(" {line}"); + } + } + } + } + } + if let (Some(trial), true) = (trial, write_baseline) { + let path = baseline.unwrap_or_else(|| root.join(trials::BASELINE)); + report::write_baseline(&path, trial, &report.measures)?; + println!("baseline: recorded `{trial}` in {}", path.display()); + } + Ok(held) } #[derive(Debug, Subcommand)] @@ -1040,6 +1120,41 @@ fn main() -> ExitCode { } }; } + Some(Top::TrialPrepare { trial, sandbox }) => { + return match trials::prepare(&root, &trial, &sandbox) { + Ok(summary) => { + println!("{summary}"); + ExitCode::SUCCESS + } + Err(error) => { + eprintln!("error: {error}"); + ExitCode::from(1) + } + }; + } + Some(Top::TrialReport { + run, + trial, + sandbox, + baseline, + write_baseline, + }) => { + return match trial_report( + &root, + &run, + trial.as_deref(), + sandbox, + baseline, + write_baseline, + ) { + Ok(true) => ExitCode::SUCCESS, + Ok(false) => ExitCode::from(1), + Err(error) => { + eprintln!("error: {error}"); + ExitCode::from(1) + } + }; + } }; match result { diff --git a/crates/agentplugins-check/src/report.rs b/crates/agentplugins-check/src/report.rs new file mode 100644 index 0000000..d97d21c --- /dev/null +++ b/crates/agentplugins-check/src/report.rs @@ -0,0 +1,862 @@ +//! What a headless trial run achieved, in numbers, and whether they got worse than the baseline. +//! +//! Until this existed a run was read by hand and nothing compared one round with the last. Every +//! number here comes from the run's own stream-json (tool calls and the output of the commands it +//! ran) or from the files it left in the sandbox, never from what the agent's final prose claims: +//! a `UNMAPPED:` marker counts when it is in a spec file on disk. + +use std::collections::{BTreeMap, BTreeSet, HashSet}; +use std::fmt::Write as _; +use std::path::{Path, PathBuf}; + +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +use crate::trials::{Definition, Measure}; + +/// A Bash call and what it printed. +struct Shell { + command: String, + output: Option, +} + +/// What the report reads from a stream-json run. +struct Run { + tool_calls: usize, + shells: Vec, + /// Paths given to `Write`, `Edit` and `MultiEdit`, in first-write order. + written: Vec, +} + +fn result_text(content: &Value) -> String { + match content { + Value::String(text) => text.clone(), + Value::Array(blocks) => blocks + .iter() + .filter_map(|block| block["text"].as_str()) + .collect::>() + .join("\n"), + _ => String::new(), + } +} + +fn parse(text: &str) -> Run { + let mut seen = HashSet::new(); + let mut tool_calls = 0; + let mut shells = Vec::new(); + let mut by_id = BTreeMap::new(); + let mut written = Vec::new(); + for line in text.lines() { + let Ok(event) = serde_json::from_str::(line) else { + continue; + }; + for block in event["message"]["content"].as_array().into_iter().flatten() { + match block["type"].as_str() { + Some("tool_use") => { + let id = block["id"].as_str().unwrap_or_default().to_owned(); + if !id.is_empty() && !seen.insert(id.clone()) { + continue; + } + tool_calls += 1; + let input = &block["input"]; + match block["name"].as_str() { + Some("Bash") => { + by_id.insert(id, shells.len()); + shells.push(Shell { + command: input["command"].as_str().unwrap_or_default().to_owned(), + output: None, + }); + } + Some("Write" | "Edit" | "MultiEdit") => { + if let Some(path) = input["file_path"].as_str() { + let path = PathBuf::from(path); + if !written.contains(&path) { + written.push(path); + } + } + } + _ => {} + } + } + Some("tool_result") => { + let id = block["tool_use_id"].as_str().unwrap_or_default(); + if let Some(&index) = by_id.get(id) { + shells[index].output = Some(result_text(&block["content"])); + } + } + _ => {} + } + } + } + Run { + tool_calls, + shells, + written, + } +} + +/// Whether `ess specify validate` ran, and what its last output said. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum Validate { + /// Never ran. + NotRun, + /// Its last output was not `valid`. + Invalid, + /// Its last output said `valid`. + Valid, +} + +/// The counts of the last synthesis. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub struct Synthesis { + /// Scenarios synthesized. + pub scenarios: u64, + /// Refusals reported. + pub refusals: u64, +} + +/// The counts of the last `go test`. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +pub struct GoTest { + /// Tests that passed. + pub passed: u64, + /// Tests that failed, and packages that did not build. + pub failed: u64, + /// Tests that were skipped. + pub skipped: u64, +} + +/// One run's numbers; a field is present exactly when the trial measures it. The same shape is a +/// trial's entry in `trials/baseline.json`. +/// +/// `synthesis` and `go_test` have three states on purpose: absent (not measured), `null` (measured, +/// never ran) and counts; a baseline that had counts and a run with `null` got worse. +#[allow(clippy::option_option)] +#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Measures { + /// Tool calls. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tool_calls: Option, + /// `ess specify validate`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub validate: Option, + /// The last synthesis; `null` when none ran. + #[serde( + default, + skip_serializing_if = "Option::is_none", + deserialize_with = "measured" + )] + pub synthesis: Option>, + /// `UNMAPPED:` markers in spec files the run wrote. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub unmapped: Option, + /// The listed outputs that exist. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub outputs: Option>, + /// The last `go test`; `null` when none ran. + #[serde( + default, + skip_serializing_if = "Option::is_none", + deserialize_with = "measured" + )] + pub go_test: Option>, +} + +/// A field that is present is measured, even when its value is `null` (the command never ran); +/// an absent field is a measure the trial does not take. +#[allow(clippy::option_option)] +fn measured<'de, D, T>(deserializer: D) -> Result>, D::Error> +where + D: serde::Deserializer<'de>, + T: Deserialize<'de>, +{ + Option::::deserialize(deserializer).map(Some) +} + +fn is_validate(command: &str) -> bool { + command.contains("specify validate") || command.contains("ess validate") +} + +/// The verdict of one validate output: the last line that states one. +fn verdict(output: &str) -> Validate { + let mut verdict = Validate::Invalid; + for line in output.lines() { + let line = line.trim(); + if line == "valid" || line.ends_with(", valid") || line.contains("\"valid\": true") { + verdict = Validate::Valid; + } else if line.contains(" was refused") || line.contains("\"valid\": false") { + verdict = Validate::Invalid; + } + } + verdict +} + +/// The number right before `word` on `line`. +fn number_before(line: &str, word: &str) -> Option { + let before = &line[..line.find(word)?]; + let digits: String = before + .trim_end() + .chars() + .rev() + .take_while(char::is_ascii_digit) + .collect(); + digits.chars().rev().collect::().parse().ok() +} + +/// `N scenario(s) … M refusal(s)` from the last synthesis that printed it. +fn synthesis(run: &Run) -> Option { + let mut last = None; + for shell in run + .shells + .iter() + .filter(|s| s.command.contains("synthesize")) + { + for line in shell.output.as_deref().unwrap_or_default().lines() { + if let (Some(scenarios), Some(refusals)) = ( + number_before(line, "scenario(s)"), + number_before(line, "refusal(s)"), + ) { + last = Some(Synthesis { + scenarios, + refusals, + }); + } + } + } + last +} + +/// Counts of one `go test` output, from `-v` lines or `-json` events; `None` when it holds neither. +fn go_counts(output: &str) -> Option { + let mut results: BTreeMap = BTreeMap::new(); + let mut broken = 0; + for line in output.lines() { + let trimmed = line.trim(); + if let Ok(event) = serde_json::from_str::(trimmed) { + if let (Some(action), Some(test)) = (event["Action"].as_str(), event["Test"].as_str()) { + if let Some(status) = ["pass", "fail", "skip"].into_iter().find(|s| *s == action) { + results.insert(test.to_owned(), status); + } + } + continue; + } + for (prefix, status) in [ + ("--- PASS: ", "pass"), + ("--- FAIL: ", "fail"), + ("--- SKIP: ", "skip"), + ] { + if let Some(rest) = trimmed.strip_prefix(prefix) { + let name = rest.rsplit_once(" (").map_or(rest, |(name, _)| name); + results.insert(name.to_owned(), status); + } + } + if trimmed.starts_with("FAIL") + && (trimmed.ends_with("[build failed]") || trimmed.ends_with("[setup failed]")) + { + broken += 1; + } + } + if results.is_empty() && broken == 0 { + return None; + } + // A top-level test with subtests is their parent; its verdict repeats theirs. + let parents: BTreeSet<&str> = results + .keys() + .filter_map(|name| name.split_once('/').map(|(top, _)| top)) + .collect(); + let mut counts = GoTest { + failed: broken, + ..GoTest::default() + }; + for (name, status) in &results { + if !name.contains('/') && parents.contains(name.as_str()) { + continue; + } + match *status { + "pass" => counts.passed += 1, + "fail" => counts.failed += 1, + _ => counts.skipped += 1, + } + } + Some(counts) +} + +fn go_test(run: &Run) -> Option { + run.shells + .iter() + .filter(|shell| shell.command.contains("go test")) + .rev() + .find_map(|shell| go_counts(shell.output.as_deref()?)) +} + +/// `UNMAPPED:` markers in the YAML files the run wrote inside the sandbox, as they are on disk. +fn unmapped(run: &Run, sandbox: &Path, workdir: &Path) -> (u64, usize) { + let mut markers = 0; + let mut files = 0; + for path in &run.written { + let path = if path.is_absolute() { + path.clone() + } else { + workdir.join(path) + }; + let yaml = path + .extension() + .is_some_and(|extension| extension == "yaml" || extension == "yml"); + let inside = std::fs::canonicalize(&path).is_ok_and(|path| path.starts_with(sandbox)); + if !yaml || !inside { + continue; + } + if let Ok(text) = std::fs::read_to_string(&path) { + files += 1; + markers += text.matches("UNMAPPED:").count() as u64; + } + } + (markers, files) +} + +fn present(path: &Path) -> bool { + if path.is_dir() { + std::fs::read_dir(path).is_ok_and(|mut entries| entries.next().is_some()) + } else { + path.is_file() + } +} + +/// A run measured, with the lines that say so. +pub struct Report { + /// The numbers. + pub measures: Measures, + /// One line per measure. + pub lines: Vec, +} + +/// Measure a run. `definition` is `None` for an ad-hoc `PROMPT=` run, which gets every measure +/// that needs no definition. +pub fn measure( + text: &str, + sandbox: &Path, + definition: Option<&Definition>, +) -> Result { + let sandbox = std::fs::canonicalize(sandbox) + .map_err(|error| format!("sandbox {}: {error}", sandbox.display()))?; + let wants = |measure: Measure| { + definition.map_or(measure != Measure::Outputs, |definition| { + definition.measures(measure) + }) + }; + let workdir = definition.map_or_else(|| sandbox.join("work"), |d| sandbox.join(d.workdir())); + let run = parse(text); + if run.tool_calls == 0 && !text.lines().any(|line| line.contains("\"type\"")) { + return Err("no stream-json events: not a trial run".to_owned()); + } + let mut measures = Measures::default(); + let mut lines = Vec::new(); + + measures.tool_calls = Some(run.tool_calls as u64); + lines.push(format!("tool calls: {}", run.tool_calls)); + + if wants(Measure::Validate) { + let runs: Vec<&Shell> = run + .shells + .iter() + .filter(|s| is_validate(&s.command)) + .collect(); + let validate = runs.last().map_or(Validate::NotRun, |shell| { + verdict(shell.output.as_deref().unwrap_or_default()) + }); + measures.validate = Some(validate); + lines.push(match validate { + Validate::NotRun => "validate: not run".to_owned(), + Validate::Invalid => format!("validate: not valid (last of {} run(s))", runs.len()), + Validate::Valid => format!("validate: valid (last of {} run(s))", runs.len()), + }); + } + if wants(Measure::Synthesis) { + let synthesis = synthesis(&run); + measures.synthesis = Some(synthesis); + lines.push(synthesis.map_or_else( + || "synthesis: not run".to_owned(), + |s| { + format!( + "synthesis: {} scenario(s), {} refusal(s)", + s.scenarios, s.refusals + ) + }, + )); + } + if wants(Measure::Unmapped) { + let (markers, files) = unmapped(&run, &sandbox, &workdir); + measures.unmapped = Some(markers); + lines.push(format!( + "unmapped: {markers} `UNMAPPED:` marker(s) in {files} YAML file(s) the run wrote" + )); + } + if let Some(definition) = definition.filter(|d| d.measures(Measure::Outputs)) { + let mut have = BTreeSet::new(); + let mut missing = Vec::new(); + for (name, path) in &definition.outputs { + if present(&workdir.join(path)) { + have.insert(name.clone()); + } else { + missing.push(format!("{name} ({path})")); + } + } + let mut line = format!( + "outputs: {}/{} present", + have.len(), + definition.outputs.len() + ); + if !missing.is_empty() { + let _ = write!(line, "; missing: {}", missing.join(", ")); + } + lines.push(line); + measures.outputs = Some(have); + } + if wants(Measure::GoTest) { + let counts = go_test(&run); + measures.go_test = Some(counts); + lines.push(counts.map_or_else( + || "go test: not run".to_owned(), + |c| { + format!( + "go test: {} passed, {} failed, {} skipped", + c.passed, c.failed, c.skipped + ) + }, + )); + } + Ok(Report { measures, lines }) +} + +/// Every way `now` is worse than `then`. Only a measure both record is compared. +#[must_use] +pub fn worse(then: &Measures, now: &Measures) -> Vec { + let mut found = Vec::new(); + if let (Some(then), Some(now)) = (then.tool_calls, now.tool_calls) { + if now * 2 > then * 3 { + found.push(format!("tool calls: {then} → {now}, up by more than 50%")); + } + } + if let (Some(Validate::Valid), Some(now)) = (then.validate, now.validate) { + if now != Validate::Valid { + found.push(format!("validate: was valid, now {now:?}")); + } + } + if let (Some(Some(then)), Some(now)) = (then.synthesis, now.synthesis) { + match now { + None => found.push("synthesis: ran before, now did not".to_owned()), + Some(now) if now.refusals > then.refusals => found.push(format!( + "synthesis: refusals {} → {}", + then.refusals, now.refusals + )), + Some(_) => {} + } + } + if let (Some(then), Some(now)) = (&then.outputs, &now.outputs) { + let lost: Vec<&str> = then.difference(now).map(String::as_str).collect(); + if !lost.is_empty() { + found.push(format!("outputs: missing now: {}", lost.join(", "))); + } + } + if let (Some(Some(then)), Some(now)) = (then.go_test, now.go_test) { + match now { + None => found.push("go test: ran before, now did not".to_owned()), + Some(now) if now.failed > then.failed => found.push(format!( + "go test: failures {} → {}", + then.failed, now.failed + )), + Some(_) => {} + } + } + found +} + +/// Read `trials/baseline.json` (or another baseline file). +pub fn read_baseline(path: &Path) -> Result, String> { + let text = std::fs::read_to_string(path) + .map_err(|error| format!("reading {}: {error}", path.display()))?; + serde_json::from_str(&text).map_err(|error| format!("parsing {}: {error}", path.display())) +} + +/// Record `measures` as `trial`'s baseline in `path`, keeping every other trial's entry. +pub fn write_baseline(path: &Path, trial: &str, measures: &Measures) -> Result<(), String> { + let mut baseline = if path.exists() { + read_baseline(path)? + } else { + BTreeMap::new() + }; + baseline.insert(trial.to_owned(), measures.clone()); + let text = serde_json::to_string_pretty(&baseline).map_err(|error| error.to_string())?; + std::fs::write(path, text + "\n") + .map_err(|error| format!("writing {}: {error}", path.display())) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::trials::Kind; + + fn tool(id: &str, name: &str, input: &str) -> String { + format!( + r#"{{"type":"assistant","message":{{"content":[{{"type":"tool_use","id":"{id}","name":"{name}","input":{input}}}]}}}}"# + ) + } + + fn result(id: &str, text: &str) -> String { + let content = serde_json::to_string(text).unwrap(); + format!( + r#"{{"type":"user","message":{{"content":[{{"type":"tool_result","tool_use_id":"{id}","content":[{{"type":"text","text":{content}}}]}}]}}}}"# + ) + } + + fn bash(id: &str, command: &str, output: &str) -> String { + let input = serde_json::json!({ "command": command }).to_string(); + format!("{}\n{}", tool(id, "Bash", &input), result(id, output)) + } + + fn scratch(label: &str) -> PathBuf { + let path = std::env::temp_dir().join(format!( + "agentplugins-check-report-{label}-{}", + std::process::id() + )); + let _ = std::fs::remove_dir_all(&path); + std::fs::create_dir_all(path.join("work")).unwrap(); + std::fs::canonicalize(path).unwrap() + } + + fn definition(measures: &[Measure], outputs: &[(&str, &str)]) -> Definition { + Definition { + name: "demo".to_owned(), + kind: Kind::EssFullPackage, + prompt: "p".to_owned(), + dir: None, + fixture: None, + remote: false, + seeded: false, + setup: Vec::new(), + measures: measures.to_vec(), + outputs: outputs + .iter() + .map(|(name, path)| ((*name).to_owned(), (*path).to_owned())) + .collect(), + } + } + + const INIT: &str = r#"{"type":"system","subtype":"init"}"#; + + #[test] + fn validate_is_the_verdict_of_the_last_run() { + let sandbox = scratch("validate"); + let valid = bash( + "a", + "ess specify validate --path spec", + "garden v1 — 2 file(s), valid", + ); + let refused = bash( + "b", + "ess specify validate --path spec", + "spec was refused:\n - domains/plot.yaml: missing field `type`", + ); + let only = definition(&[Measure::ToolCalls, Measure::Validate], &[]); + let run = [INIT, &refused, &valid].join("\n"); + let report = measure(&run, &sandbox, Some(&only)).unwrap(); + assert_eq!(report.measures.validate, Some(Validate::Valid)); + assert_eq!(report.lines[1], "validate: valid (last of 2 run(s))"); + let run = [INIT, &valid, &refused].join("\n"); + let report = measure(&run, &sandbox, Some(&only)).unwrap(); + assert_eq!(report.measures.validate, Some(Validate::Invalid)); + let json = bash( + "c", + "ess specify validate --format json", + "{\n \"valid\": true\n}", + ); + let report = measure(&[INIT, &json].join("\n"), &sandbox, Some(&only)).unwrap(); + assert_eq!(report.measures.validate, Some(Validate::Valid)); + let report = measure(INIT, &sandbox, Some(&only)).unwrap(); + assert_eq!(report.measures.validate, Some(Validate::NotRun)); + std::fs::remove_dir_all(&sandbox).unwrap(); + } + + #[test] + fn synthesis_counts_come_from_the_last_synthesize_output() { + let sandbox = scratch("synthesis"); + let first = bash( + "a", + "ess verify conform synthesize --path spec --target go --out out/conformance", + "refused: refusal[ESS-SYNTH-011]: …\n12 scenario(s) (0 authored), 5 refusal(s), 5 file(s) written to out", + ); + let second = bash( + "b", + "cd spec && ess verify conform synthesize --target ir --out suite.json", + "14 scenario(s) (0 authored), 2 refusal(s), 1 file(s) written to suite.json", + ); + let unrelated = bash( + "c", + "echo '3 scenario(s), 9 refusal(s)'", + "3 scenario(s), 9 refusal(s)", + ); + let run = [INIT, &first, &second, &unrelated].join("\n"); + let report = measure(&run, &sandbox, None).unwrap(); + assert_eq!( + report.measures.synthesis, + Some(Some(Synthesis { + scenarios: 14, + refusals: 2 + })) + ); + assert!(report + .lines + .contains(&"synthesis: 14 scenario(s), 2 refusal(s)".to_owned())); + assert_eq!(report.measures.tool_calls, Some(3)); + std::fs::remove_dir_all(&sandbox).unwrap(); + } + + #[test] + fn unmapped_markers_are_read_from_the_files_on_disk() { + let sandbox = scratch("unmapped"); + let spec = sandbox.join("work/spec"); + std::fs::create_dir_all(&spec).unwrap(); + std::fs::write( + spec.join("loans.yaml"), + "# UNMAPPED: who may lend\n# UNMAPPED: overdue rule\ndomain: x\n", + ) + .unwrap(); + std::fs::write(spec.join("system.yaml"), "format: ess/1\n").unwrap(); + std::fs::write(spec.join("notes.md"), "UNMAPPED: not a spec\n").unwrap(); + let write = |id: &str, file: &str| { + let input = serde_json::json!({ "file_path": spec.join(file), "content": "UNMAPPED: in the prose only" }); + tool(id, "Write", &input.to_string()) + }; + let edit = tool( + "d", + "Edit", + &serde_json::json!({ "file_path": spec.join("loans.yaml") }).to_string(), + ); + let outside = tool( + "e", + "Write", + &serde_json::json!({ "file_path": "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/nonexistent/elsewhere.yaml" }).to_string(), + ); + let run = [ + INIT, + &write("a", "loans.yaml"), + &write("b", "system.yaml"), + &write("c", "notes.md"), + &edit, + &outside, + ] + .join("\n"); + let report = measure(&run, &sandbox, None).unwrap(); + assert_eq!(report.measures.unmapped, Some(2)); + assert!(report.lines.contains( + &"unmapped: 2 `UNMAPPED:` marker(s) in 2 YAML file(s) the run wrote".to_owned() + )); + assert_eq!(report.measures.tool_calls, Some(5)); + std::fs::remove_dir_all(&sandbox).unwrap(); + } + + #[test] + fn go_test_counts_leaf_tests_of_the_last_run() { + let sandbox = scratch("gotest"); + let verbose = "=== RUN TestConformance\n=== RUN TestConformance/garden.plot.Plot/assign/assigned\n --- PASS: TestConformance/garden.plot.Plot/assign/assigned (0.00s)\n --- PASS: TestConformance/garden.plot.Plot (0.00s)\n --- SKIP: TestConformance/garden.plot.Release/redeliver (0.00s)\n --- FAIL: TestConformance/garden.plot.Plot/invariant (0.01s)\n--- FAIL: TestConformance (0.02s)\n--- PASS: TestStore (0.00s)\nFAIL\nFAIL\texample.com/garden/impl\t0.031s"; + let broken = bash( + "a", + "go test ./...", + "# example.com/garden/impl\nimpl/target.go:9:2: undefined: essconform.Tagret\nFAIL\texample.com/garden/impl [build failed]", + ); + let listed = bash( + "c", + "go test -list . ./impl", + "TestConformance\nok \texample.com/garden/impl\t0.002s", + ); + let only = definition(&[Measure::ToolCalls, Measure::GoTest], &[]); + let run = [ + INIT, + &broken, + &bash("b", "cd impl && go test -v ./...", verbose), + &listed, + ] + .join("\n"); + let report = measure(&run, &sandbox, Some(&only)).unwrap(); + assert_eq!( + report.measures.go_test, + Some(Some(GoTest { + passed: 3, + failed: 1, + skipped: 1 + })) + ); + assert_eq!(report.lines[1], "go test: 3 passed, 1 failed, 1 skipped"); + let report = measure(&[INIT, &broken].join("\n"), &sandbox, Some(&only)).unwrap(); + assert_eq!( + report.measures.go_test, + Some(Some(GoTest { + passed: 0, + failed: 1, + skipped: 0 + })) + ); + let json = "{\"Action\":\"run\",\"Test\":\"TestConformance\"}\n{\"Action\":\"pass\",\"Test\":\"TestConformance/a\"}\n{\"Action\":\"skip\",\"Test\":\"TestConformance/b\"}\n{\"Action\":\"pass\",\"Test\":\"TestConformance\"}"; + let report = measure( + &[INIT, &bash("d", "go test -json ./...", json)].join("\n"), + &sandbox, + Some(&only), + ) + .unwrap(); + assert_eq!( + report.measures.go_test, + Some(Some(GoTest { + passed: 1, + failed: 0, + skipped: 1 + })) + ); + std::fs::remove_dir_all(&sandbox).unwrap(); + } + + #[test] + fn outputs_are_checked_on_disk_and_only_when_listed() { + let sandbox = scratch("outputs"); + let out = sandbox.join("work/out"); + std::fs::create_dir_all(out.join("schema")).unwrap(); + std::fs::write(out.join("schema/a.json"), "{}").unwrap(); + std::fs::create_dir_all(out.join("site")).unwrap(); + std::fs::write(out.join("openapi.yaml"), "openapi: 3.1.0").unwrap(); + let listed = definition( + &[Measure::ToolCalls, Measure::Outputs], + &[ + ("schema", "out/schema"), + ("openapi", "out/openapi.yaml"), + ("site", "out/site"), + ("asyncapi", "out/asyncapi"), + ], + ); + let report = measure(INIT, &sandbox, Some(&listed)).unwrap(); + assert_eq!( + report.measures.outputs, + Some(["openapi".to_owned(), "schema".to_owned()].into()) + ); + assert_eq!( + report.lines[1], + "outputs: 2/4 present; missing: asyncapi (out/asyncapi), site (out/site)" + ); + assert_eq!(report.lines.len(), 2, "unlisted measures print nothing"); + let adhoc = measure(INIT, &sandbox, None).unwrap(); + assert!(adhoc.measures.outputs.is_none()); + assert!(measure("not json", &sandbox, None).is_err()); + std::fs::remove_dir_all(&sandbox).unwrap(); + } + + #[test] + fn a_duplicated_tool_use_is_counted_once() { + let sandbox = scratch("dupes"); + let call = tool("same", "Read", "{}"); + let report = measure(&[INIT, &call, &call].join("\n"), &sandbox, None).unwrap(); + assert_eq!(report.measures.tool_calls, Some(1)); + std::fs::remove_dir_all(&sandbox).unwrap(); + } + + fn full() -> Measures { + Measures { + tool_calls: Some(40), + validate: Some(Validate::Valid), + synthesis: Some(Some(Synthesis { + scenarios: 12, + refusals: 2, + })), + unmapped: Some(0), + outputs: Some(["go".to_owned(), "schema".to_owned()].into()), + go_test: Some(Some(GoTest { + passed: 10, + failed: 0, + skipped: 2, + })), + } + } + + #[test] + fn the_same_or_better_is_not_worse() { + let mut better = full(); + better.tool_calls = Some(60); + better.unmapped = Some(4); + better.synthesis = Some(Some(Synthesis { + scenarios: 9, + refusals: 1, + })); + better.go_test = Some(Some(GoTest { + passed: 1, + failed: 0, + skipped: 11, + })); + assert!(worse(&full(), &full()).is_empty()); + assert!( + worse(&full(), &better).is_empty(), + "{:?}", + worse(&full(), &better) + ); + assert!(worse(&Measures::default(), &full()).is_empty()); + } + + #[test] + fn every_listed_regression_is_worse() { + let now = Measures { + tool_calls: Some(61), + validate: Some(Validate::NotRun), + synthesis: Some(Some(Synthesis { + scenarios: 12, + refusals: 3, + })), + unmapped: Some(0), + outputs: Some(["schema".to_owned()].into()), + go_test: Some(Some(GoTest { + passed: 9, + failed: 1, + skipped: 2, + })), + }; + let found = worse(&full(), &now); + assert_eq!(found.len(), 5, "{found:?}"); + for expected in [ + "tool calls: 40 → 61", + "validate: was valid, now NotRun", + "refusals 2 → 3", + "missing now: go", + "failures 0 → 1", + ] { + assert!( + found.iter().any(|line| line.contains(expected)), + "{expected}: {found:?}" + ); + } + let gone = Measures { + synthesis: Some(None), + go_test: Some(None), + ..full() + }; + let found = worse(&full(), &gone); + assert_eq!(found.len(), 2, "{found:?}"); + } + + #[test] + fn a_baseline_round_trips_and_keeps_other_trials() { + let sandbox = scratch("baseline"); + let path = sandbox.join("baseline.json"); + std::fs::write(&path, "{\"other\": {\"tool_calls\": 3}}\n").unwrap(); + write_baseline(&path, "demo", &full()).unwrap(); + let read = read_baseline(&path).unwrap(); + assert_eq!(read["demo"], full()); + assert_eq!(read["other"].tool_calls, Some(3)); + let text = std::fs::read_to_string(&path).unwrap(); + assert!(text.contains("\"validate\": \"valid\""), "{text}"); + let never = Measures { + synthesis: Some(None), + go_test: Some(None), + ..Measures::default() + }; + write_baseline(&path, "never", &never).unwrap(); + assert_eq!(read_baseline(&path).unwrap()["never"], never); + std::fs::remove_dir_all(&sandbox).unwrap(); + } +} diff --git a/crates/agentplugins-check/src/trials.rs b/crates/agentplugins-check/src/trials.rs new file mode 100644 index 0000000..eb77ce2 --- /dev/null +++ b/crates/agentplugins-check/src/trials.rs @@ -0,0 +1,486 @@ +//! Trial definitions: `trials//trial.yaml`, one per trial, with an optional fixture beside +//! it, and `trials/baseline.json`, the measures of the last accepted run of each. +//! +//! A trial prompt that lives in a throwaway script cannot be re-run next round or compared with the +//! last one. Here each is data the gate validates and `task trial:run TRIAL=` executes: +//! [`prepare`] copies the fixture into a fresh sandbox, commits it, and writes the prompt and the +//! setup the Taskfile runs. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use serde::Deserialize; + +/// The directory holding every trial, relative to the repository root. +pub const TRIALS: &str = "trials"; + +/// The committed baseline, relative to the repository root. +pub const BASELINE: &str = "trials/baseline.json"; + +/// What a trial exercises. A label for readers and for the round's coverage; it does not change +/// how a run is measured, which [`Definition::measures`] decides. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum Kind { + /// A new ESS specification from a sentence. + EssNew, + /// An ESS specification of an existing service (the fixture). + EssRetrofit, + /// Generation and a synthesized suite from an existing specification. + EssPipeline, + /// Every ESS output, and an implementation held to the synthesized Go suite. + EssFullPackage, + /// AEP planning from an existing backlog. + AepBacklog, + /// Setting up `worktree` in a repository. + WorktreeOnboarding, + /// Bringing a seeded older install current. + Upgrade, +} + +/// A number `agentplugins-check trial-report` reads from a run. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Measure { + /// Tool calls the run made. + ToolCalls, + /// Whether `ess specify validate` ran and its last output said `valid`. + Validate, + /// Scenarios and refusals from the last `ess verify conform synthesize`. + Synthesis, + /// `UNMAPPED:` markers in the spec files the run wrote, read from disk. + Unmapped, + /// Which of [`Definition::outputs`] exist on disk. + Outputs, + /// Passed, failed and skipped tests of the last `go test`. + GoTest, +} + +/// One `trials//trial.yaml`. +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Definition { + /// The trial's name; equal to its directory name and the sandbox name. + pub name: String, + /// What it exercises. + pub kind: Kind, + /// What the user types. + pub prompt: String, + /// The subdirectory of the sandbox's `work/` the run starts in and the fixture is copied to. + #[serde(default)] + pub dir: Option, + /// A directory beside `trial.yaml` copied into the run's directory and committed there. + #[serde(default)] + pub fixture: Option, + /// Give the fixture a bare `origin` inside the sandbox, with `main` pushed. + #[serde(default)] + pub remote: bool, + /// The sandbox is seeded with older plugins on purpose (an upgrade trial). + #[serde(default)] + pub seeded: bool, + /// Shell lines run from the sandbox root with the sandbox's `env`, before the run: installing + /// the plugins under test, or seeding an older install. + #[serde(default)] + pub setup: Vec, + /// The numbers the run must report. + pub measures: Vec, + /// Output name → path relative to the run's directory, for [`Measure::Outputs`]. + #[serde(default)] + pub outputs: BTreeMap, +} + +impl Definition { + /// The directory the run starts in, relative to the sandbox. + #[must_use] + pub fn workdir(&self) -> PathBuf { + let work = PathBuf::from("work"); + match self.dir.as_deref() { + Some(dir) if !dir.is_empty() => work.join(dir), + _ => work, + } + } + + /// Whether the run reports `measure`. + #[must_use] + pub fn measures(&self, measure: Measure) -> bool { + self.measures.contains(&measure) + } +} + +/// A relative path that stays below the directory it is joined to. +fn contained(path: &str) -> bool { + let path = Path::new(path); + !path.as_os_str().is_empty() + && path + .components() + .all(|component| matches!(component, std::path::Component::Normal(_))) +} + +fn read(root: &Path, name: &str) -> Result { + let path = root.join(TRIALS).join(name).join("trial.yaml"); + let text = std::fs::read_to_string(&path) + .map_err(|error| format!("reading {}: {error}", path.display()))?; + serde_yaml::from_str(&text).map_err(|error| format!("parsing {}: {error}", path.display())) +} + +/// Every trial name under `trials/`, sorted. +pub fn names(root: &Path) -> Result, String> { + let directory = root.join(TRIALS); + let mut names = Vec::new(); + for entry in std::fs::read_dir(&directory) + .map_err(|error| format!("reading {}: {error}", directory.display()))? + { + let path = entry.map_err(|error| error.to_string())?.path(); + if path.is_dir() { + if let Some(name) = path.file_name().and_then(|name| name.to_str()) { + names.push(name.to_owned()); + } + } + } + names.sort(); + Ok(names) +} + +/// One trial's definition, checked. +pub fn load(root: &Path, name: &str) -> Result { + let known = names(root)?; + if !known.iter().any(|known| known == name) { + return Err(format!( + "no trial `{name}` under {TRIALS}/; the trials are: {}", + known.join(", ") + )); + } + let definition = read(root, name)?; + let problems = problems(root, &definition); + if problems.is_empty() { + Ok(definition) + } else { + Err(format!( + "{TRIALS}/{name}/trial.yaml:\n {}", + problems.join("\n ") + )) + } +} + +fn problems(root: &Path, definition: &Definition) -> Vec { + let mut problems = Vec::new(); + let directory = root.join(TRIALS).join(&definition.name); + if !directory.is_dir() { + problems.push(format!( + "`name: {}` is not the directory the definition sits in", + definition.name + )); + } + if definition.prompt.trim().is_empty() { + problems.push("the prompt is empty".to_owned()); + } + if let Some(dir) = definition.dir.as_deref() { + if !contained(dir) { + problems.push(format!("`dir: {dir}` is not a relative path below `work/`")); + } + } + if let Some(fixture) = definition.fixture.as_deref() { + if !contained(fixture) || !directory.join(fixture).is_dir() { + problems.push(format!( + "`fixture: {fixture}` is not a directory beside trial.yaml" + )); + } else if directory.join(fixture).join(".git").exists() { + problems.push(format!( + "`fixture: {fixture}` carries a `.git`; the sandbox commits it" + )); + } + } + if definition.remote && definition.fixture.is_none() { + problems.push("`remote: true` needs a fixture to push".to_owned()); + } + if definition.seeded && definition.setup.is_empty() { + problems.push("`seeded: true` needs the `setup` that seeds the install".to_owned()); + } + if !definition.measures(Measure::ToolCalls) { + problems.push("every trial measures `tool_calls`".to_owned()); + } + if definition.measures(Measure::Outputs) == definition.outputs.is_empty() { + problems.push("`outputs` are listed exactly when `measures` has `outputs`".to_owned()); + } + for (output, path) in &definition.outputs { + if !contained(path) { + problems.push(format!( + "output `{output}: {path}` is not a relative path below the run's directory" + )); + } + } + problems +} + +/// Every definition parses and holds together, and the baseline names only trials that exist. +pub fn check(root: &Path) -> Result<(), String> { + let mut found = Vec::new(); + let names = names(root)?; + for name in &names { + match read(root, name) { + Ok(definition) => { + for problem in problems(root, &definition) { + found.push(format!("{TRIALS}/{name}/trial.yaml: {problem}")); + } + } + Err(error) => found.push(error), + } + } + let baseline = root.join(BASELINE); + let text = std::fs::read_to_string(&baseline) + .map_err(|error| format!("reading {BASELINE}: {error}"))?; + match serde_json::from_str::>(&text) { + Ok(entries) => { + for trial in entries.keys() { + if !names.contains(trial) { + found.push(format!("{BASELINE} records `{trial}`, which is no trial")); + } + } + } + Err(error) => found.push(format!("parsing {BASELINE}: {error}")), + } + if found.is_empty() { + Ok(()) + } else { + Err(format!( + "{} trial definition problem(s):\n {}", + found.len(), + found.join("\n ") + )) + } +} + +fn copy_tree(from: &Path, to: &Path) -> Result<(), String> { + std::fs::create_dir_all(to).map_err(|error| format!("creating {}: {error}", to.display()))?; + for entry in + std::fs::read_dir(from).map_err(|error| format!("reading {}: {error}", from.display()))? + { + let path = entry.map_err(|error| error.to_string())?.path(); + let target = to.join(path.file_name().unwrap_or_default()); + if path.is_dir() { + copy_tree(&path, &target)?; + } else { + std::fs::copy(&path, &target) + .map_err(|error| format!("copying {}: {error}", path.display()))?; + } + } + Ok(()) +} + +/// Run `git` in `directory` as the trial identity, reading no configuration of the operator's. +fn git(sandbox: &Path, directory: &Path, args: &[&str]) -> Result<(), String> { + let output = Command::new("git") + .args([ + "-c", + "commit.gpgsign=false", + "-c", + "init.defaultBranch=main", + ]) + .args(args) + .current_dir(directory) + .env("HOME", sandbox.join("home")) + .env("GIT_CONFIG_NOSYSTEM", "1") + .env("GIT_CONFIG_GLOBAL", "/dev/null") + .env("GIT_AUTHOR_NAME", "trial") + .env("GIT_AUTHOR_EMAIL", "trial@example.invalid") + .env("GIT_COMMITTER_NAME", "trial") + .env("GIT_COMMITTER_EMAIL", "trial@example.invalid") + .output() + .map_err(|error| format!("running git: {error}"))?; + if output.status.success() { + Ok(()) + } else { + Err(format!( + "git {} in {}: {}", + args.join(" "), + directory.display(), + String::from_utf8_lossy(&output.stderr).trim() + )) + } +} + +/// Prepare a fresh sandbox (`task trial:sandbox`) for `name`: the fixture copied into the run's +/// directory and committed, and beside `env` the files `task trial:run` reads — `prompt.txt`, +/// `workdir`, `setup.sh` and, for an upgrade trial, `seeded`. +pub fn prepare(root: &Path, name: &str, sandbox: &Path) -> Result { + let definition = load(root, name)?; + if !sandbox.join("env").is_file() { + return Err(format!( + "{} is not a trial sandbox (no `env`); create it with `task trial:sandbox`", + sandbox.display() + )); + } + let work = sandbox.join(definition.workdir()); + std::fs::create_dir_all(&work) + .map_err(|error| format!("creating {}: {error}", work.display()))?; + let mut done = Vec::new(); + if let Some(fixture) = definition.fixture.as_deref() { + copy_tree(&root.join(TRIALS).join(name).join(fixture), &work)?; + git(sandbox, &work, &["init", "--quiet"])?; + git(sandbox, &work, &["add", "--all"])?; + git( + sandbox, + &work, + &["commit", "--quiet", "-m", "Initial import"], + )?; + done.push(format!("fixture committed in {}", work.display())); + if definition.remote { + let remote = sandbox.join("remote.git"); + git( + sandbox, + sandbox, + &["init", "--quiet", "--bare", &remote.to_string_lossy()], + )?; + git( + sandbox, + &work, + &["remote", "add", "origin", &remote.to_string_lossy()], + )?; + git(sandbox, &work, &["push", "--quiet", "-u", "origin", "main"])?; + done.push(format!("origin {}", remote.display())); + } + } + let write = |file: &str, text: &str| { + std::fs::write(sandbox.join(file), text).map_err(|error| format!("writing {file}: {error}")) + }; + write("prompt.txt", definition.prompt.trim_end())?; + write("workdir", &definition.dir.clone().unwrap_or_default())?; + if definition.setup.is_empty() { + let _ = std::fs::remove_file(sandbox.join("setup.sh")); + } else { + write( + "setup.sh", + &format!("set -e\n{}\n", definition.setup.join("\n")), + )?; + done.push(format!("{} setup line(s)", definition.setup.len())); + } + if definition.seeded { + write("seeded", "")?; + } else { + let _ = std::fs::remove_file(sandbox.join("seeded")); + } + Ok(format!( + "prepared trial `{name}` ({:?}) in {}{}", + definition.kind, + sandbox.display(), + if done.is_empty() { + String::new() + } else { + format!(": {}", done.join("; ")) + } + )) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn scratch(label: &str) -> PathBuf { + let path = std::env::temp_dir().join(format!( + "agentplugins-check-trials-{label}-{}", + std::process::id() + )); + let _ = std::fs::remove_dir_all(&path); + std::fs::create_dir_all(&path).expect("the scratch directory is writable"); + path + } + + fn repository() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .and_then(Path::parent) + .expect("checker is under the repository root") + .to_path_buf() + } + + #[test] + fn the_committed_trials_hold_together() { + check(&repository()).expect("every committed trial validates"); + let names = names(&repository()).expect("trials/ is readable"); + for expected in [ + "ess-new", + "ess-retrofit", + "ess-pipeline", + "ess-full-package", + "aep-backlog", + "worktree-onboarding", + "upgrade-seeded", + ] { + assert!(names.iter().any(|name| name == expected), "{expected}"); + } + } + + #[test] + fn a_definition_that_does_not_hold_together_is_refused() { + let root = scratch("bad"); + let trial = root.join(TRIALS).join("broken"); + std::fs::create_dir_all(&trial).unwrap(); + std::fs::write( + trial.join("trial.yaml"), + "name: other\nkind: ess-new\nprompt: ' '\ndir: ../escape\nseeded: true\nmeasures: [outputs]\n", + ) + .unwrap(); + std::fs::write(root.join(BASELINE), "{\"gone\": {}}").unwrap(); + let error = check(&root).unwrap_err(); + std::fs::remove_dir_all(&root).unwrap(); + for expected in [ + "not the directory", + "prompt is empty", + "not a relative path below `work/`", + "needs the `setup`", + "every trial measures `tool_calls`", + "`outputs` are listed exactly", + "records `gone`, which is no trial", + ] { + assert!(error.contains(expected), "{expected}: {error}"); + } + } + + #[test] + fn prepare_commits_the_fixture_and_writes_the_run_files() { + let root = scratch("prepare"); + let trial = root.join(TRIALS).join("demo"); + std::fs::create_dir_all(trial.join("fixture/src")).unwrap(); + std::fs::write(trial.join("fixture/src/main.go"), "package main\n").unwrap(); + std::fs::write( + trial.join("trial.yaml"), + "name: demo\nkind: ess-retrofit\nprompt: |\n Describe this service.\ndir: svc\nfixture: fixture\nremote: true\nsetup: [echo one]\nmeasures: [tool_calls, unmapped]\n", + ) + .unwrap(); + std::fs::write(root.join(BASELINE), "{}").unwrap(); + let sandbox = root.join("sandbox"); + std::fs::create_dir_all(sandbox.join("home")).unwrap(); + std::fs::write(sandbox.join("env"), "").unwrap(); + + let summary = prepare(&root, "demo", &sandbox).unwrap(); + assert!(summary.contains("fixture committed"), "{summary}"); + let work = sandbox.join("work/svc"); + assert!(work.join("src/main.go").is_file()); + let log = Command::new("git") + .args(["log", "--format=%an %s", "origin/main"]) + .current_dir(&work) + .output() + .unwrap(); + assert_eq!( + String::from_utf8_lossy(&log.stdout), + "trial Initial import\n" + ); + assert_eq!( + std::fs::read_to_string(sandbox.join("prompt.txt")).unwrap(), + "Describe this service." + ); + assert_eq!( + std::fs::read_to_string(sandbox.join("workdir")).unwrap(), + "svc" + ); + assert_eq!( + std::fs::read_to_string(sandbox.join("setup.sh")).unwrap(), + "set -e\necho one\n" + ); + assert!(!sandbox.join("seeded").exists()); + let unknown = prepare(&root, "missing", &sandbox).unwrap_err(); + std::fs::remove_dir_all(&root).unwrap(); + assert!(unknown.contains("the trials are: demo"), "{unknown}"); + } +} diff --git a/plugins/aep/.claude-plugin/plugin.json b/plugins/aep/.claude-plugin/plugin.json index f0a1f39..3896e9b 100644 --- a/plugins/aep/.claude-plugin/plugin.json +++ b/plugins/aep/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "aep", "displayName": "AEP", "description": "Plan governed work in the AEP artifact store and deliver it in reviewed waves: decomposition, plan critique, reverse engineering, story scoping, implementation and adversarial review.", - "version": "0.14.8", + "version": "0.14.9", "author": { "name": "Beyond10x" }, diff --git a/plugins/aep/.codex-plugin/plugin.json b/plugins/aep/.codex-plugin/plugin.json index dd8492e..e792fb7 100644 --- a/plugins/aep/.codex-plugin/plugin.json +++ b/plugins/aep/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "aep", - "version": "0.14.8", + "version": "0.14.9", "description": "Plan governed work in the AEP artifact store and deliver it in reviewed waves.", "author": { "name": "Beyond10x" diff --git a/plugins/b10x/.claude-plugin/plugin.json b/plugins/b10x/.claude-plugin/plugin.json index c6abc6d..f775634 100644 --- a/plugins/b10x/.claude-plugin/plugin.json +++ b/plugins/b10x/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "b10x", "displayName": "Beyond10x", "description": "Set up, upgrade and check the Beyond10x plugins and binaries, route work to them, and create portable plugins.", - "version": "0.14.8", + "version": "0.14.9", "author": { "name": "Beyond10x" }, diff --git a/plugins/b10x/.codex-plugin/plugin.json b/plugins/b10x/.codex-plugin/plugin.json index 2f9eb2d..31516e7 100644 --- a/plugins/b10x/.codex-plugin/plugin.json +++ b/plugins/b10x/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "b10x", - "version": "0.14.8", + "version": "0.14.9", "description": "Set up, upgrade and check the Beyond10x plugins and binaries, route work to them, and create portable plugins.", "author": { "name": "Beyond10x" diff --git a/plugins/connectors/.claude-plugin/plugin.json b/plugins/connectors/.claude-plugin/plugin.json index 4cf469e..8209f32 100644 --- a/plugins/connectors/.claude-plugin/plugin.json +++ b/plugins/connectors/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "connectors", - "version": "0.14.8", + "version": "0.14.9", "description": "Set up, inspect, and invoke governed integrations through the connectors CLI.", "author": { "name": "Beyond10x" }, "license": "Apache-2.0", diff --git a/plugins/connectors/.codex-plugin/plugin.json b/plugins/connectors/.codex-plugin/plugin.json index da231a8..2ab5478 100644 --- a/plugins/connectors/.codex-plugin/plugin.json +++ b/plugins/connectors/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "connectors", - "version": "0.14.8", + "version": "0.14.9", "description": "Set up, inspect, and invoke governed integrations through the connectors CLI.", "author": { "name": "Beyond10x" }, "license": "Apache-2.0", diff --git a/plugins/ess/.claude-plugin/plugin.json b/plugins/ess/.claude-plugin/plugin.json index d4e63e6..8bf3d94 100644 --- a/plugins/ess/.claude-plugin/plugin.json +++ b/plugins/ess/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "ess", "displayName": "ESS", "description": "Write, retrofit, validate and project Executable System Specifications, and hold implementations to them with conformance suites.", - "version": "0.14.8", + "version": "0.14.9", "author": { "name": "Beyond10x" }, diff --git a/plugins/ess/.codex-plugin/plugin.json b/plugins/ess/.codex-plugin/plugin.json index d1ee00b..1bdf152 100644 --- a/plugins/ess/.codex-plugin/plugin.json +++ b/plugins/ess/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "ess", - "version": "0.14.8", + "version": "0.14.9", "description": "Write, retrofit, validate and project Executable System Specifications, and hold implementations to them with conformance suites.", "author": { "name": "Beyond10x" diff --git a/plugins/worktree/.claude-plugin/plugin.json b/plugins/worktree/.claude-plugin/plugin.json index 4f91c64..fd066d7 100644 --- a/plugins/worktree/.claude-plugin/plugin.json +++ b/plugins/worktree/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "worktree", "displayName": "Worktree", "description": "Create, lease, finish, audit and safely clean isolated Git worktrees through the worktree CLI.", - "version": "0.14.8", + "version": "0.14.9", "author": { "name": "Beyond10x" }, diff --git a/plugins/worktree/.codex-plugin/plugin.json b/plugins/worktree/.codex-plugin/plugin.json index c03e0df..36d70f0 100644 --- a/plugins/worktree/.codex-plugin/plugin.json +++ b/plugins/worktree/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "worktree", - "version": "0.14.8", + "version": "0.14.9", "description": "Create, lease, finish, audit and safely clean isolated Git worktrees through the worktree CLI.", "author": { "name": "Beyond10x" diff --git a/trials/aep-backlog/fixture/README.md b/trials/aep-backlog/fixture/README.md new file mode 100644 index 0000000..b5cf34b --- /dev/null +++ b/trials/aep-backlog/fixture/README.md @@ -0,0 +1,4 @@ +# birdlog + +A command-line log for bird sightings: what you saw, where, when, and how many. Sightings are kept +in one local file; nothing leaves the machine. diff --git a/trials/aep-backlog/fixture/TODO.md b/trials/aep-backlog/fixture/TODO.md new file mode 100644 index 0000000..b078db8 --- /dev/null +++ b/trials/aep-backlog/fixture/TODO.md @@ -0,0 +1,20 @@ +# TODO + +## Recording +- [ ] `birdlog add --count N --at ` records a sighting with the current time +- [ ] species names are checked against a bundled checklist; unknown names are refused with the closest matches +- [ ] edit and delete a sighting by its number +- [x] store sightings in a single JSON Lines file under the user's data directory + +## Looking back +- [ ] list sightings, newest first, filterable by species, place and date range +- [ ] a year list: every species seen this year, with the first date it was seen +- [ ] totals per place + +## Sharing +- [ ] export a date range as CSV for the regional survey +- [ ] import the survey's CSV format, skipping rows already present + +## Chores +- [ ] the checklist is two years old; find out how often the survey republishes it +- [ ] tests for the storage format before anything else touches it diff --git a/trials/aep-backlog/trial.yaml b/trials/aep-backlog/trial.yaml new file mode 100644 index 0000000..f139d14 --- /dev/null +++ b/trials/aep-backlog/trial.yaml @@ -0,0 +1,15 @@ +name: aep-backlog +kind: aep-backlog +prompt: | + Our backlog for this little tool is just `TODO.md`. I'd like to plan it properly with AEP from + now on: turn what's there into epics and stories, leave out what's already done, and make sure + the result validates. Don't delete `TODO.md`. + + When you're done, list the skills and agents you used, quote any instruction text that confused + you, and paste the last validation output verbatim. +dir: birdlog +fixture: fixture +setup: + - b10x init aep --host claude --out plan.json + - b10x setup apply --plan plan.json --yes +measures: [tool_calls] diff --git a/trials/baseline.json b/trials/baseline.json new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/trials/baseline.json @@ -0,0 +1 @@ +{} diff --git a/trials/ess-full-package/trial.yaml b/trials/ess-full-package/trial.yaml new file mode 100644 index 0000000..c3e9951 --- /dev/null +++ b/trials/ess-full-package/trial.yaml @@ -0,0 +1,34 @@ +name: ess-full-package +kind: ess-full-package +prompt: | + I'd like to see the whole ESS package end to end on a small service. Pick a neutral domain + yourself — something with a lifecycle, a couple of commands that can be refused and at least one + event — and write its ESS specification in `spec/`. + + Then produce every output ESS offers from it, each in its own directory under `out/`: + `out/schema` (JSON Schema), `out/openapi`, `out/asyncapi`, `out/docs`, `out/site`, + `out/types-rust` (a Rust type library), `out/types-go` (a Go type library) and `out/conformance` + (the Go conformance suite). + + Then write a minimal in-memory Go implementation of the service in `impl/` that implements the + suite's `Target`, run `go test -v ./...` in `impl/`, and tell me how many scenarios passed, failed + and were skipped. Don't edit the generated suite to make it pass. + + When you're done, list the skills and agents you used, quote any instruction text that confused + you, and paste the last validation output, the last synthesis output and the last `go test` + summary verbatim. +dir: service +setup: + - b10x init ess --host claude --out plan.json + - b10x setup apply --plan plan.json --yes +measures: [tool_calls, validate, synthesis, unmapped, outputs, go_test] +outputs: + schema: out/schema + openapi: out/openapi + asyncapi: out/asyncapi + docs: out/docs + site: out/site + types-rust: out/types-rust + types-go: out/types-go + conformance: out/conformance + implementation: impl diff --git a/trials/ess-new/trial.yaml b/trials/ess-new/trial.yaml new file mode 100644 index 0000000..4b28062 --- /dev/null +++ b/trials/ess-new/trial.yaml @@ -0,0 +1,15 @@ +name: ess-new +kind: ess-new +prompt: | + I'm starting a small service for a community seed library: members borrow seed packets in + spring, and bring back seeds from their harvest in autumn so the library can restock. Some + varieties run out, and a member can only have five packets out at once. Write an ESS + specification for it in `spec/` and make sure it validates. + + When you're done, list the skills and agents you used, quote any instruction text that confused + you, and paste the last validation output verbatim. +dir: seedlib +setup: + - b10x init ess --host claude --out plan.json + - b10x setup apply --plan plan.json --yes +measures: [tool_calls, validate, unmapped] diff --git a/trials/ess-pipeline/fixture/spec/domains/booking.yaml b/trials/ess-pipeline/fixture/spec/domains/booking.yaml new file mode 100644 index 0000000..1bd9291 --- /dev/null +++ b/trials/ess-pipeline/fixture/spec/domains/booking.yaml @@ -0,0 +1,177 @@ +domain: rehearsal.booking + +summary: Holding, confirming and releasing a rehearsal room for one slot. + +naming: + wire: bookings + display: Bookings + +types: + - name: rehearsal.booking.BookingId + kind: newtype + of: Uuid + + - name: rehearsal.booking.Room + kind: enum + variants: [Studio, Hall, Booth] + + - name: rehearsal.booking.BandName + kind: newtype + of: String + +entities: + - name: rehearsal.booking.Booking + + identity: + name: booking_id + type: rehearsal.booking.BookingId + + fields: + - name: band + type: rehearsal.booking.BandName + - name: room + type: rehearsal.booking.Room + - name: starts_at + type: Timestamp + - name: hours + type: Integer + + invariants: + - hours > 0 + + lifecycle: + initial: Held + states: [Held, Confirmed, Released] + terminal: [Released] + transitions: + - name: confirm + from: [Held] + to: Confirmed + - name: release + from: [Held, Confirmed] + to: Released + +actors: + - name: rehearsal.booking.BandLeader + may: + - rehearsal.booking.HoldRoom + - rehearsal.booking.ConfirmBooking + - rehearsal.booking.ReleaseBooking + +errors: + - name: rehearsal.booking.InvalidLength + summary: The booking is not for a positive number of hours. + fields: + - name: hours + type: Integer + + - name: rehearsal.booking.BookingStateConflict + summary: The booking is not in a state this command acts from. + fields: + - name: state + type: rehearsal.booking.Booking.State + +commands: + - name: rehearsal.booking.HoldRoom + input: + - name: band + type: rehearsal.booking.BandName + - name: room + type: rehearsal.booking.Room + - name: starts_at + type: Timestamp + - name: hours + type: Integer + outcomes: + - name: held + when: hours > 0 + creates: rehearsal.booking.Booking + instance: booking_id + emits: + - rehearsal.booking.RoomHeld + payload: + rehearsal.booking.RoomHeld: + band: input.band + room: input.room + hours: input.hours + summary: The room is held for the band. + - name: refused + error: rehearsal.booking.InvalidLength + summary: The length was not positive, and nothing was held. + + - name: rehearsal.booking.ConfirmBooking + input: + - name: booking_id + type: rehearsal.booking.BookingId + outcomes: + - name: confirmed + moves: rehearsal.booking.Booking.confirm + instance: booking_id + emits: + - rehearsal.booking.BookingConfirmed + payload: + rehearsal.booking.BookingConfirmed: + booking_id: input.booking_id + summary: The band has paid and the room is theirs. + - name: wrong-state + wrong_state: true + error: rehearsal.booking.BookingStateConflict + summary: The booking is not held, so nothing was confirmed. + + - name: rehearsal.booking.ReleaseBooking + input: + - name: booking_id + type: rehearsal.booking.BookingId + outcomes: + - name: released + moves: rehearsal.booking.Booking.release + instance: booking_id + emits: + - rehearsal.booking.BookingReleased + payload: + rehearsal.booking.BookingReleased: + booking_id: input.booking_id + summary: The room is free again. + - name: wrong-state + wrong_state: true + error: rehearsal.booking.BookingStateConflict + summary: The booking was already released. + +events: + - name: rehearsal.booking.RoomHeld + fields: + - name: booking_id + type: rehearsal.booking.BookingId + - name: band + type: rehearsal.booking.BandName + - name: room + type: rehearsal.booking.Room + - name: hours + type: Integer + + - name: rehearsal.booking.BookingConfirmed + fields: + - name: booking_id + type: rehearsal.booking.BookingId + + - name: rehearsal.booking.BookingReleased + fields: + - name: booking_id + type: rehearsal.booking.BookingId + +views: + - name: rehearsal.booking.BookingById + source: rehearsal.booking.Booking + consistency: read_your_writes + fields: + - name: booking_id + type: rehearsal.booking.BookingId + - name: band + type: rehearsal.booking.BandName + - name: room + type: rehearsal.booking.Room + - name: hours + type: Integer + naming: + wire: by-id + display: Booking by id diff --git a/trials/ess-pipeline/fixture/spec/system.yaml b/trials/ess-pipeline/fixture/spec/system.yaml new file mode 100644 index 0000000..a114ce3 --- /dev/null +++ b/trials/ess-pipeline/fixture/spec/system.yaml @@ -0,0 +1,10 @@ +format: ess/1 +system: rehearsal +version: v1 + +summary: >- + Rehearsal rooms for a music school: a band holds a room for a slot, confirms it when the band + leader pays, and releases it when plans change. + +domains: + - rehearsal.booking diff --git a/trials/ess-pipeline/trial.yaml b/trials/ess-pipeline/trial.yaml new file mode 100644 index 0000000..bf01987 --- /dev/null +++ b/trials/ess-pipeline/trial.yaml @@ -0,0 +1,21 @@ +name: ess-pipeline +kind: ess-pipeline +prompt: | + There's an ESS specification for our rehearsal-room bookings in `spec/`. Generate its JSON + Schema into `out/schema`, its OpenAPI into `out/openapi` and its docs into `out/docs`, and + synthesize the conformance suite as the canonical document at `out/conformance.json`. Tell me how + many scenarios it has and whether the synthesis refused anything, and why. + + When you're done, list the skills and agents you used, quote any instruction text that confused + you, and paste the last synthesis output verbatim. +dir: rehearsal +fixture: fixture +setup: + - b10x init ess --host claude --out plan.json + - b10x setup apply --plan plan.json --yes +measures: [tool_calls, validate, synthesis, outputs] +outputs: + schema: out/schema + openapi: out/openapi + docs: out/docs + conformance: out/conformance.json diff --git a/trials/ess-retrofit/fixture/README.md b/trials/ess-retrofit/fixture/README.md new file mode 100644 index 0000000..882f5a2 --- /dev/null +++ b/trials/ess-retrofit/fixture/README.md @@ -0,0 +1,16 @@ +# toolshed + +The neighbourhood tool library: members borrow drills, ladders and saws, and bring them back. + +```console +go run . # listens on :8080 +``` + +| method | path | does | +|---|---|---| +| `POST` | `/tools` | add a tool (`{"name": "…", "category": "…"}`) | +| `POST` | `/tools/{id}/lend` | lend it to a member (`{"member": "…", "days": 7}`) | +| `POST` | `/tools/{id}/return` | take it back | +| `POST` | `/tools/{id}/retire` | take it out of circulation | +| `GET` | `/tools/{id}` | one tool | +| `GET` | `/loans/overdue` | tools out longer than agreed | diff --git a/trials/ess-retrofit/fixture/go.mod b/trials/ess-retrofit/fixture/go.mod new file mode 100644 index 0000000..52a2f8c --- /dev/null +++ b/trials/ess-retrofit/fixture/go.mod @@ -0,0 +1,3 @@ +module example.com/toolshed + +go 1.22 diff --git a/trials/ess-retrofit/fixture/main.go b/trials/ess-retrofit/fixture/main.go new file mode 100644 index 0000000..bbe8461 --- /dev/null +++ b/trials/ess-retrofit/fixture/main.go @@ -0,0 +1,76 @@ +package main + +import ( + "encoding/json" + "errors" + "log" + "net/http" + "strings" + "time" +) + +func main() { + shed := NewShed(time.Now) + log.Fatal(http.ListenAndServe(":8080", routes(shed))) +} + +func routes(shed *Shed) http.Handler { + mux := http.NewServeMux() + mux.HandleFunc("POST /tools", func(w http.ResponseWriter, r *http.Request) { + var body struct { + Name string `json:"name"` + Category string `json:"category"` + } + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + tool, err := shed.Add(body.Name, body.Category) + reply(w, tool, err) + }) + mux.HandleFunc("POST /tools/{id}/lend", func(w http.ResponseWriter, r *http.Request) { + var body struct { + Member string `json:"member"` + Days int `json:"days"` + } + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + tool, err := shed.Lend(r.PathValue("id"), body.Member, body.Days) + reply(w, tool, err) + }) + mux.HandleFunc("POST /tools/{id}/return", func(w http.ResponseWriter, r *http.Request) { + tool, err := shed.Return(r.PathValue("id")) + reply(w, tool, err) + }) + mux.HandleFunc("POST /tools/{id}/retire", func(w http.ResponseWriter, r *http.Request) { + tool, err := shed.Retire(r.PathValue("id")) + reply(w, tool, err) + }) + mux.HandleFunc("GET /tools/{id}", func(w http.ResponseWriter, r *http.Request) { + tool, err := shed.Get(r.PathValue("id")) + reply(w, tool, err) + }) + mux.HandleFunc("GET /loans/overdue", func(w http.ResponseWriter, r *http.Request) { + reply(w, shed.Overdue(), nil) + }) + return mux +} + +func reply(w http.ResponseWriter, value any, err error) { + switch { + case errors.Is(err, ErrNotFound): + http.Error(w, err.Error(), http.StatusNotFound) + case errors.Is(err, ErrOnLoan), errors.Is(err, ErrNotOnLoan), errors.Is(err, ErrRetired): + http.Error(w, err.Error(), http.StatusConflict) + case err != nil: + http.Error(w, err.Error(), http.StatusUnprocessableEntity) + default: + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(value) + } +} + +// Validation errors carry the offending value in their message only. +func invalid(message string) error { return errors.New(strings.TrimSpace(message)) } diff --git a/trials/ess-retrofit/fixture/shed.go b/trials/ess-retrofit/fixture/shed.go new file mode 100644 index 0000000..71afa42 --- /dev/null +++ b/trials/ess-retrofit/fixture/shed.go @@ -0,0 +1,165 @@ +package main + +import ( + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "sync" + "time" +) + +var ( + ErrNotFound = errors.New("no such tool") + ErrOnLoan = errors.New("tool is on loan") + ErrNotOnLoan = errors.New("tool is not on loan") + ErrRetired = errors.New("tool is retired") +) + +// Status is where a tool is: on the shelf, with a member, or gone for good. +type Status string + +const ( + Available Status = "available" + OnLoan Status = "on_loan" + Retired Status = "retired" +) + +// Categories the shed shelves tools under; anything else is refused. +var categories = map[string]bool{"power": true, "hand": true, "garden": true, "ladder": true} + +type Tool struct { + ID string `json:"id"` + Name string `json:"name"` + Category string `json:"category"` + Status Status `json:"status"` + Member string `json:"member,omitempty"` + DueAt *time.Time `json:"due_at,omitempty"` + Loans int `json:"loans"` +} + +// Event is what the shed announces; the notice board and the reminder mailer read them. +type Event struct { + Kind string `json:"kind"` + ToolID string `json:"tool_id"` + Member string `json:"member,omitempty"` + At time.Time `json:"at"` +} + +type Shed struct { + mu sync.Mutex + now func() time.Time + tools map[string]*Tool + Events []Event +} + +func NewShed(now func() time.Time) *Shed { + return &Shed{now: now, tools: map[string]*Tool{}} +} + +func newID() string { + b := make([]byte, 8) + _, _ = rand.Read(b) + return hex.EncodeToString(b) +} + +func (s *Shed) emit(kind string, tool *Tool) { + s.Events = append(s.Events, Event{Kind: kind, ToolID: tool.ID, Member: tool.Member, At: s.now()}) +} + +func (s *Shed) Add(name, category string) (Tool, error) { + s.mu.Lock() + defer s.mu.Unlock() + if name == "" { + return Tool{}, invalid("a tool needs a name") + } + if !categories[category] { + return Tool{}, invalid(fmt.Sprintf("unknown category %q", category)) + } + tool := &Tool{ID: newID(), Name: name, Category: category, Status: Available} + s.tools[tool.ID] = tool + s.emit("ToolAdded", tool) + return *tool, nil +} + +func (s *Shed) Lend(id, member string, days int) (Tool, error) { + s.mu.Lock() + defer s.mu.Unlock() + tool, ok := s.tools[id] + if !ok { + return Tool{}, ErrNotFound + } + if days < 1 || days > 28 { + return Tool{}, invalid(fmt.Sprintf("a loan is 1 to 28 days, not %d", days)) + } + switch tool.Status { + case OnLoan: + return Tool{}, ErrOnLoan + case Retired: + return Tool{}, ErrRetired + } + due := s.now().Add(time.Duration(days) * 24 * time.Hour) + tool.Status, tool.Member, tool.DueAt = OnLoan, member, &due + tool.Loans++ + s.emit("ToolLent", tool) + return *tool, nil +} + +func (s *Shed) Return(id string) (Tool, error) { + s.mu.Lock() + defer s.mu.Unlock() + tool, ok := s.tools[id] + if !ok { + return Tool{}, ErrNotFound + } + if tool.Status != OnLoan { + return Tool{}, ErrNotOnLoan + } + s.emit("ToolReturned", tool) + tool.Status, tool.Member, tool.DueAt = Available, "", nil + return *tool, nil +} + +// Retire takes a tool out of circulation. A tool on loan is retired when it comes back: the call +// succeeds and changes nothing, and the desk is expected to try again later. +func (s *Shed) Retire(id string) (Tool, error) { + s.mu.Lock() + defer s.mu.Unlock() + tool, ok := s.tools[id] + if !ok { + return Tool{}, ErrNotFound + } + switch tool.Status { + case Retired: + return Tool{}, ErrRetired + case OnLoan: + return *tool, nil + } + tool.Status = Retired + s.emit("ToolRetired", tool) + return *tool, nil +} + +func (s *Shed) Get(id string) (Tool, error) { + s.mu.Lock() + defer s.mu.Unlock() + tool, ok := s.tools[id] + if !ok { + return Tool{}, ErrNotFound + } + return *tool, nil +} + +// Overdue lists the tools whose loan ran past its due date, oldest first by insertion. +func (s *Shed) Overdue() []Tool { + s.mu.Lock() + defer s.mu.Unlock() + now := s.now() + var late []Tool + for _, tool := range s.tools { + if tool.Status == OnLoan && tool.DueAt != nil && now.After(*tool.DueAt) { + late = append(late, *tool) + } + } + return late +} diff --git a/trials/ess-retrofit/trial.yaml b/trials/ess-retrofit/trial.yaml new file mode 100644 index 0000000..1ed53ab --- /dev/null +++ b/trials/ess-retrofit/trial.yaml @@ -0,0 +1,15 @@ +name: ess-retrofit +kind: ess-retrofit +prompt: | + This repository is our tool-lending service. Write an ESS specification in `spec/` that describes + what the code actually does today — not what it should do — validate it, and synthesize its + conformance suite. Tell me what you could not map from the code, and what the synthesis refused. + + When you're done, list the skills and agents you used, quote any instruction text that confused + you, and paste the last validation output and the last synthesis output verbatim. +dir: toolshed +fixture: fixture +setup: + - b10x init ess --host claude --out plan.json + - b10x setup apply --plan plan.json --yes +measures: [tool_calls, validate, synthesis, unmapped] diff --git a/trials/upgrade-seeded/trial.yaml b/trials/upgrade-seeded/trial.yaml new file mode 100644 index 0000000..a09d481 --- /dev/null +++ b/trials/upgrade-seeded/trial.yaml @@ -0,0 +1,17 @@ +name: upgrade-seeded +kind: upgrade +prompt: | + I installed the Beyond10x plugins a while ago and haven't touched them since. Bring everything up + to date, and tell me what changed. + + When you're done, list the skills and agents you used, quote any instruction text that confused + you, and paste the last command output verbatim. +seeded: true +setup: + - git -c advice.detachedHead=false clone --quiet --depth 1 --branch 0.12.0 https://github.com/beyond10x/agentplugins.git home/seed-marketplace + - claude plugin marketplace add "$PWD/home/seed-marketplace" + - | + for plugin in $(grep -o '"\./plugins/[a-z0-9-]*"' home/seed-marketplace/.claude-plugin/marketplace.json | tr -d '"'); do + claude plugin install "${plugin##*/}@b10x" + done +measures: [tool_calls] diff --git a/trials/worktree-onboarding/fixture/README.md b/trials/worktree-onboarding/fixture/README.md new file mode 100644 index 0000000..aa8278a --- /dev/null +++ b/trials/worktree-onboarding/fixture/README.md @@ -0,0 +1,8 @@ +# recipe-scaler + +Scale a recipe up or down: give it the recipe and the number of servings you want, and it rewrites +every quantity. Metric and imperial units are both undrestood. + +```console +cargo run -- scale pancakes.txt --servings 6 +``` diff --git a/trials/worktree-onboarding/fixture/pancakes.txt b/trials/worktree-onboarding/fixture/pancakes.txt new file mode 100644 index 0000000..778fc4d --- /dev/null +++ b/trials/worktree-onboarding/fixture/pancakes.txt @@ -0,0 +1,5 @@ +Pancakes (serves 2) +150 g flour +1 egg +200 ml milk +1 tbsp sugar diff --git a/trials/worktree-onboarding/trial.yaml b/trials/worktree-onboarding/trial.yaml new file mode 100644 index 0000000..c47ade6 --- /dev/null +++ b/trials/worktree-onboarding/trial.yaml @@ -0,0 +1,16 @@ +name: worktree-onboarding +kind: worktree-onboarding +prompt: | + Set up worktree for this repository so I can work on changes in isolated checkouts. Then use it + to fix the typo in `README.md` in an isolated checkout on a branch, commit it there, and tell me + where the checkout is. Leave this checkout as it is. + + When you're done, list the skills and agents you used, quote any instruction text that confused + you, and paste the last command output verbatim. +dir: recipe-scaler +fixture: fixture +remote: true +setup: + - b10x init worktree --host claude --out plan.json + - b10x setup apply --plan plan.json --yes +measures: [tool_calls]