b10x-bot · Sep 25, 2026 · Sep 25, 2026
diff --git a/.agents/skills/improving-by-trial/SKILL.md b/.agents/skills/improving-by-trial/SKILL.md
@@ -17,22 +17,44 @@ task trial:sandbox NAME=<name>
 This creates `/var/tmp/b10x-trials-$USER/<name>/` (the Taskfile's `TRIALS`) with its own `home/`, a `b10x` built from this
 checkout, and an `env` file. The `env` file sets `HOME` to the sandbox and `B10X_MARKETPLACE` to
 this checkout, so `b10x init` installs the plugins as they are in the working tree, before any
-release. It also unsets `ANTHROPIC_API_KEY`, `TMPDIR` and `TMPPREFIX`. Put the fixture the trial
-needs (a small service, a `TODO.md`, an OpenAPI document) under `work/`, and commit it there with
-`git` when the trial needs history or a remote.
+release. It also unsets `ANTHROPIC_API_KEY`, `TMPDIR` and `TMPPREFIX`. A defined trial
+(`task trial:run TRIAL=<name>`, § 2) builds its own sandbox and fixture; for an ad-hoc one, put the
+fixture it needs (a small service, a `TODO.md`, an OpenAPI document) under `work/`, and commit it
+there with `git` when the trial needs history or a remote.
 
 To trial the released version instead, remove `home/.local/bin/b10x` and unset `B10X_MARKETPLACE`
 in `env`; the agent then follows `SETUP.md` from the release.
 
 ## 2. Run it headless, never as a sub-agent
 
+The round's trials are defined in `trials/`, one directory each:
+
+| file | holds |
+|---|---|
+| `trials/<name>/trial.yaml` | `name`, `kind`, `prompt`, optional `dir` (the run's subdirectory of `work/`), `fixture` (a directory beside it), `remote` (a bare `origin` in the sandbox), `seeded`, `setup` (shell lines run from the sandbox root with its `env` before the run), `measures`, and `outputs` (name → path below the run's directory) |
+| `trials/<name>/fixture/` | the small service or backlog the trial starts from |
+| `trials/baseline.json` | the measures of the last accepted run of each trial |
+
+`task check` validates every definition. Run one by name:
+
 ```console
-task trial:run NAME=<name> PROMPT='<the user sentence>' [DIR=<subdirectory of work>]
+task trial:run TRIAL=<name>
+```
+
+This builds a fresh sandbox (`trial:sandbox`), copies the fixture into `work/<dir>` and commits it
+there (`agentplugins-check trial-prepare`), runs the definition's `setup` (installing the plugins
+under test with `b10x init … --out plan.json` and `b10x setup apply --plan plan.json --yes`, or
+seeding an older release), then runs the prompt. An ad-hoc trial still runs in an existing sandbox:
+
+```console
+task trial:run NAME=<name> PROMPT='<the user sentence>' [DIR=<subdirectory of work>] [SEEDED=true]
 ```
 
 The task copies the operator's credentials in (mode 600), runs `claude -p` with the sandbox's
 `env`, `--strict-mcp-config` (no MCP server, including the account's claude.ai connectors) and
-stream-json output into `run.jsonl`, and deletes the credentials when it finishes.
+stream-json output into `run.jsonl`, and deletes the credentials when it finishes. The sandbox
+`PATH` has `cargo` and `go`, and `go` is an allowed tool, so a trial can build and test an
+implementation.
 
 **Never run a trial as a sub-agent of the working session.** A sub-agent gets the session's own
 agents and plugins whatever `HOME` its shell uses. In trial 3 the aep critics that ran were this
@@ -44,7 +66,7 @@ confused it, and pasting the final command output verbatim. Give it no other hin
 
 ## 3. Only an isolated run counts
 
-`trial:run` ends with:
+`trial:run` checks isolation before it measures:
 
 ```console
 agentplugins-check trial-isolation <sandbox>/run.jsonl --sandbox <sandbox> --version <version>
@@ -62,15 +84,37 @@ trial 3, sandboxes under `~/.cache` loaded the operator's `~/.claude/CLAUDE.md`
 its commit rules) and a `.claude/settings.local.json` an earlier trial left in `~/.cache`, which
 enabled `aep@b10x`. `trial:sandbox` refuses to build below any of those files; that is why `TRIALS`
 is outside `$HOME`. Neither leak shows in the `init` event, so the placement is the only guard. An upgrade trial seeds older plugins on
-purpose: run it with `SEEDED=true`, so plugins must match what the sandbox was seeded with.
+purpose (`seeded: true` in its definition, or `SEEDED=true`), so plugins must match what the sandbox
+was seeded with; a registry that did not exist before the run is read after it.
 
 ## 4. Read the run
 
-From `run.jsonl`:
+`trial:run` ends with the numbers, one line each:
+
+```console
+agentplugins-check trial-report <sandbox>/run.jsonl --trial <name> [--baseline trials/baseline.json] [--write-baseline]
+```
+
+| measure | read from |
+|---|---|
+| `tool_calls` | `tool_use` blocks in the run |
+| `validate` | the last `ess specify validate` the run ran: `valid`, not valid, or not run |
+| `synthesis` | `N scenario(s) … M refusal(s)` in the last `ess verify conform synthesize` output |
+| `unmapped` | `UNMAPPED:` markers in the YAML files the run wrote, read from disk, not from its prose |
+| `outputs` | which of the definition's `outputs` exist (a directory counts when it is not empty) |
+| `go_test` | passed, failed and skipped tests of the last `go test` (`-v` or `-json`); a package that does not build counts as a failure |
+
+A trial reports the measures its definition lists; without `--trial`, an ad-hoc run gets every
+measure but `outputs`. With `--baseline` it exits 1 when a measure got worse than the trial's entry:
+validate stops passing, refusals or `go test` failures go up, a synthesis or `go test` that ran no
+longer runs, an output the baseline had is missing, or tool calls rise by more than 50%. Other
+changes print and pass. `--write-baseline` records the run as the trial's entry; do that for an
+isolated run the round accepts, and commit `trials/baseline.json` with the fixes it led to.
+
+The numbers say what happened, not why. From `run.jsonl`, also collect:
 
 | collect | how |
 |---|---|
-| tool calls | count of `tool_use` blocks |
 | stuck points | every refusal and error in `tool_result`s, verbatim |
 | guesses | what the final report says it inferred or could not find |
 | waste | calls repeated, files read twice, commands that failed and were retried unchanged |
@@ -108,9 +152,12 @@ and the fix is in a release, not when the issue closes.
    passes when the finding does not recur. New findings go back to step 5.
 4. Release when every trial of the round passes.
 
-Each round has 3 ESS trials (a new specification; a retrofit of an existing service; generate plus
-a synthesized suite), and 1 trial each for aep planning, worktree, and onboarding or upgrade.
-Change the domain each round so the agents cannot copy the previous answer from the skills.
+Each round runs every trial in `trials/`: 4 ESS trials (`ess-new`, a new specification;
+`ess-retrofit`, an existing service; `ess-pipeline`, generation plus a synthesized suite;
+`ess-full-package`, every output plus a Go implementation held to the synthesized suite), and
+`aep-backlog`, `worktree-onboarding` and `upgrade-seeded`. Change the domains and fixtures each
+round so the agents cannot copy the previous answer from the skills; a changed trial starts a new
+baseline entry.
 
 ## 7. Clean up
 

diff --git a/AGENTS.md b/AGENTS.md
@@ -16,6 +16,7 @@ CLIs. Serves O2 (decisions as data) and O3 (any harness).
 | `website/` | public docs; must pass `task site-build` |
 | `SETUP.md` | agent bootstrap, published as a release asset |
 | `.agents/skills/improving-by-trial/` | how plugins are improved: isolated headless trials (`task trial:sandbox`, `task trial:run`), triage, fix, re-run |
+| `trials/` | the round's trial definitions (`<name>/trial.yaml`, fixtures) and `baseline.json`; `task trial:run TRIAL=<name>` runs one, `agentplugins-check trial-report` measures it against the baseline |
 
 ## Rules
 
@@ -34,7 +35,7 @@ CLIs. Serves O2 (decisions as data) and O3 (any harness).
 ## Gate
 
 ```console
-task check          # fmt, clippy, tests, agentplugins-check
+task check          # fmt, clippy, tests, agentplugins-check (includes the trial definitions)
 task site-build     # when website/ changes
 cargo run --locked --bin agentplugins-check -- tools    # network: every spelled command exists in the newest aep, ess, worktree
 ```

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,23 @@
 # Changelog
 
+## [0.14.9] — 2026-09-25
+
+Trials become measurable: their prompts live in the repository, and each run is scored against a
+committed baseline, so a drop in quality fails instead of waiting to be noticed.
+
+- `trials/<name>/trial.yaml` defines each trial (prompt, fixture, setup, measures, expected
+  outputs); `task trial:run TRIAL=<name>` builds the sandbox, prepares the fixture and runs it. Seven
+  trials: ESS new spec, retrofit, pipeline and full package; aep backlog planning; worktree
+  onboarding; upgrade from a seeded install. The gate checks the definitions.
+- New trial `ess-full-package`: every ESS output (JSON Schema, OpenAPI, AsyncAPI, docs and site,
+  Rust and Go type libraries, the Go conformance suite), then an in-memory Go implementation run
+  with `go test` against the generated suite. Sandboxes get `go`.
+- `agentplugins-check trial-report` measures a run: tool calls, the last `validate` verdict,
+  synthesis scenarios and refusals, `UNMAPPED:` markers read from the files written, which outputs
+  exist, and `go test` passed/failed/skipped. With `--baseline trials/baseline.json` it exits 1 when
+  a measure gets worse; `--write-baseline` records an accepted run. `trial:run` calls it after the
+  isolation check.
+
 ## [0.14.8] — 2026-09-25
 
 `b10x-harness` joins the catalog, and `b10x` chooses a prebuilt archive only where one exists for

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -3,7 +3,7 @@ resolver = "2"
 members = ["crates/agentplugins-check", "crates/b10x"]
 
 [workspace.package]
-version = "0.14.8"
+version = "0.14.9"
 edition = "2021"
 rust-version = "1.85"
 license = "Apache-2.0"

diff --git a/Taskfile.yml b/Taskfile.yml
@@ -45,7 +45,7 @@ tasks:
       - rm -rf '{{.T}}'
       - mkdir -p -m 700 '{{.TRIALS}}'
       - mkdir -p '{{.T}}/home/.local/bin' '{{.T}}/work' '{{.T}}/tools'
-      - for t in claude codex; do p="$(command -v $t)" && ln -sf "$(readlink -f "$p")" '{{.T}}/tools/'$t; done; true
+      - for t in claude codex go gofmt; do p="$(command -v $t)" && ln -sf "$(readlink -f "$p")" '{{.T}}/tools/'$t; done; true
       - for t in cargo rustc rustup; do ln -sf "$HOME/.cargo/bin/$t" '{{.T}}/tools/'$t; done
       - echo '{}' > '{{.T}}/home/.claude.json'
       - nice -n 19 env CARGO_BUILD_JOBS=4 cargo build --quiet --locked -p b10x
@@ -62,28 +62,42 @@ tasks:
       - echo '{{.T}}'
 
   trial:run:
-    desc: "Run one headless trial in a sandbox and check its isolation: task trial:run NAME=<name> PROMPT='…' [DIR=<subdir of work>] [SEEDED=true for an upgrade trial]."
-    requires:
-      vars: [NAME, PROMPT]
+    desc: "Run one headless trial, check its isolation and measure it: task trial:run TRIAL=<name under trials/> (in a fresh sandbox), or task trial:run NAME=<name> PROMPT='…' [DIR=<subdir of work>] [SEEDED=true for an upgrade trial] in an existing one."
     vars:
+      NAME: '{{.NAME | default .TRIAL}}'
       T: '{{.TRIALS}}/{{.NAME}}'
       DIR: '{{.DIR | default ""}}'
-      SEEDED: '{{if eq (.SEEDED | default "") "true"}}--seeded{{end}}'
       VERSION:
         sh: sed -n 's/^version = "\(.*\)"$/\1/p' Cargo.toml | head -1
+    preconditions:
+      - sh: test -n '{{.TRIAL}}' || { test -n '{{.NAME}}' && test -n "$PROMPT"; }
+        msg: "set TRIAL=<name> (a definition under trials/), or NAME=<name> and PROMPT='…'"
     cmds:
-      - cp '{{.T}}/home/.claude/plugins/installed_plugins.json' '{{.T}}/registry-at-start.json'
+      - task: trial:sandbox
+        vars: { NAME: '{{.NAME}}' }
+        if: test -n '{{.TRIAL}}'
+      - cmd: cargo run --quiet --locked --bin agentplugins-check -- trial-prepare '{{.TRIAL}}' --sandbox '{{.T}}'
+        if: test -n '{{.TRIAL}}'
+      - cmd: |
+          printf '%s' "$PROMPT" > '{{.T}}/prompt.txt'
+          printf '%s' '{{.DIR}}' > '{{.T}}/workdir'
+          rm -f '{{.T}}/setup.sh' '{{.T}}/seeded'
+          if [ '{{.SEEDED}}' = true ]; then : > '{{.T}}/seeded'; fi
+        if: test -z '{{.TRIAL}}'
+      - if [ -f '{{.T}}/setup.sh' ]; then (. '{{.T}}/env' && cd '{{.T}}' && . ./setup.sh); fi
+      - rm -f '{{.T}}/registry-at-start.json'; if [ -f '{{.T}}/home/.claude/plugins/installed_plugins.json' ]; then cp '{{.T}}/home/.claude/plugins/installed_plugins.json' '{{.T}}/registry-at-start.json'; fi
       - install -m 600 "$HOME/.claude/.credentials.json" '{{.T}}/home/.claude/.credentials.json'
       - defer: rm -f '{{.T}}/home/.claude/.credentials.json'
       - |
-        (. '{{.T}}/env' && mkdir -p '{{.T}}/work/{{.DIR}}' && cd '{{.T}}/work/{{.DIR}}' &&
-         timeout 2400 claude -p "$PROMPT" --permission-mode acceptEdits \
-           --allowedTools 'Bash(b10x:*)' 'Bash(ess:*)' 'Bash(aep:*)' 'Bash(worktree:*)' 'Bash(git:*)' \
+        (. '{{.T}}/env' && dir="$(cat '{{.T}}/workdir')" && mkdir -p "{{.T}}/work/$dir" && cd "{{.T}}/work/$dir" &&
+         timeout 2400 claude -p "$(cat '{{.T}}/prompt.txt')" --permission-mode acceptEdits \
+           --allowedTools 'Bash(b10x:*)' 'Bash(ess:*)' 'Bash(aep:*)' 'Bash(worktree:*)' 'Bash(git:*)' 'Bash(go:*)' \
              'Bash(mkdir:*)' 'Bash(ls:*)' 'Bash(cat:*)' 'Bash(find:*)' 'Bash(curl:*)' 'Bash(tar:*)' \
              'Bash(sha256sum:*)' 'Bash(mv:*)' 'Bash(chmod:*)' Read Write Edit Glob Grep Skill Task Agent \
            --strict-mcp-config --output-format stream-json --verbose > '{{.T}}/run.jsonl' 2> '{{.T}}/run.err'
          echo "EXIT=$?" >> '{{.T}}/run.err')
-      - cargo run --quiet --locked --bin agentplugins-check -- trial-isolation '{{.T}}/run.jsonl' --sandbox '{{.T}}' --marketplace '{{.ROOT_DIR}}' --version '{{.VERSION}}' {{.SEEDED}}
+      - cargo run --quiet --locked --bin agentplugins-check -- trial-isolation '{{.T}}/run.jsonl' --sandbox '{{.T}}' --marketplace '{{.ROOT_DIR}}' --version '{{.VERSION}}' $([ -f '{{.T}}/seeded' ] && echo --seeded)
+      - cargo run --quiet --locked --bin agentplugins-check -- trial-report '{{.T}}/run.jsonl' --sandbox '{{.T}}' {{if .TRIAL}}--trial '{{.TRIAL}}' --baseline '{{.ROOT_DIR}}/trials/baseline.json'{{end}}
     env:
       PROMPT: '{{.PROMPT}}'
 

diff --git a/crates/agentplugins-check/src/main.rs b/crates/agentplugins-check/src/main.rs
@@ -8,8 +8,10 @@ use clap::{Parser, Subcommand};
 mod concept;
 mod evals;
 mod readiness;
+mod report;
 mod tools;
 mod trial;
+mod trials;
 
 /// The marketplace identity in every marketplace format.
 const MARKETPLACE: &str = "b10x";
@@ -928,6 +930,7 @@ fn check(root: &Path) -> Result<(), String> {
             remote: std::collections::BTreeSet::new(),
         },
     )?;
+    trials::check(root)?;
     evals::evals(root)
 }
 
@@ -997,6 +1000,83 @@ enum Top {
         #[arg(long)]
         seeded: bool,
     },
+    /// Prepare a fresh sandbox for a trial defined under `trials/`: copy and commit its fixture,
+    /// and write the prompt, working directory and setup `task trial:run` reads.
+    TrialPrepare {
+        /// The trial's name (its directory under `trials/`).
+        trial: String,
+        /// The sandbox directory made by `task trial:sandbox`.
+        #[arg(long)]
+        sandbox: PathBuf,
+    },
+    /// Measure a trial run: tool calls, validation, synthesis, `UNMAPPED:` markers, outputs and
+    /// `go test` counts, one line each; with `--baseline`, exit 1 when a measure got worse.
+    TrialReport {
+        /// The run's stream-json output.
+        run: PathBuf,
+        /// The trial under `trials/` the run executed; without it, an ad-hoc `PROMPT=` run gets
+        /// every measure that needs no definition.
+        #[arg(long)]
+        trial: Option<String>,
+        /// The sandbox directory; default: the directory holding the run.
+        #[arg(long)]
+        sandbox: Option<PathBuf>,
+        /// Compare with this baseline (`trials/baseline.json`) and exit 1 when a measure got worse.
+        #[arg(long, requires = "trial")]
+        baseline: Option<PathBuf>,
+        /// Record this run as the trial's baseline, in `--baseline` or `trials/baseline.json`.
+        #[arg(long, requires = "trial")]
+        write_baseline: bool,
+    },
+}
+
+/// `trial-report`: print the measures, compare them with the baseline, and record them.
+fn trial_report(
+    root: &Path,
+    run: &Path,
+    trial: Option<&str>,
+    sandbox: Option<PathBuf>,
+    baseline: Option<PathBuf>,
+    write_baseline: bool,
+) -> Result<bool, String> {
+    let definition = trial.map(|name| trials::load(root, name)).transpose()?;
+    let text = std::fs::read_to_string(run)
+        .map_err(|error| format!("reading {}: {error}", run.display()))?;
+    let sandbox = sandbox
+        .or_else(|| run.parent().map(Path::to_path_buf))
+        .ok_or("no sandbox: pass --sandbox")?;
+    let report = report::measure(&text, &sandbox, definition.as_ref())?;
+    println!("trial {}: {}", trial.unwrap_or("(ad hoc)"), run.display());
+    for line in &report.lines {
+        println!("  {line}");
+    }
+    let mut held = true;
+    if let (Some(trial), Some(path)) = (trial, baseline.as_deref()) {
+        match report::read_baseline(path)?.get(trial) {
+            None => println!(
+                "baseline: none recorded for `{trial}` in {}",
+                path.display()
+            ),
+            Some(then) => {
+                let worse = report::worse(then, &report.measures);
+                if worse.is_empty() {
+                    println!("baseline: no measure got worse than {}", path.display());
+                } else {
+                    held = write_baseline;
+                    eprintln!("worse than the baseline in {}:", path.display());
+                    for line in worse {
+                        eprintln!("  {line}");
+                    }
+                }
+            }
+        }
+    }
+    if let (Some(trial), true) = (trial, write_baseline) {
+        let path = baseline.unwrap_or_else(|| root.join(trials::BASELINE));
+        report::write_baseline(&path, trial, &report.measures)?;
+        println!("baseline: recorded `{trial}` in {}", path.display());
+    }
+    Ok(held)
 }
 
 #[derive(Debug, Subcommand)]
@@ -1040,6 +1120,41 @@ fn main() -> ExitCode {
                 }
             };
         }
+        Some(Top::TrialPrepare { trial, sandbox }) => {
+            return match trials::prepare(&root, &trial, &sandbox) {
+                Ok(summary) => {
+                    println!("{summary}");
+                    ExitCode::SUCCESS
+                }
+                Err(error) => {
+                    eprintln!("error: {error}");
+                    ExitCode::from(1)
+                }
+            };
+        }
+        Some(Top::TrialReport {
+            run,
+            trial,
+            sandbox,
+            baseline,
+            write_baseline,
+        }) => {
+            return match trial_report(
+                &root,
+                &run,
+                trial.as_deref(),
+                sandbox,
+                baseline,
+                write_baseline,
+            ) {
+                Ok(true) => ExitCode::SUCCESS,
+                Ok(false) => ExitCode::from(1),
+                Err(error) => {
+                    eprintln!("error: {error}");
+                    ExitCode::from(1)
+                }
+            };
+        }
     };
 
     match result {