Launch docs: self-hosting, Claude Code, account MCP, CLI installs and a full proofread

agent-testing/decision-engine.mdx

@@ -28,7 +28,7 @@ Configurable rubric judges default to **generative**. Jev is an explicit opt-in
 after validation on representative labelled examples for that rubric. A high
 confidence value is not proof of correctness or agreement with the existing judge.
 Fixed grounding, capability classification and workshop semantic services use
-Jev with generative fallback. Existing explicit policies and snapshots are preserved.
+Jev with generative fallback.
 
 The Console's evaluator editor exposes **Decision engine** and **Minimum confidence**.
 The REST authoring request accepts `decision_policy`. General evaluator writes and
@@ -48,7 +48,7 @@ MCP `upsert_evaluator` accept the same policy under `config.decision`:
 Set `backend` to `generative` to use the generative judge throughout that evaluator.
 The policy is frozen in new evaluator snapshots and included in verdict identities.
 The requested Jev release is pinned; the provider's actual served revision is
-recorded with each response. Historical results are not relabelled or rescored.
+recorded with each response. Changing a policy does not rescore earlier results.
 
 Responses must cover every question, use only declared options, and return valid
 probability distributions. Low-confidence answers, invalid responses, unavailable

agent-testing/eval.mdx

@@ -63,7 +63,7 @@ Judges are authored behaviour-scoped from the behaviour's group in the Eval libr
 
 Author from the Eval library by hand, or describe what you want to test and let Overmind generate the evaluator (`POST /api/evaluators/generate-prompt/`). The generated prompt template references the agent's output as `{{output.<path>}}` and dataset fields as `{{reference.<path>}}`, and everything stays editable before saving.
 
-`overmind sync` preloads the capability's Default eval set for every capability it created, remounted, or whose card changed, in tiers: Tier 0, rule-based checks compiled from the capability's card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. When Tier 1 authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability's **evidence profile** — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply.
+`overmind sync` refreshes the Default eval set of every current capability, in tiers: Tier 0, rule-based checks compiled from the capability's card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. Tier 1 authors once per capability; when it authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability's **evidence profile** — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply.
 
 <Frame caption="The Eval library grouped by behaviour, with each behaviour's outcome and step coverage.">
   <img src="/images/platform/eval-library.jpg" alt="Eval library grouped by behaviour" />
@@ -204,7 +204,7 @@ Each dataset row has its own entry in the breakdown, with responses from differe
 
 The datapoint popup shows **Input** beside **Output and evaluation**. Select a model to switch both panels together. Generated output is the model's response, not a label extracted by a statistical evaluator; the **Reference answer** is shown separately as grading data. Expand **Generated messages** to inspect recorded assistant and tool messages.
 
-New generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider's rendered chat template or every subsequent replay request. Older samples show **Dataset input** from the run's pinned version and state that the exact model request was not captured. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript. Existing runs do not need to be rerun to inspect their pinned dataset input.
+Generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider's rendered chat template or every subsequent replay request. A sample without a captured request shows **Dataset input** from the run's pinned version. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript.
 
 The MCP `overmind://eval-runs/{eval_run}` resource exposes up to five bounded sample inspections with separate input, output, reference, and evaluator reasoning. `input_source` distinguishes captured requests, pinned dataset rows, and unavailable input; `samples_truncated` indicates additional samples.
 

agent-testing/optimisers.mdx

@@ -20,7 +20,7 @@ The Optimiser takes an agent you can already measure and edits the agent itself.
     Candidates are graded by the eval set you pick, or the capability's active eval set by default. The set must belong to the capability. See [Eval](/agent-testing/eval#eval-sets).
   </Step>
   <Step title="A coding agent in the repository">
-    Your agent executes in **your** environment, not Overmind's. The loop is driven from your checkout by the `overmind` CLI, and the candidate diffs are written by your coding agent. **Optimiser → New run** in the Console offers **Improve prompts & code** to test changes to prompts, tool definitions and application logic, or **Compare models** to evaluate different models while keeping prompts and logic fixed. Select the capability and eval dataset, then use **Copy prompt** and paste into your coding agent in that capability's repository. Copying does not start a run. The `/overmind optimise` and `/overmind backtest` commands are unchanged; scores and progress appear in Overmind.
+    Your agent executes in **your** environment, not Overmind's. The loop is driven from your checkout by the `overmind` CLI, and the candidate diffs are written by your coding agent. **Optimiser → New run** in the Console offers **Improve prompts & code** to test changes to prompts, tool definitions and application logic, or **Compare models** to evaluate different models while keeping prompts and logic fixed. Select the capability and eval dataset, then use **Copy prompt** and paste into your coding agent in that capability's repository. Copying does not start a run. The `/overmind optimise` and `/overmind backtest` commands drive the same runs; scores and progress appear in Overmind.
   </Step>
 </Steps>
 

core/capabilities.mdx

@@ -13,7 +13,7 @@ Your **agent** in Overmind is the product itself: one per project, mapped as a g
   lines={[
     "╔══════════════════════╗                               ╔═ Overmind ══════╗",
     "║      Local repo      ║                               ║                 ║",
-    "║chassis → cards → sync╟────POST /api/v1/sync/─────┐   ║                 ║",
+    "║chassis → cards → sync╟────POST /api/v1/sync──────┐   ║                 ║",
     "╚══════════════════════╝                           │   ║  ╔════════════╗ ║",
     "                                                   ├───╫─▶║ Capability ║ ║",
     "╔══════════════════════╗                           │   ║  ╚════════════╝ ║",
@@ -72,29 +72,29 @@ After code changes: run `/overmind setup` again; it includes the final `overmind
 
 - **Identity is carried**, in order: the toml `id` → the toml `slug` → a new record. A capability the push no longer includes becomes a leftover (`archived = true` in the toml on the next `sync down`).
 - **Behaviours are minted from the trajectory map**: one contract per mapped route, with an entry anchor, the ordered anchors that follow it, and its tool set. Behaviours are what [trace scoring](/agent-testing/trace-scoring#behaviours-and-bindings) binds production traffic to; the Console labels the column **Task**.
-- **The Default eval set is preloaded** for every created, remounted or card-changed capability: Tier-0 rule-based checks compiled from the card, Tier-1 generative LLM judges grounded in it, and per-behaviour outcome and step judges for trace scoring. See [Eval](/agent-testing/eval#authoring).
+- **The Default eval set is refreshed** for every current capability on each sync: Tier-0 rule-based checks compiled from the card (deduplicated), Tier-1 generative LLM judges grounded in it (authored once per capability), and per-behaviour outcome and step judges for trace scoring. See [Eval](/agent-testing/eval#authoring).
 
 ### A scan never deletes
 
 A capability the push no longer includes becomes a **leftover**: it keeps its id and data, leaves the agent view, shows a **Not in latest scan** badge, and remounts in place when a later push reproduces it. `slug` in `overmind.toml` is the server identity — rename a capability by editing `name`, never `slug`.
 
-To remove a capability yourself, use **Delete capability** at the bottom of its page (`DELETE /api/capabilities/{id}/`). The record is soft-deleted: it leaves lists, scans and identity lookup, its traces, datasets and runs stay, and its inference alias stops serving. Because a deleted capability is invisible to later syncs, the next push of the same code mints a fresh record.
+To remove a capability yourself, use **Delete capability** at the bottom of its page (`DELETE /api/capabilities/{id}/`). The record is soft-deleted: it leaves lists, scans and identity lookup, its traces, datasets and runs stay, and its inference alias stops serving. Because a deleted capability is invisible to later syncs, the next push of the same code mints a fresh record with a new id, and `overmind sync` writes that id to `overmind.toml`. Code that sets the old UUID as `capability_id` needs the new one; a slug keeps binding.
 
 ### Telemetry
 
 Spans carrying `overmind.capability.id` — set with [`init(capability_id=...)`](/tracing/sdk-python#init) or a `capability(..., id=)` scope — bind to that capability. The value is the capability's UUID or its toml `slug`; both resolve, and both are stable through renames. `overmind.capability.name` is a display label and never binds.
 
-Ingest never creates a capability. An identity the project does not know leaves the span **unbound**: it is stored, browsable under Observability's Unbound filter, and binds retroactively when a later push makes the identity resolvable. `confirmed_at` is stamped on the capability the first time a span binds to one of its behaviours; the capability page shows **Confirmed** or **Unconfirmed** accordingly. A capability created by hand (`POST /api/capabilities/`) carries an **Observed** badge instead, because no scan can confirm or retire it.
+Ingest never creates a capability. An identity the project does not know leaves the span **unbound**: it is stored, browsable under Observability's Unbound filter, and binds retroactively when a later push makes the identity resolvable. A capability created by hand (`POST /api/capabilities/`) carries an **Observed** badge, because no scan can retire it.
 
 ## The agent page
 
-**Agent** in the sidebar opens the product view: the capability grid, or the onboarding prompt while the project is still empty. Open a capability for its page.
+**Agent** in the sidebar opens the product view: the capability grid, or the onboarding prompt while the project is still empty. Open a capability for its page. The **Repository snapshot** in the header names the repository, branch and last scan time behind the map. A scan that recorded no revision reads **Revision unavailable**; a project with no scan reads **No repository scan**.
 
 <Frame caption="The Agent page: one card per capability with its model, traffic, and live score.">
   <img src="/images/platform/agent-home.jpg" alt="Agent page with the capability grid" />
 </Frame>
 
-The header carries the name (editable in place), copyable **capability id** and **project id** chips, the slug, the badges — **Declared** or **Inferred** (whether the manifest named it), **Confirmed** or **Unconfirmed**, **Not in latest scan** — and a **Last push** line linking to the project's scan report. **View traces** opens the capability's filtered Observability view. Below it, one scrolling page:
+The header carries the name (editable in place), copyable **capability id** and **project id** chips, the slug, and the **Not in latest scan** and **Observed** badges when they apply. **View traces** opens the capability's filtered Observability view. Below it, one scrolling page:
 
 - **Facts** — the model the capability serves through (or the live deployed model behind its alias) and its source path.
 - **Prompt** — the system prompt lifted from your code, one per mode when the capability has several.
@@ -112,7 +112,7 @@ The header carries the name (editable in place), copyable **capability id** and
   <img src="/images/platform/capability-flow.jpg" alt="Trajectory map fullscreen view" />
 </Frame>
 
-The Agent page is the record of the latest sync: each capability with its card, behaviours, and whether telemetry has confirmed it.
+The Agent page is the record of the latest sync: each capability with its card and behaviours.
 
 ## Instrumentation
 

core/datasets.mdx

@@ -62,7 +62,7 @@ Quality and capability-alignment findings remain advisory, including train/eval
 
 Ask explicitly for synthetic examples. Validated batches accumulate in one active generated version, with no draft or Apply step. Retries do not add duplicate rows; interrupted generation keeps completed batches and can continue to the same target. Used versions cannot receive more batches. Seed lineage, capability context, instructions and generation engine are recorded. Synthetic answers still need independent review and cannot be split away from their seeds into held-out evaluation. Automatic preparation never generates synthetic rows.
 
-Chat activity is chronological, with expandable thinking and approval checkpoints. Clearing chat does not remove applied versions. The agent uses the first configured engine: Cursor, OpenRouter, then OpenAI, Anthropic or Gemini. With no key, the workshop opens and reports that the agent is unavailable; see [Self-hosting](/platform/self-hosting#the-data-workshop-agent).
+Chat activity is chronological, with expandable thinking and approval checkpoints. Clearing chat does not remove applied versions. The agent runs on OpenRouter, or as a Cursor Composer session when `CURSOR_API_KEY` is set (recommended); see [Self-hosting](/platform/self-hosting#the-data-workshop-agent).
 
 ## Versions
 
@@ -83,13 +83,39 @@ Rows that cannot meet a technical contract are counted with the reason, never qu
 
 Training reports overlap without dropping selected rows. Checks include normalized inputs, trace/conversation identities, configured groups, case identifiers and synthetic seed lineage. Internal identity columns are hidden from the table, filters and column counts, not removed from stored data.
 
+### What valid rows look like
+
+<Tabs>
+  <Tab title="Train">
+    A `messages` column with a conversation that ends with the assistant, in OpenAI chat format. Rows that call tools also carry a `tools` column with the tool definitions.
+
+    ```json
+    {"messages": [
+      {"role": "system", "content": "You triage support tickets."},
+      {"role": "user", "content": "My card was declined twice."},
+      {"role": "assistant", "content": "Route to billing. Priority: high."}
+    ]}
+    ```
+  </Tab>
+  <Tab title="Eval">
+    An `input` on every row and an `expected_output` to score against. `input` can be text, an object or a conversation. Other columns are kept.
+
+    ```json
+    {"input": "My card was declined twice.", "expected_output": "billing"}
+    ```
+  </Tab>
+</Tabs>
+
+A file does not need to arrive in this shape; the agent converts it during preparation.
+
 ## From your coding agent
 
 `/overmind dataset` covers the loop: land a dataset from traces, inspect it, query it with SQL, ask its agent for a change, accept a proposal. Files and exports go through the CLI, because bytes don't travel over MCP:
 
 ```bash
-overmind dataset upload rows.jsonl --intent eval
-overmind dataset export <dataset-id> --format jsonl
+overmind dataset upload transcripts.jsonl --intent train --wait   # wait for landing and the first preparation
+overmind dataset upload tickets.jsonl --split 20 --wait           # one file as a train and an eval dataset
+overmind dataset export <dataset-id> --cell 1.2 --output rows.jsonl
 ```
 
 See [MCP](/platform/mcp) and the [CLI](/platform/cli#overmind-dataset).
@@ -107,6 +133,28 @@ An export is a plain download and never counts as a use, so it won't freeze anyt
 
 Everything the workshop does is available over REST: create a dataset, add or edit a step, run the chain, message the agent, page through rows with their diff marks, and stream events as they happen. See [Building a dataset](/platform/api#building-a-dataset).
 
+## When something is refused
+
+| Message | Fix |
+| --- | --- |
+| `Row 12 has 3 cells; the header has 2.` | A CSV cell holds the delimiter. Quote it, or use JSONL |
+| `Column 'notes' appears twice.` | Rename one of the columns |
+| `The file is not UTF-8.` | Save the file as UTF-8 and upload it again |
+| `Line 40 is not valid JSON.` | Fix that line; the file lands only when every line parses |
+| `Two rows are needed to split.` | Add rows, or create a single dataset |
+| `no messages column` / `no input column` | Select **Ask Overmind to fix it** on the cell |
+| `Longest dataset row is ≈N tokens but this model's … context is M tokens` | Drop or shorten the long rows, or pick a model with a longer context |
+| `The dataset is busy.` | The agent or a run is working on it. Wait for it to finish |
+
+## Limits
+
+| Limit | Value |
+| --- | --- |
+| One uploaded file | 2 GB |
+| One `.json` document | 256 MB; use JSONL above that |
+| One value in a row | 200,000 characters; longer values become a preview marker |
+| One cell's run time | 300 seconds |
+
 ## Next steps
 
 <CardGroup cols={2}>

core/observability.mdx

@@ -101,6 +101,8 @@ A project-scoped key pins every span to its project. Sending the same spans twic
 
 Overmind already understands the OTel GenAI, OpenLLMetry and OpenInference conventions, so models, providers, token counts, costs, tool names and message payloads from existing instrumentation land without any mapping of your own.
 
+Overmind prices each LLM span once, at ingest. A cost the span reports is kept; otherwise the cost comes from the model, token counts, cache reads and provider, at list prices for the span's start time. Spans from OTLP and from connectors are priced the same way.
+
 These are the ones you set:
 
 | Attribute | What it does |