From f7ec82d9aa76ec78611a5c20ae2abe31879c99e9 Mon Sep 17 00:00:00 2001 From: Tyler-Edwards Date: Tue, 29 Sep 2026 17:58:27 -0700 Subject: [PATCH 01/19] Document account MCP access and optional plugin --- platform/mcp.mdx | 70 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 61 insertions(+), 9 deletions(-) diff --git a/platform/mcp.mdx b/platform/mcp.mdx index 72cd1d1..1b34ed1 100644 --- a/platform/mcp.mdx +++ b/platform/mcp.mdx @@ -1,9 +1,9 @@ --- title: MCP -description: The Overmind MCP server — setup per coding agent, authentication, the 32 tools, 11 prompts and the resources, and the CLI handoff for local work. +description: The Overmind MCP server — setup per coding agent, authentication, the 36 tools, workflow prompts and the resources, and the CLI handoff for local work. --- -MCP is the platform's first-class agent surface: a project-scoped [Model Context Protocol](https://modelcontextprotocol.io) server at `/api/mcp/` that shares its services with the REST API and the Console. A coding agent connected to it can inspect health and failures, land and shape datasets, author evaluators and run evals, launch and follow training, schedule optimisation runs, and drive connectors and instrumentation — everything except the steps that need your filesystem, which it hands to the [CLI](/platform/cli). +MCP is the platform's first-class agent surface: an account-aware [Model Context Protocol](https://modelcontextprotocol.io) server at `/api/mcp/` that shares its services with the REST API and the Console. A coding agent connected to it can inspect health and failures, land and shape datasets, author evaluators and run evals, launch and follow training, schedule optimisation runs, and drive connectors and instrumentation — everything except the steps that need your filesystem, which it hands to the [CLI](/platform/cli). ## Setup @@ -56,17 +56,54 @@ claude mcp add --transport http overmind https://api.overmindlab.ai/api/mcp/ --h Codex takes the same entry in `.codex/config.toml`. Self-hosted: replace the URL with `http://localhost:8000/api/mcp/` (`overmind init --env local`). +## Optional plugin + +The Codex, Cursor and Claude Code plugins package the existing MCP connection, +Overmind branding and the shared workflow skills. MCP remains the complete +integration: tool descriptions, essential operating guidance, resources and +native prompts are available to clients that do not install a plugin. There is +no separate plugin UI. + +The bundled `.mcp.json` targets the hosted API and uses OAuth account sign-in. +The connection can access every active project your account is authorized to access, +including future memberships. Account API keys support the same model; project +API keys retain their narrower access. +For a local or self-hosted deployment, use `overmind init --env local --ide codex` +and `overmind sync`, or configure the intended endpoint directly. Keep one active +MCP connection for the intended account or project; a plugin is optional when that connection exists. + +Start with `list_projects`. When a visual view is useful, read +`overmind://project/current?project_id=ID` and open its +`console_url`. The server builds this ordinary Console link from `FRONTEND_URL` +and the authenticated project. Resource IDs can identify a more specific +Console page. The browser requires its own Console session, and opening a page +does not authorize mutations. + ## Authentication -The key must be **project-scoped** — pinned to exactly one active project the key's owner is a member of — and carry `read` and/or `write` permission. The project is bound from the key; no tool accepts a project id. A key with only `read` sees the read-only tools; the rest need `write`. A key with an IP allowlist is refused from other addresses. `Authorization: Bearer ovr_…` is accepted as an alternative to `X-Api-Key`. +OAuth connections and account API keys discover authorized projects through `list_projects`. Pass the chosen `project_id` on every project tool and as a query parameter on resource URIs. Membership is checked for each operation. There is no shared selected-project state, so concurrent work cannot switch another request's project. Returned resource links retain the project ID. A project API key can only use its configured project and can omit `project_id`. + +Read access exposes read-only tools; mutations require write access. API keys retain expiry, active-user and IP-allowlist checks. OAuth grants request `overmind:read` and/or `overmind:write` and require Console sign-in and explicit consent. Tokens use S256 PKCE, exact redirect matching, one-time codes and resource binding; OAuth tokens are accepted only by MCP. Access tokens expire after one hour. Connections remain authorized until revoked, with no refresh-token time limit. Refresh tokens still rotate; reuse revokes the connection's token family. Disabled accounts and lost project memberships continue to lose access. + +Set `MCP_SERVER_URL` to the canonical HTTPS `/api/mcp/` endpoint on a hosted deployment, with `FRONTEND_URL` pointing at its Console. HTTP is accepted only for local loopback development. Configured servers advertise OAuth discovery, public-client registration (`token_endpoint_auth_method: none`), authorization, token and revocation endpoints. API-key connections continue to use `X-Api-Key` or `Authorization: Bearer`. -The server answers `401` for authentication errors and `403` otherwise, without a `WWW-Authenticate: Bearer` challenge, so MCP clients never start OAuth registration against it. Requests must be `POST` with `application/json`, from an allowed host and origin, and at most the protocol's default body size. Errors are JSON-RPC values, never exceptions: `{"error": {code, message, retryable, fields}}`. +For OpenAI publishing, set `OPENAI_APPS_CHALLENGE` to the portal's public ownership token; the API serves it at `/.well-known/openai-apps-challenge`. Deploy the backend migration and Console consent route before configuring OAuth in the submission portal, then verify the domain and scan tools. + +MCP transport requests use `POST` with `application/json`, from an allowed host and origin, within the protocol's body limit. Authentication failures return `401`; authorization failures return `403`. Tool errors are structured results: `{"error": {code, message, retryable, fields}}`. The public surface is read and write only. There are no delete, cancel or generic API tools; `retry_deployment` is the one lifecycle exception and covers exactly the failed-or-deleted deployment case. ## Tools -Thirty-two tools, all project-scoped, each with a strict input and output schema the client reads at call time. Every result carries the same object twice — as `structuredContent` and as compact JSON text — plus resource links for the entities it touched. Arguments marked `?` are optional. +Thirty-six tools, each with a strict input and output schema the client reads at call time. Every result carries the same object twice — as `structuredContent` and as compact JSON text — plus resource links for the entities it touched. Arguments marked `?` are optional. + +Project tools also accept `project_id`, required for account connections. `list_projects` and `get_model_catalog` do not need a project. + +### Projects + +| Tool | Arguments | Returns | +| --- | --- | --- | +| `list_projects` | `limit?`, `offset?` | Authorized active projects, Console/resource links and pagination | ### Observability @@ -166,11 +203,11 @@ Eleven prompts compose the tools into multi-step workflows with human checkpoint ## Resources -Read-only, project-scoped entity state under `overmind://`. Static: +Read-only project entity state under `overmind://`. Account connections add `?project_id=ID`; follow returned links to preserve scope. Static: | URI | Content | | --- | --- | -| `overmind://project/current` | The authenticated project identity and safe metadata | +| `overmind://project/current` | The authenticated project identity, provenance, safe metadata and ordinary `console_url` | | `overmind://dataset-upload` | CLI guidance for uploading local dataset bytes | | `overmind://dataset-export` | CLI guidance for downloading a committed dataset | | `overmind://checkpoint-download` | CLI guidance for downloading an archived checkpoint | @@ -186,6 +223,21 @@ Each deployment resource also includes a current `worker` section: `state` (`war Three things never cross MCP: file bytes, repository edits, and provider secrets. A dataset file goes up with `overmind dataset upload`; a dataset or checkpoint comes down with `overmind dataset export` and `overmind model download-checkpoint`; repository discovery is `/overmind setup` then `overmind sync`; an optimisation run's candidates are generated and replayed by `overmind optimise` in your checkout; a model swap is a prompt (`get_model_swap_prompt`) the agent applies with its own editing tools; a connector's provider key is entered with `overmind connector add` in a terminal. The prompts above name the exact command at each handoff, and `--json` on the CLI returns the ids the agent needs to continue. -## The skill +## Workflow skills -`overmind init` installs the `overmind` skill next to the MCP entry, and on Cursor and Claude Code the `/overmind` slash commands (`onboard`, `setup`, `ensure-tracing`, `dataset`, `finetune`, `optimise`, `backtest`) that load its references. The skill holds the conventions the tools assume — list first, poll jobs with `get_job`, the dataset contracts that gate every run — and is the only place the local scan workflow (`/overmind setup`) lives; the native prompts cover the server-side workflows. +`overmind init` installs the shared workflow skills next to the MCP entry. The optional plugin bundles the same files. The main `overmind` skill handles local setup, repository discovery and workflows across surfaces; choose a focused skill for work in one product surface. + +| Skill | Workflow | +| --- | --- | +| `overmind-agent` | Capability map, behaviour coverage and repository provenance | +| `overmind-observability` | Trace investigations, failures, latency and instrumentation gaps | +| `overmind-datasets` | Data Workshop preparation, proposals, generation and export | +| `overmind-evaluations` | Evaluators, eval sets, runs and baseline comparisons | +| `overmind-optimiser` | Prompt/code experiments and model comparisons | +| `overmind-training` | Exact data preparation, costs, fine-tuning and held-out evaluation | +| `overmind-inference` | Serving metrics, worker state, live routing and activation | +| `overmind-integrations` | Tracing providers, boundary mappings and verified imports | + +Run `overmind skills list --verbose` to see the installed SDK's catalog. Refresh selected workflows with `overmind skills sync overmind-observability overmind-datasets --ide codex`, or run `overmind init --ide codex` to install the complete set. + +On Cursor and Claude Code, the `/overmind` slash commands (`onboard`, `setup`, `ensure-tracing`, `dataset`, `finetune`, `optimise`, `backtest`) continue to load the main skill's local references. Essential usage guidance travels in MCP initialization, tool descriptions and resources; native prompts remain the shared server-side workflows. Console navigation opens the existing product pages. From d10c331fcbaeaf85ee5ee18f88e583326521ac10 Mon Sep 17 00:00:00 2001 From: freyam Date: Wed, 30 Sep 2026 12:28:24 +0100 Subject: [PATCH 02/19] Document Windows installs of the overmind CLI Virtual environment, PATH, execution policy and the long-path limit for the tracing extra. --- platform/cli.mdx | 21 +++++++++++++++++++++ tracing/sdk-python.mdx | 4 ++-- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/platform/cli.mdx b/platform/cli.mdx index f95336b..b0b1e3f 100644 --- a/platform/cli.mdx +++ b/platform/cli.mdx @@ -28,6 +28,27 @@ Then in your coding agent, `/overmind setup` scans the repository, writes capabi When Overmind is a project dependency, run commands through that project's environment (`uv run overmind`, `poetry run overmind`, or its virtual-environment executable). An editable installation does not update skill copies already in a repository. Refresh the installed skill with `uv run overmind skills sync overmind --ide ` using the appropriate project runner, then start a new conversation so the coding agent reads it. +## Windows + +Install into a virtual environment. It keeps package paths short and puts `overmind` on `PATH` while it is active: + +```powershell +python -m venv .venv +.venv\Scripts\Activate.ps1 # cmd.exe: .venv\Scripts\activate.bat +pip install overmind +overmind --help +``` + +If PowerShell refuses to run `Activate.ps1`, allow local scripts for your user once: + +```powershell +Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser +``` + +A global `pip install` puts `overmind.exe` in the interpreter's `Scripts` folder, which is not on `PATH` by default. The command is then not found; `python -m overmind` runs the same CLI. + +`overmind[tracing]` depends on litellm, some of whose files sit more than 120 characters below `site-packages`. Windows limits a full path to 260 characters unless long paths are enabled. The Microsoft Store `Python 3.x` packages install into a folder of about 140 characters, so the install fails with `OSError: [Errno 2] No such file or directory`. Use a virtual environment in a short folder such as `C:\dev\app\.venv`, or have an administrator enable the **Enable Win32 long paths** group policy (or set `LongPathsEnabled` to `1` under `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem`) and reboot. + ## `overmind init` ```bash diff --git a/tracing/sdk-python.mdx b/tracing/sdk-python.mdx index 82f0ba2..56498ef 100644 --- a/tracing/sdk-python.mdx +++ b/tracing/sdk-python.mdx @@ -6,11 +6,11 @@ description: Reference for the overmind package — init(), provider auto-instru The `overmind` package instruments your LLM stack over OpenTelemetry and exports spans to Overmind. One `init()` call captures every supported provider call automatically; `run()` brackets one agent run; decorators trace the code around the model calls; a few one-line calls declare what the run was asked to do and what it delivered, which is what [trace scoring](/agent-testing/trace-scoring) judges. The package is MIT. ```bash -pip install "overmind[tracing]" # the tracing SDK, every provider instrumentor, LangChain coverage +pip install "overmind[tracing]" # the tracing SDK, every provider instrumentor, LangChain coverage, span cost pricing pip install overmind # the `overmind` CLI and the inference client only ``` -The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. There are no other extras. +The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. The extra also carries litellm, which prices spans from their token counts. There are no other extras. On Windows, see [Windows](/platform/cli#windows). ## `init()` From 1a56e790cd307feb735eda351058175a9dc057c1 Mon Sep 17 00:00:00 2001 From: freyam Date: Wed, 30 Sep 2026 14:03:21 +0100 Subject: [PATCH 03/19] Document server-side span pricing and drop the litellm long-path note --- core/observability.mdx | 2 +- platform/cli.mdx | 4 +--- tracing/sdk-python.mdx | 4 ++-- 3 files changed, 4 insertions(+), 6 deletions(-) diff --git a/core/observability.mdx b/core/observability.mdx index e3c8002..7655e92 100644 --- a/core/observability.mdx +++ b/core/observability.mdx @@ -99,7 +99,7 @@ A project-scoped key pins every span to its project. Sending the same spans twic ### The attributes Overmind reads -Overmind already understands the OTel GenAI, OpenLLMetry and OpenInference conventions, so models, providers, token counts, costs, tool names and message payloads from existing instrumentation land without any mapping of your own. +Overmind already understands the OTel GenAI, OpenLLMetry and OpenInference conventions, so models, providers, token counts, costs, tool names and message payloads from existing instrumentation land without any mapping of your own. A span that reports tokens but no cost is priced at ingest from its model's list price on the span's start date, counting cache reads at the cache rate; a reported cost is kept as sent. These are the ones you set: diff --git a/platform/cli.mdx b/platform/cli.mdx index b0b1e3f..389d0ca 100644 --- a/platform/cli.mdx +++ b/platform/cli.mdx @@ -30,7 +30,7 @@ When Overmind is a project dependency, run commands through that project's envir ## Windows -Install into a virtual environment. It keeps package paths short and puts `overmind` on `PATH` while it is active: +Install into a virtual environment. It puts `overmind` on `PATH` while it is active: ```powershell python -m venv .venv @@ -47,8 +47,6 @@ Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser A global `pip install` puts `overmind.exe` in the interpreter's `Scripts` folder, which is not on `PATH` by default. The command is then not found; `python -m overmind` runs the same CLI. -`overmind[tracing]` depends on litellm, some of whose files sit more than 120 characters below `site-packages`. Windows limits a full path to 260 characters unless long paths are enabled. The Microsoft Store `Python 3.x` packages install into a folder of about 140 characters, so the install fails with `OSError: [Errno 2] No such file or directory`. Use a virtual environment in a short folder such as `C:\dev\app\.venv`, or have an administrator enable the **Enable Win32 long paths** group policy (or set `LongPathsEnabled` to `1` under `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem`) and reboot. - ## `overmind init` ```bash diff --git a/tracing/sdk-python.mdx b/tracing/sdk-python.mdx index 56498ef..6a4b0ca 100644 --- a/tracing/sdk-python.mdx +++ b/tracing/sdk-python.mdx @@ -6,11 +6,11 @@ description: Reference for the overmind package — init(), provider auto-instru The `overmind` package instruments your LLM stack over OpenTelemetry and exports spans to Overmind. One `init()` call captures every supported provider call automatically; `run()` brackets one agent run; decorators trace the code around the model calls; a few one-line calls declare what the run was asked to do and what it delivered, which is what [trace scoring](/agent-testing/trace-scoring) judges. The package is MIT. ```bash -pip install "overmind[tracing]" # the tracing SDK, every provider instrumentor, LangChain coverage, span cost pricing +pip install "overmind[tracing]" # the tracing SDK, every provider instrumentor, LangChain coverage pip install overmind # the `overmind` CLI and the inference client only ``` -The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. The extra also carries litellm, which prices spans from their token counts. There are no other extras. On Windows, see [Windows](/platform/cli#windows). +The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. There are no other extras. On Windows, see [Windows](/platform/cli#windows). ## `init()` From d146414b8d2595ae51c034da1a2c2220df2db898 Mon Sep 17 00:00:00 2001 From: freyam Date: Wed, 30 Sep 2026 14:15:02 +0100 Subject: [PATCH 04/19] Put CLI install steps for macOS, Linux and Windows in one tabbed section --- platform/cli.mdx | 91 +++++++++++++++++++++++++++++------------- tracing/sdk-python.mdx | 2 +- 2 files changed, 64 insertions(+), 29 deletions(-) diff --git a/platform/cli.mdx b/platform/cli.mdx index 389d0ca..01d6c1e 100644 --- a/platform/cli.mdx +++ b/platform/cli.mdx @@ -5,47 +5,82 @@ description: The overmind command — init, sync, chassis, dataset upload and ex The `overmind` command ships with the Python package. It is the local half of every workflow that touches your filesystem: it installs the skill and MCP entry, syncs `overmind.toml` with the server, lands files as datasets, adds tracing connectors, drives optimisation runs in your working tree, and downloads weights. The MCP server never edits a repository or moves bytes; the CLI does. The package is MIT. -```bash -pip install overmind # the CLI and the inference client -pip install "overmind[tracing]" # plus the tracing SDK and every provider instrumentor -``` +## Install -```bash -uv tool install overmind -# or -pipx install overmind -``` +Install the CLI into a virtual environment, or as a standalone tool with `uv` or `pipx`. Both put `overmind` on `PATH`. -Every command that talks to the API resolves its key in the same order: `--api-key`, then the project credential in `.overmind/credentials.toml` (written by `overmind sync`), then `OVERMIND_API_KEY`. The base URL comes from `--api-url`, `OVERMIND_API_URL`, or `base-url` in `overmind.toml`; the hosted default is `https://api.overmindlab.ai`. Keys go over the wire as `X-Api-Key`. `overmind --version` prints the package version; `overmind --help` prints flags. + + + ```bash + python3 -m venv .venv + source .venv/bin/activate + pip install overmind + overmind --help + ``` -```bash -export OVERMIND_API_KEY= -overmind init --ide cursor # or claude | opencode | codex -overmind sync # create the project, save the project key -``` + As a standalone tool, outside any project: -Then in your coding agent, `/overmind setup` scans the repository, writes capabilities into `overmind.toml`, and runs `overmind sync` to push them. The installed skill recognises the repository and verified connection, previews the onboarding roadmap, and reports repository-specific findings alongside the current stage out of ten, completed stages, and stages still to follow. A pre-sync handoff names the destination, included data, credential checks performed, and configuration to review. The finish reports the synced result, outstanding verification, and one recommended next action without automatically starting it. There is no `overmind scan` command. + ```bash + uv tool install overmind # or: pipx install overmind + ``` -When Overmind is a project dependency, run commands through that project's environment (`uv run overmind`, `poetry run overmind`, or its virtual-environment executable). An editable installation does not update skill copies already in a repository. Refresh the installed skill with `uv run overmind skills sync overmind --ide ` using the appropriate project runner, then start a new conversation so the coding agent reads it. + Homebrew's Python refuses a global `pip install` with `externally-managed-environment`. Use a virtual environment or a tool install. + + + ```bash + python3 -m venv .venv + source .venv/bin/activate + pip install overmind + overmind --help + ``` + + As a standalone tool, outside any project: + + ```bash + uv tool install overmind # or: pipx install overmind + ``` -## Windows + Debian and Ubuntu mark the system Python as externally managed, so a global `pip install` fails with `externally-managed-environment`. Use a virtual environment or a tool install. + + + Install into a virtual environment. It puts `overmind` on `PATH` while it is active: -Install into a virtual environment. It puts `overmind` on `PATH` while it is active: + ```powershell + python -m venv .venv + .venv\Scripts\Activate.ps1 # cmd.exe: .venv\Scripts\activate.bat + pip install overmind + overmind --help + ``` -```powershell -python -m venv .venv -.venv\Scripts\Activate.ps1 # cmd.exe: .venv\Scripts\activate.bat -pip install overmind -overmind --help + If PowerShell refuses to run `Activate.ps1`, allow local scripts for your user once: + + ```powershell + Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser + ``` + + A global `pip install` puts `overmind.exe` in the interpreter's `Scripts` folder, which is not on `PATH` by default. The command is then not found; `python -m overmind` runs the same CLI. + + + +The base package is the CLI and the inference client. The `tracing` extra adds the tracing SDK and every provider instrumentor: + +```bash +pip install "overmind[tracing]" ``` -If PowerShell refuses to run `Activate.ps1`, allow local scripts for your user once: +When Overmind is a project dependency, run commands through that project's environment (`uv run overmind`, `poetry run overmind`, or its virtual-environment executable). An editable installation does not update skill copies already in a repository. Refresh the installed skill with `uv run overmind skills sync overmind --ide ` using the appropriate project runner, then start a new conversation so the coding agent reads it. + +## First run -```powershell -Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser +Every command that talks to the API resolves its key in the same order: `--api-key`, then the project credential in `.overmind/credentials.toml` (written by `overmind sync`), then `OVERMIND_API_KEY`. The base URL comes from `--api-url`, `OVERMIND_API_URL`, or `base-url` in `overmind.toml`; the hosted default is `https://api.overmindlab.ai`. Keys go over the wire as `X-Api-Key`. `overmind --version` prints the package version; `overmind --help` prints flags. + +```bash +export OVERMIND_API_KEY= # PowerShell: $env:OVERMIND_API_KEY = "" +overmind init --ide cursor # or claude | opencode | codex +overmind sync # create the project, save the project key ``` -A global `pip install` puts `overmind.exe` in the interpreter's `Scripts` folder, which is not on `PATH` by default. The command is then not found; `python -m overmind` runs the same CLI. +Then in your coding agent, `/overmind setup` scans the repository, writes capabilities into `overmind.toml`, and runs `overmind sync` to push them. The installed skill recognises the repository and verified connection, previews the onboarding roadmap, and reports repository-specific findings alongside the current stage out of ten, completed stages, and stages still to follow. A pre-sync handoff names the destination, included data, credential checks performed, and configuration to review. The finish reports the synced result, outstanding verification, and one recommended next action without automatically starting it. There is no `overmind scan` command. ## `overmind init` diff --git a/tracing/sdk-python.mdx b/tracing/sdk-python.mdx index 6a4b0ca..90d2eb1 100644 --- a/tracing/sdk-python.mdx +++ b/tracing/sdk-python.mdx @@ -10,7 +10,7 @@ pip install "overmind[tracing]" # the tracing SDK, every provider instrumentor pip install overmind # the `overmind` CLI and the inference client only ``` -The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. There are no other extras. On Windows, see [Windows](/platform/cli#windows). +The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. There are no other extras. Install steps for macOS, Linux and Windows are on the [CLI page](/platform/cli#install). ## `init()` From 47bdf1c89f65a7f229badfd67b4be87801606652 Mon Sep 17 00:00:00 2001 From: freyam Date: Wed, 30 Sep 2026 14:21:40 +0100 Subject: [PATCH 05/19] Draw one underline on content tabs and keep the focus ring inside the tab --- custom.css | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/custom.css b/custom.css index f3802ec..1a81867 100644 --- a/custom.css +++ b/custom.css @@ -1282,6 +1282,17 @@ html[data-page-mode="wide"] #content-area { color: var(--text) !important; border-bottom-color: var(--copper-500) !important; } +/* Mintlify underlines the inner tab button too; the tab's own border is the one indicator. */ +#content-area [data-component-part="tab-button"] { + border: 0 !important; + padding: 0 !important; + margin: 0 !important; + color: inherit !important; + font: inherit !important; +} +#content-area [role="tab"]:focus-visible { + outline-offset: -2px; +} /* ─── kbd ───────────────────────────────────────────────────────────────── */ From 13694a659425c5650dce037472289bd5ece28299 Mon Sep 17 00:00:00 2001 From: freyam Date: Wed, 30 Sep 2026 16:15:03 +0100 Subject: [PATCH 06/19] Keep code-block buttons from covering the code Hide Mintlify's fade overlay, give the button group its own chip, and keep the header rule off it. --- custom.css | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/custom.css b/custom.css index 1a81867..58f44a8 100644 --- a/custom.css +++ b/custom.css @@ -894,7 +894,7 @@ html[data-page-mode="wide"] #content-area { } /* Code-block header (filename / language tabs) as a strip on the same panel. */ -#content-area :is(.code-block, .code-group) > div:first-child:not(:has(pre)) { +#content-area :is(.code-block, .code-group) > div:first-child:not(:has(pre), [data-floating-buttons]) { background: var(--surface) !important; border-bottom: 1px solid var(--frame) !important; font-family: var(--font-mono) !important; @@ -925,9 +925,15 @@ html[data-page-mode="wide"] #content-area { color: var(--text) !important; background: var(--surface-2) !important; } +/* Mintlify's fade overlay is a gradient band wider than the buttons; the flat + canvas has no gradients, so the buttons sit on a code-bg chip instead. */ #content-area [data-fade-overlay] { - background: var(--code-bg) !important; - background-image: none !important; + display: none !important; +} +#content-area [data-floating-buttons] { + z-index: 2; + padding-left: 6px; + background: var(--code-bg); } /* ─── Tables — hairlines only ───────────────────────────────────────────── */ From b9d3498c232fbf66eb746fc4f8f2ade25a555924 Mon Sep 17 00:00:00 2001 From: freyam Date: Wed, 30 Sep 2026 16:28:38 +0100 Subject: [PATCH 07/19] Clip code before its floating buttons instead of under them Reserve a gutter for the buttons in the code viewport; drop the chip and restore the header rule's exclusion. --- custom.css | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/custom.css b/custom.css index 58f44a8..2254cc8 100644 --- a/custom.css +++ b/custom.css @@ -96,6 +96,8 @@ --text-3: var(--stone-400); --text-disabled: var(--stone-600); --code-bg: #0e0c09; + /* Floating buttons (58px) + their right inset (16px) + 8px of air. */ + --code-button-gutter: 82px; --font-display: "PP Mondwest", ui-serif, Georgia, serif; --font-nav: "NeueBit", ui-monospace, monospace; @@ -901,6 +903,14 @@ html[data-page-mode="wide"] #content-area { font-size: 12px !important; color: var(--text-3) !important; } +/* The code viewport stops at a gutter under the floating buttons, so a long line + clips before them instead of running beneath; the fade overlay has nothing to cover. */ +#content-area .code-block > [data-floating-buttons] + [data-component-part="code-block-root"] { + padding-right: var(--code-button-gutter); +} +#content-area [data-fade-overlay] { + display: none !important; +} #content-area :is(.code-block, .code-group) [role="tab"] { font-family: var(--font-mono) !important; font-size: 12px !important; @@ -914,27 +924,17 @@ html[data-page-mode="wide"] #content-area { border-bottom-color: var(--copper-500) !important; } -#content-area [data-testid="copy-code-button"] { +#content-area :is([data-testid="copy-code-button"], #ask-assistant-code-block-button) { background: transparent !important; border: 0 !important; color: var(--text-3) !important; clip-path: var(--notch); transition: color 150ms ease, background-color 150ms ease; } -#content-area [data-testid="copy-code-button"]:hover { +#content-area :is([data-testid="copy-code-button"], #ask-assistant-code-block-button):hover { color: var(--text) !important; background: var(--surface-2) !important; } -/* Mintlify's fade overlay is a gradient band wider than the buttons; the flat - canvas has no gradients, so the buttons sit on a code-bg chip instead. */ -#content-area [data-fade-overlay] { - display: none !important; -} -#content-area [data-floating-buttons] { - z-index: 2; - padding-left: 6px; - background: var(--code-bg); -} /* ─── Tables — hairlines only ───────────────────────────────────────────── */ From 235cd4dfcf3a453c5960d76597dd534547485f39 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 05:47:57 +0100 Subject: [PATCH 08/19] Drop the span pricing note from the observability page --- core/observability.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/observability.mdx b/core/observability.mdx index 7655e92..e3c8002 100644 --- a/core/observability.mdx +++ b/core/observability.mdx @@ -99,7 +99,7 @@ A project-scoped key pins every span to its project. Sending the same spans twic ### The attributes Overmind reads -Overmind already understands the OTel GenAI, OpenLLMetry and OpenInference conventions, so models, providers, token counts, costs, tool names and message payloads from existing instrumentation land without any mapping of your own. A span that reports tokens but no cost is priced at ingest from its model's list price on the span's start date, counting cache reads at the cache rate; a reported cost is kept as sent. +Overmind already understands the OTel GenAI, OpenLLMetry and OpenInference conventions, so models, providers, token counts, costs, tool names and message payloads from existing instrumentation land without any mapping of your own. These are the ones you set: From d61c671e4e8084e9efaad9824dd0c990f9e1f340 Mon Sep 17 00:00:00 2001 From: Tyler-Edwards Date: Wed, 30 Sep 2026 22:02:33 -0700 Subject: [PATCH 09/19] Document local Codex setup and training input format --- models/training.mdx | 5 +++++ platform/self-hosting.mdx | 7 +++++++ 2 files changed, 12 insertions(+) diff --git a/models/training.mdx b/models/training.mdx index cda18d6..15384af 100644 --- a/models/training.mdx +++ b/models/training.mdx @@ -27,6 +27,11 @@ Independent train and eval data gives meaningful held-out scores. Overmind repor Training wizard with dataset setup and model picker +For MCP `start_finetune`, omit `hyperparameters` to derive model-specific defaults. +When supplying settings, use `"training_type": {"type": "Lora"}` or +`"training_type": {"type": "Full"}`. A string training type is rejected before +job creation with a field-level validation error. + ## Exact preprocessing Workshop quality reviews remain advisory and are not repeated as launch approval gates. Training performs model-specific preparation, not repairs to labels, evidence or task scope. diff --git a/platform/self-hosting.mdx b/platform/self-hosting.mdx index 2fff428..e38eef0 100644 --- a/platform/self-hosting.mdx +++ b/platform/self-hosting.mdx @@ -32,6 +32,13 @@ Use any login credentials to sign in, self hosted mode doesn't require any email Start by creating a project by following the [Quickstart](/quickstart). +For Codex on this machine, `overmind init --ide codex --env local` configures +`http://localhost:8000/api/mcp/`. Then run `overmind sync` with the temporary +account key from the Console. Sync saves a project-scoped key; reload Codex +once to connect. For another self-hosted address, set `OVERMIND_API_URL` before +both commands. + + For a demo workspace with thirty days of traffic, datasets, runs and served models, run the seed: ```bash From b10340bdc5191b384bb3ceabb8b10fef42d291d1 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 08:26:10 +0100 Subject: [PATCH 10/19] Document self-hosting and Claude Code onboarding as they run Startup needs only OpenRouter; Claude Code MCP is local scope; verify by trace_id. --- platform/cli.mdx | 9 +++++---- platform/glossary.mdx | 2 +- platform/mcp.mdx | 22 +++++----------------- platform/self-hosting.mdx | 35 +++++++++++++---------------------- quickstart.mdx | 4 ++-- tracing/sdk-python.mdx | 2 +- 6 files changed, 27 insertions(+), 47 deletions(-) diff --git a/platform/cli.mdx b/platform/cli.mdx index f95336b..234c42c 100644 --- a/platform/cli.mdx +++ b/platform/cli.mdx @@ -20,6 +20,7 @@ Every command that talks to the API resolves its key in the same order: `--api-k ```bash export OVERMIND_API_KEY= +export OVERMIND_API_URL= # self-hosted only, e.g. http://localhost:8000 overmind init --ide cursor # or claude | opencode | codex overmind sync # create the project, save the project key ``` @@ -41,18 +42,18 @@ overmind init --ide claude --env local | `--env` | `production` | `production`, `staging`, or `local` (`http://localhost:8000`) | | `--api-url` | `$OVERMIND_API_URL` | Base URL; overrides `--env` for both the MCP entry and `overmind.toml` | -Installs the `overmind` skill under the client's directory (`.cursor/skills/`, `.claude/skills/`, `.opencode/skills/`, `.agents/skills/`), writes the `/overmind` slash commands (Cursor and Claude Code only: `.cursor/commands/` or `.claude/commands/`), adds the Overmind MCP entry to the client's own config file, and seeds `overmind.toml` when it is missing. Other configured MCP servers are left untouched. An existing `overmind.toml` is left unchanged. +Installs the `overmind` skill under the client's directory (`.cursor/skills/`, `.claude/skills/`, `.opencode/skills/`, `.agents/skills/`), writes the `/overmind` slash commands (Cursor only: `.cursor/commands/`), adds the Overmind MCP entry to the client's own config, and seeds `overmind.toml` when it is missing. Other configured MCP servers are left untouched. An existing `overmind.toml` is left unchanged. | Client | MCP config | Skill install | | --- | --- | --- | | Cursor | `.cursor/mcp.json` | `.cursor/skills` | -| Claude Code | `.mcp.json` | `.claude/skills` | +| Claude Code | `~/.claude.json`, local scope for this directory | `.claude/skills` | | OpenCode | `opencode.json` | `.opencode/skills` | | Codex | `.codex/config.toml` | `.agents/skills` | -The MCP entry is written with the saved project key when `.overmind/credentials.toml` already exists; otherwise it is written without a key and `overmind sync` fills it in. Any file that receives a key is added to `.git/info/exclude` and set to mode `0600`; the CLI refuses to write a key into a tracked file. Codex is limited to `production` and `staging` unless `--api-url` is set. `init` does not persist the bootstrap account key. +The MCP entry is written with the saved project key when `.overmind/credentials.toml` already exists; otherwise it is written without a key and `overmind sync` fills it in. Claude Code's entry is a local-scope server in `~/.claude.json` (or `$CLAUDE_CONFIG_DIR/.claude.json`), so it needs no `.mcp.json` approval and a committed `.mcp.json` is left untouched. Any repository file that receives a key is added to `.git/info/exclude` and set to mode `0600`; the CLI refuses to write a key into a tracked file. `init` does not persist the bootstrap account key. -Slash commands written for Cursor and Claude Code: `/overmind`, `/overmind onboard`, `/overmind setup`, `/overmind ensure-tracing`, `/overmind dataset`, `/overmind finetune`, `/overmind optimise`, `/overmind backtest`. +Slash commands: `/overmind`, `/overmind onboard`, `/overmind setup`, `/overmind ensure-tracing`, `/overmind dataset`, `/overmind finetune`, `/overmind optimise`, `/overmind backtest`. Cursor gets a command file for each; Claude Code passes the word after `/overmind` to the skill, which opens the matching reference. ## `overmind sync` diff --git a/platform/glossary.mdx b/platform/glossary.mdx index 350104e..8147be0 100644 --- a/platform/glossary.mdx +++ b/platform/glossary.mdx @@ -53,7 +53,7 @@ description: Definitions of terms used across Overmind, as they appear in the Co **Task execution** — one unit bound to a behaviour, with its verdicts, execution score, route flags, and session score. Binding source is `declared` (the span carried the identity), `anchor_join` (matched on anchors) or `unbound`. Status `completed`, `error` or `interrupted`. The default row of Observability. -**Instrumentation plan** — the placement tickets (file, qualname, decorator, scope) the `get_instrumentation_plan` MCP tool returns for a capability or behaviour; `verify_instrumentation` grades a supplied span list against them. +**Instrumentation plan** — the placement tickets (file, qualname, decorator, scope) the `get_instrumentation_plan` MCP tool returns for a capability or behaviour; `verify_instrumentation` grades an ingested trace, or a supplied span list, against them. ## Data diff --git a/platform/mcp.mdx b/platform/mcp.mdx index 72cd1d1..beac8be 100644 --- a/platform/mcp.mdx +++ b/platform/mcp.mdx @@ -22,18 +22,6 @@ MCP is the platform's first-class agent surface: a project-scoped [Model Context } ``` -```json Claude Code (.mcp.json) -{ - "mcpServers": { - "overmind": { - "type": "http", - "url": "https://api.overmindlab.ai/api/mcp/", - "headers": { "X-Api-Key": "ovr_…" } - } - } -} -``` - ```json OpenCode (opencode.json) { "$schema": "https://opencode.ai/config.json", @@ -48,13 +36,13 @@ MCP is the platform's first-class agent surface: a project-scoped [Model Context } ``` -```bash Claude Code (one line) -claude mcp add --transport http overmind https://api.overmindlab.ai/api/mcp/ --header "X-Api-Key: ovr_…" +```bash Claude Code +claude mcp add --scope local --transport http overmind https://api.overmindlab.ai/api/mcp/ --header "X-Api-Key: ovr_…" ``` -Codex takes the same entry in `.codex/config.toml`. Self-hosted: replace the URL with `http://localhost:8000/api/mcp/` (`overmind init --env local`). +Codex takes the same entry in `.codex/config.toml`. Claude Code keeps the key in local scope (`~/.claude.json`), outside the repository. Self-hosted: replace the URL with your API host plus `/api/mcp/`, for example `http://localhost:8000/api/mcp/` (`overmind init --env local`, or `--api-url`). ## Authentication @@ -138,7 +126,7 @@ A connector is created, with its provider credentials, in the Console under Inte | Tool | Arguments | Returns | | --- | --- | --- | | `get_instrumentation_plan` | `capability?`, `behaviour?` | The registered placement tickets for a capability or behaviour | -| `verify_instrumentation` | `capability?`, `spans` | Bind and grade a caller-supplied span list without writing traces or scores | +| `verify_instrumentation` | `capability?`, `trace_id` or `spans` | Bind and grade an ingested trace by id, or a caller-supplied span list, without writing traces or scores | ### Jobs @@ -188,4 +176,4 @@ Three things never cross MCP: file bytes, repository edits, and provider secrets ## The skill -`overmind init` installs the `overmind` skill next to the MCP entry, and on Cursor and Claude Code the `/overmind` slash commands (`onboard`, `setup`, `ensure-tracing`, `dataset`, `finetune`, `optimise`, `backtest`) that load its references. The skill holds the conventions the tools assume — list first, poll jobs with `get_job`, the dataset contracts that gate every run — and is the only place the local scan workflow (`/overmind setup`) lives; the native prompts cover the server-side workflows. +`overmind init` installs the `overmind` skill next to the MCP entry. `/overmind onboard`, `setup`, `ensure-tracing`, `dataset`, `finetune`, `optimise` and `backtest` load its references: Cursor through command files, Claude Code through the skill itself. The skill holds the conventions the tools assume — list first, poll jobs with `get_job`, the dataset contracts that gate every run — and is the only place the local scan workflow (`/overmind setup`) lives; the native prompts cover the server-side workflows. diff --git a/platform/self-hosting.mdx b/platform/self-hosting.mdx index 2fff428..9399049 100644 --- a/platform/self-hosting.mdx +++ b/platform/self-hosting.mdx @@ -7,35 +7,26 @@ The platform is one repository, [overmind-core/overmind](https://github.com/over ## Requirements -Docker with Compose, [Bun](https://bun.sh/) for the frontend, and[uv](https://docs.astral.sh/uv/) for anything Python outside the containers. -> The API refuses to start without a training backend's credentials (Modal or Baseten), an S3 credential pair for the checkpoint archive, and `INFERENCE_API_URL` — see [Training and serving](#training-and-serving). +Docker with Compose, and [uv](https://docs.astral.sh/uv/) for anything Python outside the containers. +> The API needs one key to start: `OPENROUTER_API_KEY` (or `CURSOR_API_KEY`). Training and serving are optional; until their keys are set, fine-tuning readiness lists the missing variables and job creation is refused — see [Training and serving](#training-and-serving). ## Start the stack ```bash git clone https://github.com/overmind-core/overmind.git cd overmind -cp .env.example .env # fill in OPENROUTER_API_KEY, the training backend, AWS keys, INFERENCE_API_URL -docker compose up -d # API, Postgres, Redis, four Celery workers, beat, Grafana +cp .env.example .env # set OPENROUTER_API_KEY +docker compose up -d # API, Console, Postgres, Redis, four Celery workers, beat, Grafana ``` -The Console is a React SPA served separately using Bun: +The Console is at [http://localhost:5173](http://localhost:5173) and the API at `http://localhost:8000`. Sign in with any email and password; self-hosted mode creates the account on first use and needs no email verification. Leave `CLERK_API_SECRET_KEY` blank. -```bash -cd frontend && \ -cp .env.example .env && \ -bun install && bun run dev # http://localhost:5173 -``` - -You can now access the Console at [http://localhost:5173](http://localhost:5173). -Use any login credentials to sign in, self hosted mode doesn't require any email verification. - -Start by creating a project by following the [Quickstart](/quickstart). +Start by creating a project by following the [Quickstart](/quickstart). The onboarding prompt in your Console already points the coding agent at your API. -For a demo workspace with thirty days of traffic, datasets, runs and served models, run the seed: +For a demo workspace with thirty days of traffic, datasets, runs and served models, run the seed with the email you sign in with: ```bash -docker compose exec -T api python manage.py shell < seed.py +docker compose exec api python manage.py seed_demo --owner you@example.com ``` ## Environment reference @@ -53,12 +44,12 @@ docker compose exec -T api python manage.py shell < seed.py | `FIELD_ENCRYPTION_KEY` | Fernet key for stored connector credentials. Unset, one is derived from `DJANGO_SECRET_KEY`; the API prints a warning at startup when `DJANGO_DEBUG=False` | | `FRONTEND_URL` | Where the Console lives (default `http://localhost:5173`), for links in prompts and emails | | `TRACE_SETTLE_SECONDS` | How long a trace with no root span stays `live` before it reads `interrupted` (default `600`) | -| `OPENROUTER_API_KEY` | Every routed model call: judges, eval-run generation, the optimiser's scoring, frontier inference | +| `OPENROUTER_API_KEY` | Every routed model call: judges, eval-run generation, the optimiser's scoring, frontier inference. Required at startup unless `CURSOR_API_KEY` is set. If the key is rejected, the capability's Evaluators tab shows `Judge authoring failed`; fix the key, then run `overmind sync` again | ### The Data Workshop agent -The dataset agent runs on the first key it finds, in this order: `CURSOR_API_KEY` (a Cursor Composer session), `OPENROUTER_API_KEY`, then `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` with a native tool-calling loop. Both engines take the same tools and emit the same events. With none set the workshop page opens and the agent reports that it is not configured. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. +The dataset agent runs on the first key it finds, in this order: `CURSOR_API_KEY` (a Cursor Composer session), `OPENROUTER_API_KEY`, then `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` with a native tool-calling loop. Both engines take the same tools and emit the same events. The API does not start without `CURSOR_API_KEY` or `OPENROUTER_API_KEY`. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. ### Decision engine @@ -72,12 +63,12 @@ capacity falls back to generative judging; Jev is not a chat or inference model. | Variable | Purpose | | --- | --- | -| `FINETUNING_BACKEND` | `modal` (settings default); `docker-compose.yml` and `.env.example`. Startup fails unless the chosen backend's credentials are set | +| `FINETUNING_BACKEND` | `modal`, the only accepted value | | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `MODAL_ENVIRONMENT` | Modal training and serving workers, deployed with `modal deploy overbae/modal/modal_vllm_worker.py`, `register_model.py` and `modal_sft_worker.py` (`--env $MODAL_ENVIRONMENT`). `MODAL_ENVIRONMENT` (default `overmind-dev`) also selects the checkpoint bucket | | `MODAL_SFT_APP_NAME` | The deployed SFT app (default `overmind-sft`) | -| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | The Modal serving endpoint printed by `modal deploy`, and the shared secret it checks. `INFERENCE_API_URL` is required at startup | +| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | The Modal serving endpoint printed by `modal deploy`, and the shared secret it checks | | `HF_TOKEN` | Gated base models on Hugging Face | -| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` | The checkpoint archive that backs weights download; the key pair is required at startup | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_BUCKET_NAME`, `AWS_REGION` | The checkpoint archive that backs weights download | ### Billing modes diff --git a/quickstart.mdx b/quickstart.mdx index dad0538..f7b7c23 100644 --- a/quickstart.mdx +++ b/quickstart.mdx @@ -9,13 +9,13 @@ Overmind has first class integration with cursor, claude code, opencode, and cod - Start by signing up at [console.overmindlab.ai](https://console.overmindlab.ai) + Start by signing up at [console.overmindlab.ai](https://console.overmindlab.ai). Self-hosted: open your own Console, for example `http://localhost:5173` — see [Self-hosting](/platform/self-hosting). Pick your coding agent — **Cursor**, **Claude Code**, **OpenCode**, or **Codex** — and hit **Copy onboarding prompt**. - Paste the prompt into your coding agent (repo root as the workspace folder) and hit enter. + Paste the prompt into your coding agent (repo root as the workspace folder) and hit enter. After the first sync, reload the agent once so it loads the Overmind MCP server; in Claude Code, exit and run `claude -c`. - Overmind will now scan your repository for [AI Agents & Capabilities](/core/capabilities#the-scan), you can see them in the [Console](https://console.overmindlab.ai) - If you want any changes in how they are grouped, evals, etc, just ask your coding agent to do it. - Next step is to ensure your codebase is instrumented properly for [tracing using OTLP endpoint](/core/observability#the-otlp-endpoint), just run `/overmind ensure-tracing` in your coding agent. diff --git a/tracing/sdk-python.mdx b/tracing/sdk-python.mdx index 82f0ba2..cb7e674 100644 --- a/tracing/sdk-python.mdx +++ b/tracing/sdk-python.mdx @@ -164,7 +164,7 @@ Two identity forms tie code to a capability declared in `overmind.toml`: - `overmind.init(capability_id="")` (or `run(capability_id=...)`) — ingest binds by the capability's UUID; a slug resolves server-side. - `@overmind.capability("", id="...")` on an entry function — at runtime the name is a label only; pass `id=` or set `capability_id` in `init()` to bind. -Verification is a bounded, read-only MCP smoke: run with a unique `conversation_id`, flush, find the trace with `query_traces(session=..., all_spans=true)`, read `overmind://traces/{trace_id}`, and hand its spans to `verify_instrumentation`, which checks capability attribution, nesting, inputs and outputs, usage, and timing without writing traces or scores. +Verification is a bounded, read-only MCP smoke: run with a unique `conversation_id`, flush, find the trace with `query_traces(session=..., all_spans=false)`, and pass its `trace_id` to `verify_instrumentation`, which grades the ingested spans on the server and checks capability attribution, nesting, inputs and outputs, usage, and timing without writing traces or scores. ### `task()` — carve a run into units From 2ed86d3d99fcc096be1f37984528750ab3bb274b Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 09:33:28 +0100 Subject: [PATCH 11/19] Make OpenRouter the one required key --- platform/self-hosting.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/platform/self-hosting.mdx b/platform/self-hosting.mdx index 9399049..0ac313a 100644 --- a/platform/self-hosting.mdx +++ b/platform/self-hosting.mdx @@ -8,7 +8,7 @@ The platform is one repository, [overmind-core/overmind](https://github.com/over ## Requirements Docker with Compose, and [uv](https://docs.astral.sh/uv/) for anything Python outside the containers. -> The API needs one key to start: `OPENROUTER_API_KEY` (or `CURSOR_API_KEY`). Training and serving are optional; until their keys are set, fine-tuning readiness lists the missing variables and job creation is refused — see [Training and serving](#training-and-serving). +> The API needs one key to start: `OPENROUTER_API_KEY`. Training and serving are optional; until their keys are set, fine-tuning readiness lists the missing variables and job creation is refused — see [Training and serving](#training-and-serving). ## Start the stack @@ -44,12 +44,12 @@ docker compose exec api python manage.py seed_demo --owner you@example.com | `FIELD_ENCRYPTION_KEY` | Fernet key for stored connector credentials. Unset, one is derived from `DJANGO_SECRET_KEY`; the API prints a warning at startup when `DJANGO_DEBUG=False` | | `FRONTEND_URL` | Where the Console lives (default `http://localhost:5173`), for links in prompts and emails | | `TRACE_SETTLE_SECONDS` | How long a trace with no root span stays `live` before it reads `interrupted` (default `600`) | -| `OPENROUTER_API_KEY` | Every routed model call: judges, eval-run generation, the optimiser's scoring, frontier inference. Required at startup unless `CURSOR_API_KEY` is set. If the key is rejected, the capability's Evaluators tab shows `Judge authoring failed`; fix the key, then run `overmind sync` again | +| `OPENROUTER_API_KEY` | Every routed model call: judges, eval-run generation, the optimiser's scoring, frontier inference and the Data Workshop agent. Required at startup. If the key is rejected, the capability's Evaluators tab shows `Judge authoring failed`; fix the key, then run `overmind sync` again | ### The Data Workshop agent -The dataset agent runs on the first key it finds, in this order: `CURSOR_API_KEY` (a Cursor Composer session), `OPENROUTER_API_KEY`, then `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` with a native tool-calling loop. Both engines take the same tools and emit the same events. The API does not start without `CURSOR_API_KEY` or `OPENROUTER_API_KEY`. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. +The dataset agent runs on OpenRouter. Set `CURSOR_API_KEY` to run it as a Cursor Composer session instead. Both engines take the same tools and emit the same events. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. ### Decision engine From acbadb1bfb1e2be5a89c5048b689800722107ce4 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 12:30:58 +0100 Subject: [PATCH 12/19] List required and optional self-host keys --- platform/self-hosting.mdx | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/platform/self-hosting.mdx b/platform/self-hosting.mdx index 0ac313a..3f410be 100644 --- a/platform/self-hosting.mdx +++ b/platform/self-hosting.mdx @@ -8,14 +8,32 @@ The platform is one repository, [overmind-core/overmind](https://github.com/over ## Requirements Docker with Compose, and [uv](https://docs.astral.sh/uv/) for anything Python outside the containers. -> The API needs one key to start: `OPENROUTER_API_KEY`. Training and serving are optional; until their keys are set, fine-tuning readiness lists the missing variables and job creation is refused — see [Training and serving](#training-and-serving). + +The API refuses to start until every required key is set; the error names each missing one. + +| Required | Used for | +| --- | --- | +| `OPENROUTER_API_KEY` | Judges, evals, trace scoring, the Data Workshop, optimiser scoring, frontier inference | +| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | Modal training and serving workers | +| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | Serving trained models: the endpoint printed by `modal deploy` and its shared secret | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_BUCKET_NAME` | The fine-tuning checkpoint archive | +| `HF_TOKEN` | Gated Hugging Face base models | + +| Optional | Effect | +| --- | --- | +| `CURSOR_API_KEY` | Recommended for the Data Workshop: runs the dataset agent as a Cursor Composer session | +| `OPENAI_API_KEY` | The `embedding_cosine` evaluator | +| `ANTHROPIC_API_KEY`, `GEMINI_API_KEY` | Data Workshop fallbacks; unused while `OPENROUTER_API_KEY` is set | +| `STRIPE_SECRET_KEY` | Paid plans and the credit cap — see [Billing modes](#billing-modes) | + +Fine-tuning and serving also need the Modal workers deployed — see [Training and serving](#training-and-serving). ## Start the stack ```bash git clone https://github.com/overmind-core/overmind.git cd overmind -cp .env.example .env # set OPENROUTER_API_KEY +cp .env.example .env # set the required keys above docker compose up -d # API, Console, Postgres, Redis, four Celery workers, beat, Grafana ``` @@ -49,7 +67,7 @@ docker compose exec api python manage.py seed_demo --owner you@example.com ### The Data Workshop agent -The dataset agent runs on OpenRouter. Set `CURSOR_API_KEY` to run it as a Cursor Composer session instead. Both engines take the same tools and emit the same events. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. +The dataset agent runs on OpenRouter. We recommend setting `CURSOR_API_KEY`, which runs it as a Cursor Composer session instead. Both engines take the same tools and emit the same events. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. ### Decision engine @@ -64,11 +82,13 @@ capacity falls back to generative judging; Jev is not a chat or inference model. | Variable | Purpose | | --- | --- | | `FINETUNING_BACKEND` | `modal`, the only accepted value | -| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `MODAL_ENVIRONMENT` | Modal training and serving workers, deployed with `modal deploy overbae/modal/modal_vllm_worker.py`, `register_model.py` and `modal_sft_worker.py` (`--env $MODAL_ENVIRONMENT`). `MODAL_ENVIRONMENT` (default `overmind-dev`) also selects the checkpoint bucket | +| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `MODAL_ENVIRONMENT` | Required (tokens). Modal training and serving workers, deployed with `modal deploy overbae/modal/modal_vllm_worker.py`, `register_model.py` and `modal_sft_worker.py` (`--env $MODAL_ENVIRONMENT`). `MODAL_ENVIRONMENT` (default `overmind-dev`) also selects the checkpoint bucket | | `MODAL_SFT_APP_NAME` | The deployed SFT app (default `overmind-sft`) | -| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | The Modal serving endpoint printed by `modal deploy`, and the shared secret it checks | -| `HF_TOKEN` | Gated base models on Hugging Face | -| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_BUCKET_NAME`, `AWS_REGION` | The checkpoint archive that backs weights download | +| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | Required. The Modal serving endpoint printed by `modal deploy`, and the shared secret it checks | +| `HF_TOKEN` | Required. Gated base models on Hugging Face | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_BUCKET_NAME`, `AWS_REGION` | Required (all but the region). The checkpoint archive that backs weights download | + +The Modal workers read their credentials from a Modal secret named `overmind-inference`: the AWS keys, `INFERENCE_API_KEY` and `HF_TOKEN`. ### Billing modes From 1769d28edf2d167061953549d6ff7db37d94778a Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 12:46:39 +0100 Subject: [PATCH 13/19] Remove statements that are no longer true --- agent-testing/decision-engine.mdx | 4 ++-- agent-testing/eval.mdx | 4 ++-- agent-testing/optimisers.mdx | 2 +- core/capabilities.mdx | 8 ++++---- core/datasets.mdx | 2 +- index.mdx | 2 +- models/inference.mdx | 6 +++--- models/training.mdx | 6 +++--- platform/administration.mdx | 4 ++-- platform/api.mdx | 6 +++--- platform/cli.mdx | 6 +----- platform/glossary.mdx | 12 ++++++------ platform/mcp.mdx | 9 ++++++--- quickstart.mdx | 4 ++-- tracing/sdk-typescript.mdx | 2 +- 15 files changed, 38 insertions(+), 39 deletions(-) diff --git a/agent-testing/decision-engine.mdx b/agent-testing/decision-engine.mdx index 0cb4cc2..b6aa0c9 100644 --- a/agent-testing/decision-engine.mdx +++ b/agent-testing/decision-engine.mdx @@ -28,7 +28,7 @@ Configurable rubric judges default to **generative**. Jev is an explicit opt-in after validation on representative labelled examples for that rubric. A high confidence value is not proof of correctness or agreement with the existing judge. Fixed grounding, capability classification and workshop semantic services use -Jev with generative fallback. Existing explicit policies and snapshots are preserved. +Jev with generative fallback. The Console's evaluator editor exposes **Decision engine** and **Minimum confidence**. The REST authoring request accepts `decision_policy`. General evaluator writes and @@ -48,7 +48,7 @@ MCP `upsert_evaluator` accept the same policy under `config.decision`: Set `backend` to `generative` to use the generative judge throughout that evaluator. The policy is frozen in new evaluator snapshots and included in verdict identities. The requested Jev release is pinned; the provider's actual served revision is -recorded with each response. Historical results are not relabelled or rescored. +recorded with each response. Changing a policy does not rescore earlier results. Responses must cover every question, use only declared options, and return valid probability distributions. Low-confidence answers, invalid responses, unavailable diff --git a/agent-testing/eval.mdx b/agent-testing/eval.mdx index 82b9592..6e7eaac 100644 --- a/agent-testing/eval.mdx +++ b/agent-testing/eval.mdx @@ -63,7 +63,7 @@ Judges are authored behaviour-scoped from the behaviour's group in the Eval libr Author from the Eval library by hand, or describe what you want to test and let Overmind generate the evaluator (`POST /api/evaluators/generate-prompt/`). The generated prompt template references the agent's output as `{{output.}}` and dataset fields as `{{reference.}}`, and everything stays editable before saving. -`overmind sync` preloads the capability's Default eval set for every capability it created, remounted, or whose card changed, in tiers: Tier 0, rule-based checks compiled from the capability's card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. When Tier 1 authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability's **evidence profile** — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply. +`overmind sync` refreshes the Default eval set of every current capability, in tiers: Tier 0, rule-based checks compiled from the capability's card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. Tier 1 authors once per capability; when it authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability's **evidence profile** — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply. Eval library grouped by behaviour @@ -204,7 +204,7 @@ Each dataset row has its own entry in the breakdown, with responses from differe The datapoint popup shows **Input** beside **Output and evaluation**. Select a model to switch both panels together. Generated output is the model's response, not a label extracted by a statistical evaluator; the **Reference answer** is shown separately as grading data. Expand **Generated messages** to inspect recorded assistant and tool messages. -New generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider's rendered chat template or every subsequent replay request. Older samples show **Dataset input** from the run's pinned version and state that the exact model request was not captured. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript. Existing runs do not need to be rerun to inspect their pinned dataset input. +Generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider's rendered chat template or every subsequent replay request. A sample without a captured request shows **Dataset input** from the run's pinned version. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript. The MCP `overmind://eval-runs/{eval_run}` resource exposes up to five bounded sample inspections with separate input, output, reference, and evaluator reasoning. `input_source` distinguishes captured requests, pinned dataset rows, and unavailable input; `samples_truncated` indicates additional samples. diff --git a/agent-testing/optimisers.mdx b/agent-testing/optimisers.mdx index e3ba393..04afd7b 100644 --- a/agent-testing/optimisers.mdx +++ b/agent-testing/optimisers.mdx @@ -20,7 +20,7 @@ The Optimiser takes an agent you can already measure and edits the agent itself. Candidates are graded by the eval set you pick, or the capability's active eval set by default. The set must belong to the capability. See [Eval](/agent-testing/eval#eval-sets). - Your agent executes in **your** environment, not Overmind's. The loop is driven from your checkout by the `overmind` CLI, and the candidate diffs are written by your coding agent. **Optimiser → New run** in the Console offers **Improve prompts & code** to test changes to prompts, tool definitions and application logic, or **Compare models** to evaluate different models while keeping prompts and logic fixed. Select the capability and eval dataset, then use **Copy prompt** and paste into your coding agent in that capability's repository. Copying does not start a run. The `/overmind optimise` and `/overmind backtest` commands are unchanged; scores and progress appear in Overmind. + Your agent executes in **your** environment, not Overmind's. The loop is driven from your checkout by the `overmind` CLI, and the candidate diffs are written by your coding agent. **Optimiser → New run** in the Console offers **Improve prompts & code** to test changes to prompts, tool definitions and application logic, or **Compare models** to evaluate different models while keeping prompts and logic fixed. Select the capability and eval dataset, then use **Copy prompt** and paste into your coding agent in that capability's repository. Copying does not start a run. The `/overmind optimise` and `/overmind backtest` commands drive the same runs; scores and progress appear in Overmind. diff --git a/core/capabilities.mdx b/core/capabilities.mdx index 831f2f6..81d7bce 100644 --- a/core/capabilities.mdx +++ b/core/capabilities.mdx @@ -72,7 +72,7 @@ After code changes: run `/overmind setup` again; it includes the final `overmind - **Identity is carried**, in order: the toml `id` → the toml `slug` → a new record. A capability the push no longer includes becomes a leftover (`archived = true` in the toml on the next `sync down`). - **Behaviours are minted from the trajectory map**: one contract per mapped route, with an entry anchor, the ordered anchors that follow it, and its tool set. Behaviours are what [trace scoring](/agent-testing/trace-scoring#behaviours-and-bindings) binds production traffic to; the Console labels the column **Task**. -- **The Default eval set is preloaded** for every created, remounted or card-changed capability: Tier-0 rule-based checks compiled from the card, Tier-1 generative LLM judges grounded in it, and per-behaviour outcome and step judges for trace scoring. See [Eval](/agent-testing/eval#authoring). +- **The Default eval set is refreshed** for every current capability on each sync: Tier-0 rule-based checks compiled from the card (deduplicated), Tier-1 generative LLM judges grounded in it (authored once per capability), and per-behaviour outcome and step judges for trace scoring. See [Eval](/agent-testing/eval#authoring). ### A scan never deletes @@ -84,7 +84,7 @@ To remove a capability yourself, use **Delete capability** at the bottom of its Spans carrying `overmind.capability.id` — set with [`init(capability_id=...)`](/tracing/sdk-python#init) or a `capability(..., id=)` scope — bind to that capability. The value is the capability's UUID or its toml `slug`; both resolve, and both are stable through renames. `overmind.capability.name` is a display label and never binds. -Ingest never creates a capability. An identity the project does not know leaves the span **unbound**: it is stored, browsable under Observability's Unbound filter, and binds retroactively when a later push makes the identity resolvable. `confirmed_at` is stamped on the capability the first time a span binds to one of its behaviours; the capability page shows **Confirmed** or **Unconfirmed** accordingly. A capability created by hand (`POST /api/capabilities/`) carries an **Observed** badge instead, because no scan can confirm or retire it. +Ingest never creates a capability. An identity the project does not know leaves the span **unbound**: it is stored, browsable under Observability's Unbound filter, and binds retroactively when a later push makes the identity resolvable. A capability created by hand (`POST /api/capabilities/`) carries an **Observed** badge, because no scan can retire it. ## The agent page @@ -94,7 +94,7 @@ Ingest never creates a capability. An identity the project does not know leaves Agent page with the capability grid -The header carries the name (editable in place), copyable **capability id** and **project id** chips, the slug, the badges — **Declared** or **Inferred** (whether the manifest named it), **Confirmed** or **Unconfirmed**, **Not in latest scan** — and a **Last push** line linking to the project's scan report. **View traces** opens the capability's filtered Observability view. Below it, one scrolling page: +The header carries the name (editable in place), copyable **capability id** and **project id** chips, the slug, and the **Not in latest scan** and **Observed** badges when they apply. **View traces** opens the capability's filtered Observability view. Below it, one scrolling page: - **Facts** — the model the capability serves through (or the live deployed model behind its alias) and its source path. - **Prompt** — the system prompt lifted from your code, one per mode when the capability has several. @@ -112,7 +112,7 @@ The header carries the name (editable in place), copyable **capability id** and Trajectory map fullscreen view -The Agent page is the record of the latest sync: each capability with its card, behaviours, and whether telemetry has confirmed it. +The Agent page is the record of the latest sync: each capability with its card and behaviours. ## Instrumentation diff --git a/core/datasets.mdx b/core/datasets.mdx index cf867ad..ca45b52 100644 --- a/core/datasets.mdx +++ b/core/datasets.mdx @@ -62,7 +62,7 @@ Quality and capability-alignment findings remain advisory, including train/eval Ask explicitly for synthetic examples. Validated batches accumulate in one active generated version, with no draft or Apply step. Retries do not add duplicate rows; interrupted generation keeps completed batches and can continue to the same target. Used versions cannot receive more batches. Seed lineage, capability context, instructions and generation engine are recorded. Synthetic answers still need independent review and cannot be split away from their seeds into held-out evaluation. Automatic preparation never generates synthetic rows. -Chat activity is chronological, with expandable thinking and approval checkpoints. Clearing chat does not remove applied versions. The agent uses the first configured engine: Cursor, OpenRouter, then OpenAI, Anthropic or Gemini. With no key, the workshop opens and reports that the agent is unavailable; see [Self-hosting](/platform/self-hosting#the-data-workshop-agent). +Chat activity is chronological, with expandable thinking and approval checkpoints. Clearing chat does not remove applied versions. The agent runs on OpenRouter, or as a Cursor Composer session when `CURSOR_API_KEY` is set (recommended); see [Self-hosting](/platform/self-hosting#the-data-workshop-agent). ## Versions diff --git a/index.mdx b/index.mdx index c5e9987..11705be 100644 --- a/index.mdx +++ b/index.mdx @@ -42,7 +42,7 @@ Overmind is built on top of standard protocols so no matter what agent you use, What Overmind does: - [Capability scan](/core/capabilities): Scanning your codebase for AI agents and different agentic capabilities. - [Live scoring](/agent-testing/trace-scoring): every trace is carved into task executions and judged against the agent's eval set -- [Datasets from traces](/core/datasets): traces from real environments become eval and training data, no synthetic data, no data poisoning, no hallucinations. +- [Datasets from traces](/core/datasets): traces from real environments become eval and training data. - [Evaluations](/agent-testing/eval): your in house quality control loop, measured using a typed score, ensuring the agent is always improving. - [Optimiser](/agent-testing/optimisers): An AI Engineer running experiments on your repository, trying to improve the agent's performance while adhering to your quality control loop. - [Models](/models/training): Finetuned models that you own, built on top of your custom data. diff --git a/models/inference.mdx b/models/inference.mdx index 7e0b855..8358115 100644 --- a/models/inference.mdx +++ b/models/inference.mdx @@ -11,8 +11,8 @@ The Console model list shows deployments from training runs. Base-model deployme A deployment moves through `queued → deploying → warming → ready`, or ends `failed`, `deleting`, `deleted`. Two shapes: -- A dense model trained on Modal with LoRA is served as an **adapter on a shared BF16 base**; deployments of the same base share one process and skip the `quantizing` stage. -- Everything else — full fine-tunes, mixture-of-experts models, models whose family has no adapter support — is **merged and quantised into a private checkpoint** (FP8 where the base supports it) and served from its own vLLM process. +- A dense or explicitly supported mixture-of-experts model trained on Modal with LoRA is served as an **adapter on a shared BF16 base**; deployments of the same base share one process and skip the `quantizing` stage. +- Everything else — full fine-tunes, other mixture-of-experts models, models whose family has no adapter support — is **merged and quantised into a private checkpoint** (FP8 where the base supports it) and served from its own vLLM process. Each process scales to zero when idle: two minutes without traffic for most models, fifteen for the largest. The first request after that cold-boots the model in two to seven minutes; the response is held open with an idle ping every 15 seconds so the connection survives. @@ -112,7 +112,7 @@ Through MCP, call `set_active_model` and poll its `model_activation` receipt wit The list shows **Model**, **Base model**, **Capability**, **Training job**, **Reference**, **Status**, **Requests**, **Tokens**, **Throughput**, **Latency** and **Last active**; latency and throughput are medians over warm calls. Status shows the deployment status until it is `ready`; a ready model shows its serving state from `live/` — **Warm** when its worker pool has running containers or it served successful traffic in the last 90 seconds, **Warming** while a cold boot is in progress, **Asleep** when its measured pool has no containers, and **Unknown** only when worker measurements are unavailable and there is no recent serving signal. The list and detail page refresh this state every five seconds; checking status does not wake the model. The separate **Live model** badge identifies the capability’s selected route, independently of worker warmth. -The model page focuses on **Inference performance**: request volume, success rate, end-to-end p50 and p95 response time, median warm generation speed, and estimated serving cost. The reporting period applies to both headline metrics and charts: **Last hour**, **Last 24 hours**, **Last 7 days**, **Last 30 days** or **All time**. **Application traffic** includes API-key requests to either the capability alias or a pinned model ID; **All traffic** also includes internal calls and historical requests whose source is unknown. Metrics refresh every 15 seconds while the page is open. +The model page focuses on **Inference performance**: request volume, success rate, end-to-end p50 and p95 response time, median warm generation speed, and estimated serving cost. The reporting period applies to both headline metrics and charts: **Last hour**, **Last 24 hours**, **Last 7 days**, **Last 30 days** or **All time**. **Application traffic** includes API-key requests to either the capability alias or a pinned model ID; **All traffic** also includes internal calls and requests with an unknown source. Metrics refresh every 15 seconds while the page is open. The charts show request traffic and failures, response-time percentiles, and generation speed. Empty traffic buckets are zero; missing timing measurements remain gaps. Failed calls do not contribute to latency or speed distributions. A single empty state replaces the charts when the selected period has no requests. Refresh errors are visible and do not turn into zero traffic. The latest failure includes its request ID. diff --git a/models/training.mdx b/models/training.mdx index cda18d6..4e10163 100644 --- a/models/training.mdx +++ b/models/training.mdx @@ -33,7 +33,7 @@ Workshop quality reviews remain advisory and are not repeated as launch approval For Modal, CPU preprocessing runs after launch and before GPU training using the matching tokenizer, chat template and loss mask. Setup does not start or wait for preprocessing. Recommendations size context from estimated row lengths; job preparation includes pinned training and enabled validation data. Exact overflow within the model's training-method limit requests a larger context and rechecks every row. Over-limit rows, invalid tool formatting and examples without supervised next-token targets still block. Nothing is silently truncated or discarded. -The GPU consumes the matching validated token artifact and saved tokenizer. Repair incompatible data in the workshop and prepare the revised version. Explicit REST `POST /api/training-preparations/` and MCP `prepare_training_data` remain available; poll with `get_job(kind="training_preparation")`. Other providers retain provider-side preprocessing. +The GPU consumes the matching validated token artifact and saved tokenizer. Repair incompatible data in the workshop and prepare the revised version. Explicit REST `POST /api/training-preparations/` and MCP `prepare_training_data` run the same preparation; poll with `get_job(kind="training_preparation")`. Worker failures are distinct from incompatible data. A processor/code mismatch requires deploying the current Modal SFT worker, not just restarting Docker. Preprocessing logs remain at `preparations//preprocess_stdout.log` on the SFT volume. Confirmed failures can retry; uncertain submissions fail closed. @@ -47,7 +47,7 @@ Serving context is planned separately from training sequence length, using the p The wizard shows the constraints for each model: -- **Context.** Training limits can be shorter than inference windows and differ between LoRA and full training. Modal uses estimates to size context, not to exclude candidates; exact preparation determines compatibility. Other providers retain their existing eligibility checks. +- **Context.** Training limits can be shorter than inference windows and differ between LoRA and full training. Estimates size context but do not exclude candidates; exact preparation determines compatibility. - **Tool calling.** If your data contains tool calls, only models that support them are offered. - **Adapters.** Larger dense models and every mixture-of-experts model train as LoRA adapters; the picker says so instead of offering a choice. @@ -84,7 +84,7 @@ Every selected evaluation uses the entire pinned eval version, the same prompt s **Judge model**, below **Eval set**, defaults to the set's per-evaluator choices. An explicit selection is saved for this run and used on both sides of the comparison without changing the set. REST/MCP use `eval_judge_model`; it cannot change after launch. The dropdown labels estimated context fit and judge-budget differences. Estimated limit overflows turn affected selectors and **Start** orange; undersized options are greyed out but remain selectable. Clicking **Start** opens a short risk summary with **Start anyway**, without requiring a model change. Unverified, unavailable and pending checks stay neutral and do not open this warning. Shared incumbent evaluations require the same judge selection as well as matching data and benchmark context. -Supported untouched bases use OpenRouter when configured and exactly matched in its catalog; otherwise they use the existing provider/Modal route. Once an eval or baseline deployment starts, its route stays pinned until an explicit retry. Trained checkpoints use their own deployment. +Supported untouched bases use OpenRouter when exactly matched in its catalog; otherwise they use a Modal deployment. Once an eval or baseline deployment starts, its route stays pinned until an explicit retry. Trained checkpoints use their own deployment. The Evals table shows selected evaluations immediately, including what each is waiting for, and keeps updating after deployment. Evaluations that never start are skipped if training fails or is cancelled. When experiments share an incumbent-before evaluation, the group view shows that run once as **Shared · N experiments**. Each experiment still has its own base-model and trained-model evaluations, and its individual view retains the shared incumbent result and comparison. diff --git a/platform/administration.mdx b/platform/administration.mdx index ebb9f75..47a3590 100644 --- a/platform/administration.mdx +++ b/platform/administration.mdx @@ -17,7 +17,7 @@ One key type serves everything programmatic: the tracing SDK, raw OTLP ingest, t - **Format**: `ovr_` followed by a random token. Only a SHA-256 hash is stored — the full key is **shown once, at creation**; the table keeps the first 12 characters as `prefix`. - **Headers**: `X-Api-Key: `, `Authorization: Bearer `, or `Authorization: Api-Key `. -- **Scope**: **project-scoped** keys see one project; **account-scoped** keys reach every project you are a member of and are the only keys that can create a project. MCP and `.overmind/credentials.toml` hold a project-scoped key; the onboarding prompt carries a temporary account-scoped key that `overmind sync` trades for a project one. +- **Scope**: **project-scoped** keys see one project; **account-scoped** keys reach every project you are a member of and are the only keys that can create a project. `overmind sync` stores a project-scoped key in `.overmind/credentials.toml` and the MCP entry; the onboarding prompt carries a temporary account-scoped key that `overmind sync` trades for a project one. - **Lifetime**: keys do not expire unless an `expires_at` is set on the row. Revocation is immediate, and each key records `last_used_at`. - **Where**: **Projects → your project → API keys** in the Console, or [over the API](/platform/api#authentication). @@ -25,7 +25,7 @@ One key type serves everything programmatic: the tracing SDK, raw OTLP ingest, t Project page with the API keys table -Scope one key per environment and keep them out of source control. `overmind sync` writes the project key to `.overmind/credentials.toml`, and `overmind init` writes it into the IDE's MCP entry; both refuse a path git already tracks, add the file to `.git/info/exclude` (not `.gitignore`), and set mode `0600`. +Scope one key per environment and keep them out of source control. `overmind sync` writes the project key to `.overmind/credentials.toml`, and `overmind init` writes it into the IDE's MCP entry; both refuse a path git already tracks, add the file to `.git/info/exclude` (not `.gitignore`), and set mode `0600`. Claude Code's MCP entry lives in `~/.claude.json`, outside the repository. ## Tracing connectors diff --git a/platform/api.mdx b/platform/api.mdx index 7255f73..dbd03f7 100644 --- a/platform/api.mdx +++ b/platform/api.mdx @@ -19,10 +19,10 @@ Three ways to authenticate; the first is right for almost all programmatic use: | Method | How | Use for | | --- | --- | --- | | **API key** | `X-Api-Key: ovr_...`, `Authorization: Bearer ovr_...`, or `Authorization: Api-Key ovr_...` | SDKs, OTLP ingest, scripts, CI, the CLI, MCP, inference, `overmind sync` | -| **JWT** | `POST /api/auth/token/` with `email` and `password` → access token (1 day) + refresh token (7 days, `POST /api/auth/token/refresh/`) | Accounts created with `POST /api/auth/register/`; guest sessions | +| **JWT** | `POST /api/auth/token/` with `email` and `password` → access token (1 day) + refresh token (7 days, `POST /api/auth/token/refresh/`) | Self-hosted accounts (`POST /api/auth/local/` signs in and creates the account on first use); guest sessions | | **Clerk session** | `Authorization: Bearer ` | The Console | -A key has a **scope**: `project` pins it to one project, `account` reaches every project the user is a member of. MCP and `.overmind/credentials.toml` use a project-scoped key; of the two, only an account-scoped key can `POST /api/projects/`. Keys are created and revoked under **Projects → your project → API keys** or via the API: +A key has a **scope**: `project` pins it to one project, `account` reaches every project the user is a member of. `.overmind/credentials.toml` holds a project-scoped key; only an account-scoped key can `POST /api/projects/`. Keys are created and revoked under **Projects → your project → API keys** or via the API: ```bash # Create (the key is returned once), inspect the calling key, list, revoke @@ -86,7 +86,7 @@ Standard REST resources follow Django REST Framework conventions: `GET` list wit | `GET /api/models/catalog/` | The OpenRouter model catalog used for judges and run variants | | `GET /api/public/models/`, `GET /api/public/models/{slug}/` | The public model library, unauthenticated | | `GET /health` | Health check (no auth) | -| `/api/auth/register/`, `/api/auth/me/`, `/api/auth/onboarding/`, `POST /api/auth/guest/`, `POST /api/auth/guest/claim/` | Account, onboarding state, guest workspaces | +| `POST /api/auth/local/`, `/api/auth/me/`, `/api/auth/onboarding/`, `POST /api/auth/guest/`, `POST /api/auth/guest/claim/` | Self-hosted sign-in, account, onboarding state, guest workspaces | | `/api/billing/subscription/`, `ledger/`, `checkout/`, `topup/`, `cancel/`, `renew/`, `webhook/` | Plans, the credits ledger, Stripe | ## Building a dataset diff --git a/platform/cli.mdx b/platform/cli.mdx index 234c42c..cb2cb3a 100644 --- a/platform/cli.mdx +++ b/platform/cli.mdx @@ -25,7 +25,7 @@ overmind init --ide cursor # or claude | opencode | codex overmind sync # create the project, save the project key ``` -Then in your coding agent, `/overmind setup` scans the repository, writes capabilities into `overmind.toml`, and runs `overmind sync` to push them. The installed skill recognises the repository and verified connection, previews the onboarding roadmap, and reports repository-specific findings alongside the current stage out of ten, completed stages, and stages still to follow. A pre-sync handoff names the destination, included data, credential checks performed, and configuration to review. The finish reports the synced result, outstanding verification, and one recommended next action without automatically starting it. There is no `overmind scan` command. +Then in your coding agent, `/overmind setup` scans the repository, writes capabilities into `overmind.toml`, and runs `overmind sync` to push them. The installed skill recognises the repository and verified connection, previews the onboarding roadmap, and reports repository-specific findings alongside the current stage out of ten, completed stages, and stages still to follow. A pre-sync handoff names the destination, included data, credential checks performed, and configuration to review. The finish reports the synced result, outstanding verification, and one recommended next action without automatically starting it. When Overmind is a project dependency, run commands through that project's environment (`uv run overmind`, `poetry run overmind`, or its virtual-environment executable). An editable installation does not update skill copies already in a repository. Refresh the installed skill with `uv run overmind skills sync overmind --ide ` using the appropriate project runner, then start a new conversation so the coding agent reads it. @@ -210,7 +210,3 @@ overmind skills sync overmind --ide cursor ## JSON output `--json` on `dataset upload`, `dataset export`, `connector add` and `model download-checkpoint` prints one machine-readable object and nothing else; on failure they print `{"error": "..."}` and exit 1. `optimise status --json` prints the experiment. This is how the `/overmind` commands hand a local step to the CLI and read the result back. - -## Not a command - -There is no `overmind scan`. Discovery is `/overmind setup` in your coding agent: it runs `overmind chassis`, authors capability cards into `overmind.toml`, then `overmind sync` uploads them. See [Agent & Capabilities](/core/capabilities#the-scan). diff --git a/platform/glossary.mdx b/platform/glossary.mdx index 8147be0..54545d7 100644 --- a/platform/glossary.mdx +++ b/platform/glossary.mdx @@ -17,9 +17,9 @@ description: Definitions of terms used across Overmind, as they appear in the Co **Scan snapshot** — the server's record of one `overmind sync`: the toml capabilities, cards, and repo summary. `GET /api/v1/sync?project_id=` reads it back. -**Cluster** — one group of functions reachable from an entry point with the same prompt set; the unit the coding agent names as a capability. +**Tool cluster** — a group of a capability's tools that serve similar tasks, named on the card; the trajectory map renders each cluster together. -**Mode** — one distinct model call inside a capability, with its own system prompt: the roles inside an orchestration. Each mode becomes one behaviour on sync. +**Mode** — one task or role inside a capability, with its own prompt: a stage of a pipeline or a worker in an orchestration. A trajectory-map entry for a mode shares its name, so the behaviour minted from it carries that identity. **Card** — the agent-authored description of a capability in `overmind.toml` (`[capabilities..capability_card]`): task, modality, input schema, output fields, expected output, success criteria, failure modes, constraints, tool protocol, and a trajectory map of anchors — each with source provenance. Filled by `/overmind setup` from source, merged onto the capability and its behaviours on `overmind sync`. @@ -27,15 +27,15 @@ description: Definitions of terms used across Overmind, as they appear in the Co **API key** — an `ovr_...` credential (stored hashed, shown once at creation) that authenticates the SDK, OTLP ingest, the REST API, MCP, the CLI, and inference. Sent as `X-Api-Key` or `Authorization: Bearer`. **Project-scoped** keys see one project; **account-scoped** keys reach every project the user is in. -**MCP** — the project-scoped Model Context Protocol server at `/api/mcp/`: the agent surface a coding agent uses for every workflow that does not need the filesystem. See [MCP](/platform/mcp). +**MCP** — the Model Context Protocol server at `/api/mcp/`: the agent surface a coding agent uses for every workflow that does not need the filesystem. See [MCP](/platform/mcp). -**Skill** — the `overmind` agent skill `overmind init` installs, with the `/overmind` commands and one reference per workflow. See [CLI](/platform/cli). +**Skill** — the agent skills `overmind init` installs: the main `overmind` skill, with the `/overmind` commands and one reference per workflow, and focused workflow skills. See [CLI](/platform/cli). ## Observability **Trace** — one end-to-end run of an agent: the set of spans sharing a `trace_id`. There is no separate trace record; the root span (the one with no parent) represents the trace in lists. -**Span** — one operation inside a trace, stored with its full OTLP payload (timing, status, attributes, events) plus platform fields computed at ingest. Spans are typed by `overmind.span_type`: `entry_point`, `workflow`, `tool_call`, `function`, `retrieval`, or `llm_call`; an untyped span is classified from its attributes and name and defaults to `llm_call`. +**Span** — one operation inside a trace, stored with its full OTLP payload (timing, status, attributes, events) plus platform fields computed at ingest. Spans are typed by `overmind.span.type`: `entry_point`, `workflow`, `tool_call`, `function`, `retrieval`, or `llm_call`; an untyped span is classified from its attributes and name and defaults to `llm_call`. **Session** — traces grouped by a shared `conversation.id` attribute, representing one multi-turn exchange. Set via `set_conversation_id()` or `run(conversation_id=...)` in the SDK. Carries a **session score** folded from the conversation ledger. See [Trace scoring](/agent-testing/trace-scoring#session-scores). @@ -111,7 +111,7 @@ description: Definitions of terms used across Overmind, as they appear in the Co **Training run** — one or more parallel experiments over a training dataset, each producing a candidate model (API resource: `finetuning-jobs`). See [Models](/models/training). -**Baseline (training)** — the capability's production model, scored on the eval dataset at submit time; the reference every trained model's delta is measured against. +**Benchmark model (training)** — the model a training run compares against: the codebase incumbent or a ready trained model, chosen per run. The headline delta uses the untouched base before training when that evaluation is selected, otherwise the benchmark. **Deployed model** — a served trained model with a copyable reference (`ft-...`) callable via the OpenAI-compatible API. Status: `queued`, `deploying`, `warming`, `ready`, `failed`, `deleting`, `deleted`. See [Inference](/models/inference). diff --git a/platform/mcp.mdx b/platform/mcp.mdx index beac8be..10e0db5 100644 --- a/platform/mcp.mdx +++ b/platform/mcp.mdx @@ -96,8 +96,10 @@ Thirty-two tools, all project-scoped, each with a strict input and output schema | `check_finetune_readiness` | `dataset`, `capability?`, `cell?`, `version?` | Prerequisites, train/eval dataset shape, recommended models, evaluators, credits | | `estimate_finetune` | `dataset`, `base_model`, `n_epochs?`, `use_lora?`, `cell?`, `version?` | Real-token cost and duration estimate; creates nothing | | `start_finetune` | `dataset`, `base_model`, `capability?`, `eval_dataset?`, `eval_set?`, `validation_dataset?`, `cell?`, `version?`, `eval_cell?`, `eval_version?`, `validation_cell?`, `validation_version?`, `validation_enabled?`, `validation_split_ratio?`, `split_method?`, `name?`, `use_case?`, `hyperparameters?`, `group_id?` | Validate and queue a training job. Returns a job receipt | +| `prepare_training_data` | `dataset`, `base_model`, `context_length`, `training_type?`, `validation_dataset?`, `cell?`, `retry_failed?` | Run exact tokenization for a model and context length; returns a cached preparation report. Poll with `get_job(kind="training_preparation")` | | `retry_deployment` | `deployment` | Retry a failed or deleted deployment. Returns a job receipt | | `set_active_model` | `capability`, `deployment?` | Verify a ready deployment, then switch the alias; poll the returned `model_activation` job. Omit deployment to clear routing | +| `set_benchmark_model` | `capability`, `deployment?` | Choose a ready trained deployment as the capability's default training benchmark; omit deployment to use the codebase incumbent. Does not change serving | | `run_inference` | `deployment`, `messages`, `temperature?`, `max_tokens?` | One chat completion against a ready deployment using production output budgets, with usage, latency and finish reason | | `get_model_swap_prompt` | `finetune`, `pin?` | A copy-paste prompt that points the capability's code at a successful fine-tune; the agent applies it locally | @@ -119,7 +121,7 @@ Thirty-two tools, all project-scoped, each with a strict input and output schema | `configure_connector` | `connector`, `source_project_id?`, `lookback_days?`, `backfill_from?`, `backfill_to?`, `capability_mapping?`, `auto_sync_enabled?`, `poll_interval_seconds?` | Save source project, lookback, capability mapping and polling settings | | `sync_connector` | `connector` | Queue the configured import. Returns a task receipt | -A connector is created, with its provider credentials, in the Console under Integrations; until that form is complete the tools answer with the `connector_setup_required` action. +A connector is created with its provider credentials through `overmind connector add` or the Console under Integrations; until then the tools answer with the `connector_setup_required` action. ### Instrumentation @@ -130,13 +132,13 @@ A connector is created, with its provider credentials, in the Console under Inte ### Jobs -Background work returns a **job receipt** — `kind`, `id`, `status` and an `overmind://jobs/{kind}/{id}` resource — that `get_job` reads back. Kinds: `dataset_run`, `connector_sync`, `eval_run`, `finetune` (alias `finetune_job`), `optimizer` (aliases `optimizer_run`, `optimizer_experiment`) and `deployment`. +Background work returns a **job receipt** — `kind`, `id`, `status` and an `overmind://jobs/{kind}/{id}` resource — that `get_job` reads back. Kinds: `dataset_run`, `connector_sync`, `eval_run`, `training_preparation`, `finetune` (alias `finetune_job`), `optimizer` (aliases `optimizer_run`, `optimizer_experiment`), `deployment` and `model_activation`. Names resolve within the project: capability arguments take an id, key or name; connectors take an id or name; dataset tools prefer the UUID `list_datasets` returned because dataset names are not unique. ## Prompts -Eleven prompts compose the tools into multi-step workflows with human checkpoints: +Twelve prompts compose the tools into multi-step workflows with human checkpoints: | Prompt | Arguments | What it drives | | --- | --- | --- | @@ -151,6 +153,7 @@ Eleven prompts compose the tools into multi-step workflows with human checkpoint | `upload-dataset-file` | `path`, `intent?`, `project_id?` | `overmind dataset upload` in the terminal, then verify through MCP | | `export-dataset` | `dataset` | `overmind dataset export` to the local filesystem | | `download-checkpoint` | `deployment` | `overmind model download-checkpoint` to the local filesystem | +| `connect-traces` | `connector_type` | `overmind connector add` in the terminal, then configure, confirm the mapping and sync through MCP | ## Resources diff --git a/quickstart.mdx b/quickstart.mdx index f7b7c23..f6702b9 100644 --- a/quickstart.mdx +++ b/quickstart.mdx @@ -5,7 +5,7 @@ description: The easiest way to get started with Overmind. Overmind has first class integration with cursor, claude code, opencode, and codex through our SKILL `/overmind` and MCP server, after signing up you can pretty much do everything from your coding agent. -> We've built the experince to be as easy and native to your coding agent as possible. +> We've built the experience to be as easy and native to your coding agent as possible. @@ -22,7 +22,7 @@ Overmind has first class integration with cursor, claude code, opencode, and cod - The pushed capabilities are the starting point. From the same chat, each command drives one loop end to end: + The pushed capabilities are the starting point. From the same chat, each command drives one loop end to end. Cursor and Claude Code take them as slash commands; in Codex and OpenCode, ask the agent for the step by name: | Command | What it does | | --- | --- | diff --git a/tracing/sdk-typescript.mdx b/tracing/sdk-typescript.mdx index 409cc31..b155934 100644 --- a/tracing/sdk-typescript.mdx +++ b/tracing/sdk-typescript.mdx @@ -4,7 +4,7 @@ description: There is no TypeScript tracing SDK; instrument Node.js with OpenTel --- -No TypeScript SDK ships with the current platform. The `@overmind-lab/trace-sdk` package on npm predates the current ingest contract — it does not stamp the attributes below. Do not install it. +There is no TypeScript SDK. Instrument Node.js with OpenTelemetry directly. Node.js services trace like any other OpenTelemetry client: From 759e1dfdaf4f6ecf1e901c1e83b5406ddb740cd6 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 12:46:47 +0100 Subject: [PATCH 14/19] Match the sync path in the scan diagram --- core/capabilities.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/capabilities.mdx b/core/capabilities.mdx index 81d7bce..1c3d012 100644 --- a/core/capabilities.mdx +++ b/core/capabilities.mdx @@ -13,7 +13,7 @@ Your **agent** in Overmind is the product itself: one per project, mapped as a g lines={[ "╔══════════════════════╗ ╔═ Overmind ══════╗", "║ Local repo ║ ║ ║", - "║chassis → cards → sync╟────POST /api/v1/sync/─────┐ ║ ║", + "║chassis → cards → sync╟────POST /api/v1/sync──────┐ ║ ║", "╚══════════════════════╝ │ ║ ╔════════════╗ ║", " ├───╫─▶║ Capability ║ ║", "╔══════════════════════╗ │ ║ ╚════════════╝ ║", From 6757d90beb1a5e0e78e53e611ecbee8af557a986 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 13:09:39 +0100 Subject: [PATCH 15/19] Fold #37 into Datasets: row shapes, refusals and limits --- core/datasets.mdx | 52 +++++++++++++++++++++++++++++++++++++++++++++-- platform/cli.mdx | 4 ++-- 2 files changed, 52 insertions(+), 4 deletions(-) diff --git a/core/datasets.mdx b/core/datasets.mdx index ca45b52..d6f652e 100644 --- a/core/datasets.mdx +++ b/core/datasets.mdx @@ -83,13 +83,39 @@ Rows that cannot meet a technical contract are counted with the reason, never qu Training reports overlap without dropping selected rows. Checks include normalized inputs, trace/conversation identities, configured groups, case identifiers and synthetic seed lineage. Internal identity columns are hidden from the table, filters and column counts, not removed from stored data. +### What valid rows look like + + + + A `messages` column with a conversation that ends with the assistant, in OpenAI chat format. Rows that call tools also carry a `tools` column with the tool definitions. + + ```json + {"messages": [ + {"role": "system", "content": "You triage support tickets."}, + {"role": "user", "content": "My card was declined twice."}, + {"role": "assistant", "content": "Route to billing. Priority: high."} + ]} + ``` + + + An `input` on every row and an `expected_output` to score against. `input` can be text, an object or a conversation. Other columns are kept. + + ```json + {"input": "My card was declined twice.", "expected_output": "billing"} + ``` + + + +A file does not need to arrive in this shape; the agent converts it during preparation. + ## From your coding agent `/overmind dataset` covers the loop: land a dataset from traces, inspect it, query it with SQL, ask its agent for a change, accept a proposal. Files and exports go through the CLI, because bytes don't travel over MCP: ```bash -overmind dataset upload rows.jsonl --intent eval -overmind dataset export --format jsonl +overmind dataset upload transcripts.jsonl --intent train --wait # wait for landing and the first preparation +overmind dataset upload tickets.jsonl --split 20 --wait # one file as a train and an eval dataset +overmind dataset export --cell 1.2 --output rows.jsonl ``` See [MCP](/platform/mcp) and the [CLI](/platform/cli#overmind-dataset). @@ -107,6 +133,28 @@ An export is a plain download and never counts as a use, so it won't freeze anyt Everything the workshop does is available over REST: create a dataset, add or edit a step, run the chain, message the agent, page through rows with their diff marks, and stream events as they happen. See [Building a dataset](/platform/api#building-a-dataset). +## When something is refused + +| Message | Fix | +| --- | --- | +| `Row 12 has 3 cells; the header has 2.` | A CSV cell holds the delimiter. Quote it, or use JSONL | +| `Column 'notes' appears twice.` | Rename one of the columns | +| `The file is not UTF-8.` | Save the file as UTF-8 and upload it again | +| `Line 40 is not valid JSON.` | Fix that line; the file lands only when every line parses | +| `Two rows are needed to split.` | Add rows, or create a single dataset | +| `no messages column` / `no input column` | Select **Ask Overmind to fix it** on the cell | +| `Longest dataset row is ≈N tokens but this model's … context is M tokens` | Drop or shorten the long rows, or pick a model with a longer context | +| `The dataset is busy.` | The agent or a run is working on it. Wait for it to finish | + +## Limits + +| Limit | Value | +| --- | --- | +| One uploaded file | 2 GB | +| One `.json` document | 256 MB; use JSONL above that | +| One value in a row | 200,000 characters; longer values become a preview marker | +| One cell's run time | 300 seconds | + ## Next steps diff --git a/platform/cli.mdx b/platform/cli.mdx index f159451..84723b9 100644 --- a/platform/cli.mdx +++ b/platform/cli.mdx @@ -145,7 +145,7 @@ Skipped directories: `.git`, `.venv`, `venv`, `node_modules`, `__pycache__`, `.t ## `overmind dataset` ```bash -overmind dataset upload rows.jsonl --intent eval --json +overmind dataset upload rows.jsonl --intent eval --wait --json overmind dataset upload traffic.parquet --split 20 --split-position random overmind dataset export --format csv --cell --output triage.csv ``` @@ -159,7 +159,7 @@ overmind dataset export --format csv --cell --output tria | `--capability` | none | Capability UUID to align the dataset with | | `--split` | off | Land a train dataset and an eval dataset; the eval share in percent (1–99). Drop `--intent`. | | `--split-position` | `tail` | Where the eval rows come from: `head`, `tail`, or `random` | -| `--wait` | off | Wait for landing to finish; exit 1 if it fails | +| `--wait` | off | Wait for the landing and the agent's first preparation; exit 1 with the dataset's error if either fails | | `--api-key`, `--api-url`, `--path` | resolved | Credentials; `--path` points at `overmind.toml` | | `--json` | off | Print one machine-readable object | From 27e19d90fd06a11ca604eaeb1887a3e85a894f74 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 13:10:57 +0100 Subject: [PATCH 16/19] Fold #40 into the introduction and quickstart --- index.mdx | 24 +++++++++++++----------- quickstart.mdx | 35 +++++++++++++++++------------------ 2 files changed, 30 insertions(+), 29 deletions(-) diff --git a/index.mdx b/index.mdx index 11705be..823168b 100644 --- a/index.mdx +++ b/index.mdx @@ -1,6 +1,6 @@ --- title: "Introduction" -description: "Overmind continuously improves production AI agents with their own data: traces in, trained models out." +description: "Overmind improves production AI agents with their own data: traces in; datasets, evals, optimisation runs and trained models out." --- import { Ascii } from "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/snippets/ascii.jsx"; @@ -33,20 +33,22 @@ import { Ascii } from "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/snippets/ascii.jsx"; ]} /> -Overmind ingests OpenTelemetry traces, scores them as they arrive, turns them into curated datasets, to be used for evaluations, optimisations and model training benchmarked against what you currently have. +Overmind ingests OpenTelemetry traces, scores them as they arrive and turns them into datasets. Those datasets drive evals, optimisation runs and training runs, and every result is benchmarked against the agent you ship today. -The platform can be used as [cloud hosted](https://console.overmindlab.ai) or [self-hosted](/platform/self-hosting) with Docker Compose. The platform is AGPL-3.0; the SDK is MIT. See [Licensing](/platform/licensing). +Use the [hosted Console](https://console.overmindlab.ai) or [self-host](/platform/self-hosting) the same code with Docker Compose. The platform is AGPL-3.0 and the SDK is MIT; see [Licensing](/platform/licensing). -Overmind is built on top of standard protocols so no matter what agent you use, you can use Overmind to improve it, with little to no effort. +Everything speaks standard protocols: OTLP/HTTP for ingest, REST JSON for the API, [MCP](/platform/mcp) for coding agents and OpenAI-compatible serving for models. The Python SDK instruments OpenAI, Anthropic, Gemini, Agno and LangChain, and any OpenTelemetry exporter can send to the same endpoint. What Overmind does: -- [Capability scan](/core/capabilities): Scanning your codebase for AI agents and different agentic capabilities. -- [Live scoring](/agent-testing/trace-scoring): every trace is carved into task executions and judged against the agent's eval set -- [Datasets from traces](/core/datasets): traces from real environments become eval and training data. -- [Evaluations](/agent-testing/eval): your in house quality control loop, measured using a typed score, ensuring the agent is always improving. -- [Optimiser](/agent-testing/optimisers): An AI Engineer running experiments on your repository, trying to improve the agent's performance while adhering to your quality control loop. -- [Models](/models/training): Finetuned models that you own, built on top of your custom data. -- [Inference](/models/inference): OpenAI-compatible API for your trained models and frontier models + +- [Capability scan](/core/capabilities): your coding agent maps the repository into capabilities +- [Live scoring](/agent-testing/trace-scoring): every trace is carved into task executions and judged against the capability's eval set +- [Datasets](/core/datasets): traces and files become eval and training data, shaped by the dataset's own agent +- [Eval](/agent-testing/eval): variants measured against a baseline with typed scores +- [Optimiser](/agent-testing/optimisers): candidate code changes replayed in your repository and scored; the winning diff is yours to apply +- [Training](/models/training): fine-tuned models you own, trained on Modal from your data +- [Inference](/models/inference): an OpenAI-compatible API for your trained models and frontier models +- [MCP](/platform/mcp) and [CLI](/platform/cli): the same workflows from Cursor, Claude Code, OpenCode or Codex ## Guides diff --git a/quickstart.mdx b/quickstart.mdx index f6702b9..3a751e7 100644 --- a/quickstart.mdx +++ b/quickstart.mdx @@ -1,47 +1,46 @@ --- title: Quickstart -description: The easiest way to get started with Overmind. +description: Sign up or self-host, paste one prompt into your coding agent, then improve the agent with /overmind commands. --- -Overmind has first class integration with cursor, claude code, opencode, and codex through our SKILL `/overmind` and MCP server, after signing up you can pretty much do everything from your coding agent. - -> We've built the experience to be as easy and native to your coding agent as possible. +Overmind onboards through your coding agent. Cursor, Claude Code, OpenCode and Codex all work: one pasted prompt installs the SDK, connects the MCP server and scans your repository into capabilities. After that, instrumentation, datasets, training and optimisation run from the same chat. - - Start by signing up at [console.overmindlab.ai](https://console.overmindlab.ai). Self-hosted: open your own Console, for example `http://localhost:5173` — see [Self-hosting](/platform/self-hosting). + + Sign up at [console.overmindlab.ai](https://console.overmindlab.ai), or open your own Console, for example `http://localhost:5173` — see [Self-hosting](/platform/self-hosting). A **project** holds your capabilities, traces, datasets, runs, models and keys; the first sync creates it. - Pick your coding agent — **Cursor**, **Claude Code**, **OpenCode**, or **Codex** — and hit **Copy onboarding prompt**. + On the **Agent** page, pick **Cursor**, **Claude Code**, **OpenCode** or **Codex**, then select **Copy onboarding prompt**. The prompt carries a temporary account key, so paste it only into your local coding agent. **Manual setup** shows the same commands for a terminal. - - Paste the prompt into your coding agent (repo root as the workspace folder) and hit enter. After the first sync, reload the agent once so it loads the Overmind MCP server; in Claude Code, exit and run `claude -c`. - - Overmind will now scan your repository for [AI Agents & Capabilities](/core/capabilities#the-scan), you can see them in the [Console](https://console.overmindlab.ai) - - If you want any changes in how they are grouped, evals, etc, just ask your coding agent to do it. - - Next step is to ensure your codebase is instrumented properly for [tracing using OTLP endpoint](/core/observability#the-otlp-endpoint), just run `/overmind ensure-tracing` in your coding agent. + + Open the repository root as the workspace and paste the prompt. The agent installs `overmind`, runs `overmind init` and `overmind sync`, and saves a project-scoped key to `.overmind/credentials.toml`. Reload the agent once so it loads the Overmind MCP server; in Claude Code, exit and run `claude -c`. + The agent then scans the repository into [capabilities](/core/capabilities#the-scan) and syncs them to the Console. To regroup capabilities or change their evals, ask the agent. - The pushed capabilities are the starting point. From the same chat, each command drives one loop end to end. Cursor and Claude Code take them as slash commands; in Codex and OpenCode, ask the agent for the step by name: + Each command drives one loop end to end. Cursor and Claude Code take them as slash commands; in OpenCode and Codex, ask for the step by name. | Command | What it does | | --- | --- | - | `/overmind ensure-tracing` | Inspect codebase and instrument it properly for tracing | - | `/overmind dataset` | Build, clean, upload, or export a dataset | - | `/overmind finetune` | Fine-tune, deploy, and smoke-test a model | + | `/overmind ensure-tracing` | Instrument the code and check that traces arrive | + | `/overmind dataset` | Build, clean, upload or export a dataset | + | `/overmind finetune` | Fine-tune, deploy and smoke-test a model | | `/overmind optimise` | Run prompt and code optimisation | | `/overmind backtest` | Compare models in the repository | + Plain `/overmind` routes to the right workflow when you describe the goal instead. - Once tracing is in, open **Observability**. + Once traces arrive, open **Observability**. **Task executions** shows one row per scored unit of work, **Root traces** one row per trace, and **Sessions** groups traces by session. - + Observability task executions view in the Overmind console +Deployed processes need `OVERMIND_API_KEY` in their runtime configuration; the saved project key covers local runs only. Not using Python or a coding agent? Any OpenTelemetry SDK can export to the [OTLP endpoint](/core/observability#the-otlp-endpoint). + ## Next steps From 8ba484ebb84ca3799f64a1b2ce1c26bdc5e59c0f Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 13:11:43 +0100 Subject: [PATCH 17/19] Document ingest pricing, the repository snapshot and re-scan after delete --- core/capabilities.mdx | 4 ++-- core/observability.mdx | 2 ++ tracing/sdk-python.mdx | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/core/capabilities.mdx b/core/capabilities.mdx index 1c3d012..ddc764c 100644 --- a/core/capabilities.mdx +++ b/core/capabilities.mdx @@ -78,7 +78,7 @@ After code changes: run `/overmind setup` again; it includes the final `overmind A capability the push no longer includes becomes a **leftover**: it keeps its id and data, leaves the agent view, shows a **Not in latest scan** badge, and remounts in place when a later push reproduces it. `slug` in `overmind.toml` is the server identity — rename a capability by editing `name`, never `slug`. -To remove a capability yourself, use **Delete capability** at the bottom of its page (`DELETE /api/capabilities/{id}/`). The record is soft-deleted: it leaves lists, scans and identity lookup, its traces, datasets and runs stay, and its inference alias stops serving. Because a deleted capability is invisible to later syncs, the next push of the same code mints a fresh record. +To remove a capability yourself, use **Delete capability** at the bottom of its page (`DELETE /api/capabilities/{id}/`). The record is soft-deleted: it leaves lists, scans and identity lookup, its traces, datasets and runs stay, and its inference alias stops serving. Because a deleted capability is invisible to later syncs, the next push of the same code mints a fresh record with a new id, and `overmind sync` writes that id to `overmind.toml`. Code that sets the old UUID as `capability_id` needs the new one; a slug keeps binding. ### Telemetry @@ -88,7 +88,7 @@ Ingest never creates a capability. An identity the project does not know leaves ## The agent page -**Agent** in the sidebar opens the product view: the capability grid, or the onboarding prompt while the project is still empty. Open a capability for its page. +**Agent** in the sidebar opens the product view: the capability grid, or the onboarding prompt while the project is still empty. Open a capability for its page. The **Repository snapshot** in the header names the repository, branch and last scan time behind the map. A scan that recorded no revision reads **Revision unavailable**; a project with no scan reads **No repository scan**. Agent page with the capability grid diff --git a/core/observability.mdx b/core/observability.mdx index e3c8002..e49e5f5 100644 --- a/core/observability.mdx +++ b/core/observability.mdx @@ -101,6 +101,8 @@ A project-scoped key pins every span to its project. Sending the same spans twic Overmind already understands the OTel GenAI, OpenLLMetry and OpenInference conventions, so models, providers, token counts, costs, tool names and message payloads from existing instrumentation land without any mapping of your own. +Overmind prices each LLM span once, at ingest. A cost the span reports is kept; otherwise the cost comes from the model, token counts, cache reads and provider, at list prices for the span's start time. Spans from OTLP and from connectors are priced the same way. + These are the ones you set: | Attribute | What it does | diff --git a/tracing/sdk-python.mdx b/tracing/sdk-python.mdx index 3a54a52..913bc7a 100644 --- a/tracing/sdk-python.mdx +++ b/tracing/sdk-python.mdx @@ -205,7 +205,7 @@ for i, doc in enumerate(documents): ### LangChain / LangGraph -`providers=["langchain"]` (or `"auto"`) mounts the OpenInference LangChain instrumentor that ships with `overmind[tracing]`, which covers LangGraph: every chain, model, and tool invocation gets a span with model, token, and cost evidence. For the scoring semantics no instrumentor can know, `overmind.integrations.langgraph.bind` maps graph nodes to behaviour units — call it on the `StateGraph` after `add_node()`, before `compile()`: +`providers=["langchain"]` (or `"auto"`) mounts the OpenInference LangChain instrumentor that ships with `overmind[tracing]`, which covers LangGraph: every chain, model, and tool invocation gets a span with model and token evidence, and Overmind prices it at ingest. For the scoring semantics no instrumentor can know, `overmind.integrations.langgraph.bind` maps graph nodes to behaviour units — call it on the `StateGraph` after `add_node()`, before `compile()`: ```python from overmind.integrations import langgraph as overmind_langgraph From 02faff41f949e63cdfb05a564043f42cbbbbbfb6 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 13:12:15 +0100 Subject: [PATCH 18/19] Make the merged MCP and training text read the same way --- models/training.mdx | 5 +---- platform/mcp.mdx | 14 ++++++++------ 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/models/training.mdx b/models/training.mdx index 0bffaa1..028a77b 100644 --- a/models/training.mdx +++ b/models/training.mdx @@ -27,10 +27,7 @@ Independent train and eval data gives meaningful held-out scores. Overmind repor Training wizard with dataset setup and model picker -For MCP `start_finetune`, omit `hyperparameters` to derive model-specific defaults. -When supplying settings, use `"training_type": {"type": "Lora"}` or -`"training_type": {"type": "Full"}`. A string training type is rejected before -job creation with a field-level validation error. +Over MCP, `start_finetune` derives model-specific defaults when `hyperparameters` is omitted. To set the method yourself, pass `"training_type": {"type": "Lora"}` or `{"type": "Full"}`; a plain string is refused before the job is created, with a field-level error. ## Exact preprocessing diff --git a/platform/mcp.mdx b/platform/mcp.mdx index 1ba58e6..d0dd020 100644 --- a/platform/mcp.mdx +++ b/platform/mcp.mdx @@ -1,6 +1,6 @@ --- title: MCP -description: The Overmind MCP server — setup per coding agent, authentication, the 36 tools, workflow prompts and the resources, and the CLI handoff for local work. +description: The Overmind MCP server — setup per coding agent, the optional plugin, authentication, the 36 tools, 12 prompts, resources, workflow skills and the CLI handoff for local work. --- MCP is the platform's first-class agent surface: an account-aware [Model Context Protocol](https://modelcontextprotocol.io) server at `/api/mcp/` that shares its services with the REST API and the Console. A coding agent connected to it can inspect health and failures, land and shape datasets, author evaluators and run evals, launch and follow training, schedule optimisation runs, and drive connectors and instrumentation — everything except the steps that need your filesystem, which it hands to the [CLI](/platform/cli). @@ -54,15 +54,17 @@ Start with `list_projects`. To see a project in the browser, read `overmind://pr ## Authentication -OAuth connections and account API keys discover authorized projects through `list_projects`. Pass the chosen `project_id` on every project tool and as a query parameter on resource URIs. Membership is checked for each operation. There is no shared selected-project state, so concurrent work cannot switch another request's project. Returned resource links retain the project ID. A project API key can only use its configured project and can omit `project_id`. +**Account connections.** OAuth connections and account API keys find the projects you can access with `list_projects`. Pass the chosen `project_id` to every project tool, and as a query parameter on resource URIs. Membership is checked on every operation, and there is no shared "current project", so parallel work cannot switch another request's project. Returned resource links keep the project id. -Read access exposes read-only tools; mutations require write access. API keys retain expiry, active-user and IP-allowlist checks. OAuth grants request `overmind:read` and/or `overmind:write` and require Console sign-in and explicit consent. Tokens use S256 PKCE, exact redirect matching, one-time codes and resource binding; OAuth tokens are accepted only by MCP. Access tokens expire after one hour. Connections remain authorized until revoked, with no refresh-token time limit. Refresh tokens still rotate; reuse revokes the connection's token family. Disabled accounts and lost project memberships continue to lose access. +**Project API keys** reach only their own project and can omit `project_id`. -Set `MCP_SERVER_URL` to the canonical HTTPS `/api/mcp/` endpoint on a hosted deployment, with `FRONTEND_URL` pointing at its Console. HTTP is accepted only for local loopback development. Configured servers advertise OAuth discovery, public-client registration (`token_endpoint_auth_method: none`), authorization, token and revocation endpoints. API-key connections continue to use `X-Api-Key` or `Authorization: Bearer`. +**Permissions.** Read access exposes the read-only tools; writes need write access. API keys keep their expiry, active-user and IP-allowlist checks, and authenticate as `X-Api-Key` or `Authorization: Bearer`. -To publish the server as an OpenAI app, set `OPENAI_APPS_CHALLENGE` to the portal's ownership token; the API serves it at `/.well-known/openai-apps-challenge`. +**OAuth.** A grant asks for `overmind:read` and/or `overmind:write`, and needs Console sign-in and explicit consent. Tokens use S256 PKCE, exact redirect matching, one-time codes and resource binding, and only MCP accepts them. Access tokens expire after one hour. A connection stays authorized until it is revoked: refresh tokens have no time limit, but they rotate, and reusing an old one revokes the connection. A disabled account or a lost project membership ends access. -MCP transport requests use `POST` with `application/json`, from an allowed host and origin, within the protocol's body limit. Authentication failures return `401`; authorization failures return `403`. Tool errors are structured results: `{"error": {code, message, retryable, fields}}`. +**Hosting.** On a public deployment, set `MCP_SERVER_URL` to the HTTPS `/api/mcp/` endpoint and `FRONTEND_URL` to the Console. Plain HTTP is accepted only on local loopback. The server then advertises OAuth discovery, public-client registration (`token_endpoint_auth_method: none`), and the authorization, token and revocation endpoints. To publish the server as an OpenAI app, set `OPENAI_APPS_CHALLENGE` to the portal's ownership token; the API serves it at `/.well-known/openai-apps-challenge`. + +**Transport.** Requests are `POST` with `application/json`, from an allowed host and origin, within the protocol's body limit. Authentication failures return `401` and authorization failures `403`. Tool errors are structured results: `{"error": {code, message, retryable, fields}}`. The public surface is read and write only. There are no delete, cancel or generic API tools; `retry_deployment` is the one lifecycle exception and covers exactly the failed-or-deleted deployment case. From 8cbcfafbb2495c441f0a9bdcd0be0c36c07224f6 Mon Sep 17 00:00:00 2001 From: freyam Date: Thu, 1 Oct 2026 13:13:44 +0100 Subject: [PATCH 19/19] List every MCP tool argument the server accepts --- platform/mcp.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/platform/mcp.mdx b/platform/mcp.mdx index d0dd020..d085e95 100644 --- a/platform/mcp.mdx +++ b/platform/mcp.mdx @@ -97,7 +97,7 @@ Project tools also accept `project_id`, required for account connections. `list_ | `list_datasets` | `capability?`, `intent?`, `state?`, `search?`, `limit?`, `offset?` | List project datasets | | `inspect_dataset` | `dataset`, `chat_limit?` | One dataset: cell chain, sample, recent agent chat, next actions | | `query_dataset` | `dataset`, `sql`, `cell?`, `limit?` | Read-only SQL over one ran cell | -| `create_dataset_from_traces` | `name`, `trace_ids?`, `filters?`, `search?`, `limit?`, `intent?`, `capability?` | Land a dataset, one row per trace; the selection is counted first and refused when empty. Returns a task receipt | +| `create_dataset_from_traces` | `name`, `trace_ids?`, `filters?`, `search?`, `limit?`, `intent?`, `capability?`, `split?` | Land a dataset, one row per trace; the selection is counted first and refused when empty. `split` lands disjoint ` train` and ` eval` datasets instead of one with an intent. Returns a task receipt | | `message_dataset_agent` | `dataset`, `message` | Queue one agent turn for an idle dataset. Returns a job receipt | | `run_dataset` | `dataset`, `proposal_cell?` | Run the dataset, optionally accepting one proposal. Returns a job receipt | @@ -105,10 +105,10 @@ Project tools also accept `project_id`, required for account connections. `list_ | Tool | Arguments | Effect | | --- | --- | --- | -| `check_evaluation_readiness` | `dataset`, `eval_set?`, `cell?`, `version?`, `mode?` | Dataset, active eval set, evaluator applicability, variable bindings, credit readiness | +| `check_evaluation_readiness` | `dataset`, `eval_set?`, `cell?`, `version?`, `mode?`, `variants?`, `judge_model?` | Dataset, active eval set, evaluator applicability, variable bindings, credit readiness | | `create_eval_set` | `name`, `evaluator_ids`, `capability?` | Atomically create an eval set and its members in their applicable roles; does not activate it or launch a run | | `upsert_evaluator` | `evaluator?`, `name?`, `kind?`, `capability?`, `scope?`, `score_type?`, `display_name?`, `description?`, `rubric_md?`, `evaluation_prompt?`, `checklist?`, `judge_model?`, `score_min?`, `score_max?`, `pass_threshold?`, `requires_reference?`, `variable_mapping?`, `config?`, `applicable_roles?`, `surface?`, `choices?` | Create or update an evaluator | -| `run_evaluation` | `name`, `dataset`, `eval_set?`, `cell?`, `version?`, `evaluator_ids?`, `variants?`, `max_items?`, `sampling?` | Create and launch an eval run. Returns a job receipt | +| `run_evaluation` | `name`, `dataset`, `eval_set?`, `cell?`, `version?`, `evaluator_ids?`, `variants?`, `max_items?`, `sampling?`, `judge_model?` | Create and launch an eval run. Returns a job receipt | | `compare_evaluations` | `run`, `baseline` | Compare a run with a project-scoped baseline | | `annotate_evaluation_sample` | `annotation?`, `sample?`, `evaluator?`, `value?`, `label?`, `note?` | Create or update a human annotation, attributed to the key's owner | @@ -119,7 +119,7 @@ Project tools also accept `project_id`, required for account connections. `list_ | `get_model_catalog` | `has_tool_calling?`, `max_context?` | Models available for fine-tuning: tiers, context limits, training methods, tool-calling, batch bounds | | `check_finetune_readiness` | `dataset`, `capability?`, `cell?`, `version?` | Prerequisites, train/eval dataset shape, recommended models, evaluators, credits | | `estimate_finetune` | `dataset`, `base_model`, `n_epochs?`, `use_lora?`, `cell?`, `version?` | Real-token cost and duration estimate; creates nothing | -| `start_finetune` | `dataset`, `base_model`, `capability?`, `eval_dataset?`, `eval_set?`, `validation_dataset?`, `cell?`, `version?`, `eval_cell?`, `eval_version?`, `validation_cell?`, `validation_version?`, `validation_enabled?`, `validation_split_ratio?`, `split_method?`, `name?`, `use_case?`, `hyperparameters?`, `group_id?` | Validate and queue a training job. Returns a job receipt | +| `start_finetune` | `dataset`, `base_model`, `capability?`, `eval_dataset?`, `eval_set?`, `validation_dataset?`, `cell?`, `version?`, `eval_cell?`, `eval_version?`, `validation_cell?`, `validation_version?`, `validation_enabled?`, `validation_split_ratio?`, `split_method?`, `name?`, `use_case?`, `hyperparameters?`, `group_id?`, `baseline_model?`, `eval_judge_model?`, `eval_incumbent_before?`, `eval_incumbent_after?`, `eval_model_before?`, `eval_model_after?` | Validate and queue a training job. `baseline_model` overrides the capability's benchmark for this job; the `eval_*` flags choose which comparison evaluations run. Returns a job receipt | | `prepare_training_data` | `dataset`, `base_model`, `context_length`, `training_type?`, `validation_dataset?`, `cell?`, `retry_failed?` | Run exact tokenization for a model and context length; returns a cached preparation report. Poll with `get_job(kind="training_preparation")` | | `retry_deployment` | `deployment` | Retry a failed or deleted deployment. Returns a job receipt | | `set_active_model` | `capability`, `deployment?` | Verify a ready deployment, then switch the alias; poll the returned `model_activation` job. Omit deployment to clear routing | @@ -142,7 +142,7 @@ Project tools also accept `project_id`, required for account connections. `list_ | Tool | Arguments | Effect | | --- | --- | --- | | `inspect_connectors` | `connector?`, `include_source_projects?`, `preview?`, `preview_source_project_id?`, `preview_lookback_days?`, `max_runs?` | List or inspect connectors: configuration, mappings, provider capabilities, sync status | -| `configure_connector` | `connector`, `source_project_id?`, `lookback_days?`, `backfill_from?`, `backfill_to?`, `capability_mapping?`, `auto_sync_enabled?`, `poll_interval_seconds?` | Save source project, lookback, capability mapping and polling settings | +| `configure_connector` | `connector`, `source_project_id?`, `lookback_days?`, `backfill_from?`, `backfill_to?`, `capability_mapping?`, `auto_sync_enabled?`, `poll_interval_seconds?`, `confirm_mapping?` | Save source project, lookback, capability mapping and polling settings; `confirm_mapping` confirms the reviewed mapping | | `sync_connector` | `connector` | Queue the configured import. Returns a task receipt | A connector is created with its provider credentials through `overmind connector add` or the Console under Integrations; until then the tools answer with the `connector_setup_required` action.