diff --git a/agent-testing/decision-engine.mdx b/agent-testing/decision-engine.mdx index 0cb4cc2..b6aa0c9 100644 --- a/agent-testing/decision-engine.mdx +++ b/agent-testing/decision-engine.mdx @@ -28,7 +28,7 @@ Configurable rubric judges default to **generative**. Jev is an explicit opt-in after validation on representative labelled examples for that rubric. A high confidence value is not proof of correctness or agreement with the existing judge. Fixed grounding, capability classification and workshop semantic services use -Jev with generative fallback. Existing explicit policies and snapshots are preserved. +Jev with generative fallback. The Console's evaluator editor exposes **Decision engine** and **Minimum confidence**. The REST authoring request accepts `decision_policy`. General evaluator writes and @@ -48,7 +48,7 @@ MCP `upsert_evaluator` accept the same policy under `config.decision`: Set `backend` to `generative` to use the generative judge throughout that evaluator. The policy is frozen in new evaluator snapshots and included in verdict identities. The requested Jev release is pinned; the provider's actual served revision is -recorded with each response. Historical results are not relabelled or rescored. +recorded with each response. Changing a policy does not rescore earlier results. Responses must cover every question, use only declared options, and return valid probability distributions. Low-confidence answers, invalid responses, unavailable diff --git a/agent-testing/eval.mdx b/agent-testing/eval.mdx index 82b9592..6e7eaac 100644 --- a/agent-testing/eval.mdx +++ b/agent-testing/eval.mdx @@ -63,7 +63,7 @@ Judges are authored behaviour-scoped from the behaviour's group in the Eval libr Author from the Eval library by hand, or describe what you want to test and let Overmind generate the evaluator (`POST /api/evaluators/generate-prompt/`). The generated prompt template references the agent's output as `{{output.}}` and dataset fields as `{{reference.}}`, and everything stays editable before saving. -`overmind sync` preloads the capability's Default eval set for every capability it created, remounted, or whose card changed, in tiers: Tier 0, rule-based checks compiled from the capability's card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. When Tier 1 authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability's **evidence profile** — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply. +`overmind sync` refreshes the Default eval set of every current capability, in tiers: Tier 0, rule-based checks compiled from the capability's card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. Tier 1 authors once per capability; when it authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability's **evidence profile** — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply. Eval library grouped by behaviour @@ -204,7 +204,7 @@ Each dataset row has its own entry in the breakdown, with responses from differe The datapoint popup shows **Input** beside **Output and evaluation**. Select a model to switch both panels together. Generated output is the model's response, not a label extracted by a statistical evaluator; the **Reference answer** is shown separately as grading data. Expand **Generated messages** to inspect recorded assistant and tool messages. -New generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider's rendered chat template or every subsequent replay request. Older samples show **Dataset input** from the run's pinned version and state that the exact model request was not captured. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript. Existing runs do not need to be rerun to inspect their pinned dataset input. +Generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider's rendered chat template or every subsequent replay request. A sample without a captured request shows **Dataset input** from the run's pinned version. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript. The MCP `overmind://eval-runs/{eval_run}` resource exposes up to five bounded sample inspections with separate input, output, reference, and evaluator reasoning. `input_source` distinguishes captured requests, pinned dataset rows, and unavailable input; `samples_truncated` indicates additional samples. diff --git a/agent-testing/optimisers.mdx b/agent-testing/optimisers.mdx index e3ba393..04afd7b 100644 --- a/agent-testing/optimisers.mdx +++ b/agent-testing/optimisers.mdx @@ -20,7 +20,7 @@ The Optimiser takes an agent you can already measure and edits the agent itself. Candidates are graded by the eval set you pick, or the capability's active eval set by default. The set must belong to the capability. See [Eval](/agent-testing/eval#eval-sets). - Your agent executes in **your** environment, not Overmind's. The loop is driven from your checkout by the `overmind` CLI, and the candidate diffs are written by your coding agent. **Optimiser → New run** in the Console offers **Improve prompts & code** to test changes to prompts, tool definitions and application logic, or **Compare models** to evaluate different models while keeping prompts and logic fixed. Select the capability and eval dataset, then use **Copy prompt** and paste into your coding agent in that capability's repository. Copying does not start a run. The `/overmind optimise` and `/overmind backtest` commands are unchanged; scores and progress appear in Overmind. + Your agent executes in **your** environment, not Overmind's. The loop is driven from your checkout by the `overmind` CLI, and the candidate diffs are written by your coding agent. **Optimiser → New run** in the Console offers **Improve prompts & code** to test changes to prompts, tool definitions and application logic, or **Compare models** to evaluate different models while keeping prompts and logic fixed. Select the capability and eval dataset, then use **Copy prompt** and paste into your coding agent in that capability's repository. Copying does not start a run. The `/overmind optimise` and `/overmind backtest` commands drive the same runs; scores and progress appear in Overmind. diff --git a/core/capabilities.mdx b/core/capabilities.mdx index 831f2f6..ddc764c 100644 --- a/core/capabilities.mdx +++ b/core/capabilities.mdx @@ -13,7 +13,7 @@ Your **agent** in Overmind is the product itself: one per project, mapped as a g lines={[ "╔══════════════════════╗ ╔═ Overmind ══════╗", "║ Local repo ║ ║ ║", - "║chassis → cards → sync╟────POST /api/v1/sync/─────┐ ║ ║", + "║chassis → cards → sync╟────POST /api/v1/sync──────┐ ║ ║", "╚══════════════════════╝ │ ║ ╔════════════╗ ║", " ├───╫─▶║ Capability ║ ║", "╔══════════════════════╗ │ ║ ╚════════════╝ ║", @@ -72,29 +72,29 @@ After code changes: run `/overmind setup` again; it includes the final `overmind - **Identity is carried**, in order: the toml `id` → the toml `slug` → a new record. A capability the push no longer includes becomes a leftover (`archived = true` in the toml on the next `sync down`). - **Behaviours are minted from the trajectory map**: one contract per mapped route, with an entry anchor, the ordered anchors that follow it, and its tool set. Behaviours are what [trace scoring](/agent-testing/trace-scoring#behaviours-and-bindings) binds production traffic to; the Console labels the column **Task**. -- **The Default eval set is preloaded** for every created, remounted or card-changed capability: Tier-0 rule-based checks compiled from the card, Tier-1 generative LLM judges grounded in it, and per-behaviour outcome and step judges for trace scoring. See [Eval](/agent-testing/eval#authoring). +- **The Default eval set is refreshed** for every current capability on each sync: Tier-0 rule-based checks compiled from the card (deduplicated), Tier-1 generative LLM judges grounded in it (authored once per capability), and per-behaviour outcome and step judges for trace scoring. See [Eval](/agent-testing/eval#authoring). ### A scan never deletes A capability the push no longer includes becomes a **leftover**: it keeps its id and data, leaves the agent view, shows a **Not in latest scan** badge, and remounts in place when a later push reproduces it. `slug` in `overmind.toml` is the server identity — rename a capability by editing `name`, never `slug`. -To remove a capability yourself, use **Delete capability** at the bottom of its page (`DELETE /api/capabilities/{id}/`). The record is soft-deleted: it leaves lists, scans and identity lookup, its traces, datasets and runs stay, and its inference alias stops serving. Because a deleted capability is invisible to later syncs, the next push of the same code mints a fresh record. +To remove a capability yourself, use **Delete capability** at the bottom of its page (`DELETE /api/capabilities/{id}/`). The record is soft-deleted: it leaves lists, scans and identity lookup, its traces, datasets and runs stay, and its inference alias stops serving. Because a deleted capability is invisible to later syncs, the next push of the same code mints a fresh record with a new id, and `overmind sync` writes that id to `overmind.toml`. Code that sets the old UUID as `capability_id` needs the new one; a slug keeps binding. ### Telemetry Spans carrying `overmind.capability.id` — set with [`init(capability_id=...)`](/tracing/sdk-python#init) or a `capability(..., id=)` scope — bind to that capability. The value is the capability's UUID or its toml `slug`; both resolve, and both are stable through renames. `overmind.capability.name` is a display label and never binds. -Ingest never creates a capability. An identity the project does not know leaves the span **unbound**: it is stored, browsable under Observability's Unbound filter, and binds retroactively when a later push makes the identity resolvable. `confirmed_at` is stamped on the capability the first time a span binds to one of its behaviours; the capability page shows **Confirmed** or **Unconfirmed** accordingly. A capability created by hand (`POST /api/capabilities/`) carries an **Observed** badge instead, because no scan can confirm or retire it. +Ingest never creates a capability. An identity the project does not know leaves the span **unbound**: it is stored, browsable under Observability's Unbound filter, and binds retroactively when a later push makes the identity resolvable. A capability created by hand (`POST /api/capabilities/`) carries an **Observed** badge, because no scan can retire it. ## The agent page -**Agent** in the sidebar opens the product view: the capability grid, or the onboarding prompt while the project is still empty. Open a capability for its page. +**Agent** in the sidebar opens the product view: the capability grid, or the onboarding prompt while the project is still empty. Open a capability for its page. The **Repository snapshot** in the header names the repository, branch and last scan time behind the map. A scan that recorded no revision reads **Revision unavailable**; a project with no scan reads **No repository scan**. Agent page with the capability grid -The header carries the name (editable in place), copyable **capability id** and **project id** chips, the slug, the badges — **Declared** or **Inferred** (whether the manifest named it), **Confirmed** or **Unconfirmed**, **Not in latest scan** — and a **Last push** line linking to the project's scan report. **View traces** opens the capability's filtered Observability view. Below it, one scrolling page: +The header carries the name (editable in place), copyable **capability id** and **project id** chips, the slug, and the **Not in latest scan** and **Observed** badges when they apply. **View traces** opens the capability's filtered Observability view. Below it, one scrolling page: - **Facts** — the model the capability serves through (or the live deployed model behind its alias) and its source path. - **Prompt** — the system prompt lifted from your code, one per mode when the capability has several. @@ -112,7 +112,7 @@ The header carries the name (editable in place), copyable **capability id** and Trajectory map fullscreen view -The Agent page is the record of the latest sync: each capability with its card, behaviours, and whether telemetry has confirmed it. +The Agent page is the record of the latest sync: each capability with its card and behaviours. ## Instrumentation diff --git a/core/datasets.mdx b/core/datasets.mdx index cf867ad..d6f652e 100644 --- a/core/datasets.mdx +++ b/core/datasets.mdx @@ -62,7 +62,7 @@ Quality and capability-alignment findings remain advisory, including train/eval Ask explicitly for synthetic examples. Validated batches accumulate in one active generated version, with no draft or Apply step. Retries do not add duplicate rows; interrupted generation keeps completed batches and can continue to the same target. Used versions cannot receive more batches. Seed lineage, capability context, instructions and generation engine are recorded. Synthetic answers still need independent review and cannot be split away from their seeds into held-out evaluation. Automatic preparation never generates synthetic rows. -Chat activity is chronological, with expandable thinking and approval checkpoints. Clearing chat does not remove applied versions. The agent uses the first configured engine: Cursor, OpenRouter, then OpenAI, Anthropic or Gemini. With no key, the workshop opens and reports that the agent is unavailable; see [Self-hosting](/platform/self-hosting#the-data-workshop-agent). +Chat activity is chronological, with expandable thinking and approval checkpoints. Clearing chat does not remove applied versions. The agent runs on OpenRouter, or as a Cursor Composer session when `CURSOR_API_KEY` is set (recommended); see [Self-hosting](/platform/self-hosting#the-data-workshop-agent). ## Versions @@ -83,13 +83,39 @@ Rows that cannot meet a technical contract are counted with the reason, never qu Training reports overlap without dropping selected rows. Checks include normalized inputs, trace/conversation identities, configured groups, case identifiers and synthetic seed lineage. Internal identity columns are hidden from the table, filters and column counts, not removed from stored data. +### What valid rows look like + + + + A `messages` column with a conversation that ends with the assistant, in OpenAI chat format. Rows that call tools also carry a `tools` column with the tool definitions. + + ```json + {"messages": [ + {"role": "system", "content": "You triage support tickets."}, + {"role": "user", "content": "My card was declined twice."}, + {"role": "assistant", "content": "Route to billing. Priority: high."} + ]} + ``` + + + An `input` on every row and an `expected_output` to score against. `input` can be text, an object or a conversation. Other columns are kept. + + ```json + {"input": "My card was declined twice.", "expected_output": "billing"} + ``` + + + +A file does not need to arrive in this shape; the agent converts it during preparation. + ## From your coding agent `/overmind dataset` covers the loop: land a dataset from traces, inspect it, query it with SQL, ask its agent for a change, accept a proposal. Files and exports go through the CLI, because bytes don't travel over MCP: ```bash -overmind dataset upload rows.jsonl --intent eval -overmind dataset export --format jsonl +overmind dataset upload transcripts.jsonl --intent train --wait # wait for landing and the first preparation +overmind dataset upload tickets.jsonl --split 20 --wait # one file as a train and an eval dataset +overmind dataset export --cell 1.2 --output rows.jsonl ``` See [MCP](/platform/mcp) and the [CLI](/platform/cli#overmind-dataset). @@ -107,6 +133,28 @@ An export is a plain download and never counts as a use, so it won't freeze anyt Everything the workshop does is available over REST: create a dataset, add or edit a step, run the chain, message the agent, page through rows with their diff marks, and stream events as they happen. See [Building a dataset](/platform/api#building-a-dataset). +## When something is refused + +| Message | Fix | +| --- | --- | +| `Row 12 has 3 cells; the header has 2.` | A CSV cell holds the delimiter. Quote it, or use JSONL | +| `Column 'notes' appears twice.` | Rename one of the columns | +| `The file is not UTF-8.` | Save the file as UTF-8 and upload it again | +| `Line 40 is not valid JSON.` | Fix that line; the file lands only when every line parses | +| `Two rows are needed to split.` | Add rows, or create a single dataset | +| `no messages column` / `no input column` | Select **Ask Overmind to fix it** on the cell | +| `Longest dataset row is ≈N tokens but this model's … context is M tokens` | Drop or shorten the long rows, or pick a model with a longer context | +| `The dataset is busy.` | The agent or a run is working on it. Wait for it to finish | + +## Limits + +| Limit | Value | +| --- | --- | +| One uploaded file | 2 GB | +| One `.json` document | 256 MB; use JSONL above that | +| One value in a row | 200,000 characters; longer values become a preview marker | +| One cell's run time | 300 seconds | + ## Next steps diff --git a/core/observability.mdx b/core/observability.mdx index e3c8002..e49e5f5 100644 --- a/core/observability.mdx +++ b/core/observability.mdx @@ -101,6 +101,8 @@ A project-scoped key pins every span to its project. Sending the same spans twic Overmind already understands the OTel GenAI, OpenLLMetry and OpenInference conventions, so models, providers, token counts, costs, tool names and message payloads from existing instrumentation land without any mapping of your own. +Overmind prices each LLM span once, at ingest. A cost the span reports is kept; otherwise the cost comes from the model, token counts, cache reads and provider, at list prices for the span's start time. Spans from OTLP and from connectors are priced the same way. + These are the ones you set: | Attribute | What it does | diff --git a/custom.css b/custom.css index f3802ec..2254cc8 100644 --- a/custom.css +++ b/custom.css @@ -96,6 +96,8 @@ --text-3: var(--stone-400); --text-disabled: var(--stone-600); --code-bg: #0e0c09; + /* Floating buttons (58px) + their right inset (16px) + 8px of air. */ + --code-button-gutter: 82px; --font-display: "PP Mondwest", ui-serif, Georgia, serif; --font-nav: "NeueBit", ui-monospace, monospace; @@ -894,13 +896,21 @@ html[data-page-mode="wide"] #content-area { } /* Code-block header (filename / language tabs) as a strip on the same panel. */ -#content-area :is(.code-block, .code-group) > div:first-child:not(:has(pre)) { +#content-area :is(.code-block, .code-group) > div:first-child:not(:has(pre), [data-floating-buttons]) { background: var(--surface) !important; border-bottom: 1px solid var(--frame) !important; font-family: var(--font-mono) !important; font-size: 12px !important; color: var(--text-3) !important; } +/* The code viewport stops at a gutter under the floating buttons, so a long line + clips before them instead of running beneath; the fade overlay has nothing to cover. */ +#content-area .code-block > [data-floating-buttons] + [data-component-part="code-block-root"] { + padding-right: var(--code-button-gutter); +} +#content-area [data-fade-overlay] { + display: none !important; +} #content-area :is(.code-block, .code-group) [role="tab"] { font-family: var(--font-mono) !important; font-size: 12px !important; @@ -914,21 +924,17 @@ html[data-page-mode="wide"] #content-area { border-bottom-color: var(--copper-500) !important; } -#content-area [data-testid="copy-code-button"] { +#content-area :is([data-testid="copy-code-button"], #ask-assistant-code-block-button) { background: transparent !important; border: 0 !important; color: var(--text-3) !important; clip-path: var(--notch); transition: color 150ms ease, background-color 150ms ease; } -#content-area [data-testid="copy-code-button"]:hover { +#content-area :is([data-testid="copy-code-button"], #ask-assistant-code-block-button):hover { color: var(--text) !important; background: var(--surface-2) !important; } -#content-area [data-fade-overlay] { - background: var(--code-bg) !important; - background-image: none !important; -} /* ─── Tables — hairlines only ───────────────────────────────────────────── */ @@ -1282,6 +1288,17 @@ html[data-page-mode="wide"] #content-area { color: var(--text) !important; border-bottom-color: var(--copper-500) !important; } +/* Mintlify underlines the inner tab button too; the tab's own border is the one indicator. */ +#content-area [data-component-part="tab-button"] { + border: 0 !important; + padding: 0 !important; + margin: 0 !important; + color: inherit !important; + font: inherit !important; +} +#content-area [role="tab"]:focus-visible { + outline-offset: -2px; +} /* ─── kbd ───────────────────────────────────────────────────────────────── */ diff --git a/index.mdx b/index.mdx index c5e9987..823168b 100644 --- a/index.mdx +++ b/index.mdx @@ -1,6 +1,6 @@ --- title: "Introduction" -description: "Overmind continuously improves production AI agents with their own data: traces in, trained models out." +description: "Overmind improves production AI agents with their own data: traces in; datasets, evals, optimisation runs and trained models out." --- import { Ascii } from "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/snippets/ascii.jsx"; @@ -33,20 +33,22 @@ import { Ascii } from "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/snippets/ascii.jsx"; ]} /> -Overmind ingests OpenTelemetry traces, scores them as they arrive, turns them into curated datasets, to be used for evaluations, optimisations and model training benchmarked against what you currently have. +Overmind ingests OpenTelemetry traces, scores them as they arrive and turns them into datasets. Those datasets drive evals, optimisation runs and training runs, and every result is benchmarked against the agent you ship today. -The platform can be used as [cloud hosted](https://console.overmindlab.ai) or [self-hosted](/platform/self-hosting) with Docker Compose. The platform is AGPL-3.0; the SDK is MIT. See [Licensing](/platform/licensing). +Use the [hosted Console](https://console.overmindlab.ai) or [self-host](/platform/self-hosting) the same code with Docker Compose. The platform is AGPL-3.0 and the SDK is MIT; see [Licensing](/platform/licensing). -Overmind is built on top of standard protocols so no matter what agent you use, you can use Overmind to improve it, with little to no effort. +Everything speaks standard protocols: OTLP/HTTP for ingest, REST JSON for the API, [MCP](/platform/mcp) for coding agents and OpenAI-compatible serving for models. The Python SDK instruments OpenAI, Anthropic, Gemini, Agno and LangChain, and any OpenTelemetry exporter can send to the same endpoint. What Overmind does: -- [Capability scan](/core/capabilities): Scanning your codebase for AI agents and different agentic capabilities. -- [Live scoring](/agent-testing/trace-scoring): every trace is carved into task executions and judged against the agent's eval set -- [Datasets from traces](/core/datasets): traces from real environments become eval and training data, no synthetic data, no data poisoning, no hallucinations. -- [Evaluations](/agent-testing/eval): your in house quality control loop, measured using a typed score, ensuring the agent is always improving. -- [Optimiser](/agent-testing/optimisers): An AI Engineer running experiments on your repository, trying to improve the agent's performance while adhering to your quality control loop. -- [Models](/models/training): Finetuned models that you own, built on top of your custom data. -- [Inference](/models/inference): OpenAI-compatible API for your trained models and frontier models + +- [Capability scan](/core/capabilities): your coding agent maps the repository into capabilities +- [Live scoring](/agent-testing/trace-scoring): every trace is carved into task executions and judged against the capability's eval set +- [Datasets](/core/datasets): traces and files become eval and training data, shaped by the dataset's own agent +- [Eval](/agent-testing/eval): variants measured against a baseline with typed scores +- [Optimiser](/agent-testing/optimisers): candidate code changes replayed in your repository and scored; the winning diff is yours to apply +- [Training](/models/training): fine-tuned models you own, trained on Modal from your data +- [Inference](/models/inference): an OpenAI-compatible API for your trained models and frontier models +- [MCP](/platform/mcp) and [CLI](/platform/cli): the same workflows from Cursor, Claude Code, OpenCode or Codex ## Guides diff --git a/models/inference.mdx b/models/inference.mdx index 7e0b855..8358115 100644 --- a/models/inference.mdx +++ b/models/inference.mdx @@ -11,8 +11,8 @@ The Console model list shows deployments from training runs. Base-model deployme A deployment moves through `queued → deploying → warming → ready`, or ends `failed`, `deleting`, `deleted`. Two shapes: -- A dense model trained on Modal with LoRA is served as an **adapter on a shared BF16 base**; deployments of the same base share one process and skip the `quantizing` stage. -- Everything else — full fine-tunes, mixture-of-experts models, models whose family has no adapter support — is **merged and quantised into a private checkpoint** (FP8 where the base supports it) and served from its own vLLM process. +- A dense or explicitly supported mixture-of-experts model trained on Modal with LoRA is served as an **adapter on a shared BF16 base**; deployments of the same base share one process and skip the `quantizing` stage. +- Everything else — full fine-tunes, other mixture-of-experts models, models whose family has no adapter support — is **merged and quantised into a private checkpoint** (FP8 where the base supports it) and served from its own vLLM process. Each process scales to zero when idle: two minutes without traffic for most models, fifteen for the largest. The first request after that cold-boots the model in two to seven minutes; the response is held open with an idle ping every 15 seconds so the connection survives. @@ -112,7 +112,7 @@ Through MCP, call `set_active_model` and poll its `model_activation` receipt wit The list shows **Model**, **Base model**, **Capability**, **Training job**, **Reference**, **Status**, **Requests**, **Tokens**, **Throughput**, **Latency** and **Last active**; latency and throughput are medians over warm calls. Status shows the deployment status until it is `ready`; a ready model shows its serving state from `live/` — **Warm** when its worker pool has running containers or it served successful traffic in the last 90 seconds, **Warming** while a cold boot is in progress, **Asleep** when its measured pool has no containers, and **Unknown** only when worker measurements are unavailable and there is no recent serving signal. The list and detail page refresh this state every five seconds; checking status does not wake the model. The separate **Live model** badge identifies the capability’s selected route, independently of worker warmth. -The model page focuses on **Inference performance**: request volume, success rate, end-to-end p50 and p95 response time, median warm generation speed, and estimated serving cost. The reporting period applies to both headline metrics and charts: **Last hour**, **Last 24 hours**, **Last 7 days**, **Last 30 days** or **All time**. **Application traffic** includes API-key requests to either the capability alias or a pinned model ID; **All traffic** also includes internal calls and historical requests whose source is unknown. Metrics refresh every 15 seconds while the page is open. +The model page focuses on **Inference performance**: request volume, success rate, end-to-end p50 and p95 response time, median warm generation speed, and estimated serving cost. The reporting period applies to both headline metrics and charts: **Last hour**, **Last 24 hours**, **Last 7 days**, **Last 30 days** or **All time**. **Application traffic** includes API-key requests to either the capability alias or a pinned model ID; **All traffic** also includes internal calls and requests with an unknown source. Metrics refresh every 15 seconds while the page is open. The charts show request traffic and failures, response-time percentiles, and generation speed. Empty traffic buckets are zero; missing timing measurements remain gaps. Failed calls do not contribute to latency or speed distributions. A single empty state replaces the charts when the selected period has no requests. Refresh errors are visible and do not turn into zero traffic. The latest failure includes its request ID. diff --git a/models/training.mdx b/models/training.mdx index cda18d6..028a77b 100644 --- a/models/training.mdx +++ b/models/training.mdx @@ -27,13 +27,15 @@ Independent train and eval data gives meaningful held-out scores. Overmind repor Training wizard with dataset setup and model picker +Over MCP, `start_finetune` derives model-specific defaults when `hyperparameters` is omitted. To set the method yourself, pass `"training_type": {"type": "Lora"}` or `{"type": "Full"}`; a plain string is refused before the job is created, with a field-level error. + ## Exact preprocessing Workshop quality reviews remain advisory and are not repeated as launch approval gates. Training performs model-specific preparation, not repairs to labels, evidence or task scope. For Modal, CPU preprocessing runs after launch and before GPU training using the matching tokenizer, chat template and loss mask. Setup does not start or wait for preprocessing. Recommendations size context from estimated row lengths; job preparation includes pinned training and enabled validation data. Exact overflow within the model's training-method limit requests a larger context and rechecks every row. Over-limit rows, invalid tool formatting and examples without supervised next-token targets still block. Nothing is silently truncated or discarded. -The GPU consumes the matching validated token artifact and saved tokenizer. Repair incompatible data in the workshop and prepare the revised version. Explicit REST `POST /api/training-preparations/` and MCP `prepare_training_data` remain available; poll with `get_job(kind="training_preparation")`. Other providers retain provider-side preprocessing. +The GPU consumes the matching validated token artifact and saved tokenizer. Repair incompatible data in the workshop and prepare the revised version. Explicit REST `POST /api/training-preparations/` and MCP `prepare_training_data` run the same preparation; poll with `get_job(kind="training_preparation")`. Worker failures are distinct from incompatible data. A processor/code mismatch requires deploying the current Modal SFT worker, not just restarting Docker. Preprocessing logs remain at `preparations//preprocess_stdout.log` on the SFT volume. Confirmed failures can retry; uncertain submissions fail closed. @@ -47,7 +49,7 @@ Serving context is planned separately from training sequence length, using the p The wizard shows the constraints for each model: -- **Context.** Training limits can be shorter than inference windows and differ between LoRA and full training. Modal uses estimates to size context, not to exclude candidates; exact preparation determines compatibility. Other providers retain their existing eligibility checks. +- **Context.** Training limits can be shorter than inference windows and differ between LoRA and full training. Estimates size context but do not exclude candidates; exact preparation determines compatibility. - **Tool calling.** If your data contains tool calls, only models that support them are offered. - **Adapters.** Larger dense models and every mixture-of-experts model train as LoRA adapters; the picker says so instead of offering a choice. @@ -84,7 +86,7 @@ Every selected evaluation uses the entire pinned eval version, the same prompt s **Judge model**, below **Eval set**, defaults to the set's per-evaluator choices. An explicit selection is saved for this run and used on both sides of the comparison without changing the set. REST/MCP use `eval_judge_model`; it cannot change after launch. The dropdown labels estimated context fit and judge-budget differences. Estimated limit overflows turn affected selectors and **Start** orange; undersized options are greyed out but remain selectable. Clicking **Start** opens a short risk summary with **Start anyway**, without requiring a model change. Unverified, unavailable and pending checks stay neutral and do not open this warning. Shared incumbent evaluations require the same judge selection as well as matching data and benchmark context. -Supported untouched bases use OpenRouter when configured and exactly matched in its catalog; otherwise they use the existing provider/Modal route. Once an eval or baseline deployment starts, its route stays pinned until an explicit retry. Trained checkpoints use their own deployment. +Supported untouched bases use OpenRouter when exactly matched in its catalog; otherwise they use a Modal deployment. Once an eval or baseline deployment starts, its route stays pinned until an explicit retry. Trained checkpoints use their own deployment. The Evals table shows selected evaluations immediately, including what each is waiting for, and keeps updating after deployment. Evaluations that never start are skipped if training fails or is cancelled. When experiments share an incumbent-before evaluation, the group view shows that run once as **Shared · N experiments**. Each experiment still has its own base-model and trained-model evaluations, and its individual view retains the shared incumbent result and comparison. diff --git a/platform/administration.mdx b/platform/administration.mdx index ebb9f75..47a3590 100644 --- a/platform/administration.mdx +++ b/platform/administration.mdx @@ -17,7 +17,7 @@ One key type serves everything programmatic: the tracing SDK, raw OTLP ingest, t - **Format**: `ovr_` followed by a random token. Only a SHA-256 hash is stored — the full key is **shown once, at creation**; the table keeps the first 12 characters as `prefix`. - **Headers**: `X-Api-Key: `, `Authorization: Bearer `, or `Authorization: Api-Key `. -- **Scope**: **project-scoped** keys see one project; **account-scoped** keys reach every project you are a member of and are the only keys that can create a project. MCP and `.overmind/credentials.toml` hold a project-scoped key; the onboarding prompt carries a temporary account-scoped key that `overmind sync` trades for a project one. +- **Scope**: **project-scoped** keys see one project; **account-scoped** keys reach every project you are a member of and are the only keys that can create a project. `overmind sync` stores a project-scoped key in `.overmind/credentials.toml` and the MCP entry; the onboarding prompt carries a temporary account-scoped key that `overmind sync` trades for a project one. - **Lifetime**: keys do not expire unless an `expires_at` is set on the row. Revocation is immediate, and each key records `last_used_at`. - **Where**: **Projects → your project → API keys** in the Console, or [over the API](/platform/api#authentication). @@ -25,7 +25,7 @@ One key type serves everything programmatic: the tracing SDK, raw OTLP ingest, t Project page with the API keys table -Scope one key per environment and keep them out of source control. `overmind sync` writes the project key to `.overmind/credentials.toml`, and `overmind init` writes it into the IDE's MCP entry; both refuse a path git already tracks, add the file to `.git/info/exclude` (not `.gitignore`), and set mode `0600`. +Scope one key per environment and keep them out of source control. `overmind sync` writes the project key to `.overmind/credentials.toml`, and `overmind init` writes it into the IDE's MCP entry; both refuse a path git already tracks, add the file to `.git/info/exclude` (not `.gitignore`), and set mode `0600`. Claude Code's MCP entry lives in `~/.claude.json`, outside the repository. ## Tracing connectors diff --git a/platform/api.mdx b/platform/api.mdx index 7255f73..dbd03f7 100644 --- a/platform/api.mdx +++ b/platform/api.mdx @@ -19,10 +19,10 @@ Three ways to authenticate; the first is right for almost all programmatic use: | Method | How | Use for | | --- | --- | --- | | **API key** | `X-Api-Key: ovr_...`, `Authorization: Bearer ovr_...`, or `Authorization: Api-Key ovr_...` | SDKs, OTLP ingest, scripts, CI, the CLI, MCP, inference, `overmind sync` | -| **JWT** | `POST /api/auth/token/` with `email` and `password` → access token (1 day) + refresh token (7 days, `POST /api/auth/token/refresh/`) | Accounts created with `POST /api/auth/register/`; guest sessions | +| **JWT** | `POST /api/auth/token/` with `email` and `password` → access token (1 day) + refresh token (7 days, `POST /api/auth/token/refresh/`) | Self-hosted accounts (`POST /api/auth/local/` signs in and creates the account on first use); guest sessions | | **Clerk session** | `Authorization: Bearer ` | The Console | -A key has a **scope**: `project` pins it to one project, `account` reaches every project the user is a member of. MCP and `.overmind/credentials.toml` use a project-scoped key; of the two, only an account-scoped key can `POST /api/projects/`. Keys are created and revoked under **Projects → your project → API keys** or via the API: +A key has a **scope**: `project` pins it to one project, `account` reaches every project the user is a member of. `.overmind/credentials.toml` holds a project-scoped key; only an account-scoped key can `POST /api/projects/`. Keys are created and revoked under **Projects → your project → API keys** or via the API: ```bash # Create (the key is returned once), inspect the calling key, list, revoke @@ -86,7 +86,7 @@ Standard REST resources follow Django REST Framework conventions: `GET` list wit | `GET /api/models/catalog/` | The OpenRouter model catalog used for judges and run variants | | `GET /api/public/models/`, `GET /api/public/models/{slug}/` | The public model library, unauthenticated | | `GET /health` | Health check (no auth) | -| `/api/auth/register/`, `/api/auth/me/`, `/api/auth/onboarding/`, `POST /api/auth/guest/`, `POST /api/auth/guest/claim/` | Account, onboarding state, guest workspaces | +| `POST /api/auth/local/`, `/api/auth/me/`, `/api/auth/onboarding/`, `POST /api/auth/guest/`, `POST /api/auth/guest/claim/` | Self-hosted sign-in, account, onboarding state, guest workspaces | | `/api/billing/subscription/`, `ledger/`, `checkout/`, `topup/`, `cancel/`, `renew/`, `webhook/` | Plans, the credits ledger, Stripe | ## Building a dataset diff --git a/platform/cli.mdx b/platform/cli.mdx index f95336b..84723b9 100644 --- a/platform/cli.mdx +++ b/platform/cli.mdx @@ -3,30 +3,85 @@ title: CLI description: The overmind command — init, sync, chassis, dataset upload and export, connectors, the optimise loop, model checkpoints, and skills. --- -The `overmind` command ships with the Python package. It is the local half of every workflow that touches your filesystem: it installs the skill and MCP entry, syncs `overmind.toml` with the server, lands files as datasets, adds tracing connectors, drives optimisation runs in your working tree, and downloads weights. The MCP server never edits a repository or moves bytes; the CLI does. The package is MIT. +The `overmind` command ships with the Python package. It is the local half of every workflow that touches your filesystem: it installs the workflow skills and MCP entry, syncs `overmind.toml` with the server, lands files as datasets, adds tracing connectors, drives optimisation runs in your working tree, and downloads weights. The MCP server never edits a repository or moves bytes; the CLI does. The package is MIT. -```bash -pip install overmind # the CLI and the inference client -pip install "overmind[tracing]" # plus the tracing SDK and every provider instrumentor -``` +## Install + +Install the CLI into a virtual environment, or as a standalone tool with `uv` or `pipx`. Both put `overmind` on `PATH`. + + + + ```bash + python3 -m venv .venv + source .venv/bin/activate + pip install overmind + overmind --help + ``` + + As a standalone tool, outside any project: + + ```bash + uv tool install overmind # or: pipx install overmind + ``` + + Homebrew's Python refuses a global `pip install` with `externally-managed-environment`. Use a virtual environment or a tool install. + + + ```bash + python3 -m venv .venv + source .venv/bin/activate + pip install overmind + overmind --help + ``` + + As a standalone tool, outside any project: + + ```bash + uv tool install overmind # or: pipx install overmind + ``` + + Debian and Ubuntu mark the system Python as externally managed, so a global `pip install` fails with `externally-managed-environment`. Use a virtual environment or a tool install. + + + Install into a virtual environment. It puts `overmind` on `PATH` while it is active: + + ```powershell + python -m venv .venv + .venv\Scripts\Activate.ps1 # cmd.exe: .venv\Scripts\activate.bat + pip install overmind + overmind --help + ``` + + If PowerShell refuses to run `Activate.ps1`, allow local scripts for your user once: + + ```powershell + Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser + ``` + + A global `pip install` puts `overmind.exe` in the interpreter's `Scripts` folder, which is not on `PATH` by default. The command is then not found; `python -m overmind` runs the same CLI. + + + +The base package is the CLI and the inference client. The `tracing` extra adds the tracing SDK and every provider instrumentor: ```bash -uv tool install overmind -# or -pipx install overmind +pip install "overmind[tracing]" ``` +When Overmind is a project dependency, run commands through that project's environment (`uv run overmind`, `poetry run overmind`, or its virtual-environment executable). An editable installation does not update skill copies already in a repository. Refresh the installed skill with `uv run overmind skills sync overmind --ide ` using the appropriate project runner, then start a new conversation so the coding agent reads it. + +## First run + Every command that talks to the API resolves its key in the same order: `--api-key`, then the project credential in `.overmind/credentials.toml` (written by `overmind sync`), then `OVERMIND_API_KEY`. The base URL comes from `--api-url`, `OVERMIND_API_URL`, or `base-url` in `overmind.toml`; the hosted default is `https://api.overmindlab.ai`. Keys go over the wire as `X-Api-Key`. `overmind --version` prints the package version; `overmind --help` prints flags. ```bash -export OVERMIND_API_KEY= +export OVERMIND_API_KEY= # PowerShell: $env:OVERMIND_API_KEY = "" +export OVERMIND_API_URL= # self-hosted only, e.g. http://localhost:8000 overmind init --ide cursor # or claude | opencode | codex overmind sync # create the project, save the project key ``` -Then in your coding agent, `/overmind setup` scans the repository, writes capabilities into `overmind.toml`, and runs `overmind sync` to push them. The installed skill recognises the repository and verified connection, previews the onboarding roadmap, and reports repository-specific findings alongside the current stage out of ten, completed stages, and stages still to follow. A pre-sync handoff names the destination, included data, credential checks performed, and configuration to review. The finish reports the synced result, outstanding verification, and one recommended next action without automatically starting it. There is no `overmind scan` command. - -When Overmind is a project dependency, run commands through that project's environment (`uv run overmind`, `poetry run overmind`, or its virtual-environment executable). An editable installation does not update skill copies already in a repository. Refresh the installed skill with `uv run overmind skills sync overmind --ide ` using the appropriate project runner, then start a new conversation so the coding agent reads it. +Then in your coding agent, `/overmind setup` scans the repository, writes capabilities into `overmind.toml`, and runs `overmind sync` to push them. The installed skill recognises the repository and verified connection, previews the onboarding roadmap, and reports repository-specific findings alongside the current stage out of ten, completed stages, and stages still to follow. A pre-sync handoff names the destination, included data, credential checks performed, and configuration to review. The finish reports the synced result, outstanding verification, and one recommended next action without automatically starting it. ## `overmind init` @@ -41,18 +96,18 @@ overmind init --ide claude --env local | `--env` | `production` | `production`, `staging`, or `local` (`http://localhost:8000`) | | `--api-url` | `$OVERMIND_API_URL` | Base URL; overrides `--env` for both the MCP entry and `overmind.toml` | -Installs the `overmind` skill under the client's directory (`.cursor/skills/`, `.claude/skills/`, `.opencode/skills/`, `.agents/skills/`), writes the `/overmind` slash commands (Cursor and Claude Code only: `.cursor/commands/` or `.claude/commands/`), adds the Overmind MCP entry to the client's own config file, and seeds `overmind.toml` when it is missing. Other configured MCP servers are left untouched. An existing `overmind.toml` is left unchanged. +Installs the workflow skills under the client's directory (`.cursor/skills/`, `.claude/skills/`, `.opencode/skills/`, `.agents/skills/`), writes the `/overmind` slash commands (Cursor only: `.cursor/commands/`), adds the Overmind MCP entry to the client's own config, and seeds `overmind.toml` when it is missing. Other configured MCP servers are left untouched. An existing `overmind.toml` is left unchanged. | Client | MCP config | Skill install | | --- | --- | --- | | Cursor | `.cursor/mcp.json` | `.cursor/skills` | -| Claude Code | `.mcp.json` | `.claude/skills` | +| Claude Code | `~/.claude.json`, local scope for this directory | `.claude/skills` | | OpenCode | `opencode.json` | `.opencode/skills` | | Codex | `.codex/config.toml` | `.agents/skills` | -The MCP entry is written with the saved project key when `.overmind/credentials.toml` already exists; otherwise it is written without a key and `overmind sync` fills it in. Any file that receives a key is added to `.git/info/exclude` and set to mode `0600`; the CLI refuses to write a key into a tracked file. Codex is limited to `production` and `staging` unless `--api-url` is set. `init` does not persist the bootstrap account key. +The MCP entry is written with the saved project key when `.overmind/credentials.toml` already exists; otherwise it is written without a key and `overmind sync` fills it in. Claude Code's entry is a local-scope server in `~/.claude.json` (or `$CLAUDE_CONFIG_DIR/.claude.json`), so it needs no `.mcp.json` approval and a committed `.mcp.json` is left untouched. Any repository file that receives a key is added to `.git/info/exclude` and set to mode `0600`; the CLI refuses to write a key into a tracked file. `init` does not persist the bootstrap account key. -Slash commands written for Cursor and Claude Code: `/overmind`, `/overmind onboard`, `/overmind setup`, `/overmind ensure-tracing`, `/overmind dataset`, `/overmind finetune`, `/overmind optimise`, `/overmind backtest`. +Slash commands: `/overmind`, `/overmind onboard`, `/overmind setup`, `/overmind ensure-tracing`, `/overmind dataset`, `/overmind finetune`, `/overmind optimise`, `/overmind backtest`. Cursor gets a command file for each; Claude Code passes the word after `/overmind` to the skill, which opens the matching reference. ## `overmind sync` @@ -90,7 +145,7 @@ Skipped directories: `.git`, `.venv`, `venv`, `node_modules`, `__pycache__`, `.t ## `overmind dataset` ```bash -overmind dataset upload rows.jsonl --intent eval --json +overmind dataset upload rows.jsonl --intent eval --wait --json overmind dataset upload traffic.parquet --split 20 --split-position random overmind dataset export --format csv --cell --output triage.csv ``` @@ -104,7 +159,7 @@ overmind dataset export --format csv --cell --output tria | `--capability` | none | Capability UUID to align the dataset with | | `--split` | off | Land a train dataset and an eval dataset; the eval share in percent (1–99). Drop `--intent`. | | `--split-position` | `tail` | Where the eval rows come from: `head`, `tail`, or `random` | -| `--wait` | off | Wait for landing to finish; exit 1 if it fails | +| `--wait` | off | Wait for the landing and the agent's first preparation; exit 1 with the dataset's error if either fails | | `--api-key`, `--api-url`, `--path` | resolved | Credentials; `--path` points at `overmind.toml` | | `--json` | off | Print one machine-readable object | @@ -209,7 +264,3 @@ overmind skills sync overmind --ide cursor ## JSON output `--json` on `dataset upload`, `dataset export`, `connector add` and `model download-checkpoint` prints one machine-readable object and nothing else; on failure they print `{"error": "..."}` and exit 1. `optimise status --json` prints the experiment. This is how the `/overmind` commands hand a local step to the CLI and read the result back. - -## Not a command - -There is no `overmind scan`. Discovery is `/overmind setup` in your coding agent: it runs `overmind chassis`, authors capability cards into `overmind.toml`, then `overmind sync` uploads them. See [Agent & Capabilities](/core/capabilities#the-scan). diff --git a/platform/glossary.mdx b/platform/glossary.mdx index 350104e..54545d7 100644 --- a/platform/glossary.mdx +++ b/platform/glossary.mdx @@ -17,9 +17,9 @@ description: Definitions of terms used across Overmind, as they appear in the Co **Scan snapshot** — the server's record of one `overmind sync`: the toml capabilities, cards, and repo summary. `GET /api/v1/sync?project_id=` reads it back. -**Cluster** — one group of functions reachable from an entry point with the same prompt set; the unit the coding agent names as a capability. +**Tool cluster** — a group of a capability's tools that serve similar tasks, named on the card; the trajectory map renders each cluster together. -**Mode** — one distinct model call inside a capability, with its own system prompt: the roles inside an orchestration. Each mode becomes one behaviour on sync. +**Mode** — one task or role inside a capability, with its own prompt: a stage of a pipeline or a worker in an orchestration. A trajectory-map entry for a mode shares its name, so the behaviour minted from it carries that identity. **Card** — the agent-authored description of a capability in `overmind.toml` (`[capabilities..capability_card]`): task, modality, input schema, output fields, expected output, success criteria, failure modes, constraints, tool protocol, and a trajectory map of anchors — each with source provenance. Filled by `/overmind setup` from source, merged onto the capability and its behaviours on `overmind sync`. @@ -27,15 +27,15 @@ description: Definitions of terms used across Overmind, as they appear in the Co **API key** — an `ovr_...` credential (stored hashed, shown once at creation) that authenticates the SDK, OTLP ingest, the REST API, MCP, the CLI, and inference. Sent as `X-Api-Key` or `Authorization: Bearer`. **Project-scoped** keys see one project; **account-scoped** keys reach every project the user is in. -**MCP** — the project-scoped Model Context Protocol server at `/api/mcp/`: the agent surface a coding agent uses for every workflow that does not need the filesystem. See [MCP](/platform/mcp). +**MCP** — the Model Context Protocol server at `/api/mcp/`: the agent surface a coding agent uses for every workflow that does not need the filesystem. See [MCP](/platform/mcp). -**Skill** — the `overmind` agent skill `overmind init` installs, with the `/overmind` commands and one reference per workflow. See [CLI](/platform/cli). +**Skill** — the agent skills `overmind init` installs: the main `overmind` skill, with the `/overmind` commands and one reference per workflow, and focused workflow skills. See [CLI](/platform/cli). ## Observability **Trace** — one end-to-end run of an agent: the set of spans sharing a `trace_id`. There is no separate trace record; the root span (the one with no parent) represents the trace in lists. -**Span** — one operation inside a trace, stored with its full OTLP payload (timing, status, attributes, events) plus platform fields computed at ingest. Spans are typed by `overmind.span_type`: `entry_point`, `workflow`, `tool_call`, `function`, `retrieval`, or `llm_call`; an untyped span is classified from its attributes and name and defaults to `llm_call`. +**Span** — one operation inside a trace, stored with its full OTLP payload (timing, status, attributes, events) plus platform fields computed at ingest. Spans are typed by `overmind.span.type`: `entry_point`, `workflow`, `tool_call`, `function`, `retrieval`, or `llm_call`; an untyped span is classified from its attributes and name and defaults to `llm_call`. **Session** — traces grouped by a shared `conversation.id` attribute, representing one multi-turn exchange. Set via `set_conversation_id()` or `run(conversation_id=...)` in the SDK. Carries a **session score** folded from the conversation ledger. See [Trace scoring](/agent-testing/trace-scoring#session-scores). @@ -53,7 +53,7 @@ description: Definitions of terms used across Overmind, as they appear in the Co **Task execution** — one unit bound to a behaviour, with its verdicts, execution score, route flags, and session score. Binding source is `declared` (the span carried the identity), `anchor_join` (matched on anchors) or `unbound`. Status `completed`, `error` or `interrupted`. The default row of Observability. -**Instrumentation plan** — the placement tickets (file, qualname, decorator, scope) the `get_instrumentation_plan` MCP tool returns for a capability or behaviour; `verify_instrumentation` grades a supplied span list against them. +**Instrumentation plan** — the placement tickets (file, qualname, decorator, scope) the `get_instrumentation_plan` MCP tool returns for a capability or behaviour; `verify_instrumentation` grades an ingested trace, or a supplied span list, against them. ## Data @@ -111,7 +111,7 @@ description: Definitions of terms used across Overmind, as they appear in the Co **Training run** — one or more parallel experiments over a training dataset, each producing a candidate model (API resource: `finetuning-jobs`). See [Models](/models/training). -**Baseline (training)** — the capability's production model, scored on the eval dataset at submit time; the reference every trained model's delta is measured against. +**Benchmark model (training)** — the model a training run compares against: the codebase incumbent or a ready trained model, chosen per run. The headline delta uses the untouched base before training when that evaluation is selected, otherwise the benchmark. **Deployed model** — a served trained model with a copyable reference (`ft-...`) callable via the OpenAI-compatible API. Status: `queued`, `deploying`, `warming`, `ready`, `failed`, `deleting`, `deleted`. See [Inference](/models/inference). diff --git a/platform/mcp.mdx b/platform/mcp.mdx index 72cd1d1..d085e95 100644 --- a/platform/mcp.mdx +++ b/platform/mcp.mdx @@ -1,9 +1,9 @@ --- title: MCP -description: The Overmind MCP server — setup per coding agent, authentication, the 32 tools, 11 prompts and the resources, and the CLI handoff for local work. +description: The Overmind MCP server — setup per coding agent, the optional plugin, authentication, the 36 tools, 12 prompts, resources, workflow skills and the CLI handoff for local work. --- -MCP is the platform's first-class agent surface: a project-scoped [Model Context Protocol](https://modelcontextprotocol.io) server at `/api/mcp/` that shares its services with the REST API and the Console. A coding agent connected to it can inspect health and failures, land and shape datasets, author evaluators and run evals, launch and follow training, schedule optimisation runs, and drive connectors and instrumentation — everything except the steps that need your filesystem, which it hands to the [CLI](/platform/cli). +MCP is the platform's first-class agent surface: an account-aware [Model Context Protocol](https://modelcontextprotocol.io) server at `/api/mcp/` that shares its services with the REST API and the Console. A coding agent connected to it can inspect health and failures, land and shape datasets, author evaluators and run evals, launch and follow training, schedule optimisation runs, and drive connectors and instrumentation — everything except the steps that need your filesystem, which it hands to the [CLI](/platform/cli). ## Setup @@ -22,18 +22,6 @@ MCP is the platform's first-class agent surface: a project-scoped [Model Context } ``` -```json Claude Code (.mcp.json) -{ - "mcpServers": { - "overmind": { - "type": "http", - "url": "https://api.overmindlab.ai/api/mcp/", - "headers": { "X-Api-Key": "ovr_…" } - } - } -} -``` - ```json OpenCode (opencode.json) { "$schema": "https://opencode.ai/config.json", @@ -48,25 +36,49 @@ MCP is the platform's first-class agent surface: a project-scoped [Model Context } ``` -```bash Claude Code (one line) -claude mcp add --transport http overmind https://api.overmindlab.ai/api/mcp/ --header "X-Api-Key: ovr_…" +```bash Claude Code +claude mcp add --scope local --transport http overmind https://api.overmindlab.ai/api/mcp/ --header "X-Api-Key: ovr_…" ``` -Codex takes the same entry in `.codex/config.toml`. Self-hosted: replace the URL with `http://localhost:8000/api/mcp/` (`overmind init --env local`). +Codex takes the same entry in `.codex/config.toml`. Claude Code keeps the key in local scope (`~/.claude.json`), outside the repository. Self-hosted: replace the URL with your API host plus `/api/mcp/`, for example `http://localhost:8000/api/mcp/` (`overmind init --env local`, or `--api-url`). + +## Optional plugin + +The Cursor, Claude Code and Codex plugins bundle the MCP connection, Overmind branding and the [workflow skills](#workflow-skills). MCP alone is the complete integration: tool descriptions, operating guidance, resources and prompts reach every client, with or without a plugin. + +The plugin's `.mcp.json` points at the hosted API and signs in with OAuth. The connection reaches every active project your account can access, including projects you join later. For a self-hosted deployment, run `overmind init --env local --ide ` and `overmind sync` instead. Keep one active Overmind MCP connection per client. + +Start with `list_projects`. To see a project in the browser, read `overmind://project/current?project_id=ID` and open its `console_url`, which the server builds from `FRONTEND_URL`. The browser needs its own Console sign-in, and opening a page grants no write access. ## Authentication -The key must be **project-scoped** — pinned to exactly one active project the key's owner is a member of — and carry `read` and/or `write` permission. The project is bound from the key; no tool accepts a project id. A key with only `read` sees the read-only tools; the rest need `write`. A key with an IP allowlist is refused from other addresses. `Authorization: Bearer ovr_…` is accepted as an alternative to `X-Api-Key`. +**Account connections.** OAuth connections and account API keys find the projects you can access with `list_projects`. Pass the chosen `project_id` to every project tool, and as a query parameter on resource URIs. Membership is checked on every operation, and there is no shared "current project", so parallel work cannot switch another request's project. Returned resource links keep the project id. + +**Project API keys** reach only their own project and can omit `project_id`. + +**Permissions.** Read access exposes the read-only tools; writes need write access. API keys keep their expiry, active-user and IP-allowlist checks, and authenticate as `X-Api-Key` or `Authorization: Bearer`. -The server answers `401` for authentication errors and `403` otherwise, without a `WWW-Authenticate: Bearer` challenge, so MCP clients never start OAuth registration against it. Requests must be `POST` with `application/json`, from an allowed host and origin, and at most the protocol's default body size. Errors are JSON-RPC values, never exceptions: `{"error": {code, message, retryable, fields}}`. +**OAuth.** A grant asks for `overmind:read` and/or `overmind:write`, and needs Console sign-in and explicit consent. Tokens use S256 PKCE, exact redirect matching, one-time codes and resource binding, and only MCP accepts them. Access tokens expire after one hour. A connection stays authorized until it is revoked: refresh tokens have no time limit, but they rotate, and reusing an old one revokes the connection. A disabled account or a lost project membership ends access. + +**Hosting.** On a public deployment, set `MCP_SERVER_URL` to the HTTPS `/api/mcp/` endpoint and `FRONTEND_URL` to the Console. Plain HTTP is accepted only on local loopback. The server then advertises OAuth discovery, public-client registration (`token_endpoint_auth_method: none`), and the authorization, token and revocation endpoints. To publish the server as an OpenAI app, set `OPENAI_APPS_CHALLENGE` to the portal's ownership token; the API serves it at `/.well-known/openai-apps-challenge`. + +**Transport.** Requests are `POST` with `application/json`, from an allowed host and origin, within the protocol's body limit. Authentication failures return `401` and authorization failures `403`. Tool errors are structured results: `{"error": {code, message, retryable, fields}}`. The public surface is read and write only. There are no delete, cancel or generic API tools; `retry_deployment` is the one lifecycle exception and covers exactly the failed-or-deleted deployment case. ## Tools -Thirty-two tools, all project-scoped, each with a strict input and output schema the client reads at call time. Every result carries the same object twice — as `structuredContent` and as compact JSON text — plus resource links for the entities it touched. Arguments marked `?` are optional. +Thirty-six tools, each with a strict input and output schema the client reads at call time. Every result carries the same object twice — as `structuredContent` and as compact JSON text — plus resource links for the entities it touched. Arguments marked `?` are optional. + +Project tools also accept `project_id`, required for account connections. `list_projects` and `get_model_catalog` do not need a project. + +### Projects + +| Tool | Arguments | Returns | +| --- | --- | --- | +| `list_projects` | `limit?`, `offset?` | Authorized active projects, Console/resource links and pagination | ### Observability @@ -85,7 +97,7 @@ Thirty-two tools, all project-scoped, each with a strict input and output schema | `list_datasets` | `capability?`, `intent?`, `state?`, `search?`, `limit?`, `offset?` | List project datasets | | `inspect_dataset` | `dataset`, `chat_limit?` | One dataset: cell chain, sample, recent agent chat, next actions | | `query_dataset` | `dataset`, `sql`, `cell?`, `limit?` | Read-only SQL over one ran cell | -| `create_dataset_from_traces` | `name`, `trace_ids?`, `filters?`, `search?`, `limit?`, `intent?`, `capability?` | Land a dataset, one row per trace; the selection is counted first and refused when empty. Returns a task receipt | +| `create_dataset_from_traces` | `name`, `trace_ids?`, `filters?`, `search?`, `limit?`, `intent?`, `capability?`, `split?` | Land a dataset, one row per trace; the selection is counted first and refused when empty. `split` lands disjoint ` train` and ` eval` datasets instead of one with an intent. Returns a task receipt | | `message_dataset_agent` | `dataset`, `message` | Queue one agent turn for an idle dataset. Returns a job receipt | | `run_dataset` | `dataset`, `proposal_cell?` | Run the dataset, optionally accepting one proposal. Returns a job receipt | @@ -93,10 +105,10 @@ Thirty-two tools, all project-scoped, each with a strict input and output schema | Tool | Arguments | Effect | | --- | --- | --- | -| `check_evaluation_readiness` | `dataset`, `eval_set?`, `cell?`, `version?`, `mode?` | Dataset, active eval set, evaluator applicability, variable bindings, credit readiness | +| `check_evaluation_readiness` | `dataset`, `eval_set?`, `cell?`, `version?`, `mode?`, `variants?`, `judge_model?` | Dataset, active eval set, evaluator applicability, variable bindings, credit readiness | | `create_eval_set` | `name`, `evaluator_ids`, `capability?` | Atomically create an eval set and its members in their applicable roles; does not activate it or launch a run | | `upsert_evaluator` | `evaluator?`, `name?`, `kind?`, `capability?`, `scope?`, `score_type?`, `display_name?`, `description?`, `rubric_md?`, `evaluation_prompt?`, `checklist?`, `judge_model?`, `score_min?`, `score_max?`, `pass_threshold?`, `requires_reference?`, `variable_mapping?`, `config?`, `applicable_roles?`, `surface?`, `choices?` | Create or update an evaluator | -| `run_evaluation` | `name`, `dataset`, `eval_set?`, `cell?`, `version?`, `evaluator_ids?`, `variants?`, `max_items?`, `sampling?` | Create and launch an eval run. Returns a job receipt | +| `run_evaluation` | `name`, `dataset`, `eval_set?`, `cell?`, `version?`, `evaluator_ids?`, `variants?`, `max_items?`, `sampling?`, `judge_model?` | Create and launch an eval run. Returns a job receipt | | `compare_evaluations` | `run`, `baseline` | Compare a run with a project-scoped baseline | | `annotate_evaluation_sample` | `annotation?`, `sample?`, `evaluator?`, `value?`, `label?`, `note?` | Create or update a human annotation, attributed to the key's owner | @@ -107,9 +119,11 @@ Thirty-two tools, all project-scoped, each with a strict input and output schema | `get_model_catalog` | `has_tool_calling?`, `max_context?` | Models available for fine-tuning: tiers, context limits, training methods, tool-calling, batch bounds | | `check_finetune_readiness` | `dataset`, `capability?`, `cell?`, `version?` | Prerequisites, train/eval dataset shape, recommended models, evaluators, credits | | `estimate_finetune` | `dataset`, `base_model`, `n_epochs?`, `use_lora?`, `cell?`, `version?` | Real-token cost and duration estimate; creates nothing | -| `start_finetune` | `dataset`, `base_model`, `capability?`, `eval_dataset?`, `eval_set?`, `validation_dataset?`, `cell?`, `version?`, `eval_cell?`, `eval_version?`, `validation_cell?`, `validation_version?`, `validation_enabled?`, `validation_split_ratio?`, `split_method?`, `name?`, `use_case?`, `hyperparameters?`, `group_id?` | Validate and queue a training job. Returns a job receipt | +| `start_finetune` | `dataset`, `base_model`, `capability?`, `eval_dataset?`, `eval_set?`, `validation_dataset?`, `cell?`, `version?`, `eval_cell?`, `eval_version?`, `validation_cell?`, `validation_version?`, `validation_enabled?`, `validation_split_ratio?`, `split_method?`, `name?`, `use_case?`, `hyperparameters?`, `group_id?`, `baseline_model?`, `eval_judge_model?`, `eval_incumbent_before?`, `eval_incumbent_after?`, `eval_model_before?`, `eval_model_after?` | Validate and queue a training job. `baseline_model` overrides the capability's benchmark for this job; the `eval_*` flags choose which comparison evaluations run. Returns a job receipt | +| `prepare_training_data` | `dataset`, `base_model`, `context_length`, `training_type?`, `validation_dataset?`, `cell?`, `retry_failed?` | Run exact tokenization for a model and context length; returns a cached preparation report. Poll with `get_job(kind="training_preparation")` | | `retry_deployment` | `deployment` | Retry a failed or deleted deployment. Returns a job receipt | | `set_active_model` | `capability`, `deployment?` | Verify a ready deployment, then switch the alias; poll the returned `model_activation` job. Omit deployment to clear routing | +| `set_benchmark_model` | `capability`, `deployment?` | Choose a ready trained deployment as the capability's default training benchmark; omit deployment to use the codebase incumbent. Does not change serving | | `run_inference` | `deployment`, `messages`, `temperature?`, `max_tokens?` | One chat completion against a ready deployment using production output budgets, with usage, latency and finish reason | | `get_model_swap_prompt` | `finetune`, `pin?` | A copy-paste prompt that points the capability's code at a successful fine-tune; the agent applies it locally | @@ -128,27 +142,27 @@ Thirty-two tools, all project-scoped, each with a strict input and output schema | Tool | Arguments | Effect | | --- | --- | --- | | `inspect_connectors` | `connector?`, `include_source_projects?`, `preview?`, `preview_source_project_id?`, `preview_lookback_days?`, `max_runs?` | List or inspect connectors: configuration, mappings, provider capabilities, sync status | -| `configure_connector` | `connector`, `source_project_id?`, `lookback_days?`, `backfill_from?`, `backfill_to?`, `capability_mapping?`, `auto_sync_enabled?`, `poll_interval_seconds?` | Save source project, lookback, capability mapping and polling settings | +| `configure_connector` | `connector`, `source_project_id?`, `lookback_days?`, `backfill_from?`, `backfill_to?`, `capability_mapping?`, `auto_sync_enabled?`, `poll_interval_seconds?`, `confirm_mapping?` | Save source project, lookback, capability mapping and polling settings; `confirm_mapping` confirms the reviewed mapping | | `sync_connector` | `connector` | Queue the configured import. Returns a task receipt | -A connector is created, with its provider credentials, in the Console under Integrations; until that form is complete the tools answer with the `connector_setup_required` action. +A connector is created with its provider credentials through `overmind connector add` or the Console under Integrations; until then the tools answer with the `connector_setup_required` action. ### Instrumentation | Tool | Arguments | Returns | | --- | --- | --- | | `get_instrumentation_plan` | `capability?`, `behaviour?` | The registered placement tickets for a capability or behaviour | -| `verify_instrumentation` | `capability?`, `spans` | Bind and grade a caller-supplied span list without writing traces or scores | +| `verify_instrumentation` | `capability?`, `trace_id` or `spans` | Bind and grade an ingested trace by id, or a caller-supplied span list, without writing traces or scores | ### Jobs -Background work returns a **job receipt** — `kind`, `id`, `status` and an `overmind://jobs/{kind}/{id}` resource — that `get_job` reads back. Kinds: `dataset_run`, `connector_sync`, `eval_run`, `finetune` (alias `finetune_job`), `optimizer` (aliases `optimizer_run`, `optimizer_experiment`) and `deployment`. +Background work returns a **job receipt** — `kind`, `id`, `status` and an `overmind://jobs/{kind}/{id}` resource — that `get_job` reads back. Kinds: `dataset_run`, `connector_sync`, `eval_run`, `training_preparation`, `finetune` (alias `finetune_job`), `optimizer` (aliases `optimizer_run`, `optimizer_experiment`), `deployment` and `model_activation`. Names resolve within the project: capability arguments take an id, key or name; connectors take an id or name; dataset tools prefer the UUID `list_datasets` returned because dataset names are not unique. ## Prompts -Eleven prompts compose the tools into multi-step workflows with human checkpoints: +Twelve prompts compose the tools into multi-step workflows with human checkpoints: | Prompt | Arguments | What it drives | | --- | --- | --- | @@ -163,14 +177,15 @@ Eleven prompts compose the tools into multi-step workflows with human checkpoint | `upload-dataset-file` | `path`, `intent?`, `project_id?` | `overmind dataset upload` in the terminal, then verify through MCP | | `export-dataset` | `dataset` | `overmind dataset export` to the local filesystem | | `download-checkpoint` | `deployment` | `overmind model download-checkpoint` to the local filesystem | +| `connect-traces` | `connector_type` | `overmind connector add` in the terminal, then configure, confirm the mapping and sync through MCP | ## Resources -Read-only, project-scoped entity state under `overmind://`. Static: +Read-only project entity state under `overmind://`. Account connections add `?project_id=ID`; follow returned links to preserve scope. Static: | URI | Content | | --- | --- | -| `overmind://project/current` | The authenticated project identity and safe metadata | +| `overmind://project/current` | The authenticated project identity, provenance, safe metadata and ordinary `console_url` | | `overmind://dataset-upload` | CLI guidance for uploading local dataset bytes | | `overmind://dataset-export` | CLI guidance for downloading a committed dataset | | `overmind://checkpoint-download` | CLI guidance for downloading an archived checkpoint | @@ -186,6 +201,21 @@ Each deployment resource also includes a current `worker` section: `state` (`war Three things never cross MCP: file bytes, repository edits, and provider secrets. A dataset file goes up with `overmind dataset upload`; a dataset or checkpoint comes down with `overmind dataset export` and `overmind model download-checkpoint`; repository discovery is `/overmind setup` then `overmind sync`; an optimisation run's candidates are generated and replayed by `overmind optimise` in your checkout; a model swap is a prompt (`get_model_swap_prompt`) the agent applies with its own editing tools; a connector's provider key is entered with `overmind connector add` in a terminal. The prompts above name the exact command at each handoff, and `--json` on the CLI returns the ids the agent needs to continue. -## The skill +## Workflow skills + +`overmind init` installs the workflow skills next to the MCP entry, and the optional plugin bundles the same files. The main `overmind` skill covers setup, repository discovery and work that spans several surfaces. Each focused skill covers one surface: -`overmind init` installs the `overmind` skill next to the MCP entry, and on Cursor and Claude Code the `/overmind` slash commands (`onboard`, `setup`, `ensure-tracing`, `dataset`, `finetune`, `optimise`, `backtest`) that load its references. The skill holds the conventions the tools assume — list first, poll jobs with `get_job`, the dataset contracts that gate every run — and is the only place the local scan workflow (`/overmind setup`) lives; the native prompts cover the server-side workflows. +| Skill | Workflow | +| --- | --- | +| `overmind-agent` | Capability map, behaviour coverage and repository provenance | +| `overmind-observability` | Trace investigations, failures, latency and instrumentation gaps | +| `overmind-datasets` | Data Workshop preparation, proposals, generation and export | +| `overmind-evaluations` | Evaluators, eval sets, runs and baseline comparisons | +| `overmind-optimiser` | Prompt and code experiments, and model comparisons | +| `overmind-training` | Exact data preparation, costs, fine-tuning and held-out evaluation | +| `overmind-inference` | Serving metrics, worker state, live routing and activation | +| `overmind-integrations` | Tracing providers, boundary mappings and verified imports | + +`overmind skills list --verbose` prints the installed catalog. `overmind skills sync --ide ` refreshes the named skills, and `overmind init --ide ` installs the full set. + +`/overmind onboard`, `setup`, `ensure-tracing`, `dataset`, `finetune`, `optimise` and `backtest` load the main skill's references: Cursor through command files, Claude Code through the skill itself. In OpenCode and Codex, ask for the step by name. The main skill is the only place the local scan (`/overmind setup`) lives; the native prompts cover the server-side workflows. diff --git a/platform/self-hosting.mdx b/platform/self-hosting.mdx index 2fff428..0aa7a20 100644 --- a/platform/self-hosting.mdx +++ b/platform/self-hosting.mdx @@ -7,35 +7,46 @@ The platform is one repository, [overmind-core/overmind](https://github.com/over ## Requirements -Docker with Compose, [Bun](https://bun.sh/) for the frontend, and[uv](https://docs.astral.sh/uv/) for anything Python outside the containers. -> The API refuses to start without a training backend's credentials (Modal or Baseten), an S3 credential pair for the checkpoint archive, and `INFERENCE_API_URL` — see [Training and serving](#training-and-serving). +Docker with Compose, and [uv](https://docs.astral.sh/uv/) for anything Python outside the containers. + +The API refuses to start until every required key is set; the error names each missing one. + +| Required | Used for | +| --- | --- | +| `OPENROUTER_API_KEY` | Judges, evals, trace scoring, the Data Workshop, optimiser scoring, frontier inference | +| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | Modal training and serving workers | +| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | Serving trained models: the endpoint printed by `modal deploy` and its shared secret | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_BUCKET_NAME` | The fine-tuning checkpoint archive | +| `HF_TOKEN` | Gated Hugging Face base models | + +| Optional | Effect | +| --- | --- | +| `CURSOR_API_KEY` | Recommended for the Data Workshop: runs the dataset agent as a Cursor Composer session | +| `OPENAI_API_KEY` | The `embedding_cosine` evaluator | +| `ANTHROPIC_API_KEY`, `GEMINI_API_KEY` | Data Workshop fallbacks; unused while `OPENROUTER_API_KEY` is set | +| `STRIPE_SECRET_KEY` | Paid plans and the credit cap — see [Billing modes](#billing-modes) | + +Fine-tuning and serving also need the Modal workers deployed — see [Training and serving](#training-and-serving). ## Start the stack ```bash git clone https://github.com/overmind-core/overmind.git cd overmind -cp .env.example .env # fill in OPENROUTER_API_KEY, the training backend, AWS keys, INFERENCE_API_URL -docker compose up -d # API, Postgres, Redis, four Celery workers, beat, Grafana +cp .env.example .env # set the required keys above +docker compose up -d # API, Console, Postgres, Redis, four Celery workers, beat, Grafana ``` -The Console is a React SPA served separately using Bun: +The Console is at [http://localhost:5173](http://localhost:5173) and the API at `http://localhost:8000`. Sign in with any email and password; self-hosted mode creates the account on first use and needs no email verification. Leave `CLERK_API_SECRET_KEY` blank. -```bash -cd frontend && \ -cp .env.example .env && \ -bun install && bun run dev # http://localhost:5173 -``` +Start by creating a project by following the [Quickstart](/quickstart). The onboarding prompt in your Console already points the coding agent at your API. -You can now access the Console at [http://localhost:5173](http://localhost:5173). -Use any login credentials to sign in, self hosted mode doesn't require any email verification. +To connect a repository by hand instead, run `overmind init --ide --env local`, then `overmind sync` with the temporary account key from the Console. `--env local` points the MCP entry at `http://localhost:8000/api/mcp/`. Sync saves a project-scoped key; reload the coding agent once to connect. For another self-hosted address, set `OVERMIND_API_URL` before both commands. -Start by creating a project by following the [Quickstart](/quickstart). - -For a demo workspace with thirty days of traffic, datasets, runs and served models, run the seed: +For a demo workspace with thirty days of traffic, datasets, runs and served models, run the seed with the email you sign in with: ```bash -docker compose exec -T api python manage.py shell < seed.py +docker compose exec api python manage.py seed_demo --owner you@example.com ``` ## Environment reference @@ -53,12 +64,12 @@ docker compose exec -T api python manage.py shell < seed.py | `FIELD_ENCRYPTION_KEY` | Fernet key for stored connector credentials. Unset, one is derived from `DJANGO_SECRET_KEY`; the API prints a warning at startup when `DJANGO_DEBUG=False` | | `FRONTEND_URL` | Where the Console lives (default `http://localhost:5173`), for links in prompts and emails | | `TRACE_SETTLE_SECONDS` | How long a trace with no root span stays `live` before it reads `interrupted` (default `600`) | -| `OPENROUTER_API_KEY` | Every routed model call: judges, eval-run generation, the optimiser's scoring, frontier inference | +| `OPENROUTER_API_KEY` | Every routed model call: judges, eval-run generation, the optimiser's scoring, frontier inference and the Data Workshop agent. Required at startup. If the key is rejected, the capability's Evaluators tab shows `Judge authoring failed`; fix the key, then run `overmind sync` again | ### The Data Workshop agent -The dataset agent runs on the first key it finds, in this order: `CURSOR_API_KEY` (a Cursor Composer session), `OPENROUTER_API_KEY`, then `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` with a native tool-calling loop. Both engines take the same tools and emit the same events. With none set the workshop page opens and the agent reports that it is not configured. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. +The dataset agent runs on OpenRouter. We recommend setting `CURSOR_API_KEY`, which runs it as a Cursor Composer session instead. Both engines take the same tools and emit the same events. `OPENAI_API_KEY` also serves embeddings (`text-embedding-3-small`) for the `embedding_cosine` statistical evaluator. ### Decision engine @@ -72,12 +83,14 @@ capacity falls back to generative judging; Jev is not a chat or inference model. | Variable | Purpose | | --- | --- | -| `FINETUNING_BACKEND` | `modal` (settings default); `docker-compose.yml` and `.env.example`. Startup fails unless the chosen backend's credentials are set | -| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `MODAL_ENVIRONMENT` | Modal training and serving workers, deployed with `modal deploy overbae/modal/modal_vllm_worker.py`, `register_model.py` and `modal_sft_worker.py` (`--env $MODAL_ENVIRONMENT`). `MODAL_ENVIRONMENT` (default `overmind-dev`) also selects the checkpoint bucket | +| `FINETUNING_BACKEND` | `modal`, the only accepted value | +| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, `MODAL_ENVIRONMENT` | Required (tokens). Modal training and serving workers, deployed with `modal deploy overbae/modal/modal_vllm_worker.py`, `register_model.py` and `modal_sft_worker.py` (`--env $MODAL_ENVIRONMENT`). `MODAL_ENVIRONMENT` (default `overmind-dev`) also selects the checkpoint bucket | | `MODAL_SFT_APP_NAME` | The deployed SFT app (default `overmind-sft`) | -| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | The Modal serving endpoint printed by `modal deploy`, and the shared secret it checks. `INFERENCE_API_URL` is required at startup | -| `HF_TOKEN` | Gated base models on Hugging Face | -| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` | The checkpoint archive that backs weights download; the key pair is required at startup | +| `INFERENCE_API_URL`, `INFERENCE_API_KEY` | Required. The Modal serving endpoint printed by `modal deploy`, and the shared secret it checks | +| `HF_TOKEN` | Required. Gated base models on Hugging Face | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_BUCKET_NAME`, `AWS_REGION` | Required (all but the region). The checkpoint archive that backs weights download | + +The Modal workers read their credentials from a Modal secret named `overmind-inference`: the AWS keys, `INFERENCE_API_KEY` and `HF_TOKEN`. ### Billing modes diff --git a/quickstart.mdx b/quickstart.mdx index dad0538..3a751e7 100644 --- a/quickstart.mdx +++ b/quickstart.mdx @@ -1,47 +1,46 @@ --- title: Quickstart -description: The easiest way to get started with Overmind. +description: Sign up or self-host, paste one prompt into your coding agent, then improve the agent with /overmind commands. --- -Overmind has first class integration with cursor, claude code, opencode, and codex through our SKILL `/overmind` and MCP server, after signing up you can pretty much do everything from your coding agent. - -> We've built the experince to be as easy and native to your coding agent as possible. +Overmind onboards through your coding agent. Cursor, Claude Code, OpenCode and Codex all work: one pasted prompt installs the SDK, connects the MCP server and scans your repository into capabilities. After that, instrumentation, datasets, training and optimisation run from the same chat. - - Start by signing up at [console.overmindlab.ai](https://console.overmindlab.ai) + + Sign up at [console.overmindlab.ai](https://console.overmindlab.ai), or open your own Console, for example `http://localhost:5173` — see [Self-hosting](/platform/self-hosting). A **project** holds your capabilities, traces, datasets, runs, models and keys; the first sync creates it. - Pick your coding agent — **Cursor**, **Claude Code**, **OpenCode**, or **Codex** — and hit **Copy onboarding prompt**. + On the **Agent** page, pick **Cursor**, **Claude Code**, **OpenCode** or **Codex**, then select **Copy onboarding prompt**. The prompt carries a temporary account key, so paste it only into your local coding agent. **Manual setup** shows the same commands for a terminal. - - Paste the prompt into your coding agent (repo root as the workspace folder) and hit enter. - - Overmind will now scan your repository for [AI Agents & Capabilities](/core/capabilities#the-scan), you can see them in the [Console](https://console.overmindlab.ai) - - If you want any changes in how they are grouped, evals, etc, just ask your coding agent to do it. - - Next step is to ensure your codebase is instrumented properly for [tracing using OTLP endpoint](/core/observability#the-otlp-endpoint), just run `/overmind ensure-tracing` in your coding agent. + + Open the repository root as the workspace and paste the prompt. The agent installs `overmind`, runs `overmind init` and `overmind sync`, and saves a project-scoped key to `.overmind/credentials.toml`. Reload the agent once so it loads the Overmind MCP server; in Claude Code, exit and run `claude -c`. + The agent then scans the repository into [capabilities](/core/capabilities#the-scan) and syncs them to the Console. To regroup capabilities or change their evals, ask the agent. - The pushed capabilities are the starting point. From the same chat, each command drives one loop end to end: + Each command drives one loop end to end. Cursor and Claude Code take them as slash commands; in OpenCode and Codex, ask for the step by name. | Command | What it does | | --- | --- | - | `/overmind ensure-tracing` | Inspect codebase and instrument it properly for tracing | - | `/overmind dataset` | Build, clean, upload, or export a dataset | - | `/overmind finetune` | Fine-tune, deploy, and smoke-test a model | + | `/overmind ensure-tracing` | Instrument the code and check that traces arrive | + | `/overmind dataset` | Build, clean, upload or export a dataset | + | `/overmind finetune` | Fine-tune, deploy and smoke-test a model | | `/overmind optimise` | Run prompt and code optimisation | | `/overmind backtest` | Compare models in the repository | + Plain `/overmind` routes to the right workflow when you describe the goal instead. - Once tracing is in, open **Observability**. + Once traces arrive, open **Observability**. **Task executions** shows one row per scored unit of work, **Root traces** one row per trace, and **Sessions** groups traces by session. - + Observability task executions view in the Overmind console +Deployed processes need `OVERMIND_API_KEY` in their runtime configuration; the saved project key covers local runs only. Not using Python or a coding agent? Any OpenTelemetry SDK can export to the [OTLP endpoint](/core/observability#the-otlp-endpoint). + ## Next steps diff --git a/tracing/sdk-python.mdx b/tracing/sdk-python.mdx index 82f0ba2..913bc7a 100644 --- a/tracing/sdk-python.mdx +++ b/tracing/sdk-python.mdx @@ -10,7 +10,7 @@ pip install "overmind[tracing]" # the tracing SDK, every provider instrumentor pip install overmind # the `overmind` CLI and the inference client only ``` -The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. There are no other extras. +The base package is the [CLI](/platform/cli) and the OpenAI-compatible [inference client](/models/inference#calling-your-model). Tracing is the one extra: on a bare install, accessing `overmind.init` or any tracing name raises `ImportError` with the install line to run. There are no other extras. Install steps for macOS, Linux and Windows are on the [CLI page](/platform/cli#install). ## `init()` @@ -164,7 +164,7 @@ Two identity forms tie code to a capability declared in `overmind.toml`: - `overmind.init(capability_id="")` (or `run(capability_id=...)`) — ingest binds by the capability's UUID; a slug resolves server-side. - `@overmind.capability("", id="...")` on an entry function — at runtime the name is a label only; pass `id=` or set `capability_id` in `init()` to bind. -Verification is a bounded, read-only MCP smoke: run with a unique `conversation_id`, flush, find the trace with `query_traces(session=..., all_spans=true)`, read `overmind://traces/{trace_id}`, and hand its spans to `verify_instrumentation`, which checks capability attribution, nesting, inputs and outputs, usage, and timing without writing traces or scores. +Verification is a bounded, read-only MCP smoke: run with a unique `conversation_id`, flush, find the trace with `query_traces(session=..., all_spans=false)`, and pass its `trace_id` to `verify_instrumentation`, which grades the ingested spans on the server and checks capability attribution, nesting, inputs and outputs, usage, and timing without writing traces or scores. ### `task()` — carve a run into units @@ -205,7 +205,7 @@ for i, doc in enumerate(documents): ### LangChain / LangGraph -`providers=["langchain"]` (or `"auto"`) mounts the OpenInference LangChain instrumentor that ships with `overmind[tracing]`, which covers LangGraph: every chain, model, and tool invocation gets a span with model, token, and cost evidence. For the scoring semantics no instrumentor can know, `overmind.integrations.langgraph.bind` maps graph nodes to behaviour units — call it on the `StateGraph` after `add_node()`, before `compile()`: +`providers=["langchain"]` (or `"auto"`) mounts the OpenInference LangChain instrumentor that ships with `overmind[tracing]`, which covers LangGraph: every chain, model, and tool invocation gets a span with model and token evidence, and Overmind prices it at ingest. For the scoring semantics no instrumentor can know, `overmind.integrations.langgraph.bind` maps graph nodes to behaviour units — call it on the `StateGraph` after `add_node()`, before `compile()`: ```python from overmind.integrations import langgraph as overmind_langgraph diff --git a/tracing/sdk-typescript.mdx b/tracing/sdk-typescript.mdx index 409cc31..b155934 100644 --- a/tracing/sdk-typescript.mdx +++ b/tracing/sdk-typescript.mdx @@ -4,7 +4,7 @@ description: There is no TypeScript tracing SDK; instrument Node.js with OpenTel --- -No TypeScript SDK ships with the current platform. The `@overmind-lab/trace-sdk` package on npm predates the current ingest contract — it does not stamp the attributes below. Do not install it. +There is no TypeScript SDK. Instrument Node.js with OpenTelemetry directly. Node.js services trace like any other OpenTelemetry client: