From 2ecc2845d30ce8122e7e837bd0bbc1942aac35ff Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 16 Jul 2026 18:31:07 +0000 Subject: [PATCH] docs(agenteye): sync from FailproofAI/agenteye@465793a --- docs/agenteye/alerts.mdx | 6 +- docs/agenteye/api-keys.mdx | 36 +- docs/agenteye/assistant.mdx | 10 +- docs/agenteye/audits.mdx | 4 +- docs/agenteye/cli-and-agents.mdx | 6 +- docs/agenteye/cli-recipes.mdx | 8 +- docs/agenteye/cli-skill.mdx | 18 +- docs/agenteye/cli.mdx | 16 +- docs/agenteye/collector-installation.mdx | 464 ------- docs/agenteye/collector-migration.mdx | 179 --- docs/agenteye/concepts.mdx | 19 +- docs/agenteye/dashboards.mdx | 2 +- docs/agenteye/deployment-options.mdx | 58 - docs/agenteye/deployment.mdx | 460 ------- docs/agenteye/error-tracking.mdx | 10 +- docs/agenteye/evaluation-suite.mdx | 157 +-- docs/agenteye/evaluations.mdx | 10 +- docs/agenteye/event-stream.mdx | 4 +- docs/agenteye/faq.mdx | 53 - docs/agenteye/getting-started.mdx | 292 ----- docs/agenteye/github-token.mdx | 152 --- docs/agenteye/health-monitoring.mdx | 186 --- docs/agenteye/incidents.mdx | 4 +- docs/agenteye/kubernetes-deployment.mdx | 1429 ---------------------- docs/agenteye/managed-deployment.mdx | 204 --- docs/agenteye/observability.mdx | 4 +- docs/agenteye/overview.mdx | 53 +- docs/agenteye/python-sdk.mdx | 54 +- docs/agenteye/queries.mdx | 6 +- docs/agenteye/security.mdx | 24 +- docs/agenteye/sessions.mdx | 6 +- docs/agenteye/single-pod-deployment.mdx | 455 ------- docs/agenteye/telemetry.mdx | 4 +- docs/agenteye/tenant-management.mdx | 179 --- docs/agenteye/troubleshooting.mdx | 741 ----------- 35 files changed, 165 insertions(+), 5148 deletions(-) delete mode 100644 docs/agenteye/collector-installation.mdx delete mode 100644 docs/agenteye/collector-migration.mdx delete mode 100644 docs/agenteye/deployment-options.mdx delete mode 100644 docs/agenteye/deployment.mdx delete mode 100644 docs/agenteye/faq.mdx delete mode 100644 docs/agenteye/getting-started.mdx delete mode 100644 docs/agenteye/github-token.mdx delete mode 100644 docs/agenteye/health-monitoring.mdx delete mode 100644 docs/agenteye/kubernetes-deployment.mdx delete mode 100644 docs/agenteye/managed-deployment.mdx delete mode 100644 docs/agenteye/single-pod-deployment.mdx delete mode 100644 docs/agenteye/tenant-management.mdx delete mode 100644 docs/agenteye/troubleshooting.mdx diff --git a/docs/agenteye/alerts.mdx b/docs/agenteye/alerts.mdx index e06ee05b2..92779080d 100644 --- a/docs/agenteye/alerts.mdx +++ b/docs/agenteye/alerts.mdx @@ -1,10 +1,10 @@ --- title: "Alerts" -description: "AgentEye Alerts documentation." +description: "FailproofAI Observability Alerts documentation." --- -Find out the moment something crosses your line, on the channel your team already watches, instead of hearing about it from a customer. Set a rule once and AgentEye checks it on a schedule, then pages you by email, Slack, webhook, or right in the dashboard. +Find out the moment something crosses your line, on the channel your team already watches, instead of hearing about it from a customer. Set a rule once and FailproofAI Observability checks it on a schedule, then pages you by email, Slack, webhook, or right in the dashboard. ![The Alerts page: a grid of alert-rule cards, each showing its trigger, evaluation window, channels, and an info, warning, or critical severity badge](/agenteye/images/alerts.png) *Every alert rule at a glance: what it watches, how often, where it pages, and how urgent.* @@ -22,7 +22,7 @@ Attach any combination to a single rule, and its severity (info, warning, or cri ## Build the rule in a form, not JSON -You describe what "broken" means in a form, and AgentEye writes the underlying rule for you. The JSON spec is just what that form produces under the hood, so you can read it to understand a rule but you rarely type it. +You describe what "broken" means in a form, and FailproofAI Observability writes the underlying rule for you. The JSON spec is just what that form produces under the hood, so you can read it to understand a rule but you rarely type it. ![The new-alert form: name and description, an enabled toggle, and a trigger picker offering metric threshold, custom SQL, evaluation score, compound eval, and per-event conditions](/agenteye/images/alert-new.png) *Pick a trigger and the form swaps in the right fields; Save writes the rule.* diff --git a/docs/agenteye/api-keys.mdx b/docs/agenteye/api-keys.mdx index 753e1eeb5..774978f0d 100644 --- a/docs/agenteye/api-keys.mdx +++ b/docs/agenteye/api-keys.mdx @@ -1,18 +1,18 @@ --- title: "API Keys" -description: "AgentEye API Keys documentation." +description: "FailproofAI Observability API Keys documentation." --- -API keys control who and what can reach your AgentEye server, so a collector can send events without ever gaining read or admin powers. Each key carries one or more permissions, and each permission gates specific server routes; you grant only the few a job needs. Most deployments create just three kinds of key. +API keys control who and what can reach your FailproofAI Observability server, so a collector can send events without ever gaining read or admin powers. Each key carries one or more permissions, and each permission gates specific server routes; you grant only the few a job needs. Most deployments create just three kinds of key. ## The 3 keys most deployments need | Key | Permissions | Who uses it | |---|---|---| -| Collector key | `events:add` | The `agenteye-collector` on each agent machine, to send events. See [Collector installation](/agenteye/collector-installation). | +| Collector key | `events:add` | The `agenteye-collector` on each agent machine, to send events. | | Dashboard read key | `events:read`, `keys:read` | A read-only operator or integration that queries data without changing it. | -| Bootstrap admin key | all permissions | The operator who first brings the instance up (and the dashboard container). Seeded from the `ADMIN_KEY` environment variable. See [Bootstrap admin key](#bootstrap-admin-key). | +| Bootstrap admin key | all permissions | The operator who first brings the instance up (and the dashboard). Seeded from the `ADMIN_KEY` environment variable. See [Bootstrap admin key](#bootstrap-admin-key). | Start here. Reach for the full permission catalogue below only when you need a narrower, custom-scoped key. See also [Recommended key layout](#recommended-key-layout) and [Creating keys](#creating-keys). @@ -50,7 +50,7 @@ The server enforces a fixed catalogue of permissions; each one gates specific HT | Permission | HTTP routes | What it allows | |---|---|---| -| `queries:read` | `GET /queries`, `GET /queries/:id`, `GET /queries/schema` | List saved queries, load one, and inspect the read-only ClickHouse schema the composer targets. | +| `queries:read` | `GET /queries`, `GET /queries/:id`, `GET /queries/schema` | List saved queries, load one, and inspect the read-only schema the composer targets. | | `queries:write` | `POST /queries`, `PUT /queries/:id` | Create and edit saved queries. SQL is still routed through the same read-only role and guarded SQL checks as a `queries:run` call. | | `queries:delete` | `DELETE /queries/:id` | Delete a saved query. | | `queries:run` | `POST /queries/run` | Execute saved or ad-hoc SQL against the read-only role used by the composer. | @@ -138,8 +138,8 @@ Sets are surfaced in the dashboard and managed over the API at `GET /permission- Set membership is what backs two other features: -- **`DEFAULT_USER_PERMISSIONS`** (the grant preselected when an admin opens **+ new user**) defaults to the `standard` set. See [Deployment](/agenteye/deployment). -- **The `--set` flag** on `agenteye-orgctl` (operator member management) starts a member from a named set, which you then fine-tune with `--add` / `--remove`. See [Tenant Management](/agenteye/tenant-management). +- **`DEFAULT_USER_PERMISSIONS`** (the grant preselected when an admin opens **+ new user**) defaults to the `standard` set. +- **The `--set` flag** on `agenteye-orgctl` (operator member management) starts a member from a named set, which you then fine-tune with `--add` / `--remove`. > **Note:** When a set includes a permission that is not key-assignable (for example a custom set carrying `keys:update`), seeding a key from that set drops the non-assignable tokens; the server would otherwise reject the key with HTTP 422. Dashboard users are not subject to that restriction. @@ -157,9 +157,9 @@ To rotate: change `ADMIN_KEY` to a new secret and restart the server. ## Organization scoping -**Organizations themselves are created and managed out-of-band by an operator, not through this keys API.** Org and member lifecycle (create / rename / delete / purge an org; add / update / remove a member) is done with the **`agenteye-orgctl`** CLI that runs inside the server pod; there is no HTTP API or dashboard button for it. See [Tenant Management](/agenteye/tenant-management). What *is* unchanged: **per-org API keys are still minted in the dashboard (or via this keys API)** by org members. +**Organizations themselves are created and managed out-of-band by an operator, not through this keys API.** Org and member lifecycle (create / rename / delete / purge an org; add / update / remove a member) is done with the **`agenteye-orgctl`** CLI; there is no HTTP API or dashboard button for it. What *is* unchanged: **per-org API keys are still minted in the dashboard (or via this keys API)** by org members. -In a multi-org deployment, every key an org member creates (through this keys API or the dashboard **Keys** page) belongs to **one organization** and can only ever read or write that org's data; the org is stamped on the key at creation and enforced on every request. The two bootstrap keys are the only exception: the `admin` key (seeded from `ADMIN_KEY`) and the `dashboard-assistant` key (seeded from `AGENT_API_KEY`) are **instance-scoped** (they carry no org). The dashboard container authenticates with the `admin` key so it can proxy per-org requests on behalf of signed-in members. Single-tenant deployments need not think about this; all keys belong to the built-in `default` org. +In a multi-org deployment, every key an org member creates (through this keys API or the dashboard **Keys** page) belongs to **one organization** and can only ever read or write that org's data; the org is stamped on the key at creation and enforced on every request. The two bootstrap keys are the only exception: the `admin` key (seeded from `ADMIN_KEY`) and the `dashboard-assistant` key (seeded from `AGENT_API_KEY`) are **instance-scoped** (they carry no org). The dashboard authenticates with the `admin` key so it can proxy per-org requests on behalf of signed-in members. Single-tenant deployments need not think about this; all keys belong to the built-in `default` org. --- @@ -170,7 +170,7 @@ Use the admin key (or any key with `keys:create` permission) to create additiona ### Collector key (ingest only) ```bash -curl -s -X POST http://your-server:8080/keys \ +curl -s -X POST http://your-server/keys \ -H "Authorization: Bearer $ADMIN_KEY" \ -H "Content-Type: application/json" \ -d '{ @@ -183,7 +183,7 @@ curl -s -X POST http://your-server:8080/keys \ ### Dashboard key (read only) ```bash -curl -s -X POST http://your-server:8080/keys \ +curl -s -X POST http://your-server/keys \ -H "Authorization: Bearer $ADMIN_KEY" \ -H "Content-Type: application/json" \ -d '{ @@ -209,7 +209,7 @@ When you create a key over the HTTP API, you provide the `key` value yourself; c ## Listing Keys ```bash -curl -s http://your-server:8080/keys \ +curl -s http://your-server/keys \ -H "Authorization: Bearer $ADMIN_KEY" ``` @@ -222,7 +222,7 @@ Key secrets are not returned in list responses, only IDs, names, and permissions Disabling revokes access immediately without deleting the key record. ```bash -curl -s -X POST http://your-server:8080/keys//disable \ +curl -s -X POST http://your-server/keys//disable \ -H "Authorization: Bearer $ADMIN_KEY" ``` @@ -233,7 +233,7 @@ curl -s -X POST http://your-server:8080/keys//disable \ Generates a new secret for an existing key. The old secret is invalidated immediately. ```bash -curl -s -X POST http://your-server:8080/keys//regenerate \ +curl -s -X POST http://your-server/keys//regenerate \ -H "Authorization: Bearer $ADMIN_KEY" ``` @@ -255,9 +255,9 @@ When you create a key from the dashboard you do not supply the secret; the dashb | Key | Permissions | Used by | |---|---|---| -| `admin` (bootstrap via `ADMIN_KEY` env var) | all | Ops/setup, and the dashboard container (authenticates with `ADMIN_KEY`, proxies user requests with permission checks) | +| `admin` (bootstrap via `ADMIN_KEY` env var) | all | Ops/setup, and the dashboard (authenticates with `ADMIN_KEY`, proxies user requests with permission checks) | | Per-host collector key | `events:add` | Collector on each agent machine | -| `dashboard-assistant` (bootstrap via `AGENT_API_KEY` env var) | `events:read`, `evaluations:read`, `dashboards:read`, `dashboards:write`, `queries:read`, `queries:write`, `queries:run` | AI assistant container, seeded automatically, **protected**; can't be edited through the API | +| `dashboard-assistant` (bootstrap via `AGENT_API_KEY` env var) | `events:read`, `evaluations:read`, `dashboards:read`, `dashboards:write`, `queries:read`, `queries:write`, `queries:run` | AI assistant, seeded automatically, **protected**; can't be edited through the API | | Assistant telemetry key (optional) | `events:add` | AI assistant self-instrumentation, if enabled | > **Note:** The assistant's key is **seeded automatically** by the server from the `AGENT_API_KEY` env var (the same secret the agent presents as `AGENTEYE_API_KEY`); there is no manual key-minting step and no admin key involved. Its permissions are fixed in source code so scope can't be widened by misconfiguration: read across events / evaluations / dashboards, plus dashboards-write and queries-read / write / run for the "Ask AI to write a query" authoring flow. All SQL still goes through the same read-only role and guarded SQL path as a user-written query, so this widens the *authoring surface*, not the data surface; destructive operations (`queries:delete`, `dashboards:delete`) deliberately stay off the assistant key. Like the `admin` key, it is **protected**: it can't be disabled or regenerated through the keys API, only rotated by changing `AGENT_API_KEY` and restarting. Dashboard *users* additionally need the `agent:use` permission to see and use the assistant. If you enable self-instrumentation, give the assistant a separate `events:add`-only key. @@ -276,7 +276,5 @@ You only need these if you are upgrading an existing instance; new deployments c ## Next steps -- [Collector installation](/agenteye/collector-installation): where the `events:add` collector key is configured. - [Python SDK](/agenteye/python-sdk): how your agent code authenticates when sending events. -- [Tenant management](/agenteye/tenant-management): create and manage organizations and their members. -- [Deployment](/agenteye/deployment): set `ADMIN_KEY`, `AGENT_API_KEY`, and the default permission set. +- [Security](/agenteye/security): how sign-in, access control, and per-organization data isolation work. diff --git a/docs/agenteye/assistant.mdx b/docs/agenteye/assistant.mdx index d16f4d8c7..09b5f2e9f 100644 --- a/docs/agenteye/assistant.mdx +++ b/docs/agenteye/assistant.mdx @@ -1,12 +1,12 @@ --- title: "AI Assistant" -description: "AgentEye AI Assistant documentation." +description: "FailproofAI Observability AI Assistant documentation." --- -Ask your agent data a question in plain English and get an answer that links straight to the evidence. No SQL to write, no dashboards to dig through: it is the fastest way for anyone on your team to get answers about your agents. +Ask your agent data a question in plain English and get an answer that links straight to the evidence. No SQL to write, no dashboards to dig through — the **FailproofAI Observability** assistant is the fastest way for anyone on your team to get answers about your agents. -![The AgentEye AI assistant answering a plain-English question inside the dashboard, showing a live Agent Activity table, a per-agent model-usage breakdown, and written takeaways, with the queries it ran shown inline](/agenteye/images/assistant.png) +![The FailproofAI Observability assistant answering a plain-English question inside the dashboard, showing a live Agent Activity table, a per-agent model-usage breakdown, and written takeaways, with the queries it ran shown inline](/agenteye/images/assistant.png) *Ask in plain English and get an answer built from your own data. Here it breaks down which agents are busiest and which models they use, and shows the queries it ran so you can verify every number.* There is nothing to learn. Open the chat, type what you want to know, and follow the links it hands back: @@ -36,7 +36,7 @@ When an answer is worth keeping, ask the assistant to save it. It drafts the SQL On the **Queries** page it goes a step further and becomes a SQL author: describe the query you want ("show error rate by agent for the last 7 days") and it streams SQL straight into the editor, opening a diff view so you can **Accept** or **Reject** the change before it lands. -![The AgentEye Queries page and its SQL editor](/agenteye/images/query-lab.png) +![The Observability Queries page and its SQL editor](/agenteye/images/query-lab.png) *The Queries page: this editor is where the assistant streams a draft, read-only query for you to accept or reject.* Authoring SQL by asking here uses the `queries:run` permission, the same one behind the editor's **Run** button. Chat everywhere else needs `agent:use`. @@ -49,7 +49,7 @@ You can open the assistant up to everyone without worrying about what it might t - **Every write waits for you.** Saved queries and dashboards are created only after your explicit Approve click, and there is no setting that turns that gate off. - **It can never delete anything.** No delete tool is exposed and the assistant holds no delete permission. Deletions stay in your hands, in the dashboard. - **It stays inside your org.** The assistant only ever sees the organization you are currently viewing. -- **Your questions stay yours.** Prompts and answers live in your own AgentEye database; product analytics records usage metadata only, never your prompt text. +- **Your questions stay yours.** Prompts and answers live in your own Observability database; product analytics records usage metadata only, never your prompt text. ## Where to find it diff --git a/docs/agenteye/audits.mdx b/docs/agenteye/audits.mdx index ad3346910..10ccff829 100644 --- a/docs/agenteye/audits.mdx +++ b/docs/agenteye/audits.mdx @@ -1,10 +1,10 @@ --- title: "Audits: your automatic reliability analyst" -description: "AgentEye Audits: your automatic reliability analyst documentation." +description: "FailproofAI Observability Audits: your automatic reliability analyst documentation." --- -AgentEye goes looking for the failures you never wrote a rule for and hands you a ranked, evidence-backed to-do list of exactly what to fix. It is like having an analyst comb your logs every night, then leaving the short list on your desk by morning. +FailproofAI Observability goes looking for the failures you never wrote a rule for and hands you a ranked, evidence-backed to-do list of exactly what to fix. It is like having an analyst comb your logs every night, then leaving the short list on your desk by morning.
diff --git a/docs/agenteye/cli-and-agents.mdx b/docs/agenteye/cli-and-agents.mdx index 430a778f1..8af4742c9 100644 --- a/docs/agenteye/cli-and-agents.mdx +++ b/docs/agenteye/cli-and-agents.mdx @@ -1,10 +1,10 @@ --- title: "CLI" -description: "AgentEye CLI documentation." +description: "FailproofAI Observability CLI documentation." --- -Your entire AgentEye deployment, one command away. Check production, cut an API key, or ack an incident without leaving your terminal, then script any of it into CI, or let a coding agent do it for you in plain English. +Your entire FailproofAI Observability deployment, one command away. Check production, cut an API key, or ack an incident without leaving your terminal, then script any of it into CI, or let a coding agent do it for you in plain English. ```bash pipx install agenteye @@ -47,7 +47,7 @@ It is built to run unattended. Confirmation prompts auto-skip when no terminal i Better yet, you should not have to remember any of these flags at all. The **CLI skill** is a small Agent Skill folder named `agenteye-cli` that teaches a coding agent such as Claude Code or Codex to drive the CLI from plain-English requests. Ask "is anything broken today?" and the agent picks the command, runs it as you, and answers in prose. -For Claude Code, drop the `agenteye-cli` folder into `~/.claude/skills/` and it is auto-discovered. AgentEye provides the folder; there is nothing to deploy and no image to pull, because it only drives the CLI you already installed. Log in yourself first: the skill cannot complete the emailed-code login for you. +For Claude Code, drop the `agenteye-cli` folder into `~/.claude/skills/` and it is auto-discovered. FailproofAI Observability provides the folder; there is nothing extra to install, because it only drives the CLI you already installed. Log in yourself first: the skill cannot complete the emailed-code login for you. Because the agent runs the CLI as you, it can do everything your login permits, reads and writes alike: create keys, change settings, resolve incidents. The CLI's "are you sure?" prompt does not fire for an agent, so the skill is written to state the exact command and wait for your OK before any change. You are the confirmation step. diff --git a/docs/agenteye/cli-recipes.mdx b/docs/agenteye/cli-recipes.mdx index 57a00d714..fc6c9ad16 100644 --- a/docs/agenteye/cli-recipes.mdx +++ b/docs/agenteye/cli-recipes.mdx @@ -1,12 +1,12 @@ --- title: "CLI recipes for agents" -description: "AgentEye CLI recipes for agents documentation." +description: "FailproofAI Observability CLI recipes for agents documentation." --- -Pull session, event, and evaluation data (and trigger re-evaluations) straight from a script or coding agent, with clean JSON on stdout that pipes directly into `jq`. These recipes turn AgentEye's observability data into something a terminal user or an AI coding agent (Claude Code, Cursor) can query and automate, without clicking through the dashboard. +Pull session, event, and evaluation data (and trigger re-evaluations) straight from a script or coding agent, with clean JSON on stdout that pipes directly into `jq`. These recipes turn FailproofAI Observability's data into something a terminal user or an AI coding agent (Claude Code, Cursor) can query and automate, without clicking through the dashboard. -The patterns below are copy-paste ready for the AgentEye CLI (`agenteye`). For installation, authentication, and the full option list see [CLI](/agenteye/cli); run `agenteye -h` or `agenteye -h` for the built-in help. +The patterns below are copy-paste ready for the FailproofAI Observability CLI (`agenteye`). For installation, authentication, and the full option list see [CLI](/agenteye/cli); run `agenteye -h` or `agenteye -h` for the built-in help. ## Golden rules @@ -176,4 +176,4 @@ Each command's `--fields` accepts exactly its own item's field names. The set di - [CLI](/agenteye/cli): installation, authentication, and the full option reference for every command. - [CLI agent skill](/agenteye/cli-skill): package these recipes as a skill your coding agent can load. - [API keys](/agenteye/api-keys): create and scope the keys the CLI, SDK, and collector authenticate with. -- [Python SDK](/agenteye/python-sdk): send events into AgentEye so there is data for these recipes to query. +- [Python SDK](/agenteye/python-sdk): send events into FailproofAI Observability so there is data for these recipes to query. diff --git a/docs/agenteye/cli-skill.mdx b/docs/agenteye/cli-skill.mdx index 71aaa6701..c36ffff3f 100644 --- a/docs/agenteye/cli-skill.mdx +++ b/docs/agenteye/cli-skill.mdx @@ -1,25 +1,25 @@ --- -title: "AgentEye CLI Agent Skill" -description: "AgentEye AgentEye CLI Agent Skill documentation." +title: "FailproofAI Observability CLI Agent Skill" +description: "FailproofAI Observability FailproofAI Observability CLI Agent Skill documentation." --- -Ask your coding agent *"is anything broken today?"* and let it answer from your live AgentEye data, with no commands to memorize. The **AgentEye CLI skill** (`agenteye-cli`) is an *Agent Skill*: a small folder of instructions that a coding agent such as Claude Code or Codex loads on demand. It teaches the agent to operate your AgentEye deployment through the [`agenteye` CLI](/agenteye/cli) from plain-English requests like *"give CI a key that can only push events"* or *"ack the firing incident and assign it to me."* +Ask your coding agent *"is anything broken today?"* and let it answer from your live FailproofAI Observability data, with no commands to memorize. The **FailproofAI Observability CLI skill** (`agenteye-cli`) is an *Agent Skill*: a small folder of instructions that a coding agent such as Claude Code or Codex loads on demand. It teaches the agent to operate your Observability deployment through the [`agenteye` CLI](/agenteye/cli) from plain-English requests like *"give CI a key that can only push events"* or *"ack the firing incident and assign it to me."* It is **not** a service or a separate binary; there is nothing to deploy. It rides on top of the CLI you have already installed: the agent shells out to `agenteye --json …`, parses the clean JSON, and answers you in prose. Everything it can do, you could do yourself by typing the same commands. --- -## How it relates to the other AgentEye interfaces +## How it relates to the other FailproofAI Observability interfaces -AgentEye gives you four ways to reach the same data and controls. They complement each other: +FailproofAI Observability gives you four ways to reach the same data and controls. They complement each other: | Interface | What it is | Where it runs | Reach for it when | |---|---|---|---| | **[CLI](/agenteye/cli)** | The command/flag reference for `agenteye` | Your terminal | You want to run or script a specific command | | **[CLI recipes](/agenteye/cli-recipes)** | Copy-paste `jq`/pipeline patterns | Your terminal / scripts | You're wiring the CLI into automation | | **CLI skill** (this doc) | A natural-language front door on the CLI | Your coding agent, on your workstation | You want to *just ask* and let the agent pick the command | -| **[In-dashboard AI assistant](/agenteye/assistant)** | A chat embedded in the dashboard | The agent service (a container in your cluster) | You want in-dashboard Q&A over your data | +| **[In-dashboard AI assistant](/agenteye/assistant)** | A chat embedded in the dashboard | Server-side (in the dashboard) | You want in-dashboard Q&A over your data | The skill itself has no privileges of its own; it just turns your words into CLI calls that run as you: @@ -27,7 +27,7 @@ The skill itself has no privileges of its own; it just turns your words into CLI flowchart TD YOU["you: 'ack the firing incident'"] --> AGENT["coding agent (Claude Code / Codex)
loads the agenteye-cli skill"] AGENT --> CLI["agenteye --json incidents ack ..."] - CLI -->|your authenticated CLI session| API["AgentEye dashboard API"] + CLI -->|your authenticated CLI session| API["Observability dashboard API"] ``` ### vs. the in-dashboard AI assistant: an important distinction @@ -54,7 +54,7 @@ Agent Skills are folders that contain a `SKILL.md` (plus optional references). Y - **Claude Code**: copy the `agenteye-cli/` folder into `~/.claude/skills/` (available in every project) or into `/.claude/skills/` (scoped to that repo). Claude Code auto-discovers it; verify with the `/skills` list, or just ask a question that matches its description. - **Codex (OpenAI)**: Codex reads the same `SKILL.md`. The bundled `agents/openai.yaml` sets `allow_implicit_invocation: true`, so Codex auto-selects the skill when a task matches; otherwise invoke it explicitly as `$agenteye-cli`. -The skill is maintained alongside the `agenteye` CLI but ships as a **separate folder**, not inside the `pipx install agenteye` package, so don't look for it there. AgentEye delivers the `agenteye-cli/` folder to you on its own; if you don't have it, ask your AgentEye contact. Nothing about it is gated: it needs no Docker image and no credential, because it only drives the **public** `agenteye` CLI against your own dashboard. +The skill is maintained alongside the `agenteye` CLI but ships as a **separate folder**, not inside the `pipx install agenteye` package, so don't look for it there. FailproofAI Observability delivers the `agenteye-cli/` folder to you on its own; if you don't have it, ask your FailproofAI contact. Nothing about it is gated: it needs no credential at all, because it only drives the **public** `agenteye` CLI against your own dashboard. --- @@ -64,7 +64,7 @@ The skill is maintained alongside the `agenteye` CLI but ships as a **separate f The `agenteye` CLI normally asks *"are you sure?"* before a destructive action. It **auto-skips that confirmation whenever it is not attached to a terminal (which is exactly how a coding agent runs it), and `--json` skips it too.** So the safety prompt will **not** fire for the agent. -The skill is written to compensate: it is instructed to state the exact command it will run and get your explicit **OK before any state change**. Keep that discipline. When you drive AgentEye through an agent, *you* are the confirmation step. The state-changing commands to watch for: +The skill is written to compensate: it is instructed to state the exact command it will run and get your explicit **OK before any state change**. Keep that discipline. When you drive FailproofAI Observability through an agent, *you* are the confirmation step. The state-changing commands to watch for: - `keys create` / `update` / `disable` / `regenerate` - `users create` / `update` / `disable` / `enable` diff --git a/docs/agenteye/cli.mdx b/docs/agenteye/cli.mdx index 392656e31..189b910e5 100644 --- a/docs/agenteye/cli.mdx +++ b/docs/agenteye/cli.mdx @@ -1,10 +1,10 @@ --- title: "CLI" -description: "AgentEye CLI documentation." +description: "FailproofAI Observability CLI documentation." --- -Drive your whole AgentEye deployment from the terminal or a script: no dashboard round-trips. The `agenteye` CLI queries your data (sessions, event logs, evaluations) and administers your org (API keys, users, settings, alerts, incidents, saved queries), so reach for it when you want to automate a check, wire AgentEye into CI, or let a coding agent inspect production. Every command supports a `--json` flag, so it works equally well for you at a prompt or for a coding agent (Claude Code, Cursor) shelling out and parsing the result. +Drive all of FailproofAI Observability from the terminal or a script: no dashboard round-trips. The `agenteye` CLI queries your data (sessions, event logs, evaluations) and administers your org (API keys, users, settings, alerts, incidents, saved queries), so reach for it when you want to automate a check, wire Observability into CI, or let a coding agent inspect production. Every command supports a `--json` flag, so it works equally well for you at a prompt or for a coding agent (Claude Code, Cursor) shelling out and parsing the result. With one binary you can: @@ -13,7 +13,7 @@ With one binary you can: - **Run analytics**: saved SQL and an ad-hoc query runner (`query`). - **Ask the AI assistant**: the same read-only analyst you chat with in the dashboard (`agent`). -> **Note:** This is the `agenteye` CLI, a different tool from the collector daemon (`agenteye-collector`). The CLI talks to your dashboard; the collector ships events to the server. See [Collector installation](/agenteye/collector-installation) for the collector. +> **Note:** This is the `agenteye` CLI, a different tool from the collector daemon (`agenteye-collector`). The CLI talks to your dashboard; the collector ships events to the server. --- @@ -62,14 +62,14 @@ pipx install agenteye uv tool install agenteye ``` -It requires Python 3.10+ and needs no GitHub token. The installed command is **`agenteye`**: +It requires Python 3.10+. The installed command is **`agenteye`**: ```bash agenteye --version agenteye --help ``` -> **Note:** The AgentEye Python SDK also uses the `agenteye` distribution name. Installing the CLI with `pipx` or `uv tool` (rather than `pip install` into a shared virtualenv) keeps the two from colliding. A plain `pip install agenteye` is fine only if the SDK is not installed in the same environment. +> **Note:** The FailproofAI Observability Python SDK also uses the `agenteye` distribution name. Installing the CLI with `pipx` or `uv tool` (rather than `pip install` into a shared virtualenv) keeps the two from colliding. A plain `pip install agenteye` is fine only if the SDK is not installed in the same environment. --- @@ -91,7 +91,7 @@ agenteye logout # revoke the session and clear the stored token `whoami` never errors on a missing or expired session; it reports `logged_in: false` instead, so a script or agent can probe auth state safely (it can still exit non-zero if no base URL is set or the dashboard is unreachable). -**Requirements:** your email must be permitted to sign in to the dashboard (ask your AgentEye administrator), and the dashboard must be reachable at its base URL (see [Configuration](#configuration)). If you request a code and none arrives, your email is likely not yet enabled for dashboard access. +**Requirements:** your email must be permitted to sign in to the dashboard (ask your FailproofAI Observability administrator), and the dashboard must be reachable at its base URL (see [Configuration](#configuration)). If you request a code and none arrives, your email is likely not yet enabled for dashboard access. --- @@ -302,7 +302,7 @@ agenteye incidents subscribe ; agenteye incidents unsubscribe ; agente ### Analytics & assistant: `query` · `agent` -**`query`**: saved SQL against your analytics store (ClickHouse, the database that holds your events) plus an ad-hoc runner. Saved queries are referenced by **name**; the SQL is validated server-side (SELECT/WITH only, statement timeout, row cap). +**`query`**: saved SQL against your analytics store plus an ad-hoc runner. Saved queries are referenced by **name**; the SQL is validated server-side (SELECT/WITH only, statement timeout, row cap). ```bash agenteye query schema [TABLE] # column layout of the analytics views @@ -345,6 +345,6 @@ These make the CLI safe to script: a coding agent can branch on a `4` to prompt ## Next steps - **[CLI recipes for agents](/agenteye/cli-recipes)**: copy-paste query patterns, `jq` one-liners, `--fields` projections, exit-code handling, and JSON output shapes, written for coding agents driving the CLI. -- **[CLI agent skill](/agenteye/cli-skill)**: package this CLI as an installable Claude Code / Codex *skill* so a coding agent drives AgentEye from plain-English requests. +- **[CLI agent skill](/agenteye/cli-skill)**: package this CLI as an installable Claude Code / Codex *skill* so a coding agent drives FailproofAI Observability from plain-English requests. - **[API keys](/agenteye/api-keys)**: the permission model behind `keys create --add …`. - **[AI assistant](/agenteye/assistant)**: enabling the assistant that `agent ask` talks to. diff --git a/docs/agenteye/collector-installation.mdx b/docs/agenteye/collector-installation.mdx deleted file mode 100644 index efcdb68fe..000000000 --- a/docs/agenteye/collector-installation.mdx +++ /dev/null @@ -1,464 +0,0 @@ ---- -title: "Collector Installation" -description: "AgentEye Collector Installation documentation." ---- - - -The `agenteye-collector` daemon makes sure your agents' telemetry reaches AgentEye without ever blocking your application. Your code writes events to a local directory and moves on; the collector takes ownership from there, uploading each file within milliseconds and surviving restarts, network outages, and transient server errors. The result is durable, fire-and-forget delivery: your agents run at full speed while the collector guarantees no events are lost in transit. - -Mechanically, the collector is a lightweight daemon that watches `$AGENTEYE_HOME/events/` (default: `~/.agenteye/events/`) for `.jsonl` files written by the Python SDK and uploads them to the AgentEye server. Failed uploads are retried with exponential backoff, and a periodic recovery sweep re-queues anything left behind by a crash or deploy. - -> **Note:** The collector command is now **`agenteye-collector`** (it used to be `agenteye`). The short `agenteye` name now belongs to the AgentEye CLI. If you are upgrading an existing install, see [Collector migration](/agenteye/collector-migration). - ---- - -## Prerequisites - -- Your `AGENTEYE_TOKEN`: a GitHub PAT you generate yourself (see [GitHub token setup](/agenteye/github-token)) -- The server URL and a collector API key (see [API keys](/agenteye/api-keys)) - ---- - -## Quickstart - -The smallest path to a working collector on Linux x86_64. The detailed sections below cover every other platform, config option, and deployment pattern. - -1. Install the binary (see [Option A](#option-a-binary-recommended) for other platforms): - - ```bash - VERSION=0.0.1-beta.14 - GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download "collector/v${VERSION}" \ - --repo agenteye-enterprise/releases \ - --pattern 'agenteye-collector-linux-x86_64' - chmod +x agenteye-collector-linux-x86_64 - sudo mv agenteye-collector-linux-x86_64 /usr/local/bin/agenteye-collector - ``` - -2. Point it at your server with your collector API key: - - ```bash - export AGENTEYE_URL=https://ingest.example.com/events - export AGENTEYE_KEY=YOUR_COLLECTOR_KEY - ``` - -3. Generate one event by running an agent instrumented with the [Python SDK](/agenteye/python-sdk), so a `.jsonl` file lands in `~/.agenteye/events/`. - -4. Confirm it uploads with a one-shot flush and look for an `[UPLOADED]` line: - - ```bash - agenteye-collector flush - ``` - -5. Run the daemon so new events keep uploading: - - ```bash - agenteye-collector start - ``` - -Then open the AgentEye dashboard and confirm your agent's session appears (see [Getting started](/agenteye/getting-started)). - ---- - -## Option A: Binary (recommended) - -Pre-built static binaries are available for Linux, macOS, and Windows (x86_64 and arm64). Download the binary for your platform directly from the `agenteye-enterprise/releases` repo under the latest `collector/v` release tag. - -Available artifact names: - -| Platform | Artifact | -|---|---| -| Linux x86_64 | `agenteye-collector-linux-x86_64` | -| Linux arm64 | `agenteye-collector-linux-arm64` | -| macOS x86_64 | `agenteye-collector-darwin-x86_64` | -| macOS arm64 | `agenteye-collector-darwin-arm64` | -| Windows x86_64 | `agenteye-collector-windows-x86_64.exe` | -| Windows arm64 | `agenteye-collector-windows-arm64.exe` | - -**Download with the `gh` CLI** (replace the version and pick your platform's artifact): - -```bash -VERSION=0.0.1-beta.14 -GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download "collector/v${VERSION}" \ - --repo agenteye-enterprise/releases \ - --pattern 'agenteye-collector-linux-x86_64' - -chmod +x agenteye-collector-linux-x86_64 -sudo mv agenteye-collector-linux-x86_64 /usr/local/bin/agenteye-collector -``` - -**Or with `curl`:** - -```bash -VERSION=0.0.1-beta.14 -ARTIFACT=agenteye-collector-linux-x86_64 -curl -fsSL \ - -H "Authorization: token $AGENTEYE_TOKEN" \ - -L "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/agenteye-enterprise/releases/releases/download/collector%2Fv${VERSION}/${ARTIFACT}" \ - -o agenteye-collector -chmod +x agenteye-collector -sudo mv agenteye-collector /usr/local/bin/agenteye-collector -``` - ---- - -## Option B: Docker - -```bash -echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin -docker pull ghcr.io/agenteye-enterprise/collector:beta-latest -``` - -> **Note:** Pulling the collector image requires signed enterprise access to `ghcr.io/agenteye-enterprise/*`. Authenticate with your `AGENTEYE_TOKEN` as shown above. - -> **Tip:** Current beta builds publish the floating `:beta-latest` tag; `:latest` is assigned only to stable releases. For repeatable deployments, prefer a pinned version tag such as `:v0.0.1-beta.14`. - -**Run:** - -```bash -docker run -d --restart unless-stopped \ - --name agenteye-collector \ - -e AGENTEYE_URL=https://ingest.example.com/events \ - -e AGENTEYE_KEY="$AGENTEYE_KEY" \ - -e AGENTEYE_HOME=/data \ - -v "$HOME/.agenteye:/data" \ - ghcr.io/agenteye-enterprise/collector:beta-latest \ - start -``` - -The official image runs as a non-root user, so set `AGENTEYE_HOME` explicitly and mount the host spool to it. The volume mount shares the same `~/.agenteye/` directory that the Python SDK writes to on the host. If you already set `AGENTEYE_HOME` elsewhere on the host, mount that directory instead of `$HOME/.agenteye`. - -> **Warning:** The image runs as UID 65532, so the mounted host directory must be writable by that user. If it is not, the collector cannot write `health.json` or manage the spool, and the daemon fails to start. - ---- - -## Configuration - -All options can be set three ways (highest priority first): - -1. CLI flag: `agenteye-collector start --url https://...` -2. Environment variable: `AGENTEYE_URL=https://...` -3. Config file: `~/.agenteye/config.json` - -### Required options - -| Option | CLI flag | Env var | config.json key | -|---|---|---|---| -| Backend URL | `--url ` | `AGENTEYE_URL` | `"url"` | -| API key | `--key ` | `AGENTEYE_KEY` | `"key"` | - -### Optional options (with defaults) - -| Option | CLI flag | Env var | config.json key | Default | -|---|---|---|---|---| -| Max concurrent uploads | `--max-concurrent-uploads ` | `AGENTEYE_MAX_CONCURRENT_UPLOADS` | `"max_concurrent_uploads"` | `64` | -| Sweeper interval (s) | `--sweep-interval ` | `AGENTEYE_SWEEP_INTERVAL` | `"sweep_interval_secs"` | `60` | -| Sweeper min file age (s) | `--sweep-min-age ` | `AGENTEYE_SWEEP_MIN_AGE` | `"sweep_min_age_secs"` | `120` | -| Max files per sweep | `--sweep-max-files ` | `AGENTEYE_SWEEP_MAX_FILES` | `"sweep_max_files"` | `64` | -| Max upload attempts | `--max-retries ` | `AGENTEYE_MAX_RETRIES` | `"max_retries"` | `5` | -| Retry base delay (ms) | `--retry-base-delay ` | `AGENTEYE_RETRY_BASE_DELAY` | `"retry_base_delay_ms"` | `1000` | - -### mTLS options (optional) - -For deployments that require mutual TLS (mTLS), the collector can present a client certificate during the TLS handshake. When these options are not set, the collector uses standard HTTPS. - -| Option | CLI flag | Env var | config.json key | -|---|---|---|---| -| Client certificate (PEM) | `--tls-cert ` | `AGENTEYE_TLS_CERT` | `"tls_cert"` | -| Client private key (PEM) | `--tls-key ` | `AGENTEYE_TLS_KEY` | `"tls_key"` | -| Custom CA cert (PEM) | `--tls-ca ` | `AGENTEYE_TLS_CA` | `"tls_ca"` | - -`--tls-cert` and `--tls-key` must be set together. The files must be PEM-encoded. - -`--tls-ca` is independent and only needed when the AgentEye server presents a TLS certificate that isn't issued by a publicly-trusted CA (e.g. self-signed by an in-cluster `cert-manager` issuer when you don't have a real DNS domain). The collector adds the supplied CA as an additional trust anchor; the standard public roots remain trusted, so existing deployments are unaffected. The file may contain a single PEM cert or a full chain (multiple concatenated PEM blocks). - -**Running the collector as a sidecar in your application pod?** See [Single-pod deployment](/agenteye/single-pod-deployment) for the end-to-end EKS pattern: mTLS bundle delivered via AWS Secrets Manager + Secrets Store CSI Driver + IRSA, with automatic rotation. - -When running in Kubernetes with the Secret hand-off pattern, mount the certificate Secret as a volume and point these paths to the mounted files: - -```yaml -# Example: collector Deployment snippet -volumes: - - name: mtls-certs - secret: - secretName: agenteye-collector-mtls -containers: - - name: collector - env: - - name: AGENTEYE_TLS_CERT - value: /etc/agenteye/tls/tls.crt - - name: AGENTEYE_TLS_KEY - value: /etc/agenteye/tls/tls.key - # Only when the server cert isn't publicly-trusted (e.g. in-cluster - # self-signed CA). The same Secret typically carries ca.crt alongside - # tls.crt/tls.key. - - name: AGENTEYE_TLS_CA - value: /etc/agenteye/tls/ca.crt - volumeMounts: - - name: mtls-certs - mountPath: /etc/agenteye/tls - readOnly: true -``` - -### Example `~/.agenteye/config.json` - -```json -{ - "url": "https://ingest.example.com/events", - "key": "sk-...", - "max_concurrent_uploads": 16, - "sweep_interval_secs": 30 -} -``` - -With mTLS: - -```json -{ - "url": "https://ingest.example.com/events", - "key": "sk-...", - "tls_cert": "/etc/agenteye/tls/tls.crt", - "tls_key": "/etc/agenteye/tls/tls.key" -} -``` - -With mTLS plus a custom CA (self-signed AgentEye server): - -```json -{ - "url": "https://ingest.example.com/events", - "key": "sk-...", - "tls_cert": "/etc/agenteye/tls/tls.crt", - "tls_key": "/etc/agenteye/tls/tls.key", - "tls_ca": "/etc/agenteye/tls/ca.crt" -} -``` - -If `AGENTEYE_HOME` is set, that directory is used instead of `~/.agenteye`. - ---- - -## First-time Setup - -After installing, configure the collector with your server URL and API key: - -```bash -mkdir -p ~/.agenteye -cat > ~/.agenteye/config.json <<'EOF' -{ - "url": "https://ingest.example.com/events", - "key": "YOUR_COLLECTOR_KEY" -} -EOF -``` - -> Use `https` for any deployment that crosses an untrusted network so events are not sent in plaintext. The plaintext `http://your-server-host:8080/events` form is appropriate only for purely local testing against a server on the same host. - -**Confirm uploads are working** (one-shot flush, exits after draining pending events): - -```bash -agenteye-collector flush -``` - -`flush` reports its progress to stdout. When the spool (your `~/.agenteye/events/` directory) is empty it prints `No pending files.` and exits `0`, but note that an empty spool makes **no network call at all**, so this on its own does not confirm your connection. Otherwise it prints one line per file (`[UPLOADED] ` or `[FAILED] ()`), followed by a `Done: / uploaded, failed.` summary. - -> **Tip:** To actually validate your URL, key, and TLS settings, first run your agent (or the [Python SDK](/agenteye/python-sdk)) so at least one `.jsonl` event lands in `~/.agenteye/events/`, then run `agenteye-collector flush` and look for an `[UPLOADED]` line. - ---- - -## Running as a Daemon - -### Direct - -```bash -agenteye-collector start -``` - -### Container / Docker - -When the collector and your application share a container, run them under a process supervisor. The simplest option is `supervisord`; it ships in every major distro, restarts crashed processes, forwards signals, and waits for graceful shutdown. - -**`Dockerfile`:** - -```Dockerfile -FROM python:3.11-slim - -RUN apt-get update && apt-get install -y --no-install-recommends supervisor \ - && rm -rf /var/lib/apt/lists/* - -# Pull the agenteye-collector binary from the official image. -# Pin a specific tag (:beta-latest for current betas, or a :v tag); -# :latest is published only for stable releases. -COPY --from=ghcr.io/agenteye-enterprise/collector:beta-latest \ - /usr/local/bin/agenteye-collector /usr/local/bin/agenteye-collector - -COPY supervisord.conf /etc/supervisor/conf.d/agenteye.conf -COPY my_agent.py /app/my_agent.py - -CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"] -``` - -**`supervisord.conf`:** - -```ini -[supervisord] -nodaemon=true -user=root - -[program:agenteye-collector] -command=/usr/local/bin/agenteye-collector start -autorestart=true -stopwaitsecs=30 -stdout_logfile=/dev/stdout -stdout_logfile_maxbytes=0 -stderr_logfile=/dev/stderr -stderr_logfile_maxbytes=0 - -[program:app] -command=python /app/my_agent.py -autorestart=unexpected -stopwaitsecs=30 -stdout_logfile=/dev/stdout -stdout_logfile_maxbytes=0 -stderr_logfile=/dev/stderr -stderr_logfile_maxbytes=0 -``` - -Why these settings: - -- `autorestart=true` on agenteye-collector: restart on any exit (crash, panic, OOM). -- `autorestart=unexpected` on the app: restart only on non-zero exit, so a one-shot agent that exits 0 doesn't loop. -- `stopwaitsecs=30`: gives the collector room to drain pending uploads on SIGTERM before supervisord escalates to SIGKILL. -- `stdout_logfile=/dev/stdout`, `*_maxbytes=0`: stream both programs' output to the container stdout; no log files inside the container. - -Pass `AGENTEYE_URL` / `AGENTEYE_KEY` (and any TLS env vars) on `docker run -e` as before; supervisord inherits the environment. - -> **Note:** If you run the collector as its own container (Docker Compose service, Kubernetes sidecar, etc.), don't use supervisord; the container runtime's restart policy already does this job. See [Single-pod deployment](/agenteye/single-pod-deployment) for the EKS sidecar pattern. - -**Kubernetes liveness probe** (applies whether the collector runs alone or under supervisord): - -```yaml -livenessProbe: - exec: - command: ["agenteye-collector", "health"] - initialDelaySeconds: 10 - periodSeconds: 30 -``` - -The running daemon writes a heartbeat to `$AGENTEYE_HOME/health.json` every 30 seconds. `agenteye-collector health` reads that file and exits `0` (healthy) only when the heartbeat is fresh and the upload tasks are running normally; it exits `1` (unhealthy) when the heartbeat is older than 90 seconds (for example, the daemon has stopped) or while the watcher and sweeper are restarting after an unexpected exit. The heartbeat is written only by `start`, so run the probe against the long-lived daemon rather than the one-shot `flush` command. - -### systemd (Linux, recommended for production) - -```ini -# /etc/systemd/system/agenteye-collector.service -[Unit] -Description=AgentEye Collector -After=network.target - -[Service] -ExecStart=/usr/local/bin/agenteye-collector start -Restart=on-failure -RestartSec=5 -EnvironmentFile=/etc/agenteye/env - -[Install] -WantedBy=multi-user.target -``` - -Create `/etc/agenteye/env`: - -```ini -AGENTEYE_URL=https://ingest.example.com/events -AGENTEYE_KEY=YOUR_COLLECTOR_KEY -``` - -```bash -sudo systemctl daemon-reload -sudo systemctl enable --now agenteye-collector -sudo systemctl status agenteye-collector -``` - -### launchd (macOS) - -```xml - - - - - - Label - ai.befailproof.agenteye-collector - ProgramArguments - - /usr/local/bin/agenteye-collector - start - - EnvironmentVariables - - AGENTEYE_URL - https://ingest.example.com/events - AGENTEYE_KEY - YOUR_COLLECTOR_KEY - - RunAtLoad - - KeepAlive - - - -``` - -```bash -launchctl load ~/Library/LaunchAgents/ai.befailproof.agenteye-collector.plist -``` - -### Confirm it's working - -With the daemon running, run an agent so it emits events, then open the AgentEye dashboard and check the sessions list. Your agent's session should appear within a few seconds of the events being written. - -![AgentEye sessions list showing recently arrived agent sessions](/agenteye/images/sessions-list.png) - -If nothing shows up, see [Troubleshooting](/agenteye/troubleshooting) and [Getting started](/agenteye/getting-started) for the full SDK → collector → server path. - ---- - -## Upgrading the Collector - -The collector does not update itself. To upgrade: - -- **Binary:** download the new `agenteye-collector--` artifact from the latest `collector/v` release (see [Option A](#option-a-binary-recommended)), replace `/usr/local/bin/agenteye-collector`, then restart the service (`sudo systemctl restart agenteye-collector`, re-`launchctl load`, or restart your supervisor). -- **Docker:** `docker pull ghcr.io/agenteye-enterprise/collector:beta-latest` (or a pinned `:v` tag; `:latest` exists only for stable releases) and recreate the container. - -`AGENTEYE_TOKEN` is required to download new binaries/images from the private releases repo, but is **not** needed by the running daemon. - ---- - -## Subcommands - -| Command | Description | -|---|---| -| `agenteye-collector start` | Start the long-lived daemon. On startup it flushes any events left over from a previous run, then watches for new files and uploads them. The watcher and sweeper auto-restart on unexpected exit, and a heartbeat is written to `health.json` every 30 seconds. | -| `agenteye-collector flush` | One-shot: upload all pending files and exit. Prints `No pending files.` when the spool is empty, otherwise a per-file `[UPLOADED]`/`[FAILED]` log and a `Done: / uploaded, failed.` summary. | -| `agenteye-collector health` | Read the daemon's `health.json` heartbeat. Exit `0` when fresh and healthy; exit `1` when the heartbeat is stale (older than 90s) or the tasks are restarting. | - ---- - -## Directory Layout - -```text -~/.agenteye/ -├── config.json <- optional config file -├── events/ <- .jsonl files written by the SDK, picked up by the collector -└── failed/ <- files that failed all upload attempts -``` - -Files in `failed/` are not automatically retried. To manually re-queue them, move them back to `events/` and run `agenteye-collector flush`. - ---- - -## Next steps - -- [Python SDK](/agenteye/python-sdk): instrument your agents so they write the events the collector uploads. -- [API keys](/agenteye/api-keys): create and scope the collector key the daemon authenticates with. -- [Getting started](/agenteye/getting-started): the full setup, end to end. -- [Troubleshooting](/agenteye/troubleshooting): fixes for events that aren't reaching AgentEye. diff --git a/docs/agenteye/collector-migration.mdx b/docs/agenteye/collector-migration.mdx deleted file mode 100644 index bf6feb3b6..000000000 --- a/docs/agenteye/collector-migration.mdx +++ /dev/null @@ -1,179 +0,0 @@ ---- -title: "Migrating to `agenteye-collector`" -description: "AgentEye Migrating to `agenteye-collector` documentation." ---- - - -The collector binary has been renamed from `agenteye` to `agenteye-collector`. Migrating is non-destructive (no downtime, no data loss) and it frees the short `agenteye` name for the [AgentEye CLI](/agenteye/cli), so the collector daemon and the CLI can coexist on one machine. Do it the next time you update an existing collector install. - -> **Note:** You only need this page if you already have an `agenteye` collector daemon running. Installing for the first time? See [Collector installation](/agenteye/collector-installation) instead. None of the migration steps below apply to a fresh install. - -The short `agenteye` name now belongs to the AgentEye CLI, a separate tool for querying sessions, events, and evaluations from your terminal. This guide walks you through migrating an existing collector install. - ---- - -## What changed - -| | Before | After | -|---|---|---| -| Command / binary | `agenteye` | `agenteye-collector` | -| Default install path | `/usr/local/bin/agenteye` | `/usr/local/bin/agenteye-collector` | -| Subcommands | `start`, `flush`, `health`, `update` | `start`, `flush`, `health` | -| Self-update (`agenteye update`) | built in | **removed**: download the new binary or pull the new image | -| Install script (`install.sh`) | provided | **removed**: download the binary directly (see [Collector Installation](/agenteye/collector-installation)) | -| `AGENTEYE_TOKEN` | needed to download **and** for background update checks | needed only to **download** binaries/images | - -Configuration is unchanged: the same `~/.agenteye/config.json`, the same `AGENTEYE_URL` / `AGENTEYE_KEY` / `AGENTEYE_HOME` / TLS environment variables, and the same `~/.agenteye/events/` spool. **No config edits are required.** - -> **Note:** If you run the renamed binary under the old name `agenteye`, it still works but prints a one-line deprecation warning to stderr reminding you to switch to `agenteye-collector`. - ---- - -## Before you start - -- Your **existing `agenteye` install keeps running**; nothing breaks the moment you upgrade. Migrate deliberately, then remove the old binary last. -- Follow this order to avoid downtime: - 1. Install the new `agenteye-collector` binary (or pull the new image). - 2. Update your service definition / health probe / scripts to call `agenteye-collector`. - 3. Reload and restart the service; confirm it is healthy. - 4. **Only then** remove the old `/usr/local/bin/agenteye` binary. - ---- - -## 1. Install the new binary - -Download the artifact for your platform (`agenteye-collector-linux-x86_64`, `agenteye-collector-darwin-arm64`, and so on; see [Collector Installation → Option A](/agenteye/collector-installation#option-a-binary-recommended) for the full list) from the latest `collector/v` release and place it at `/usr/local/bin/agenteye-collector`. Docker users: `docker pull ghcr.io/agenteye-enterprise/collector:beta-latest` (or a pinned `:v` tag, which is preferred; `:latest` exists only for stable releases). - -Verify: - -```bash -agenteye-collector --version -``` - ---- - -## 2. Update your deployment - -### systemd (Linux) - -Edit `/etc/systemd/system/agenteye-collector.service` so `ExecStart` points at the new binary: - -```ini -[Service] -ExecStart=/usr/local/bin/agenteye-collector start -``` - -Then reload and restart: - -```bash -sudo systemctl daemon-reload -sudo systemctl restart agenteye-collector -sudo systemctl status agenteye-collector -``` - -### launchd (macOS) - -> **Note:** If your existing plist is at the older path -> `~/Library/LaunchAgents/host.exosphere.agenteye-collector.plist`, rename -> the file to `ai.befailproof.agenteye-collector.plist` and also change the -> `Label` value inside the file to the new identifier before -> reloading. - -In `~/Library/LaunchAgents/ai.befailproof.agenteye-collector.plist`, change the first `ProgramArguments` entry from `/usr/local/bin/agenteye` to `/usr/local/bin/agenteye-collector`, then reload: - -```bash -launchctl unload ~/Library/LaunchAgents/ai.befailproof.agenteye-collector.plist -launchctl load ~/Library/LaunchAgents/ai.befailproof.agenteye-collector.plist -``` - -### supervisord - -In your `supervisord` program block, set `command` to the new binary: - -```ini -[program:agenteye-collector] -command=/usr/local/bin/agenteye-collector start -``` - -Then `supervisorctl reread && supervisorctl update`. - -### Docker / Kubernetes - -Pull the new image (`ghcr.io/agenteye-enterprise/collector:beta-latest` or a pinned `:v`, which is preferred; `:latest` exists only for stable releases). The image entrypoint is already `agenteye-collector`, so the same `docker run` command with the `start` subcommand keeps working with no change. - -> **Warning:** Update your health probes. If you use a Kubernetes liveness/readiness probe (or any `docker exec`) that runs the binary by name, change the command to `agenteye-collector`: - -```yaml -livenessProbe: - exec: - command: ["agenteye-collector", "health"] -``` - -The new image does **not** ship an `agenteye` alias, so a probe still calling `agenteye` will fail. Update the probe in the same rollout as the new image. - -### Cron / manual scripts - -Replace any `agenteye start|flush|health` invocations with the matching `agenteye-collector start|flush|health` command. **Delete any `agenteye update` cron jobs**; that subcommand no longer exists (see [Upgrades from now on](#upgrades-from-now-on)). - ---- - -## 3. Remove the old binary (last) - -Once the service runs on `agenteye-collector` and reports healthy: - -```bash -sudo rm /usr/local/bin/agenteye -``` - -This matters especially if you also use the AgentEye CLI, which installs its own `agenteye` command; leaving the old collector binary at `/usr/local/bin/agenteye` would make the `agenteye` name ambiguous on your `PATH`. - ---- - -## Upgrades from now on - -The collector no longer updates itself. To upgrade: - -- **Binary:** download the new artifact for your platform (for example `agenteye-collector-linux-x86_64`; see [Collector Installation → Option A](/agenteye/collector-installation#option-a-binary-recommended) for the full list), replace `/usr/local/bin/agenteye-collector`, and restart the service. -- **Docker:** `docker pull ghcr.io/agenteye-enterprise/collector:beta-latest` (or a pinned `:v` tag, which is preferred; `:latest` exists only for stable releases) and recreate the container. - -`AGENTEYE_TOKEN` is still required to download from the private releases repo, but the running daemon no longer needs it. - ---- - -## Verify - -```bash -agenteye-collector --version # new binary is on PATH -agenteye-collector health # exit 0 = healthy -agenteye-collector flush # forwards any queued events and exits cleanly -``` - -Then open your AgentEye dashboard and confirm events are still flowing. The live events stream (the **Events** page) and the **Sessions** list should both show fresh activity with recent timestamps. When they do, your migration is complete. - -![Freshly ingested events in the dashboard live events stream](/agenteye/images/events-stream.png) - ---- - -## Rollback - -The migration is non-destructive. If you need to roll back, point your service definition back at the old `/usr/local/bin/agenteye` binary (as long as you have not removed it yet) and restart. The event spool and config are shared and unaffected. - ---- - -## Troubleshooting - -| Symptom | Cause | Fix | -|---|---|---| -| `warning: the collector binary is now agenteye-collector …` on every run | You are invoking the binary under the old `agenteye` name | Call `agenteye-collector` instead; update service files and scripts. | -| systemd fails: `.../agenteye: No such file or directory` | You removed the old binary before updating `ExecStart` | Set `ExecStart=/usr/local/bin/agenteye-collector start`, then `sudo systemctl daemon-reload`. | -| Kubernetes pod crash-loops after the image upgrade | Liveness probe still runs `agenteye` | Change the probe command to `["agenteye-collector", "health"]`. | -| `agenteye: command not found`, but `agenteye-collector` works | Scripts/aliases still reference the old name | Update them to `agenteye-collector`. | -| Running `agenteye` starts the CLI, not the collector | You have the AgentEye CLI installed; it owns `agenteye` | Use `agenteye-collector` for the daemon and remove any leftover old collector binary at `/usr/local/bin/agenteye`. | - ---- - -## Next steps - -- [Collector installation](/agenteye/collector-installation): the full install and configuration reference for `agenteye-collector`. -- [CLI](/agenteye/cli): the tool that now owns the short `agenteye` command. -- [Troubleshooting](/agenteye/troubleshooting): fixes for events that stop appearing after the switch. diff --git a/docs/agenteye/concepts.mdx b/docs/agenteye/concepts.mdx index 7d95acdbe..e09deae4f 100644 --- a/docs/agenteye/concepts.mdx +++ b/docs/agenteye/concepts.mdx @@ -1,10 +1,10 @@ --- title: "Concepts" -description: "AgentEye Concepts documentation." +description: "FailproofAI Observability Concepts documentation." --- -This page defines the vocabulary AgentEye uses. If a term in another guide is unfamiliar, it's defined here. You don't need to read it end to end: skim it, or jump back when you hit a word you want pinned down. +This page defines the vocabulary FailproofAI Observability uses. If a term in another guide is unfamiliar, it's defined here. You don't need to read it end to end: skim it, or jump back when you hit a word you want pinned down. --- @@ -17,13 +17,13 @@ The smallest unit of data. One event records a single step your agent took: a `t One agent run, identified by a `session_id`. A session is all the events that share that id, rolled up into a single row on the **Sessions** page and drawn as an execution graph on its detail page. A session usually starts with `agent_start` and ends with `agent_end`. **Agent** -A named actor inside a run, identified by an `agent_id`. A run can involve several agents: a planner that spawns a summarizer sub-agent, for example. Sub-agents carry a `parent_id`, which is what lets AgentEye draw them on their own lanes in the execution graph. +A named actor inside a run, identified by an `agent_id`. A run can involve several agents: a planner that spawns a summarizer sub-agent, for example. Sub-agents carry a `parent_id`, which is what lets FailproofAI Observability draw them on their own lanes in the execution graph. **Environment** A label for where the run happened: `production`, `staging`, `dev`. You set it once when you configure the SDK. Almost every dashboard page can filter by environment. **Context-window fill** -The percentage of a model's context window a response consumed. AgentEye stamps it on `model_response` events for models it recognizes, so prompt growth and impending compaction are visible right in the event stream. +The percentage of a model's context window a response consumed. FailproofAI Observability stamps it on `model_response` events for models it recognizes, so prompt growth and impending compaction are visible right in the event stream. --- @@ -36,7 +36,7 @@ A quality score for a finished session, produced by a scoring service you run. E The name of one dimension an evaluator reports, such as `helpfulness`. Alerts and audits can watch a specific score key over time. **Evaluator** -Your scoring service. AgentEye POSTs a finished run's transcript to it and stores the scores it returns. AgentEye does not ship a default evaluator; the scoring logic is yours. +Your scoring service. FailproofAI Observability POSTs a finished run's transcript to it and stores the scores it returns. It does not ship a default evaluator; the scoring logic is yours. --- @@ -55,7 +55,7 @@ An open issue created when an alert rule fires. Incidents have a lifecycle (ackn A recurring investigation (hourly to weekly) that mines your logs *across* sessions for failure patterns you haven't written a rule for: error clusters, low scores, latency outliers, tool-call loops, and runs that never finished. Where an alert watches a metric you already know about, an audit tells you what to look at next. See [Audits](/agenteye/audits). **Finding** -One ranked, evidence-backed result from an audit run. A finding names a pattern, links to the exact sessions behind it, and carries a triage lifecycle (acknowledge, resolve, mute, dismiss). AgentEye deduplicates findings run-over-run so a known pattern updates instead of piling up. +One ranked, evidence-backed result from an audit run. A finding names a pattern, links to the exact sessions behind it, and carries a triage lifecycle (acknowledge, resolve, mute, dismiss). FailproofAI Observability deduplicates findings run-over-run so a known pattern updates instead of piling up. **The AI assistant** The in-dashboard chat that answers questions about your agents in plain English, over your own data. It is read-only by default; anything it creates (a saved query, a dashboard) is approval-gated, and it can never delete. See [AI assistant](/agenteye/assistant). @@ -65,16 +65,16 @@ The in-dashboard chat that answers questions about your agents in plain English, ## Running it **Organization (tenant)** -An isolated workspace. One AgentEye instance can host many organizations, each with its own users, keys, and data. Every dashboard URL is scoped under your org slug (`//…`). See [Tenant management](/agenteye/tenant-management). +An isolated workspace. One FailproofAI Observability instance can host many organizations, each with its own users, keys, and data. Every dashboard URL is scoped under your org slug (`//…`). **Collector** -`agenteye-collector`, the lightweight daemon that runs on each agent machine, batches the events the SDK writes to disk, and ships them to the server. See [Collector installation](/agenteye/collector-installation). +`agenteye-collector`, the lightweight daemon that runs on each agent machine, batches the events the SDK writes to disk, and ships them to the server. **API key** A scoped token that authenticates a client against the server. Keys carry granular permissions (for example `events:add` for the collector, read-only scopes for a dashboard key). See [API keys](/agenteye/api-keys). **Server** -The ingest and API service. It receives events, writes them to ClickHouse, keeps operational state in PostgreSQL, and serves the dashboard and CLI. +The ingest and API service. It ingests events, stores operational state in your databases, and serves the dashboard and CLI. **Dashboard** The web UI. Every page is scoped to an organization and reads through the server's API. @@ -84,5 +84,4 @@ The web UI. Every page is scoped to an organization and reads through the server ## Next steps - [Overview](/agenteye/overview): how these pieces fit together. -- [Getting started](/agenteye/getting-started): put the concepts to work end to end. - [Observability](/agenteye/observability): the observe surfaces (Events, Sessions, Models, Tools, Hooks, Errors). diff --git a/docs/agenteye/dashboards.mdx b/docs/agenteye/dashboards.mdx index c07e6dbb0..8db1905f8 100644 --- a/docs/agenteye/dashboards.mdx +++ b/docs/agenteye/dashboards.mdx @@ -1,6 +1,6 @@ --- title: "Dashboards" -description: "AgentEye Dashboards documentation." +description: "FailproofAI Observability Dashboards documentation." --- diff --git a/docs/agenteye/deployment-options.mdx b/docs/agenteye/deployment-options.mdx deleted file mode 100644 index 750f793e5..000000000 --- a/docs/agenteye/deployment-options.mdx +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: "Deployment options" -description: "AgentEye Deployment options documentation." ---- - - -AgentEye runs entirely in your own environment, and there are four ways to install it. This page helps you pick the right one in about 30 seconds, then points you at the full guide. If you're just trying it out, start with Docker Compose. - ---- - -## Pick your path - -| Path | Best for | Who operates it | Effort | -|---|---|---|---| -| **Docker Compose** | Trying AgentEye, small deployments, a single host | You | Lowest: one stack, a few minutes | -| **Managed** | Production, without running the platform yourself | Failproof AI, on your cluster | Low: you integrate, we operate | -| **Kubernetes** | Production at scale, full control | You | Highest: HA, storage, TLS, upgrades | -| **Single-pod sidecar** | Co-locating the collector with one app on Kubernetes | You | Medium: one pod, no cluster-wide install | - ---- - -## Docker Compose - -The fastest way to see AgentEye. One Compose stack brings up the server, dashboard, ClickHouse, PostgreSQL, and an optional Redis cache on a single host. Ideal for evaluations, demos, and small production deployments. - -Start here: [Getting started](/agenteye/getting-started). - -## Managed - -Failproof AI operates the AgentEye platform on a cluster you designate, so you get production hosting without running the databases and upgrades yourself. You still keep your data in your own cloud account. You install the collector and integrate the SDK; everything else is operated for you. - -See [Managed deployment](/agenteye/managed-deployment). - -## Kubernetes - -A full, self-managed production install with high availability, persistent storage, TLS, and your own upgrade cadence. This is the most involved path and the most flexible. - -See [Kubernetes deployment](/agenteye/kubernetes-deployment). - -## Single-pod sidecar - -Run `agenteye-collector` as a sidecar next to a single application pod, shipping that app's events without a cluster-wide install. A good fit when you want to instrument one workload on Kubernetes without operating the whole platform. - -See [Single-pod deployment](/agenteye/single-pod-deployment). - ---- - -> **Tip:** All four paths report into the same dashboard and use the same [Python SDK](/agenteye/python-sdk), [collector](/agenteye/collector-installation), and [API keys](/agenteye/api-keys). Switching paths later does not change how you instrument your agents. - -Whichever path you choose, pulling the AgentEye container images and downloading release artifacts requires signed enterprise access. If you don't have access yet, request a demo: email [nivedit@exosphere.host](mailto:nivedit@exosphere.host), [nikita@exosphere.host](mailto:nikita@exosphere.host), [nivedit@befailproof.ai](mailto:nivedit@befailproof.ai), or [nikita@befailproof.ai](mailto:nikita@befailproof.ai). - ---- - -## Next steps - -- [Getting started](/agenteye/getting-started): the Docker Compose quick path, end to end. -- [Security](/agenteye/security): how AgentEye isolates and protects your data. -- [Collector installation](/agenteye/collector-installation): install the collector on any host. diff --git a/docs/agenteye/deployment.mdx b/docs/agenteye/deployment.mdx deleted file mode 100644 index 0f475462e..000000000 --- a/docs/agenteye/deployment.mdx +++ /dev/null @@ -1,460 +0,0 @@ ---- -title: "Deployment" -description: "AgentEye Deployment documentation." ---- - - -Run AgentEye on your own infrastructure, so your agent data never leaves it. AgentEye is a self-hosted platform for observing, evaluating, and improving your AI agents in production. Use this page for the full picture (the components, the datastores they need, and every configuration knob) plus a copy-paste Docker Compose quick start. - -> **Tip:** In a hurry? Skip to the [Docker Compose quick start](#docker-compose-recommended). It brings up Postgres, the server, and the dashboard with one command. - -## Prerequisites - -Before you start, make sure you have: - -- **PostgreSQL 15+**: the relational store. 15+ is required; see [Architecture overview](#architecture-overview). -- **ClickHouse 24+**: the analytics store for every ingested event. -- **A host for the containers**: Docker with Compose, or a Kubernetes cluster. -- **Object storage** (S3, GCS, or Azure Blob): only if you want automated backups. -- **Signed enterprise access**: the Docker images live in a private registry (`ghcr.io/agenteye-enterprise/*`); pulling them needs a signed enterprise credential. - -Redis 7+ is optional (a shared cache and rate-limit backend); AgentEye runs fine without it. - -## Choose your path - -This page covers the general Docker and Docker Compose deployment plus the complete configuration reference. For a more opinionated route, start with one of these instead: - -- **[Single-pod deployment](/agenteye/single-pod-deployment)**: the whole stack in one pod for a fast evaluation. -- **[Kubernetes deployment](/agenteye/kubernetes-deployment)**: production-grade manifests with backups and scaling. -- **[Managed deployment](/agenteye/managed-deployment)**: let Exosphere run it for you. - ---- - -## Architecture Overview - -```mermaid -flowchart LR - subgraph agents["AI agent machines"] - SDK["Python SDK
writes JSONL"] --> COL["agenteye-collector"] - end - subgraph infra["Your infrastructure"] - DASH["Dashboard"] -->|API| SRV["Server"] - SRV <--> PG[("PostgreSQL 15+
relational store")] - SRV <--> CH[("ClickHouse 24+
events / analytics")] - SRV -.-> RDS[("Redis 7+
optional")] - end - COL -->|HTTP| SRV -``` - -- **Server**: Rust HTTP service; receives event batches, writes them to ClickHouse, and maintains relational state in PostgreSQL. -- **Dashboard**: Next.js web app; reads and writes exclusively through the server API. -- **agenteye-collector**: deployed on agent machines, not the server host. -- **Postgres 15+**: REQUIRED. Upgrade before deploying this version (the multi-tenant schema uses a Postgres 15+ feature and will not run on 14). Stores the transactional (OLTP) tables the app reads and writes constantly: `api_keys`, `users`, `sessions`, `evaluation_jobs` (queue), `dashboards`, `saved_queries`, `otp_codes`, plus the multi-tenant tables `orgs`, `org_memberships`, `org_settings`. -- **ClickHouse 24+**: REQUIRED. The analytics store for every ingested event. Engine: `ReplacingMergeTree`, partitioned by month, ordered by `(session_id, ts, dedup_key)`. The server connects via `CLICKHOUSE_URL`; the bundled `deploy/base/clickhouse/` ships a perf-tuned single-node configuration. **Multi-tenant requirement:** the bundled config enables SQL access management + `users_without_row_policies_can_read_rows=false` so the server can create one read-only ClickHouse user + row policy per organization (the engine-enforced isolation boundary for the SQL editor and the AI assistant). If you supply your own ClickHouse config, carry these settings over (see `deploy/base/clickhouse/configmap.yaml`). -- **Redis 7+**: *optional* shared cache + rate-limit backend. Server and dashboard both connect via `REDIS_URL`. If absent, both degrade gracefully to Postgres-only paths. See **Redis (optional cache)** below. - ---- - -## Server - -### Pull the image - -> **Note:** The images live in a private registry. Pulling them needs signed enterprise access: a `AGENTEYE_TOKEN` credential you receive as a signed customer. If you are still evaluating, skip ahead and read on; the configuration reference below applies once you have access. - -```bash -echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin -docker pull ghcr.io/agenteye-enterprise/server:beta-latest -``` - -> **Note:** Current builds publish under `beta-latest`; `latest` is assigned only to stable releases. For production, pin a specific `:v` tag; see [Available Image Tags](#available-image-tags). - -### Environment variables - -> **Note:** Minimum required to boot: `DATABASE_URL` + `CLICKHOUSE_URL`. Everything else in the table below is optional and has a sensible default. Set only what you need. - -| Variable | Required | Default | Description | -|---|---|---|---| -| `DATABASE_URL` | Yes | none | Postgres connection string in standard libpq DSN format (the ordinary `postgres` URL scheme your Postgres client already uses). Supports `?sslmode=require` and other libpq parameters. The secret in it must not contain `/`, `+`, or `=`; use `openssl rand -hex` to generate a URL-safe one. | -| `ADMIN_KEY` | No | none | Bootstrap admin API key. Upserted with all permissions on every startup. Rotate by changing the value and restarting. | -| `LISTEN_ADDR` | No | `0.0.0.0:8080` | TCP address to bind | -| `MAX_BODY_BYTES` | No | `134217728` (128 MB) | Maximum request body size | -| `ADMIN_EMAIL` | No | none | Bootstrap admin user email. Upserted with all permissions on every startup and marked protected: cannot be disabled or have permissions modified via the dashboard/API. To rotate the bootstrap admin, change `ADMIN_EMAIL` and restart; the new email is upserted as protected, and the previous one retains its protection until manually cleared in the database. | -| `ALLOWED_EMAILS` | No | none (all blocked) | Comma-separated list of allowed emails for user creation and login. Supports exact addresses (`user@example.com`) and domain wildcards (`*@example.com`). If unset, no users can be created or log in. **First-boot seed only**: seeds the default org's allowlist on first boot; thereafter each org's [`//settings`](#operational-settings) page is the source of truth and changing this env var has no effect. | -| `SMTP_HOST` | No | none | SMTP server hostname for sending OTP emails. If unset, OTP codes are logged to stdout instead. | -| `SMTP_PORT` | No | `587` | SMTP server port | -| `SMTP_USERNAME` | No | none | SMTP authentication username | -| `SMTP_PASSWORD` | No | none | SMTP authentication password | -| `SMTP_FROM` | No | none | Sender email address for OTP emails | -| `SMTP_TLS` | No | STARTTLS | STARTTLS is used unless you explicitly turn it off: `false` or `0` sends plaintext (no TLS); any other value (including unset) enables STARTTLS. | -| `DASHBOARD_URL` | No | built-in default | Dashboard origin used to build both the OTP-email magic link and the incident magic-links in alert notifications. If unset it falls back to a built-in default (and, for OTP only, to the dashboard-derived request origin first). Set this for split-domain setups so both email and Slack/incident links point at your dashboard. See **Email magic-link URL** below; most operators do not need to set this. | -| `SESSION_TTL_SECS` | No | `86400` (24 h) | Dashboard session duration in seconds. **First-boot seed only**: edit per org via [`//settings`](#operational-settings) after the first deploy. | -| `OTP_TTL_SECS` | No | `600` (10 min) | OTP code validity period in seconds. **First-boot seed only**: edit per org via [`//settings`](#operational-settings) after the first deploy. | -| `REDIS_URL` | No | none | Optional shared cache + rate-limit backend, e.g. `redis://redis:6379/0`. When set, the server caches authenticated API-key lookups, the dashboard's `/models` aggregate, the sessions list, and the env-list facet; it also moves OTP-request rate limiting off Postgres COUNT and onto Redis INCR. If unset or unreachable, the server runs without the cache (the OTP limit falls back to Postgres, every other cache call falls through to the source of truth). See **Redis (optional cache)** below. | -| `CLICKHOUSE_URL` | **Yes** | none | Base URL of the ClickHouse instance, e.g. `http://clickhouse:8123`. The server applies its events schema to this database on every startup and refuses to boot if it can't reach ClickHouse. See **ClickHouse (required analytics store)** below. | -| `CLICKHOUSE_DATABASE` | No | `agenteye` | ClickHouse database (schema) name. The server creates it on startup if it doesn't exist. | -| `ORG_CH_SECRET` | No (single-tenant) / **Yes (multi-org)** | dev default | HMAC key from which each organization's per-tenant ClickHouse password is derived. The SQL editor and the AI assistant's `run_query` execute as the org's own read-only ClickHouse user, whose row policy enforces tenant isolation in the engine. Single-tenant deployments boot fine on the built-in dev default; **before provisioning a second org you MUST set a strong, stable value**, because the `agenteye-orgctl org create` CLI refuses to run on the built-in dev default. Rotating it orphans every org's ClickHouse user until the next startup re-provisions them (the boot-time reconcile heals this automatically). Keep it secret and unchanged across replicas. Org provisioning itself is operator-only; see **Organizations (multi-tenancy)** below. | -| `DEFAULT_ORG_NAME` | No | `Default` | Display name seeded for the built-in default org. **First-boot seed only**, and only while the org still carries its freshly-migrated generic identity, applied on startup, then ignored. Once you rename the org (`agenteye-orgctl org rename`) the rename is authoritative and this env var has no further effect. | -| `DEFAULT_ORG_SLUG` | No | `default` | URL slug for the built-in default org, the dashboard path it lives at (`//…`). Same first-boot-only / pristine-only semantics as `DEFAULT_ORG_NAME`. Must be 1-40 lowercase alphanumerics with single internal hyphens and not a [reserved word](#organizations-multi-tenancy); an invalid value is ignored (the org keeps `default`). Lets a single-tenant install present as e.g. `/acme` instead of `/default` without any post-deploy CLI step. | -| `RUST_LOG` | No | `info` | Log verbosity (`debug`, `warn`, `error`, `agenteye_server=trace`) | -| `EVALUATOR_ENDPOINT` | No | none | Base URL of your evaluator service (e.g. `http://evaluator:9000`). When unset the entire evaluation pipeline is a no-op; no queue rows are written, no workers run. See [Evaluation Suite](/agenteye/evaluation-suite). | -| `EVALUATOR_TOKEN` | No | none | Sent as `Authorization: Bearer ` to the evaluator. **Must equal the same value the evaluator service is configured with.** Optional only if your evaluator is configured with no token. | -| `EVALUATOR_WORKERS` | No | `2` | Concurrency: number of worker tasks per server instance that dispatch evaluations. Safe to run across multiple horizontally-scaled servers. | -| `EVALUATOR_CLAIM_BATCH` | No | `4` | Maximum number of evaluations a single worker claims per tick. Batches are dispatched **concurrently**, so total concurrency on your evaluator endpoint is `EVALUATOR_WORKERS × EVALUATOR_CLAIM_BATCH`. | -| `EVALUATOR_POLL_IDLE_SECS` | No | `2` | How long a worker sleeps between dispatch attempts when nothing is due. | -| `EVALUATOR_POLLING_INTERVAL_SECS` | No | `10` | Final fallback cadence (seconds) for `GET /evaluate/{id}` polls when the evaluator does not return a per-response `next_poll_secs` and does not advertise a `default_poll_interval_secs` from `GET /config`. | -| `EVALUATOR_REQUEST_TIMEOUT_MS` | No | `30000` | Per-HTTP-request timeout against the evaluator (milliseconds). | -| `EVALUATOR_MAX_ATTEMPTS` | No | `5` | After this many failed attempts an evaluation is recorded as terminal `error` (or `timeout` if the failures were request timeouts). | -| `EVALUATOR_CONFIG_REFRESH_SECS` | No | `300` (5 min) | How often the server re-fetches `GET /config` from the evaluator. | -| `EVALUATOR_MAX_POLL_DURATION_SECS` | No | `3600` (1 h) | Maximum wallclock time a session may remain in the polling queue before AgentEye terminates it as `timeout`. Guards against an evaluator that returns `pending` forever. | -| `ALERT_WORKERS` | No | `1` | Concurrency: number of worker tasks per server instance that evaluate alert rules. See [Alerts](/agenteye/alerts). | -| `ALERT_CLAIM_BATCH` | No | `16` | Maximum number of alerts a single worker claims per tick. | -| `ALERT_POLL_IDLE_SECS` | No | `5` | How long an alerts worker sleeps when the queue is empty. | -| `ALERT_REQUEST_TIMEOUT_MS` | No | `15000` | Per-trigger evaluation timeout (ClickHouse queries + outbound channel HTTP). | -| `ALERT_MAX_ATTEMPTS` | No | `5` | Consecutive transient failures before an alert reschedules at its normal cadence instead of exponential backoff. | -| `AUDIT_WORKERS` | No | `1` | Concurrency: number of worker tasks per server instance that execute audits. See [Audits](/agenteye/audits). | -| `AUDIT_CLAIM_BATCH` | No | `1` | Maximum number of due audits a single worker claims per tick. An agentic investigation is one long loop, so the default is 1. | -| `AUDIT_POLL_IDLE_SECS` | No | `30` | How long an audits worker sleeps when no audit is due. | -| `AUDIT_REQUEST_TIMEOUT_MS` | No | `30000` | Per-policy-query timeout against ClickHouse (milliseconds). | -| `AUDIT_LLM_TIMEOUT_MS` | No | `1440000` | Timeout for the agentic investigation call to the AI assistant service. A full agent loop runs for minutes; keep this ABOVE the agent's own `AGENTEYE_AUDIT_TIMEOUT_MS` so the agent returns its partial findings before the server gives up. | -| `AUDIT_MAX_ATTEMPTS` | No | `5` | Consecutive transient failures before an audit reschedules at its normal cadence instead of exponential backoff. | -| `AGENTEYE_AGENT_URL` / `AGENTEYE_AGENT_TOKEN` | No | none | The audit's agentic investigation calls the AI-assistant `agent` service, **reusing the same connection as the assistant**. Set these two on the **server** as well (the bundled manifests/compose do). Both set ⇒ audits run the AI investigation; either unset ⇒ audits run **policy-only** (the deterministic SQL policy pass still runs), regardless of the per-audit `llm_enabled` flag. The agent must also have an LLM configured. See the [AI assistant](/agenteye/assistant) guide. | - -**AI assistant service: audit + sandbox settings.** The agentic investigation and its in-pod Python sandbox are tuned on the **agent service** (not the server), all on the `AGENTEYE_AUDIT_*` prefix and all optional: - -| Variable | Default | Meaning | -|---|---|---| -| `AGENTEYE_AUDIT_MAX_STEPS` | `200` | Max agent turns per investigation. | -| `AGENTEYE_AUDIT_TIMEOUT_MS` | `1200000` | Wall-clock for one investigation (20 min). Must stay **below** the server's `AUDIT_LLM_TIMEOUT_MS`. | -| `AGENTEYE_AUDIT_MAX_CONCURRENCY` | `1` | Concurrent investigations per agent pod (separate from the chat assistant's budget). | -| `AGENTEYE_AUDIT_SANDBOX_TIMEOUT_MS` / `_MEM_MB` / `_CPU_SECS` / `_OUTPUT_MAX_BYTES` / `_SCRIPT_MAX_BYTES` | `20000` / `768` / `10` / `64000` / `64000` | Per-script limits for the bubblewrap sandbox. | - -**Sandbox platform requirement.** The audit code sandbox runs the model's Python inside a lightweight isolation jail (bubblewrap), which needs the Linux kernel feature called **unprivileged user namespaces**. The agent pod must permit the syscalls it uses: relax the default syscall filter by setting `seccompProfile: Unconfined` (Kubernetes) or `security_opt: [seccomp:unconfined]` (Compose) on the agent. Where the node kernel disables unprivileged user namespaces (e.g. some GKE COS images), the sandbox **preflight fails and the auditor degrades to SQL-only automatically** (no error, just a `sandbox_available: false` on the agent's `/health`). - -### Run - -Set `DATABASE_URL` in your environment, then pass it through to the container: - -```bash -docker run -d --restart unless-stopped \ - --name agenteye-server \ - -e DATABASE_URL="$DATABASE_URL" \ - -e ADMIN_KEY="$ADMIN_KEY" \ - -p 8080:8080 \ - ghcr.io/agenteye-enterprise/server:beta-latest -``` - -The server runs database migrations automatically on startup; no separate migration step needed. - -### Health check - -```text -GET /health # liveness - always {"status":"ok"} once the process is up -GET /ready # readiness - 200 when Postgres + ClickHouse are reachable, else 503 -``` - -No authentication required. Use `/health` for **liveness** probes and `/ready` for **readiness** / load-balancer probes. `/ready` checks the hard dependencies the server cannot serve without (Postgres + ClickHouse), so a server that is running but cannot reach its database is taken out of rotation and shows as `NotReady`; Redis is reported but never fails readiness. On the bundled Kubernetes manifests the readiness probe already points at `/ready` and liveness stays on `/health`. See [Health monitoring](/agenteye/health-monitoring) for the full picture, including opt-in Kubernetes-native pod-failure alerting to Slack. - -### Email magic-link URL - -OTP login emails contain a one-tap **open the dashboard** button. Clicking it lands the user on `/login?token=&email=
`; the dashboard exchanges that pair for a session and redirects to the app, with no manual code re-entry. The server resolves the dashboard origin used to build the link in three tiers: - -1. **`X-AgentEye-Dashboard-Url` header**: set automatically by the dashboard's `/api/auth/otp/request` proxy from its own public origin. In a same-origin deployment (server and dashboard share a host behind one ingress that forwards proxy headers), **no configuration is required**. -2. **`DASHBOARD_URL` env var**: set this if your dashboard is reachable on a different origin than the one the server's OTP request endpoint sees (split `api.example.com` / `app.example.com`), or if your ingress doesn't propagate the public host into the dashboard pod (so `request.nextUrl.origin` would otherwise resolve to a wildcard bind like `0.0.0.0:3000`). Example: `DASHBOARD_URL=https://app.example.com`. -3. **Default**: `https://app.befailproof.ai`, used only if neither of the above is present. - -The header value is validated: only `https://*` and loopback (`http://localhost*`, `http://127.0.0.1*`) origins are accepted, and wildcard bind addresses (`0.0.0.0`, `[::]`) are rejected even with the `https://` scheme. Anything else falls through to tier 2. - -Set it on a running cluster with a one-liner; no file, no kustomize rebuild: - -```bash -kubectl set env deployment/server -n agenteye \ - DASHBOARD_URL=https://app.example.com -``` - -This triggers a rollout; the new pods pick the value up on first request. Note that the override lives only on the Deployment; a subsequent `kustomize build | kubectl apply` against the overlay will wipe it unless you add the same env var to your overlay's `server-env.yaml` patch. - ---- - -## Dashboard - -### Pull the image - -```bash -docker pull ghcr.io/agenteye-enterprise/dashboard:beta-latest -``` - -### Environment variables - -| Variable | Required | Default | Description | -|---|---|---|---| -| `AGENTEYE_SERVER_URL` | Yes | none | Base URL of the server, e.g. `http://localhost:8080` | -| `AGENTEYE_API_KEY` | Yes | none | API key the dashboard uses to authenticate to the server. Needs all permissions (admin key recommended). | -| `AE_LOG_LEVEL` | No | `info` | Server-side log verbosity: `debug`, `info`, `warn`, `error`. Set to `debug` to see upstream-request/response lines and session-validation traces when diagnosing issues. | -| `AE_LOG_JSON` | No | auto | `1` forces JSON-per-line output; `0` forces human-readable output. When unset, JSON is enabled automatically if `NODE_ENV=production`. JSON is recommended in production so logs parse cleanly with `jq` or a log aggregator. | -| `AE_ANALYTICS_DISABLED` | No | none | Set to `1`/`true` to disable the dashboard's anonymous product-usage telemetry. See [Telemetry & privacy](#telemetry--privacy) below. | -| `REDIS_URL` | No | none | Optional shared cache backend, e.g. `redis://redis:6379/0`. When set, the dashboard caches `validateSession()` results across replicas and shares the Next.js fetch cache for the latency-aggregate / env-list proxy routes. Edge-side OTP request and verify rate limits also use Redis when present (falling open if Redis is unreachable; the server-side limit is the security backstop). See **Redis (optional cache)** below. | -| `AGENTEYE_AGENT_URL` | No | none | Base URL of the optional AI-assistant `agent` service, e.g. `http://agent:9100`. **Leave it unset to hide the assistant entirely**: no assistant bubble appears in the dashboard. See the [AI assistant](/agenteye/assistant) guide. | -| `AGENTEYE_AGENT_TOKEN` | No | none | Shared secret the dashboard presents to the `agent` service. Must match the `AGENTEYE_AGENT_TOKEN` configured on the agent. See the [AI assistant](/agenteye/assistant) guide. | - -### Run - -```bash -docker run -d --restart unless-stopped \ - --name agenteye-dashboard \ - -e AGENTEYE_SERVER_URL="http://your-server-host:8080" \ - -e AGENTEYE_API_KEY="$ADMIN_KEY" \ - -p 3000:3000 \ - ghcr.io/agenteye-enterprise/dashboard:beta-latest -``` - -### Telemetry & privacy - -The dashboard sends **anonymous product-usage analytics** to Exosphere's analytics service (PostHog): which dashboard pages are viewed and a handful of UI actions such as creating an API key or re-evaluating a session. This usage signal informs which features are prioritised. - -- **No agent, session, or event data ever leaves your infrastructure.** Only dashboard UI usage is reported. Page URLs are stripped of identifiers before sending, and operators are identified only by an opaque internal id, never by email. -- Telemetry is **enabled by default**. To turn it off completely, set `AE_ANALYTICS_DISABLED=1` on the dashboard container and restart. -- Analytics are sent to the dashboard's own `/ingest` path, which the dashboard reverse-proxies to PostHog (`https://us.i.posthog.com`). Keeping requests first-party means browser ad-blockers don't drop them. The **dashboard container** needs outbound access to PostHog; if it's blocked, telemetry silently does nothing and the dashboard is unaffected. - ---- - -## AI Assistant (optional) - -An in-dashboard AI assistant lets your team ask questions of their agent data in plain language (summarising sessions, drafting SQL for the `/queries` editor, and turning saved queries into dashboard tiles) without leaving the dashboard. It runs as a separate internal `agent` container (on the Claude Agent SDK) that only the dashboard can reach, and stays **disabled until you configure an LLM endpoint**. - -To enable it you set, on the `agent` service, an LLM connection (**Portkey** via `PORTKEY_API_KEY` + a model-catalog slug `AGENTEYE_AGENT_MODEL=@/`, direct Anthropic via `ANTHROPIC_API_KEY`, another gateway via `ANTHROPIC_BASE_URL`, or Bedrock/Vertex), a **dedicated** data key, and a shared `AGENTEYE_AGENT_TOKEN` matching the dashboard. Dashboard users additionally need the `agent:use` permission. - -For the assistant's data key you don't mint anything by hand: pick a random secret, set it as `AGENTEYE_API_KEY` on the `agent` **and** as `AGENT_API_KEY` on the `server`, and the server seeds it on startup with a fixed permission set. Its data access is read-only (`events:read`, `evaluations:read`, `dashboards:read`, `queries:read`), and it additionally holds approval-gated authoring scopes (`dashboards:write`, `queries:write`, `queries:run`) so it can draft and validate saved queries and build dashboard tiles on the user's behalf; all SQL still runs through the org's read-only ClickHouse role, so this widens what the assistant can author, not what data it can reach. The scopes are fixed in code and cannot be widened by configuration. That key is protected; it can't be disabled or regenerated via the API, only rotated by changing the value and restarting. Never reuse the admin/dashboard key for this. - -Full setup, the complete environment-variable reference, telemetry options, and the security model are in the **[AI assistant](/agenteye/assistant)** guide. - ---- - -## ClickHouse (required analytics store) - -ClickHouse keeps your dashboards responsive at high event volumes and lets the `/queries` SQL editor join across events, evaluations, and sessions in a single store. It is the required canonical store for every ingested event, every terminal evaluation outcome, and the derived per-session aggregates. PostgreSQL holds the relational / mutable-state tables (api_keys, users, otp_codes, evaluation_jobs, dashboards, saved_queries); the analytical surface lives in ClickHouse so the dashboard's rollups and your own SQL queries can scan and join it natively, without cross-database round-trips. The server refuses to boot without `CLICKHOUSE_URL`. - -### Schema - -Three ClickHouse objects are created at server startup, all idempotent (`CREATE IF NOT EXISTS`): - -- **`agenteye.events`**: `ReplacingMergeTree(ingested_at)`, partitioned by `toYYYYMM(ts)`, ordered by `(session_id, ts, dedup_key)`. Duplicate inserts (collector retries) collapse to a single row at merge time; the server computes a deterministic SHA-256 `dedup_key` for every event so retries are safe. -- **`agenteye.evaluations`**: `ReplacingMergeTree(ingested_at)`, partitioned by `toYYYYMM(finished_at)`, ordered by `(session_id, finished_at, dedup_key)`. Written once per terminal evaluation outcome by the evaluator pipeline. Same dedup-key model as `events`. -- **`agenteye.agent_sessions`**: a **VIEW** over `agenteye.events`, not a physical table. Every column is derived (`started_at = min(ts)`, `last_event_at = max(ts)`, `ended_at = max(if event_type='agent_end', ts, NULL)`, `event_count = count()`, etc.). No per-event upsert and no separate backfill; the view auto-reflects whatever is in `events`. - -For backwards-compat with saved queries that reference `analytics.evaluations` / `analytics.sessions`, the server also creates an `analytics` ClickHouse database with views over the `agenteye.*` tables; `analytics.events`, `analytics.evaluations`, `analytics.agent_sessions`, `analytics.sessions` all resolve correctly. - -### Configuration - -The bundled docker-compose and `deploy/base/clickhouse/` ship a ClickHouse service tuned for AgentEye's workload: - -- 2 GiB requested / 4 GiB limit memory in the shipped base overlay (sized to fit small POC/staging nodes); production customers should overlay up. The recommended floor is 2c / 4Gi request, 6c / 8Gi limit. `max_server_memory_usage_to_ram_ratio=0.9` -- 256 MiB mark cache + 256 MiB uncompressed cache (sized to coexist with the 4 GiB pod limit) -- `background_pool_size=16`, `background_merges_mutations_concurrency_ratio=2` -- MergeTree: `parts_to_throw_insert=3000`, `parts_to_delay_insert=1500`, `non_replicated_deduplication_window=1000` -- `local_io_method=auto` (io_uring on supported kernels) -- `fsync_metadata=0`: acceptable because of at-least-once ingest + ReplacingMergeTree dedup -- `query_log` enabled with 30-day TTL; `query_thread_log` removed (expensive at high QPS) -- Query time caps: the default user profile caps at 60s; the read-only profile used for untrusted `/queries/run` caps at 10s -- 100 GiB persistent volume (PVC) attached to the ClickHouse StatefulSet, the Kubernetes controller that gives ClickHouse stable per-pod storage. Customer overlays SHOULD override it to a fast SSD storage class for production. - -### Backups - -Your full dataset is captured nightly in a single restorable archive, so a cluster or storage loss is recoverable. ClickHouse is backed up automatically by the daily `agenteye-backup` CronJob, which dumps both PostgreSQL and ClickHouse in one pass. ClickHouse is read over its HTTP API: `agenteye.events` and `agenteye.evaluations` are dumped in ClickHouse-native format (the views and row policies are recreated by the server on startup, so the table data is the complete picture) and bundled with the Postgres dump into a single compressed archive uploaded to your object storage. - -The destination bucket and cloud credentials are configured per overlay. See the **Backups** section of [Kubernetes deployment](/agenteye/kubernetes-deployment) for upload configuration and restore steps. - ---- - -## Redis (optional cache) - -Redis is an **optional** shared cache + rate-limit backend used by the server and dashboard. With Redis deployed and `REDIS_URL` set on both services: - -- **Server** caches authenticated API-key lookups, the `/events/environments` + `/evaluations/environments` lists, the `/events/latency_aggregate` rollup (the heaviest query the dashboard polls), the `/sessions` list, and switches OTP-request rate limiting from a Postgres `COUNT(*)` to a Redis `INCR + EXPIRE`. -- **Dashboard** caches `validateSession()` results so the 10-20 authed API calls a typical page load issues all share one upstream session check. It also rate-limits OTP-request and OTP-verify at the dashboard edge. - -**Both services degrade gracefully if Redis is unreachable.** Every cache call returns `Err` within a bounded timeout and the caller falls back to the source of truth (Postgres on the server, the upstream Rust server on the dashboard). OTP rate limiting falls back to the Postgres `COUNT(*)` path on the server (the security property is preserved); the dashboard's edge OTP limit fails open while the server-side limit still holds. Redis being down degrades latency, not correctness. - -### Configuration - -The docker-compose bundle already includes a Redis service and wires `REDIS_URL=redis://redis:6379/0` into the server and dashboard. To use an external Redis, set `REDIS_URL` to your endpoint and remove the `redis` service from the compose file. - -### Memory + persistence - -The bundled Redis image runs with `--appendonly yes --appendfsync everysec --maxmemory 256mb --maxmemory-policy allkeys-lru`. AOF persistence means the cache survives container restarts; `everysec` is the right durability/perf balance because losing the last second of cache writes is harmless. LRU eviction caps memory growth. - -### When NOT to deploy Redis - -- Single-instance dev/QA. The in-process caches on the server alone deliver most of the per-replica benefit; Redis adds the cross-replica sharing that single-instance setups don't need. -- Air-gapped installs where the operational cost of running one more service outweighs the latency win. - ---- - -## Docker Compose (recommended) - -A `docker-compose.yml` is available in the `agenteye-enterprise/releases` repo. It brings up Postgres, the server, and the dashboard with a single command. - -```bash -GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download \ - --repo agenteye-enterprise/releases \ - --pattern 'docker-compose.yml' \ - --dir ./agenteye -cd agenteye -``` - -**Override defaults via `.env`:** - -```ini -# Use URL-safe passwords (no /, +, or = characters). -# Generate with: openssl rand -hex 24 -POSTGRES_PASSWORD=your-db-password -ADMIN_KEY=your-admin-secret - -# Dashboard authentication -ADMIN_EMAIL=admin@yourcompany.com -ALLOWED_EMAILS=*@yourcompany.com - -# SMTP for OTP emails (omit to log OTP codes to stdout) -# SMTP_HOST=smtp.yourprovider.com -# SMTP_PORT=587 -# SMTP_USERNAME=your-smtp-user -# SMTP_PASSWORD=your-smtp-password -# SMTP_FROM=noreply@yourcompany.com - -RUST_LOG=info -``` - -```bash -docker compose up -d -``` - -**First login.** Once the stack is up, open the dashboard at `http://localhost:3000`, enter an email that matches `ALLOWED_EMAILS`, and request a one-time code. If you didn't configure SMTP, the code is printed to the server logs. Read it with `docker compose logs server`. Enter the code and you land in the app. For a guided first-run walkthrough, see [Getting started](/agenteye/getting-started). - -**Stop (keeps data volume):** - -```bash -docker compose down -``` - -**Stop and wipe all data:** - -```bash -docker compose down -v -``` - ---- - -## Operational settings - -A small set of operational knobs that used to be pinned by env vars are now editable per organization from the dashboard's **`//settings`** page; each org configures its own. Changes take effect within seconds, with no restart and no redeploy. - -![The per-org settings page, where operational knobs like allowed sign-ins and session lifetime are edited without a restart](/agenteye/images/settings.png) - -| Setting | Bootstrap env var | What it controls | -|---|---|---| -| Allowed sign-ins | `ALLOWED_EMAILS` | Emails (or `*@domain.com` wildcards) permitted to receive an OTP and be added as users | -| Default user permissions | `DEFAULT_USER_PERMISSIONS` | The **name** of the permission set preselected when an admin opens **+ new user** (not a list of individual tokens). Defaults to `standard`: read-only access plus the everyday on-call actions (trigger re-evaluations, run queries, ack incidents, use the assistant). See [API keys](/agenteye/api-keys) for the permission sets available. | -| Session lifetime | `SESSION_TTL_SECS` | How long a dashboard login stays valid before re-auth. The dashboard re-checks the upstream session every 5 seconds, so a permission update on `//users` takes effect on the affected user's next request, with no relogin. | -| One-time-code lifetime | `OTP_TTL_SECS` | How long an OTP / magic-link stays usable | -| Alert notification channels | `ALERTS_ENABLED_CHANNELS` | Comma-separated list of channel kinds the alert dispatcher is allowed to use: `email`, `slack`, `webhook`. Per-alert configuration is still authored on `//alerts/`, but the dispatcher filters every outbound delivery through this set; a channel disabled here short-circuits with a `skipped_disabled` audit row. The `dashboard` channel (the local audit insert) is always allowed. Defaults to all three on. | - -### How the bootstrap works - -Settings are stored per organization in `org_settings`. On first boot, the server seeds the default org's missing rows from the matching env var (or a sensible default if the env var is unset). After that, **the stored value is the source of truth and the env var is ignored**; changing the env var on a later restart will not affect a live org's value, and additional orgs start from defaults and configure their own. - -This means: - -- For a fresh deploy, set the env vars as shown above and the default org reads them on first boot. -- To change a value later, log into the dashboard and edit it under `//settings`. The change applies within seconds across all server replicas; no restart needed. -- A startup log line records what got seeded vs. what was already present, so you can confirm the bootstrap took effect: - - ```text - INFO settings bootstrap: seeded default-org row key=allowed_sign_ins env_var=ALLOWED_EMAILS seeded_from_env=true - ``` - -#### Sign-in semantics across organizations - -A session and an OTP are global to the user, not to a single org, so two rules reconcile per-org settings at sign-in time: - -- **Session / OTP lifetime**: the strictest (shortest) lifetime among the orgs the user belongs to wins. -- **Allowed sign-ins**: the gate ORs every org's allowlist together with org membership: a user may request an OTP if any org's allowlist admits their email **or** they are already a member of any org. - -### Permissions - -Access to a `//settings` page is gated by two permissions: - -- `settings:read`: see the page and current values. -- `settings:write`: save changes. - -The bootstrap admin user (seeded from `ADMIN_EMAIL`) gets both automatically along with every other permission. Grant them to other users from `//users` as needed. - ---- - -## Organizations (multi-tenancy) - -A single deployment can serve multiple isolated **organizations** (tenants); every row of data belongs to exactly one org and isolation is enforced in the database engine. A single-tenant install needs nothing here; all data lives in a built-in `default` org. (You can give that org a friendlier name and URL slug, so it lives at e.g. `/acme` instead of `/default`, by setting `DEFAULT_ORG_NAME` / `DEFAULT_ORG_SLUG` before the first boot, or by renaming it anytime with `agenteye-orgctl org rename`.) - -**Tenant provisioning is operator-only.** Organizations and their memberships are created and managed with the **`agenteye-orgctl`** CLI, which ships **inside the server image** (alongside `agenteye-server`) and runs **inside the existing server pod**; there is **no separate pod/Job, no HTTP API, and no dashboard button**. It reuses the server's `DATABASE_URL`, `CLICKHOUSE_URL`, and `ORG_CH_SECRET`. - -```bash -# Docker Compose - exec into the running server service: -docker compose exec server agenteye-orgctl org create --slug acme --name "Acme Corp" -docker compose exec server agenteye-orgctl member add --org acme --email alice@acme.example --set admin - -# Kubernetes - exec into the running server Deployment: -kubectl -n agenteye exec deploy/server -- agenteye-orgctl org list -``` - -Available verbs: `org create | list | rename | delete | purge` and `member add | list | update | remove`, with builtin permission sets `admin`, `standard`, and `read-only`. Added members get an OTP on first dashboard login. - -**Before creating a second org:** set a strong, stable `ORG_CH_SECRET` (the `org create` command refuses to run on the built-in dev default) and ensure Postgres is **15+**. **Unchanged:** per-org API keys are still minted in the dashboard/API by org members; only the org + member lifecycle moved to the CLI. Full command reference and a worked example: **[Tenant management](/agenteye/tenant-management)**. - ---- - -## Context-window fill - -Each `model_response` event shows a **context-fill pill**: input plus output tokens as a percentage of that model's context window. The bands are `healthy` (0-24%), `watch` (25-49%), `compacting` (50-74%), and `reset context` (75-100%). AgentEye resolves common model IDs automatically, so no initial configuration is required. - -Every model an organization sends appears under **Settings → model context windows**. Users with `settings:write` can override its window or add a private/proxy model (0-1,000,000 tokens); `0` means “unknown” and suppresses the pill. Changes apply to newly ingested events. Users with `settings:read` can view the list. - -New events get the fill from the moment you upgrade. To also populate **historical** events (and the per-model list) for an existing deployment, run the one-off backfill. It ships inside the server image (like `agenteye-orgctl`) and runs in the existing server pod: - -```bash -# preview (prints the per-org mutation, changes nothing): -kubectl -n agenteye exec deploy/server -- agenteye-backfill-context-window --dry-run -# apply: -kubectl -n agenteye exec deploy/server -- agenteye-backfill-context-window -# docker compose: -docker compose exec server agenteye-backfill-context-window -``` - -It is idempotent (safe to re-run) and re-uses `DATABASE_URL` / `CLICKHOUSE_URL` / `REDIS_URL` from the pod. Re-run it after editing model windows if you want existing events recomputed. - ---- - -## Production Considerations - -- **Postgres**: Use a managed Postgres service or a dedicated instance with regular backups. The `DATABASE_URL` supports all standard libpq parameters, including `sslmode=require` for encrypted connections. -- **TLS**: Put the server and dashboard behind a reverse proxy (nginx, Caddy, Traefik) that terminates TLS. -- **Firewall**: The server port (default 8080) should only be reachable from collector machines and the dashboard host, not the public internet. -- **Admin key**: Set `ADMIN_KEY` to a strong random secret. After bootstrapping, create dedicated scoped keys for collectors and the dashboard rather than using the admin key everywhere. -- **Image tags**: Pin to the version in your release manifests (for example, `server:v0.0.1-beta.48`) in production rather than a floating tag to avoid unintended upgrades. Current beta builds publish under `beta-latest`; `latest` is assigned only to stable releases. -- **Health monitoring**: On Kubernetes the readiness probe uses `/ready` (Postgres + ClickHouse reachability) while liveness stays on `/health`. For fleet-wide "is AgentEye itself up?" alerting to Slack, enable the opt-in Robusta add-on; see [Health monitoring](/agenteye/health-monitoring). - ---- - -## Available Image Tags - -| Tag | Description | -|-----|-------------| -| `latest` | Latest stable release | -| `beta-latest` | Latest pre-release (beta) | -| `v` | Pinned version, e.g. `v0.0.1-beta.48` (recommended for production) | - ---- - -## Next steps - -- [Getting started](/agenteye/getting-started): send your first events and confirm they land in the dashboard. -- [Collector installation](/agenteye/collector-installation): install `agenteye-collector` on your agent machines. -- [Kubernetes deployment](/agenteye/kubernetes-deployment): production manifests, backups, and scaling. -- [Tenant management](/agenteye/tenant-management): provision additional organizations on one deployment. diff --git a/docs/agenteye/error-tracking.mdx b/docs/agenteye/error-tracking.mdx index e2d51cfdc..ac110a3d6 100644 --- a/docs/agenteye/error-tracking.mdx +++ b/docs/agenteye/error-tracking.mdx @@ -1,6 +1,6 @@ --- title: "Error Tracking" -description: "AgentEye Error Tracking documentation." +description: "FailproofAI Observability Error Tracking documentation." --- @@ -13,7 +13,7 @@ See every failure your agents produce in one place, grouped so a noisy burst rea When an agent breaks, you should not have to scroll a live event stream hoping to catch the red rows before they scroll away. The **Errors** page does the collecting for you. It pulls together everything the dashboard would paint red into one triage surface, so the first thing you see is what is failing, not where to go looking for it. -And it catches more than the obvious ones. Alongside explicit `error` events, AgentEye surfaces the quiet failures too: any `tool_result`, `hook_completed`, or `agent_end` whose payload carries a failure shows up here. A tool that returned an error, or a hook that exited badly, no longer slips past you just because nothing threw a loud exception. +And it catches more than the obvious ones. Alongside explicit `error` events, FailproofAI Observability surfaces the quiet failures too: any `tool_result`, `hook_completed`, or `agent_end` whose payload carries a failure shows up here. A tool that returned an error, or a hook that exited badly, no longer slips past you just because nothing threw a loud exception. Across the top, a histogram plots errors over time. One look tells you whether this is a steady background trickle or a spike that started a few minutes ago, so you know right away whether to drop what you are doing. @@ -23,13 +23,13 @@ Like every observe surface, the Errors page is scoped to your organization and f A single broken dependency can fire the same error hundreds of times a minute. Left raw, that is a wall of near-identical lines that buries the one thing you actually need to see. -AgentEye collapses repeat failures that share the same session and error type into a single row. A burst reads as one incident. You end up counting problems, not log lines, and the signal that matters stays on top instead of being drowned out by its own volume. +FailproofAI Observability collapses repeat failures that share the same session and error type into a single row. A burst reads as one incident. You end up counting problems, not log lines, and the signal that matters stays on top instead of being drowned out by its own volume. ## From "something is red" to the exact event Click any row to land straight inside that run's session, positioned on the exact event that failed. No copying session IDs, no scrolling to hunt for the moment it went wrong: you arrive right on it, with the full execution graph one glance away so you can see what the agent did in the moments before it broke. -If you have `alerts:write`, every row also carries a **+ alert** button. Click it and AgentEye opens a new alert rule already filled in to catch that same failure again. The incident you just triaged becomes the one that pages you next time, instead of surprising you twice. +If you have `alerts:write`, every row also carries a **+ alert** button. Click it and Observability opens a new alert rule already filled in to catch that same failure again. The incident you just triaged becomes the one that pages you next time, instead of surprising you twice. **Where to find it:** the **Errors** page lives in the observe section of the dashboard, at `//errors`. @@ -38,4 +38,4 @@ If you have `alerts:write`, every row also carries a **+ alert** button. Click i - [Alerts](/agenteye/alerts): turn any failure into a paging rule. - [Incidents](/agenteye/incidents): track a firing alert from open to resolved. - [Sessions](/agenteye/sessions): open the full run behind any error. -- [Audits](/agenteye/audits): let AgentEye find failure patterns across your runs for you. +- [Audits](/agenteye/audits): let Observability find failure patterns across your runs for you. diff --git a/docs/agenteye/evaluation-suite.mdx b/docs/agenteye/evaluation-suite.mdx index f5300803e..ab1b4aefc 100644 --- a/docs/agenteye/evaluation-suite.mdx +++ b/docs/agenteye/evaluation-suite.mdx @@ -1,17 +1,17 @@ --- title: "Evaluation Suite" -description: "AgentEye Evaluation Suite documentation." +description: "FailproofAI Observability Evaluation Suite documentation." --- -AgentEye can automatically score every finished agent run for quality: you supply a small scoring service, and AgentEye handles the rest. Use it to track the dimensions you care about (helpfulness, tool efficiency, factuality, safety; you choose), catch regressions early, and compare agents or environments at a glance. Scoring is opt-in: the pipeline does nothing until you set `EVALUATOR_ENDPOINT` on the server. +FailproofAI Observability can automatically score every finished agent run for quality: you supply a small scoring service, and Observability handles the rest. Use it to track the dimensions you care about (helpfulness, tool efficiency, factuality, safety; you choose), catch regressions early, and compare agents or environments at a glance. Scoring is opt-in: the pipeline does nothing until you set `EVALUATOR_ENDPOINT` on the server. -> **Note:** You define the score dimensions. Your evaluator can return any numeric keys it likes; AgentEye stores, trends, and displays whatever you send back. +> **Note:** You define the score dimensions. Your evaluator can return any numeric keys it likes; Observability stores, trends, and displays whatever you send back. ## At a glance -1. **Write a scorer.** Stand up a small HTTP service that reads a session transcript and returns scores. AgentEye ships a working reference you can copy. See [Writing an evaluator with the SDK](#writing-an-evaluator-with-the-sdk). -2. **Point AgentEye at it.** Set `EVALUATOR_ENDPOINT` (and a shared `EVALUATOR_TOKEN`) on the server process. +1. **Write a scorer.** Stand up a small HTTP service that reads a session transcript and returns scores. Observability ships a working reference you can copy. See [Writing an evaluator with the SDK](#writing-an-evaluator-with-the-sdk). +2. **Point Observability at it.** Set `EVALUATOR_ENDPOINT` (and a shared `EVALUATOR_TOKEN`) on the server process. 3. **Watch the scores land.** Every completed session is scored automatically; results show up on the session detail page, the sessions grid, and saved dashboards. ![A session detail view with the evaluation summary, per-dimension score bars, and reasoning text in the right rail](/agenteye/images/session-detail.png) @@ -24,7 +24,7 @@ AgentEye can automatically score every finished agent run for quality: you suppl ```mermaid flowchart LR - ING["ingest /events
agent_end"] --> SRV["AgentEye server"] + ING["ingest /events
agent_end"] --> SRV["Observability server"] SRV -->|"POST /evaluate"| EV["Evaluator service"] EV -->|"done or pending"| SRV SRV -->|"poll GET /evaluate/{job_id}"| EV @@ -32,26 +32,26 @@ flowchart LR SRV --> RES["evaluations
terminal results"] ``` -When the AgentEye SDK emits an `agent_end` event for a session, the server +When the Observability SDK emits an `agent_end` event for a session, the server schedules an evaluation. It then POSTs the full event transcript to your evaluator service, which can either: - **Return the result inline** with `{"status":"done", "scores":{...}, "reasoning":{...}, "summary":"..."}`. The result is appended to the session's evaluation timeline. `reasoning` and `summary` are optional. -- **Defer** with `{"status":"pending", "job_id":"abc-123"}`. AgentEye then +- **Defer** with `{"status":"pending", "job_id":"abc-123"}`. Observability then calls `GET {EVALUATOR_ENDPOINT}/evaluate/abc-123` until your evaluator returns `{"status":"done", ...}` or `{"status":"error", "error":"..."}`. The polling cadence is per-job: a `pending` response may include - `next_poll_secs` to override; otherwise AgentEye uses the + `next_poll_secs` to override; otherwise Observability uses the `default_poll_interval_secs` value from `GET /config`; otherwise the server falls back to `EVALUATOR_POLLING_INTERVAL_SECS` (default 10s). All values are clamped to [1s, 1h]. Sessions that never emit `agent_end` (for example, a crashed agent process) can also be picked up: the evaluator's `GET /config` may return -`{"inactivity_timeout_secs": 1800}`, and AgentEye will evaluate any session +`{"inactivity_timeout_secs": 1800}`, and Observability will evaluate any session that has gone idle for that long. Set the field to `null` or omit it to disable this fallback. @@ -74,7 +74,7 @@ past `inactivity_timeout_secs`, a fresh evaluation is enqueued. Transient failures (5xx, 429, timeouts, network errors) are retried with exponential backoff up to `EVALUATOR_MAX_ATTEMPTS`; 4xx responses are -terminal. AgentEye is safe to run with multiple horizontally-scaled server +terminal. Observability is safe to run with multiple horizontally-scaled server instances; work is partitioned so the same session is never dispatched twice concurrently. @@ -85,7 +85,7 @@ twice concurrently. Every authenticated route uses **bearer token auth**. The same value must be configured on both sides: -- AgentEye server: env var `EVALUATOR_TOKEN` +- Observability server: env var `EVALUATOR_TOKEN` - Evaluator service: configured the same way (the `agenteye-evaluator` SDK reads `EVALUATOR_TOKEN` by convention) @@ -169,18 +169,10 @@ You don't have to implement the HTTP contract by hand. The `agenteye-evaluator` Python package gives you a typed FastAPI wrapper that handles auth, routing, and the request/response shapes for you. -AgentEye also ships a **working reference evaluator** -(`deploy/examples/evaluator/evaluator.py`) that scores `helpfulness`, -`tool_efficiency`, and `factuality` from the shape of the transcript. Copy it as -a starting point and swap in your own logic: an LLM judge, a rule engine, -whatever fits your quality bar. - -> **Note:** The `agenteye-evaluator` SDK is delivered to signed enterprise -> customers as a wheel/sdist inside each release tarball on the -> [agenteye-enterprise releases page](https://github.com/agenteye-enterprise/releases). -> It is not published to public PyPI. Install it from the wheel you download, -> or from the bundled `evaluator-sdk/` source, as the reference Dockerfile below -> does. +FailproofAI Observability also ships a **working reference evaluator** that +scores `helpfulness`, `tool_efficiency`, and `factuality` from the shape of the +transcript. Copy it as a starting point and swap in your own logic: an LLM +judge, a rule engine, whatever fits your quality bar. Minimum viable evaluator: @@ -204,108 +196,31 @@ def run(req: EvalRequest) -> EvalResponse: The `app` instance runs under any ASGI server, so `uvicorn module:app` starts it. For evaluators that need to defer expensive work, return `JobPending` -instead and register a `@app.job_lookup` handler; the AgentEye server +instead and register a `@app.job_lookup` handler; the Observability server polls `GET /evaluate/{job_id}` until you return a terminal status or the `EVALUATOR_MAX_POLL_DURATION_SECS` cap (default 1 h) elapses. -The full API reference, async pattern, and event schema ship with the SDK: the -`agenteye-evaluator` README is included in each release tarball on the -[agenteye-enterprise releases page](https://github.com/agenteye-enterprise/releases). +The full API reference, async pattern, and event schema are documented in the +`agenteye-evaluator` SDK's README. --- -## Running an evaluator on Kubernetes - -The evaluator is **your service**: AgentEye does not ship a default -evaluator container. The signed enterprise release includes reference Kubernetes -manifests under `deploy/examples/evaluator/` that you can apply as-is after -swapping in your image and a shared bearer token. - -> **Note:** The `deploy/` example bundle referenced throughout this section ships -> with the signed enterprise release. If you're reading from the public docs and -> don't have it yet, treat the manifests below as a template. The important -> shapes (Deployment, Service, secret) are shown inline. - -### 1. Containerize your evaluator - -A minimal Dockerfile for your evaluator: - -```dockerfile -FROM python:3.12-slim -WORKDIR /app -# Install the agenteye-evaluator SDK from the bundled source that ships in your -# release (it is not on public PyPI). Build with the repo root as the context so -# this COPY can see the evaluator-sdk/ directory. -COPY evaluator-sdk /app/evaluator-sdk -RUN pip install --no-cache-dir /app/evaluator-sdk 'uvicorn[standard]>=0.30' -COPY my_evaluator.py . -RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin evaluator \ - && chown -R evaluator:evaluator /app -USER evaluator -EXPOSE 9000 -CMD ["uvicorn", "my_evaluator:app", "--host", "0.0.0.0", "--port", "9000"] -``` - -`runAsNonRoot` (UID 10001) keeps the container compatible with Pod -Security restricted profiles. - -### 2. Create the shared bearer token - -```bash -kubectl -n agenteye create secret generic evaluator-token \ - --from-literal=token="$(openssl rand -hex 32)" -``` - -Use the same value as `EVALUATOR_TOKEN` on the AgentEye server. The -server sends `Authorization: Bearer ` on every request; the SDK -uses `hmac.compare_digest` for a constant-time check and rejects -mismatches with HTTP 401. - -### 3. Apply the example manifests - -```bash -# Edit deploy/examples/evaluator/deployment.yaml first to point -# `image:` at your registry, then: -kubectl apply -k deploy/examples/evaluator/ -``` - -The example includes: - -- A 2-replica Deployment with `runAsNonRoot`, read-only root filesystem, - all capabilities dropped, liveness + readiness on `/health` -- A ClusterIP Service on port 9000 -- A `secret.example.yaml` template (intentionally excluded from the - Kustomization; create the real secret out-of-band so no token lands - in git) - -### 4. Wire AgentEye to it - -On the AgentEye server, set: - -```bash -EVALUATOR_ENDPOINT=http://evaluator:9000 -EVALUATOR_TOKEN= -``` - -The server fans out `EVALUATOR_WORKERS × EVALUATOR_CLAIM_BATCH` -concurrent requests across all evaluator pods (defaults: `2 × 4 = 8`). -Scale `replicas` and per-pod resource limits in concert with these -server-side knobs. +## Running your evaluator -### Verification +The evaluator is **your service** — FailproofAI Observability does not ship a +default evaluator, so you build and run it wherever you run your own services. +It runs under any ASGI server (for example `uvicorn my_evaluator:app`); serve +the `/health`, `/config`, and `/evaluate` routes from the +[HTTP contract](#http-contract), then point the server at it (see +[Configuring the server](#configuring-the-server)). -```bash -kubectl -n agenteye port-forward svc/evaluator 9000:9000 -curl -s http://localhost:9000/health # → {"status":"ok"} -``` - -After an agent runs end-to-end, `GET /evaluations` on the AgentEye -server should return a row with `status: "done"` and the scores your -evaluator produced. +Once the evaluator is reachable, `GET /health` returns `{"status":"ok"}`. After +an agent runs end-to-end, `GET /evaluations` on the server returns a row with +`status: "done"` and the scores your evaluator produced. --- -## Configuring the AgentEye server +## Configuring the server Set on the server process: @@ -322,11 +237,12 @@ Set on the server process: | `EVALUATOR_CONFIG_REFRESH_SECS` | `GET /config` cadence (default 300). | | `EVALUATOR_MAX_POLL_DURATION_SECS` | Maximum wallclock time a session may remain in the polling queue before it's terminated as `timeout` (default 3600s). Guards against an evaluator that keeps returning `pending` forever. | -To turn on automatic scoring across the whole instance, provision the `agenteye-evaluator` Secret with both keys set. On the bundled Kubernetes manifests, the server reads `EVALUATOR_ENDPOINT` and `EVALUATOR_TOKEN` from this optional Secret. Create it via your organization's standard secret-management process, then restart the server Deployment to pick up the change. - -The tuning knobs above are not wired by default; expose the corresponding environment variables on the server container in your Deployment manifest if you need to override the defaults. +To turn on automatic scoring, set both `EVALUATOR_ENDPOINT` and +`EVALUATOR_TOKEN` on the server, then restart it to pick up the change. With +`EVALUATOR_ENDPOINT` unset the pipeline stays a no-op. -See [Deployment](/agenteye/deployment) for the full env var table. +The tuning knobs above are optional; set the corresponding environment +variables on the server only if you need to override the defaults. --- @@ -481,6 +397,5 @@ longer than the default. ## Next steps - [Python SDK](/agenteye/python-sdk): emit the `agent_end` events that trigger scoring. -- [Deployment](/agenteye/deployment): the full server env var reference, including the evaluator knobs. - [API keys](/agenteye/api-keys): the `evaluations:read` and `evaluations:trigger` permissions. -- [Audits](/agenteye/audits): AgentEye's other automated quality feature, for policy-based review. +- [Audits](/agenteye/audits): Observability's other automated quality feature, for policy-based review. diff --git a/docs/agenteye/evaluations.mdx b/docs/agenteye/evaluations.mdx index 5589adc9a..fe701604d 100644 --- a/docs/agenteye/evaluations.mdx +++ b/docs/agenteye/evaluations.mdx @@ -1,10 +1,10 @@ --- title: "Evaluations" -description: "AgentEye Evaluations documentation." +description: "FailproofAI Observability Evaluations documentation." --- -Quality problems find you now, instead of you hearing about them in a user complaint. Connect your own scoring service once and AgentEye grades every finished run automatically, so a drop in helpfulness or a spike in hallucinations shows up on its own, before a customer feels it. +Quality problems find you now, instead of you hearing about them in a user complaint. Connect your own scoring service once and FailproofAI Observability grades every finished run automatically, so a drop in helpfulness or a spike in hallucinations shows up on its own, before a customer feels it. ![The Sessions grid with a score column: each run carries an evaluation status pill and colour-coded helpfulness, factuality, and tool-efficiency badges](/agenteye/images/sessions-list.png) @@ -12,7 +12,7 @@ Quality problems find you now, instead of you hearing about them in a user compl ## Stop sampling runs by hand -You used to spot-check a handful of runs and hope the rest were fine. Now every completed session is scored the moment it finishes, on the dimensions you care about: helpfulness, tool efficiency, factuality, safety, whatever your quality bar is. You define the score keys; AgentEye stores, trends, and displays whatever your evaluator sends back. No run slips through unscored, and you stop learning about a regression from a support ticket. +You used to spot-check a handful of runs and hope the rest were fine. Now every completed session is scored the moment it finishes, on the dimensions you care about: helpfulness, tool efficiency, factuality, safety, whatever your quality bar is. You define the score keys; FailproofAI Observability stores, trends, and displays whatever your evaluator sends back. No run slips through unscored, and you stop learning about a regression from a support ticket. The scores ride along on the sessions grid at **`//sessions`** (sidebar → *observe* → *sessions*), one badge cluster per row. Want just the runs that fell short? Filter the grid by score range, say helpfulness below 0.5, and pull up exactly the runs worth reading. Viewing scores needs the `evaluations:read` permission. @@ -38,11 +38,11 @@ Dashboards live at **`//dashboards`** (sidebar → *analyze* → *dash ## Connect an evaluator once -Scoring is opt-in and stays completely off until you point AgentEye at a scorer. You stand up one small HTTP service (AgentEye ships a working reference you can copy), set two values on your server, and every run from then on is scored for you. The full walkthrough, the scoring contract, and the SDK live in the deep guide. +Scoring is opt-in and stays completely off until you point FailproofAI Observability at a scorer. You stand up one small HTTP service (Observability ships a working reference you can copy), set two values on your server, and every run from then on is scored for you. The full walkthrough, the scoring contract, and the SDK live in the deep guide. ## Related - [Evaluation suite](/agenteye/evaluation-suite): connect your evaluator, the scoring contract, and the SDK. - [Sessions](/agenteye/sessions): the run-by-run grid where scores appear. - [Dashboards](/agenteye/dashboards): save and share quality trends across your org. -- [Audits](/agenteye/audits): AgentEye's other automatic quality feature, for cross-session investigations. +- [Audits](/agenteye/audits): Observability's other automatic quality feature, for cross-session investigations. diff --git a/docs/agenteye/event-stream.mdx b/docs/agenteye/event-stream.mdx index 72965d8df..566717ad8 100644 --- a/docs/agenteye/event-stream.mdx +++ b/docs/agenteye/event-stream.mdx @@ -1,6 +1,6 @@ --- title: "Event Stream" -description: "AgentEye Event Stream documentation." +description: "FailproofAI Observability Event Stream documentation." --- @@ -37,7 +37,7 @@ Free-text search cuts straight to a message, a tool name, or an id you already h The Event Stream is your org home. Sign in and it is the first surface you land on, at `//`, so triage starts the second you arrive. -Behind it, your agents emit events through the SDK, the collector ships them to your AgentEye server, and the stream tails them as they arrive in infrastructure you control. When you want the rolled-up view instead of the raw trail, each run's events collapse into a single row on Sessions, one click away. +Behind it, your agents emit events through the SDK, the collector ships them to your FailproofAI Observability server, and the stream tails them as they arrive in infrastructure you control. When you want the rolled-up view instead of the raw trail, each run's events collapse into a single row on Sessions, one click away. This is the raw source of truth that every other observe surface builds on, so when a number looks wrong elsewhere, the stream is where you confirm what actually happened. diff --git a/docs/agenteye/faq.mdx b/docs/agenteye/faq.mdx deleted file mode 100644 index 8a94c89a5..000000000 --- a/docs/agenteye/faq.mdx +++ /dev/null @@ -1,53 +0,0 @@ ---- -title: "FAQ" -description: "AgentEye FAQ documentation." ---- - - -Quick answers to the questions teams ask most when evaluating AgentEye. For the vocabulary used below, see [Concepts](/agenteye/concepts). - ---- - -## Is AgentEye self-hosted? Does my data leave my environment? - -Yes, it's self-hosted, and no. You run the server, dashboard, and databases in your own infrastructure. Events, prompts, model responses, and analytics stay in storage you control. Even in the [Managed](/agenteye/managed-deployment) option, your data lives in your own cloud account. See [Security](/agenteye/security). - -## Which databases do I need? - -Two: **ClickHouse** (required: it stores all events and analytics) and **PostgreSQL** 15+ (required: it stores organizations, users, API keys, dashboards, and saved queries). **Redis** is optional; it's a cache and rate-limit backend, and everything degrades gracefully without it. - -## What does AgentEye instrument? - -Your agent code, through the [Python SDK](/agenteye/python-sdk). You add `agenteye.event.*` calls at the points you care about: agent start/end, model requests and responses, tool calls, hooks, errors, and human-in-the-loop steps. Instrumentation is explicit, so you decide exactly what is recorded. - -## Do I have to score my runs? What's an "evaluator"? - -Scoring is opt-in. Sessions are always recorded, but they're only scored once you connect an **evaluator**, a small scoring service you run that AgentEye POSTs each finished run to. AgentEye doesn't ship a default evaluator; the scoring logic is yours. Until you connect one, sessions simply appear unscored. See [Evaluation suite](/agenteye/evaluation-suite). - -## What's the difference between alerts and audits? - -An **alert** watches a metric you already know about (error rate, latency, cost, an evaluator score) and pages you when it crosses a threshold. An **audit** is the other direction: a recurring investigation that mines your logs across sessions for failure patterns you *haven't* written a rule for, and hands you ranked, evidence-backed findings. Use alerts for known risks and audits to discover unknown ones. See [Alerts](/agenteye/alerts) and [Audits](/agenteye/audits). - -## Can multiple teams share one instance? - -Yes. One AgentEye instance can host many **organizations**, each isolated at the database layer with its own users, keys, and data. See [Tenant management](/agenteye/tenant-management). - -## Does the AI assistant have write access to my data? - -Only in a limited, approval-gated way. The assistant is read-only by default; its SQL is restricted to `SELECT`/`WITH` queries, anything it creates (a saved query or dashboard) requires your explicit approval, and it can never delete. See [AI assistant](/agenteye/assistant). - -## How is AgentEye different from Failproof AI Guards? - -AgentEye is for **observing and improving** agents after the fact: tracing runs, scoring quality, finding failures, and alerting. Guards is a separate Failproof AI product focused on **policies and guardrails** that act on agent behavior. They're complementary; this documentation set covers AgentEye. - -## How do I get access? - -AgentEye is an enterprise product from Failproof AI; pulling the container images and downloading release artifacts requires signed enterprise access. To request a demo or get access, email [nivedit@exosphere.host](mailto:nivedit@exosphere.host), [nikita@exosphere.host](mailto:nikita@exosphere.host), [nivedit@befailproof.ai](mailto:nivedit@befailproof.ai), or [nikita@befailproof.ai](mailto:nikita@befailproof.ai). - ---- - -## Next steps - -- [Overview](/agenteye/overview): what AgentEye is and how it fits together. -- [Getting started](/agenteye/getting-started): deploy the stack and see your first event. -- [Deployment options](/agenteye/deployment-options): pick the right install path. diff --git a/docs/agenteye/getting-started.mdx b/docs/agenteye/getting-started.mdx deleted file mode 100644 index 61a7a989d..000000000 --- a/docs/agenteye/getting-started.mdx +++ /dev/null @@ -1,292 +0,0 @@ ---- -title: "Getting Started with AgentEye" -description: "AgentEye Getting Started with AgentEye documentation." ---- - - -Stand up AgentEye and watch your first agent run appear in the dashboard, end to end, in about 15 minutes. By the end you'll have the server and dashboard running in your own infrastructure, a collector shipping events from an agent machine, and your Python agent instrumented so its runs show up as searchable sessions. - -This guide uses the single-node **Docker Compose** path, the fastest way to try AgentEye. Running somewhere else? Pick your path: [Managed deployment](/agenteye/managed-deployment) (we operate it for you), [Kubernetes deployment](/agenteye/kubernetes-deployment) (production, multi-node), or [Single-pod deployment](/agenteye/single-pod-deployment) (one co-located pod). - -> **AgentEye is an enterprise product from Failproof AI.** The steps below pull private container images, which needs signed enterprise access. To request a demo or get access, email [nivedit@exosphere.host](mailto:nivedit@exosphere.host), [nikita@exosphere.host](mailto:nikita@exosphere.host), [nivedit@befailproof.ai](mailto:nivedit@befailproof.ai), or [nikita@befailproof.ai](mailto:nikita@befailproof.ai). - ---- - -## What is AgentEye? - -AgentEye is a self-hosted platform for observing, evaluating, and improving your AI agents in production. It records what your agents do (every step of a run) so you can see how they behave in production and catch regressions before your users do. Connect your own evaluator service and it will also score the quality of each completed run automatically. - -Here's how the data flows, in one direction: - -```mermaid -flowchart LR - SDK["Python SDK"] --> COL["agenteye-collector"] --> SRV["server"] --> DASH["dashboard"] - SRV --> CH[("ClickHouse
events and analytics")] - SRV --> PG[("Postgres
orgs, users, API keys,
dashboards, saved queries")] -``` - -Your agent code emits **events** through the **Python SDK**. A lightweight **collector** daemon batches them and ships them to the **server**. The server stores events and analytics in **ClickHouse** and operational state (organizations, users, API keys, dashboards, saved queries) in **Postgres**, and the **dashboard** reads it all back for you to explore. - -What you get: - -- **Events**: the raw, per-step trail of every agent run (tool calls, model calls, hooks, errors). -- **Sessions**: those events rolled up into one row per run, each ready to be scored automatically once you connect an evaluator. -- **Evaluations**: quality scores produced by your own evaluator services, so quality drops surface without manual review. -- **Queries & dashboards**: saved ClickHouse SQL over your data, charted into shared, org-scoped dashboards. -- **Alerts & incidents**: threshold rules that page you (email, Slack, webhook, in-dashboard) plus an incident workflow to triage them. -- **CLI & AI assistant**: a terminal client (`agenteye`) and the in-dashboard **AI assistant** for asking questions in plain English. - -
- -
- -*See a live agent run trace itself in the AgentEye dashboard: every tool call, model request, and hook on one timeline.* - -You run all of it in your own infrastructure, as a single Docker Compose stack (this guide), a production Kubernetes install, or a single co-located pod. The rest of this guide sets up the Compose stack end to end. - ---- - -## Before you begin - -The Docker Compose path takes about 15 minutes. On the machine where you'll run AgentEye, and on each agent machine, make sure you have: - -- **Docker** and **Docker Compose**: to run the server, dashboard, and datastores (Steps 2-3). -- **GitHub CLI (`gh`)** and **curl**: to download the compose file, collector binary, and SDK (Steps 2, 4, 5). -- **Python 3** and **pip**: to install the SDK and instrument your agent (Steps 5-6). -- Signed enterprise access to pull the private AgentEye packages. See [GitHub token setup](/agenteye/github-token). - -> **Note:** The collector download in Step 4 is a Linux x86_64 binary. For macOS, Linux arm64, or container installs, see [Collector installation](/agenteye/collector-installation). - ---- - -## Step 1: Authenticate - -AgentEye's packages are private, so this step authenticates you as a signed enterprise customer to pull them. All artifacts are distributed from the `agenteye-enterprise` GitHub organization; you generate your own GitHub personal access token (PAT) to fetch them. Follow the [GitHub token setup](/agenteye/github-token) guide for exact steps and required permissions. - -```bash -export AGENTEYE_TOKEN= - -# Authenticate Docker against GHCR -echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin -``` - ---- - -## Step 2: Deploy the Server and Dashboard - -The server receives events from collectors and makes them queryable; the dashboard is where you explore them. Ingested events and analytics live in ClickHouse (the required analytics store), while Postgres holds operational state such as organizations, users, API keys, dashboards, and saved queries. - -**Download the published compose file:** - -```bash -mkdir -p ./agenteye -curl -fsSL \ - -H "Authorization: token $AGENTEYE_TOKEN" \ - https://raw.githubusercontent.com/agenteye-enterprise/releases/main/docker-compose.yml \ - -o ./agenteye/docker-compose.yml -cd agenteye -``` - -**Set your secrets:** - -Create a `.env` file so the deployment does not run on the default `admin` credential. At minimum set `ADMIN_KEY` and `POSTGRES_PASSWORD`: - -```bash -POSTGRES_PASSWORD=your-db-password -ADMIN_KEY=your-admin-secret -``` - -**Start the stack:** - -```bash -docker compose up -d -``` - -This brings up the full stack, including the required ClickHouse analytics store and an optional Redis cache, alongside the server and dashboard. ClickHouse must be healthy for the server to start. - -The server is now listening at `http://localhost:8080` and the dashboard at `http://localhost:3000`. - -For production deployments (custom Postgres, TLS, reverse proxy), see [Deployment](/agenteye/deployment). - ---- - -## Step 3: Create an API Key for the Collector - -Each collector authenticates with a scoped API key. Use the `ADMIN_KEY` you set in Step 2 to create one: - -```bash -curl -s -X POST http://localhost:8080/keys \ - -H "Authorization: Bearer $ADMIN_KEY" \ - -H "Content-Type: application/json" \ - -d '{"name":"prod-collector","key":"your-collector-secret","permissions":["events:add"]}' -``` - -You provide the `key` value yourself; use it in the collector config in Step 4. See [API keys](/agenteye/api-keys) for full key management. - ---- - -## Step 4: Install the Collector - -On every machine that runs your AI agents, install the collector daemon. - -**Download the binary (Linux x86_64):** - -```bash -# Check the agenteye-enterprise/releases page for the latest collector/v* tag. -VERSION=0.0.1-beta.14 -GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download "collector/v${VERSION}" \ - --repo agenteye-enterprise/releases \ - --pattern 'agenteye-collector-linux-x86_64' -chmod +x agenteye-collector-linux-x86_64 -sudo mv agenteye-collector-linux-x86_64 /usr/local/bin/agenteye-collector -``` - -> **Note:** This downloads the **Linux x86_64** build. For macOS (Apple Silicon or Intel), Linux arm64, or Docker / systemd / launchd setup, see [Collector installation](/agenteye/collector-installation), which lists the download for each platform. The command above installs a Linux binary that will not run elsewhere. - -**Configure:** - -```bash -mkdir -p ~/.agenteye -cat > ~/.agenteye/config.json < **Tip:** To confirm the full path works before opening the dashboard, run your agent, then run `agenteye-collector flush` on the agent machine, and refresh the Events page (Step 7). If nothing appears, that tells you which hop to check: the collector, the server URL, or the API key. - ---- - -## Step 7: View Events in the Dashboard - -Open `http://your-dashboard-host:3000` and sign in. AgentEye emails you a single-use code (or a one-click magic link), so there's no password to manage. - -![The AgentEye sign-in screen, which sends a single-use code to your email](/agenteye/images/login.png) - -Once you're in, the **Events** page shows a live trail of all ingested events. Filter by `session_id` or `agent_id` to drill into a specific run. - -![The live Events stream, colour-coded by event type and filterable by environment, agent, and session](/agenteye/images/events-stream.png) - -The **Sessions** page rolls those events up into one row per run. Once you connect an evaluator service, AgentEye scores every completed run automatically and the latest score appears on each row, so quality regressions surface without manual review. Until you set one up, sessions still capture the full run; they just don't carry a score yet. - -![The Sessions list, one row per run, with status pills and evaluation score badges (scores appear once an evaluator is connected)](/agenteye/images/sessions-list.png) - -To turn on scoring and configure how sessions are evaluated, see the [Evaluation suite](/agenteye/evaluation-suite). - -Click any session to open its **execution graph**, a git-style view of how agents, tools, hooks, and model calls unfolded over time, with parallel sub-agents on their own lanes and a per-run breakdown in the right rail: - -![A session's git-style execution graph beside its event timeline, with the tool/model/hook breakdown panel](/agenteye/images/session-detail.png) - ---- - -## Step 8: Connect your coding agents - -The events you are now browsing are one plain-English question away from your terminal. The **`agenteye` CLI** reads and administers your deployment (sessions, events, evals, keys, alerts, incidents), and the **[CLI skill](/agenteye/cli-skill)** teaches a coding agent like Claude Code or Codex to drive it for you, so you can just ask "is anything broken today?" instead of memorizing commands. - -Install the CLI and sign in. This is a separate tool from the collector: the collector ships events to the server, while the CLI talks to your dashboard. - -```bash -pipx install agenteye -agenteye --base-url http://your-dashboard-host:3000 login --email you@example.com # emailed 6-digit code -agenteye whoami # confirm user and active org -agenteye --json sessions --since 24h # the runs from Step 7, as JSON -``` - -To let an agent answer for you, place the `agenteye-cli/` skill folder (AgentEye provides it) in `~/.claude/skills/`; Claude Code discovers it automatically. Then ask in plain English: - -```text -you ▸ Why did session run-001 fail? -agent ▸ Running: agenteye --json events --session-id run-001 --all - checkout-agent hit TimeoutError on its third tool call -``` - -The agent runs the CLI as you, so it can do only what your login permits, and it can change things (rotate a key, resolve an incident) as well as read. It states each command and waits for your OK before any write. See [CLI and agents](/agenteye/cli-and-agents) for the full picture, the [CLI reference](/agenteye/cli) for every command, and [CLI recipes](/agenteye/cli-recipes) for automation patterns. - ---- - -## Step 9: Explore, chart, and alert - -With events flowing, the **analyze** pages turn raw activity into answers, so you can measure agent behavior, share findings across the team, and get paged the moment something regresses. Dashboard pages are organization-scoped, so the URLs you see in the address bar are prefixed with your org slug (`//…`). - -- **Queries** (`//queries`): start from a library of saved, reusable queries over your events and evaluations (built-in presets plus your own)… - -![The saved-queries library: a grid of reusable queries, both built-in presets and custom ones](/agenteye/images/queries.png) - - …then open one in the SQL composer to tweak it and run it with live results: - -![The SQL query composer running a saved query, with a schema sidebar and a live result grid](/agenteye/images/query-lab.png) - -- **Dashboards** (`//dashboards`): pin queries as line, bar, area, or pie tiles into shared, org-wide dashboards. - -![A dashboard built from saved queries: an events-per-hour line, an errors-by-type bar, a latency area chart, and tokens-by-model](/agenteye/images/dashboard-fleet.png) - -- **Alerts** (`//alerts`): promote any threshold into a paging rule that notifies by email, Slack, webhook, or in-dashboard. See [Alerts](/agenteye/alerts). - -![The Alerts list: threshold rules with their notification channels and current status](/agenteye/images/alerts.png) - ---- - -## Next steps - -- [Evaluation suite](/agenteye/evaluation-suite): connect an evaluator so sessions get scored. -- [Deployment](/agenteye/deployment): harden the stack for production. -- [API keys](/agenteye/api-keys): manage collector and user access. -- [Troubleshooting](/agenteye/troubleshooting): diagnose events that aren't showing up. diff --git a/docs/agenteye/github-token.mdx b/docs/agenteye/github-token.mdx deleted file mode 100644 index ae27847a4..000000000 --- a/docs/agenteye/github-token.mdx +++ /dev/null @@ -1,152 +0,0 @@ ---- -title: "GitHub Token Setup" -description: "AgentEye GitHub Token Setup documentation." ---- - - -A GitHub Personal Access Token (PAT) is the one credential you need to download and run AgentEye. With a single token you can pull the Docker images, download release binaries, and install the Python SDK: no per-component logins and no shared secrets to circulate. Set it up once per machine and rotate it on your own schedule, so access stays auditable and revocable per person. - -All AgentEye artifacts are distributed from the `agenteye-enterprise` GitHub organization. Once your organization is granted access, each developer or operator generates and rotates their own token. - -> **Note:** Pulling the private Docker images requires signed enterprise access. Before any token will work, a member of the `agenteye-enterprise` organization must grant your GitHub user or organization access. To request access, email `support@exosphere.host`. - -You have two ways to create a token. Most teams should use **Option A (classic)**. It has the broadest support for GHCR's `docker login` flow. Choose **Option B (fine-grained)** only if your security policy requires fine-grained tokens. - ---- - -## Option A: Classic Token (Recommended) - -A classic token is the most reliable choice for AgentEye, because GHCR's `docker login` and image-pull flow has the broadest, most consistent support for classic tokens. Two scopes cover everything you need (pulling images and downloading release assets), so you authenticate once and move on without troubleshooting registry quirks. One of them, `read:packages`, is genuinely read-only; the other, `repo`, is the only classic scope that grants access to private release assets, and it is deliberately broad. GitHub defines it as full control (read and write) of private repositories. - -### 1. Create the token - -Go to **GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic) → Generate new token (classic)**. - -| Field | Value | -|---|---| -| **Note** | `agenteye-` (e.g. `agenteye-prod-server`) | -| **Expiration** | Set an expiry appropriate for your security policy; 90 days is a reasonable default | - -> **Note:** GitHub labels this field **Note** for classic tokens and **Token name** for fine-grained tokens. They serve the same purpose: a human-readable identifier for later auditing and revocation. - -### 2. Select scopes - -| Scope | Why it is needed | -|---|---| -| `read:packages` | Pull Docker images from `ghcr.io/agenteye-enterprise/` and download package assets | -| `repo` | Read private repository contents, raw files, and release assets from `agenteye-enterprise/releases`. This is GitHub's broad "Full control of private repositories" scope (read and write), not a read-only scope. It is simply the only classic scope that grants access to private release assets | - -No other scopes are required. - -### 3. Generate and copy the token - -Click **Generate token** and copy the value immediately; it is shown only once. Store it in your secret manager or environment. - ---- - -## Option B: Fine-Grained Token - -Fine-grained tokens scope access to specific repositories and permissions, making them the tightest least-privilege option. Choose this path when your organization's security policy mandates fine-grained tokens. - -> **Note:** GHCR support for fine-grained tokens is less consistent than for classic tokens. If `docker login` or `docker pull` fails after following these steps, fall back to a classic token (Option A). - -### 1. Create the token - -Go to **GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token**. - -| Field | Value | -|---|---| -| **Token name** | `agenteye-` (e.g. `agenteye-prod-server`) | -| **Expiration** | Set an expiry appropriate for your security policy; 90 days is a reasonable default | -| **Resource owner** | `agenteye-enterprise` | -| **Repository access** | **Only select repositories** → `agenteye-enterprise/releases` | - -### 2. Set repository permissions - -Under **Permissions → Repository permissions**, set: - -| Permission | Access | -|---|---| -| **Contents** | Read-only | -| **Packages** | Read-only | - -All other permissions can remain **No access**. - -> **Note:** If the container images (`ghcr.io/agenteye-enterprise/...`) are published as organization-level packages rather than repository-linked packages, Docker login may fail with repository-scoped permissions alone. In that case, add an organization-level permission: **Permissions → Organization permissions → Packages: Read-only**. - -### 3. What each permission grants - -| Permission | Used for | -|---|---| -| Contents: Read-only | Downloading `docker-compose.yml`, release binaries, and Python wheels from `agenteye-enterprise/releases` | -| Packages: Read-only | Pulling Docker images from `ghcr.io/agenteye-enterprise/` | - -### 4. Generate and copy the token - -Click **Generate token** and copy the value immediately; it is shown only once. Store it in your secret manager or environment. - ---- - -## Set the Token on Your Machine - -Once you have created a token (Option A or B above), set it as an environment variable and Docker credential once per machine: - -```bash -export AGENTEYE_TOKEN= -echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin -``` - -> **Note:** GHCR ignores the `docker login` username and authenticates entirely from the token, so any non-empty value works. These docs use `-u x` for brevity; deployment manifests that create a Kubernetes image-pull secret may use a more descriptive username such as `agenteye-enterprise`. Either is accepted. - ---- - -## Rotating a Token - -Rotating tokens on a schedule keeps access auditable and limits the blast radius if a credential ever leaks. Tokens can also expire or be revoked at any time, so rotation is the routine way to stay authenticated. To rotate: - -1. Generate a new token using the steps above. -2. Update `AGENTEYE_TOKEN` in your environment or secret manager. -3. Re-authenticate Docker: `echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin` -4. Revoke the old token in GitHub → Settings → Developer settings → Personal access tokens, then open the **Tokens (classic)** or **Fine-grained tokens** sub-page matching the token's type and delete it. - ---- - -## Verify Your Token - -Confirm the token works before wiring it into a deployment, so authentication failures surface here rather than mid-rollout. Each command exercises one of the scopes above: - -```bash -# Packages scope - authenticate Docker against GHCR -echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin - -# Contents scope - fetch a raw release file -curl -fsSL \ - -H "Authorization: token $AGENTEYE_TOKEN" \ - https://raw.githubusercontent.com/agenteye-enterprise/releases/main/docker-compose.yml \ - -o /tmp/agenteye-compose-check.yml -``` - -A successful `docker login` confirms the package scope; a downloaded file confirms the contents scope. - ---- - -## Troubleshooting - -| Symptom | Likely cause | Fix | -|---|---|---| -| `docker login` returns 401 | Token missing `Packages: Read-only` (fine-grained) or `read:packages` (classic) | Add the package scope and regenerate | -| `curl` returns 404 on raw GitHub URLs | Token missing `Contents: Read-only` or `repo` scope | Add the contents scope and regenerate | -| `gh release download` returns 403 | Token not authorized for `agenteye-enterprise/releases` | Verify the repo is included in the fine-grained token's repository access, or use a classic token with `repo` scope | -| Token accepted but images not found | Org-level package permission missing on fine-grained token | Add organization-level `Packages: Read-only` permission | - -For access issues, contact `support@exosphere.host`. - ---- - -## Next steps - -With your token ready, continue setting up AgentEye: - -- [Getting started](/agenteye/getting-started): deploy AgentEye and sign in to the dashboard. -- [Collector installation](/agenteye/collector-installation): install the `agenteye-collector` using your token. -- [Python SDK](/agenteye/python-sdk): instrument your agents to send events to AgentEye. diff --git a/docs/agenteye/health-monitoring.mdx b/docs/agenteye/health-monitoring.mdx deleted file mode 100644 index d6bc19277..000000000 --- a/docs/agenteye/health-monitoring.mdx +++ /dev/null @@ -1,186 +0,0 @@ ---- -title: "Health Monitoring" -description: "AgentEye Health Monitoring documentation." ---- - - -AgentEye watches your agents. This page is about watching AgentEye *itself*, so -you get alerted the moment the product is down or degraded, not just when your -agents misbehave. Detection is **independent of AgentEye**: it reads pod state -from the Kubernetes control plane and checks AgentEye's hard dependencies, so it -still fires when the server, ClickHouse, or Postgres is the thing that's down. - -> **Note:** This is the Kubernetes story. If you run the single-pod or Docker -> Compose deployment, the built-in `/health` and `/ready` probes in the next -> section are all you need, and they behave identically there. Robusta (layer 2) -> is an opt-in Kubernetes-fleet add-on, not a requirement. See the -> [single-pod deployment guide](/agenteye/single-pod-deployment). - -There are two layers. The first is built in; the second is opt-in. - -```mermaid -flowchart LR - subgraph L1["Layer 1 (built in): Kubernetes probes the server"] - RP["readinessProbe"] --> READY["GET /ready
checks Postgres + ClickHouse"] - LP["livenessProbe"] --> HEALTH["GET /health
is the process alive?"] - end - subgraph L2["Layer 2 (opt-in): Robusta watches the control plane"] - ROB["Robusta"] --> POD["reads pod state directly"] --> SLACK["posts failures to Slack"] - end -``` - -Because layer 2 reads the control plane instead of asking AgentEye, it still alerts when the server, ClickHouse, or Postgres is the thing that is down. - -## 1. Dependency-aware readiness (built in) - -The server exposes two probe endpoints with deliberately different jobs: - -| Endpoint | Probe | Checks | Auth | -|---|---|---|---| -| `GET /health` | liveness | process is alive (always `{"status":"ok"}`) | none | -| `GET /ready` | readiness | can actually serve: **Postgres + ClickHouse** reachable | none | - -`/ready` returns `200` with `"status":"ready"` and every check `"ok"` when both -hard dependencies are reachable, and `503` with `"status":"not_ready"` when -either is unreachable. Both responses carry a small body: - -```json -{ "status": "not_ready", - "checks": { "postgres": "ok", "clickhouse": "down", "redis": "not_configured" } } -``` - -Redis is an optional cache the server degrades past, so it is reported for -information but **never** fails readiness. It shows `"ok"` when a cache is -configured and `"not_configured"` otherwise; it is never `"down"`. - -On the bundled Kubernetes manifests the **readiness** probe points at `/ready` -and **liveness** stays on `/health`. The effect: a server that is *running but -cannot reach its database* is taken out of the Service and shows as `NotReady`, -a state your cluster monitoring (below) can alert on. Liveness stays cheap on -`/health`, so a brief dependency blip never restarts the pod, while the readiness -probe's generous failure threshold tolerates about 30 seconds of blips before -pulling a replica out of rotation. - -**Verify the built-in probes yourself.** From anywhere that can reach the server -(swap in your server's address and port, `8080` in the bundled manifests): - -```bash -# Liveness: always 200 while the process is up -curl -s http://localhost:8080/health -# {"status":"ok"} - -# Readiness: 200 when Postgres + ClickHouse are reachable, 503 otherwise -curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8080/ready -# 200 body: {"status":"ready", "checks": { ... }} -# 503 body: {"status":"not_ready", "checks": { ... }} if a hard dependency is down -``` - -## 2. Pod-failure alerting with Robusta (opt-in) - -[Robusta](https://github.com/robusta-dev/robusta) is a Kubernetes-native monitor -that watches the API server and posts pod failures (`CrashLoopBackOff`, -`OOMKilled`, `ImagePullBackOff`, `Pending`/`NotReady`, `Failed`, evictions) to -Slack. Because it observes the control plane rather than asking AgentEye, it -alerts even when AgentEye cannot serve at all. - -> **Note:** The rest of this section is operator setup: Helm, a Slack app, and a -> values file. If you are evaluating AgentEye rather than installing it, skip to -> [What it reports](#what-it-reports) to see the payoff. - -Robusta ships as an opt-in add-on in the release bundle. Enable it with the -standard Robusta Helm chart and the small values file shown below: - -1. Add the chart repo and get a Slack **bot token** (`xoxb-…`) for the channel: - - ```bash - helm repo add robusta https://robusta-charts.storage.googleapis.com - helm repo update - ``` - - Because the configuration below keeps everything in-cluster - (`disableCloudRouting: true`), the token comes from a self-hosted Slack app: - create an app at `https://api.slack.com/apps`, add the `chat:write` bot scope, - install it to your workspace, copy the **Bot User OAuth Token** (`xoxb-…`), and - invite the bot to the channel (`/invite @your-app`). - -2. Create a `values.yaml` with a per-deployment label (`clusterName`) and your - Slack channel, scoped to the `agenteye` namespace: - - ```yaml - clusterName: "acme-prod" # per-deployment label; appears on every alert - enablePrometheusStack: false # pod-crash alerts only; no metric stack - disableCloudRouting: true # deliver to Slack directly, in-cluster - sinksConfig: - - slack_sink: - name: vendor_slack - slack_channel: "agenteye-fleet-health" - api_key: "REPLACE_WITH_SLACK_BOT_TOKEN" # xoxb-… (prefer --set or a secret) - scope: - include: - - namespace: [agenteye] # only AgentEye-namespace alerts; remove to widen - ``` - -3. Install, pinning `--version` to a known-good Robusta chart release - ([releases](https://github.com/robusta-dev/robusta/releases)) so you never - install an untested chart: - - ```bash - helm install robusta robusta/robusta \ - --namespace robusta --create-namespace \ - --version \ - -f values.yaml \ - --set sinksConfig[0].slack_sink.api_key=$ROBUSTA_SLACK_TOKEN - ``` - -### What it reports - -- Kubernetes **pod state** (which AgentEye pod is failing and why) and each pod's - **image tag**, i.e. the running component **version**. -- **No AgentEye event data and no customer data** ever leaves the cluster. -- The bundled values restrict alerts to the **`agenteye` namespace**, so unrelated - workloads in the same cluster are not reported. - -Here is what a pod-failure alert looks like when it lands in Slack: - -```text -#agenteye-fleet-health - -🔴 Robusta · Pod is crash looping - Cluster: acme-prod ← clusterName label (which deployment) - Namespace: agenteye - Pod: agenteye-server-7d9c8b6f5-2xk4t - Container: server (ghcr.io/agenteye-enterprise/server:0.0.1-beta.18) - Reason: CrashLoopBackOff · last state OOMKilled (exit 137) - Restarts: 5 in the last 3m -``` - -The `clusterName` tells you *which* deployment is affected at a glance, and the -image tag (`…/server:0.0.1-beta.18`) tells you exactly which version is running. - -### One place for every deployment - -Point every deployment's Robusta at **one shared Slack channel**, each with its -own `clusterName`. Every alert is tagged with that label, so a single channel -shows the health of your whole fleet, and you can tell which deployment is -affected at a glance. - -### Total-cluster outages - -A purely in-cluster watcher cannot report a **whole-cluster or network outage** -(it goes down with the cluster). If you need that, enable the optional **Robusta -UI sink**: set `disableCloudRouting: false` and add a `robusta_sink` (with a -token from `robusta gen-config`) to `sinksConfig`. It adds an aggregated -multi-cluster dashboard and flags any cluster that stops checking in. - -## Troubleshooting - -See the Health Monitoring section of the -[troubleshooting guide](/agenteye/troubleshooting#health-monitoring-issues) for "no -alerts arriving" and "server keeps flapping `NotReady`". - -## Next steps - -- [Kubernetes deployment](/agenteye/kubernetes-deployment): where the readiness and liveness probes and the Robusta add-on are wired in. -- [Single-pod deployment](/agenteye/single-pod-deployment): the `/health` and `/ready` probes for Compose and single-node setups. -- [Alerts](/agenteye/alerts): alert on your agents' behavior, the complement to this page's infrastructure health. -- [Troubleshooting](/agenteye/troubleshooting#health-monitoring-issues): fixes for missing alerts and `NotReady` flapping. diff --git a/docs/agenteye/incidents.mdx b/docs/agenteye/incidents.mdx index 26749a8e7..6f46a4575 100644 --- a/docs/agenteye/incidents.mdx +++ b/docs/agenteye/incidents.mdx @@ -1,6 +1,6 @@ --- title: "Incidents" -description: "AgentEye Incidents documentation." +description: "FailproofAI Observability Incidents documentation." --- @@ -20,7 +20,7 @@ When the incident is over, you already have the write-up. Open any incident and ![An incident detail view: the parent alert and breach summary, assignees and subscribers, an attributed activity timeline, and a comment thread](/agenteye/images/incident-detail.png) *Everything that happened, in order, each line signed by whoever did it.* -Every action (opened, acknowledged, resolved, and so on) is written to that timeline and never edited away. Each entry is attributed: to the operator who took it, by email, or to **automated** for anything AgentEye did on its own, like opening the incident on the breach. Nothing is anonymous and nothing is lost, so the post-mortem more or less writes itself. +Every action (opened, acknowledged, resolved, and so on) is written to that timeline and never edited away. Each entry is attributed: to the operator who took it, by email, or to **automated** for anything FailproofAI Observability did on its own, like opening the incident on the breach. Nothing is anonymous and nothing is lost, so the post-mortem more or less writes itself. ## How an incident moves diff --git a/docs/agenteye/kubernetes-deployment.mdx b/docs/agenteye/kubernetes-deployment.mdx deleted file mode 100644 index 0191a8391..000000000 --- a/docs/agenteye/kubernetes-deployment.mdx +++ /dev/null @@ -1,1429 +0,0 @@ ---- -title: "Kubernetes Deployment Guide" -description: "AgentEye Kubernetes Deployment Guide documentation." ---- - - -AgentEye is a self-hosted platform for observing, evaluating, and improving your AI agents in production. This guide is for platform and infrastructure teams who want to run AgentEye at production scale on their own Kubernetes cluster. By the end you'll have a highly available, mutual-TLS-secured deployment: two replicas each of the API server and dashboard, dedicated data stores, and automated daily backups. - -Prefer not to operate the cluster yourself? [Managed deployment](/agenteye/managed-deployment) has Exosphere run it for you. Just want a lightweight single-node trial or a collector sidecar? See [Single-pod deployment](/agenteye/single-pod-deployment). - -This guide deploys the full AgentEye stack: - -- **ClickHouse 24.8** -- the canonical events and evaluations analytics store (StatefulSet with 100Gi persistent volume). Required: the server refuses to start without it. -- **PostgreSQL 16** -- relational/metadata store for organizations, API keys, users, dashboards, saved queries, and auth (StatefulSet with 50Gi persistent volume) -- **Redis 7.2** -- optional shared cache and rate-limit backend; the server and dashboard degrade gracefully if it is unavailable -- **AgentEye Server** -- Rust API for event ingestion, analytics, and key management (2 replicas) -- **AgentEye Dashboard** -- Next.js web UI (2 replicas) -- **AI assistant (agent service)** -- optional read-only in-dashboard assistant on port 9100; inert until an LLM endpoint is configured -- **Traefik (public)** -- ingress controller for collector traffic, mTLS-protected -- **Traefik (dashboard)** -- ingress controller for the dashboard, VPN/IP-allowlist-only -- **cert-manager** -- TLS certificates and mTLS CA -- **Backup CronJob** -- daily combined dump of PostgreSQL + ClickHouse at 03:00 UTC -- **Cert Renewal Monitor** -- alerts when client certificates are near expiry - -**Estimated time:** 60--90 minutes for a first deployment. - -For the managed deployment model where Exosphere handles all of this on your behalf, see [Managed deployment](/agenteye/managed-deployment). - ---- - -## Prerequisites - -Run each verification command before starting. Every check must pass. - -| Requirement | Minimum | Verification Command | Expected | -|---|---|---|---| -| Kubernetes cluster | 1.27+ | `kubectl version` | Server Version >= v1.27 | -| Kustomize (bundled with kubectl) | Kustomize v1.14+ (ships inside kubectl 1.27+) | `kubectl kustomize --help` | Prints usage text | -| Helm | v3 | `helm version` | `Version:"v3.x.x"` | -| cluster-admin RBAC | -- | `kubectl auth can-i create namespaces` | `yes` | -| Default StorageClass | -- | `kubectl get storageclass` | At least one row marked `(default)` | -| LoadBalancer support | -- | Cloud-dependent (EKS, GKE, AKS all support this by default) | -- | -| GitHub PAT | -- | `echo $AGENTEYE_TOKEN` | Non-empty (see [GitHub token setup](/agenteye/github-token)) | -| openssl | -- | `openssl version` | OpenSSL 1.x or 3.x | -| Cloud storage bucket | -- | For PostgreSQL + ClickHouse backups (S3, GCS, or Azure Blob) | -- | - -**Cluster sizing:** Minimum 3 nodes, 4 vCPU / 8 GB RAM each. See [Managed deployment](/agenteye/managed-deployment) for full requirements. - -### Run all checks at once - -```bash -echo "--- Prerequisites Check ---" -kubectl version --client -o yaml 2>/dev/null | grep -q gitVersion && echo "PASS: kubectl" || echo "FAIL: kubectl not found" -helm version --short 2>/dev/null | grep -q v3 && echo "PASS: helm v3" || echo "FAIL: helm v3 not found" -kubectl auth can-i create namespaces 2>/dev/null | grep -q yes && echo "PASS: cluster-admin" || echo "FAIL: no cluster-admin" -kubectl get storageclass 2>/dev/null | grep -q default && echo "PASS: default StorageClass" || echo "FAIL: no default StorageClass" -[ -n "$AGENTEYE_TOKEN" ] && echo "PASS: AGENTEYE_TOKEN set" || echo "FAIL: AGENTEYE_TOKEN not set" -openssl version >/dev/null 2>&1 && echo "PASS: openssl" || echo "FAIL: openssl not found" -echo "---" -``` - -### Deployment shape - -The **ingest endpoint** is served on a hostname you control (e.g. `ingest.your-company.example`). cert-manager requests a publicly-trusted TLS certificate from Let's Encrypt over HTTP-01, so collectors verify the server cert against the system trust store, with no per-customer CA pinning. - -The **dashboard endpoint** works the same way: it is served on a second hostname you control (e.g. `agenteye.your-company.example`) pointing at the dashboard Traefik LoadBalancer, and cert-manager issues its Let's Encrypt certificate through that LoadBalancer. Browsers get a trusted certificate with no warning. - -> **Warning:** Certificate issuance and renewal validate over HTTP-01, so both LoadBalancers must be reachable from the public internet on port 80. If you need to IP-restrict the dashboard LoadBalancer, coordinate a DNS-01 solver with support first. Otherwise renewals fail silently and the certificate expires. - ---- - -## Get the Manifests - -> **Note:** Cloning the releases repo and pulling the `ghcr.io/agenteye-enterprise/*` images both require signed enterprise access. See [GitHub token setup](/agenteye/github-token) to obtain your token. - -```bash -git clone https://x:${AGENTEYE_TOKEN}@github.com/agenteye-enterprise/releases.git -cd releases/deploy -``` - -**Test it:** - -```bash -ls base/kustomization.yaml -``` - -Expected: the file exists. If it doesn't, the clone failed -- check your `AGENTEYE_TOKEN`. - -**Directory structure:** - -```text -deploy/ - base/ Shared Kustomize base (all K8s resources) - overlays/ Cluster-specific overrides (image tags, hostnames, resources) - third-party/ Helm values for Traefik, cert-manager, and (opt-in) Robusta health monitoring -``` - -The **base** contains every resource needed for a full deployment, including Let's Encrypt certificates for the two public hostnames you configure in Phase 3.1. An **overlay** patches the base for a specific environment (e.g. custom image tags, resource limits, env wiring). The **third-party** directory contains Helm values files for external infrastructure. - -> **Tip:** The server's readiness probe already reflects Postgres + ClickHouse health, and `third-party/robusta/` adds opt-in Kubernetes-native pod-failure alerting to Slack. See [Health monitoring](/agenteye/health-monitoring). - ---- - -## Phase 1 -- Third-Party Infrastructure (~30 min) - -### 1.1 Install cert-manager - -cert-manager manages TLS certificates for HTTPS and the private CA used for mTLS client certificates. - -```bash -helm repo add jetstack https://charts.jetstack.io -helm repo update - -helm install cert-manager jetstack/cert-manager \ - --namespace cert-manager --create-namespace \ - --set crds.install=true -``` - -**Test it:** - -```bash -kubectl get pods -n cert-manager -``` - -Expected: 3 pods all `Running` -- `cert-manager`, `cert-manager-cainjector`, `cert-manager-webhook`. - -```bash -kubectl get crds | grep cert-manager -``` - -Expected: at least `certificates.cert-manager.io`, `clusterissuers.cert-manager.io`, `issuers.cert-manager.io`. - -**If it fails:** Pods in `CrashLoopBackOff` usually means CRDs weren't installed. Re-run with `--set crds.install=true`. If webhook pods fail readiness, wait 30 seconds and check again -- they can take a moment to start. - ---- - -### 1.2 Install Traefik -- Public Ingest Controller - -This Traefik instance handles collector traffic on an **external** LoadBalancer. It terminates TLS and enforces mTLS (client certificate verification) on the ingest endpoint. - -```bash -helm repo add traefik https://traefik.github.io/charts -helm repo update - -helm install traefik-public traefik/traefik \ - --namespace traefik-public --create-namespace \ - --version 39.0.8 \ - -f third-party/traefik/values-public.yaml -``` - -**Test it:** - -```bash -kubectl get pods -n traefik-public -``` - -Expected: 1 pod `Running`. - -```bash -kubectl get ingressclass traefik-public -``` - -Expected: the IngressClass exists (it is not the default class). - -**If it fails:** Check `kubectl describe pod -n traefik-public ` for image pull errors or resource constraints. - ---- - -### 1.3 Install Traefik -- Dashboard Controller - -This Traefik instance serves the dashboard on a dedicated LoadBalancer, restricted by IP allowlist. - -> **Note:** Two allowlist mechanisms ship for this instance. This guide uses `values-dashboard.yaml`, which restricts access with the portable `service.loadBalancerSourceRanges` field. A parallel `values-internal.yaml` is also provided for AWS environments that prefer the `service.beta.kubernetes.io/aws-load-balancer-source-ranges` annotation instead. Pick one and use it consistently; the steps below assume `values-dashboard.yaml`. - -**Before installing**, edit `third-party/traefik/values-dashboard.yaml` to set the allowed source IPs. The `loadBalancerSourceRanges` field controls which IPs can reach the dashboard. By default it is set to `0.0.0.0/0` (all IPs); restrict it to your VPN, office, or known egress IPs. - -#### Whitelist a single IP - -```yaml -service: - loadBalancerSourceRanges: - - "/32" -``` - -#### Whitelist multiple IPs - -Add one entry per IP or CIDR block. A `/32` suffix matches a single IPv4 address; a CIDR block (e.g. `/24`) matches a range. You can mix individual IPs and ranges freely: - -```yaml -service: - loadBalancerSourceRanges: - - "203.0.113.10/32" # office gateway - - "203.0.113.11/32" # backup office gateway - - "198.51.100.0/24" # VPN pool - - "192.0.2.50/32" # on-call engineer home IP -``` - -Tips when maintaining the list: - -- Keep one entry per line and add a short `#` comment identifying each IP's owner or purpose; this is what future operators use to decide whether an entry is still needed. -- Always use CIDR notation. A bare IP like `203.0.113.10` is rejected by the cloud provider; use `203.0.113.10/32`. -- For IPv6 ranges, use the equivalent `/128` (single address) or larger CIDR, e.g. `2001:db8::1/128`. Not all cloud providers support IPv6 source ranges; check your provider's LoadBalancer documentation. -- The list is an **OR**: traffic is allowed if the source matches any entry. - -After editing the file, proceed to `helm install` below. If the controller is already installed, run `helm upgrade` with the same flags, or patch the Service at runtime (next section). - -#### Update the whitelist at runtime - -You can change the allowed IPs without a Helm upgrade by patching the Service directly. **The patch replaces the entire list**; always include every IP you want to keep, not just the new one. - -To replace the list with a new set of IPs: - -```bash -kubectl patch svc traefik-dashboard -n traefik-dashboard \ - -p '{"spec":{"loadBalancerSourceRanges":["203.0.113.10/32","198.51.100.0/24","192.0.2.50/32"]}}' -``` - -To safely **append** an IP without losing existing entries, read the current list first, then patch with the combined set: - -```bash -# 1. Show the current allowlist -kubectl get svc traefik-dashboard -n traefik-dashboard \ - -o jsonpath='{.spec.loadBalancerSourceRanges}' - -# 2. Patch with the full list including the new IP -kubectl patch svc traefik-dashboard -n traefik-dashboard \ - -p '{"spec":{"loadBalancerSourceRanges":["/32","/32","/32"]}}' -``` - -> Runtime patches are not persisted back to `values-dashboard.yaml`. To keep the change across future Helm upgrades, also update the values file and commit it. - -Then install: - -```bash -helm install traefik-dashboard traefik/traefik \ - --namespace traefik-dashboard --create-namespace \ - --version 39.0.8 \ - -f third-party/traefik/values-dashboard.yaml -``` - -**Test it:** - -```bash -kubectl get pods -n traefik-dashboard -``` - -Expected: 1 pod `Running`. - -```bash -kubectl get ingressclass traefik-dashboard -``` - -Expected: the IngressClass exists. - ---- - -### 1.4 Wait for LoadBalancers - -Both Traefik instances need external IPs before proceeding. - -```bash -kubectl get svc -n traefik-public -kubectl get svc -n traefik-dashboard -``` - -**Test it:** Both services show an `EXTERNAL-IP` (not ``). - -If still pending, watch for assignment: - -```bash -kubectl get svc -n traefik-public -w -``` - -Press `Ctrl+C` once the IP appears. IP assignment typically takes 2--5 minutes. - -**If it fails:** `` after 10 minutes usually means the cloud provider can't provision a LoadBalancer. Both `traefik-public` and `traefik-dashboard` are external LoadBalancers, so check the usual provisioning blockers: subnet tags (EKS requires `kubernetes.io/role/elb`), VPC configuration, and LoadBalancer service quotas. - ---- - -## Phase 2 -- Create Secrets (~10 min) - -All secrets are created manually before deploying the application. This ensures sensitive values never appear in manifest files. - -### 2.1 Create the namespace - -```bash -kubectl create namespace agenteye -``` - -**Test it:** - -```bash -kubectl get namespace agenteye -``` - -Expected: status `Active`. - ---- - -### 2.2 Image pull secret - -This secret authenticates with `ghcr.io` to pull the AgentEye container images. See [GitHub token setup](/agenteye/github-token) for how to generate your PAT. - -```bash -kubectl create secret docker-registry agenteye-image-pull \ - --namespace agenteye \ - --docker-server=ghcr.io \ - --docker-username=agenteye-enterprise \ - --docker-password="${AGENTEYE_TOKEN}" -``` - -**Test it:** - -```bash -kubectl get secret agenteye-image-pull -n agenteye -o jsonpath='{.type}' -``` - -Expected: `kubernetes.io/dockerconfigjson`. - -**Test it (deep)** -- verify the token can actually pull images: - -Use the `server` image tag pinned in your overlay's `kustomization.yaml` (currently `v0.0.1-beta.48` in both the bundled `acme` overlay and base deployment). Substitute the tag below for the one you are deploying so this check does not drift across releases: - -```bash -kubectl run test-pull \ - --image=ghcr.io/agenteye-enterprise/server:v0.0.1-beta.48 \ - --overrides='{"spec":{"imagePullSecrets":[{"name":"agenteye-image-pull"}]}}' \ - --restart=Never -n agenteye --command -- echo ok - -# Wait a few seconds for the pull, then: -kubectl logs test-pull -n agenteye -kubectl delete pod test-pull -n agenteye -``` - -Expected: `ok` printed in the logs. - -**If it fails:** `ErrImagePull` or `401 Unauthorized` means the PAT is invalid or lacks `read:packages` scope. Re-check [GitHub token setup](/agenteye/github-token). - ---- - -### 2.3 PostgreSQL credentials - -```bash -POSTGRES_PASSWORD=$(openssl rand -hex 24) - -kubectl create secret generic agenteye-postgres \ - --namespace agenteye \ - --from-literal=POSTGRES_USER=postgres \ - --from-literal=POSTGRES_PASSWORD="${POSTGRES_PASSWORD}" \ - --from-literal=POSTGRES_DB=agenteye -``` - -> **Warning:** We use `-hex` (not `-base64`) to generate the password. Base64 output can contain `+`, `/`, and `=` which break the `DATABASE_URL` connection string. See [Troubleshooting](/agenteye/troubleshooting) for details. - -> **Note:** Store `POSTGRES_PASSWORD` in your secrets manager immediately. You will need it if you ever restore from a backup or connect to the database directly. - -**Test it:** - -```bash -kubectl get secret agenteye-postgres -n agenteye -``` - -Expected: the secret exists. - -```bash -kubectl get secret agenteye-postgres -n agenteye \ - -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d | wc -c -``` - -Expected: `48` (24 hex bytes = 48 characters). - ---- - -### 2.4 Admin API key - -```bash -ADMIN_KEY=$(openssl rand -hex 32) - -kubectl create secret generic agenteye-admin-key \ - --namespace agenteye \ - --from-literal=ADMIN_KEY="${ADMIN_KEY}" -``` - -The admin key is the bootstrap credential. The server upserts it on every startup with all permissions. Use it to create scoped collector keys in Phase 7. See [API keys](/agenteye/api-keys) for the full permissions model. - -> **Note:** Store `ADMIN_KEY` in your secrets manager immediately. - -**Test it:** - -```bash -kubectl get secret agenteye-admin-key -n agenteye -``` - -Expected: the secret exists. - ---- - -### 2.5 Auth configuration (dashboard login) - -The dashboard uses email + OTP for user login. Without this secret the server still starts and the `ADMIN_KEY` API path keeps working, but **no user can log in via the UI**. - -All keys are referenced as `optional: true` in the base manifest, so partial secrets (or no secret at all) are fine; the server falls back to the documented defaults. Bundling everything into one `agenteye-auth` secret keeps the auth surface rotatable in a single place. - -```bash -kubectl create secret generic agenteye-auth \ - --namespace agenteye \ - --from-literal=ADMIN_EMAIL="admin@yourcompany.com" \ - --from-literal=ALLOWED_EMAILS="*@yourcompany.com" \ - --from-literal=SMTP_HOST="smtp.yourprovider.com" \ - --from-literal=SMTP_PORT="587" \ - --from-literal=SMTP_USERNAME="your-smtp-user" \ - --from-literal=SMTP_PASSWORD="your-smtp-password" \ - --from-literal=SMTP_FROM="noreply@yourcompany.com" \ - --from-literal=SMTP_TLS="starttls" \ - --from-literal=DEFAULT_ORG_NAME="Acme Corp" \ - --from-literal=DEFAULT_ORG_SLUG="acme" -``` - -| Key | Purpose | -|---|---| -| `ADMIN_EMAIL` | Bootstrap admin user. Upserted on every startup with all permissions and protected from deletion/permission edits via the dashboard. Without it, no admin is seeded and first login is impossible. | -| `ALLOWED_EMAILS` | Comma-separated allowlist. Supports exact addresses (`user@example.com`) and domain wildcards (`*@example.com`). Without it, **no user can log in or be created**. | -| `SMTP_HOST`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD`, `SMTP_FROM` | SMTP relay for sending OTP codes. If `SMTP_HOST` is unset, OTP codes are logged to the server's stdout instead of emailed (useful for first-boot smoke tests). Provide all SMTP keys together for real email delivery. | -| `SMTP_TLS` | One of `starttls` (default), `tls`, or `none`. | -| `DEFAULT_ORG_NAME`, `DEFAULT_ORG_SLUG` | Optional. Give the built-in `default` organization a friendly display name and URL slug so it lives at e.g. `/acme` instead of `/default`. Applied on **first boot only**; once you rename the org with `agenteye-orgctl org rename` (see §7.6) these are ignored. The slug must be 1--40 lowercase alphanumerics with single internal hyphens. Leave both unset to keep the generic `default`. | - -> **Note:** Store the SMTP credentials in your secrets manager. - -**Test it:** - -```bash -kubectl get secret agenteye-auth -n agenteye \ - -o jsonpath='{.data}' | grep -o '"[A-Z_]*"' | sort -u -``` - -Expected: the keys you populated appear in the output. - ---- - -### 2.6 Multi-tenant org isolation key (optional) - -Skip this for a single-tenant deployment; the server runs on a built-in dev default and serves the one `default` org fine. **Before you create a second organization**, set a strong, stable `ORG_CH_SECRET`: each org's ClickHouse password is derived as `HMAC(ORG_CH_SECRET, org_id)`, so the publicly-known dev default would yield publicly-derivable per-org credentials. The `agenteye-orgctl org create` command (see [§7.6 Provision organizations](#76-provision-organizations-multi-tenant)) refuses to run while the server is still on the built-in dev default. - -```bash -kubectl create secret generic agenteye-org-ch-secret \ - --namespace agenteye \ - --from-literal=ORG_CH_SECRET="$(openssl rand -base64 48)" - -# Roll the server so it picks up the new value. -kubectl -n agenteye rollout restart deployment/server -``` - -The server reads this via an **optional** `secretKeyRef`, so a single-tenant cluster that never creates it still boots normally. Keep the value **stable and identical across all replicas**; rotating it invalidates every org's derived ClickHouse password until the boot-time reconcile re-provisions the users (a rolling restart with the value consistent everywhere heals it). See `deploy/base/server/secret.example.yaml`. - -> **Warning:** Store `ORG_CH_SECRET` in your secrets manager and don't rotate it casually. - ---- - -### 2.7 Verify all secrets - -```bash -kubectl get secrets -n agenteye -o custom-columns='NAME:.metadata.name,TYPE:.type' -``` - -Expected output (among any default secrets): - -```text -NAME TYPE -agenteye-admin-key Opaque -agenteye-auth Opaque -agenteye-image-pull kubernetes.io/dockerconfigjson -agenteye-postgres Opaque -agenteye-org-ch-secret Opaque # only if you completed §2.6 (multi-tenant) -``` - -The four core secrets (`agenteye-admin-key`, `agenteye-auth`, `agenteye-image-pull`, `agenteye-postgres`) must be present before continuing. `agenteye-org-ch-secret` is only required for multi-tenant deployments (see §2.6). - ---- - -## Phase 3 -- Deploy the Application (~5 min) - -### 3.1 Configure the public hostnames - -cert-manager needs the ingest and dashboard hostnames before it can request their Let's Encrypt certificates. Copy the template and set both: - -```bash -cp base/certificates/domain.env.example base/certificates/domain.env -# Edit base/certificates/domain.env and set: -# INGEST_DOMAIN=ingest.your-company.example (resolves to the public Traefik LB) -# DASHBOARD_DOMAIN=agenteye.your-company.example (resolves to the dashboard Traefik LB) -``` - -`domain.env` is gitignored; it stays local to each deployment. The kustomize build fails loudly if either key is missing. - -> **Note:** DNS must resolve first. You don't have to point DNS at the LBs yet (they don't exist until Phase 1.2 is complete), but ACME issuance in step 3.2 will retry until each hostname resolves to its LoadBalancer. You can either set DNS now (using the LB hostnames captured in Phase 1.4) or proceed and add the records in Phase 4. - ---- - -### 3.2 Apply manifests - -Apply the base directly for a fresh install, or an overlay if you've cut one for this environment (overlays just pin image tags, env vars, and resource limits; they inherit the base's certs and routing): - -```bash -kubectl apply -k base/ -# or -kubectl apply -k overlays// -``` - -The overlay includes the base automatically; apply one, not both. - ---- - -### 3.3 Wait for pods - -```bash -kubectl wait --for=condition=Ready pod -l 'app in (server,dashboard,postgres,clickhouse)' \ - -n agenteye --timeout=180s -``` - -The wait is scoped to the core data-plane pods. The optional `agent` (AI assistant) and `redis` pods come up alongside them; the assistant stays inert until you supply its LLM endpoint (see [AI assistant](/agenteye/assistant)), and Redis is a best-effort cache, so neither needs to be Ready for the platform to serve traffic. - -**Test it:** - -```bash -kubectl get pods -n agenteye -``` - -Expected (the optional `agent` and `redis` pods also appear and reach `Running`): - -```text -NAME READY STATUS RESTARTS AGE -agent-xxxxxxxxxx-xxxxx 1/1 Running 0 ... -clickhouse-0 1/1 Running 0 ... -dashboard-xxxxxxxxxx-xxxxx 1/1 Running 0 ... -dashboard-xxxxxxxxxx-xxxxx 1/1 Running 0 ... -postgres-0 1/1 Running 0 ... -redis-0 1/1 Running 0 ... -server-xxxxxxxxxx-xxxxx 1/1 Running 0 ... -server-xxxxxxxxxx-xxxxx 1/1 Running 0 ... -``` - -**If it fails:** - -| Pod Status | Likely Cause | Debug Command | -|---|---|---| -| `ImagePullBackOff` | Bad image pull secret or PAT | `kubectl describe pod -n agenteye` | -| `CrashLoopBackOff` | Bad environment variables (e.g. DATABASE_URL) | `kubectl logs -n agenteye` | -| `Pending` | Insufficient CPU/memory or no nodes | `kubectl describe pod -n agenteye` (check Events) | - ---- - -### 3.4 Verify storage - -```bash -kubectl get pvc -n agenteye -``` - -Expected, both with status `Bound`: - -| PVC | Capacity | Backs | -|---|---|---| -| `postgres-data-postgres-0` | `50Gi` | PostgreSQL relational/metadata store | -| `clickhouse-data-clickhouse-0` | `100Gi` | ClickHouse events + evaluations analytics store | - -A `redis-data-redis-0` PVC (1Gi) also appears for the optional cache. - -**If it fails:** `Pending` means no StorageClass can provision the volume. Check `kubectl get storageclass` and ensure a default exists. For production, overlay the ClickHouse volume onto a fast SSD StorageClass (e.g. gp3 on AWS, pd-ssd on GCP); compaction throughput suffers on slow disks. - ---- - -### 3.5 Verify certificates - -```bash -kubectl get certificates -n agenteye -``` - -Expected: 3 certificates, all `Ready: True`: - -| Name | Issuer | Purpose | -|---|---|---| -| `mtls-ca` | `selfsigned` | Private CA for issuing mTLS client certs (10-year validity) | -| `ingest-tls` | `letsencrypt-prod` | Public TLS certificate for the ingest endpoint (90-day, auto-renewed) | -| `dashboard-tls` | `letsencrypt-prod` | Public TLS certificate for the dashboard (90-day, auto-renewed) | - -**If `ingest-tls` or `dashboard-tls` is not Ready:** - -`kubectl describe certificate -n agenteye` and read the Events. The common causes: - -- **DNS not yet pointing at the LB.** Let's Encrypt resolves the hostname and hits port 80 to validate. `INGEST_DOMAIN` must resolve to the public LB, `DASHBOARD_DOMAIN` to the dashboard LB. Until the CNAME/Alias propagates, the order stays `pending`. Once DNS is correct, cert-manager retries automatically (no need to delete the Certificate). -- **Hostname not substituted.** If `dnsNames` still reads `INGEST_DOMAIN_PLACEHOLDER` / `DASHBOARD_DOMAIN_PLACEHOLDER`, you skipped step 3.1 -- create `base/certificates/domain.env` and re-apply. -- **Dashboard Traefik can't serve the challenge** (`dashboard-tls` only). The dashboard Traefik instance must be installed with the bundled values file (Phase 1.2), which enables the scoped Ingress provider that serves cert-manager's HTTP-01 solver. An instance installed without it leaves the challenge unroutable and the order `pending` forever. - -**If `mtls-ca` is not Ready:** cert-manager itself is unhealthy. Re-check the cert-manager pods from step 1.1. - ---- - -### 3.6 Verify CronJobs - -```bash -kubectl get cronjobs -n agenteye -``` - -Expected: - -| Name | Schedule | Purpose | -|---|---|---| -| `agenteye-backup` | `0 3 * * *` | Daily Postgres + ClickHouse backup at 03:00 UTC | -| `cert-renewal-check` | `0 3,15 * * *` | Certificate expiry alerts at 03:00 and 15:00 UTC | - ---- - -### 3.7 Verify server started correctly - -```bash -kubectl logs -n agenteye -l app=server --tail=20 -``` - -**Test it:** Look for a startup line indicating the server is listening on port 8080. There should be no database connection errors (the server requires both PostgreSQL and ClickHouse to be reachable before it reports Ready). - -**If it fails:** The most common cause is a `POSTGRES_PASSWORD` containing URL-unsafe characters that break the `DATABASE_URL`. See [Troubleshooting](/agenteye/troubleshooting). - ---- - -### 3.8 Verify dashboard connected to server - -```bash -kubectl logs -n agenteye -l app=dashboard --tail=20 -``` - -**Test it:** Look for `Ready` in the output with no `ECONNREFUSED` or similar errors. - -**If it fails:** Check that the `server` Service exists (`kubectl get svc server -n agenteye`) and that `AGENTEYE_SERVER_URL` is set to `http://server:8080` in the dashboard deployment. - ---- - -## Phase 4 -- Network Access (~5 min) - -### 4.1 Retrieve LoadBalancer addresses - -```bash -PUBLIC_IP=$(kubectl get svc -n traefik-public \ - -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}') - -INTERNAL_IP=$(kubectl get svc -n traefik-dashboard \ - -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}') -``` - -> On AWS EKS, LoadBalancers return a hostname instead of an IP. Replace `.ip` with `.hostname` in the commands above. - -**Test it:** - -```bash -echo "Public (ingest): $PUBLIC_IP" -echo "Internal (dashboard): $INTERNAL_IP" -``` - -Both must be non-empty. - ---- - -### 4.2 Point DNS at the LoadBalancers - -Create DNS records so the hostnames from `base/certificates/domain.env` resolve to their LoadBalancers (`INGEST_DOMAIN` to the **public** Traefik LB, `DASHBOARD_DOMAIN` to the **dashboard** Traefik LB): - -- **AWS Route 53:** `A` record with `Alias = Yes`, target = the LB hostname. Don't use plain A → IP; ELB IPs rotate. -- **Any other provider:** `CNAME` from the hostname to the LB hostname. - -Verify: - -```bash -dig +short ingest.your-company.example -dig +short agenteye.your-company.example -``` - -Should return the same addresses as `$PUBLIC_IP` and `$INTERNAL_IP` respectively (or, on EKS, resolve to the same `*.elb.amazonaws.com` hostnames). - -Once DNS resolves, cert-manager finishes the pending ACME orders from Phase 3.5 within a minute. Re-run `kubectl get certificates -n agenteye` until both `ingest-tls` and `dashboard-tls` show `Ready: True`. - ---- - -### 4.3 Reach the ingest endpoint - -The public ingest endpoint enforces mutual TLS, so every request (including `/health`) must present a client certificate. You issue your first client certificate in Phase 5; if you have one already, verify reachability now: - -```bash -curl -s --cert issued//client.crt \ - --key issued//client.key \ - https://ingest.your-company.example/health -``` - -Expected: `{"status":"ok"}`. No `-k` is needed -- the server certificate chains to a public CA for `INGEST_DOMAIN`, so it validates against the system trust store. Reach the ingest endpoint by its `INGEST_DOMAIN` hostname (which matches the issued certificate), not by the raw LoadBalancer IP/hostname. - -The dashboard endpoint is served on `DASHBOARD_DOMAIN` with a publicly-trusted certificate and is not behind mTLS, so no `-k` and no client certificate are needed: - -```bash -curl -s https://agenteye.your-company.example/ -o /dev/null -w '%{http_code}\n' -``` - -Reach the dashboard by its hostname, not the raw LB address. The certificate is bound to `DASHBOARD_DOMAIN`, so the raw address shows a certificate-name mismatch. - -**If it fails:** If `curl` hangs, check that the LB is reachable from your machine (VPN, security groups, firewall rules). A `certificate required` handshake error on the ingest hostname means no client certificate was presented; complete Phase 5 first. A TLS validation error on the ingest hostname means the server cert hasn't finished issuing; go back to Phase 3.5 and resolve the issue there. - ---- - -## Phase 5 -- Issue mTLS Client Certificates (~10 min per cluster) - -Collectors authenticate with **two factors**: a client certificate (transport layer, proves the request comes from an authorized cluster) and an API key (application layer, proves the request is from a collector with `events:add` permission). A leaked key is useless without the cert; a stolen cert is useless without a valid key. - -### 5.1 Issue a certificate - -Each cluster running collectors needs its own client certificate. From the manifests directory: - -```bash -cd base/certificates/client-certs -./issue-client-cert.sh -``` - -Replace `` with a meaningful identifier (e.g. `us-east-1-prod`, `staging`). - -**Test it:** The script prints `==> Done!` and lists the output files. - -```bash -kubectl get certificate mtls-client- -n agenteye -``` - -Expected: `Ready: True`. - -Output files in `issued//`: - -| File | Purpose | -|---|---| -| `client.crt` | Client certificate (90-day validity) | -| `client.key` | Client private key | -| `ca.crt` | CA certificate for server verification | -| `collector-mtls-secret.yaml` | Ready-to-apply Kubernetes Secret for the collector cluster | - ---- - -### 5.1b Alternative delivery: AWS Secrets Manager - -If the consumer of the cert is a Kubernetes Pod that needs `client.crt` and `client.key` on disk -- the typical case when you run the agenteye-collector as a sidecar in your application pod -- push the cert bundle into AWS Secrets Manager. The application pod then mounts it via the [Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/) with IRSA (IAM Roles for Service Accounts, AWS's mechanism for giving a pod scoped AWS permissions), and certificate rotation is fully hands-off. - -```bash -cd base/certificates/client-certs -export AWS_REGION=us-east-1 # region where your workload runs -./issue-client-cert.sh --save-to aws-secrets-manager -``` - -On re-run (renewal), the script calls `PutSecretValue` on the same secret, so the ARN and name stay stable. The CSI Driver picks up the new version on its next rotation poll and rewrites the files inside the pod. - -**Prerequisites:** - -- `aws` CLI v2 authenticated to your AWS account. -- `jq` installed. -- `AWS_REGION` environment variable set. -- IAM permissions on your caller identity (scope `Resource` to `arn:aws:secretsmanager:::secret:agenteye/mtls-client/*`): - - `secretsmanager:CreateSecret` - - `secretsmanager:DescribeSecret` - - `secretsmanager:PutSecretValue` - - `secretsmanager:TagResource` - -**What the script does in this mode:** - -| Step | Action | -|---|---| -| 1 | Issues / re-extracts the cert via cert-manager (same as default mode). | -| 2 | Calls `DescribeSecret` on `agenteye/mtls-client/` to decide create-vs-update. | -| 3 | On first run: `CreateSecret` with a three-key JSON payload (`client.crt`, `client.key`, `ca.crt`), tagged `AgentEyeCluster=`. On subsequent runs: `PutSecretValue` to publish a new version; tag refreshed via `TagResource`. | -| 4 | Deletes `issued//` only after a successful upload. On any failure, the directory is preserved so you can retry. | - -**If the secret is scheduled for deletion**, the script fails with a clear error telling you to run `aws secretsmanager restore-secret --secret-id agenteye/mtls-client/` before retrying. - -For full pod wiring (SecretProviderClass, IRSA setup, rotation behavior, troubleshooting) see [Single-pod deployment](/agenteye/single-pod-deployment). - ---- - -### 5.2 Verify the certificate works - -Test the issued certificate against the mTLS ingress: - -```bash -curl -sk --cert issued//client.crt \ - --key issued//client.key \ - https://${PUBLIC_IP}/health -``` - -Expected: `{"status":"ok"}` - -**If it fails:** - -| Error | Cause | Fix | -|---|---|---| -| `certificate required` | Cert not being presented | Check file paths in the `curl` command | -| `bad certificate` | CA mismatch | Verify `mtls-ca-issuer` issued the cert: `kubectl describe certificate mtls-client- -n agenteye` | -| `connection refused` | Wrong hostname or LB not reachable | Check `/etc/hosts` or DNS | - ---- - -### 5.3 Deliver to the collector cluster - -Send `collector-mtls-secret.yaml` to the team operating the collector cluster. They apply it: - -```bash -kubectl apply -f collector-mtls-secret.yaml -n -``` - -Then configure the collector to mount the secret and use the cert paths: - -```json -{ - "tls_cert": "/etc/agenteye/tls/client.crt", - "tls_key": "/etc/agenteye/tls/client.key" -} -``` - -See [Collector installation](/agenteye/collector-installation) for complete collector setup including Kubernetes volume mounts. - -**Test it (in the collector cluster):** - -```bash -kubectl get secret agenteye-collector-mtls -n -``` - -Expected: the secret exists with 3 data keys (`client.crt`, `client.key`, `ca.crt`). - ---- - -### 5.4 Certificate lifecycle - -| Property | Value | -|---|---| -| Client cert validity | 90 days | -| Auto-renewal | cert-manager renews 15 days before expiry | -| CA validity | 10 years | -| Expiry alerts | CronJob alerts 30 days before expiry (Phase 6) | - -cert-manager auto-renews the certificate on the **AgentEye cluster**, but the renewed cert must be re-delivered to the collector cluster. Re-run `issue-client-cert.sh` and re-apply `collector-mtls-secret.yaml` before the old cert expires. - -If you are using `--save-to aws-secrets-manager` (see § 5.1b), re-run the same command. The script calls `PutSecretValue` on the same secret; pods mounting the secret via the Secrets Store CSI Driver pick up the new version on their next rotation poll (default: every hour), with no pod restart required. - ---- - -### 5.5 Revoke a certificate - -To immediately block a cluster's collector access: - -```bash -kubectl delete certificate mtls-client- -n agenteye -``` - -**Test it:** The `curl` command from step 5.2 now fails with a TLS handshake error. - ---- - -## Phase 6 -- Certificate Renewal Monitoring (~2 min) - -A built-in CronJob runs every 12 hours (03:00 and 15:00 UTC) and checks all client certificates labeled `agenteye.io/cert-type=mtls-client`. It alerts when any certificate is within 30 days of expiry. - -### 6.1 Enable Slack notifications (optional) - -```bash -kubectl create secret generic cert-renewal-notify-config \ - --namespace agenteye \ - --from-literal=SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL" -``` - -Without this secret, the CronJob still runs and logs certificate status to stdout. - -**Test it:** - -```bash -kubectl get secret cert-renewal-notify-config -n agenteye -``` - -Expected: the secret exists. - ---- - -### 6.2 Test the CronJob - -```bash -kubectl create job --from=cronjob/cert-renewal-check test-cert-check -n agenteye - -kubectl wait --for=condition=Complete job/test-cert-check -n agenteye --timeout=60s - -kubectl logs -n agenteye -l job-name=test-cert-check -``` - -Expected: a list of certificates with their expiry status. If the Slack webhook is configured, check the Slack channel for the alert message. - -**If it fails:** Check RBAC -- the CronJob's ServiceAccount needs `get, list` permissions on cert-manager Certificate resources. Verify with: `kubectl describe role cert-renewal-check -n agenteye`. - -Clean up the test job: - -```bash -kubectl delete job test-cert-check -n agenteye -``` - ---- - -## Phase 7 -- Verify End-to-End - -This phase confirms the entire pipeline works: health check, key creation, event ingestion, and dashboard display. - -> **Note:** The examples below reach the ingest endpoint by its raw LoadBalancer address (`${PUBLIC_IP}`) for convenience, which is why they pass `-k`; the server certificate is bound to `INGEST_DOMAIN`, not to the LB IP, so the hostname check is skipped. The ingest endpoint enforces mutual TLS on **every** path, so every call must also present a client certificate (`--cert`/`--key`). To validate the public certificate as well, target `https://ingest.your-company.example/...` instead of `${PUBLIC_IP}` and drop the `-k`. - -### 7.1 Health check - -```bash -curl -sk --cert issued//client.crt \ - --key issued//client.key \ - https://${PUBLIC_IP}/health -``` - -Expected: `{"status":"ok"}` with HTTP 200. - ---- - -### 7.2 Create scoped collector keys - -The admin key is for bootstrap and management. Create dedicated `events:add` keys for collectors: - -```bash -COLLECTOR_KEY=$(openssl rand -hex 32) - -curl -sk -X POST https://${PUBLIC_IP}/keys \ - --cert issued//client.crt \ - --key issued//client.key \ - -H "Authorization: Bearer ${ADMIN_KEY}" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "prod-collector", - "key": "'"${COLLECTOR_KEY}"'", - "permissions": ["events:add"] - }' -``` - -**Test it:** Response includes `"id"`, `"name": "prod-collector"`, `"permissions": ["events:add"]`, `"created_at"`. - -**Test it:** Verify the key appears in the key list: - -```bash -curl -sk https://${PUBLIC_IP}/keys \ - --cert issued//client.crt \ - --key issued//client.key \ - -H "Authorization: Bearer ${ADMIN_KEY}" -``` - -Expected: `prod-collector` appears in the response. - -See [API keys](/agenteye/api-keys) for the full key management reference. - ---- - -### 7.3 Ingest a test event - -```bash -echo '{"session_id":"test","agent_id":"smoke-test","type":"test","timestamp":"2026-04-20T00:00:00Z"}' \ - | curl -sk --cert issued//client.crt \ - --key issued//client.key \ - -H "Authorization: Bearer ${COLLECTOR_KEY}" \ - -H "Content-Type: application/x-ndjson" \ - --data-binary @- \ - https://${PUBLIC_IP}/events -``` - -Expected: `{"accepted":1,"skipped":0}` with HTTP 200. - -**If it fails:** - -| HTTP Status | Cause | -|---|---| -| 401 | Invalid or missing API key | -| 403 | Key lacks `events:add` permission | -| TLS handshake error | Client certificate issue -- see Phase 5 troubleshooting | - ---- - -### 7.4 Verify event appears in dashboard - -Open `https://agenteye.your-company.example` (your `DASHBOARD_DOMAIN`) in a browser. The certificate is publicly trusted, so there is no warning. - -> If the dashboard LoadBalancer is restricted by IP allowlist and you can't connect, verify your IP is allowed: -> ```bash -> kubectl get svc traefik-dashboard -n traefik-dashboard -o jsonpath='{.spec.loadBalancerSourceRanges}' -> ``` -> Keep in mind that Let's Encrypt renews the dashboard certificate over HTTP-01 on port 80, and source ranges apply to the whole LoadBalancer. Before restricting it to corporate ranges, coordinate a DNS-01 solver with support or renewals fail silently. - -**Test it:** The smoke-test event should appear in the event list with session `test` and agent `smoke-test`. - -**If it fails:** Check dashboard logs (`kubectl logs -n agenteye -l app=dashboard --tail=50`). Verify `AGENTEYE_SERVER_URL` and `AGENTEYE_API_KEY` are set correctly. - ---- - -### 7.5 Test the backup CronJob - -```bash -kubectl create job --from=cronjob/agenteye-backup test-backup -n agenteye - -kubectl wait --for=condition=Complete job/test-backup -n agenteye --timeout=300s - -kubectl logs -n agenteye -l job-name=test-backup -``` - -Expected: `Backup created: agenteye-YYYYMMDD-HHMMSS.tar.gz (NNN)` in the logs; the archive bundles the Postgres dump and the ClickHouse tables. - -> The S3 upload step ships wired into the CronJob and runs whenever `BACKUP_BUCKET` is set (the base ships a default bucket value). It is skipped only when `BACKUP_BUCKET` is empty or literally `PLACEHOLDER`. Point it at your own bucket and grant the `agenteye-backup` ServiceAccount write access before relying on it (see Backups section below). - -Clean up: - -```bash -kubectl delete job test-backup -n agenteye -``` - ---- - -### 7.6 Provision organizations (multi-tenant) - -Skip this for a single-tenant deployment; all data lives in the built-in `default` org and nothing here is required. - -If you are running multiple isolated tenants, organizations and their memberships are created with the **`agenteye-orgctl`** CLI. It ships **inside the server image** (alongside `agenteye-server`) and you run it **inside the existing `server` Deployment with `kubectl exec`; there is no separate pod, Job, or Deployment, and no HTTP API or dashboard button for tenant lifecycle.** Running it in the server pod means it reuses the pod's `DATABASE_URL`, `CLICKHOUSE_URL`, and the `ORG_CH_SECRET` from §2.6. - -> **Note:** Complete §2.6 first. `org create` refuses to run while the server is still on the built-in dev `ORG_CH_SECRET`, and the per-org ClickHouse user it provisions depends on that secret being strong and stable. - -**Create an org and add its first admin:** - -```bash -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl org create --slug acme --name "Acme Corp" - -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl member add --org acme --email alice@acme.example --set admin -``` - -The new member receives an OTP on first dashboard login and then works entirely in the UI under the org's URL prefix (e.g. `/acme/...`). - -**Other commands** (run the same `kubectl -n agenteye exec deploy/server -- agenteye-orgctl …` way): - -| Command | What it does | -|---|---| -| `org list` | List organizations and their state. | -| `org rename --slug --name ` | Rename an org (slug unchanged). | -| `org delete --slug ` | Soft-delete + drop the org's ClickHouse user; **data retained**. | -| `org purge --slug ` | Irreversible data wipe; org must be `delete`d first; never the `default` org. | -| `member list --org ` | List members and their permissions. | -| `member update --org --email [--set ...] [--add ...] [--remove ...]` | Change a member's permissions. | -| `member remove --org --email ` | Remove a member from the org. | - -Builtin permission sets are `admin`, `standard`, and `read-only`. **Per-org API keys are still minted in the dashboard/API by org members (§7.2 shows the keys API); only the org + member lifecycle is operator-only.** Full reference and a worked example: [Tenant management](/agenteye/tenant-management). - ---- - -## Post-Deployment Checklist - -Use this checklist to confirm everything is working. Every item should be checked before handing off to collectors. - -- [ ] All pods `Running` in `agenteye` namespace -- [ ] PostgreSQL PVC bound (50Gi) and ClickHouse PVC bound (100Gi) -- [ ] All 3 certificates `Ready: True` -- [ ] Both LoadBalancer IPs assigned -- [ ] DNS or `/etc/hosts` configured and resolving -- [ ] `/health` returns HTTP 200 -- [ ] mTLS cert test passes (`curl` with client cert to `/health`) -- [ ] Scoped collector key created and tested -- [ ] Test event ingested (`accepted: 1`) -- [ ] Event visible in dashboard -- [ ] Client certificates issued for each collector cluster -- [ ] Backup CronJob tested manually -- [ ] Cert renewal CronJob tested manually -- [ ] Slack webhook for cert alerts configured (optional) -- [ ] Backup bucket configured in overlay (see below) -- [ ] Admin key and Postgres password stored in secrets manager - ---- - -## Backups - -A single `agenteye-backup` CronJob runs daily at 03:00 UTC. It dumps **both** stores: PostgreSQL (relational state) and ClickHouse (the `events` + `evaluations` analytics tables), then streams them as one compressed archive straight to the object storage you wire in your overlay. The `.tar.gz` is piped directly to the upload command and is never written to the pod's disk. - -Each run produces one object, `agenteye-.tar.gz`, which unpacks to: - -```text -postgres.sql # pg_dump of the relational database -events.sql # ClickHouse events table DDL -events.native # ClickHouse events data (Native format) -evaluations.sql # ClickHouse evaluations table DDL -evaluations.native # ClickHouse evaluations data -``` - -ClickHouse is read over its HTTP API (the same endpoint the server uses), so the job needs no ClickHouse client. Only the two physical tables are dumped; the server recreates all views (`agent_sessions`, the `analytics.*` aliases) and row policies on startup, so those tables are the complete picture. - -### Configure cloud upload - -The backup CronJob streams the compressed archive straight to object storage, so the shipped pipeline ends in an AWS S3 upload that reads from **stdin**. The base sets a default `BACKUP_BUCKET` you must override with your own bucket. The pipeline runs under `set -euo pipefail`, so a broken pipe aborts the job instead of reporting a false success. - -**On AWS:** set `BACKUP_BUCKET` to your bucket and grant the `agenteye-backup` ServiceAccount write access via IRSA (no script change needed). The shipped upload pipeline is: - -```bash -tar c -C "${DIR}" . | pv -f -i 15 -s "${DUMPS_BYTES}" | gzip \ - | aws s3 cp - "s3://${BACKUP_BUCKET}/${FILENAME}" -``` - -**On GCP / Azure:** swap only the final stage (`aws s3 cp -`) for the provider equivalent below. Replace it (don't append) because a leftover `aws s3 cp` stage under `set -euo pipefail` fails the job. Each equivalent reads the archive from stdin, matching the streaming pipeline; there is no on-disk `/tmp/${FILENAME}` to point at, because the archive is never written to disk. - -| Cloud | Final pipeline stage (reads stdin) | -|---|---| -| **AWS S3** | `aws s3 cp - "s3://${BACKUP_BUCKET}/${FILENAME}"` (ships by default) | -| **GCP Cloud Storage** | `gsutil cp - "gs://${BACKUP_BUCKET}/${FILENAME}"` | -| **Azure Blob** | `az storage blob upload --file /dev/stdin -c backups -n "${FILENAME}"` | - -One object per run means a bucket lifecycle rule (e.g. "delete after 30 days") prunes whole backups cleanly. - -The `agenteye-backup` ServiceAccount needs IAM permissions to write to the bucket. On AWS use IRSA; on GKE Workload Identity; on AKS Pod Identity. Annotate the ServiceAccount in your overlay accordingly. - -The raw dumps are buffered in a 30 GiB `emptyDir` at `/tmp`; the ClickHouse `events` dump dominates its size. If your data is larger, raise the volume's `sizeLimit` in your overlay. - -### Manual backup - -```bash -kubectl create job --from=cronjob/agenteye-backup manual-backup -n agenteye -``` - -### Restore from backup - -1. Download the archive from your bucket and unpack it: - ```bash - tar xzf agenteye-.tar.gz - # -> postgres.sql, events.sql, events.native, evaluations.sql, evaluations.native - ``` - -2. Scale the server down to stop writes during the restore: - ```bash - kubectl scale deployment/server -n agenteye --replicas=0 - ``` - -3. Restore PostgreSQL: - ```bash - kubectl cp postgres.sql agenteye/postgres-0:/tmp/postgres.sql - kubectl exec -n agenteye postgres-0 -- bash -c 'psql -U postgres -d agenteye < /tmp/postgres.sql' - ``` - -4. Restore ClickHouse. The schema already exists (the server creates it on startup; on a bare cluster, apply the `.sql` DDL files first). Re-inserting is safe; `ReplacingMergeTree` collapses duplicates on merge: - ```bash - kubectl cp events.native agenteye/clickhouse-0:/tmp/events.native - kubectl cp evaluations.native agenteye/clickhouse-0:/tmp/evaluations.native - kubectl exec -n agenteye clickhouse-0 -- sh -c \ - 'clickhouse-client --query "INSERT INTO agenteye.events FORMAT Native" < /tmp/events.native' - kubectl exec -n agenteye clickhouse-0 -- sh -c \ - 'clickhouse-client --query "INSERT INTO agenteye.evaluations FORMAT Native" < /tmp/evaluations.native' - ``` - -5. Scale the server back up: - ```bash - kubectl scale deployment/server -n agenteye --replicas=2 - ``` - -6. **Test it:** hit `/health` and confirm the dashboard shows historical events and evaluations. - ---- - -## Upgrading - -1. **Pull the latest manifests:** - ```bash - git pull - ``` - -2. **Check release notes** for breaking changes or new environment variables. - - Upgrading from a pre-`v0.0.1-beta.6` deploy: dashboard login now uses email + OTP and reads its config from a new `agenteye-auth` secret. Create it per §2.5 before applying, or no user will be able to log in via the UI. API-key traffic is unaffected either way. - -3. **Update image tags** in your overlay's `kustomization.yaml` if applicable: - ```yaml - images: - - name: ghcr.io/agenteye-enterprise/server - newTag: - - name: ghcr.io/agenteye-enterprise/dashboard - newTag: - ``` - -4. **Take a pre-upgrade backup:** - ```bash - kubectl create job --from=cronjob/agenteye-backup pre-upgrade-backup -n agenteye - kubectl wait --for=condition=Complete job/pre-upgrade-backup -n agenteye --timeout=120s - ``` - -5. **Apply:** - ```bash - kubectl apply -k overlays// - ``` - -6. **Test it -- monitor rollout:** - ```bash - kubectl rollout status deployment/server -n agenteye - kubectl rollout status deployment/dashboard -n agenteye - ``` - -7. **Test it -- health check:** - ```bash - curl -sk https://${PUBLIC_IP}/health - ``` - Expected: `{"status":"ok"}` - -8. **Test it -- dashboard:** Open in browser, verify it loads and shows existing events. - -### Rollback - -If something is wrong after upgrading: - -```bash -kubectl rollout undo deployment/server -n agenteye -kubectl rollout undo deployment/dashboard -n agenteye -``` - -**Test it:** `/health` returns 200, dashboard loads and shows events. - ---- - -## Creating Overlays for New Environments - -To deploy to a new cluster, create a new overlay by copying the existing `acme` overlay as a template: - -```bash -cp -r overlays/acme overlays/ -``` - -Files to customize in `overlays//`: - -| File | What to Change | -|---|---| -| `kustomization.yaml` | Image tags (`newTag` values) | -| `patches/server-env.yaml` | `DATABASE_URL` sslmode if needed | -| `patches/resource-limits.yaml` | CPU and memory requests/limits | -| `patches/cert-renewal-env.yaml` | Slack webhook URL | - -**Test it -- dry run:** Render the manifests without applying to verify correctness: - -```bash -kubectl kustomize overlays// -``` - -Inspect the output for correct hostnames, image tags, and resource values. Then apply: - -```bash -kubectl apply -k overlays// -``` - ---- - -## Architecture Reference - -### Kubernetes resources created - -| Kind | Name | Namespace | Purpose | -|---|---|---|---| -| Namespace | `agenteye` | -- | All AgentEye resources | -| StatefulSet | `clickhouse` | agenteye | ClickHouse 24.8 with 100Gi PVC -- canonical events + evaluations store | -| StatefulSet | `postgres` | agenteye | PostgreSQL 16 with 50Gi PVC -- relational/metadata store | -| StatefulSet | `redis` | agenteye | Redis 7.2 with 1Gi PVC -- optional shared cache | -| Deployment | `server` | agenteye | Rust/Axum API (2 replicas, pod anti-affinity) | -| Deployment | `dashboard` | agenteye | Next.js UI (2 replicas) | -| Deployment | `agent` | agenteye | Optional AI assistant (1 replica); inert until an LLM endpoint is configured | -| Service | `clickhouse` | agenteye | ClusterIP, port 8123 (HTTP) | -| Service | `postgres` | agenteye | ClusterIP, port 5432 | -| Service | `redis` | agenteye | ClusterIP, port 6379 | -| Service | `server` | agenteye | ClusterIP, port 8080 | -| Service | `dashboard` | agenteye | ClusterIP, port 3000 | -| Service | `agent` | agenteye | ClusterIP, port 9100 (internal-only) | -| PodDisruptionBudget | `clickhouse` | agenteye | Keeps at least one ClickHouse pod available during disruptions | -| NetworkPolicy | `agent` | agenteye | Restricts ingress to the AI assistant service | -| Certificate | `mtls-ca` | agenteye | Self-signed CA (10-year, ECDSA P-256) | -| Issuer | `mtls-ca-issuer` | agenteye | Issues client certs from the CA | -| Certificate | `ingest-tls` | agenteye | Server TLS for ingest endpoint (90-day, Let's Encrypt) | -| Certificate | `dashboard-tls` | agenteye | Server TLS for dashboard (90-day, Let's Encrypt) | -| IngressRoute | `ingest` | agenteye | Routes `/events`, `/health`, `/keys` via traefik-public with mTLS | -| IngressRoute | `dashboard` | agenteye | Routes `/` via traefik-dashboard (no mTLS) | -| TLSOption | `mtls-ingest` | agenteye | Requires and verifies client certificates | -| CronJob | `agenteye-backup` | agenteye | Daily Postgres + ClickHouse backup at 03:00 UTC | -| CronJob | `cert-renewal-check` | agenteye | Certificate expiry alerts, twice daily | -| ServiceAccount | `agenteye-backup` | agenteye | IAM-annotatable SA for backup bucket access | -| ServiceAccount | `cert-renewal-check` | agenteye | SA for cert-manager Certificate read access | -| Role | `cert-renewal-check` | agenteye | `get, list` on cert-manager Certificates | -| RoleBinding | `cert-renewal-check` | agenteye | Binds role to ServiceAccount | - -### PostgreSQL sizing and tuning - -The base manifests ship a single-replica PostgreSQL 16 StatefulSet with a tuned `postgresql.conf` mounted from a ConfigMap, so the dashboard's hot read paths get a real buffer cache and `work_mem` budget out of the box rather than the image defaults. - -**Default resources (acme overlay):** - -| Resource | Request | Limit | -|---|---|---| -| CPU | 1 | 4 | -| Memory | 4 Gi | 8 Gi | -| Storage | 50 Gi PVC (RWO, default StorageClass) | -- | - -**Default `postgresql.conf` highlights** (full file in `postgres/configmap.yaml`): - -| Setting | Value | Rationale | -|---|---|---| -| `shared_buffers` | `2GB` | ~25% of the 8 Gi limit | -| `effective_cache_size` | `6GB` | ~75% of the limit | -| `work_mem` | `32MB` | Headroom for dashboard sorts (events / evaluations history) | -| `maintenance_work_mem` | `512MB` | Faster `CREATE INDEX`, `VACUUM` | -| `max_connections` | `100` | Server pool (10 × 2 replicas) + readonly pool (4) + headroom | -| `random_page_cost` | `1.1` | SSD-backed PVC | -| `jit` | `off` | OLTP-shaped workload | -| `shared_preload_libraries` | `pg_stat_statements` | Per-query statistics | -| `log_min_duration_statement` | `250ms` | Surface slow queries in pod logs | - -**Override for a different pod size:** patch the `postgres-config` ConfigMap in your overlay. Keep `shared_buffers` around 25% of the memory limit and `effective_cache_size` around 75%. After editing, restart the pod (`kubectl -n agenteye rollout restart statefulset/postgres`) for the new `postgresql.conf` to take effect. - -**Inspect slow queries** once the pod is running. `pg_stat_statements` aggregates per-query timing, so this is the fastest way to find what is loading the database. Its views live in the `extensions` schema (not `public`) so they stay hidden from the read-only role that backs the dashboard's ad-hoc query workbench: - -```bash -kubectl exec -n agenteye postgres-0 -- psql -U postgres -d agenteye -c \ - "SELECT round(total_exec_time::numeric, 0) AS total_ms, calls, - round(mean_exec_time::numeric, 1) AS mean_ms, query - FROM extensions.pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20;" -``` - -### Environment variables - -**Server:** - -| Variable | Source | Value | -|---|---|---| -| `DATABASE_URL` | Constructed from `agenteye-postgres` secret | Built at deploy time from secret values (see `server/deployment.yaml`). Points at the relational/metadata store. | -| `CLICKHOUSE_URL` | Deployment manifest | `http://clickhouse:8123` -- **required**; the canonical events + evaluations store. The server refuses to start without it. | -| `CLICKHOUSE_DATABASE` | Deployment manifest | `agenteye` | -| `REDIS_URL` | Deployment manifest | `redis://redis:6379/0` -- optional cache and rate-limit backend. If Redis is unreachable, auth and OTP fall back to PostgreSQL (graceful degradation). | -| `ADMIN_KEY` | `agenteye-admin-key` secret | Bootstrap admin credential | -| `AGENT_API_KEY` | `agenteye-agent` secret (optional) | Seeds the protected `dashboard-assistant` key the AI assistant's agent service uses (the same value the agent reads as `AGENTEYE_API_KEY`). Unset means the key isn't seeded and the assistant stays off. See [AI assistant](/agenteye/assistant). | -| `AGENTEYE_AGENT_URL` | Deployment manifest | `http://agent:9100` -- where the **server** reaches the agent service to run the audit's agentic investigation (the same connection the dashboard uses). Unset ⇒ audits run policy-only. See [AI assistant](/agenteye/assistant). | -| `AGENTEYE_AGENT_TOKEN` | `agenteye-agent` secret (optional) | Shared secret the server presents to the agent for audit investigation calls; must match the agent's `AGENTEYE_AGENT_TOKEN`. Unset ⇒ audits run policy-only. | -| `ADMIN_EMAIL` | `agenteye-auth` secret (optional) | Bootstrap admin user email, upserted as protected on every startup | -| `ALLOWED_EMAILS` | `agenteye-auth` secret (optional) | Comma-separated allowlist for user creation/login; supports `*@domain` wildcards | -| `SMTP_HOST`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD`, `SMTP_FROM`, `SMTP_TLS` | `agenteye-auth` secret (optional) | SMTP relay for OTP delivery; if `SMTP_HOST` is unset, OTPs are logged to stdout | -| `DEFAULT_ORG_NAME`, `DEFAULT_ORG_SLUG` | `agenteye-auth` secret (optional) | First-boot-only display name and URL slug for the built-in `default` org; ignored after an org rename. Slug must be 1--40 lowercase alphanumerics with single internal hyphens. See §2.5. | -| `EVALUATOR_ENDPOINT`, `EVALUATOR_TOKEN` | `agenteye-evaluator` secret (optional) | Wire the evaluation pipeline. With the secret absent the pipeline is a no-op. See [Evaluation suite](/agenteye/evaluation-suite). | -| `SESSION_TTL_SECS` | Default | `86400` (24 h) | -| `OTP_TTL_SECS` | Default | `600` (10 min) | -| `ORG_CH_SECRET` | `agenteye-org-ch-secret` secret (optional; **required before a 2nd org**) | HMAC key from which each organization's per-tenant ClickHouse password is derived. Single-tenant clusters run on the built-in dev default; **set a strong, stable value in your own secret before provisioning a second organization** (see §2.6), and keep it identical across all server replicas; rotating it orphans every org's ClickHouse user until the next startup re-provisions them. | -| `RUST_LOG` | Deployment manifest | `info` | -| `LISTEN_ADDR` | Default | `0.0.0.0:8080` | -| `MAX_BODY_BYTES` | Default | `134217728` (128 MB) | - -> **Multi-tenant ClickHouse.** Per-org isolation requires SQL access management on ClickHouse; the bundled `deploy/base/clickhouse/` config enables it (access management + `users_without_row_policies_can_read_rows=false`) so the server can create one read-only ClickHouse user and row policy per organization. If you replace that config, carry those settings over. - -**Dashboard:** - -| Variable | Source | Value | -|---|---|---| -| `AGENTEYE_SERVER_URL` | Deployment manifest | `http://server:8080` | -| `AGENTEYE_API_KEY` | `agenteye-admin-key` secret | Same as `ADMIN_KEY` | -| `AGENTEYE_AGENT_URL` | Deployment manifest | `http://agent:9100` -- where the dashboard reaches the optional AI assistant | -| `AGENTEYE_AGENT_TOKEN` | `agenteye-agent` secret (optional) | Shared secret the dashboard presents to the AI assistant; the assistant stays inert until its LLM endpoint is configured. See [AI assistant](/agenteye/assistant). | -| `REDIS_URL` | Deployment manifest | `redis://redis:6379/0` -- optional; caches `validateSession()` results across replicas and shares the Next.js fetch cache. Unset => every authed request falls through to the server (correct, just slower). | -| `HOSTNAME` | Deployment manifest | `0.0.0.0` | - -### Traffic flow - -```mermaid -flowchart LR - subgraph collectors["Collector Clusters"] - COL["Collector Pod
+ client cert + API key"] - end - BR["Browser (VPN)
Developer"] - subgraph ae["AgentEye Cluster"] - TP["Traefik Public (mTLS)"] --> SRV["Server"] --> PG[("PostgreSQL")] - TD["Traefik Dashboard"] --> DASH["Dashboard"] - end - COL -->|"POST /events, 443"| TP - BR -->|"HTTPS, 443"| TD -``` - ---- - -## Troubleshooting - -The most useful commands for deployment-specific issues: - -```bash -# Pod details and events -kubectl describe pod -n agenteye - -# Application logs -kubectl logs -n agenteye -l app=server --tail=200 -kubectl logs -n agenteye -l app=dashboard --tail=200 - -# Certificate status -kubectl describe certificate -n agenteye - -# Recent cluster events -kubectl get events -n agenteye --sort-by=.lastTimestamp -``` - -For a comprehensive list of common issues and solutions, see [Troubleshooting](/agenteye/troubleshooting). - ---- - -## Related Documentation - -| Guide | Description | -|---|---| -| [Managed Deployment](/agenteye/managed-deployment) | Setup guide for managed deployment on your Kubernetes cluster | -| [Getting Started](/agenteye/getting-started) | End-to-end walkthrough with Docker Compose | -| [Deployment](/agenteye/deployment) | Docker-based deployment, env vars, configuration | -| [Tenant Management](/agenteye/tenant-management) | Provision organizations & members with the operator-only `agenteye-orgctl` CLI | -| [GitHub Token Setup](/agenteye/github-token) | Generate your GitHub PAT to access artifacts | -| [Collector Installation](/agenteye/collector-installation) | All collector install methods | -| [Python SDK](/agenteye/python-sdk) | Full SDK API reference | -| [API Keys](/agenteye/api-keys) | Creating and managing API keys | -| [Troubleshooting](/agenteye/troubleshooting) | Common issues and fixes | - ---- - -## Support - -Email `support@exosphere.host` for help with deployment. - ---- - -## Next steps - -- [Collector installation](/agenteye/collector-installation): start sending events from your agents into the cluster you just deployed. -- [API keys](/agenteye/api-keys): mint scoped keys for collectors and dashboard users. -- [Tenant management](/agenteye/tenant-management): provision isolated organizations if you run multiple teams. -- [Troubleshooting](/agenteye/troubleshooting): resolve pod, certificate, or backup issues. diff --git a/docs/agenteye/managed-deployment.mdx b/docs/agenteye/managed-deployment.mdx deleted file mode 100644 index c678085fe..000000000 --- a/docs/agenteye/managed-deployment.mdx +++ /dev/null @@ -1,204 +0,0 @@ ---- -title: "Managed Deployment on Your Kubernetes Cluster" -description: "AgentEye Managed Deployment on Your Kubernetes Cluster documentation." ---- - - -Get the full AgentEye platform running in your own cloud without operating a single database, certificate, or upgrade yourself. You provide a dedicated Kubernetes cluster, and Exosphere deploys, configures, runs, backs up, and upgrades every component for you, with all data staying inside your cloud account. Choose this path when you want AgentEye's value with none of the day-to-day operational work. - -AgentEye is a self-hosted platform for observing, evaluating, and improving your AI agents in production. It captures agent sessions, tool calls, model requests, and errors, turns them into searchable analytics and evaluations, and surfaces the results in a dashboard with an optional AI assistant. - ---- - -## Which deployment guide should I use? - -AgentEye runs several ways. Pick the one that matches who you want operating it: - -| Guide | Who operates it | Best when | -|---|---|---| -| [Managed deployment](/agenteye/managed-deployment) (this page) | Exosphere, in your cluster | You want the full platform with zero operational work | -| [Getting started](/agenteye/getting-started) | You (Docker Compose) | You want the fastest way to try AgentEye on one machine | -| [Single-pod deployment](/agenteye/single-pod-deployment) | You | You run the collector as a sidecar next to each agent for low-latency, in-pod telemetry | -| [Kubernetes deployment](/agenteye/kubernetes-deployment) | You | You want to deploy and scale every component on your own cluster | - -The rest of this page covers the **managed** model, where Exosphere operates everything inside a cluster you own. - ---- - -## Prerequisites - -- A **GitHub PAT** for pulling container images and downloading artifacts (see [GitHub token setup](/agenteye/github-token)) -- A **dedicated Kubernetes cluster** (see requirements below) -- A **storage bucket** for database backups -- **Network connectivity**: port 443 inbound to the cluster's load balancer - ---- - -## Step 1: Provision a Dedicated Kubernetes Cluster - -Create a Kubernetes cluster dedicated to AgentEye. It should not be shared with other workloads, so the full platform (application services, databases, analytics, and caching) runs in isolation without impacting your existing infrastructure. - -| Requirement | Details | -|---|---| -| **Distribution** | Any conformant Kubernetes: EKS, GKE, AKS, or self-managed | -| **Version** | 1.27 or later | -| **Node pool** | Minimum: **3 nodes, 4 vCPU / 8 GB RAM each** (standard general-purpose instances) | -| **Storage** | A default StorageClass that provisions block volumes (e.g. `gp3` on AWS, `pd-ssd` on GCP) | -| **Load Balancer** | The cluster must be able to provision cloud LoadBalancer services (default on EKS, GKE, AKS) | - -> **Note:** Exosphere installs and manages everything else inside the cluster: ingress controllers, TLS certificates, databases, caching, monitoring, and all application deployments. - ---- - -## Step 2: Grant Access to the AgentEye Team - -Exosphere needs cluster-admin access (or equivalent broad RBAC) to manage namespaces, custom resource definitions, ingress controllers, and storage provisioners. - -| Requirement | Details | -|---|---| -| **Access method** | IAM role (preferred for EKS/GKE), kubeconfig, or SSO-based access | -| **VPN / bastion** | If the Kubernetes API server is private, provide VPN credentials or bastion access for the Exosphere operations team | - ---- - -## Step 3: Configure Network Connectivity - -Your network team needs to allow inbound traffic on **port 443** to the cluster's load balancers. The deployment runs two separate load balancers: one for event ingestion (mTLS-protected) and one for the dashboard: - -| Traffic | Source | Destination | Security | -|---|---|---|---| -| **Event ingestion** | Collector pods in your clusters | Ingest LoadBalancer, port 443 | mTLS (client certificate) + API key | -| **Dashboard** | Developer browsers | Dashboard LoadBalancer, port 443 | HTTPS on your domain, passwordless email OTP sign-in | - -The ingest endpoint is protected by mutual TLS; collectors must present a valid client certificate **and** a valid API key on every request. The dashboard runs on its own load balancer and hostname, with sign-in restricted to your allowlisted email addresses/domains. - -**DNS records (one-time):** you create two CNAME records under a domain you control, one for the ingest endpoint and one for the dashboard (e.g. `agenteye.your-company.example`), pointing at the load balancer hostnames Exosphere provides. Exosphere then provisions publicly-trusted TLS certificates for both hostnames automatically, including renewals. - -> **Note:** Automated certificate issuance and renewal validate over HTTP on port 80 of each load balancer. If your security posture requires restricting the dashboard load balancer to corporate IP ranges, tell Exosphere first. We switch certificate validation to a DNS-based method (one extra DNS record on your side) so renewals keep working behind the restriction. - -> **Note:** Cluster nodes need internet access to pull container images from `ghcr.io`. Pulling the private platform images needs signed enterprise access. If your network restricts outbound traffic, allowlist `ghcr.io` or mirror images to your internal registry. - ---- - -## Step 4: Provide a Backup Storage Bucket - -Database backups are stored in a cloud storage bucket that you own. - -| Requirement | Details | -|---|---| -| **Service** | S3 (AWS), GCS (GCP), or Azure Blob Storage | -| **Access** | Grant write access to the cluster's nodes via IAM role for service accounts (IRSA on EKS, Workload Identity on GKE) or provide credentials | -| **Retention** | You control the bucket's lifecycle policy (retention period, archival rules). Exosphere writes backups; you decide how long to keep them | - -A single daily backup dumps both PostgreSQL (relational state) and ClickHouse (events and evaluations) into one compressed archive and uploads it to your bucket. Backups also run before every upgrade. - ---- - -## Step 5: Designate a Point of Contact - -Provide one person or Slack/Teams channel on your side for cluster-level issues: node health, cloud account limits, network changes. Day-to-day operations do not involve this contact. - ---- - -## What We Deploy - -Here is how the managed topology fits together. Everything inside the dashed boundary runs in the cluster you own, so your data never leaves your cloud account: - -```mermaid -flowchart TD - AC["Your agent clusters
collectors send events"] -->|"mTLS + API key, 443"| ILB - BR["Developer browsers
view the dashboard"] -->|"HTTPS, 443"| DLB - subgraph cloud["Your cloud account (data stays here)"] - ILB["Ingest LoadBalancer
mTLS ingress"] --> SRV["AgentEye Server"] - DLB["Dashboard LoadBalancer
HTTPS ingress"] --> DASH["Dashboard (+ AI assistant)"] - SRV <--> DASH - SRV --> CH[("ClickHouse")] - SRV --> PG[("Postgres")] - SRV --> RDS[("Redis cache")] - SRV --> BK["Daily backup to
your storage bucket"] - CM["cert-manager
TLS + mTLS certs"] - end -``` - -Once Exosphere has cluster access, the following components are deployed and managed for you: - -| Component | Role | -|---|---| -| **AgentEye Server** | HTTP API that receives events from collectors, runs analytics, and serves data to the dashboard | -| **Dashboard** | Web interface for viewing agent sessions, tool calls, model requests, and errors; hosts the optional read-only AI assistant | -| **ClickHouse** | Required canonical store for ingested events, analytics, and evaluations | -| **PostgreSQL** | Relational store for organizations, API keys, users, dashboards, and saved queries | -| **Redis** | Optional shared cache and rate-limit backend; the platform degrades gracefully if it is unavailable | -| **AI assistant (optional)** | Internal read-only assistant container; stays disabled until an LLM endpoint is configured | -| **Ingress controllers** | Two load balancers (one for mTLS-protected ingest, one for the dashboard) terminating TLS with publicly-trusted, auto-renewed certificates and enforcing mTLS on the ingest endpoint | -| **cert-manager** | Automates TLS certificate provisioning and mTLS client-certificate issuance | -| **Certificate monitoring** | A scheduled job checks certificate expiry and sends alerts (e.g. to Slack) as certificates approach renewal | - -The managed offering also operates the platform's evaluation pipeline, which scores agent activity against your evaluation criteria. See [AI assistant](/agenteye/assistant) and [Evaluation suite](/agenteye/evaluation-suite) for what these capabilities deliver. - ---- - -## What We Provide to You - -After deployment is complete, you receive: - -| Item | Details | -|---|---| -| **Dashboard URL** | A hostname under your domain (e.g. `https://agenteye.your-company.example`), served with a publicly-trusted, auto-renewed TLS certificate. You create one CNAME to the load balancer hostname we provide; sign-in is passwordless email OTP | -| **Collector endpoint** | The ingest hostname's `/events` path (e.g. `https://ingest.your-company.example/events`), mTLS-protected | -| **Client certificate bundle** | Per-cluster: client cert, private key, and CA cert delivered as a Kubernetes Secret manifest. Apply it once per cluster | -| **GitHub PAT** | For downloading collector binaries and Python SDK packages | -| **Collector API keys** | Scoped keys with `events:add` permission, one per collector deployment | -| **Installation guides** | Step-by-step docs for the collector and Python SDK | - ---- - -## What You Do After Setup - -Your only ongoing work is on your own agent machines, not the AgentEye cluster: - -1. **Install the collector** in each Kubernetes cluster running AI agents: mount the client certificate and configure the endpoint URL and API key. See [Collector installation](/agenteye/collector-installation). -2. **Integrate the Python SDK** into your agent code. See [Python SDK](/agenteye/python-sdk). -3. **Open the dashboard** in your browser to view agent activity. - -No cluster operations, no database management, no certificate renewals, no upgrades. - ---- - -## Security - -- **Data stays in your cloud account.** The cluster, storage, and databases all run in your environment. No data leaves your boundary. -- **You control access.** The cluster is in your account. You can audit, monitor, or revoke Exosphere's access at any time. All operations go through your cloud's audit log (CloudTrail, GCP Audit Logs, etc.). -- **mTLS on event ingestion.** Every collector request requires both a valid client certificate and an API key. A leaked key is useless without the cert; a stolen cert is useless without a valid key. -- **Dashboard access control.** The dashboard runs on its own load balancer, separate from event ingestion, and sign-in is passwordless email OTP restricted to the email addresses/domains you allowlist. An IP source-range allowlist on the load balancer is available on request; because automated certificate renewal must reach the load balancer, Exosphere pairs the restriction with DNS-based certificate validation so renewals keep working. -- **Per-cluster certificates.** Each of your clusters receives its own client certificate. If one cluster is compromised, that certificate is revoked independently without affecting others. - ---- - -## Deployment Timeline - -| Phase | Duration | Your involvement | -|---|---|---| -| **Cluster provisioning** | 1-2 days | Provision the cluster and grant Exosphere access | -| **Platform setup** | 1 day | None; Exosphere installs all infrastructure components | -| **Application deployment** | 1 day | None; Exosphere deploys the server, dashboard, and creates API keys | -| **Collector rollout** | 1-3 days | Install collectors in your clusters (with guidance from Exosphere) | -| **Production burn-in** | 1 week | None; Exosphere monitors and tunes | - -Typical total: **~2 weeks** from kickoff to production-ready. - ---- - -## Support - -For questions or issues, contact Exosphere at `support@exosphere.host`. - ---- - -## Next steps - -- [Getting started](/agenteye/getting-started): end-to-end walkthrough -- [Collector installation](/agenteye/collector-installation): install and configure the collector -- [Python SDK](/agenteye/python-sdk): instrument your agent code -- [API keys](/agenteye/api-keys): manage access and permissions -- [Troubleshooting](/agenteye/troubleshooting): common issues and fixes diff --git a/docs/agenteye/observability.mdx b/docs/agenteye/observability.mdx index 971075974..7074f91d5 100644 --- a/docs/agenteye/observability.mdx +++ b/docs/agenteye/observability.mdx @@ -1,6 +1,6 @@ --- title: "Observe" -description: "AgentEye Observe documentation." +description: "FailproofAI Observability Observe documentation." --- @@ -19,5 +19,5 @@ Four surfaces, each with its own page: - [Evaluations](/agenteye/evaluations): score every run for quality. - [Alerts](/agenteye/alerts): turn any threshold into a paging rule. -- [Audits](/agenteye/audits): let AgentEye find failure patterns across sessions for you. +- [Audits](/agenteye/audits): let FailproofAI Observability find failure patterns across sessions for you. - [CLI and agents](/agenteye/cli-and-agents): the same observability from your terminal. diff --git a/docs/agenteye/overview.mdx b/docs/agenteye/overview.mdx index 82930aecd..9fa02b953 100644 --- a/docs/agenteye/overview.mdx +++ b/docs/agenteye/overview.mdx @@ -1,16 +1,16 @@ --- -title: "What is AgentEye?" -description: "AgentEye What is AgentEye? documentation." +title: "What is FailproofAI Observability?" +description: "FailproofAI Observability What is FailproofAI Observability? documentation." --- -AgentEye is a self-hosted platform for observing, evaluating, and improving your AI agents in production. It records everything your agents do (every tool call, model request, hook, and error), scores the quality of each run, and surfaces the failures you didn't know to look for, all in a dashboard you run inside your own infrastructure. +FailproofAI Observability is a self-hosted platform for observing, evaluating, and improving your AI agents in production. It records everything your agents do (every tool call, model request, hook, and error), scores the quality of each run, and surfaces the failures you didn't know to look for, all in a dashboard you run inside your own infrastructure. -If you ship AI agents and you're tired of guessing why a run went wrong, this is the page to start on. It explains what AgentEye gives you and how the pieces fit together, before you install anything. +If you ship AI agents and you're tired of guessing why a run went wrong, this is the page to start on. It explains what FailproofAI Observability gives you and how the pieces fit together, before you install anything. -> **AgentEye is an enterprise product from Failproof AI.** Want to see it in action? Request a demo: email [nivedit@exosphere.host](mailto:nivedit@exosphere.host), [nikita@exosphere.host](mailto:nikita@exosphere.host), [nivedit@befailproof.ai](mailto:nivedit@befailproof.ai), or [nikita@befailproof.ai](mailto:nikita@befailproof.ai). +> **FailproofAI Observability is an enterprise product from FailproofAI.** Want to see it in action? Request a demo: email [nikita@befailproof.ai](mailto:nikita@befailproof.ai). -![An AgentEye session drawn as a git-style execution graph beside its event timeline, with a per-run breakdown of tools, models, and hooks in the right rail](/agenteye/images/session-detail.png) +![A FailproofAI Observability session drawn as a git-style execution graph beside its event timeline, with a per-run breakdown of tools, models, and hooks in the right rail](/agenteye/images/session-detail.png) *Every agent run is drawn as a git-style execution graph (left) beside its event timeline. Parallel sub-agents each get their own lane; the right rail breaks down the tools, models, hooks, and token spend for the run.* @@ -21,7 +21,7 @@ If you ship AI agents and you're tired of guessing why a run went wrong, this is Two short videos show the two things teams reach for first: tracing a run, and finding failures automatically.
- +
*Agent tracing: follow a single run step by step, from goal to tools to final answer.* @@ -30,24 +30,24 @@ Two short videos show the two things teams reach for first: tracing a run, and f
-*Failproof Audit: let AgentEye mine your logs across sessions and tell you what to fix.* +*Failproof Audit: let FailproofAI Observability mine your logs across sessions and tell you what to fix.* --- ## Why teams use it - **See what your agent actually did.** Every run becomes a readable, git-style execution graph: which tools ran in parallel, which sub-agents branched off, where it stalled, and what it spent. -- **Catch quality regressions automatically.** Connect a small scoring service and AgentEye scores every finished run, so a drop in helpfulness or a spike in hallucinations shows up on its own. +- **Catch quality regressions automatically.** Connect a small scoring service and FailproofAI Observability scores every finished run, so a drop in helpfulness or a spike in hallucinations shows up on its own. - **Find failures you didn't write a rule for.** Recurring audits mine your logs across sessions for error clusters, latency outliers, low scores, and stuck runs, then hand you ranked, evidence-backed findings. - **Get paged when it matters.** Threshold rules fire on error rate, latency, cost, or evaluator scores and open incidents you can acknowledge, assign, and resolve. - **Ask questions in plain English.** An in-dashboard AI assistant answers "how is quality trending in prod this week?" over your own data. Any change it makes is approval-gated. -- **Keep your data.** AgentEye is self-hosted: events, prompts, and analytics stay in infrastructure you control. +- **Keep your data.** FailproofAI Observability is self-hosted: events, prompts, and analytics stay in infrastructure you control. --- ## What you get -AgentEye is organized around three ideas (**observe**, **analyze**, and **admin**), mirrored in the dashboard's left sidebar. +FailproofAI Observability is organized around three ideas (**observe**, **analyze**, and **admin**), mirrored in the dashboard's left sidebar. **Observe** (the raw truth of what happened): @@ -75,46 +75,33 @@ AgentEye is organized around three ideas (**observe**, **analyze**, and **admin* **Admin** (run it for your team): - **[API keys](/agenteye/api-keys)**: scoped tokens for the collector, the dashboard, and the assistant. -- **[Users](/agenteye/tenant-management)**: passwordless, email-based sign-in with an allowlist. +- **Users**: passwordless, email-based sign-in with an allowlist. - **Settings**: per-org configuration, including model context-window overrides. --- ## How the pieces fit -Data flows in one direction, from your agent code to the dashboard: - -```mermaid -flowchart TD - SDK["Your agent
Python SDK"] -->|emits events| COL["agenteye-collector
batches and ships"] - COL --> SRV["AgentEye server
ingest + API"] - SRV --> CH[("ClickHouse
events + analytics")] - SRV --> PG[("PostgreSQL
orgs, users, keys, dashboards")] - SRV --> RDS[("Redis
optional cache")] - SRV --> DASH["AgentEye dashboard"] -``` +Data flows in one direction, from your agent code to the dashboard: your agent (via the Python SDK) emits events to the agenteye-collector, which ships them to the server, which serves the dashboard. Two optional services round it out — a scoring service (evaluations) and an AI assistant service (the in-dashboard chat). - **Python SDK**: you add a few `agenteye.event.*` calls to your agent; events are buffered locally. - **agenteye-collector**: a lightweight daemon on each agent machine that batches events and ships them to the server. -- **Server**: ingests events into ClickHouse (the analytics store) and keeps operational state in PostgreSQL; Redis is an optional cache. -- **Dashboard**: where you explore everything, reachable only through the server's API. +- **Server**: ingests your events and keeps operational state in your own databases. +- **Dashboard**: where you explore everything. - **Optional services**: a scoring service (evaluations), and an AI assistant service (the in-dashboard chat). For the vocabulary used throughout the docs (*event, session, evaluation, audit, finding, incident*), see [Concepts](/agenteye/concepts). --- -## Getting AgentEye - -AgentEye is an enterprise product from Failproof AI. It runs entirely in your own environment; you can deploy it as a single Docker Compose stack, a production Kubernetes install, a managed install on your cluster, or a single co-located pod. If you don't have access to the packages yet, request a demo and we'll get you set up: email [nivedit@exosphere.host](mailto:nivedit@exosphere.host), [nikita@exosphere.host](mailto:nikita@exosphere.host), [nivedit@befailproof.ai](mailto:nivedit@befailproof.ai), or [nikita@befailproof.ai](mailto:nikita@befailproof.ai). +## Getting FailproofAI Observability -Not sure which install path fits? See [Deployment options](/agenteye/deployment-options). +FailproofAI Observability is an enterprise product from FailproofAI, and it works alongside FailproofAI Enforcement — the policy and guardrail product — under the FailproofAI brand. It runs entirely in your own environment. If you don't have access to the packages yet, request a demo and we'll get you set up: email [nikita@befailproof.ai](mailto:nikita@befailproof.ai). --- ## Next steps -- [Concepts](/agenteye/concepts): the AgentEye vocabulary in one place. -- [Getting started](/agenteye/getting-started): deploy the stack and get your first event flowing. -- [Deployment options](/agenteye/deployment-options): Compose vs. Managed vs. Kubernetes vs. Single-pod. -- [Security](/agenteye/security): how AgentEye keeps your data isolated and in your control. +- [Concepts](/agenteye/concepts): the FailproofAI Observability vocabulary in one place. +- [Observability](/agenteye/observability): follow what your agents do, run by run. +- [Security](/agenteye/security): how FailproofAI Observability keeps your data isolated and in your control. diff --git a/docs/agenteye/python-sdk.mdx b/docs/agenteye/python-sdk.mdx index e2b9a8583..eeac22cff 100644 --- a/docs/agenteye/python-sdk.mdx +++ b/docs/agenteye/python-sdk.mdx @@ -1,55 +1,27 @@ --- title: "Python SDK" -description: "AgentEye Python SDK documentation." +description: "FailproofAI Observability Python SDK documentation." --- -See exactly what your AI agents did in production: every agent run, tool call, model request, hook, and human intervention. The AgentEye Python SDK records that trail from inside your agent code so you can debug, audit, and evaluate what happened. Use it whenever you want AgentEye to observe your agents. +See exactly what your AI agents did in production: every agent run, tool call, model request, hook, and human intervention. The FailproofAI Observability Python SDK records that trail from inside your agent code so you can debug, audit, and evaluate what happened. Use it whenever you want FailproofAI Observability to observe your agents. Under the hood, the SDK writes structured events to local JSONL files, and the collector daemon picks them up and ships them to the platform automatically. You do not manage those files yourself. -> **Tip:** New to AgentEye? Start with [Getting started](/agenteye/getting-started) for the full picture. This page is the complete SDK event reference. +> **Tip:** New to FailproofAI Observability? This page is the complete SDK event reference.
- +
--- ## Installation -Download the wheel from GitHub Releases using your `AGENTEYE_TOKEN`. If you do not yet have a token, see [GitHub token setup](/agenteye/github-token) for the setup steps and required permissions. - -**Using `gh` CLI + pip:** - -```bash -VERSION=0.0.1b9 -GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download "python-sdk/v${VERSION}" \ - --repo agenteye-enterprise/releases \ - --pattern 'agenteye-*.whl' -pip install "agenteye-${VERSION}-py3-none-any.whl" -``` - -**Using `gh` CLI + uv:** - -```bash -VERSION=0.0.1b9 -GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download "python-sdk/v${VERSION}" \ - --repo agenteye-enterprise/releases \ - --pattern 'agenteye-*.whl' -uv add "./agenteye-${VERSION}-py3-none-any.whl" -``` - -**Using curl (no `gh` CLI):** +Install the SDK from PyPI: ```bash -VERSION=0.0.1b9 -curl -fsSL \ - -H "Authorization: Bearer $AGENTEYE_TOKEN" \ - -L \ - "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/agenteye-enterprise/releases/releases/download/python-sdk%2Fv${VERSION}/agenteye-${VERSION}-py3-none-any.whl" \ - -o "agenteye-${VERSION}-py3-none-any.whl" -pip install "agenteye-${VERSION}-py3-none-any.whl" +pip install agenteye ``` --- @@ -84,7 +56,7 @@ agenteye.event.agent_end(session_id="run-001", agent_id="planner", outcome="succ ### Instrumenting a real call -In practice you wrap your existing agent code. Bracket a model call with `model_request` before and `model_response` after, so the two events span the real request and AgentEye can pair them: +In practice you wrap your existing agent code. Bracket a model call with `model_request` before and `model_response` after, so the two events span the real request and FailproofAI Observability can pair them: ```python import anthropic @@ -142,8 +114,7 @@ Call once before any `event.*` call. Safe to omit; defaults work out of the box. When `base_dir` is `None` (the default), the SDK reads `$AGENTEYE_HOME` if set, otherwise falls back to `~/.agenteye`. This matches the collector's own resolution, so a single `AGENTEYE_HOME` env var configures the shared event spool for both -the SDK and the collector, required for sidecar / single-pod deployments where -both processes must agree on the spool path. +the SDK and the collector. --- @@ -165,7 +136,7 @@ export AGENTEYE_ENVIRONMENT=production **Priority:** `configure(environment=...)` wins over the environment variable. If neither is set, defaults to `"dev"`. -The environment value appears as a first-class filter in the dashboard and is stored as an indexed column on the server for fast queries. +The environment value appears as a first-class filter in the dashboard and is stored on the server for fast queries. > **Warning:** Environment values must not contain a literal `,` comma. The dashboard filters use comma-separated multi-select on the wire (`?environment=prod,staging`), so an environment named `prod,blue` would be split into two values. Events with comma-containing environments are rejected at ingest time. @@ -181,7 +152,7 @@ That makes redaction your choice and your responsibility. If a prompt or tool pa ## Event Reference -Most events come in start/end pairs that share a correlation ID: `tool_use` and `tool_result` share a `tool_call_id`, `hook_triggered` and `hook_completed` share a `hook_id`, and `human_wait` and `human_input` share an `input_id`. Emit the start event, do the work, then emit the end event with the same ID. AgentEye matches the pair and computes `duration_ms` for you, so you never pass `duration_ms` yourself. +Most events come in start/end pairs that share a correlation ID: `tool_use` and `tool_result` share a `tool_call_id`, `hook_triggered` and `hook_completed` share a `hook_id`, and `human_wait` and `human_input` share an `input_id`. Emit the start event, do the work, then emit the end event with the same ID. FailproofAI Observability matches the pair and computes `duration_ms` for you, so you never pass `duration_ms` yourself. ![A session's git-style execution graph beside its event timeline, reconstructed from the paired events, with the tool/model/hook breakdown panel](/agenteye/images/session-detail.png) @@ -455,6 +426,5 @@ Each file is written atomically: the SDK writes to a temporary file and then ren ## Next steps -- [Collector installation](/agenteye/collector-installation): install the collector so these events actually ship to AgentEye. -- [Getting started](/agenteye/getting-started): the full setup, from install to seeing your first traces. -- [Troubleshooting](/agenteye/troubleshooting): what to check if events do not appear in the dashboard. +- [Event stream](/agenteye/event-stream): watch these events arrive live, colour-coded and filterable by environment, agent, and session. +- [Sessions](/agenteye/sessions): see how the paired events reconstruct each agent run as an execution graph and timeline. diff --git a/docs/agenteye/queries.mdx b/docs/agenteye/queries.mdx index 8652cb096..79475a3b8 100644 --- a/docs/agenteye/queries.mdx +++ b/docs/agenteye/queries.mdx @@ -1,10 +1,10 @@ --- title: "Queries" -description: "AgentEye Queries documentation." +description: "FailproofAI Observability Queries documentation." --- -Ask any question of your agent data and get an answer in seconds. AgentEye gives you a library of saved, ready-to-run queries over your events and evaluations, so you start from a working example instead of a blank SQL editor. +Ask any question of your agent data and get an answer in seconds. FailproofAI Observability gives you a library of saved, ready-to-run queries over your events and evaluations, so you start from a working example instead of a blank SQL editor. ![The saved-queries library: a grid of reusable queries, both built-in presets and custom ones](/agenteye/images/queries.png) @@ -28,7 +28,7 @@ Open any query and it lands in the SQL composer, where you can adjust it and see - **A schema sidebar** lays out the analytics tables and their columns, so you can shape a query without hunting for field names. - **A live result grid** returns rows the moment you run, so you iterate in seconds rather than guessing and re-guessing. -- **Read-only by design.** Queries run against your analytics store (ClickHouse) and are validated on the server: only `SELECT` and `WITH` statements are allowed, with a statement timeout and a row cap. An exploratory query can never modify your data, and a runaway one gets stopped for you. +- **Read-only by design.** Queries run against your event store and are validated on the server: only `SELECT` and `WITH` statements are allowed, with a statement timeout and a row cap. An exploratory query can never modify your data, and a runaway one gets stopped for you. Happy with the result? Save it back to the library so the whole team inherits it, or pin its output onto a dashboard as a line, bar, area, or pie tile. diff --git a/docs/agenteye/security.mdx b/docs/agenteye/security.mdx index 366eddc99..ecebf7a35 100644 --- a/docs/agenteye/security.mdx +++ b/docs/agenteye/security.mdx @@ -1,35 +1,35 @@ --- title: "Security" -description: "AgentEye Security documentation." +description: "FailproofAI Observability Security documentation." --- -AgentEye is built to sit close to your production agents, which means it sees your prompts, tool inputs, and outputs. This page explains how it keeps that data isolated, controlled, and in your hands. If you're evaluating AgentEye for a security review, start here. +FailproofAI Observability is built to sit close to your production agents, which means it sees your prompts, tool inputs, and outputs. This page explains how it keeps that data isolated, controlled, and in your hands. If you're evaluating FailproofAI Observability for a security review, start here. --- ## Your data stays in your environment -AgentEye is self-hosted. Events, prompts, model responses, and analytics are stored in databases you run: ClickHouse for events and analytics, PostgreSQL for operational state. Nothing is sent to a third-party SaaS for storage. In the [Managed](/agenteye/managed-deployment) model, Failproof AI operates the platform on a cluster you designate, and your data still lives in your own cloud account. +FailproofAI Observability is self-hosted. Events, prompts, model responses, and analytics are stored in your own databases, in your own environment. Nothing is sent to a third-party SaaS for storage, and your data stays in your own cloud account. --- ## Tenant isolation -One AgentEye instance can host many organizations, and each is isolated at the database layer, not just in the UI: +One FailproofAI Observability instance can host many organizations, and each is isolated at the storage layer — enforced by the database, not just the UI: -- **PostgreSQL** uses row-level security so an organization's operational data (users, keys, dashboards, saved queries) is scoped to that org. -- **ClickHouse** gives each organization a read-only user constrained by a row policy, so cross-org event reads are blocked at the database, and every ingested event is stamped with its owning org. +- An organization's operational data (users, keys, dashboards, saved queries) is scoped to that org, and cross-org reads are blocked by the database itself. +- Every ingested event is stamped with its owning org, so one organization's events can never be read by another. -Every dashboard route is scoped under an org slug (`//…`). See [Tenant management](/agenteye/tenant-management) for the isolation model in detail. +Every dashboard route is scoped under an org slug (`//…`). --- ## Sign-in -AgentEye uses passwordless, email-based sign-in. There is no password to phish or leak. A user requests a one-time code (or a one-click magic link), which is emailed to them and expires quickly. Sign-in is gated by an **allowlist**: only email addresses (or domains) you permit can authenticate. +FailproofAI Observability uses passwordless, email-based sign-in. There is no password to phish or leak. A user requests a one-time code (or a one-click magic link), which is emailed to them and expires quickly. Sign-in is gated by an **allowlist**: only email addresses (or domains) you permit can authenticate. -![The AgentEye sign-in screen, which sends a single-use code to your email](/agenteye/images/login.png) +![The FailproofAI Observability sign-in screen, which sends a single-use code to your email](/agenteye/images/login.png) --- @@ -57,12 +57,12 @@ So a teammate can ask "which agents errored most this week?" and act on the answ ## In transit -For production installs you terminate TLS at your ingress and run the collector-to-server and browser-to-server traffic over HTTPS. The [Managed](/agenteye/managed-deployment) and [Kubernetes](/agenteye/kubernetes-deployment) guides cover certificates and per-cluster TLS in detail. +All traffic runs over HTTPS. You terminate TLS with your own certificates, so collector-to-server and browser-to-server traffic is encrypted in transit. --- ## Next steps -- [Tenant management](/agenteye/tenant-management): how organizations are isolated. +- [Overview](/agenteye/overview): how FailproofAI Observability fits together. - [API keys](/agenteye/api-keys): scope access for the collector, dashboard, and assistant. -- [Deployment options](/agenteye/deployment-options): pick a path that fits your security posture. +- [Observability](/agenteye/observability): what FailproofAI Observability captures from your agents. diff --git a/docs/agenteye/sessions.mdx b/docs/agenteye/sessions.mdx index d8c44632f..066831cb4 100644 --- a/docs/agenteye/sessions.mdx +++ b/docs/agenteye/sessions.mdx @@ -1,17 +1,17 @@ --- title: "Sessions & Execution Graph" -description: "AgentEye Sessions & Execution Graph documentation." +description: "FailproofAI Observability Sessions & Execution Graph documentation." --- -Stop guessing why a run failed. AgentEye rolls every event from a run into one readable row, then draws the whole run as a git-style picture you can read in seconds, so you see exactly what your agent did, step by step. +Stop guessing why a run failed. FailproofAI Observability rolls every event from a run into one readable row, then draws the whole run as a git-style picture you can read in seconds, so you see exactly what your agent did, step by step. ![The Sessions list: one row per run, across environments and agents, with status pills and evaluation score badges](/agenteye/images/sessions-list.png) *One row per run: the status pill tells you how the run ended at a glance, and a score badge rides along once an evaluator is connected.*
- +
*Agent tracing: follow a single run step by step, from goal to tools to final answer.* diff --git a/docs/agenteye/single-pod-deployment.mdx b/docs/agenteye/single-pod-deployment.mdx deleted file mode 100644 index 912cfcd9d..000000000 --- a/docs/agenteye/single-pod-deployment.mdx +++ /dev/null @@ -1,455 +0,0 @@ ---- -title: "Single-Pod Deployment: Collector + Application Sidecar on EKS" -description: "AgentEye Single-Pod Deployment: Collector + Application Sidecar on EKS documentation." ---- - - -Run the AgentEye collector in the same Kubernetes Pod as your application so it can ship your agent's event logs to AgentEye without telemetry ever crossing a network boundary to be collected. Reach for this pattern when your app should hand off telemetry in-process rather than call across the network: you get low latency, tight lifecycle coupling, and per-tenant pod isolation. New to AgentEye? Read [Getting started](/agenteye/getting-started) first, then come back for the EKS sidecar setup. - -Here, your app and the collector share one in-pod event spool (a shared folder): your SDK writes event files into it, and the collector reads and uploads them. The handoff is a plain file read: no localhost port to expose, no service mesh to traverse. The mTLS (mutual-TLS) client certificate the collector uses to authenticate to your backend is delivered straight into the pod from AWS Secrets Manager, so credential rotation needs no manual file shuffling on your side. - -> **Tip:** This sidecar + shared-spool model is cloud-agnostic. Two containers sharing an `emptyDir` spool work on any Kubernetes distribution. Only the certificate-delivery path (AWS Secrets Manager + the Secrets Store CSI Driver + IAM Roles for Service Accounts, or IRSA) is specific to AWS / EKS. If you run elsewhere, keep the pod and spool layout and swap in your platform's secret-mount mechanism for Phases 2 and 3. - -> **Note:** Pick single-pod when your application should not call across a network boundary to reach the collector. For multi-app fleets that share one collector per node or per cluster, see the [Kubernetes deployment](/agenteye/kubernetes-deployment) guide instead. - ---- - -## At a glance - -```mermaid -flowchart TD - subgraph pod["Pod"] - APP["your-app
SDK writes JSONL"] -->|writes| SPOOL["emptyDir spool
/var/lib/agenteye (events + failed)
AGENTEYE_HOME"] - COL["agenteye-collector
sweeps events/, uploads over mTLS"] -->|reads| SPOOL - CSI["CSI mount (read-only)
/etc/agenteye/tls: client.crt, client.key, ca.crt"] -->|reads cert| COL - end - SM["AWS Secrets Manager
agenteye/mtls-client/(cluster)"] -->|"mounted by Secrets Store CSI (AWS + IRSA)"| CSI - EXO["Exosphere delivers the cert bundle
on issue / renewal"] --> SM - COL -->|"outbound: mTLS to AGENTEYE_URL"| INGEST["AgentEye ingest"] -``` - -Two data flows, two volumes: - -- **Events (in-pod):** your app's SDK writes `.jsonl` files into the shared `emptyDir` at `$AGENTEYE_HOME/events/`; the collector sweeper reads them and uploads. No localhost port, no loopback, pure shared-filesystem handoff. -- **mTLS cert (pod ← cloud):** the Secrets Store CSI Driver mounts the cert bundle from Secrets Manager into a read-only volume at `/etc/agenteye/tls/`, scoped to the collector container. - -**Two independent parties:** - -| Party | Responsibility | -|---|---| -| Exosphere | Issues the mTLS client certificate and delivers the bundle into **your** AWS account's Secrets Manager under a stable name. Re-publishes the renewed bundle into the same secret before expiry. | -| You | Install the Secrets Store CSI Driver, grant the pod's ServiceAccount read access to the secret via IRSA, and apply the Pod manifest. That's it. | - ---- - -## Prerequisites - -### In your AWS account / EKS cluster - -- An EKS cluster with an **OIDC provider** associated. Confirm with: - - ```bash - aws eks describe-cluster --name \ - --query "cluster.identity.oidc.issuer" --output text - ``` - - If the command returns an `https://oidc.eks.…` URL, OIDC is enabled. If not, associate one: - - ```bash - eksctl utils associate-iam-oidc-provider \ - --cluster --approve - ``` - -- The [Secrets Store CSI Driver](https://secrets-store-csi-driver.sigs.k8s.io/) and the [AWS provider](https://github.com/aws/secrets-store-csi-driver-provider-aws) installed in the cluster (see § Phase 2). - -- AWS CLI v2 and `kubectl` on your workstation. - -### Coordination with Exosphere - -Before you deploy, Exosphere delivers the mTLS client bundle into your AWS account's Secrets Manager and provides: - -- The **secret name** (convention: `agenteye/mtls-client/`) -- The **AWS region** the secret lives in -- The **AgentEye backend URL** to configure the collector with -- Your collector **API key** (see [API keys](/agenteye/api-keys)) - ---- - -## Phase 1: What Exosphere delivers - -You do not generate the mTLS client certificate yourself. Exosphere issues it and delivers the bundle directly into your AWS account's Secrets Manager, so the only credential material that ever lands in your environment is the finished, ready-to-mount secret. - -What arrives in your account: - -| Property | Value | -|---|---| -| Secret name | `agenteye/mtls-client/` (stable across renewals) | -| Region | The AWS region you nominated for your EKS cluster | -| Payload | A single JSON secret with three keys (`client.crt`, `client.key`, and `ca.crt`), each holding the PEM-encoded material | -| Tag | `AgentEyeCluster=` | - -On renewal, the same secret is updated in place with a new version, so the ARN and name never change; your `SecretProviderClass` and IAM policy keep working untouched. For the certificate lifecycle (validity, renewal cadence, expiry alerting) see the [Kubernetes deployment](/agenteye/kubernetes-deployment) guide. - ---- - -## Phase 2: Install the Secrets Store CSI Driver + AWS provider - -The Secrets Store CSI Driver is a Kubernetes add-on that mounts secrets from an external store (here, AWS Secrets Manager) into a pod as read-only files. Skip this step if you already run another workload that mounts AWS secrets via CSI. - -```bash -# CSI Driver -helm repo add secrets-store-csi-driver \ - https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts -helm install -n kube-system csi-secrets-store \ - secrets-store-csi-driver/secrets-store-csi-driver \ - --set syncSecret.enabled=true \ - --set enableSecretRotation=true \ - --set rotationPollInterval=1h - -# AWS provider -kubectl apply -f \ - https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml -``` - -**Verify:** - -```bash -kubectl get pods -n kube-system | grep -E 'csi-secrets-store|aws-provider' -``` - -Expected: `Running` for every pod. - -> **Note:** `rotationPollInterval=1h` sets how often the CSI Driver re-reads the secret. When Exosphere publishes a renewed certificate, Secrets Manager is updated in place; the CSI Driver picks it up on this interval and re-writes the mounted files. The collector reads the certificate files once at startup, so it begins presenting the renewed certificate only after a process restart; see § Certificate rotation for how to trigger one. - ---- - -## Phase 3: Grant the pod read access to the secret (IRSA) - -### 3.1 Create the IAM policy - -Save as `agenteye-mtls-reader-policy.json`: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "ReadAgentEyeMtlsBundle", - "Effect": "Allow", - "Action": [ - "secretsmanager:GetSecretValue", - "secretsmanager:DescribeSecret" - ], - "Resource": "arn:aws:secretsmanager:::secret:agenteye/mtls-client/-*" - } - ] -} -``` - -Substitute ``, ``, and ``. The trailing `-*` matches the six-character random suffix AWS appends to every secret ARN. - -Create the policy: - -```bash -aws iam create-policy \ - --policy-name AgentEyeMtlsReader- \ - --policy-document file://agenteye-mtls-reader-policy.json -``` - -### 3.2 Create the IAM role and bind it to the pod's ServiceAccount - -```bash -eksctl create iamserviceaccount \ - --name agenteye-pod \ - --namespace \ - --cluster \ - --role-name AgentEyePodRole- \ - --attach-policy-arn arn:aws:iam:::policy/AgentEyeMtlsReader- \ - --approve -``` - -This creates a `ServiceAccount` named `agenteye-pod` with the `eks.amazonaws.com/role-arn` annotation pointing at the new role. - -### 3.3 Required IAM permissions: summary - -| Permission | Scope | Why | -|---|---|---| -| `secretsmanager:GetSecretValue` | `arn:aws:secretsmanager:::secret:agenteye/mtls-client/-*` | CSI Driver reads the cert bundle on every mount + rotation tick. | -| `secretsmanager:DescribeSecret` | same | CSI Driver calls `DescribeSecret` to detect version changes between polls. | - -**Do NOT grant** `secretsmanager:PutSecretValue`, `secretsmanager:UpdateSecret`, or `secretsmanager:DeleteSecret` to the pod. The pod only ever reads the secret; writing new versions into it is handled by Exosphere when the certificate is issued or renewed. - -If the secret is encrypted with a customer-managed KMS key (not the default `aws/secretsmanager` key), also grant: - -```json -{ - "Effect": "Allow", - "Action": ["kms:Decrypt"], - "Resource": "arn:aws:kms:::key/", - "Condition": { - "StringEquals": { - "kms:ViaService": "secretsmanager..amazonaws.com" - } - } -} -``` - ---- - -## Phase 4: Deploy the Pod - -### 4.1 SecretProviderClass - -`agenteye-mtls-spc.yaml`: - -```yaml -apiVersion: secrets-store.csi.x-k8s.io/v1 -kind: SecretProviderClass -metadata: - name: agenteye-mtls - namespace: -spec: - provider: aws - parameters: - objects: | - - objectName: "agenteye/mtls-client/" - objectType: "secretsmanager" - jmesPath: - - path: '"client.crt"' - objectAlias: "client.crt" - - path: '"client.key"' - objectAlias: "client.key" - - path: '"ca.crt"' - objectAlias: "ca.crt" -``` - -The `jmesPath` block tells the AWS provider to split the JSON secret into three separate files on disk. The quoting in `'"client.crt"'` is required because JMESPath treats `.` as a sub-expression operator. - -```bash -kubectl apply -f agenteye-mtls-spc.yaml -``` - -### 4.2 Pod / Deployment manifest - -**How the two containers talk to each other.** The AgentEye SDK and the collector do not communicate over a network socket; there is no local HTTP port. The SDK writes event batches as `.jsonl` files into `$AGENTEYE_HOME/events/`, and the collector continuously watches that directory and uploads each file. For a sidecar pod this means: - -- Both containers mount the **same** `emptyDir` volume at the **same** path. -- Both containers set `AGENTEYE_HOME` to that path. -- Your application image must have the AgentEye SDK installed and configured (see [Python SDK](/agenteye/python-sdk)). - -> **Warning:** When `AGENTEYE_HOME` is unset, both the SDK and the collector default to `~/.agenteye`, and the two containers have different home directories, so they would land on two separate spools and the handoff would silently fail. Set `AGENTEYE_HOME` to the same explicit path on **both** containers. The §4.3 verification and the matching Troubleshooting row catch this if it is missed. - -`agenteye-pod.yaml` (Deployment with one replica, scale as needed): - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: my-app-with-collector - namespace: -spec: - replicas: 1 - selector: - matchLabels: - app: my-app-with-collector - template: - metadata: - labels: - app: my-app-with-collector - spec: - serviceAccountName: agenteye-pod - containers: - - name: app - image: - env: - # SDK writes event JSONL files into $AGENTEYE_HOME/events/ - - name: AGENTEYE_HOME - value: /var/lib/agenteye - volumeMounts: - - name: agenteye-spool - mountPath: /var/lib/agenteye - - - name: agenteye-collector - # Pin to a versioned :v tag for production; :beta-latest - # tracks the current beta build (:latest exists only for stable releases). - image: ghcr.io/agenteye-enterprise/collector:beta-latest - args: ["start"] - env: - # Must match the app container's AGENTEYE_HOME so the collector - # sweeps the same directory the SDK writes to. - - name: AGENTEYE_HOME - value: /var/lib/agenteye - - name: AGENTEYE_URL - value: "https://ingest.example.agenteye.com/events" - - name: AGENTEYE_KEY - valueFrom: - secretKeyRef: - name: agenteye-collector-api-key - key: key - - name: AGENTEYE_TLS_CERT - value: /etc/agenteye/tls/client.crt - - name: AGENTEYE_TLS_KEY - value: /etc/agenteye/tls/client.key - # Only when the AgentEye server presents a cert not signed by a - # publicly-trusted CA (e.g. self-signed by an in-cluster issuer - # because you have no real DNS domain). The CSI mount already - # exposes ca.crt alongside the client cert/key. - - name: AGENTEYE_TLS_CA - value: /etc/agenteye/tls/ca.crt - volumeMounts: - - name: agenteye-spool - mountPath: /var/lib/agenteye - - name: agenteye-mtls - mountPath: /etc/agenteye/tls - readOnly: true - livenessProbe: - exec: - command: ["agenteye-collector", "health"] - initialDelaySeconds: 10 - periodSeconds: 30 - - volumes: - # Shared event spool between app and collector. emptyDir is fine: - # events are transient and the collector drains them before pod - # termination on graceful shutdown. - - name: agenteye-spool - emptyDir: {} - # mTLS cert bundle, mounted read-only into the collector only. - - name: agenteye-mtls - csi: - driver: secrets-store.csi.k8s.io - readOnly: true - volumeAttributes: - secretProviderClass: agenteye-mtls -``` - -The `agenteye-collector-api-key` Secret holds the collector's API key (see [API keys](/agenteye/api-keys) for provisioning). - -**Apply:** - -```bash -kubectl apply -f agenteye-pod.yaml -``` - -### 4.3 Verify - -```bash -# Pod should be Running with 2/2 containers ready -kubectl get pods -n -l app=my-app-with-collector - -# Confirm the cert bundle was mounted -kubectl exec -n deploy/my-app-with-collector \ - -c agenteye-collector -- ls -l /etc/agenteye/tls/ -``` - -Expected: `client.crt`, `client.key`, `ca.crt` all present and read-only, owned by the container user. - -**Confirm the shared event spool is visible to both containers:** - -```bash -# In the collector, should show the events/ and failed/ subdirs that -# the collector auto-creates on startup: -kubectl exec -n deploy/my-app-with-collector \ - -c agenteye-collector -- ls /var/lib/agenteye/ - -# In the app, should show the same directory contents: -kubectl exec -n deploy/my-app-with-collector \ - -c app -- ls /var/lib/agenteye/ -``` - -If the two listings diverge, the volume is not mounted in both containers (or `AGENTEYE_HOME` differs); see § Troubleshooting. - -**End-to-end smoke test:** - -```bash -kubectl exec -n deploy/my-app-with-collector \ - -c agenteye-collector -- agenteye-collector flush -``` - -Expected: the collector uploads any queued events and prints a `Done: N/N uploaded, 0 failed.` summary. On an empty spool it still validates your configuration and reads and parses the TLS certificate first, then prints `No pending files.` and exits without ever exercising the upload/network path. So `flush` is a meaningful end-to-end smoke test only after your app has flushed at least one event. - -Note that `flush` exits non-zero **only** for local setup faults: missing configuration (no URL/key resolved) or an unreadable/unparseable TLS cert (check § Troubleshooting). A **wrong API key does not change the exit code**. The upload gets a `401`, the file is moved to `failed/`, and the command still prints `[FAILED] …` per file plus `Done: 0/N uploaded, N failed.` and exits `0`. To detect a bad key or a rejected upload, read the `Done:`/`[FAILED]` output or check for files landing in `$AGENTEYE_HOME/failed/`, not the exit code. - -Once events flow through, they show up in your AgentEye dashboard's events stream: - -![AgentEye dashboard events stream showing telemetry arriving after the smoke test](/agenteye/images/events-stream.png) - ---- - -## Certificate rotation - -The client certificate is valid for 90 days and is renewed automatically about 15 days before expiry; Exosphere then publishes the renewed bundle into the same Secrets Manager secret. From there, the in-pod flow is: - -1. Secrets Manager's secret gets a new `AWSCURRENT` version. ARN and name are unchanged. -2. Within `rotationPollInterval` (1h by default; see § Phase 2), the CSI Driver reads the new version and rewrites the files under `/etc/agenteye/tls/`. -3. The collector loads the certificate files **once at startup**, so it keeps presenting the previous certificate until the process restarts. To switch to the renewed material, restart the collector; a rolling restart is sufficient: - - ```bash - kubectl rollout restart deploy/my-app-with-collector -n - ``` - - To make this automatic, add a sidecar that watches `/etc/agenteye/tls/` (for example with `inotifywait`) and triggers the rollout when the files change. - -Because the previous certificate stays valid for roughly 15 days after renewal, you have a wide window to perform the restart with no interruption to ingestion. Exosphere publishes the renewed bundle for you; the only routine action on your side is to ensure the collector restarts within that window. - ---- - -## Troubleshooting - -| Symptom | Likely cause | Fix | -|---|---|---| -| Pod stuck in `ContainerCreating`, events show `MountVolume.SetUp failed for volume "agenteye-mtls"` | CSI provider can't reach Secrets Manager | Check IRSA is correctly bound: `kubectl describe sa agenteye-pod -n ` shows the `eks.amazonaws.com/role-arn` annotation. Check CloudTrail for the AssumeRole call. | -| Error: `AccessDeniedException: not authorized to perform secretsmanager:GetSecretValue` | IAM policy is scoped to the wrong ARN | The secret ARN suffix is random; use `agenteye/mtls-client/-*` with the wildcard, not the exact ARN. | -| Error: `ParameterNotFound` from AWS provider | Secret name mismatch between `SecretProviderClass.objects[].objectName` and the secret Exosphere delivered | Confirm the exact name with `aws secretsmanager list-secrets --filters Key=tag-key,Values=AgentEyeCluster`. | -| `jmesPath` error, only one file mounted | JMESPath syntax | The dots in the JSON keys require double-quoting: `'"client.crt"'`, not `client.crt`. | -| Collector logs `tls: bad certificate` after a renewal | The CSI Driver hasn't polled the new version yet, or the collector is still running with the previous certificate it loaded at startup | Confirm the mounted files have updated (`ls -l /etc/agenteye/tls/`), then restart the collector to load them: `kubectl rollout restart deploy/my-app-with-collector -n `. See § Certificate rotation. | -| Collector container crashloops with `no such file or directory: /etc/agenteye/tls/client.crt` | Volume not yet populated on first start; startup probe too aggressive | Add a small initial delay or use an init container that waits for the file to exist: `until [ -f /etc/agenteye/tls/client.crt ]; do sleep 1; done`. | -| CSI Driver pod `OOMKilled` | Default memory limits too low for clusters with many SecretProviderClasses | Bump `--set linux.resources.limits.memory=200Mi` in the Helm install. | -| App runs cleanly, `agenteye-collector flush` reports `No pending files.`, but your AgentEye dashboard shows no events | The app and collector are not sharing the event spool | Check that (a) both containers mount the same `agenteye-spool` emptyDir at the same path, and (b) both set `AGENTEYE_HOME` to that path. Run the two `ls /var/lib/agenteye/` checks from § 4.3; the listings must match. | - -**Logs to grab first:** - -```bash -kubectl logs -n kube-system -l app=secrets-store-csi-driver --tail=100 -kubectl logs -n kube-system -l app=csi-secrets-store-provider-aws --tail=100 -kubectl describe pod -n -l app=my-app-with-collector -``` - ---- - -## Reference: files on disk in the pod - -The pod has two data paths on disk: - -### mTLS cert bundle: `/etc/agenteye/tls/` (CSI, read-only, collector only) - -Mounted by the Secrets Store CSI Driver from AWS Secrets Manager. - -| File | Contents | Used by collector as | -|---|---|---| -| `client.crt` | PEM-encoded client certificate | `AGENTEYE_TLS_CERT` | -| `client.key` | PEM-encoded private key | `AGENTEYE_TLS_KEY` | -| `ca.crt` | PEM-encoded CA cert | `AGENTEYE_TLS_CA` (optional, only when the AgentEye server cert isn't publicly-trusted) | - -All three are mounted read-only and owned by the container user. They are re-written by the CSI Driver when the secret rotates. - -### Event spool: `$AGENTEYE_HOME/` (emptyDir, shared read-write between both containers) - -Shared via an `emptyDir` volume named `agenteye-spool`. - -| Path | Written by | Read by | Purpose | -|---|---|---|---| -| `$AGENTEYE_HOME/events/*.jsonl` | App (AgentEye SDK) | Collector sweeper | Event batches the SDK has flushed, awaiting upload. | -| `$AGENTEYE_HOME/failed/` | Collector (on upload failure) | You (when debugging) | JSONL files the collector could not upload after retries. | -| `$AGENTEYE_HOME/config.json` | You (optional) | Collector | Optional collector config file (alternative to env vars). | - -Both the `events/` and `failed/` subdirectories are auto-created by the collector on startup; no `initContainer` needed. - ---- - -## Next steps - -- [Collector installation](/agenteye/collector-installation): collector binary options, mTLS config reference, daemon modes. -- [Kubernetes deployment](/agenteye/kubernetes-deployment): multi-pod deployment, cert issuance internals, lifecycle and expiry alerts. -- [API keys](/agenteye/api-keys): provisioning the collector API key consumed by the pod. -- [Troubleshooting](/agenteye/troubleshooting): cluster-wide troubleshooting index. diff --git a/docs/agenteye/telemetry.mdx b/docs/agenteye/telemetry.mdx index a993790d7..5f5d32fe3 100644 --- a/docs/agenteye/telemetry.mdx +++ b/docs/agenteye/telemetry.mdx @@ -1,6 +1,6 @@ --- title: "Performance Metrics" -description: "AgentEye Performance Metrics documentation." +description: "FailproofAI Observability Performance Metrics documentation." --- @@ -24,7 +24,7 @@ A shared hover crosshair links the heat-map and the band, so a tail spike lines The Models page (shown up top) answers the two questions a bill always raises: which model, and how much. On top of the shared latency view, it adds **per-model token consumption**, **estimated cost**, and **context-window fill**, so runaway prompt growth and an impending compaction are visible before they surprise you. -AgentEye recognizes common model IDs automatically. If a window looks wrong, or you run a private model of your own, correct it or add one under **Settings**, in **model context windows**, and the fill readouts follow. +FailproofAI Observability recognizes common model IDs automatically. If a window looks wrong, or you run a private model of your own, correct it or add one under **Settings**, in **model context windows**, and the fill readouts follow. ## Tools: tell the slow apart from the broken diff --git a/docs/agenteye/tenant-management.mdx b/docs/agenteye/tenant-management.mdx deleted file mode 100644 index 6ad220d87..000000000 --- a/docs/agenteye/tenant-management.mdx +++ /dev/null @@ -1,179 +0,0 @@ ---- -title: "Tenant Management (organizations & members)" -description: "AgentEye Tenant Management (organizations & members) documentation." ---- - - -One AgentEye instance can host many fully isolated **organizations** (tenants), so a single deployment can serve separate teams, business units, or customers, with no tenant ever able to see another's data. You only need this page when you add a **second** organization. (Curious how the walls between tenants are enforced? See [Isolation model](#isolation-model) below.) - -Tenant lifecycle is operator-controlled, while everything members do day-to-day stays self-service in the dashboard. Organizations and their memberships are created and managed with the **`agenteye-orgctl`** CLI, which ships inside the server image and runs **inside the existing server pod**. Tenant creation and deletion are deliberately kept off the dashboard and HTTP API: there is **no HTTP API and no dashboard button** for tenant lifecycle, so it is gated behind cluster/pod shell access rather than the application surface. - -Within an org, members work entirely in the dashboard and API: they sign in, switch between the orgs they belong to, manage their own API keys, build dashboards and saved queries, and configure alerts for their org. The division is clean: operators provision and decommission tenants and their members via the CLI; members run everything inside a tenant through the UI. - -> **Note:** Single-tenant deployments need none of this. A single-tenant install runs without any operator action. All data, users, and keys live in a built-in `default` organization that is provisioned automatically. You only need this page when you decide to add a second org. - ---- - -## Prerequisites - -Before you create your **second** organization (the built-in `default` org needs nothing): - -- **PostgreSQL 15+.** The org-membership schema uses a column-list `ON DELETE SET NULL` foreign key that requires PostgreSQL 15+. Upgrade PostgreSQL before provisioning a second org. -- **A strong, stable `ORG_CH_SECRET`.** Each org's ClickHouse password is derived as `HMAC(ORG_CH_SECRET, org_id)`, so the publicly-known built-in dev default would yield publicly-derivable per-org credentials. `agenteye-orgctl org create` **refuses to run while `ORG_CH_SECRET` is unset or left at the built-in dev default**. Set your own value first (see [Deployment → environment variables](/agenteye/deployment) and, on Kubernetes, [§2.6 of the Kubernetes guide](/agenteye/kubernetes-deployment#26-multi-tenant-org-isolation-key-optional)). Keep it identical across all server replicas and don't rotate it casually; rotating it orphans every org's ClickHouse user until the next startup re-provisions them. - ---- - -## Running the CLI - -`agenteye-orgctl` is shipped in the **same image as the server** (alongside `agenteye-server`). You do **not** deploy a separate pod, Job, or Deployment for it; you exec it inside the server pod that is already running, so it reads the same `DATABASE_URL`, `CLICKHOUSE_URL`, and `ORG_CH_SECRET` the server uses. - -**Kubernetes:** - -```bash -kubectl -n agenteye exec deploy/server -- agenteye-orgctl -``` - -**Docker Compose:** - -```bash -docker compose exec server agenteye-orgctl -``` - -The examples below show the bare `agenteye-orgctl ` for brevity; prefix each with whichever of the two lines above matches your deployment. - ---- - -## Command reference - -### Organizations - -| Command | What it does | -|---|---| -| `org create --slug --name ` | Create a new organization. Refuses to run while `ORG_CH_SECRET` is unset or left at the built-in dev default (set your own first, see Prerequisites). Provisions the org's read-only ClickHouse user + row policy. | -| `org list` | List all organizations (slug, name, and lifecycle state). | -| `org rename --slug --name ` | Change an organization's display name. The slug (used in URLs and keys) is unchanged. | -| `org delete --slug ` | **Soft-delete** the org and drop its ClickHouse user. Data is **retained**. This revokes access and frees the per-org ClickHouse credential, but does not erase events. Reversible by ops; safe first step before a purge. | -| `org purge --slug ` | **Irreversible data wipe.** The org must already be `delete`d. Never permitted on the built-in `default` org. Use only when you are certain the tenant's data should be destroyed. | - -> **Note:** A slug must be 1-40 characters, using lowercase letters, digits, and single internal hyphens only (no leading, trailing, or doubled hyphens). Reserved top-level route names are rejected. The slug appears in URLs and API keys and can't be changed later (`org rename` only changes the display name), so pick it deliberately. - -### Members - -| Command | What it does | -|---|---| -| `member add --org --email [--set ] [--add perm1,perm2] [--remove perm3] [--protected]` | Add a member to an org. Optionally start from a builtin permission set, then add/remove individual permissions. `--protected` pins the member so the dashboard can't remove or demote them (see below). The new member gets a one-time passcode (OTP) on their first dashboard login. | -| `member list --org ` | List the org's members. The output columns are `EMAIL`, `SET` (the builtin set the member started from, or `-`), `PROT` (whether the member is protected), and `PERMISSIONS` (their effective permissions). An email shown with a trailing `*` is an instance admin; they have access to every org. | -| `member update --org --email [--set ...] [--add ...] [--remove ...] [--protected \| --unprotect]` | Change a member's permissions and/or protected flag. `--set` replaces from a builtin set; `--add` / `--remove` adjust individual permissions; `--protected` / `--unprotect` toggle protection. Passing only `--protected`/`--unprotect` (no grant flags) changes protection alone and leaves existing permissions untouched. | -| `member remove --org --email ` | Remove a member from the org. Refuses if the member is protected; `--unprotect` them first. (A person can be a member of several orgs; this only affects the named org.) | - -A person can be a member of more than one org with **different** permissions in each, e.g. an admin in one org and read-only in another. Each membership is administered independently per org: granting or changing a person's permissions in one org has no effect on their membership in any other. - -### Protected members (an un-removable org admin) - -Protection guarantees an org can never accidentally lock itself out of self-management. By default an org's own admins can add and remove each other through the dashboard's self-service users page, so they could remove the last admin and leave the org with no one able to manage it. - -![The Users page: a card per dashboard user with their email, granted permissions, and edit/disable controls](/agenteye/images/users.png) - -To prevent that, mark one member **protected**: - -```bash -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl member add --org acme --email owner@acme.example --set admin --protected -``` - -A protected member **cannot be removed or demoted through the dashboard**; those actions return an error. Only an operator can change them, and only via this CLI: run `member update --org acme --email owner@acme.example --unprotect` first, then remove or demote. This guarantees every org keeps at least one admin its own members can't lock out, while keeping tenant control operator-only. Protection is **per-org**; protecting someone in one org has no effect on their membership in another. - -### Builtin permission sets - -`--set` accepts one of three builtin sets, applied per org: - -| Set | Intended for | -|---|---| -| `admin` | Full access within the org, including managing the org's API keys and users. | -| `standard` | Day-to-day use: read everything, run queries, trigger evaluations, acknowledge incidents, and use the AI assistant. Authoring dashboards and saved queries needs the `admin` set (`dashboards:write` / `queries:write`). | -| `read-only` | View-only access to the org's data and dashboards. | - -Start from a set with `--set`, then fine-tune with `--add` / `--remove` using the individual permission tokens listed in [API Keys](/agenteye/api-keys). The permission tokens themselves are identical to those used for API keys. - ---- - -## Worked example - -Provision a new `acme` tenant, add its first admin, let them mint a key, then decommission the org. - -**1. Create the org** (`ORG_CH_SECRET` must already be set to a strong, stable value, not unset or the built-in dev default): - -```bash -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl org create --slug acme --name "Acme Corp" -``` - -**2. Add the first member as an org admin:** - -```bash -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl member add --org acme --email alice@acme.example --set admin -``` - -Alice receives an OTP the first time she signs in to the dashboard. From then on she works entirely in the UI under her org's URL prefix (e.g. `/acme/sessions`). - -**3. Mint a per-org API key (in the dashboard):** - -The operator does **not** mint per-org data keys from the CLI. Alice (or any org member with `keys:create`) creates collector / dashboard keys for the `acme` org from the dashboard's **Keys** page. Every key she creates is automatically stamped with her org and can only ever read or write `acme`'s data. See [API Keys](/agenteye/api-keys). - -**4. Adjust a member later:** - -```bash -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl member update --org acme --email alice@acme.example --add alerts:write - -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl member list --org acme -``` - -**5. Soft-delete the org** (revokes access + drops its ClickHouse user; data retained): - -```bash -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl org delete --slug acme -``` - -**6. Purge the org** (irreversible; only after a soft-delete; never the `default` org): - -```bash -kubectl -n agenteye exec deploy/server -- \ - agenteye-orgctl org purge --slug acme -``` - -On Docker Compose, replace each `kubectl -n agenteye exec deploy/server --` prefix with `docker compose exec server`. - ---- - -## Division of responsibilities - -Everything an org member needs day-to-day is self-service in the dashboard and API, scoped automatically to their current org: - -- **Per-org API keys** are created and managed by org members in the dashboard (or via the keys API with a key that carries `keys:create`). The CLI does **not** mint data keys. See [API Keys](/agenteye/api-keys). -- **Org switching** is built into the dashboard; members switch between the orgs they belong to from the org switcher, and org-scoped pages live under `//…`. -- **Dashboards, saved queries, alerts, and all data use** happen entirely in the UI and API, scoped to the member's current org. - -The operator, using `agenteye-orgctl`, owns only the org + member **lifecycle**: create / rename / delete / purge an org, and add / list / update / remove a member. - ---- - -## Isolation model - -Every row of data (events, evaluations, sessions, dashboards, saved queries, alerts, API keys, and members) belongs to exactly one org. Three layers keep tenants apart: - -- **Application-level scoping (primary).** Every request is scoped to its org with explicit `org_id` predicates, so the server only ever reads or writes the caller's own org. -- **ClickHouse per-org row policy.** The high-volume events and evaluations live in ClickHouse, where each org gets a dedicated read-only ClickHouse user with a per-org row policy. Even untrusted analytics SQL a member runs can never read another tenant's rows. -- **PostgreSQL row-level security.** On the read-only query path (`/queries/run`), Postgres row-level security adds defence-in-depth, narrowing what that path can see even if an application-level filter were ever missing. The server's own write connection runs as the table owner and relies on the app-level `org_id` scoping above. - ---- - -## Next steps - -- [Deployment](/agenteye/deployment): `ORG_CH_SECRET` and the rest of the server environment. -- [Kubernetes deployment](/agenteye/kubernetes-deployment): §2.6 creates the `agenteye-org-ch-secret` Secret before your first multi-tenant org. -- [API keys](/agenteye/api-keys): the per-org key model and the permission tokens used by `--add` / `--remove`. -- [Troubleshooting](/agenteye/troubleshooting): multi-tenant provisioning and ClickHouse-isolation issues. diff --git a/docs/agenteye/troubleshooting.mdx b/docs/agenteye/troubleshooting.mdx deleted file mode 100644 index dd8dffda9..000000000 --- a/docs/agenteye/troubleshooting.mdx +++ /dev/null @@ -1,741 +0,0 @@ ---- -title: "Troubleshooting" -description: "AgentEye Troubleshooting documentation." ---- - - -Is something in your AgentEye deployment misbehaving? This page maps the symptom you are seeing to a concrete cause and fix, using only the logs and commands you already have, with no extra monitoring stack to stand up. Reach for it whenever the dashboard, collector, SDK, or server does something unexpected, from a first-run `docker pull` failure to a ClickHouse memory error under load. - -The fixes below cover three stages of running AgentEye yourself: pre-deploy (downloading and authenticating), day-to-day operations on Kubernetes or Docker, and SDK/collector instrumentation. Dashboard pages are org-scoped under `//…`, and the events stream is the org home (`//`). Page names in this guide (for example `/sessions`, `/queries`) refer to those org-scoped routes. - -## Start here - -Not sure which section you need? Match what you are seeing: - -| Symptom | Go to | -|---|---| -| No events show up in the dashboard | [Collector issues](#collector-issues), then [Dashboard issues](#dashboard-issues) | -| Can't sign in, or "OTP rate-limited" in the logs | [Server issues](#server-issues) | -| `docker pull` or `gh release download` fails | [Authentication issues](#authentication-issues) | -| Dashboard analytics pages are slow or erroring under load | [ClickHouse-backed evaluations / sessions / queries](#clickhouse-backed-evaluations--sessions--queries) | -| The AI assistant bubble is missing or errors | [AI assistant issues](#ai-assistant-issues) | - -## What's on this page - -- [Viewing logs](#viewing-logs): read server and dashboard logs, trace one request, turn up verbosity -- [Authentication issues](#authentication-issues): `docker pull` and release-download failures -- [Server issues](#server-issues): startup, health probes, Redis, first-boot settings -- [Collector issues](#collector-issues): events not uploading, TLS and certificate errors -- [Dashboard issues](#dashboard-issues): blank page, missing events, slow analytics pages -- [AI assistant issues](#ai-assistant-issues): the assistant bubble, "not configured" errors -- [CLI issues](#cli-issues): install and startup failures -- [Python SDK issues](#python-sdk-issues): events not written, reserved-field errors -- [Health monitoring issues](#health-monitoring-issues): Slack alerts, readiness flapping -- [Certificate monitoring issues](#certificate-monitoring-issues): renewal-check CronJob, expired client certs -- [Backup issues](#backup-issues): disk space, upload failures -- [ClickHouse-backed evaluations / sessions / queries](#clickhouse-backed-evaluations--sessions--queries): empty sidebar, memory limits -- [Multi-tenancy (organizations)](#multi-tenancy-organizations): upgrade errors, cross-org isolation -- [Audits](#audits): audits not running, no findings, email delivery - ---- - -## Viewing Logs - -AgentEye does not bundle a logging or monitoring stack. Both the server and dashboard write structured logs to **stdout**, so you can read them directly with `kubectl` or `docker`; no aggregator required. - -### Kubernetes - -Follow live logs for the server and dashboard: - -```bash -kubectl logs -n agenteye -l app=server -f --timestamps -kubectl logs -n agenteye -l app=dashboard -f --timestamps -``` - -Useful variants: - -| Goal | Command | -|---|---| -| Last 200 lines (no follow) | `kubectl logs -n agenteye -l app=server --tail=200 --timestamps` | -| Logs from the previous crash | `kubectl logs -n agenteye --previous` | -| Tail all replicas at once | `kubectl logs -n agenteye -l app=server --max-log-requests=10 -f` | -| Postgres (StatefulSet) | `kubectl logs -n agenteye postgres-0 -f` | - -### Docker Compose - -```bash -docker logs -f agenteye-server -docker logs -f agenteye-dashboard -``` - -### Correlating a single request across dashboard and server - -Every dashboard request is tagged with a `request_id` and propagated to the server via the `x-request-id` header. The server echoes it in its response headers and in every log line it emits for that request. To trace one request end-to-end: - -1. Capture the id from the response header, e.g.: - ```bash - curl -i https://dashboard.example.com/api/events | grep -i x-request-id - # x-request-id: 9a7b0d6e-5e9b-4c0a-9f8a-5f1e4b5c0f3a - ``` -2. Grep both pods' logs for that id: - ```bash - REQ=9a7b0d6e-5e9b-4c0a-9f8a-5f1e4b5c0f3a - kubectl logs -n agenteye -l app=dashboard --tail=5000 | grep "$REQ" - kubectl logs -n agenteye -l app=server --tail=5000 | grep "$REQ" - ``` - -You will see the dashboard's `proxy passthrough`, `withAuth: authorized`, and `upstream response` lines alongside the server's `http request received` / `http request completed` pair, all sharing the same `request_id`. - -### JSON logs and `jq` - -Set `AE_LOG_JSON=1` on the dashboard (it is on by default when `NODE_ENV=production`) to emit one JSON object per line. Then filter structurally: - -```bash -kubectl logs -n agenteye -l app=dashboard --tail=5000 \ - | jq 'select(.level == "warn" or .level == "error")' - -kubectl logs -n agenteye -l app=dashboard --tail=5000 \ - | jq 'select(.route == "POST /api/keys")' -``` - -The Rust server emits tracing `key=value` pairs that grep well without `jq`: - -```bash -kubectl logs -n agenteye -l app=server --tail=5000 | grep 'status=5' # 5xx -kubectl logs -n agenteye -l app=server --tail=5000 | grep 'actor_key_id=' -``` - -### Turning up verbosity - -| Component | Env var | Example | -|---|---|---| -| Server | `RUST_LOG` | `RUST_LOG=debug` or `RUST_LOG=agenteye_server=debug,info` | -| Dashboard | `AE_LOG_LEVEL` | `AE_LOG_LEVEL=debug` | - -`debug` on the server adds a per-auth `api key authenticated` line. `debug` on the dashboard adds `upstream request`, `session validated`, and `proxy passthrough` lines. - -### Log retention - -Container stdout is ephemeral; kubelet rotates log files (default ~10 MiB per container) and keeps a small number on disk. Once a pod is deleted the logs are gone. If you need longer retention or cross-pod search, point your cluster at a log collector (Loki, CloudWatch, Cloud Logging, Datadog, etc.) that tails `/var/log/containers/`. AgentEye does not require or prescribe any specific choice. - ---- - -## Authentication Issues - -### `docker pull` fails with "unauthorized" - -Ensure you have authenticated Docker against GHCR with your `AGENTEYE_TOKEN`: - -```bash -echo $AGENTEYE_TOKEN | docker login ghcr.io -u x --password-stdin -``` - -The token must have `read:packages` permission on the `agenteye-enterprise` org. Contact `support@exosphere.host` if your token does not work. - -> **Note:** Pulling the `ghcr.io/agenteye-enterprise/*` images requires signed enterprise access. If you are still evaluating and have not signed, you can read these docs but not pull the images yet. - -### `gh release download` returns 404 or 401 - -- Confirm `AGENTEYE_TOKEN` is exported in your shell: `echo $AGENTEYE_TOKEN` -- Confirm you are using `GITHUB_TOKEN=$AGENTEYE_TOKEN gh release download ...` (the `gh` CLI reads `GITHUB_TOKEN`) -- The token needs `contents:read` on `agenteye-enterprise/releases` - ---- - -## Server Issues - -### Server fails with "invalid port number" - -The `POSTGRES_PASSWORD` (or another credential) contains URL-special characters (`/`, `+`, `=`) that break `DATABASE_URL` parsing. Regenerate the password using hex encoding: - -```bash -NEW_PASS=$(openssl rand -hex 24) -``` - -Then update the Kubernetes secret and the password inside Postgres (or recreate the `.env` for Docker Compose), and restart the server. See the full steps in the [Kubernetes deployment guide](/agenteye/kubernetes-deployment) § "PostgreSQL credentials". - -### Server exits immediately on startup - -Check the container logs: - -```bash -docker logs agenteye-server -``` - -Common causes: -- `DATABASE_URL` not set or malformed: the server will log the error and exit. -- Postgres is not reachable: confirm the Postgres container or managed DB is running and the host/port are correct. -- Migrations failed: check logs for SQL errors. - -### `GET /health` returns non-200 or times out - -The server may still be running migrations on first start. Wait a few seconds and retry: - -```bash -curl http://localhost:8080/health -``` - -If the issue persists, check `docker logs agenteye-server` for errors. - -### `GET /ready` returns 503 - -`/ready` is the readiness probe: it returns `503` when the server cannot reach **Postgres or ClickHouse**. The body names the failing dependency: - -```bash -curl -s http://localhost:8080/ready -# {"status":"not_ready","checks":{"postgres":"ok","clickhouse":"down","redis":"ok"}} -``` - -Fix whichever dependency it reports as `down`: is the ClickHouse/Postgres pod `Running`? Is `CLICKHOUSE_URL` / `DATABASE_URL` correct and reachable? On Kubernetes the pod reads `NotReady` until `/ready` recovers; that is expected and is exactly the signal health monitoring alerts on. Redis is never a cause: it is reported but does not fail readiness. - -### Collector returns 401 Unauthorized - -The collector's API key does not have `events:add` permission, or the key has been disabled. Create a new key with the correct permission: - -```bash -curl -s -X POST http://your-server:8080/keys \ - -H "Authorization: Bearer $ADMIN_KEY" \ - -H "Content-Type: application/json" \ - -d '{"name":"new-collector","permissions":["events:add"]}' -``` - -### Authed requests suddenly got slow (~200ms instead of ~5ms) - -This is the symptom of Redis being down while `REDIS_URL` is set. Every cache call times out after 100ms then falls through to Postgres; on auth and OTP paths the request makes two such fall-throughs. - -Confirm in the server logs: - -```text -auth cache: L2 get failed error=redis call timed out -``` - -Resolution: - -1. `redis-cli -h ping` to confirm Redis is reachable on the cluster network. -2. If Redis was briefly down and is now back, **restart the server pods**. The `redis::aio::ConnectionManager` does not reliably re-establish after the underlying connection drops; a pod restart picks up the new connection cleanly. The same applies to the dashboard. -3. If you don't want to run Redis right now, unset `REDIS_URL` in the deployment and restart. Both services run without the cache (correctness is preserved; latency reverts to pre-Redis baseline). - -### Server reports `OTP request rate-limited` in logs but the user says they only tried once - -Check whether Redis was unreachable. The fallback path uses `SELECT COUNT(*) FROM otp_codes WHERE created_at > now() - interval '15 minutes'`, which sees previously-generated OTP rows. If the user has been click-spamming "Resend" for an hour, the 15-minute window may still contain ≥5 codes. Resolve by either waiting for the window to roll over or `DELETE FROM otp_codes WHERE user_id = $1 AND created_at > now() - interval '15 minutes'` (operator console). - -### I changed `ALLOWED_EMAILS` / `SESSION_TTL_SECS` / `OTP_TTL_SECS` and restarted; nothing happened - -These env vars are **first-boot seeds only**. Once the `settings` table has a row for the matching key, that row is the source of truth; the env var is read once at first boot and then ignored on every subsequent restart. - -To change them after first boot, log into the dashboard and edit them under `/settings`. The change applies within seconds across all replicas; no restart required. - -If you need to force a re-seed from env (rare, typically only useful in development), `DELETE FROM settings WHERE key = ''` and restart the server. The bootstrap will pick up the current env-var value on the next boot. Editing via `/settings` is the supported path in production. - ---- - -## Collector Issues - -### Collector starts but events are not appearing in the dashboard - -1. Confirm the collector is running: `systemctl status agenteye-collector` (Linux) or check the process. -2. Confirm `AGENTEYE_URL` points to `http(s)://your-server-host:8080/events` (note: `/events` path). -3. Run a one-shot flush to see immediate output: - ```bash - agenteye-collector flush - ``` -4. Check that the Python SDK is actually writing files: `ls ${AGENTEYE_HOME:-~/.agenteye}/events/` -5. If files exist in `${AGENTEYE_HOME:-~/.agenteye}/failed/`, the uploads are failing. Check the collector logs for the error, likely a 4xx (bad key or URL) or network issue. - -### Files are accumulating in `$AGENTEYE_HOME/events/` and not being uploaded - -- The collector may not be running. Start it: `agenteye-collector start`; it automatically flushes pre-existing events on startup. -- Check collector health: `agenteye-collector health` -- The collector may be running but unable to reach the server. Check firewall rules between collector and server hosts. - -### Files in `$AGENTEYE_HOME/failed/` - -Files move to `failed/` after all retry attempts are exhausted (default: 5 attempts with exponential backoff). This means either: -- The server returned a 4xx error (bad key, wrong URL, or payload issue) -- The server was unreachable for the entire retry window - -Fix the underlying issue, then manually re-queue: - -```bash -mv ${AGENTEYE_HOME:-~/.agenteye}/failed/*.jsonl ${AGENTEYE_HOME:-~/.agenteye}/events/ -agenteye-collector flush -``` - -### Collector reports `network error` on every upload (TLS handshake fails) - -> **Tip:** Quick fix: the durable answer is to serve a publicly-trusted (Let's Encrypt) certificate on the ingest hostname, after which the collector needs no extra CA config. The steps below explain how, plus a short-term unblock if you are still on a self-signed setup. - -If `curl -k` against `AGENTEYE_URL` succeeds but the collector binary fails every upload with `error sending request for url (...)`, the AgentEye server is presenting a TLS certificate that isn't signed by a publicly-trusted CA. - -The **production path** is the ACME ingest hostname configured in `deploy/base/certificates/domain.env` (see the [Kubernetes deployment guide](/agenteye/kubernetes-deployment) Phase 3.1 / 4.2). Once `INGEST_DOMAIN` resolves to the public Traefik load balancer and cert-manager (the Kubernetes controller that obtains and renews TLS certificates) has issued the Let's Encrypt cert, collectors verify the server cert against the system trust store with **no `AGENTEYE_TLS_CA` needed**; clear it from your collector config if it was set against an older self-signed deployment. - -**Symptom: collector worked yesterday, fails today after a ~90-day gap.** This means the deployment is still on the legacy `selfsigned` issuer for `ingest-tls`. The 90-day cert rotated and the pinned CA file is stale. Fix permanently by switching the cluster to the ACME issuer (Phase 3.1 of the deployment guide). Short-term unblock: re-extract the current server cert and update `AGENTEYE_TLS_CA`: - -```bash -kubectl get secret ingest-tls-cert -n agenteye \ - -o jsonpath='{.data.tls\.crt}' | base64 -d > /etc/agenteye/server-ca.crt -``` - -```bash -export AGENTEYE_TLS_CA=/etc/agenteye/server-ca.crt -agenteye-collector flush -``` - -`AGENTEYE_TLS_CA` adds an additional trust anchor; the standard public roots are still trusted. - -### `ingest-tls` Certificate is stuck `Ready: False` after deploy - -```bash -kubectl describe certificate ingest-tls -n agenteye -``` - -Look at the `Events` and the referenced `Order` / `Challenge`. Common causes: - -- **DNS not resolving to the public LB.** The HTTP-01 challenge (where Let's Encrypt proves you control the domain by fetching a token from it over port 80) can't reach `INGEST_DOMAIN`. Verify with `dig +short INGEST_DOMAIN`; it should resolve to the same address as the `traefik-public` LoadBalancer's `EXTERNAL-IP`. cert-manager retries automatically once DNS propagates; no need to delete the Certificate. -- **Port 80 blocked at the load balancer / security group.** HTTP-01 requires port 80 to be reachable from Let's Encrypt's public validators. If you have an upstream WAF or SG restricting `:80`, open it (the Traefik config redirects to HTTPS, but Boulder, Let's Encrypt's validation server, follows the redirect and accepts the response). -- **`dnsNames` not substituted.** If `kubectl get certificate ingest-tls -n agenteye -o jsonpath='{.spec.dnsNames}'` shows `INGEST_DOMAIN_PLACEHOLDER`, you skipped the `domain.env` step; create it from `domain.env.example` and re-apply. -- **Rate limited by Let's Encrypt.** Repeated failed orders for the same hostname trip the duplicate-certificate or failed-validation limits. Wait at least an hour before retrying; check the Order status for the exact rate-limit message. - -### `dashboard-tls` Certificate is stuck `Ready: False` / browser still shows a warning - -Same diagnosis flow as `ingest-tls` above (`kubectl describe certificate dashboard-tls -n agenteye`); the DNS, port-80, placeholder, and rate-limit causes all apply, plus two dashboard-specific ones: - -- **`DASHBOARD_DOMAIN` resolves to the wrong LoadBalancer.** It must point at the *dashboard* Traefik LB, not the public ingest one. `dig +short` the hostname and compare against the dashboard LB address. -- **The dashboard Traefik instance can't serve the challenge.** It must be installed with the bundled dashboard values file, which enables a scoped Ingress provider for cert-manager's HTTP-01 solver. Without it the solver is unroutable and the Order stays `pending` forever. Upgrade the instance with the provided values; the pending challenge then completes on its own. -- **The LoadBalancer was IP-restricted.** Source ranges apply to port 80 too, which blocks Let's Encrypt's validators: both first issuance and every ~75-day renewal. Re-open the LB, or coordinate a DNS-01 solver with support before locking it down. - -While issuance is failing, the dashboard keeps serving its previous certificate (or the ingress default on a fresh install). Access is degraded by a browser warning, never down. - -### CLI still skips TLS verification after the dashboard got a trusted certificate - -`--insecure` is persisted to `cli.json` at login. Once the dashboard serves a publicly-trusted certificate, log in again with `agenteye --base-url https:// --secure login`; verification is saved back on and the startup warning disappears. - ---- - -## Dashboard Issues - -### Can't disable or edit the `ADMIN_EMAIL` user - -By design. The user matching `ADMIN_EMAIL` is marked protected on every server startup: the dashboard hides the Disable button for that row, and the API rejects `DELETE /users/:id` and `PUT /users/:id` against it with `403 Forbidden`. A database trigger also rejects direct `UPDATE` statements that would disable the protected row. - -To rotate the bootstrap admin, change `ADMIN_EMAIL` in your environment and restart the server. The new email is upserted as protected. The previous admin retains the protected flag until cleared in the database (typically fine, since the previous email is still a valid admin until you explicitly remove them). - -### Dashboard shows no events - -1. Confirm the server URL and API key are correct in the dashboard's environment variables (`AGENTEYE_SERVER_URL`, `AGENTEYE_API_KEY`). -2. The dashboard API key needs `events:read` permission. -3. Confirm events have actually been ingested: `curl http://your-server:8080/events -H "Authorization: Bearer $ADMIN_KEY"` - -### `/errors` is empty but `/events` shows red rows - -![The /errors page, which collects every failure the events stream paints red](/agenteye/images/errors.png) - -Newer SDK versions emit failures as `agent_end` / `tool_result` / `hook_completed` events with `outcome: "error"` in the payload, rather than as a dedicated `event_type: "error"` row. The `/errors` page now matches both: any row the `/events` stream paints red (explicit `event_type='error'`, payload `outcome`/`status` in the failure set, `is_error: true`, or a truthy `error` field) appears on `/errors`. If you previously saw "no errors in this window" while red rows were visible on `/events`, upgrade the dashboard + server together (the broadened filter is `errored=true` on `GET /events`) and the two views will agree. - -### `/models`, `/tools`, or `/hooks` is slow or fails to load on wide time ranges - -**Symptom:** on a large events table (millions of rows), opening `/models`, `/tools`, or `/hooks` (or widening the time range to `7d`, `30d`, or `all`), the charts spin and then show a load error. The server logs a ClickHouse `MEMORY_LIMIT_EXCEEDED` (Code 241) or a query timeout for the `latency_aggregate` request. - -**Cause:** older builds computed these pages' latency and distribution rollups with a query that read the full raw event `payload` and paired request/response events with an in-memory sort-and-join. Peak query memory therefore grew with the size of the window, so on a busy tenant a wide range could exceed ClickHouse's per-query memory ceiling. - -**Fix:** upgrade to a build that includes this fix. The rollup now reads only the compact promoted columns and pairs events with a streaming aggregation, so peak memory no longer scales with the raw payload; wide windows stay well within the memory ceiling and return in a fraction of the time. The improvement is entirely query-side: it applies to all existing data on the next page load, with no re-ingest or backfill. - -### Dashboard fails to load / blank page - -Check the dashboard container logs: - -```bash -docker logs agenteye-dashboard -``` - -The most common cause is `AGENTEYE_SERVER_URL` or `AGENTEYE_API_KEY` being missing or pointing to an unreachable server. - -### Dashboard analytics / telemetry - -The dashboard sends anonymous product-usage analytics to PostHog by default, routed through the dashboard's own `/ingest` path (a reverse proxy to `https://us.i.posthog.com`). Sending them first-party means browser ad-blockers don't drop them. This is independent of the dashboard's core functionality: - -- The **dashboard container** (not the browser) is what reaches PostHog. If its outbound access to `https://us.i.posthog.com` is blocked, telemetry silently no-ops; the dashboard works normally and no errors are surfaced to users. -- No agent, session, or event data is ever included, only dashboard UI usage. -- To disable telemetry entirely, set `AE_ANALYTICS_DISABLED=1` on the dashboard container and restart. See [Telemetry & privacy](/agenteye/deployment#telemetry--privacy) in the deployment guide. - -### CLI analytics / telemetry - -The `agenteye` CLI sends anonymous usage analytics to PostHog by default: which commands run, success/exit status, and duration. This is independent of the CLI's functionality: - -- The **machine running the CLI** reaches `https://us.i.posthog.com` directly. If its outbound access is blocked, telemetry silently no-ops (sending is time-bounded, so it never delays a command) and the CLI works normally. -- No agent, session, or event data is ever included: command **arguments and flag values** (dashboard URL, token, email, session ids, query filters) are never sent. -- To disable it, set `AGENTEYE_ANALYTICS_DISABLED=1` (or the cross-tool `DO_NOT_TRACK=1`) in the CLI's environment. See [Telemetry & privacy](/agenteye/cli#telemetry--privacy) in the CLI guide. - ---- - -## AI Assistant Issues - -See the [AI assistant guide](/agenteye/assistant) for full setup. - -### The assistant bubble doesn't appear - -The bubble is hidden unless **all** of these hold: - -- The signed-in user has the `agent:use` permission. -- `AGENTEYE_AGENT_URL` is set on the dashboard and the `agent` service is reachable. -- An LLM endpoint is configured on the `agent` service (`ANTHROPIC_API_KEY`, a gateway via `ANTHROPIC_BASE_URL`, or Bedrock/Vertex). With none set, the agent reports "not configured" and the bubble stays hidden. - -Check the agent's health from the dashboard host: `curl http://agent:9100/health` should return `{"status":"ok","llm_configured":true,...}`. - -### The assistant says it can't read something - -Tools are gated per user. If a user lacks `evaluations:read` (or `events:read`, `dashboards:read`), the matching tools aren't offered and the assistant will say it can't read that data. Grant the relevant read permission. - -### "assistant not configured" (HTTP 503) when sending - -The `agent` container has no LLM endpoint configured, or the dashboard's `AGENTEYE_AGENT_TOKEN` doesn't match the agent's. Set both and restart. - -### The `agent` container restarts / OOMs under load - -Each conversation spawns a short-lived child process. Ensure the container runs with an init process (the image uses `tini`; in Compose set `init: true`) and give it adequate memory limits. Reduce `AGENTEYE_AGENT_MAX_STEPS` if needed. - ---- - -## CLI Issues - -### `agenteye` fails to start with `ModuleNotFoundError: No module named 'click'` - -A fresh install of the `agenteye` CLI at version **0.1.6** can crash on startup with: - -```text -ModuleNotFoundError: No module named 'click' -``` - -0.1.6 relied on `click` being installed indirectly by `typer`; current `typer` releases no longer -pull it in, so a clean environment ends up missing the package. **Upgrade to 0.1.7 or newer**, -which depends on `click` directly: - -```bash -pipx upgrade agenteye # if installed with pipx (or: pipx install --force agenteye) -uv tool upgrade agenteye # if installed with uv -pip install --upgrade agenteye -``` - -See the [CLI guide](/agenteye/cli) for installation guidance. - ---- - -## Python SDK Issues - -### No files appearing in `$AGENTEYE_HOME/events/` - -The SDK buffers events and flushes every 500 ms by default. If your process exits before the flush, events may be lost. Call `agenteye.configure(flush_interval=0.1)` for faster flushing in short-lived scripts, or ensure your process runs long enough for a flush cycle. - -If `AGENTEYE_HOME` is set, verify the SDK is writing to `$AGENTEYE_HOME/events/` and not `~/.agenteye/events/` (requires SDK ≥ 0.0.1b5). - -### `ValueError: Reserved field names cannot be used as custom fields` - -The names `timestamp`, `type`, and `environment` are reserved and cannot be used as custom fields. Passing any of them raises: - -```text -ValueError: Reserved field names cannot be used as custom fields: [...] -``` - -Rename the offending custom field. Note that `session_id` and `agent_id` are explicit parameters of the event call, not custom fields; passing either again as a custom field raises `TypeError`. - ---- - -## Health Monitoring Issues - -### No alerts arriving in Slack (Robusta) - -Robusta (the optional Kubernetes alerting add-on described in the [Health monitoring guide](/agenteye/health-monitoring)) is **opt-in**; it sends nothing until installed and pointed at a Slack channel. Verify the release and its sink: - -```bash -kubectl get pods -n robusta # robusta-runner + robusta-forwarder should be Running -kubectl logs -n robusta -l app=robusta-runner --tail=50 -``` - -Common causes: the Slack `api_key` / `slack_channel` were not set (or the token was revoked); the `api_key` is a Robusta cloud-relay token (`robusta integrations slack`) but the bundled `disableCloudRouting: true` needs a self-hosted Slack **bot token** (`xoxb-…`), or set `disableCloudRouting: false`; the sink `scope` excludes the namespace your pods run in (the bundled values scope to `agenteye`); or no failure has occurred yet. Force a test alert by taking a pod down: - -```bash -kubectl -n agenteye delete pod -l app=clickhouse # it will be recreated -``` - -See the [Health monitoring guide](/agenteye/health-monitoring#2-pod-failure-alerting-with-robusta-opt-in) for install and configuration. - -### Server keeps flapping `NotReady` - -The readiness probe hits `/ready`, which fails when Postgres or ClickHouse is unreachable. If the server cycles in and out of `NotReady`, a dependency is intermittently unavailable; check the ClickHouse and Postgres pods and the server's `CLICKHOUSE_URL` / `DATABASE_URL`. Confirm what `/ready` reports: - -```bash -kubectl -n agenteye exec deploy/server -- sh -c 'curl -s localhost:8080/ready' -``` - -This probe is deliberately tolerant (a generous failure threshold), so sustained flapping indicates a real dependency problem rather than an over-aggressive probe. Liveness stays on `/health`, so flapping readiness will **not** restart the pod. - -## Certificate Monitoring Issues - -### CronJob is not sending Slack notifications - -The `cert-renewal-check` CronJob requires a Slack webhook URL stored in a Secret. Verify it exists: - -```bash -kubectl get secret cert-renewal-notify-config -n agenteye -``` - -If missing, create it: - -```bash -kubectl create secret generic cert-renewal-notify-config \ - --namespace agenteye \ - --from-literal=SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL" -``` - -Without the secret, the CronJob still runs and logs results to stdout. Check logs with: - -```bash -kubectl logs -n agenteye -l job-name --tail=50 -``` - -### Client certificate expired before a notification was received - -The CronJob runs every 12 hours. If it has not been running, check its status: - -```bash -kubectl get cronjob cert-renewal-check -n agenteye -``` - -Trigger a manual check: - -```bash -kubectl create job --from=cronjob/cert-renewal-check manual-cert-check -n agenteye -kubectl logs -n agenteye -l job-name=manual-cert-check -``` - -To re-issue the expired certificate immediately: - -```bash -cd base/certificates/client-certs -./issue-client-cert.sh -``` - -Then apply the regenerated `collector-mtls-secret.yaml` in the cluster(s) running your collectors and restart them: - -```bash -kubectl apply -f collector-mtls-secret.yaml -n -``` - ---- - -## Backup Issues - -### `agenteye-backup` fails with "No space left on device" - -The `agenteye-backup` CronJob dumps Postgres + ClickHouse into a `backup-tmp` `emptyDir` scratch volume (default `30Gi`), then **streams** the `tar` archive straight to S3. The compressed archive is never written back to scratch, so scratch only has to hold the *raw dumps*, not dumps + a second on-disk archive copy. A pod evicted / `No space left on device` therefore means the **raw dumps** exceed the scratch size (the ClickHouse `events` dump dominates and grows over time). Check the failed job's logs: - -```bash -kubectl logs -n agenteye -l job-name= -``` - -Fix: in your overlay, raise the CronJob's `backup-tmp` `emptyDir` `sizeLimit` above your raw dump total, and make sure the node's ephemeral storage can actually hold it (`sizeLimit` is a cap, not a reservation). If the dumps outgrow a single node's disk, replace the `emptyDir` with a PVC (EBS/PD) for `backup-tmp`, or compress the dumps at the source. - -> **Note:** Older releases wrote the `.tar.gz` into the *same* `20Gi` scratch as the dumps, so `dumps + archive` overflowed it and the pod was evicted **before** the upload ran, which looks like an S3 failure but is really disk. Streaming the upload removes that doubling. - -### `agenteye-backup` fails installing `curl` - -The job runs on the `postgres:16` image and installs `curl` at startup for the ClickHouse HTTP dump. On a cluster with no egress to the Debian package mirrors, the `apt-get` step fails. Either allow that egress from the backup pod, or bake `curl` into a mirrored/custom backup image and reference it in your overlay. - -### `agenteye-backup` runs but nothing lands in object storage - -The base ships a real `BACKUP_BUCKET` (`ts-prod-agenteye/backups`) and the `agenteye-backup` ServiceAccount. The job **streams** the archive to S3 (`tar cz … | aws s3 cp - s3://…`). If the backup pod has no write access to the bucket, the upload errors, and because the script runs under `set -euo pipefail`, a failure anywhere in that pipe **fails** the whole job at the `upload` step rather than silently no-op'ing (the pod's EXIT trap logs `backup FAILED during step: upload`). This is also the step you reach *after* fixing a scratch-space eviction, so if backups were previously evicted at the archive step, verify the upload now lands. Grep the failed job's logs for the S3 access error: - -```bash -kubectl logs -n agenteye -l job-name= | grep -iE 's3|upload|denied' -``` - -Fix: in your overlay set `BACKUP_BUCKET` to a bucket you own and annotate the existing `agenteye-backup` ServiceAccount with write access (IRSA / Workload Identity / Pod Identity). See the **Backups** section of the [Kubernetes deployment guide](/agenteye/kubernetes-deployment). - ---- - -## ClickHouse-backed evaluations / sessions / queries - -### The `/queries` page sidebar is empty after upgrading - -![The /queries SchemaBrowser sidebar, which should list the events, evaluations, and agent_sessions tables](/agenteye/images/queries.png) - -Three tables (`events`, `evaluations`, `agent_sessions`) are expected. If the SchemaBrowser sidebar is empty after upgrade, the server failed to apply the ClickHouse DDL at startup. Check the server logs for `failed to apply CH DDL statement`: - -```bash -kubectl logs -n agenteye deploy/server | grep -E 'clickhouse|CH DDL' -``` - -The most common cause is ClickHouse being unreachable while migrations run. The server refuses to start if it can't reach CH, so a stuck pod usually has a `CrashLoopBackOff` rather than a silently broken queries page, but a partial DDL apply (one statement OK, the next 5xx) leaves the schema half-baked. Restart the server pod after CH is verified reachable: - -```bash -kubectl rollout restart deploy/server -n agenteye -``` - -### New evaluations are not appearing in `/sessions` or `/queries` - -After the upgrade, new evaluations are written to ClickHouse, not Postgres, and surface under `/sessions` (gated on `evaluations:read`) and in `/queries`. If they don't appear: - -1. Confirm the evaluator pipeline is enabled (`EVALUATOR_ENDPOINT` set on the server) and producing terminal outcomes; check for `evaluation_finalized` log lines. -2. Confirm CH is reachable from the server: `kubectl exec -n agenteye deploy/server -- curl -fsS http://clickhouse:8123/ping`. -3. Spot-check the CH table: `kubectl exec -n agenteye clickhouse-0 -- clickhouse-client -q 'SELECT count() FROM agenteye.evaluations'`. - -### Queries fail under load with "Memory limit exceeded", or ClickHouse is `OOMKilled` - -> **Tip:** Quick fix: this is almost always too little RAM for the ClickHouse pod. Raise the pod's memory limit in your overlay (the "scale ClickHouse's memory" steps below). Read on first if you want to confirm memory (not CPU or disk) is the real cause before spending on a bigger node. - -**Symptom:** under heavy dashboard/query load, analytical pages (the events stream, `/sessions`, the models/latency view, the SQL editor) start failing or timing out; the server briefly flaps `NotReady`; and the ClickHouse pod shows a rising restart count. This is almost always **memory**, not CPU or disk. - -**Confirm it's memory** (not a throughput problem that replication would fix): - -1. Check the pod for out-of-memory kills: - ```bash - kubectl -n agenteye describe pod clickhouse-0 | grep -iE 'Restart Count|Last State|Reason|OOMKilled' - ``` - `Reason: OOMKilled` / `Exit Code: 137` with a climbing restart count is the tell. - -2. Ask ClickHouse what it's rejecting: - ```bash - kubectl -n agenteye exec clickhouse-0 -- clickhouse-client -q \ - "SELECT name, value FROM system.errors WHERE value>0 ORDER BY value DESC" - ``` - A large `MEMORY_LIMIT_EXCEEDED` count is the signature. The message reads *"maximum: N GiB"*; that **N is `0.9 × the pod's memory limit`** (the `max_server_memory_usage_to_ram_ratio` in `deploy/base/clickhouse/configmap.yaml`). If your heavy reads need more than N, they're rejected. - -3. Rule out the things that *aren't* the problem. If CPU, part count, and disk are all low, adding replicas/sharding would be wasted cost: - ```bash - kubectl -n agenteye top pod clickhouse-0 - kubectl -n agenteye exec clickhouse-0 -- clickhouse-client -q \ - "SELECT table, count() parts FROM system.parts WHERE active AND database='agenteye' GROUP BY table" - ``` - -**Cause:** the ClickHouse pod's memory limit is too small for the analytical working set. The heaviest reads pull the raw JSON `payload` column, run `JSONExtract*` over it, and use `FINAL`. Each can need several GiB. If the configured caches (`mark_cache_size` + `uncompressed_cache_size`) are larger than the pod, they compound it: caches are charged against the same budget and crowd out query memory. - -**Fix (scale ClickHouse's memory):** - -1. Raise the ClickHouse memory limit in your overlay by patching the `clickhouse` StatefulSet's container `resources` (the same overlay mechanism used for the other components' `resources`). The usable server budget is `0.9 × limit`, so a `6Gi` limit gives ~5.4 GiB, `16Gi` gives ~14 GiB. Set `requests.memory` to a real floor too, so the scheduler reserves it. Applying this **recreates the CH pod** (single replica → ~30-60s of analytics downtime); do it in a low-traffic window. -2. Keep the caches in `deploy/base/clickhouse/configmap.yaml` proportionate to the limit. Small caches (a few hundred MiB) are safe on a small pod; only raise them alongside a matching memory-limit increase. Per-query `max_memory_usage` is set explicitly in the `users.xml` profile (see the fixed-node section below) and is kept below the server-level cap (`0.9 × limit`) so no single query is *allowed* more RAM than the container has. -3. If the node itself is the ceiling, check the host memory ClickHouse can see: - ```bash - kubectl -n agenteye exec clickhouse-0 -- clickhouse-client -q \ - "SELECT formatReadableSize(value) FROM system.asynchronous_metrics WHERE metric='OSMemoryTotal'" - ``` - If that's only a little above the pod limit, move ClickHouse to a larger (memory-optimized) node, via a node selector/affinity in your overlay, before raising the limit further. - -**When you can't add memory: run queries in RAM and fail fast; don't spill on a slow disk.** If the node is fixed and the pod can't grow, cap what any single query may use (so one query can't take the whole node) and, on a **slow (non-SSD) data disk**, do **not** let large aggregations/sorts spill to disk. Spilling to a slow disk is slower than the server's client read timeout, so a spilling query returns a dashboard `500` mid-flight while ClickHouse keeps grinding. Keeping queries in RAM and rejecting the rare over-budget one *fast* (`MEMORY_LIMIT_EXCEEDED`, sub-second) is what restores loading. Note a ClickHouse gotcha for applying these: - -- **These are *profile* settings, and ClickHouse reads `` only from `users_config` (`users.xml` / `users.d/*.xml`), never from `config.d`.** A `` block placed in `config.d/agenteye.xml` is **silently ignored** (`max_execution_time`, `max_memory_usage`, etc. simply don't apply). The bundled config therefore ships them as a `users.xml` key on the `clickhouse-config` ConfigMap, mounted at `/etc/clickhouse-server/users.d/agenteye.xml`. -- The shipped defaults: `max_memory_usage` (per-query ceiling: one query can't consume the whole server budget), `max_bytes_before_external_group_by` / `max_bytes_before_external_sort` = **`0` (spill disabled)** so queries stay in RAM instead of crawling on the slow disk, and `max_execution_time` (runaway guard, aligned with the server's client read timeout). -- **Verify they're live** (this is also how you detect the config.d gotcha): - ```bash - kubectl -n agenteye exec clickhouse-0 -- clickhouse-client -q \ - "SELECT name, value FROM system.settings - WHERE name IN ('max_memory_usage','max_bytes_before_external_group_by','max_execution_time')" - ``` - Expect a non-zero `max_memory_usage` and `max_bytes_before_external_group_by = 0`. If `max_memory_usage` reads `0`/default, the profile isn't being applied. Check the settings live in a `users.d` mount, not `config.d`. - -Trade-off: with spill disabled, a query whose working set exceeds `max_memory_usage` is **rejected** (`MEMORY_LIMIT_EXCEEDED`) rather than completing slowly. On a slow disk that fast rejection is preferable, because a spilling query would exceed the client timeout and fail anyway. If your data disk is **fast (SSD)**, you can instead raise the `max_bytes_before_external_*` thresholds to let big queries spill to disk and complete. - ---- - -## Multi-tenancy (organizations) - -### Errors during the upgrade that enables organizations (mixed old/new server pods) - -**Symptom:** during a rolling deploy of the org-enabling release, some requests fail: server logs show `there is no unique or exclusion constraint matching the ON CONFLICT specification` on the `api_keys` path, and/or alert/Slack/webhook channels stop firing while the rollout is in flight. - -**Cause:** the upgrade replaces the old instance-wide unique index on `api_keys(name)` with per-org partial indexes, and moves the alert-channel settings (and `default_user_permissions`) out of the global `settings` table into per-org `org_settings`. An **old** server pod still issues `ON CONFLICT (name)` (now no matching constraint) and still reads channel config from the old `settings` rows (now empty). Old and new pods cannot safely coexist for these two paths. - -**Fix:** do not slow-roll this particular upgrade across mixed versions. Cut over cleanly: scale the old server to zero (or use a brief maintenance window) and bring the new version up together with its migrations, rather than running old and new replicas side by side. Normal traffic and ingest resume immediately after cutover; this only affects the version-transition window. - -### Provisioning an organization fails on `CREATE USER` / `CREATE ROW POLICY`, or one org can read another org's data - -**Symptom:** creating an org returns an error mentioning `CREATE USER`, `CREATE ROW POLICY`, or "access management is disabled"; or, worse, members of one org see another org's events/evaluations in the SQL editor or assistant. - -**Cause:** per-org isolation is enforced by a dedicated ClickHouse user + row policy per org. That requires SQL **access management** to be enabled and `users_without_row_policies_can_read_rows=false` on ClickHouse. With access management off, provisioning can't create the user/policy; with the row-policy default left at its permissive value, a user that has SELECT but no policy reads **all** rows (fail-open). - -**Fix:** use the bundled `deploy/base/clickhouse/` config, which sets both. If you run your own ClickHouse config, enable SQL access management on the server-internal user and set `users_without_row_policies_can_read_rows=false` (see `deploy/base/clickhouse/configmap.yaml`), then restart ClickHouse and re-create the org with the `agenteye-orgctl` CLI (see the [Tenant management guide](/agenteye/tenant-management)). - -### Org users lose ClickHouse access after changing `ORG_CH_SECRET` - -**Symptom:** the SQL editor and AI assistant suddenly return ClickHouse authentication failures for every organization, immediately after `ORG_CH_SECRET` was changed or set inconsistently across replicas. - -**Cause:** each org's ClickHouse password is derived as an HMAC (a keyed hash) of `ORG_CH_SECRET`. Rotating it (or running replicas with different values) invalidates every org's stored ClickHouse credential; the derived password no longer matches the provisioned user. - -**Fix:** set `ORG_CH_SECRET` to a single strong value **before** provisioning a second org and keep it stable and identical on every server replica. The server's boot-time reconcile re-provisions each org's ClickHouse user from the current secret on startup, so a server restart across all replicas (with the secret consistent) heals the orphaned users. Treat the value as a long-lived secret; don't rotate it casually. As a safety net, if `ORG_CH_SECRET` is left at the built-in dev default (i.e. unset), the boot-time reconcile **skips** non-default organizations and logs an error rather than rewriting their ClickHouse credentials to the publicly-known dev value, so a single replica that restarts without the secret can't break the other replicas. Set the secret consistently and restart to provision those orgs. - -### The AI assistant returns 400 / refuses to chat after enabling organizations - -**Symptom:** the assistant dock loads but every message comes back as an error (HTTP `400`), and the agent logs a rejected org-less `/chat` request. - -**Cause:** the agent is org-aware and fails closed; it rejects a `/chat` that carries no organization context. This happens during a transitional rollout where the agent has been upgraded but the dashboard sending the request is not yet org-aware. - -**Fix:** finish the rollout so the dashboard sends org context (the normal end state, no flag needed). To bridge the gap while a not-yet-org-aware dashboard talks to an org-aware agent, set `AGENTEYE_AGENT_ALLOW_NO_ORG=1` on the `agent` service so it falls back to the `default` org instead of refusing, and clear it once the dashboard upgrade lands. See the env reference in the [AI assistant guide](/agenteye/assistant#environment-variable-reference). - ---- - -## Audits - -### An audit never runs (next run keeps slipping, no run history) - -![The Audits page with run history and per-run stats such as policy_hits and improvements](/agenteye/images/audits.png) - -**Symptom:** the audit page shows *last run: never*, or `next run` keeps moving into the future without a row appearing in run history. - -**Cause:** the audit is disabled (disabled audits have no queue entry), or the server's audit workers are failing to claim work. - -**Fix:** confirm the audit is **enabled** (the run-now button requires it). Then check the server logs for `audits pipeline started` at boot and for `audits:` errors. A `claim_due failed` line points at Postgres connectivity. `AUDIT_WORKERS` defaults to `1`; it must be ≥ 1 for any audit to run. - -### Audit runs succeed but find nothing - -**Symptom:** run history shows `succeeded` with `findings: 0` even though `/errors` clearly shows failures. - -**Cause:** the scan window doesn't cover the failures, or the scope filters them out. - -**Fix:** check the run row's window (`window_from → window_to`) against when the failures happened. In `since_last` mode each run only scans since the previous successful run, so older failures are only seen by the *first* run or a `fixed`-window audit. Widen `scope` (environments / agent ids). The run stats show `policy_hits` (how many deterministic policies fired) and `improvements` (how many the AI investigation recorded). If both are 0, the window/scope genuinely saw nothing. - -### The run says `analysis_unavailable` and produces only policy findings - -**Symptom:** run stats include `analysis_unavailable` and the only findings are `kind: policy`; no AI improvements appear. - -**Cause:** the agentic investigation couldn't run: the server can't reach the agent service (`AGENTEYE_AGENT_URL` / `AGENTEYE_AGENT_TOKEN` unset on the **server**; the audit reuses the assistant's connection), the assistant service has no LLM configured, or the call errored/timed out (the `analysis_unavailable` string has the detail). The deterministic policy pass is the floor (it always runs), so the audit still succeeds with its security findings. - -**Fix:** set `AGENTEYE_AGENT_URL` (e.g. `http://agent:9100`) and `AGENTEYE_AGENT_TOKEN` on the **server** (the same values the dashboard assistant already uses; the bundled manifests/compose now wire them), and configure an LLM on the assistant service (see the [AI assistant guide](/agenteye/assistant)), then re-run. A large investigation may need a bigger `AUDIT_LLM_TIMEOUT_MS` (server); keep it above the agent's `AGENTEYE_AUDIT_TIMEOUT_MS`. - -### The audit code sandbox is disabled (`sandbox_available: false`) - -**Symptom:** the agent's `/health` shows `sandbox_available: false`, and audit runs note the sandbox is unavailable; the AI investigates with SQL only. - -**Cause:** the in-pod bubblewrap sandbox (a lightweight Linux sandbox for running the AI's generated code) needs **unprivileged user namespaces**, which the pod's seccomp profile (the kernel's syscall-filtering policy) or the node kernel is blocking. - -**Fix:** set `seccompProfile: Unconfined` (k8s) or `security_opt: [seccomp:unconfined]` (compose) on the agent, and confirm the node kernel allows unprivileged user namespaces (some managed images, e.g. GKE COS, disable them). Where you can't enable it, this is expected and safe: the auditor degrades to SQL-only automatically. See the [Deployment guide](/agenteye/deployment). - -### Audit email report not delivered - -**Symptom:** an audit surfaced new findings but no email arrived. - -**Cause:** the audit has no **email** channel attached, email is disabled org-wide in `alerts.enabled_channels`, there are no recipients, or SMTP is unconfigured. - -**Fix:** attach an email channel to the audit, ensure `email` is in `alerts.enabled_channels`, set recipients (on the channel or via `alerts.email_default_recipients`), and configure SMTP (the same transport the alerts + OTP emails use). Email is sent only when a run produces **at least one new** finding. - -### A muted or dismissed pattern keeps its old finding page but never re-ranks - -**Symptom:** after muting a finding, later runs never surface that pattern again, even though it still occurs. - -**Cause:** that is the designed behaviour: mute/dismiss are durable suppressions keyed on the pattern's fingerprint. - -**Fix:** open the finding and use **reopen** to clear the suppression; the next run will rank the pattern again. Use **resolve** (not mute) for "fixed" patterns you'd want to hear about if they regress. - ---- - -## Getting Help - -Contact `support@exosphere.host` with: -- Your AgentEye version (from the release tag) -- Relevant container logs (`docker logs `) -- A description of the issue and what you have already tried - -## Next steps - -- [Deployment](/agenteye/deployment): configuration options and the full environment-variable reference behind these fixes -- [Kubernetes deployment](/agenteye/kubernetes-deployment): the cluster runbook many sections here point back to -- [Health monitoring](/agenteye/health-monitoring): set up readiness and Slack alerting so you catch these issues early -- [CLI](/agenteye/cli): the `agenteye` commands used throughout this page