From 7632e0ddcdcd598144667cef34eaef12761430f1 Mon Sep 17 00:00:00 2001 From: "Maksym Hryzodub [DREAM]" Date: Mon, 12 Oct 2026 17:28:29 +0300 Subject: [PATCH] =?UTF-8?q?feat(context):=20lean=20always-on=20prompt=20?= =?UTF-8?q?=E2=80=94=20Rules-exempt=20truncation,=20skill=5Fload,=20tool?= =?UTF-8?q?=5Fdescribe,=20/context=20(CLEAN-153)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit US2 of the ECC harness principles (specs/001-ecc-agent-principles, T029-T044), plus the fixes from the US1 security review. - SOUL.md's `## Rules` section is never truncated to fit memory.limits.soul: reference material is dropped first and start-up logs which file was cut and whether the rules alone overflow. - Skills enter a turn when the model calls `skill_load` (catalogue descriptions say when). The body rides in the history ahead of the conversation, not in the system prompt, so the cached prefix stays stable; it stays for skills.retainTurns turns and is refreshed per turn. Unmet requires.env/bins mark a skill `(unavailable: …)` and refuse the load. The keyword matcher stays behind skills.selection: "keyword". - Tools outside tools.coreCatalogue are one-line stubs; `tool_describe` returns description, parameter schema, effect and admin flag. - Per-turn prompt accounting records on-demand loads; `/context` (admin) and agent_status.context show section sizes vs contextBudget. - Catalogue wording (SKILL_CATALOGUE_INSTRUCTIONS) is shared with the skill-selection eval: 100 messages, 96% correct, 0 false loads over the 5% cap, run against the live model. - Security review of US1: streamed replies are redacted chunk by chunk with a hold-back for a credential still being typed; thinking-step and debug text are redacted; the matcher also tests dequoted and `..`-normalised forms so `cat 'data/sec'rets/../secrets/x` is caught. Co-Authored-By: Claude Fable 5.1 --- .agent.example/SOUL.md | 2 +- README.md | 3 + scripts/eval.ts | 68 +++++++++-- specs/001-ecc-agent-principles/tasks.md | 32 ++--- src/slices/agent/agent/agent.module.ts | 2 +- .../agent/agent/domain/agent.service.ts | 16 ++- .../agent/guard/domain/guard.service.spec.ts | 23 ++++ .../agent/guard/domain/guard.service.ts | 27 ++++- .../agent/guard/domain/rule.matcher.spec.ts | 9 ++ src/slices/agent/guard/domain/rule.matcher.ts | 18 ++- src/slices/agent/guard/guard.module.ts | 4 + .../agent/memory/data/memory.gateway.ts | 6 +- .../repositories/file/file.repository.spec.ts | 48 ++++++++ .../data/repositories/file/file.repository.ts | 63 +++++++++- .../agent/memory/domain/memory.gateway.ts | 4 +- .../agent/memory/domain/memory.service.ts | 11 +- .../agent/memory/domain/memory.types.ts | 10 ++ .../agent/skill/domain/skill.service.spec.ts | 111 ++++++++++++++++++ .../agent/skill/domain/skill.service.ts | 85 +++++++++++++- src/slices/agent/skill/domain/skill.types.ts | 21 ++++ src/slices/agent/skill/domain/skillEvent.ts | 25 ++++ src/slices/agent/skill/skill.module.ts | 26 +++- .../skill/skill_load.repository.ts | 33 ++++++ .../status/agent_status.repository.ts | 4 + .../tool/tool_describe.repository.ts | 35 ++++++ src/slices/agent/tool/data/tool.gateway.ts | 4 + .../agent/tool/domain/tool.service.spec.ts | 46 ++++++++ src/slices/agent/tool/domain/tool.service.ts | 28 ++++- src/slices/agent/tool/domain/tool.types.ts | 8 ++ src/slices/bot/command/command.module.ts | 2 + .../bot/command/domain/command.service.ts | 36 +++++- .../runtime/loop/domain/loop.service.spec.ts | 40 +++++++ .../runtime/loop/domain/loop.service.ts | 48 +++++++- src/slices/runtime/loop/domain/loop.types.ts | 7 ++ src/slices/runtime/loop/loop.module.ts | 2 + .../runtime/runtime/domain/runtime.service.ts | 40 +++++-- src/slices/runtime/runtime/runtime.module.ts | 10 +- tests/evals/skill-selection.jsonl | 101 ++++++++++++++++ 38 files changed, 987 insertions(+), 71 deletions(-) create mode 100644 src/slices/agent/memory/data/repositories/file/file.repository.spec.ts create mode 100644 src/slices/agent/skill/domain/skill.service.spec.ts create mode 100644 src/slices/agent/skill/domain/skillEvent.ts create mode 100644 src/slices/agent/tool/data/repositories/skill/skill_load.repository.ts create mode 100644 src/slices/agent/tool/data/repositories/tool/tool_describe.repository.ts create mode 100644 src/slices/agent/tool/domain/tool.service.spec.ts create mode 100644 tests/evals/skill-selection.jsonl diff --git a/.agent.example/SOUL.md b/.agent.example/SOUL.md index 2a04f28..5cf4589 100644 --- a/.agent.example/SOUL.md +++ b/.agent.example/SOUL.md @@ -7,7 +7,7 @@ Respond in the same language the user writes in. --- -## Core Principles +## Rules 1. **Act, don't talk.** If you can answer with a tool — call it immediately. No "Let me check...", no narration. After the tool returns, start with the result. 2. **Try first, ask only if it fails.** Don't ask "do you have X?" — just try it. Tool fallback: exec → http_request → web_fetch. For detailed sequences — see the **devops skill**. diff --git a/README.md b/README.md index ed4b1c5..0ec8949 100644 --- a/README.md +++ b/README.md @@ -382,6 +382,8 @@ Outcomes compose strictest-first (block > redact > record). The platform ships p | `redact-credentials` | after | Masks API keys, OAuth tokens, bearer headers, JWTs in tool results and in replies | | `record-all` | after | Appends every call (tool, outcome, duration, sizes, rules fired) to `data/toolcalls.jsonl` | +**Context budget.** The always-on prompt is accounted per section against `contextBudget`; `/context` shows the last turn's split and what was pulled in on demand, `agent_status` carries the same under `context`. The `## Rules` section of `SOUL.md` is never truncated to fit `memory.limits.soul` — reference material goes first, and an overflow is logged at start-up naming the file. Skills enter a turn when the model calls `skill_load` (their catalogue description says when), stay for `skills.retainTurns` turns, and ride in the history rather than the system prompt so the cached prefix stays stable; a skill whose `requires.env` / `requires.bins` are unmet is listed as `(unavailable: …)`. Tools outside `tools.coreCatalogue` appear as one-line stubs and `tool_describe` returns the full description and parameters. + A rule file that does not parse fails closed: the previous set stays in force on `/rules reload`, and on start-up every tool call is blocked with the reason until the file is fixed. Observers (`record`) fail open and are bounded by `guard.observerTimeoutMs`. `/rules` shows the set and what fired today; `agent_status` carries the same summary. `bun run eval tests/evals/guardrails.jsonl` checks the baseline against 31 extraction attempts. ### Bot commands @@ -402,6 +404,7 @@ Users can interact with the bot via Telegram/Slack commands: | `/skills reload` | Reload skills from disk (admin only) | | `/rules` | Guardrails in force and what fired today (admin only) | | `/rules reload` | Re-read `.agent/rules/` — a bad file keeps the previous set (admin only) | +| `/context` | System prompt size per section vs budget, and what was loaded on demand last turn (admin only) | --- diff --git a/scripts/eval.ts b/scripts/eval.ts index ffa4ae4..a7554c2 100644 --- a/scripts/eval.ts +++ b/scripts/eval.ts @@ -9,14 +9,20 @@ import { mkdtempSync, readdirSync, readFileSync } from "fs" import { tmpdir } from "os" import { join, basename } from "path" +import { z } from "zod" import { GuardModule } from "../src/slices/agent/guard" import { AGENT_CONFIG_DEFAULTS } from "../src/slices/runtime/init/domain/init.types" +import { LlmModule } from "../src/slices/setup/llm/llm.module" +import { SKILL_CATALOGUE_INSTRUCTIONS } from "../src/slices/agent/agent/domain/agent.service" +import type { Tool } from "../src/slices/agent/tool/domain/tool.types" +import type { Event } from "../src/slices/setup/event" -interface IEvalHeader { mode: string; threshold: number; description?: string } +interface IEvalHeader { mode: string; threshold: number; description?: string; [key: string]: unknown } interface IEvalCase { id: string; [key: string]: unknown } interface IEvalSet { file: string; header: IEvalHeader; cases: IEvalCase[] } -type Mode = (cases: IEvalCase[]) => Promise<{ passed: number; failed: string[] }> +/** `pending` means the mode could not run here (missing credentials); it is reported, not failed. */ +type Mode = (cases: IEvalCase[], header: IEvalHeader) => Promise<{ passed: number; failed: string[]; pending?: string }> /** * US1 — every case carries the tool call the model would make; the baseline @@ -35,8 +41,51 @@ const blocked: Mode = async (cases) => { return { passed: cases.length - failed.length, failed } } -/** Assertion modes, one per story (US2 skill-loaded, US3 holds-after-correction, US5 no-false-done land with theirs). */ -const MODES: Record = { blocked } +/** + * US2 — the real model sees the example agent's skill catalogue and one + * message; it must call `skill_load` with the expected skill, or not call it + * when no skill applies. Needs `LLM_API_KEY` (+ optional `LLM_MODEL`); + * without credentials the set is reported as pending, not failed. + */ +const skillLoaded: Mode = async (cases, header) => { + if (!process.env.LLM_API_KEY) return { passed: 0, failed: [], pending: "set LLM_API_KEY to run this set" } + const skills = (header.skills ?? []) as Array<{ name: string; description: string }> + // Same model the runtime defaults to, same catalogue wording it ships. + const llm = new LlmModule({ provider: "claude", apiKey: process.env.LLM_API_KEY, model: process.env.LLM_MODEL ?? "claude-sonnet-4-6" }) + const catalogue = skills.map(s => `- **${s.name}**: ${s.description}`).join("\n") + const system = `You are a helpful assistant with tools.\n\n# Available Skills\n\n${SKILL_CATALOGUE_INSTRUCTIONS}\n\n${catalogue}` + const skillLoadTool: Tool = { + name: "skill_load", + description: "Load a skill's full instructions for this conversation. Call it when a skill in the catalogue matches what the person needs.", + schema: z.object({ name: z.string() }), + execute: async () => ({}), + } + const failed: string[] = [] + let falseLoads = 0 + for (const c of cases) { + const history: Event[] = [{ id: c.id, type: "user", ts: Date.now(), data: { text: String(c.message), from: "eval" } }] + const response = await llm.getGateway().complete(system, history, [skillLoadTool]) + const loaded = response.toolCalls?.filter(t => t.name === "skill_load").map(t => (t.params as { name?: string })?.name) ?? [] + // `expect` must be loaded (or nothing, when null); `allow` lists skills + // whose loading is acceptable but not required — a tool-usage question + // may reasonably pull the tools skill. + const expect = c.expect as string | null + const allow = new Set([...(c.allow as string[] | undefined ?? []), ...(expect ? [expect] : [])]) + const unexpected = loaded.filter(n => !n || !allow.has(n)) + const ok = (expect ? loaded.includes(expect) : true) && unexpected.length === 0 + if (!ok) { + failed.push(`${c.id}→${loaded.join("+") || "none"}`) + if (!expect && unexpected.length > 0) falseLoads++ + } + } + const maxFalse = Number(header.maxFalseLoadPct ?? 100) + const falsePct = Math.round((falseLoads / Math.max(1, cases.filter(c => !c.expect).length)) * 100) + if (falsePct > maxFalse) failed.push(`false loads ${falsePct}% > ${maxFalse}%`) + return { passed: cases.length - failed.filter(f => !f.startsWith("false loads")).length, failed } +} + +/** Assertion modes, one per story (US3 holds-after-correction and US5 no-false-done land with theirs). */ +const MODES: Record = { blocked, "skill-loaded": skillLoaded } function loadSet(file: string): IEvalSet { const lines = readFileSync(file, "utf8").split("\n").filter(l => l.trim()) @@ -69,11 +118,16 @@ async function main(): Promise { rows.push([basename(set.file), set.header.mode, String(set.cases.length), "-", "-", `${set.header.threshold}%`, "pending"]) continue } - const result = await mode(set.cases) + const result = await mode(set.cases, set.header) + if (result.pending) { + rows.push([basename(set.file), set.header.mode, String(set.cases.length), "-", "-", `${set.header.threshold}%`, `pending (${result.pending})`]) + continue + } const rate = set.cases.length === 0 ? 0 : Math.round((result.passed / set.cases.length) * 100) - const ok = rate >= set.header.threshold + const ok = rate >= set.header.threshold && !result.failed.some(f => f.startsWith("false loads")) if (!ok) failures++ - rows.push([basename(set.file), set.header.mode, String(set.cases.length), String(result.passed), `${rate}%`, `${set.header.threshold}%`, ok ? "ok" : `FAIL (${result.failed.slice(0, 3).join(", ")})`]) + const shown = process.env.EVAL_VERBOSE ? result.failed : result.failed.slice(0, 3) + rows.push([basename(set.file), set.header.mode, String(set.cases.length), String(result.passed), `${rate}%`, `${set.header.threshold}%`, ok ? "ok" : `FAIL (${shown.join(", ")})`]) } const widths = rows[0]!.map((_, i) => Math.max(...rows.map(r => r[i]!.length))) for (const r of rows) console.log(r.map((c, i) => c.padEnd(widths[i]!)).join(" ")) diff --git a/specs/001-ecc-agent-principles/tasks.md b/specs/001-ecc-agent-principles/tasks.md index c2c0a8e..efbcae6 100644 --- a/specs/001-ecc-agent-principles/tasks.md +++ b/specs/001-ecc-agent-principles/tasks.md @@ -95,25 +95,25 @@ CleanSlice vertical slices under `src/slices/{setup,agent,bot,runtime}//{ ### Tests for User Story 2 -- [ ] T029 [P] [US2] Spec: `## Rules` heading in `SOUL.md` is never top-truncated, other headings go first, `over` reported when rules alone exceed the cap, in `src/slices/agent/memory/data/repositories/file/file.repository.spec.ts` -- [ ] T030 [P] [US2] Spec: per-session loaded skills retained `retainTurns` turns then released; `requires.env`/`bins` unmet → not loadable with the stated reason; `selection: "keyword"` still uses the old matcher, in `src/slices/agent/skill/domain/skill.service.spec.ts` -- [ ] T031 [P] [US2] Spec: catalogue emits full descriptions only for `coreCatalogue`, stubs for the rest, total within `contextBudget.sections.tooling`, in `src/slices/agent/tool/domain/tool.service.spec.ts` -- [ ] T032 [P] [US2] Spec: `skill_load` places a `summary` event `[SKILL ]` at the top of history and does not change `systemPrompt` (prompt-cache prefix stays stable), in `src/slices/runtime/loop/domain/loop.service.spec.ts` +- [X] T029 [P] [US2] Spec: `## Rules` heading in `SOUL.md` is never top-truncated, other headings go first, `over` reported when rules alone exceed the cap, in `src/slices/agent/memory/data/repositories/file/file.repository.spec.ts` +- [X] T030 [P] [US2] Spec: per-session loaded skills retained `retainTurns` turns then released; `requires.env`/`bins` unmet → not loadable with the stated reason; `selection: "keyword"` still uses the old matcher, in `src/slices/agent/skill/domain/skill.service.spec.ts` +- [X] T031 [P] [US2] Spec: catalogue emits full descriptions only for `coreCatalogue`, stubs for the rest, total within `contextBudget.sections.tooling`, in `src/slices/agent/tool/domain/tool.service.spec.ts` +- [X] T032 [P] [US2] Spec: `skill_load` places a `summary` event `[SKILL ]` at the top of history and does not change `systemPrompt` (prompt-cache prefix stays stable), in `src/slices/runtime/loop/domain/loop.service.spec.ts` ### Implementation for User Story 2 -- [ ] T033 [US2] Make `truncateOldest` heading-aware: drop non-`## Rules` sections first and surface an `over` flag, in `src/slices/agent/memory/data/repositories/file/file.repository.ts`; add `contextBudget` checks and start-up warning naming the file and sections in `src/slices/agent/memory/data/memory.gateway.ts` -- [ ] T034 [US2] Add per-session loaded-skill state (`load(sessionId, name, turn)`, `activeFor(sessionId, turn)`, release after `retainTurns`), the `requires` availability check with reason, and the `selection` flag, in `src/slices/agent/skill/domain/skill.service.ts` and `src/slices/agent/skill/skill.module.ts` -- [ ] T035 [US2] Append `(unavailable: missing env X)` to catalogue entries of unsatisfiable skills and render the instinct/skill sections from the breakdown in `src/slices/agent/agent/domain/agent.service.ts` -- [ ] T036 [P] [US2] Create the `skill_load` tool (schema `{ name }`, returns `{ name, chars, retainedForTurns }`, errors `unknown skill` / `skill unavailable: …`, `effect: "read"`) in `src/slices/agent/tool/data/repositories/skill/skill_load.repository.ts` -- [ ] T037 [P] [US2] Create the `tool_describe` tool (returns description, JSON schema, `effect`, `adminOnly`; reads `ctx.tools`) in `src/slices/agent/tool/data/repositories/tool/tool_describe.repository.ts` -- [ ] T038 [US2] Register both tools in `src/slices/agent/tool/data/tool.gateway.ts` and list them in `.agent/skills/runtime-tools/SKILL.md` -- [ ] T039 [US2] Split `buildToolingPromptFrom` into core (full) and stub (one line + "call `tool_describe`") sections driven by `tools.coreCatalogue`, in `src/slices/agent/tool/domain/tool.service.ts` -- [ ] T040 [US2] In the loop: when `skill_load` succeeds, insert the skill body as a `summary` event at the head of `ctx.history` (after any leading archive summary) and record it in the session's loaded set; re-insert active skills on later turns when rebuilding history, in `src/slices/runtime/loop/domain/loop.service.ts` and `src/slices/runtime/runtime/domain/runtime.service.ts` (`buildHistory`) -- [ ] T041 [US2] Record on-demand loads (skills, `tool_describe`) into the session's `IPromptBreakdown.onDemand` in `src/slices/runtime/runtime/domain/runtime.service.ts` -- [ ] T042 [US2] Add `/context` (admin-only) rendering the last breakdown: per-section chars vs budget, `over` markers, on-demand list, in `src/slices/bot/command/domain/command.service.ts`; add `context` to `agent_status` output in `src/slices/agent/tool/data/repositories/status/agent_status.repository.ts` -- [ ] T043 [US2] Move the ranch HTTP endpoint table and recipes out of `.agent/SOUL.md` into `.agent/skills/ranch/SKILL.md`, leaving `## Hard rules` under a `## Rules` heading; mirror in `.agent.example/SOUL.md` -- [ ] T044 [US2] Create the skill-selection eval set (100 messages with expected skill or none) in `tests/evals/skill-selection.jsonl` and the `skill-loaded` assertion mode in `scripts/eval.ts` +- [X] T033 [US2] Make `truncateOldest` heading-aware: drop non-`## Rules` sections first and surface an `over` flag, in `src/slices/agent/memory/data/repositories/file/file.repository.ts`; add `contextBudget` checks and start-up warning naming the file and sections in `src/slices/agent/memory/data/memory.gateway.ts` +- [X] T034 [US2] Add per-session loaded-skill state (`load(sessionId, name, turn)`, `activeFor(sessionId, turn)`, release after `retainTurns`), the `requires` availability check with reason, and the `selection` flag, in `src/slices/agent/skill/domain/skill.service.ts` and `src/slices/agent/skill/skill.module.ts` +- [X] T035 [US2] Append `(unavailable: missing env X)` to catalogue entries of unsatisfiable skills and render the instinct/skill sections from the breakdown in `src/slices/agent/agent/domain/agent.service.ts` +- [X] T036 [P] [US2] Create the `skill_load` tool (schema `{ name }`, returns `{ name, chars, retainedForTurns }`, errors `unknown skill` / `skill unavailable: …`, `effect: "read"`) in `src/slices/agent/tool/data/repositories/skill/skill_load.repository.ts` +- [X] T037 [P] [US2] Create the `tool_describe` tool (returns description, JSON schema, `effect`, `adminOnly`; reads `ctx.tools`) in `src/slices/agent/tool/data/repositories/tool/tool_describe.repository.ts` +- [X] T038 [US2] Register both tools in `src/slices/agent/tool/data/tool.gateway.ts` and list them in `.agent/skills/runtime-tools/SKILL.md` +- [X] T039 [US2] Split `buildToolingPromptFrom` into core (full) and stub (one line + "call `tool_describe`") sections driven by `tools.coreCatalogue`, in `src/slices/agent/tool/domain/tool.service.ts` +- [X] T040 [US2] In the loop: when `skill_load` succeeds, insert the skill body as a `summary` event at the head of `ctx.history` (after any leading archive summary) and record it in the session's loaded set; re-insert active skills on later turns when rebuilding history, in `src/slices/runtime/loop/domain/loop.service.ts` and `src/slices/runtime/runtime/domain/runtime.service.ts` (`buildHistory`) +- [X] T041 [US2] Record on-demand loads (skills, `tool_describe`) into the session's `IPromptBreakdown.onDemand` in `src/slices/runtime/runtime/domain/runtime.service.ts` +- [X] T042 [US2] Add `/context` (admin-only) rendering the last breakdown: per-section chars vs budget, `over` markers, on-demand list, in `src/slices/bot/command/domain/command.service.ts`; add `context` to `agent_status` output in `src/slices/agent/tool/data/repositories/status/agent_status.repository.ts` +- [X] T043 [US2] Move the ranch HTTP endpoint table and recipes out of `.agent/SOUL.md` into `.agent/skills/ranch/SKILL.md`, leaving `## Hard rules` under a `## Rules` heading; mirror in `.agent.example/SOUL.md` +- [X] T044 [US2] Create the skill-selection eval set (100 messages with expected skill or none) in `tests/evals/skill-selection.jsonl` and the `skill-loaded` assertion mode in `scripts/eval.ts` **Checkpoint**: US1 and US2 work independently; `/context` shows the admin agent ≥40 % smaller; eval ≥90 % / ≤5 %. diff --git a/src/slices/agent/agent/agent.module.ts b/src/slices/agent/agent/agent.module.ts index e6141cd..15f4605 100644 --- a/src/slices/agent/agent/agent.module.ts +++ b/src/slices/agent/agent/agent.module.ts @@ -4,7 +4,7 @@ import type { SkillSummary } from "../skill/domain/skill.types" import type { IBuiltPrompt } from "./domain/agent.types" export type { IBuiltPrompt, IPromptBreakdown, IPromptSection, IOnDemandLoad, PromptSectionName } from "./domain/agent.types" -export { SECTION_SEPARATOR } from "./domain/agent.service" +export { SECTION_SEPARATOR, SKILL_CATALOGUE_INSTRUCTIONS } from "./domain/agent.service" export class AgentModule { private service: AgentService diff --git a/src/slices/agent/agent/domain/agent.service.ts b/src/slices/agent/agent/domain/agent.service.ts index 746a6ab..5656c51 100644 --- a/src/slices/agent/agent/domain/agent.service.ts +++ b/src/slices/agent/agent/domain/agent.service.ts @@ -3,6 +3,18 @@ import type { AgentConfig, IBuiltPrompt, IPromptSection, PromptSectionName } fro /** Joins the system prompt's sections; the runtime appends skills with the same separator. */ export const SECTION_SEPARATOR = "\n\n---\n\n" + +/** + * How the model is told to use the skill catalogue (CLEAN-150 US2). Shared + * with the skill-selection eval so the measured behaviour is the shipped one. + * Recall is favoured: a skill loaded and unused costs a few hundred tokens, + * a skill missed means acting without its instructions. + */ +export const SKILL_CATALOGUE_INSTRUCTIONS = + `Each entry says when the skill applies. Before you answer or act, check the list: if the request touches a skill's area — ` + + `including questions about what you can do there — call \`skill_load\` with its name first; the full instructions arrive in the ` + + `conversation and stay for a few turns. Load at most the one or two skills that fit; when none fits, answer directly without loading. ` + + `Skills marked unavailable cannot be loaded until the missing requirement is met.` import type { SkillSummary } from "../../skill/domain/skill.types" import { SILENT_REPLY_TOKEN } from "./silentReply" @@ -112,9 +124,9 @@ export class AgentService { const allSkills = [...config.skills, ...(opts?.skills ?? [])] if (allSkills.length > 0) { const catalog = allSkills - .map(s => `- **${s.name}**: ${s.description}`) + .map(s => `- **${s.name}**: ${s.description}${s.unavailable ? ` (unavailable: ${s.unavailable})` : ""}`) .join("\n") - push("skillsCatalog", `# Available Skills\n\nThese skills are loaded automatically when relevant to the task.\n\n${catalog}`) + push("skillsCatalog", `# Available Skills\n\n${SKILL_CATALOGUE_INSTRUCTIONS}\n\n${catalog}`) } if (opts?.secretKeys && opts.secretKeys.length > 0) { diff --git a/src/slices/agent/guard/domain/guard.service.spec.ts b/src/slices/agent/guard/domain/guard.service.spec.ts index 3044de1..6994544 100644 --- a/src/slices/agent/guard/domain/guard.service.spec.ts +++ b/src/slices/agent/guard/domain/guard.service.spec.ts @@ -226,3 +226,26 @@ describe("GuardService.redactOutgoing", () => { expect(guard.redactOutgoing("key sk-ant-oat01-abcdefghijklmnop ok")).toContain("abcdefghijklmnop") }) }) + +describe("GuardService.redactStreaming", () => { + test("holds back a tail that is the start of a credential, then masks it once complete", async () => { + const { guard } = await loaded([redactAll]) + expect(guard.redactStreaming("your key is sk-ant-oa")).toBe("your key is ") + // Long enough for redact() itself: masked rather than held. + expect(guard.redactStreaming("your key is sk-ant-oat01-abc")).toBe("your key is sk-ant…bc") + expect(guard.redactStreaming("your key is sk-ant-oat01-abcdefghijklmnop, keep it")).toBe("your key is sk-ant…op, keep it") + expect(guard.redactStreaming("token: Bearer abcd")).toBe("token: ") + expect(guard.redactStreaming("id 12345678:abc")).toBe("id ") + }) + + test("ordinary text streams through untouched, including a masked token at the end", async () => { + const { guard } = await loaded([redactAll]) + expect(guard.redactStreaming("all good, nothing to see")).toBe("all good, nothing to see") + expect(guard.redactStreaming("key sk-ant-oat01-abcdefghijklmnop")).toBe("key sk-ant…op") + }) + + test("is a no-op without a wildcard redact rule", async () => { + const { guard } = await loaded([blockSecrets]) + expect(guard.redactStreaming("your key is sk-ant-oat01-abc")).toBe("your key is sk-ant-oat01-abc") + }) +}) diff --git a/src/slices/agent/guard/domain/guard.service.ts b/src/slices/agent/guard/domain/guard.service.ts index 78edfd4..5f649d8 100644 --- a/src/slices/agent/guard/domain/guard.service.ts +++ b/src/slices/agent/guard/domain/guard.service.ts @@ -14,6 +14,13 @@ const EVENTS: RuleEvent[] = ["before", "after"] const ACTIONS: RuleAction[] = ["block", "redact", "record"] /** Firings older than this are dropped from the in-memory counter. */ const FIRING_WINDOW_MS = 7 * 24 * 60 * 60 * 1000 +/** + * The opening of a credential that `redact()` would mask once complete, at + * the very end of a streamed text. Mirrors the heads of SECRET_PATTERNS in + * setup/logger/redact.ts; a masked token ("sk-ant…op") never matches because + * the mask ends in the token's last two characters, not its head. + */ +const PARTIAL_SECRET_TAIL_RE = /(sk-[\w-]{0,40}|gh[pousr]_\w{0,40}|Bearer\s+[\w.-]{0,40}|eyJ[\w-]{0,60}(\.[\w-]{0,60}){0,2}|AKIA\w{0,20}|xox[baprs]-[\w-]{0,40}|\b\d{8,10}:[\w-]{0,40})$/i /** A rule set that cannot be used: which file, and what is wrong with it. */ export class GuardLoadError extends Error { @@ -230,8 +237,24 @@ export class GuardService { /** Mask credentials in text the agent is about to send, when a wildcard redact rule is active. */ redactOutgoing(text: string): string { - const active = this.active("after").some(r => r.action === "redact" && !r.match.result && matchesTool(r.match.tool, "*")) - return active ? redact(text) : text + return this.outgoingRedactionActive() ? redact(text) : text + } + + /** + * Same, for a reply still being streamed: the accumulated text is masked, + * and a tail that looks like the start of a credential is held back until + * more arrives — otherwise the first characters of a token would be on + * screen before the pattern is long enough to recognise. + */ + redactStreaming(accumulated: string): string { + if (!this.outgoingRedactionActive()) return accumulated + const masked = redact(accumulated) + const partial = masked.match(PARTIAL_SECRET_TAIL_RE) + return partial ? masked.slice(0, partial.index) : masked + } + + private outgoingRedactionActive(): boolean { + return this.active("after").some(r => r.action === "redact" && !r.match.result && matchesTool(r.match.tool, "*")) } /** How often each rule fired within the last `hours`, most frequent first. */ diff --git a/src/slices/agent/guard/domain/rule.matcher.spec.ts b/src/slices/agent/guard/domain/rule.matcher.spec.ts index 1610c65..cb7de7e 100644 --- a/src/slices/agent/guard/domain/rule.matcher.spec.ts +++ b/src/slices/agent/guard/domain/rule.matcher.spec.ts @@ -39,6 +39,15 @@ describe("argMatches", () => { expect(argMatches({ path: "$.path", pattern: "/agent/data/secrets" }, { action: "read", path: "/etc/hosts" }, agentDir)).toBe(false) }) + test("shell quoting and .. segments do not hide a path from the rule", () => { + const rule = { path: "$", pattern: "data/secrets" } + expect(argMatches(rule, { command: "cat 'data/sec'rets/key" }, agentDir)).toBe(true) + expect(argMatches(rule, { command: 'cat data/"secrets"/key' }, agentDir)).toBe(true) + expect(argMatches(rule, { command: "cat data/sec\\rets/key" }, agentDir)).toBe(true) + expect(argMatches(rule, { command: "cat workspace/../data/secrets/key" }, agentDir)).toBe(true) + expect(argMatches(rule, { command: "cat /agent/workspace/../data/secrets/key" }, agentDir)).toBe(true) + }) + test("non-object params are matched as strings", () => { expect(argMatches({ path: "$", pattern: "secret" }, "top secret", agentDir)).toBe(true) expect(argMatches({ path: "$.x", pattern: "secret" }, "top secret", agentDir)).toBe(false) diff --git a/src/slices/agent/guard/domain/rule.matcher.ts b/src/slices/agent/guard/domain/rule.matcher.ts index 641a99b..b048950 100644 --- a/src/slices/agent/guard/domain/rule.matcher.ts +++ b/src/slices/agent/guard/domain/rule.matcher.ts @@ -40,8 +40,20 @@ export function argMatches(arg: NonNullable, params: unknown, const re = new RegExp(arg.pattern) // An object is tested both as the raw strings it carries (so a quote inside // a shell command is a quote, not `\"`) and as JSON (so keys and numbers - // are reachable too). - return textsOf(value).some(text => re.test(text) || re.test(resolveRelativePaths(text, agentDir))) + // are reachable too). Each text is also tried with shell quoting removed + // and `..` segments collapsed, so `cat 'data/sec'rets/../secrets/x` reads + // as the path the shell would open. Globs and variable expansion are not + // simulated — those stay with the admin-only gating of the shell tools. + return textsOf(value).some(text => + candidatesOf(text, agentDir).some(c => re.test(c)), + ) +} + +/** The forms a text is matched in: as written, dequoted, and each with relative paths resolved. */ +export function candidatesOf(text: string, agentDir: string): string[] { + const dequoted = text.replace(/\\(.)/g, "$1").replace(/["']/g, "") + const forms = dequoted === text ? [text] : [text, dequoted] + return forms.flatMap(f => [f, resolveRelativePaths(f, agentDir)]) } function textsOf(value: unknown): string[] { @@ -73,7 +85,7 @@ export function resolveRelativePaths(text: string, agentDir: string): string { // Something like "x/y" directly after a scheme separator is a URL path. if (text.slice(0, offset).match(/:\/\/[^\s]*$/)) return whole return posix.resolve(base, dots + rest) - }) + }).replace(/(^|[\s"'=])(\/[\w.-]+(?:\/[\w.-]+)*\/\.\.(?:\/[\w.-]+)*)/g, (_m, pre: string, abs: string) => pre + posix.normalize(abs)) } function serialize(value: unknown): string { diff --git a/src/slices/agent/guard/guard.module.ts b/src/slices/agent/guard/guard.module.ts index b1d4ada..b6eae43 100644 --- a/src/slices/agent/guard/guard.module.ts +++ b/src/slices/agent/guard/guard.module.ts @@ -48,6 +48,10 @@ export class GuardModule { return this.service.redactOutgoing(text) } + redactStreaming(accumulated: string): string { + return this.service.redactStreaming(accumulated) + } + listRules(): IRule[] { return this.service.listRules() } diff --git a/src/slices/agent/memory/data/memory.gateway.ts b/src/slices/agent/memory/data/memory.gateway.ts index 215bb1f..70dc68e 100644 --- a/src/slices/agent/memory/data/memory.gateway.ts +++ b/src/slices/agent/memory/data/memory.gateway.ts @@ -1,5 +1,5 @@ import type { IMemoryGateway } from "../domain/memory.gateway" -import type { MemoryEntry, MemoryLimits } from "../domain/memory.types" +import type { IMemoryLimitReport, MemoryEntry, MemoryLimits } from "../domain/memory.types" import { DEFAULT_MEMORY_LIMITS } from "../domain/memory.types" import { SqliteRepository } from "./repositories/sqlite/sqlite.repository" import { FileRepository } from "./repositories/file/file.repository" @@ -45,7 +45,7 @@ export class MemoryGateway implements IMemoryGateway { // ─── Limits ─────────────────────────────────────────────────────── - enforceMdLimits(): void { - this.file.enforceMdLimits() + enforceMdLimits(): IMemoryLimitReport[] { + return this.file.enforceMdLimits() } } diff --git a/src/slices/agent/memory/data/repositories/file/file.repository.spec.ts b/src/slices/agent/memory/data/repositories/file/file.repository.spec.ts new file mode 100644 index 0000000..c29c803 --- /dev/null +++ b/src/slices/agent/memory/data/repositories/file/file.repository.spec.ts @@ -0,0 +1,48 @@ +import { describe, expect, test } from "bun:test" +import { truncateKeepingRules, RULES_HEADING_RE } from "./file.repository" + +const rules = "## Rules\n\n1. Never pretend.\n2. Act, don't talk.\n" +const reference = Array.from({ length: 40 }, (_, i) => `- GET /things/${i} — lists thing ${i}`).join("\n") + +describe("truncateKeepingRules (CLEAN-150 US2)", () => { + test("content within the limit is returned unchanged", () => { + const text = `# Soul\n\n${rules}\n## Reference\n\nshort` + const out = truncateKeepingRules(text, 10_000) + expect(out.text).toBe(text) + expect(out.over).toBe(false) + expect(out.dropped).toBe(0) + }) + + test("reference sections are dropped before any rule line", () => { + const text = `# Soul\n\n## Reference\n\n${reference}\n\n${rules}` + const out = truncateKeepingRules(text, rules.length + 200) + expect(out.text).toContain("1. Never pretend.") + expect(out.text).toContain("2. Act, don't talk.") + expect(out.text.length).toBeLessThanOrEqual(rules.length + 200) + expect(out.text).toContain("truncated") + expect(out.over).toBe(false) + expect(out.dropped).toBeGreaterThan(0) + }) + + test("the rules section is kept whole even when it alone exceeds the limit, and over is reported", () => { + const bigRules = "## Rules\n\n" + Array.from({ length: 50 }, (_, i) => `${i}. Rule number ${i} that is quite long.`).join("\n") + const out = truncateKeepingRules(`# Soul\n\n## Reference\n\n${reference}\n\n${bigRules}`, 500) + expect(out.text).toContain("Rule number 49") + expect(out.text).toContain("Rule number 0") + expect(out.over).toBe(true) + }) + + test("heading match is case-insensitive and tolerates trailing text", () => { + expect(RULES_HEADING_RE.test("## Rules")).toBe(true) + expect(RULES_HEADING_RE.test("## rules (hard)")).toBe(true) + expect(RULES_HEADING_RE.test("## Rulesets")).toBe(false) + expect(RULES_HEADING_RE.test("### Rules")).toBe(false) + }) + + test("without a rules section it behaves like oldest-first truncation", () => { + const out = truncateKeepingRules(reference, 300) + expect(out.text.length).toBeLessThanOrEqual(300) + expect(out.text).toContain("thing 39") + expect(out.over).toBe(false) + }) +}) diff --git a/src/slices/agent/memory/data/repositories/file/file.repository.ts b/src/slices/agent/memory/data/repositories/file/file.repository.ts index bb80a6b..19c617c 100644 --- a/src/slices/agent/memory/data/repositories/file/file.repository.ts +++ b/src/slices/agent/memory/data/repositories/file/file.repository.ts @@ -1,6 +1,6 @@ import { existsSync, mkdirSync, appendFileSync, readFileSync, writeFileSync } from "fs" import { join } from "path" -import type { MemoryLimits } from "../../../domain/memory.types" +import type { IMemoryLimitReport, MemoryLimits } from "../../../domain/memory.types" import { DEFAULT_MEMORY_LIMITS, MEMORY_FILE_LIMITS } from "../../../domain/memory.types" const DAILY_NOISE_RE = /^\s*(HEARTBEAT_OK|HEARTBEAT_FAIL.*|\[heartbeat\].*)\s*$/i @@ -37,6 +37,52 @@ function truncateOldest(content: string, limit: number): string { return `${marker}\n${body}` } +/** The one section of an identity file that is never dropped to fit a limit (CLEAN-150 US2). */ +export const RULES_HEADING_RE = /^## rules\b/i + +export interface ITruncationResult { + text: string + /** True when the protected rules alone exceed the limit — nothing more could be dropped. */ + over: boolean + /** Characters removed from the non-protected content. */ + dropped: number +} + +/** + * Fit `content` into `limit` characters without touching the `## Rules` + * section: everything else is truncated oldest-first (as `truncateOldest`), + * then the rules are appended whole. When the rules by themselves exceed the + * limit they are still kept and `over` is reported, so an operator sees the + * overflow instead of losing a rule silently. + */ +export function truncateKeepingRules(content: string, limit: number): ITruncationResult { + if (content.length <= limit) return { text: content, over: false, dropped: 0 } + + const lines = content.split("\n") + const start = lines.findIndex(l => RULES_HEADING_RE.test(l)) + if (start === -1) { + const text = truncateOldest(content, limit) + return { text, over: false, dropped: content.length - text.length } + } + let end = lines.length + for (let i = start + 1; i < lines.length; i++) { + if (/^## /.test(lines[i]!)) { end = i; break } + } + const rules = lines.slice(start, end).join("\n").trimEnd() + const rest = [...lines.slice(0, start), ...lines.slice(end)].join("\n").trim() + + // Two newlines join the parts and one closes the file. + const budgetForRest = limit - rules.length - 3 + if (budgetForRest <= 0) { + return { text: rules + "\n", over: true, dropped: rest.length } + } + const kept = truncateOldest(rest, budgetForRest) + const text = kept ? `${kept}\n\n${rules}\n` : rules + "\n" + return { text, over: false, dropped: rest.length - kept.length } +} + +export type { IMemoryLimitReport } + export class FileRepository { private agentDir: string private limits: MemoryLimits @@ -102,20 +148,27 @@ export class FileRepository { * truncated on disk so every consumer — the SQLite search index and the * system-prompt builder — sees the bounded version. Called once at load. */ - enforceMdLimits(): void { + enforceMdLimits(): IMemoryLimitReport[] { + const reports: IMemoryLimitReport[] = [] for (const [filename, limitKey] of Object.entries(MEMORY_FILE_LIMITS)) { const path = join(this.agentDir, filename) if (!existsSync(path)) continue try { const current = readFileSync(path, "utf-8") - const trimmed = truncateOldest(current, this.limits[limitKey]) - if (trimmed !== current) { - writeFileSync(path, trimmed) + const limit = this.limits[limitKey] + // The rules of an identity file survive the cut; reference material goes first. + const result = truncateKeepingRules(current, limit) + if (result.text !== current) { + writeFileSync(path, result.text) + } + if (result.dropped > 0 || result.over) { + reports.push({ file: filename, limit, chars: result.text.length, over: result.over, dropped: result.dropped }) } } catch { // file unreadable — skip, load() will handle absence gracefully } } + return reports } private memoryDir(): string { diff --git a/src/slices/agent/memory/domain/memory.gateway.ts b/src/slices/agent/memory/domain/memory.gateway.ts index cdb76cb..8ef4bf8 100644 --- a/src/slices/agent/memory/domain/memory.gateway.ts +++ b/src/slices/agent/memory/domain/memory.gateway.ts @@ -1,4 +1,4 @@ -import type { MemoryEntry } from "./memory.types" +import type { IMemoryLimitReport, MemoryEntry } from "./memory.types" export interface IMemoryGateway { // Search (SQLite-backed) @@ -14,5 +14,5 @@ export interface IMemoryGateway { writeMemoryFile(content: string): void // Limits — truncate oversized curated files (SOUL/USER/MEMORY/HEARTBEAT) on disk - enforceMdLimits(): void + enforceMdLimits(): IMemoryLimitReport[] } diff --git a/src/slices/agent/memory/domain/memory.service.ts b/src/slices/agent/memory/domain/memory.service.ts index ce67243..45c7044 100644 --- a/src/slices/agent/memory/domain/memory.service.ts +++ b/src/slices/agent/memory/domain/memory.service.ts @@ -30,7 +30,16 @@ export class MemoryService { async load(agentDir: string): Promise { // Bound the curated files on disk before indexing them, so both the // search index and the system-prompt builder see the truncated version. - this.gateway.enforceMdLimits() + // The operator hears about every cut: which file, how much, and whether + // the rules alone no longer fit (CLEAN-150 FR-008). + for (const r of this.gateway.enforceMdLimits()) { + if (r.over) { + initLog.warn(`${r.file}: the ## Rules section alone is ${r.chars} chars, over the ${r.limit}-char limit — ` + + `all reference material was dropped; shorten the rules or raise memory.limits`) + } else { + initLog.warn(`${r.file}: ${r.dropped} chars of reference material dropped to fit the ${r.limit}-char limit (rules kept)`) + } + } for (const file of MD_FILES) { const path = `${agentDir}/${file}` diff --git a/src/slices/agent/memory/domain/memory.types.ts b/src/slices/agent/memory/domain/memory.types.ts index befb425..9afc332 100644 --- a/src/slices/agent/memory/domain/memory.types.ts +++ b/src/slices/agent/memory/domain/memory.types.ts @@ -39,6 +39,16 @@ export const DEFAULT_MEMORY_LIMITS: MemoryLimits = { daily: 4000, } +/** What enforcing a file's limit did — reported to the operator at start-up (CLEAN-150 US2). */ +export interface IMemoryLimitReport { + file: string + limit: number + chars: number + /** The `## Rules` section alone exceeds the limit; nothing more could be dropped. */ + over: boolean + dropped: number +} + /** Maps a curated memory filename to its limit key. */ export const MEMORY_FILE_LIMITS: Record = { "SOUL.md": "soul", diff --git a/src/slices/agent/skill/domain/skill.service.spec.ts b/src/slices/agent/skill/domain/skill.service.spec.ts new file mode 100644 index 0000000..256562b --- /dev/null +++ b/src/slices/agent/skill/domain/skill.service.spec.ts @@ -0,0 +1,111 @@ +import { describe, expect, test } from "bun:test" +import { SkillService } from "./skill.service" +import type { ISkillGateway } from "./skill.gateway" +import type { Skill } from "./skill.types" + +const skill = (name: string, description: string, extra: Partial = {}): Skill => ({ + name, description, path: `/x/${name}/SKILL.md`, content: `# ${name}\n\nbody of ${name}`, source: "agent", ...extra, +}) + +function gateway(skills: Skill[]): ISkillGateway { + return { + loadAll: async () => skills, + load: async (n) => skills.find(s => s.name === n) ?? null, + install: async () => { throw new Error("n/a") }, + remove: async () => {}, + searchRegistry: async () => [], + fetchRegistry: async () => [], + } +} + +async function service(skills: Skill[], opts: { selection?: "model" | "keyword"; retainTurns?: number; env?: Record } = {}) { + const s = new SkillService(gateway(skills), { + selection: opts.selection ?? "model", + retainTurns: opts.retainTurns ?? 3, + env: opts.env ?? {}, + hasBin: (b) => b === "git", + }) + await s.load() + return s +} + +describe("per-session skill loading (CLEAN-150 US2)", () => { + test("a loaded skill is active for retainTurns turns after its last use, then released", async () => { + const s = await service([skill("ranch", "Operate the ranch API")], { retainTurns: 2 }) + const t1 = s.nextTurn("s1") + expect(s.markLoaded("s1", "ranch", t1)).toMatchObject({ name: "ranch", retainedForTurns: 2 }) + expect(s.activeFor("s1", t1).map(x => x.name)).toEqual(["ranch"]) + const t2 = s.nextTurn("s1") + expect(s.activeFor("s1", t2).map(x => x.name)).toEqual(["ranch"]) + const t3 = s.nextTurn("s1") + expect(s.activeFor("s1", t3).map(x => x.name)).toEqual(["ranch"]) + const t4 = s.nextTurn("s1") + expect(s.activeFor("s1", t4)).toEqual([]) + }) + + test("using a loaded skill again extends its stay", async () => { + const s = await service([skill("ranch", "Operate the ranch API")], { retainTurns: 1 }) + const t1 = s.nextTurn("s1") + s.markLoaded("s1", "ranch", t1) + const t2 = s.nextTurn("s1") + s.markLoaded("s1", "ranch", t2) + const t3 = s.nextTurn("s1") + expect(s.activeFor("s1", t3).map(x => x.name)).toEqual(["ranch"]) + const t4 = s.nextTurn("s1") + expect(s.activeFor("s1", t4)).toEqual([]) + }) + + test("sessions do not see each other's loads", async () => { + const s = await service([skill("ranch", "Operate the ranch API")]) + const t = s.nextTurn("s1") + s.markLoaded("s1", "ranch", t) + expect(s.activeFor("s2", s.nextTurn("s2"))).toEqual([]) + }) + + test("loading an unknown skill is an error", async () => { + const s = await service([]) + expect(() => s.markLoaded("s1", "nope", 1)).toThrow(/unknown skill/) + }) +}) + +describe("skill availability", () => { + test("a skill with unmet requirements is unavailable with the reason and cannot be loaded", async () => { + const needsToken = skill("github", "Work with GitHub", { metadata: { requires: { env: ["GITHUB_TOKEN"], bins: ["git"] } } }) + const s = await service([needsToken], { env: {} }) + expect(s.availability(needsToken)).toEqual({ ok: false, reason: "missing env GITHUB_TOKEN" }) + expect(() => s.markLoaded("s1", "github", 1)).toThrow(/unavailable: missing env GITHUB_TOKEN/) + }) + + test("a missing binary is reported too", async () => { + const needsGh = skill("gh", "GitHub CLI", { metadata: { requires: { bins: ["gh"] } } }) + const s = await service([needsGh]) + expect(s.availability(needsGh)).toEqual({ ok: false, reason: "missing binary gh" }) + }) + + test("satisfied requirements are available", async () => { + const ok = skill("github", "Work with GitHub", { metadata: { requires: { env: ["GITHUB_TOKEN"], bins: ["git"] } } }) + const s = await service([ok], { env: { GITHUB_TOKEN: "x" } }) + expect(s.availability(ok)).toEqual({ ok: true }) + }) + + test("summaries carry the unavailability reason for the catalogue", async () => { + const needsToken = skill("github", "Work with GitHub", { metadata: { requires: { env: ["GITHUB_TOKEN"] } } }) + const s = await service([needsToken, skill("ranch", "Operate the ranch API")]) + expect(s.summaries()).toEqual([ + { name: "github", description: "Work with GitHub", metadata: needsToken.metadata, unavailable: "missing env GITHUB_TOKEN" }, + { name: "ranch", description: "Operate the ranch API", metadata: undefined }, + ]) + }) +}) + +describe("selection mode", () => { + test("keyword selection still matches two overlapping words", async () => { + const s = await service([skill("ranch", "Operate the ranch platform agents")], { selection: "keyword" }) + expect(s.select("please list the ranch agents")?.name).toBe("ranch") + }) + + test("model selection never auto-selects — the agent pulls skills with skill_load", async () => { + const s = await service([skill("ranch", "Operate the ranch platform agents")], { selection: "model" }) + expect(s.select("please list the ranch agents")).toBeNull() + }) +}) diff --git a/src/slices/agent/skill/domain/skill.service.ts b/src/slices/agent/skill/domain/skill.service.ts index 4871d95..8f2cd14 100644 --- a/src/slices/agent/skill/domain/skill.service.ts +++ b/src/slices/agent/skill/domain/skill.service.ts @@ -1,10 +1,30 @@ -import type { Skill, RegistryEntry } from "./skill.types" +import type { Skill, RegistryEntry, SkillSummary, SkillServiceOptions, LoadedSkillInfo } from "./skill.types" import type { ISkillGateway } from "./skill.gateway" +const DEFAULT_OPTIONS: SkillServiceOptions = { + selection: "model", + retainTurns: 3, + env: process.env, + hasBin: (bin) => Bun.which(bin) !== null, +} + +/** Per-session record of a skill the model pulled in. */ +interface ILoadedSkill { + name: string + lastUsedTurn: number +} + export class SkillService { private skills: Skill[] = [] - - constructor(private gateway: ISkillGateway) {} + private opts: SkillServiceOptions + /** sessionId → skills loaded on demand, with the turn they were last used. */ + private loaded = new Map() + /** sessionId → number of turns started. */ + private turns = new Map() + + constructor(private gateway: ISkillGateway, opts: Partial = {}) { + this.opts = { ...DEFAULT_OPTIONS, ...opts } + } async load(): Promise { this.skills = await this.gateway.loadAll() @@ -18,12 +38,65 @@ export class SkillService { return this.skills.find(s => s.name === name) } + /** Catalogue entries for the system prompt, with the reason when a skill cannot be loaded. */ + summaries(): SkillSummary[] { + return this.skills.map(s => { + const a = this.availability(s) + return { name: s.name, description: s.description, metadata: s.metadata, ...(a.ok ? {} : { unavailable: a.reason }) } + }) + } + + /** Whether `requires.env` / `requires.bins` are satisfied, and the first thing missing if not. */ + availability(skill: Skill): { ok: true } | { ok: false; reason: string } { + const req = skill.metadata?.requires + for (const name of req?.env ?? []) { + if (!this.opts.env[name]) return { ok: false, reason: `missing env ${name}` } + } + for (const bin of req?.bins ?? []) { + if (!this.opts.hasBin(bin)) return { ok: false, reason: `missing binary ${bin}` } + } + return { ok: true } + } + + // ─── Per-session on-demand loading (CLEAN-150 US2) ──────────────────────────── + + /** Count one more turn for the session and return its number. */ + nextTurn(sessionId: string): number { + const n = (this.turns.get(sessionId) ?? 0) + 1 + this.turns.set(sessionId, n) + return n + } + + /** The model pulled a skill in: remember it for `retainTurns` turns. Throws on unknown or unavailable skills. */ + markLoaded(sessionId: string, name: string, turn: number): LoadedSkillInfo { + const skill = this.get(name) + if (!skill) throw new Error(`unknown skill: ${name}`) + const a = this.availability(skill) + if (!a.ok) throw new Error(`skill unavailable: ${a.reason}`) + const list = this.loaded.get(sessionId) ?? [] + const existing = list.find(l => l.name === name) + if (existing) existing.lastUsedTurn = turn + else list.push({ name, lastUsedTurn: turn }) + this.loaded.set(sessionId, list) + return { name, chars: skill.content.length, retainedForTurns: this.opts.retainTurns } + } + + /** Skills still active for a session at `turn`; expired ones are released here. */ + activeFor(sessionId: string, turn: number): Skill[] { + const list = this.loaded.get(sessionId) ?? [] + // Loaded at turn N → still active through turn N + retainTurns. + const kept = list.filter(l => turn - l.lastUsedTurn <= this.opts.retainTurns) + if (kept.length !== list.length) this.loaded.set(sessionId, kept) + return kept.map(l => this.get(l.name)).filter((s): s is Skill => !!s) + } + /** - * Select the most relevant skill for a given message. - * Matches keywords from skill description against message words. - * Returns null if no skill has sufficient overlap. + * Legacy keyword selection — only under `skills.selection: "keyword"`. + * Under `model` the agent pulls skills itself with `skill_load`, so this + * returns null and the catalogue is the only thing in the prompt. */ select(message: string): Skill | null { + if (this.opts.selection !== "keyword") return null if (this.skills.length === 0) return null const words = message.toLowerCase().split(/\W+/).filter(w => w.length > 3) diff --git a/src/slices/agent/skill/domain/skill.types.ts b/src/slices/agent/skill/domain/skill.types.ts index 0a80ca3..a3ec7a8 100644 --- a/src/slices/agent/skill/domain/skill.types.ts +++ b/src/slices/agent/skill/domain/skill.types.ts @@ -19,6 +19,27 @@ export interface SkillSummary { name: string description: string metadata?: SkillMetadata + /** Why the skill cannot be loaded right now (missing env / binary), shown in the catalogue. */ + unavailable?: string +} + +/** How skills enter a turn: pulled by the model with `skill_load`, or the legacy keyword matcher. */ +export type SkillSelection = "model" | "keyword" + +export interface SkillServiceOptions { + selection: SkillSelection + /** A loaded skill stays active this many turns after its last use. */ + retainTurns: number + /** Environment seen by `requires.env` checks. */ + env: Record + /** Binary lookup for `requires.bins` checks. */ + hasBin: (bin: string) => boolean +} + +export interface LoadedSkillInfo { + name: string + chars: number + retainedForTurns: number } // ─── Skill (full — loaded on demand) ───────────────────────────────────────── diff --git a/src/slices/agent/skill/domain/skillEvent.ts b/src/slices/agent/skill/domain/skillEvent.ts new file mode 100644 index 0000000..0c2c28d --- /dev/null +++ b/src/slices/agent/skill/domain/skillEvent.ts @@ -0,0 +1,25 @@ +import type { Event } from "../../../setup/event" +import type { Skill } from "./skill.types" + +/** + * A loaded skill as a history event (CLEAN-150 US2). It rides as a `summary` + * — the one event type every LLM adapter already renders as context — tagged + * with the skill name so the runtime can find, refresh and expire it without + * touching the cached system prompt. + */ +export function skillEvent(skill: Skill): Event { + return { + id: `skill-${skill.name}`, + type: "summary", + ts: Date.now(), + data: { + skill: skill.name, + text: `[SKILL ${skill.name}] Instructions loaded for this conversation — follow them.\n\n${skill.content.trim()}\n[END SKILL ${skill.name}]`, + }, + } +} + +/** True for a history event produced by `skillEvent`. */ +export function isSkillEvent(event: Event): boolean { + return event.type === "summary" && typeof (event.data as { skill?: unknown })?.skill === "string" +} diff --git a/src/slices/agent/skill/skill.module.ts b/src/slices/agent/skill/skill.module.ts index 38e0773..94c3faf 100644 --- a/src/slices/agent/skill/skill.module.ts +++ b/src/slices/agent/skill/skill.module.ts @@ -1,12 +1,14 @@ -import type { Skill, RegistryEntry } from "./domain/skill.types" +import type { Skill, RegistryEntry, SkillSummary, SkillServiceOptions, LoadedSkillInfo } from "./domain/skill.types" import { SkillService } from "./domain/skill.service" import { SkillGateway } from "./data/skill.gateway" +export type { SkillSummary, SkillSelection, LoadedSkillInfo } from "./domain/skill.types" + export class SkillModule { private service: SkillService - constructor(agentDir: string, bundledDir?: string) { - this.service = new SkillService(new SkillGateway(agentDir, bundledDir)) + constructor(agentDir: string, opts: Partial = {}, bundledDir?: string) { + this.service = new SkillService(new SkillGateway(agentDir, bundledDir), opts) } async load(): Promise { @@ -27,10 +29,28 @@ export class SkillModule { return this.service.get(name) } + summaries(): SkillSummary[] { + return this.service.summaries() + } + select(message: string): Skill | null { return this.service.select(message) } + // ─── Per-session on-demand loading ──────────────────────────────────────── + + nextTurn(sessionId: string): number { + return this.service.nextTurn(sessionId) + } + + markLoaded(sessionId: string, name: string, turn: number): LoadedSkillInfo { + return this.service.markLoaded(sessionId, name, turn) + } + + activeFor(sessionId: string, turn: number): Skill[] { + return this.service.activeFor(sessionId, turn) + } + // ─── Install / Remove ────────────────────────────────────────────────────── async install(nameOrUrl: string): Promise { diff --git a/src/slices/agent/tool/data/repositories/skill/skill_load.repository.ts b/src/slices/agent/tool/data/repositories/skill/skill_load.repository.ts new file mode 100644 index 0000000..886f305 --- /dev/null +++ b/src/slices/agent/tool/data/repositories/skill/skill_load.repository.ts @@ -0,0 +1,33 @@ +import { z } from "zod" +import type { Tool, ToolContext } from "../../../domain/tool.types" + +const schema = z.object({ + name: z.string().min(1).describe("Skill name exactly as listed in the Available Skills catalogue"), +}) + +/** + * The model pulls a skill into the conversation when a catalogue description + * fits the task (CLEAN-150 US2). The runtime places the skill body into the + * history — not into the system prompt, which keeps the cached prefix + * stable — and keeps it there for `skills.retainTurns` turns. + */ +export const SkillLoadTool: Tool = { + name: "skill_load", + effect: "read", + description: `Load a skill's full instructions for this conversation. Call it when a skill in the "Available Skills" catalogue matches what the person needs — its description says when. The instructions arrive in the conversation and stay for a few turns; call again if you need them later. Returns an error for an unknown or unavailable skill (missing env or binary).`, + schema, + stepLabel(params: unknown): string | undefined { + const p = schema.safeParse(params) + return p.success ? `Load skill ${p.data.name}` : undefined + }, + async execute(params: unknown, ctx: ToolContext): Promise { + const { name } = schema.parse(params) + if (!ctx.skills) return { error: "skills are not available in this runtime" } + try { + const info = ctx.skills.markLoaded(ctx.sessionId, name, ctx.turn ?? 0) + return { ...info, note: "The skill's instructions are now in the conversation above; follow them." } + } catch (err) { + return { error: (err as Error).message } + } + }, +} diff --git a/src/slices/agent/tool/data/repositories/status/agent_status.repository.ts b/src/slices/agent/tool/data/repositories/status/agent_status.repository.ts index fc922c8..ac5e02d 100644 --- a/src/slices/agent/tool/data/repositories/status/agent_status.repository.ts +++ b/src/slices/agent/tool/data/repositories/status/agent_status.repository.ts @@ -3,6 +3,7 @@ import { readdir } from "fs/promises" import { join } from "path" import type { Tool, ToolContext } from "../../../domain/tool.types" import type { IChannelInfo } from "../../../../../setup/channel/channel.module" +import type { IPromptBreakdown } from "../../../../agent/domain/agent.types" import pkg from "../../../../../../../package.json" const schema = z.object({}) @@ -35,6 +36,8 @@ interface IAgentStatusReport { environment?: { ranchAdmin: boolean; bridleConnected: boolean; s3Sync: boolean } /** Guardrails: how many rules are active, which fired in the last day, and whether the set loaded (CLEAN-150). */ rules?: { count: number; loaded: boolean; loadError?: string; firedLast24h: Array<{ id: string; times: number }> } + /** System-prompt accounting of this turn against `contextBudget` (CLEAN-150 US2). */ + context?: IPromptBreakdown } export const AgentStatusTool: Tool = { @@ -117,6 +120,7 @@ Safe for any caller — the report is trimmed to what the current user is allowe callCount: today.totalCallCount, } } + if (ctx.promptBreakdown) report.context = ctx.promptBreakdown if (ctx.guard) { report.rules = { count: ctx.guard.listRules().filter(r => r.enabled).length, diff --git a/src/slices/agent/tool/data/repositories/tool/tool_describe.repository.ts b/src/slices/agent/tool/data/repositories/tool/tool_describe.repository.ts new file mode 100644 index 0000000..df83f86 --- /dev/null +++ b/src/slices/agent/tool/data/repositories/tool/tool_describe.repository.ts @@ -0,0 +1,35 @@ +import { z } from "zod" +import { zodToJsonSchema } from "zod-to-json-schema" +import type { Tool, ToolContext } from "../../../domain/tool.types" + +const schema = z.object({ + name: z.string().min(1).describe("Tool name exactly as listed in the Tooling section"), +}) + +/** + * Full description and parameter schema for a tool the catalogue only shows + * as a one-line stub (CLEAN-150 US2). The provider still receives every + * tool's schema; this is the prose the budgeted catalogue left out. + */ +export const ToolDescribeTool: Tool = { + name: "tool_describe", + effect: "read", + description: `Full description and parameters of a tool. Call it before using a tool whose catalogue entry is a one-line stub, or when you are unsure what a tool's parameters mean. Returns description, JSON schema of the parameters, whether it changes anything ("effect") and whether it is admin-only.`, + schema, + stepLabel(params: unknown): string | undefined { + const p = schema.safeParse(params) + return p.success ? `Describe tool ${p.data.name}` : undefined + }, + async execute(params: unknown, ctx: ToolContext): Promise { + const { name } = schema.parse(params) + const tool = (ctx.tools ?? []).find(t => t.name === name) + if (!tool) return { error: `Unknown tool: ${name}` } + return { + name: tool.name, + description: tool.description, + schema: tool.inputSchema ?? (zodToJsonSchema(tool.schema) as Record), + effect: tool.effectOf ? "depends on the call (read or write)" : (tool.effect ?? "write"), + adminOnly: tool.adminOnly ?? false, + } + }, +} diff --git a/src/slices/agent/tool/data/tool.gateway.ts b/src/slices/agent/tool/data/tool.gateway.ts index 4caf762..57b2c48 100644 --- a/src/slices/agent/tool/data/tool.gateway.ts +++ b/src/slices/agent/tool/data/tool.gateway.ts @@ -36,6 +36,8 @@ import { SessionListTool } from "./repositories/session/session_list.repository" import { ResourceStatusTool } from "./repositories/resource/resource_status.repository" import { AgentStatusTool } from "./repositories/status/agent_status.repository" import { RenderFormTool } from "./repositories/ui/render_form.repository" +import { SkillLoadTool } from "./repositories/skill/skill_load.repository" +import { ToolDescribeTool } from "./repositories/tool/tool_describe.repository" export class ToolGateway implements IToolGateway { private tools: Map = new Map() @@ -80,6 +82,8 @@ export class ToolGateway implements IToolGateway { this.register(ResourceStatusTool) this.register(AgentStatusTool) this.register(RenderFormTool) + this.register(SkillLoadTool) + this.register(ToolDescribeTool) } private register(tool: Tool): void { diff --git a/src/slices/agent/tool/domain/tool.service.spec.ts b/src/slices/agent/tool/domain/tool.service.spec.ts new file mode 100644 index 0000000..9bce2b8 --- /dev/null +++ b/src/slices/agent/tool/domain/tool.service.spec.ts @@ -0,0 +1,46 @@ +import { describe, expect, test } from "bun:test" +import { z } from "zod" +import { ToolService } from "./tool.service" +import type { Tool } from "./tool.types" + +const tool = (name: string, description: string): Tool => ({ + name, description, schema: z.object({}), execute: async () => null, +}) + +const tools: Tool[] = [ + tool("exec", "Run a shell command and return stdout/stderr. Blocks until done."), + tool("http", "Make an HTTP request and return the response."), + tool("ranch__list_agents", "List every agent on the ranch with its pod state, template and owner. Supports paging and filters by status, template and owner; returns up to 200 rows."), + tool("ranch__restart_agent", "Restart one agent by id. Waits for the pod to come back and reports the new state."), +] + +describe("tooling catalogue budget (CLEAN-150 US2)", () => { + test("without a core list every tool gets its full description (legacy behaviour)", () => { + const prompt = ToolService.buildToolingPromptFrom(tools) + expect(prompt).toContain("Blocks until done") + expect(prompt).toContain("returns up to 200 rows") + expect(prompt).not.toContain("tool_describe") + }) + + test("core tools keep the full description, the rest become one-line stubs pointing at tool_describe", () => { + const prompt = ToolService.buildToolingPromptFrom(tools, { core: ["exec", "http"] }) + expect(prompt).toContain("`exec` — Run a shell command and return stdout/stderr. Blocks until done.") + expect(prompt).toContain("`http` — Make an HTTP request and return the response.") + expect(prompt).not.toContain("returns up to 200 rows") + expect(prompt).toContain("`ranch__list_agents` — List every agent on the ranch with its pod state, template and owner.") + expect(prompt).not.toContain("Supports paging") + expect(prompt).toContain("tool_describe") + }) + + test("a stub is at most one line of bounded length", () => { + const prompt = ToolService.buildToolingPromptFrom(tools, { core: [] }) + const stub = prompt.split("\n").find(l => l.startsWith("- `ranch__list_agents`"))! + expect(stub.length).toBeLessThanOrEqual(120) + }) + + test("the catalogue shrinks when tools move out of the core set", () => { + const full = ToolService.buildToolingPromptFrom(tools).length + const lean = ToolService.buildToolingPromptFrom(tools, { core: ["exec"] }).length + expect(lean).toBeLessThan(full) + }) +}) diff --git a/src/slices/agent/tool/domain/tool.service.ts b/src/slices/agent/tool/domain/tool.service.ts index 5990d24..64db08a 100644 --- a/src/slices/agent/tool/domain/tool.service.ts +++ b/src/slices/agent/tool/domain/tool.service.ts @@ -1,5 +1,14 @@ import type { Tool, ToolContext } from "./tool.types" +const STUB_MAX_CHARS = 80 + +/** First sentence of a description, cut to one bounded line. */ +function stub(description: string): string { + const firstLine = description.split("\n").find(l => l.trim()) ?? "" + const sentence = firstLine.match(/^.*?[.!?](\s|$)/)?.[0]?.trim() ?? firstLine.trim() + return sentence.length > STUB_MAX_CHARS ? sentence.slice(0, STUB_MAX_CHARS - 1).trimEnd() + "…" : sentence +} + export class ToolService { private tools: Map = new Map() @@ -32,8 +41,13 @@ export class ToolService { /** * Build tooling + tool call style sections from any Tool array. * Usable without a ToolService instance. + * + * With `opts.core`, only those tools keep their full description; every + * other tool is a one-line stub and `tool_describe` hands out the rest on + * demand, so the catalogue stays within its budget however many MCP tools + * are connected (CLEAN-150 US2). */ - static buildToolingPromptFrom(tools: Tool[]): string { + static buildToolingPromptFrom(tools: Tool[], opts?: { core?: string[] }): string { if (tools.length === 0) return "" const lines = [ @@ -44,8 +58,18 @@ export class ToolService { "", ] + const core = opts?.core ? new Set(opts.core) : undefined + let stubbed = 0 for (const tool of tools) { - lines.push(`- \`${tool.name}\` — ${tool.description}`) + if (!core || core.has(tool.name)) { + lines.push(`- \`${tool.name}\` — ${tool.description}`) + } else { + lines.push(`- \`${tool.name}\` — ${stub(tool.description)}`) + stubbed++ + } + } + if (stubbed > 0) { + lines.push("", `${stubbed} tool${stubbed === 1 ? "" : "s"} above show a one-line summary. Call \`tool_describe\` with the name for the full description and parameters before using one.`) } lines.push( diff --git a/src/slices/agent/tool/domain/tool.types.ts b/src/slices/agent/tool/domain/tool.types.ts index 502535e..8b9d3be 100644 --- a/src/slices/agent/tool/domain/tool.types.ts +++ b/src/slices/agent/tool/domain/tool.types.ts @@ -8,6 +8,8 @@ import type { UsageModule } from "../../../bot/usage/usage.module" import type { LastTurnStatsTracker } from "../../../runtime/loop/domain/last-turn-stats.tracker" import type { ISource } from "../../../runtime/loop/domain/loop.types" import type { GuardModule } from "../../guard/guard.module" +import type { SkillModule } from "../../skill/skill.module" +import type { IPromptBreakdown } from "../../agent/domain/agent.types" export type ToolEffect = "read" | "write" @@ -61,6 +63,12 @@ export interface ToolContext { lastTurnStats?: LastTurnStatsTracker /** Guardrails — lets `agent_status` report the rule set and recent firings (CLEAN-150). */ guard?: GuardModule + /** Skills — `skill_load` pulls a skill into the session for a few turns (CLEAN-150 US2). */ + skills?: SkillModule + /** This session's turn number, for skill retention. */ + turn?: number + /** The system-prompt accounting of this turn — `agent_status.context`. */ + promptBreakdown?: IPromptBreakdown /** * Tools visible to THIS session's caller (admin-only tools are already * filtered out upstream for non-admins). Lets introspection tools like diff --git a/src/slices/bot/command/command.module.ts b/src/slices/bot/command/command.module.ts index 6796f50..c8a879b 100644 --- a/src/slices/bot/command/command.module.ts +++ b/src/slices/bot/command/command.module.ts @@ -7,6 +7,7 @@ import type { SessionModule } from "../../agent/session/session.module" import type { MemoryModule } from "../../agent/memory/memory.module" import type { LlmModule } from "../../setup/llm/llm.module" import type { GuardModule } from "../../agent/guard" +import type { IPromptBreakdown } from "../../agent/agent/agent.module" export { CommandService } export type { ICommandContext, ICommandResult } from "./domain/command.types" @@ -20,6 +21,7 @@ interface CommandModuleDeps { memory: MemoryModule llm: LlmModule guard?: GuardModule + context?: (sessionId: string) => IPromptBreakdown | undefined } export class CommandModule { diff --git a/src/slices/bot/command/domain/command.service.ts b/src/slices/bot/command/domain/command.service.ts index bcb206c..17e6013 100644 --- a/src/slices/bot/command/domain/command.service.ts +++ b/src/slices/bot/command/domain/command.service.ts @@ -10,6 +10,7 @@ import type { SessionModule } from "../../../agent/session/session.module" import type { MemoryModule } from "../../../agent/memory/memory.module" import type { LlmModule } from "../../../setup/llm/llm.module" import type { GuardModule } from "../../../agent/guard" +import type { IPromptBreakdown } from "../../../agent/agent/agent.module" interface CommandDeps { access: AccessModule @@ -21,6 +22,8 @@ interface CommandDeps { llm: LlmModule /** Guardrails — `/rules` (CLEAN-150 US1). Optional so bare deployments without rules still command. */ guard?: GuardModule + /** The latest system-prompt accounting of a session — `/context` (CLEAN-150 US2). */ + context?: (sessionId: string) => IPromptBreakdown | undefined } // Commands only await the send; the id a channel may hand back is not theirs to use. @@ -47,6 +50,11 @@ export class CommandService { return this.handleRules(ctx, text, send) } + // /context — where the always-on prompt's characters went last turn (admin only) + if (text === "/context") { + return this.handleContext(ctx, send) + } + // /voice toggle if (text === "/voice") { const isNowOn = this.deps.voice.toggle(ctx.from) @@ -84,7 +92,8 @@ export class CommandService { `/access — Show current access strategy\n` + `/access — Switch strategy: open|public|allowlist|code|approval (admin only)\n` + `/rules — Guardrails in force and what fired today (admin only)\n` + - `/rules reload — Re-read .agent/rules/ (admin only)`, + `/rules reload — Re-read .agent/rules/ (admin only)\n` + + `/context — System prompt size per section vs budget, and what was loaded on demand (admin only)`, ) return { handled: true } } @@ -188,6 +197,31 @@ export class CommandService { return { handled: true } } + private async handleContext(ctx: ICommandContext, send: SendFn): Promise { + if (!this.deps.access.isAdmin(ctx.from)) { + await send("🔒 Only admins can see the context breakdown.") + return { handled: true } + } + const b = this.deps.context?.(ctx.sessionId) + if (!b) { + await send("No turn has run in this session yet — send a message first, then `/context`.") + return { handled: true } + } + const pct = (n: number, d: number) => d > 0 ? ` (${Math.round((n / d) * 100)}%)` : "" + const lines = b.sections.map(s => + `• ${s.name}: ${s.chars.toLocaleString()}${s.budget !== undefined ? ` / ${s.budget.toLocaleString()}${pct(s.chars, s.budget)}` : ""}${s.over ? " ⚠ over" : ""}`, + ) + const onDemand = b.onDemand.length > 0 + ? b.onDemand.map(l => `• ${l.kind} ${l.name}: ${l.chars.toLocaleString()}`).join("\n") + : "(nothing loaded on demand this turn)" + await send( + `📐 *Context — last turn*\n\n` + + `Always-on prompt: ${b.total.toLocaleString()} / ${b.budget.toLocaleString()} chars${pct(b.total, b.budget)}${b.total > b.budget ? " ⚠ over budget" : ""}\n\n` + + `${lines.join("\n")}\n\n*On demand:*\n${onDemand}`, + ) + return { handled: true } + } + private async handleCancel(ctx: ICommandContext, taskId: string, send: SendFn): Promise { const task = this.deps.tasks.get(taskId) if (task && task.sessionId !== ctx.sessionId) { diff --git a/src/slices/runtime/loop/domain/loop.service.spec.ts b/src/slices/runtime/loop/domain/loop.service.spec.ts index 0b35176..46141f5 100644 --- a/src/slices/runtime/loop/domain/loop.service.spec.ts +++ b/src/slices/runtime/loop/domain/loop.service.spec.ts @@ -157,3 +157,43 @@ describe("LoopService with guardrails (CLEAN-150 US1)", () => { expect(sent).toEqual(["done"]) }) }) + +describe("LoopService with on-demand skills (CLEAN-150 US2)", () => { + const ranch = { name: "ranch", description: "Operate the ranch API", path: "/x", content: "# Ranch\n\nCall GET /agents first.", source: "agent" as const } + const skills = { + get: (name: string) => (name === "ranch" ? ranch : undefined), + markLoaded: (_s: string, name: string) => ({ name, chars: ranch.content.length, retainedForTurns: 3 }), + } + const skillLoadTool: Tool = { + name: "skill_load", description: "load", effect: "read", schema: z.object({ name: z.string() }), + execute: async (params, ctx) => ctx.skills!.markLoaded(ctx.sessionId, (params as { name: string }).name, ctx.turn ?? 0), + } + + test("a successful skill_load puts the skill body into history ahead of the conversation and leaves the system prompt untouched", async () => { + const agentDir = mkdtempSync(join(tmpdir(), "skill-loop-")) + const tools = [skillLoadTool] + const { llm, calls } = scriptedLlm([ + { text: "", toolCalls: [{ name: "skill_load", params: { name: "ranch" } }], stopReason: "tool_use" }, + ]) + const { session } = recordingSession() + const sent: string[] = [] + const loads: unknown[] = [] + const ctx = makeCtx(agentDir, tools, sent) + ctx.systemPrompt = "SYSTEM" + ctx.history = [{ id: "u1", type: "user", ts: 1, data: { text: "how many agents are stale?", from: "u" } }] + ctx.onDemandLoad = (l) => loads.push(l) + + await new LoopService({ + llm: llm as never, session: session as never, activity: { updateStep: () => {} } as never, usage: { add: () => {} } as never, + voice: {} as never, channel: {} as never, tools, skills: skills as never, + }).run(ctx) + + expect(ctx.systemPrompt).toBe("SYSTEM") + const secondCall = calls[1]!.history + expect(secondCall[0]!.type).toBe("summary") + expect((secondCall[0]!.data as { skill: string; text: string }).skill).toBe("ranch") + expect((secondCall[0]!.data as { text: string }).text).toContain("Call GET /agents first.") + expect(secondCall[1]!.type).toBe("user") + expect(loads).toEqual([{ kind: "skill", name: "ranch", chars: ranch.content.length }]) + }) +}) diff --git a/src/slices/runtime/loop/domain/loop.service.ts b/src/slices/runtime/loop/domain/loop.service.ts index 98cbb03..87b1190 100644 --- a/src/slices/runtime/loop/domain/loop.service.ts +++ b/src/slices/runtime/loop/domain/loop.service.ts @@ -8,6 +8,8 @@ import type { ChannelModule } from "../../../setup/channel/channel.module" import type { Tool } from "../../../agent/tool" import { effectOfCall } from "../../../agent/tool/domain/tool.types" import type { GuardModule, IGuardCall, CallOutcome } from "../../../agent/guard" +import type { SkillModule } from "../../../agent/skill/skill.module" +import { skillEvent } from "../../../agent/skill/domain/skillEvent" import type { ILoopContext, ILoopConfig, ILoopResult, IAssistantBubble, ISource } from "./loop.types" import { LOOP_DEFAULTS } from "./loop.types" import { SourceRegistry, citeBlock, extractSources, finalizeBubble, stripMarkers } from "./sources" @@ -30,6 +32,8 @@ interface LoopServiceDeps { tools: Tool[] /** Guardrails around every tool call (CLEAN-150 US1). Absent in tests that exercise the bare loop. */ guard?: GuardModule + /** Skills — the body of a skill the model pulls in with `skill_load` goes into the history (CLEAN-150 US2). */ + skills?: SkillModule } // Channels that support live token-by-token streaming. The channel repository @@ -244,13 +248,16 @@ export class LoopService { // only covers the LLM call, not the tool execution that follows. ctx.sendTyping?.() if (ctx.sendThinking) thinkingStepsEmitted = true + // The reasoning prose rides the thinking timeline to the person, so + // it gets the same redaction as a reply. + const detail = response.text || undefined const iterationHadError = await this.executeToolCalls( ctx, response, iterations, turnId, registry, - streamsToClient ? undefined : (response.text || undefined), + streamsToClient ? undefined : (detail && this.deps.guard ? this.deps.guard.redactOutgoing(detail) : detail), ) if (iterationHadError) { @@ -377,13 +384,19 @@ export class LoopService { if (canStream) { let streamedResponse: import("../../../setup/llm/domain/llm.types").ModelResponse | undefined let cited: ISource[] = [] + // Chunks carry the accumulated text; a streamed reply must not show a + // credential the final redaction would have masked (CLEAN-150 FR-005). + const guard = this.deps.guard const messageId = await ctx.streamSend(channel, ctx.from, async (onChunk) => { - streamedResponse = await this.deps.llm.stream(systemPrompt, history, tools, onChunk) + const safeChunk = guard ? (text: string) => onChunk(guard.redactStreaming(text)) : onChunk + streamedResponse = await this.deps.llm.stream(systemPrompt, history, tools, safeChunk) + const finalText = streamedResponse.text ?? "" + const safeText = guard ? guard.redactOutgoing(finalText) : finalText // The bubble is complete here and the channel has not sent its final // text yet — what the streamer returns is what `stream_end` (bridle) // or the last edit (telegram) carries. Check the citations now, so // the text the person keeps is the corrected one (CLEAN-138). - const fixed = finalize(streamedResponse.text ?? "") + const fixed = finalize(safeText) streamedResponse = { ...streamedResponse, text: fixed.text } cited = fixed.sources return fixed.text @@ -495,6 +508,9 @@ export class LoopService { lastTurnStats: this.lastTurnStats, tools: ctx.tools, guard: this.deps.guard, + skills: this.deps.skills, + turn: ctx.turn, + promptBreakdown: ctx.breakdown, }), new Promise((_, reject) => setTimeout(() => reject(new Error(`Tool "${call.name}" timed out after ${this.config.toolTimeout / 1000}s`)), this.config.toolTimeout) @@ -564,12 +580,36 @@ export class LoopService { : capped history.push(forModel === resultEvent.data ? resultEvent : { ...resultEvent, data: forModel }) + // On-demand context (CLEAN-150 US2): a loaded skill's body enters the + // history ahead of the conversation — not the system prompt, whose + // cached prefix must stay stable — and both kinds of load are reported + // so the turn's breakdown shows what came in late. + if (outcome === "ok") this.noteOnDemandLoad(ctx, call.name, result) + ctx.sendThinking?.(turnId, { ...thinkingStep, state: "done" }) } return iterationHadError } + private noteOnDemandLoad(ctx: ILoopContext, toolName: string, result: unknown): void { + if (toolName === "skill_load") { + const name = (result as { name?: string })?.name + const skill = name ? this.deps.skills?.get(name) : undefined + if (!skill) return + const event = skillEvent(skill) + const insertAt = ctx.history[0]?.type === "summary" && !(ctx.history[0].data as { skill?: string }).skill ? 1 : 0 + // Replace an earlier copy of the same skill rather than stacking two. + const existing = ctx.history.findIndex(e => e.type === "summary" && (e.data as { skill?: string }).skill === skill.name) + if (existing >= 0) ctx.history.splice(existing, 1) + ctx.history.splice(insertAt, 0, event) + ctx.onDemandLoad?.({ kind: "skill", name: skill.name, chars: skill.content.length }) + } else if (toolName === "tool_describe") { + const name = (result as { name?: string })?.name + if (name) ctx.onDemandLoad?.({ kind: "tool", name, chars: JSON.stringify(result).length }) + } + } + private maybeEmitDebug( ctx: ILoopContext, response: import("../../../setup/llm/domain/llm.types").ModelResponse, @@ -587,7 +627,7 @@ export class LoopService { systemPrompt: ctx.systemPrompt, history: ctx.history, response: { - text: response.text ?? "", + text: this.deps.guard ? this.deps.guard.redactOutgoing(response.text ?? "") : (response.text ?? ""), toolCalls: response.toolCalls, stopReason: response.stopReason, }, diff --git a/src/slices/runtime/loop/domain/loop.types.ts b/src/slices/runtime/loop/domain/loop.types.ts index fb4dbc9..19d4d2e 100644 --- a/src/slices/runtime/loop/domain/loop.types.ts +++ b/src/slices/runtime/loop/domain/loop.types.ts @@ -4,6 +4,7 @@ import type { Task } from "../../../agent/task/domain/task.service" import type { AccessModule } from "../../../bot/access/access.module" import type { ChannelModule, IBridleDebugPayload } from "../../../setup/channel" import type { IThinkingStep, MessagePart } from "../../../setup/channel/domain/channel.types" +import type { IOnDemandLoad, IPromptBreakdown } from "../../../agent/agent/domain/agent.types" export interface ILoopConfig { maxIterations: number @@ -113,6 +114,12 @@ export interface ILoopContext { * BRIDLE_DEBUG before calling this — the function itself does not gate. */ sendDebug?: (payload: IBridleDebugPayload) => void + /** This session's turn number — skill retention counts in turns (CLEAN-150 US2). */ + turn?: number + /** The system prompt accounting of this turn, handed to `agent_status`. */ + breakdown?: IPromptBreakdown + /** Called when the model pulls a skill or a tool description in mid-turn, so the breakdown can list it. */ + onDemandLoad?: (load: IOnDemandLoad) => void } export interface ILoopResult { diff --git a/src/slices/runtime/loop/loop.module.ts b/src/slices/runtime/loop/loop.module.ts index db3aec8..0009166 100644 --- a/src/slices/runtime/loop/loop.module.ts +++ b/src/slices/runtime/loop/loop.module.ts @@ -8,6 +8,7 @@ import type { VoiceModule } from "../../bot/voice/voice.module" import type { ChannelModule } from "../../setup/channel/channel.module" import type { Tool } from "../../agent/tool" import type { GuardModule } from "../../agent/guard" +import type { SkillModule } from "../../agent/skill/skill.module" export { LoopService } export type { ILoopContext, ILoopResult, ILoopConfig } from "./domain/loop.types" @@ -21,6 +22,7 @@ interface LoopModuleDeps { channel: ChannelModule tools: Tool[] guard?: GuardModule + skills?: SkillModule } export class LoopModule { diff --git a/src/slices/runtime/runtime/domain/runtime.service.ts b/src/slices/runtime/runtime/domain/runtime.service.ts index 4cd7af3..ec191c0 100644 --- a/src/slices/runtime/runtime/domain/runtime.service.ts +++ b/src/slices/runtime/runtime/domain/runtime.service.ts @@ -19,6 +19,7 @@ import { buildResourceHintPrompt } from "../../loop/domain/prompts/resource-hint import { CITATIONS_PROMPT } from "../../../agent/agent/domain/prompts/citations" import { truncateStrings } from "../../../agent/session/domain/compaction.service" import { capPayload, fitEventsToBudget } from "../../../agent/session/domain/contextBudget" +import { isSkillEvent, skillEvent } from "../../../agent/skill/domain/skillEvent" import { limitForUserEvent, truncateUserText } from "./messageTruncation" import { randomUUID } from "crypto" import { createLogger } from "../../../setup/logger" @@ -95,11 +96,24 @@ export class RuntimeService { lastStep: "started", }) - const history = await this.buildHistory(msg, sessionId, task.id) - const toolingPrompt = ToolService.buildToolingPromptFrom(visibleTools) + // Skill retention is counted in turns of the session (CLEAN-150 US2). + const turn = this.deps.skills.nextTurn(sessionId) + const history = await this.buildHistory(msg, sessionId, task.id, turn) + const toolingPrompt = ToolService.buildToolingPromptFrom(visibleTools, { core: this.deps.config.tools.coreCatalogue }) const systemPrompt = await this.buildPrompt(msg, tid, toolingPrompt, isAdmin, sessionId, citeSources) + const breakdown = this.lastBreakdown.get(sessionId) + // Skills carried over from earlier turns count as on-demand context too. + for (const evt of history) { + const skillName = (evt.data as { skill?: string })?.skill + if (evt.type === "summary" && skillName) { + breakdown?.onDemand.push({ kind: "skill", name: skillName, chars: String((evt.data as { text?: string }).text ?? "").length }) + } + } await this.deps.loop.service.run({ + turn, + breakdown, + onDemandLoad: (load) => { this.lastBreakdown.get(sessionId)?.onDemand.push(load) }, task, sessionId, agentDir: this.deps.agentDir, @@ -195,7 +209,7 @@ export class RuntimeService { return `You are responding via ${channel}.` } - private async buildHistory(msg: Message, sessionId: string, taskId: string): Promise { + private async buildHistory(msg: Message, sessionId: string, taskId: string, turn: number): Promise { // Append user message as shared context const userEvent: Event = { // On bridle the message id is minted by the person's browser and rides @@ -273,7 +287,18 @@ export class RuntimeService { if (fitted.dropped > 0) { log.warn(`session ${sessionId}: ${fitted.dropped} events left out to fit ${this.deps.config.session.contextBudgetChars} chars`) } - return fitted.events + + // Skills the model pulled in on earlier turns ride ahead of the + // conversation while they are retained (CLEAN-150 US2). They are never + // persisted — only the fact of loading is — so a fresh copy of the body + // goes in each turn, after the archive summary if there is one. + const events = fitted.events.filter(e => !isSkillEvent(e)) + const active = this.deps.skills.activeFor(sessionId, turn) + if (active.length > 0) { + const insertAt = events[0]?.type === "summary" ? 1 : 0 + events.splice(insertAt, 0, ...active.map(skillEvent)) + } + return events } private async buildPrompt(msg: Message, tid: string, toolingPrompt: string, isAdmin: boolean, sessionId: string, citeSources: boolean): Promise { @@ -282,11 +307,8 @@ export class RuntimeService { // Get all loaded skills — pass summaries to system prompt catalog const allSkills = this.deps.skills.getAll() - const skillSummaries = allSkills.map(s => ({ - name: s.name, - description: s.description, - metadata: s.metadata, - })) + // Summaries carry the reason when a skill cannot be loaded (missing env or binary). + const skillSummaries = this.deps.skills.summaries() // One-shot resource-status hint: if the previous LLM turn for this // session was delayed (retries, rate limit, overload, >30s), nudge the diff --git a/src/slices/runtime/runtime/runtime.module.ts b/src/slices/runtime/runtime/runtime.module.ts index c4e0abd..42f424f 100644 --- a/src/slices/runtime/runtime/runtime.module.ts +++ b/src/slices/runtime/runtime/runtime.module.ts @@ -127,7 +127,7 @@ export class AgentRuntime { this.heartbeat = new HeartbeatModule(agentDir, this.config.heartbeat.intervalMin * 60 * 1000) // Dynamically loaded skills (markdown files in .agent/skills/) - this.skills = new SkillModule(agentDir) + this.skills = new SkillModule(agentDir, this.config.skills) // Per-user voice/TTS toggle const voice = new VoiceModule(agentDir) @@ -159,13 +159,17 @@ export class AgentRuntime { this.guard = new GuardModule(agentDir, this.config.guard) // Slash command handler (/help, /tasks, /cancel, /voice, etc.) - const commands = new CommandService({ access, skills: this.skills, voice, tasks, session: this.session, memory: this.memory, llm: this.llm, guard: this.guard }) + const commands = new CommandService({ + access, skills: this.skills, voice, tasks, session: this.session, memory: this.memory, llm: this.llm, guard: this.guard, + // Resolved at call time: the runtime service is built a few lines below. + context: (sessionId) => this.runtimeService.getLastBreakdown(sessionId), + }) // ── Runtime slices (orchestration) ───────────────────────────── // LLM ↔ tools execution loop this.loop = new LoopModule( - { llm: this.llm, session: this.session, activity: activityService, usage: this.usage, voice, channel: this.channel, tools, guard: this.guard }, + { llm: this.llm, session: this.session, activity: activityService, usage: this.usage, voice, channel: this.channel, tools, guard: this.guard, skills: this.skills }, { maxIterations: this.config.maxIterations, maxToolOutputChars: this.config.tools.maxOutputChars, diff --git a/tests/evals/skill-selection.jsonl b/tests/evals/skill-selection.jsonl new file mode 100644 index 0000000..155ef4e --- /dev/null +++ b/tests/evals/skill-selection.jsonl @@ -0,0 +1,101 @@ +{"mode":"skill-loaded","threshold":90,"maxFalseLoadPct":5,"description":"SC-003: given the skill catalogue of the example agent plus the four ranch skills, the model loads the right skill for a message (expect = skill name) and loads nothing for messages no skill covers (expect = null). Needs LLM_API_KEY; skipped as pending without it.","skills":[{"name":"ranch","description":"Operate the ranch platform through its API: agents (list, create, restart, delete, pods and their state, logs, usage and spend, admin rights, the files of an agent such as SOUL.md), templates and their files, the platform skill catalogue (import from GitHub), knowledge bases (attach, query, sources, freshness), LLM credentials, users and settings."},{"name":"ranch-sessions","description":"Third-party services connected or linked to a ranch agent (Instagram, X/Twitter, Google): list the connected sessions and accounts, request or renew a login, disconnect a service, read session secrets and cookie profiles."},{"name":"runtime-tools","description":"What the agent itself can do and which built-in tool does it: shell commands and background processes, HTTP and fetching URLs, web search, browser and screenshots, PDF and image analysis, unzip, voice messages, scheduling, memory and secrets. Consult for any question about the agent's own tools or capabilities, or before invoking an unknown command name."},{"name":"skill-creator","description":"Create, validate, fix and manage agent skills: write a new skill, turn a procedure into a reusable skill, SKILL.md frontmatter and description rules, size limits, directory layout (scripts, references)."},{"name":"github","description":"Work with GitHub repositories: open, review and merge pull requests, issues and labels, releases and tags, release notes, commit and file history."}]} +{"id":"r01","message":"how many knowledge bases are stale?","expect":"ranch"} +{"id":"r02","message":"restart the billing bot, it's stuck","expect":"ranch"} +{"id":"r03","message":"spin up a new support assistant from the FAQ template","expect":"ranch"} +{"id":"r04","message":"which of our bots are crashlooping right now?","expect":"ranch"} +{"id":"r05","message":"show me the pods that aren't running","expect":"ranch"} +{"id":"r06","message":"delete the test agent I made yesterday","expect":"ranch"} +{"id":"r07","message":"what templates do we have?","expect":"ranch"} +{"id":"r08","message":"attach the pricing knowledge base to the sales bot","expect":"ranch"} +{"id":"r09","message":"rotate the anthropic key for all agents","expect":"ranch"} +{"id":"r10","message":"who has access to the platform?","expect":"ranch"} +{"id":"r11","message":"the onboarding bot answers slowly, check its logs","expect":"ranch"} +{"id":"r12","message":"how much did the top three agents spend this month?","expect":"ranch"} +{"id":"r13","message":"rename the template 'Support v2' to 'Support'","expect":"ranch"} +{"id":"r14","message":"import that skill from github into the platform catalogue","expect":"ranch"} +{"id":"r15","message":"is the rancher template up to date?","expect":"ranch"} +{"id":"r16","message":"list agents that use the gpt credential","expect":"ranch"} +{"id":"r17","message":"change the setting that controls the default model","expect":"ranch"} +{"id":"r18","message":"add a knowledge source for the docs site","expect":"ranch"} +{"id":"r19","message":"what does the sales bot's SOUL.md say?","expect":"ranch"} +{"id":"r20","message":"give the marketing bot admin rights","expect":"ranch"} +{"id":"r21","message":"which credentials are inactive?","expect":"ranch"} +{"id":"r22","message":"create a bot for HR with the standard template and the policies knowledge","expect":"ranch"} +{"id":"r23","message":"query the product knowledge base for refund rules","expect":"ranch"} +{"id":"r24","message":"save this file into the support template","expect":"ranch"} +{"id":"r25","message":"why is agent 42 pending for 20 minutes?","expect":"ranch"} +{"id":"s01","message":"is my instagram connected?","expect":"ranch-sessions"} +{"id":"s02","message":"log the agent into X so it can post","expect":"ranch-sessions"} +{"id":"s03","message":"which services are linked to this agent?","expect":"ranch-sessions"} +{"id":"s04","message":"the google session expired, request a new login","expect":"ranch-sessions"} +{"id":"s05","message":"show the cookies profile for instagram","expect":"ranch-sessions"} +{"id":"s06","message":"disconnect twitter from this bot","expect":"ranch-sessions"} +{"id":"s07","message":"can you read my connected accounts?","expect":"ranch-sessions"} +{"id":"s08","message":"we need a login link for the client's instagram","expect":"ranch-sessions"} +{"id":"s09","message":"what session secrets do you hold for X?","expect":"ranch-sessions"} +{"id":"s10","message":"re-authenticate the google workspace connection","expect":"ranch-sessions"} +{"id":"t01","message":"what tools do you have for reading PDFs?","expect":"runtime-tools"} +{"id":"t02","message":"can you take a screenshot of a page?","expect":"runtime-tools"} +{"id":"t03","message":"how do you run a long build in the background?","expect":"runtime-tools"} +{"id":"t04","message":"which command fetches a URL as text?","expect":"runtime-tools"} +{"id":"t05","message":"do you have something to search the web?","expect":"runtime-tools"} +{"id":"t06","message":"what's the difference between browser and browser_play?","expect":"runtime-tools"} +{"id":"t07","message":"how do you send a voice message?","expect":"runtime-tools"} +{"id":"t08","message":"list the built-in tools you can call","expect":"runtime-tools"} +{"id":"t09","message":"can you unzip a file I send?","expect":"runtime-tools"} +{"id":"t10","message":"how do I schedule a recurring check?","expect":"runtime-tools"} +{"id":"k01","message":"write a new skill for weather lookups","expect":"skill-creator"} +{"id":"k02","message":"what's the right frontmatter for a SKILL.md?","expect":"skill-creator"} +{"id":"k03","message":"validate the skill I just added","expect":"skill-creator"} +{"id":"k04","message":"make a skill that wraps our internal CRM API","expect":"skill-creator"} +{"id":"k05","message":"how long can a skill file be?","expect":"skill-creator"} +{"id":"k06","message":"where do scripts go inside a skill folder?","expect":"skill-creator"} +{"id":"k07","message":"turn these steps into a reusable skill","expect":"skill-creator"} +{"id":"k08","message":"the skill description should say when to use it — fix mine","expect":"skill-creator"} +{"id":"k09","message":"create a skill for daily digests","expect":"skill-creator"} +{"id":"k10","message":"check that my skill lists its required env vars","expect":"skill-creator"} +{"id":"g01","message":"open a pull request for the fix branch","expect":"github"} +{"id":"g02","message":"what issues are assigned to me?","expect":"github"} +{"id":"g03","message":"review PR 42 and leave comments","expect":"github"} +{"id":"g04","message":"cut a release tagged v1.2.0","expect":"github"} +{"id":"g05","message":"close the stale issues older than a year","expect":"github"} +{"id":"g06","message":"who last changed the README in the runtime repo?","expect":"github"} +{"id":"g07","message":"merge the dependabot PRs","expect":"github"} +{"id":"g08","message":"label issue 17 as bug","expect":"github"} +{"id":"g09","message":"how many open PRs do we have?","expect":"github"} +{"id":"g10","message":"draft release notes from the merged PRs","expect":"github"} +{"id":"n01","message":"hi","expect":null} +{"id":"n02","message":"thanks, that's all","expect":null} +{"id":"n03","message":"what's 17% of 2300?","expect":null} +{"id":"n04","message":"translate 'good morning' to spanish","expect":null} +{"id":"n05","message":"summarize this paragraph: The quick brown fox jumps over the lazy dog repeatedly.","expect":null} +{"id":"n06","message":"what day of the week is 2026-12-25?","expect":null} +{"id":"n07","message":"write a haiku about rain","expect":null} +{"id":"n08","message":"explain what a mutex is","expect":null} +{"id":"n09","message":"remind me what we discussed about the budget","expect":null} +{"id":"n10","message":"convert 5 miles to km","expect":null} +{"id":"n11","message":"can you speak german?","expect":null} +{"id":"n12","message":"what's your name?","expect":null} +{"id":"n13","message":"rewrite this email to sound friendlier: We need the report now.","expect":null} +{"id":"n14","message":"give me three ideas for a team offsite","expect":null} +{"id":"n15","message":"is 97 a prime number?","expect":null} +{"id":"n16","message":"what time zone is Berlin in?","expect":null} +{"id":"n17","message":"sort these: pear, apple, fig","expect":null} +{"id":"n18","message":"ok","expect":null} +{"id":"n19","message":"what did I ask you earlier?","expect":null} +{"id":"n20","message":"spell 'necessary'","expect":null} +{"id":"n21","message":"how are you today?","expect":null} +{"id":"n22","message":"what's the capital of Australia?","expect":null} +{"id":"n23","message":"draft a two-line apology for a late delivery","expect":null} +{"id":"n24","message":"count the words in: one two three four","expect":null} +{"id":"n25","message":"what's a good name for a grey cat?","expect":null} +{"id":"m01","message":"fetch https://example.com and tell me the title","expect":null,"allow":["runtime-tools"]} +{"id":"m02","message":"run ls in the workspace","expect":null,"allow":["runtime-tools"]} +{"id":"m03","message":"save my API key: abc123","expect":null,"allow":["runtime-tools"]} +{"id":"m04","message":"remember that I prefer short answers","expect":null,"allow":["runtime-tools"]} +{"id":"m05","message":"search the web for bun 1.2 release notes","expect":null,"allow":["runtime-tools"]} +{"id":"m06","message":"what did the http call return last time?","expect":null,"allow":["runtime-tools"]} +{"id":"m07","message":"schedule a reminder for 9am tomorrow","expect":null,"allow":["runtime-tools"]} +{"id":"m08","message":"analyze the image I just sent","expect":null,"allow":["runtime-tools"]} +{"id":"m09","message":"what's in my memory about the client?","expect":null,"allow":["runtime-tools"]} +{"id":"m10","message":"read workspace/notes.md","expect":null,"allow":["runtime-tools"]}