diff --git a/.changeset/run-menu-custom-model-picker.md b/.changeset/run-menu-custom-model-picker.md new file mode 100644 index 000000000..3b1f45d97 --- /dev/null +++ b/.changeset/run-menu-custom-model-picker.md @@ -0,0 +1,28 @@ +--- +'aicodeman': minor +--- + +**Custom model endpoints: Run-menu picker, and hardening from real llama-swap validation** (#430, follow-up to #393's HTTP-API-only cut). With **Custom model endpoints** on (App Settings → Models) and at least one saved endpoint carrying a discovered model, the Run dropdown grows a **Custom Endpoints** section generated live off the CLI registry's own `capabilities.customModelInjection` — one entry per (harness that can redirect to a custom endpoint, saved endpoint). Picking one launches that harness and applies the endpoint to it; with two or more discovered models a small, scrollable dialog asks which one first, the endpoint's `defaultModelId` marked but never auto-chosen. Endpoints also now re-discover themselves automatically every 5 minutes in the background, one unreachable endpoint never blocking the others. + +Everything below was found and fixed against a **real llama-swap server**, not just unit tests: + +- **Session-busy false refusal.** A freshly launched CLI reports itself `busy` for its own startup (spinner, workspace-trust check) well before the apply call would reach it, and the apply route correctly refuses to restart a session mid-turn — indistinguishable from a fresh boot. The picker now waits for the new session to go idle (bounded at 20s, never an error on timeout) before applying. +- **Errors and confirmations you can actually read.** Toasts now default to sticky with a close button (errors always were meant to stay, but a fixed 3s timer silently hid them); a failed apply's real server-side reason (not a generic message) reaches the toast. +- **"Both claude.ai and ANTHROPIC_API_KEY set" warning.** A custom-model Claude session now runs with an isolated `CLAUDE_CONFIG_DIR` (empty, no real credentials in it) so the injected API key never coexists with a stored OAuth login — `projects` is symlinked back to the real config dir so the response viewer/subagent windows/Read My Mind keep working. That isolated, otherwise-empty directory has none of a real profile's prior "Detected a custom API key — use it?" approvals either, which would otherwise re-ask on _every_ launch with nobody at a TTY to answer (and silently refuse the key on its own default); the apply step now pre-seeds that exact approval field the same way answering the prompt once by hand would. +- **Context-window overflow.** Claude Code assumes a large default context window for a model id it doesn't recognize and never compacts, so a real local model's much smaller context silently overflowed (confirmed live: a stock ~33.7K-token system prompt against a 16384-token model). Discovery now also learns each model's real context length and applies it as `CLAUDE_CODE_MAX_CONTEXT_TOKENS` — sourced primarily from llama-swap's own `GET /running`, whose `cmd` field carries the launch flags (`--fit-ctx`/`-c`/`--ctx-size`) actually in effect, since `GET /props`'s `n_ctx` was confirmed live to report the model's theoretical/trained maximum rather than the real `--fit-ctx`-shrunk runtime context (a 154112-vs-16384 discrepancy, caught only because the fixed value still overflowed) — `/props` is now a fallback for a plain llama.cpp server with no `/running` at all. +- **Context floor too small for Claude Code to even start.** Fixing the overflow above surfaced a second, unfixable-by-injection failure: Claude Code's own system prompt and tool schemas cost roughly 36.4K tokens on their own (confirmed live via an `in:0 out:0` failure on the very first message), which can exceed a small model's entire real context before any conversation history exists to trim — no `CLAUDE_CODE_MAX_CONTEXT_TOKENS` value fixes that, since it only governs when history gets compacted. Applying such a model now returns a warning (gated on the CLI registry declaring a `contextLengthVar`, so it's a no-op for every other harness) instead of launching straight into a guaranteed first-message failure, and the Run-menu picker shows it as an in-app dialog naming the model, its discovered context and the ~40K safe floor, with the actual fix spelled out: give the model an explicit larger `-c`/`--ctx-size` in llama-swap's config instead of relying on auto-fit, which optimizes for the biggest model that fits rather than the biggest context. "Launch anyway" is still one click away. +- **The real root cause of "it still says opus, not my model."** llama.cpp runs exactly one model at a time; llama-swap unloads and reloads it on demand, which can take anywhere from a few seconds to well over a minute — long enough that a session mid-swap is indistinguishable from one that never left the native backend. Applying a selection now checks llama-swap's own `GET /running` first (feature-detected; a plain llama.cpp/OpenAI-compatible server has no such endpoint and is never checked); if switching would unload a model **another live session is actively using**, the apply is refused with a warning naming that session instead of silently switching, and a confirmation retry proceeds anyway. Either way, a sticky "loading model…" toast now covers the actual swap window until llama-swap reports the target model ready, so a prompt sent mid-swap reads as "loading," never as silence or an answer from whatever was loaded a moment before. + +- **Claude's whole first-run sequence, on every single launch.** A fresh, otherwise-empty `CLAUDE_CONFIG_DIR` isn't just missing the API-key approval above — Claude Code treats it as a brand-new profile and replays the theme picker, the security-notes screen, the per-project "trust this folder?" dialog, and (running bypassed) a one-time permissions-bypass warning, every time, confirmed live. None of that shows up again for a real, already-onboarded profile. `customModelInjection`'s new `skipFirstRunPrompts` (claude's entry only) pre-seeds that same "already been through this" state — `hasCompletedOnboarding` and this session's own project trust into the same `.claude.json` the API-key approval merges into, `skipDangerousModePermissionPrompt` into `settings.json` — so a custom-model launch reaches the conversation exactly as fast as a native cloud one, with nobody there to click through a wizard. + +Two more, from actually clicking through the swap-confirm and context-warning dialogs live: their z-index sat under the centred status banner, so a dialog could render fully hidden behind "Claude started — switching to llama-swap…"; and their Cancel/confirm buttons stacked instead of sitting side by side (`.btn-toolbar`'s own `display: flex` needs a row-layout parent it never had). Both dialogs now clear the banner and lay their buttons out centred, side by side. + +- **A session's model getting silently swapped out later, not just at launch.** The conflict check above only ever runs at the moment a session is created or a model applied — confirmed live: a second Codex session picking a different model launched with no warning at all, because nothing conflicted at that exact instant, yet it silently evicted the first session's model regardless (llama.cpp runs one model at a time). There was no mechanism to catch a swap caused by a DIFFERENT session's own later, ordinary use. A new periodic sweep (`detectCustomModelSwapDisplacements`, every 20s, one `GET /running` per distinct endpoint with a live custom-model session) now compares each such session's own model against what's actually loaded, and a new `custom-model:swapped-out` SSE event drives a global toast naming the displaced session and what's now loaded instead — so you find out before typing into a session that's about to trigger yet another reload. Notifies once per displacement, clearing once a session's own model is loaded and ready again so a later, genuinely new displacement notifies again. + +- **The loading banner's second line is now the real backend log line, not just a countdown.** llama-swap's `GET /api/events` SSE stream carries the actual `llama-server` process's own stdout (`load_model: loading model ''`, `llama_server: model loaded`, tokenizer warnings, all of it) tagged `source: "upstream"`, distinct from llama-swap's own `source: "proxy"` request-access lines — confirmed live end-to-end through a real forced swap, and it correctly stays on the last thing llama.cpp said once the load goes quiet rather than clearing to blank. ⚠️ This feature's own first cut targeted `GET /logs` instead (the name that suggested it) and shipped a live-tested implementation against it before this live check caught that `/logs` carries ONLY the proxy request log and never once showed a single backend line, even seconds after a real, confirmed swap — corrected before merge, not after. + +Remote (SSH) and Docker sessions are refused for now (400) — their restart reattaches the durable remote/in-container tmux rather than relaunching the agent. + +- **The loading banner's countdown is gone, replaced by a generic disclaimer and a Cancel button.** Its size-scaled expected-time estimate and matching auto-timeout were both a guess dressed up as a fact — real load time depends on hardware this feature has no way to know, and a fixed number could kill a genuinely slow load partway through. The banner now says "this can take a while depending on your hardware and the model size", polls indefinitely, and carries a **Cancel** button that ends the wait and closes the session on the user's own call rather than a guessed deadline. + +**One more, from watching it launch live: opencode, Codex, Gemini, Pi, Grok, DeepSeek and OMP now launch directly on the endpoint, with no restart at all.** Picking one of these seven from the Run-menu picker used to launch natively first, wait for it to settle, then restart it in place with the endpoint applied — a deliberate two-step design, but visibly a native boot immediately followed by a second one, worst on a CLI whose TUI fully reinitializes on a restart (confirmed live on Codex). `POST /api/quick-start` now accepts a `customModel` field and computes the same injection _before_ the session exists, launching straight onto the endpoint the first time — no visible relaunch, and it also runs the same llama-swap conflict check (warns before unloading a model another live session is using) at create time. Claude still uses the original launch-then-restart path for now (its own `--resume`-based restart is far less jarring, and `runClaude()`'s multi-tab and docker-config-drift-retry logic make folding it into the one-shot path separate work). diff --git a/CLAUDE.md b/CLAUDE.md index 4ef118e5f..fb83c9f54 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -227,7 +227,9 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **DeepSeek web UI** (`POST`/`GET`/`DELETE /api/deepseek/web`, `deepseek-web-server.ts`): the Run menu's "DeepSeek web UI..." entry supervises ONE background `dsh web` child process, deliberately **NOT a shell session**. The session version worked and was still wrong in use: it put a terminal tab on screen next to the web tab the user actually asked for, every single time, and nothing about a long-lived HTTP server needs to be a tab. ⚠️ What a session gave for free now has to be paid for explicitly, and every piece is load-bearing: **exactly one** server (a second click REUSES it rather than racing it for a port, which two sessions structurally could not do), **restarted when the browser authority changes** (`--trusted-host` fences dsh's `/api` against the browser authority, and a Codeman reachable at both loopback and a tailnet name has two, so whoever asks last wins: the asker is by definition the origin about to load the page), **killed on shutdown** (`stopDeepSeekWeb()` in the server teardown, because the child is detached so its whole plugin tree can be signalled at once, which also means it would OUTLIVE Codeman and hold its port against the next start), and **failures returned to the caller**, since with no tab there is nowhere for a stack trace to land. ⚠️ The port search starts at dsh's own default 3080 and walks 40, never fixed: that default is precisely the port most likely to be taken already by the user's own `dsh web`, and hardcoding it killed this feature with EADDRINUSE once. Free-port detection BINDS rather than connects (a connect probe cannot tell "free" from "listening but not answering yet"), so it is racy by nature and the caller still waits for the server to really answer before reporting success. ⚠️ Both `POST` and `DELETE` sit at the **same privilege bar as the profile installer** (`canUsernameRunPrivilegedCommands`) even though the action reads as "open a page": booting a dsh profile executes the plugin code in it, and the server is a single shared instance, so stopping it in multi-user mode takes it out from under other users' tabs. -**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `docs/custom-model-endpoints-plan.md`; backend + HTTP API only until the Run-menu picker lands, and the setting is read by nothing yet): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET /v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×. ⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified. ⚠️ Applying a selection **restarts the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` kills a WORKING pane, so a CLI whose launch declares a `fallback` chain (claude) gets the live conversation id pinned as `resumeSessionId` for that one respawn: `--session-id ` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume || --session-id ` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/` for pi/omp, grok's `[model.codeman-custom]` block name): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600. ⚠️ **Security**: every env var this feature can redirect (`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, `GROK_HOME`, `HOME` for pi/omp, `OPENCODE_CONFIG_CONTENT`, etc.) is in that CLI's `privilegedEnvKeys` — several of these were reachable via the generic `envOverrides` field's prefix allowlist BEFORE this feature existed (the env allowlist is global and prefix-based, not per-CLI-scoped), so building this surfaced and closed a pre-existing gap rather than opening a new one. `ANTHROPIC_*` is deliberately NOT in claude's `allowedPrefixes` at all — Anthropic-traffic redirection can only happen through this feature's own admin-configured, SSRF-guarded route, never a plain client-supplied `envOverrides`. **Confidence, verified end-to-end against a real llama-swap server via the DYNAMIC `scripts/test-local-llm-harnesses.ts`** (reads the live CLI registry, so a registry change needs zero script edits): claude/opencode/pi/grok/omp **PASS**; codex config structure is correct but codex only speaks the Responses API since Feb 2026, which llama.cpp/llama-swap don't implement — a confirmed protocol gap, not a bug; gemini fails with `Invalid auth method selected` (an undocumented `GATEWAY` AuthType gemini-cli selects once `GOOGLE_GEMINI_BASE_URL` is set — unresolved after real investigation); deepseek reaches the server but gets a consistent `HTTP_404` (root cause not identified); antigravity has no known mechanism at all. See the confidence table in `docs/custom-model-endpoints-plan.md` for the full detail on each. +**Custom Model Endpoint Profiles** (opt-in, `customModelEndpointsEnabled`, SYNCED, default OFF; `docs/custom-model-endpoints.md`, design doc `docs/custom-model-endpoints-plan.md`; full stack — settings-panel CRUD + the Run-menu picker, on top of the backend below): points a session at a user-configured custom OpenAI-compatible endpoint — local (llama.cpp, DGX Spark, Strix Halo) or cloud (Azure AI Foundry, OpenRouter) — instead of its harness's native cloud backend. Endpoints are a read/write-array store (`custom-model-hosts.ts`, `~/.codeman/custom-model-hosts.json`) discovered via `GET /v1/models`; `CustomModelHost.authStyle` is `'bearer'` (default, `Authorization: Bearer`) or `'api-key'` (Azure's convention) — **never both**, live-tested against a real server: sending both headers on one request reliably hangs it indefinitely, reproduced 3×. ⚠️ The actual per-CLI redirect is `capabilities.customModelInjection` on the CLI registry (four kinds: `env` for claude/gemini/deepseek, `configContentEnv` reusing opencode's existing `OPENCODE_CONFIG_CONTENT`, `configDir` for codex/pi/grok/omp — writes an isolated per-session config file, NEVER the user's real `~/.codex`/`~/.pi`/`~/.omp`/grok config — and `unsupported` for antigravity, which has no known mechanism), computed by the pure `custom-model-injection.ts` (mirrors `session-cli-builder.ts`'s no-IO discipline). ⚠️ `PI_CONFIG_DIR` does NOTHING for pi or omp (grepped pi's entire bundled JS source — the string appears nowhere); both hardcode `~/.pi/agent/models.json` / `~/.omp/agent/models.yml` with no dedicated override, so the real redirect for both is the child process's own **`HOME`**, and both need `models` as an ARRAY of `{id}` objects (an object keyed by id silently loads zero models). Grok's real mechanism turned out to be a `config.toml` `[model.]` block redirected via `GROK_HOME` — its original env-var-based recipe was flat-out wrong (produced "Not signed in" against a real binary), not just unverified. ⚠️ **Two launch paths, chosen by mechanism, not preference — see the second paragraph below for why**: opencode/codex/gemini/pi/grok/deepseek/omp apply the selection ONE-SHOT, before the session/process ever exists, with no restart at all; claude alone still applies a selection by **restarting the session's CLI process in place** via `Session.restartCli()` — a de-restricted `reattachRemote()` reusing the same `respawn-pane -k` primitive local/remote respawns already share — because every one of these harnesses reads its endpoint config at process start, never per-turn, so there is no live hot-swap; `Session.setCustomModel()` undoes the PREVIOUS selection's env keys (and deletes its old `configDir`) before merging the new ones in, so switching endpoints or clearing back to native cloud never leaves a stale key behind. ⚠️ Deleting a key from `_envOverrides` is NOT enough on its own: `tmux setenv` persists at the tmux-session level and is inherited by `respawn-pane` (measured: `setenv FOO bar` survived two successive `respawn-pane -k`), so the retired keys are queued (`_pendingEnvUnsets`) and ride `RespawnPaneOptions.unsetEnvKeys` into `applyEnvOverrides()`, which `setenv -u`s them BEFORE re-applying the live overrides. ⚠️ `restartCli()` kills a WORKING pane, so a CLI whose launch declares a `fallback` chain (claude) gets the live conversation id pinned as `resumeSessionId` for that one respawn: `--session-id ` refuses an id that already has a transcript (`Session ID ... is already in use`), and without the `--resume || --session-id ` shape the docker/remote pane commands already use, applying a model killed the pane and lost the session. ⚠️ pi, omp and grok need the config file AND a `model` launch param (`custom/` for pi/omp, grok's `[model.codeman-custom]` block name): that is the registry's `customModelInjection.launchModel` template, applied onto the respawn options through `legacyConfigField` by `_withCustomModelLaunchModel()`, never by id, and a model id the CLI's `model` token pattern cannot carry is refused with a 400 rather than silently dropped by the argv engine. ⚠️ Remote (SSH) and Docker sessions are REFUSED (400): their `restartCli()` reattaches a durable tmux rather than restarting the agent and the env lands on the local pane, so they used to report `restarted:true` and change nothing. The selection survives a Codeman restart as the disk-only `__customModel` (bookkeeping: env KEYS, config dir, launch model; never the values, which carry the API key and are re-derived from the endpoint store on recovery), the config dir is removed with the session, and every secret-bearing file (`custom-model-hosts.json`, the per-session config dir) is written 0600. ⚠️ **Security**: every env var this feature can redirect (`ANTHROPIC_BASE_URL`, `GOOGLE_GEMINI_BASE_URL`, `CODEX_HOME`, `GROK_HOME`, `HOME` for pi/omp, `OPENCODE_CONFIG_CONTENT`, etc.) is in that CLI's `privilegedEnvKeys` — several of these were reachable via the generic `envOverrides` field's prefix allowlist BEFORE this feature existed (the env allowlist is global and prefix-based, not per-CLI-scoped), so building this surfaced and closed a pre-existing gap rather than opening a new one. `ANTHROPIC_*` is deliberately NOT in claude's `allowedPrefixes` at all — Anthropic-traffic redirection can only happen through this feature's own admin-configured, SSRF-guarded route, never a plain client-supplied `envOverrides`. **Confidence, verified end-to-end against a real llama-swap server via the DYNAMIC `scripts/test-local-llm-harnesses.ts`** (reads the live CLI registry, so a registry change needs zero script edits): claude/opencode/pi/grok/omp **PASS**; codex config structure is correct, and codex only speaks the Responses API since Feb 2026 (`wire_api = "responses"`) — re-verified live against a llama-swap deployment that DOES answer `/v1/responses` (an earlier test's harder failure against a different deployment does not reproduce everywhere): a plain, no-tool-call chat turn gets a real reply, but a real tool-call attempt came back as `agent_message` TEXT (the tool-call JSON printed as the answer) rather than an executable `function_call` item — confirmed via `codex exec --json`'s raw event stream. Tool execution is what makes codex a coding agent, so it remains not usable for real work either way, just with a more precise failure mode than a flat protocol break; gemini fails with `Invalid auth method selected` (an undocumented `GATEWAY` AuthType gemini-cli selects once `GOOGLE_GEMINI_BASE_URL` is set — unresolved after real investigation); deepseek's originally-reported `HTTP_404` is root-caused and fixed — its bundled `@deepseek-ai/dsh-llm-deepseek` module builds `${DEEPSEEK_BASE_URL}/chat/completions` with no `/v1` of its own (confirmed by installing the real package and reading its source), so a new `appendV1Suffix` flag on its registry entry (alone — claude/gemini must not get it) runs `endpoint.baseUrl` through `withV1Suffix()` before writing it, live-confirmed against llama-swap (`.../chat/completions` 404s, `.../v1/chat/completions` succeeds) though not yet re-run through an actual `dsh` binary, which isn't installable in this environment; antigravity has no known mechanism at all. See the confidence table in `docs/custom-model-endpoints-plan.md` for the full detail on each. ⚠️ **The Run-menu picker generates entries from `window.__codemanCustomModelClis`** (`server.ts`, injected at page render from `enabledClis().filter(kind==='agent' && customModelInjection.kind!=='unsupported')`, JSON-escaped against a literal `` via the exported `escapeScriptJson()` since `label` is a user-`clis.json`-settable string unlike the neighbouring booleans-only `__codemanCliAvailable`), never a hardcoded per-CLI id list in the frontend — the same "no branching on CLI id outside stock.ts" discipline the registry itself enforces. One entry per (capable, INSTALLED CLI, saved endpoint) pair, e.g. "Claude Code (llama.cpp)", filtered through `isCliAvailable()` like the stock entries. Clicking one calls `selectCustomModelEntry(mode, endpointId)` (`session-ui.js`), which re-fetches the endpoint (never trusts anything cached from the dropdown's render — the 5-minute sweep below or a settings edit may have changed it since) and decides the model: exactly one discovered model launches straight away, two or more open `#customModelPickModal` to ask, with `defaultModelId` marked but never auto-chosen (asking exists so ONE launch can deliberately differ from the saved default). Either way the actual launch (`runCustomModelEntry`) routes through `run()` itself via a temporary `_runMode` swap — never `setRunMode()`, which would persist it as the user's new default — rather than a parallel dispatch table, which is what gives a custom-model launch the same `_runInFlight` lock every other Run click gets and means a CLI whose injection recipe lands later needs no update here. It then GETs `/api/sessions/:id/wait?until=idle&timeout=20000` on that session BEFORE applying — measured live, a freshly launched CLI reports itself `busy` for its own startup (boot spinner, workspace-trust check) well before the apply call would otherwise reach it, and the apply route's `isBusy()` guard correctly can't tell that apart from a real turn in progress, so every fresh launch failed with `SESSION_BUSY` until this wait was added. A timeout there is a normal 200 per the wait endpoint's own contract, never an error, so a session still busy after 20s just reaches the apply call anyway and gets that route's own honest error. It then calls `POST /api/sessions/:id/custom-model` on the session `run()` produced, guarded by snapshotting `activeSessionId` before the call and requiring it to have actually changed after — every `run*()` handles its own failure internally and returns normally rather than throwing, so a declined/failed launch must not silently re-point and restart whatever session was already open. ⚠️ The apply call reads the response body itself (`_api()`) rather than `_apiJson()`, which unwraps success but silently discards a failure body — losing the one thing (`error`) that distinguishes "still busy", "not a discovered model", "remote/Docker session" and everything else the route can report; the resulting toast is `type: 'error'`, which `showToast()` now defaults to STICKY (no auto-dismiss, an explicit close button) precisely so a message worth diagnosing survives long enough to be read — a 3s default hid the real reason behind every one of these failures until it was fixed. Entries are hidden for a remote/docker active case (the apply route refuses both) and for an endpoint with no discovered models at all (nothing to launch with). ⚠️ **Every saved endpoint's models also re-discover themselves automatically**, a `this.cleanup.setInterval` in `server.ts` (`CUSTOM_MODEL_REDISCOVER_INTERVAL_MS`, 5 minutes, off under `testMode` like the Codex plan-usage poll beside it) calling the exported `refreshAllCustomModelHosts()` (`custom-model-routes.ts`) — one endpoint unreachable on a cycle never blocks the others, and a read-modify-write PER HOST (re-reading the store before each splice, keyed by id) means an admin's concurrent edit or delete wins over a sweep that started before it, never the reverse. + +**Everything below landed after the initial backend + picker cut, each confirmed live against a real llama-swap deployment.** ⚠️ **llama-swap runs one model at a time, and switching can disrupt ANOTHER live session** — before applying, both apply routes call llama-swap's own `GET /running` (feature-detected via `getLlamaSwapStatus()`, `custom-model-routes.ts`; a plain llama.cpp/OpenAI-compatible server has no such endpoint and is simply never checked). If a different model is loaded and ready AND another live session's own selection is using it, the apply returns `{requiresConfirmation, currentlyLoadedModel, affectedSessions}` instead of switching silently; retrying with `confirmed: true` skips the check, and switching with nothing else affected proceeds immediately. llama-swap also has no dedicated "switch model" endpoint — the only thing that actually starts a swap is a real inference request naming the model (confirmed live: applying a selection alone never reached llama-swap's own logs, since nothing had asked it to load anything) — so both routes also fire `triggerLlamaSwapLoad()`, a fire-and-forget `POST /v1/chat/completions` with `max_tokens: 1`, whenever the target model isn't already loaded and ready. ⚠️ **That launch-time check cannot catch a swap caused by a DIFFERENT session's LATER, ordinary use** — confirmed live: a second Codex session picking a different model launched with no warning at all (nothing conflicted at that exact instant), yet it silently evicted the first session's model regardless, since llama-swap has no push notification of its own. `detectCustomModelSwapDisplacements()` (`custom-model-routes.ts`) is a separate periodic sweep (`server.ts`, `CUSTOM_MODEL_SWAP_CHECK_INTERVAL_MS` = 20s) that compares each live custom-model session's own `modelId` against what `/running` actually reports loaded, broadcasting a `custom-model:swapped-out` SSE event — shown as a global toast, never tied to the displaced session's own tab, since the whole point is telling the user before they type into it — the first time a mismatch appears, via a caller-owned de-dupe `Set` cleared once that session's own model is loaded and ready again so a later, genuinely new displacement notifies again rather than staying silently un-notified forever after the first one. ⚠️ **Context length is read from the REAL launch command, never `/props`** — `/props?model=`'s `default_generation_settings.n_ctx` was confirmed live to report a `--fit-ctx`-launched backend's theoretical/trained maximum rather than the real runtime-configured size (a measured 154112-vs-16384 discrepancy, caught only because the unfixed value still overflowed), so discovery parses the actual configured size straight out of `/running`'s own `cmd` field instead (`parseCtxFromCmd`: `--fit-ctx ` first, then plain llama.cpp `-c`/`--ctx-size`), falling back to `/props` only when `cmd` states no recognizable flag at all. ⚠️ **Claude alone gets a context-window FLOOR check, on top of the ceiling `contextLengthVar` already fixes** — `exceedsSafeContextFloor()` (gated on the registry declaring `contextLengthVar`, so a no-op for every other CLI by construction) compares a model's discovered context against `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000): confirmed live, twice, that Claude Code's own system prompt and tool schemas cost roughly 36.4K tokens on the very first message, before any conversation history exists to compact, so a smaller real context fails outright regardless of what `CLAUDE_CODE_MAX_CONTEXT_TOKENS` says (that var only controls when HISTORY gets compacted, and there is none yet on message one). Below the floor, the apply returns `{requiresContextWarning, modelId, contextLength, minSafeContextTokens}` instead of launching, shown as an in-app dialog naming the actual fix: give the model an explicit larger `-c`/`--ctx-size` in llama-swap's config instead of relying on `--fit-ctx` auto-fit, which optimizes for the biggest MODEL that fits rather than the biggest CONTEXT. ⚠️ **A fresh, isolated `CLAUDE_CONFIG_DIR` looks like a brand-new Claude Code profile and replays its ENTIRE first-run sequence on every launch** — the theme picker, the security-notes screen, the per-project "trust this folder?" dialog, and (running with a bypass-permissions flag) a one-time warning about it, confirmed live, none of which a real, already-onboarded profile shows again. `skipFirstRunPrompts` (claude's entry only, requires `apiKeyTrustFile` since it reuses the same file) pre-seeds that same "already been through this" state: `hasCompletedOnboarding` and this session's own `projects[workingDir].hasTrustDialogAccepted` merge into the same `.claude.json` the API-key trust file already writes to, and `skipDangerousModePermissionPrompt` merges into `settings.json` (a different file, same corrupt-tolerant merge). ⚠️ **The loading banner shows the REAL backend log line, not a guess, and has no countdown or auto-timeout at all.** `getLatestLlamaSwapLogLine()` holds one `GET /api/events` SSE connection open per endpoint (confirmed live to stay open indefinitely — read past 220KB over 8s with no `done`; idle-closed after 30s via `pruneIdleLlamaSwapLogTails`, same 20s sweep as the swap-displacement check above), parsing `logData` frames and keeping only `source: "upstream"` (the real `llama-server` process's own stdout) lines, never `source: "proxy"` (llama-swap's own request-access log). ⚠️ `GET /logs` — the endpoint this feature's own first cut targeted, since the name suggested it — was confirmed live to carry ONLY the proxy log and never a single backend line, even seconds after a real, verified model swap; caught and corrected by a live check before merge, not after. The banner itself dropped its size-scaled expected-time estimate and matching auto-timeout (a guess dressed up as a fact that could kill a genuinely slow load partway through on slower hardware) for a generic hardware/model-size disclaimer plus a user-driven **Cancel** button (`_showCenterStatus`'s `onCancel` option, a real button distinct from the plain "×" close glyph an `'error'`-type banner gets) that ends the wait and closes the session on the user's own call rather than a guessed deadline. **Run launch synchronization**: the Run entrypoint holds an in-flight lock and disables `#runBtn` for the whole launch (≥500ms), so a double click cannot create duplicate sessions with the same `w-` name. `_ensureCreatedSessionVisible()` runs before `selectSession()`, and `_onSessionCreated()` stays an idempotent upsert, so POST-first and SSE-first ordering both produce exactly one rendered tab. ⚠️ **Closing has the mirror-image race and one owner**: `closeSession()` reads `wasActive` BEFORE its `await` and announces the delete via `_closingSessions`, while `_onSessionDeleted` skips the active-session handoff for an id in that set. Both used to read `activeSessionId` after the fact, so the `session_deleted` broadcast for your own delete could null it first and closing the tab you were on landed on the welcome screen instead of the next session, on the same build, depending on timing. The fallback also picks the first order entry that is still in `sessions` (a dead id can linger in `sessionOrder`, same reason Alt+N indexes a live-filtered list). A delete from ANOTHER client still shows the welcome screen, which is the honest answer when what you were looking at was taken away. Tests: `test/session-close-fallback.test.ts`. → [architecture-invariants#run-launch-synchronization](docs/architecture-invariants.md#run-launch-synchronization) diff --git a/docs/api-reference.md b/docs/api-reference.md index dcd0d631a..00d60d8f9 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -66,17 +66,17 @@ The single source of truth is `ErrorStatus` / `httpStatusForErrorCode()` in `src/types/api.ts`. Clients should branch on `errorCode` (stable) and may rely on the HTTP status. -| `errorCode` | HTTP | Meaning | -|-------------|------|---------| -| `INVALID_INPUT` | 400 | Malformed request / failed validation | -| `UNAUTHORIZED` | 401 | Authentication required or failed | -| `NOT_FOUND` | 404 | Resource does not exist | -| `SESSION_BUSY` | 409 | Session is busy | -| `CONFLICT` | 409 | Conflicts with current state (e.g. already running) | -| `ALREADY_EXISTS` | 409 | Resource already exists | -| `OPERATION_FAILED` | 422 | Well-formed but could not be completed | -| `RATE_LIMITED` | 429 | Too many requests | -| `INTERNAL_ERROR` | 500 | Unexpected server error | +| `errorCode` | HTTP | Meaning | +| ------------------ | ---- | --------------------------------------------------- | +| `INVALID_INPUT` | 400 | Malformed request / failed validation | +| `UNAUTHORIZED` | 401 | Authentication required or failed | +| `NOT_FOUND` | 404 | Resource does not exist | +| `SESSION_BUSY` | 409 | Session is busy | +| `CONFLICT` | 409 | Conflicts with current state (e.g. already running) | +| `ALREADY_EXISTS` | 409 | Resource already exists | +| `OPERATION_FAILED` | 422 | Well-formed but could not be completed | +| `RATE_LIMITED` | 429 | Too many requests | +| `INTERNAL_ERROR` | 500 | Unexpected server error | Adding a new error code is non-breaking; removing or renaming one is a major change. @@ -87,10 +87,10 @@ exist because SSE is Codeman's only other "tell me when" channel, and an agent driving the API from a shell tool cannot practically hold a stream and parse events inline. -| Call | Blocks until | -|------|--------------| -| `GET /api/v1/sessions/:id/wait` | one of a set of lifecycle signals fires | -| `GET /api/v1/sessions/:id/wait-output` | a literal string appears in the session's output | +| Call | Blocks until | +| --------------------------------------------- | -------------------------------------------------- | +| `GET /api/v1/sessions/:id/wait` | one of a set of lifecycle signals fires | +| `GET /api/v1/sessions/:id/wait-output` | a literal string appears in the session's output | | `POST /api/v1/sessions/:id/input` with `wait` | the input is delivered **and then** a signal fires | `POST .../input` with `wait` is not the same as a `POST` followed by a separate @@ -140,13 +140,13 @@ contract is a **marker unique to each call** (`MARK="DONE_$RANDOM"`, send ### Signals -| Signal | Source | Actually fires for | -|--------|--------|--------------------| -| `idle` | the session's own `idle` event | `claude`: yes, on ❯-prompt detection after activity. `shell`: **once only**, ~500 ms after start, and never again. External CLIs: not guaranteed (they render their own TUIs and readiness is output stabilization) | -| `working` | the session's own `working` event | `claude` only in practice (spinner and work-keyword detection are Claude output formats) | -| `stop` | the Claude Code `stop` hook, the definitive end-of-turn signal | `claude` only | -| `blocked` | a `permission_prompt` or `elicitation_dialog` hook | `claude` only, and rarer than it looks: see below | -| `exit` | no process is behind the session | every mode | +| Signal | Source | Actually fires for | +| --------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `idle` | the session's own `idle` event | `claude`: yes, on ❯-prompt detection after activity. `shell`: **once only**, ~500 ms after start, and never again. External CLIs: not guaranteed (they render their own TUIs and readiness is output stabilization) | +| `working` | the session's own `working` event | `claude` only in practice (spinner and work-keyword detection are Claude output formats) | +| `stop` | the Claude Code `stop` hook, the definitive end-of-turn signal | `claude` only | +| `blocked` | a `permission_prompt` or `elicitation_dialog` hook | `claude` only, and rarer than it looks: see below | +| `exit` | no process is behind the session | every mode | `stop` is the signal to orchestrate on where it exists; `idle` is a heuristic fallback that can flap mid-turn when a spinner pauses. The default set when `until` @@ -156,12 +156,12 @@ can no longer happen). On a `claude` worker, prefer an explicit `until=stop,exit once the session is up: the default set's `idle` also resolves on a spinner pause, and on a fresh session the **startup** `idle` (emitted when the CLI first comes up) can land inside your first wait window and report a turn that never ran. Measured: -a session parked on the trust dialog emits no *further* `idle`, so it is the +a session parked on the trust dialog emits no _further_ `idle`, so it is the startup transition, not the dialog, that produces the false success below. ⚠️ **`exit` means "nothing is running", which includes "not started yet".** The server answers from `pid === null` plus a mux-layer pane-death probe, and that -covers a session that exited — including a worker that died *inside* its tmux pane +covers a session that exited — including a worker that died _inside_ its tmux pane while the local attach client (and therefore `pid`) lives on — one that was detached, and one that was **created but never started**. So the first wait after `POST /api/v1/sessions` returns `{"signal":"exit","immediate":true}` in @@ -184,7 +184,7 @@ blocked, and polling `blocked` alone will sit at its timeout. ⚠️ **On a `shell` session, only `exit` and marker-matching are dependable.** A shell session emits its one `idle` at startup and then stays `status: "idle"` forever, -whatever the pane is doing, so it never emits a *transition*. Since send-and-wait +whatever the pane is doing, so it never emits a _transition_. Since send-and-wait requires a transition (and so does `fresh=1`), both can only time out there: a documented default `wait` on a shell worker running `sleep 4` times out at the full 25 s. Synchronize hook-less sessions with `wait-output` and a unique marker @@ -218,11 +218,11 @@ with `from=buffer` keeps matching long after the dialog is gone. A worked versio ### `GET /api/v1/sessions/:id/wait` -| Param | Type | Default | Notes | -|-------|------|---------|-------| -| `until` | comma-separated list of `idle,working,stop,blocked,exit` | `stop,idle,exit` | resolves on the first to fire. An unknown token is a `400` naming it, never a silent fallback | -| `timeout` | positive integer ms | `60000` | **validated first, clamped second.** `0`, a negative value and a fractional value are all `400`s, not clamps; a valid value outside `[1000, 600000]` is clamped and echoed as `wait.timeoutMs` | -| `fresh` | `0` \| `1` \| `false` \| `true` | `0` | `1` requires an actual transition, ignoring the state at call time | +| Param | Type | Default | Notes | +| --------- | -------------------------------------------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `until` | comma-separated list of `idle,working,stop,blocked,exit` | `stop,idle,exit` | resolves on the first to fire. An unknown token is a `400` naming it, never a silent fallback | +| `timeout` | positive integer ms | `60000` | **validated first, clamped second.** `0`, a negative value and a fractional value are all `400`s, not clamps; a valid value outside `[1000, 600000]` is clamped and echoed as `wait.timeoutMs` | +| `fresh` | `0` \| `1` \| `false` \| `true` | `0` | `1` requires an actual transition, ignoring the state at call time | ```bash curl -s "$API/api/v1/sessions/$SID/wait?until=stop,exit&timeout=60000" @@ -239,12 +239,12 @@ a plain signal wait, so check the endpoint path before blaming the parameters. ### `GET /api/v1/sessions/:id/wait-output` -| Param | Type | Default | Notes | -|-------|------|---------|-------| -| `match` | literal string, 1 to 200 chars | required | substring match against the PTY stream with ANSI escapes stripped. A match spanning two PTY chunks is found | -| `nocase` | `0` \| `1` \| `false` \| `true` | `0` | case-insensitive compare. The returned snippet keeps the terminal's original casing | -| `from` | `now` \| `buffer` | `now` | `buffer` scans the tail of the existing terminal buffer (bounded, 256 KB by default) before blocking | -| `timeout` | positive integer ms | `60000` | same validation and clamp as `/wait` | +| Param | Type | Default | Notes | +| --------- | ------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------- | +| `match` | literal string, 1 to 200 chars | required | substring match against the PTY stream with ANSI escapes stripped. A match spanning two PTY chunks is found | +| `nocase` | `0` \| `1` \| `false` \| `true` | `0` | case-insensitive compare. The returned snippet keeps the terminal's original casing | +| `from` | `now` \| `buffer` | `now` | `buffer` scans the tail of the existing terminal buffer (bounded, 256 KB by default) before blocking | +| `timeout` | positive integer ms | `60000` | same validation and clamp as `/wait` | **Matching is literal, never a pattern.** A `regex` parameter is rejected with a `400` rather than ignored, so a caller that assumed otherwise finds out immediately @@ -296,10 +296,10 @@ hand-written query string decodes to a space. Two optional fields on the existing endpoint: -| Field | Type | Notes | -|-------|------|-------| -| `wait` | `true` or the same comma grammar as `until` | `true` means the default signal set. Omitted keeps the historical fire-and-forget behavior, unchanged. `null`, `false` and an empty string are all read as **absent**, not as an error and not as "wait for the default" | -| `waitTimeout` | positive integer ms | same validation **and** clamp as `timeout`: `0`, a negative and a fractional value are `400`s, anything valid is clamped into `[1000, 600000]` and echoed as `wait.timeoutMs` | +| Field | Type | Notes | +| ------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `wait` | `true` or the same comma grammar as `until` | `true` means the default signal set. Omitted keeps the historical fire-and-forget behavior, unchanged. `null`, `false` and an empty string are all read as **absent**, not as an error and not as "wait for the default" | +| `waitTimeout` | positive integer ms | same validation **and** clamp as `timeout`: `0`, a negative and a fractional value are `400`s, anything valid is clamped into `[1000, 600000]` and echoed as `wait.timeoutMs` | Both are `nullish`, so an explicit `null` from `JSON.stringify` is accepted as "absent" rather than failing validation. That is deliberate: `.optional()` would @@ -330,16 +330,24 @@ All three nest the wait result under `data.wait`, so one client helper works aga any of them: ```json -{ "success": true, "data": { - "sessionId": "28325fd3-caa7-4178-82bf-87dfebf0f464", - "status": "idle", - "limitPaused": false, - "wait": { - "signal": "stop", "until": ["stop", "idle", "exit"], - "timedOut": false, "immediate": false, "ended": false, "aborted": false, - "waitedMs": 8421, "timeoutMs": 60000 +{ + "success": true, + "data": { + "sessionId": "28325fd3-caa7-4178-82bf-87dfebf0f464", + "status": "idle", + "limitPaused": false, + "wait": { + "signal": "stop", + "until": ["stop", "idle", "exit"], + "timedOut": false, + "immediate": false, + "ended": false, + "aborted": false, + "waitedMs": 8421, + "timeoutMs": 60000 + } } -}} +} ``` `POST .../input` returns the same `wait` object alongside `delivered`, `duplicate`, @@ -353,21 +361,21 @@ redelivery (harmless, the turn it refers to may be long over), while with client that reads `delivered === false` as "duplicate" silently treats a failed send as a success. -| Field | Type | Meaning | -|-------|------|---------| -| `wait.signal` | signal \| `null` | the signal that fired (`/wait` and `/input` only) | -| `wait.until` | array of signals | what the server actually waited on, after narrowing the default set for the session's mode (`/wait` and `/input` only) | -| `wait.matched` | boolean | the string appeared (`/wait-output` only) | -| `wait.match` | string | the literal that was searched for (`/wait-output` only) | -| `wait.snippet` | string \| `null` | bounded window of output around the match, blank runs collapsed for readability (`/wait-output` only) | -| `wait.timedOut` | boolean | the wait hit its timeout. Still a `200` | -| `wait.immediate` | boolean | the condition already held at call time, so nothing was waited for (`waitedMs` is 0) | -| `wait.ended` | boolean | the session went away (deleted or torn down) before the condition was met | -| `wait.aborted` | boolean | the client hung up, so the waiter was released without resolving — and by that definition a client never reads `true`. When the **server** abandons a wait itself (send-and-wait against a session with no PTY), it answers in about a millisecond with `ended: true`, `delivered: false`, `duplicate: false` and `aborted: false`: `delivered`/`ended` carry that story, and `aborted` stays the transport flag. Present for completeness; treat a `true` as "this wait answered nothing", never as an outcome | -| `wait.waitedMs` | number | wall-clock ms actually spent waiting | -| `wait.timeoutMs` | number | the timeout **after clamping**, which is what was applied | -| `status` | `SessionStatus` | the session's status after the wait, so a caller that timed out still learns where things stand | -| `limitPaused` | boolean | the session is paused on a usage limit and will emit nothing until its reset, so a timeout here is expected rather than a stall worth retrying hard | +| Field | Type | Meaning | +| ---------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `wait.signal` | signal \| `null` | the signal that fired (`/wait` and `/input` only) | +| `wait.until` | array of signals | what the server actually waited on, after narrowing the default set for the session's mode (`/wait` and `/input` only) | +| `wait.matched` | boolean | the string appeared (`/wait-output` only) | +| `wait.match` | string | the literal that was searched for (`/wait-output` only) | +| `wait.snippet` | string \| `null` | bounded window of output around the match, blank runs collapsed for readability (`/wait-output` only) | +| `wait.timedOut` | boolean | the wait hit its timeout. Still a `200` | +| `wait.immediate` | boolean | the condition already held at call time, so nothing was waited for (`waitedMs` is 0) | +| `wait.ended` | boolean | the session went away (deleted or torn down) before the condition was met | +| `wait.aborted` | boolean | the client hung up, so the waiter was released without resolving — and by that definition a client never reads `true`. When the **server** abandons a wait itself (send-and-wait against a session with no PTY), it answers in about a millisecond with `ended: true`, `delivered: false`, `duplicate: false` and `aborted: false`: `delivered`/`ended` carry that story, and `aborted` stays the transport flag. Present for completeness; treat a `true` as "this wait answered nothing", never as an outcome | +| `wait.waitedMs` | number | wall-clock ms actually spent waiting | +| `wait.timeoutMs` | number | the timeout **after clamping**, which is what was applied | +| `status` | `SessionStatus` | the session's status after the wait, so a caller that timed out still learns where things stand | +| `limitPaused` | boolean | the session is paused on a usage limit and will emit nothing until its reset, so a timeout here is expected rather than a stall worth retrying hard | Read the outcome by discriminator, in this order: @@ -390,12 +398,12 @@ read the timeout as "the worker is wedged" and kill a session that was working f ### Errors -| `errorCode` | HTTP | When | -|-------------|------|------| -| `INVALID_INPUT` | 400 | unknown `until` / `wait` token; `stop` or `blocked` requested explicitly on a mode that installs no hooks (the message names the mode); `regex=` on `/wait-output`; `match` outside 1 to 200 chars; a non-numeric `timeout` | -| `NOT_FOUND` | 404 | no such session, or one this caller does not own | -| `SESSION_BUSY` | 409 | this session's waiter cap is full | -| `RATE_LIMITED` | 429 | a per-owner or process-wide waiter cap is full. Retry later; the session you named is not the problem | +| `errorCode` | HTTP | When | +| --------------- | ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `INVALID_INPUT` | 400 | unknown `until` / `wait` token; `stop` or `blocked` requested explicitly on a mode that installs no hooks (the message names the mode); `regex=` on `/wait-output`; `match` outside 1 to 200 chars; a non-numeric `timeout` | +| `NOT_FOUND` | 404 | no such session, or one this caller does not own | +| `SESSION_BUSY` | 409 | this session's waiter cap is full | +| `RATE_LIMITED` | 429 | a per-owner or process-wide waiter cap is full. Retry later; the session you named is not the problem | The two capacity codes are deliberately different. A process-wide cap reported as `SESSION_BUSY` would tell the caller to switch sessions, which cannot help. The @@ -446,9 +454,9 @@ Design: [`approvals-inbox-plan.md`](approvals-inbox-plan.md). - `GET /api/v1/approvals` → `{ approvals: ApprovalItem[] }`, oldest first, ownership-scoped in multi-user mode. `ApprovalItem`: `{ id, sessionId, - sessionName, kind: 'permission'|'question'|'idle', createdAt, toolName?, - toolSummary?, message?, cwd?, context?, options?: {n, label}[], - acknowledgedAt? }`. `context` is the ANSI-stripped visible pane frame; +sessionName, kind: 'permission'|'question'|'idle', createdAt, toolName?, +toolSummary?, message?, cwd?, context?, options?: {n, label}[], +acknowledgedAt? }`. `context` is the ANSI-stripped visible pane frame; `options` is present only when the dialog's numbered choices parsed confidently; `acknowledgedAt` marks an item a human has already looked at (see `/viewed` below) and tells clients not to re-arm its tab alert. Listing @@ -466,7 +474,7 @@ Design: [`approvals-inbox-plan.md`](approvals-inbox-plan.md). first, `422 OPERATION_FAILED` when the session refused input. - `POST /api/v1/approvals/:id/dismiss` removes the item without keystrokes. - `POST /api/v1/approvals/session/:sessionId/viewed` → `{ sessionId, - acknowledged: itemId | null }`. Marks the session's pending **idle** item as +acknowledged: itemId | null }`. Marks the session's pending **idle** item as seen by a human (the web UI calls it when you open the session's tab): the item stays pending and answerable, but stops arming the yellow tab alert on every client, including after a reload. Permission/question items are never @@ -491,7 +499,7 @@ user guide: [`readmymind.md`](readmymind.md). - `GET /api/v1/sessions/:id/intent` -> `{ intent: IntentProfile }` for the session's case. `IntentProfile`: `{ key, workingDir, updatedAt, goals, - recentPrompts: { ts, sessionId, text }[] }` (prompts oldest first, FIFO cap +recentPrompts: { ts, sessionId, text }[] }` (prompts oldest first, FIFO cap 50, each <= 500 chars). A case with nothing recorded answers an empty profile with `updatedAt: 0`; nothing is persisted by reads. - `PUT /api/v1/sessions/:id/intent` with `{ goals }` (<= 8192 chars, strict @@ -516,6 +524,105 @@ All four enforce session ownership in multi-user mode; a foreign session id answers `404 NOT_FOUND` (no existence leak), and profiles of two owners of the same directory are distinct by construction. +## Custom Model Endpoints + +Points a session's harness at a user-configured OpenAI-compatible endpoint — +local (llama.cpp, vLLM, DGX Spark) or cloud (Azure AI Foundry, OpenRouter) — +instead of its native cloud backend, gated by the opt-in +`customModelEndpointsEnabled` setting (default OFF). Endpoints are +machine-level infra, like remote/docker hosts: writes are admin-only in +multi-user mode. Design: [`custom-model-endpoints-plan.md`](custom-model-endpoints-plan.md); +user guide: [`custom-model-endpoints.md`](custom-model-endpoints.md). + +- `GET /api/v1/model-endpoints` -> `CustomModelHost[]`, an unwrapped bare + array like every other list route (still riding the standard `{success, +data}` envelope on the wire — unwrap it the same way). Answers `[]` for a + non-admin in multi-user mode. `apiKey` is never returned; `apiKeySet: +boolean` reports whether one is stored, so a client can render "unchanged + if left blank" without ever holding the real value. +- `POST /api/v1/model-endpoints` with `{ id, label, baseUrl, apiKey?, +authStyle?, defaultModelId? }` creates one. `id` must match + `^[a-zA-Z0-9_-]+$`; `authStyle` is `bearer` (default) or `api-key`, never + both (a real server hung indefinitely when sent both headers on one + request); `baseUrl` must be `http(s)`, carry no embedded credentials, and + is refused if it points at (or resolves to) a link-local or + cloud-metadata address. `409 ALREADY_EXISTS` on a duplicate id. +- `PUT /api/v1/model-endpoints/:id` updates one. An **absent** `apiKey` + keeps the stored one rather than clearing it — the client never receives + the real value to resend deliberately unchanged, so omission is the only + way to say "leave it alone"; there is no way to clear a key back to unset + this way. `defaultModelId`, when set, must be one of that endpoint's own + `models` (`400 INVALID_INPUT` otherwise). +- `DELETE /api/v1/model-endpoints/:id` removes one. +- `POST /api/v1/model-endpoints/:id/discover-models` fetches the endpoint's + own `GET /v1/models` and stores the result as `models`, updating + `lastDiscoveredAt`, plus (best-effort, only for a model llama-swap's own + response already reports loaded) `modelContextLengths` and `modelSizesGB`. + A `defaultModelId` that no longer appears in the fresh list is dropped + rather than carried forward invalid. Failures answer `502 OPERATION_FAILED` + with the underlying connection error, or a named egress refusal if the + resolved address turned out to be blocked. The same refresh also runs + automatically for every saved endpoint every 5 minutes in the background + (`refreshAllCustomModelHosts()`, `custom-model-routes.ts`, started from + `server.ts`), so there is no route for triggering "refresh all" — one + endpoint being unreachable on a cycle never blocks the others. +- `GET /api/v1/model-endpoints/:id/running-status` -> `{ isLlamaSwap, +running: [{model, state, cmd?}], logLine? }`, read-only, no admin gate + (any session owner who could already point a session at this endpoint can + equally ask what it currently has loaded). `isLlamaSwap` is + feature-detected via the endpoint's own `GET /running` — a plain + llama.cpp/OpenAI-compatible server has none and always answers `false`. + `logLine`, present only when `isLlamaSwap` is true, is the most recent + REAL backend `llama-server` process log line (`load_model: ...`, + `llama_server: model loaded`, etc.), sourced from the endpoint's own + `GET /api/events` SSE stream and filtered to `source: "upstream"` frames + only (never llama-swap's own `source: "proxy"` request-access log) — one + connection is held open per endpoint and reused across every poller, + idle-closed after 30s of nobody asking. This is what the Run-menu + picker's loading banner polls once a second while a model is loading. +- `POST /api/v1/sessions/:id/custom-model` with `{ endpointId, modelId, +confirmed? } | { clear: true }` applies (or clears) the session's + selection and **restarts the session's CLI process in place** — every + supported harness reads its endpoint config at process start, never per + turn, so there is no live hot-swap. (`POST /api/v1/quick-start`'s own + `customModel: { endpointId, modelId, confirmed? }` field is the + no-restart equivalent for a session that doesn't exist yet — see below.) + A Claude session resumes its existing conversation across the restart; + pi/omp/grok additionally get a forced `--model`/`-m` value, since for + those three the config file alone does not select it. `400 INVALID_INPUT` + for a remote (SSH) or Docker session — both restart their agent + differently under the hood, and applying to one would report success + while changing nothing. Two more responses replace the normal + `{customModel, restarted}` shape, neither an error — both require + retrying the same call with `confirmed: true` to proceed anyway, and + neither restarts or creates anything on the first ask: + - `{requiresConfirmation: true, currentlyLoadedModel, affectedSessions}` — + llama.cpp/llama-swap only runs one model at a time, and switching would + unload a model another **live session's own selection** is actively + using. Never returned for a plain (non-llama-swap) server, and never + just because a swap is needed at all — only when it would disrupt + someone else. + - `{requiresContextWarning: true, modelId, contextLength, +minSafeContextTokens}` — Claude Code's own fixed per-turn overhead + (system prompt + tool schemas) can exceed a small model's entire + discovered context on its own, before any conversation history exists + to compact, guaranteeing the very first message fails regardless of + `CLAUDE_CODE_MAX_CONTEXT_TOKENS`. Gated on the CLI registry declaring a + `contextLengthVar` (claude only today), so it never fires for another + harness. +- `POST /api/v1/quick-start`'s `customModel: { endpointId, modelId, +confirmed? }` field (alongside its normal `caseName`/`mode`/etc. body) + computes the same injection **before** the session exists and launches + directly on the endpoint — no restart, because there was never a + native-backend boot to restart away from. Runs the identical checks as + the dedicated route above (`requiresConfirmation`/`requiresContextWarning`, + same shapes, same `confirmed: true` retry), and is refused the same way + for a remote or Docker case. This is what the Run-menu picker uses for + opencode, Codex, Gemini, Pi, Grok, DeepSeek and OMP; Claude still uses the + dedicated restart route above (its `--resume`-based restart is far less + jarring than a full relaunch, and folding it into the one-shot path is + separate work — see `docs/custom-model-endpoints-plan.md`). + ## Voice dictation Browser dictation transcribed through this server's Claude Code login, i.e. the @@ -524,7 +631,7 @@ same speech-to-text service the CLI's own `/voice` mode uses. Gated on the synce [`claude-voice-plan.md`](claude-voice-plan.md). - `GET /api/v1/voice/status` -> `{ available, reason?, subscriptionType?, - expiresAt? }`. `reason` is `disabled` (setting off), `no-credentials` (nobody +expiresAt? }`. `reason` is `disabled` (setting off), `no-credentials` (nobody signed in to Claude Code on the server), `expired` (the access token elapsed; running any Claude session refreshes it) or `malformed`. The OAuth token itself is never returned by this or any other endpoint. diff --git a/docs/custom-model-endpoints-plan.md b/docs/custom-model-endpoints-plan.md index 85407adf7..5078f17b0 100644 --- a/docs/custom-model-endpoints-plan.md +++ b/docs/custom-model-endpoints-plan.md @@ -104,17 +104,17 @@ declared capability, never an `if (mode === 'claude')` branch. ## Per-CLI injection recipes (confidence-ranked) -| CLI | Mechanism | Confidence | -| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `claude` | Env vars: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, `ANTHROPIC_DEFAULT_SONNET_MODEL`/`_HAIKU_MODEL`/`_OPUS_MODEL` (all set to the chosen model/deployment name) | **Verified end-to-end** against a real llama-swap server — a real "hello world" reply came back. ⚠️ Non-interactive (`-p`) invocations also fire an async session-title-generation call that reuses `ANTHROPIC_DEFAULT_HAIKU_MODEL` and validates it against Claude Code's OWN internal recognized-model list, printing `[claude-code:unrecognized_model]` and, in `-p` mode, hanging the whole invocation rather than just warning. `--settings '{"autoTitle":false}'` does NOT stop this (confirmed); `--bare` does (the warning still prints, but the real prompt runs) — but `--bare` ALSO disables hooks, LSP, plugin sync, and CLAUDE.md auto-discovery, so it is only safe for the standalone one-shot test script, NEVER for a real interactive Codeman session (which depends on hooks for idle detection, trust-dialog auto-accept, etc. — see the External CLI modes section of CLAUDE.md). Whether an INTERACTIVE claude session with a custom model hits the same hang (vs. just a background warning) is untested and should be checked before calling chunk 5/6 done for claude | -| `opencode` | `OPENCODE_CONFIG_CONTENT` env var (already a registry mechanism, `stock.ts:342`) holding a JSON blob: `{"provider":{"custom":{"options":{"baseURL":...,"apiKey":...},"models":{"":{}}}},"model":"custom/"}` | **Verified by user** | -| `codex` | TOML `config.toml`: top-level `model = ""` + `[model_providers.custom]` (`base_url`, `env_key` naming an env var the real API key rides in — never a literal TOML field, since codex's schema has no such field). Written to an isolated dir via `CODEX_HOME` (`stock.ts:405-415`) so the user's own `~/.codex/config.toml` is never touched | **Config STRUCTURE verified** against a real codex binary (an earlier `[model].default` table shape was rejected: "invalid type: map, expected a string" — caught live). **Protocol CONFIRMED BROKEN against llama.cpp/llama-swap**: codex only speaks the Responses API (`wire_api = "responses"`, the only value it accepts since it dropped `"chat"` support in Feb 2026), and a real llama-swap server does not implement `/v1/responses` — a live run against it failed with repeated `Reconnecting...` then `high demand` errors. Codex support therefore needs a Responses-API-compatible endpoint (most local llama.cpp/Ollama/vLLM setups do not qualify); do not present this as working against a generic OpenAI-Chat-Completions box | -| `gemini` | Env vars `GOOGLE_GEMINI_BASE_URL` + `GEMINI_API_KEY` + `GEMINI_MODEL`; CLI needs a restart to pick them up | **Confirmed BROKEN against llama.cpp/llama-swap, unresolved after real investigation.** Setting `GOOGLE_GEMINI_BASE_URL` makes gemini-cli internally select an `AuthType.GATEWAY` auth path (undocumented — inferred from behaviour) with validation requirements distinct from every normal auth mode; a real run against llama-swap fails with `Invalid auth method selected` regardless of what key/format is supplied. Tried and all failed: a Google-format dummy API key, `GOOGLE_GENAI_USE_VERTEXAI=false`, a `GEMINI_DEFAULT_AUTH_TYPE` override, and hand-writing `settings.json` directly. `--skip-trust` was a real, separate fix (without it a trust-folder check silently overrides `--approval-mode yolo` back to `default`) but does not touch this auth failure. Documented as an open gap, not shipped as working — the registry entry and injection code exist and are exercised by the test script, but end-to-end gemini support needs upstream investigation of `GATEWAY` AuthType before it can be called done | -| `pi` | Config file `~/.pi/agent/models.json` with a custom provider whose `models` is an **array** of `{id}` objects (not an object keyed by id) plus `authHeader: true`. Redirected via the child process's own `HOME` env var, isolated per test/session — **not** `PI_CONFIG_DIR`, which does nothing for pi (grepped pi's entire bundled JS source: the string appears nowhere) | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back. Two real bugs found and fixed before this worked: (1) `PI_CONFIG_DIR` is not read by pi at all — pi hardcodes `~/.pi/agent/models.json` with no dedicated override, so the actual redirect has to be the child process's `HOME`; (2) `models` must be an array of `{id}` objects per pi's own bundled `docs/models.md`, not an object keyed by model id (silently loaded zero models). Also requires an explicit `--model custom/` on invocation — without it pi falls back to its own default provider and fails with "No API key found for the selected model" | -| `grok` | TOML `config.toml`: a fixed `[model.codeman-custom]` block (`base_url`, `env_key` naming an env var the key rides in, never a literal TOML field) written to an isolated dir via `GROK_HOME`. Invoked with `-m codeman-custom` | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back. The ORIGINAL recipe in this table (env vars `GROK_BASE_URL`/`XAI_API_KEY`/`GROK_MODEL`) was flat-out **wrong**, not just unverified: it produced "Not signed in" against a real binary. Grok's real mechanism, confirmed against xAI's own docs and a live binary, is a `config.toml` with a `[model.]` block, redirected via `GROK_HOME`; the key still rides as an env var (`XAI_API_KEY` via `env_key`), just referenced from the TOML rather than read directly | -| `deepseek` | Reuse the **existing** `DEEPSEEK_BASE_URL` + `DEEPSEEK_API_KEY` keys (already declared in `stock.ts`). Only `DEEPSEEK_BASE_URL` is in `privilegedEnvKeys` — `DEEPSEEK_API_KEY` deliberately stays clamp-exempt, since a non-granted owner supplying their OWN key removes privilege rather than granting it (adding it to the clamp list was a real regression, caught by `test/deepseek-mode.test.ts` and fixed before merge). No model-selection var — dsh model is a profile composition entry, not a flag/env var | **Confirmed reaching the server, but failing — unresolved.** A real run against llama-swap returns `dsh: HTTP_404: DeepSeek API error (HTTP 404)` consistently (confirmed the env vars are read: the request reaches the network rather than failing locally). Root cause not identified — plausible explanation by analogy with codex's Responses-API gap is that `dsh --profile headless` expects DeepSeek's official API response shape/path structure rather than a generic OpenAI-compatible `/v1/chat/completions` endpoint, but this was not confirmed by reading dsh's own bundled source (unlike pi/grok, where that grep resolved the question directly). Documented as best-effort/unknown, not shipped as verified working | -| `omp` | Config file `~/.omp/agent/models.yml` with the same array-shaped `models` + `authHeader: true` fix as pi. Redirected via `HOME`, same reasoning as pi (`PI_CONFIG_DIR` does not relocate omp's config either, despite an earlier CLAUDE.md note claiming it does) | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back, after applying the same two fixes as pi (array-shaped `models`, `HOME`-redirect instead of `PI_CONFIG_DIR`) plus an explicit `--model custom/` on invocation. Unverified against omp's own official docs (none are bundled in the install), but empirically confirmed working live | -| `antigravity` | No CLI/env/config mechanism found — Antigravity's docs describe only a GUI settings panel, and explicitly say a custom endpoint "cannot currently" become the core reasoning model. **Not implemented**; toolbar entry stays disabled for this mode with an explanatory tooltip | No known mechanism | +| CLI | Mechanism | Confidence | +| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `claude` | Env vars: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, `ANTHROPIC_DEFAULT_SONNET_MODEL`/`_HAIKU_MODEL`/`_OPUS_MODEL` (all set to the chosen model/deployment name) | **Verified end-to-end** against a real llama-swap server — a real "hello world" reply came back. ⚠️ Non-interactive (`-p`) invocations also fire an async session-title-generation call that reuses `ANTHROPIC_DEFAULT_HAIKU_MODEL` and validates it against Claude Code's OWN internal recognized-model list, printing `[claude-code:unrecognized_model]` and, in `-p` mode, hanging the whole invocation rather than just warning. `--settings '{"autoTitle":false}'` does NOT stop this (confirmed); `--bare` does (the warning still prints, but the real prompt runs) — but `--bare` ALSO disables hooks, LSP, plugin sync, and CLAUDE.md auto-discovery, so it is only safe for the standalone one-shot test script, NEVER for a real interactive Codeman session (which depends on hooks for idle detection, trust-dialog auto-accept, etc. — see the External CLI modes section of CLAUDE.md). Whether an INTERACTIVE claude session with a custom model hits the same hang (vs. just a background warning) is untested and should be checked before calling chunk 5/6 done for claude | +| `opencode` | `OPENCODE_CONFIG_CONTENT` env var (already a registry mechanism, `stock.ts:342`) holding a JSON blob: `{"provider":{"custom":{"options":{"baseURL":...,"apiKey":...},"models":{"":{}}}},"model":"custom/"}` | **Verified by user** | +| `codex` | TOML `config.toml`: top-level `model = ""` + `[model_providers.custom]` (`base_url`, `env_key` naming an env var the real API key rides in — never a literal TOML field, since codex's schema has no such field). Written to an isolated dir via `CODEX_HOME` (`stock.ts:405-415`) so the user's own `~/.codex/config.toml` is never touched | **Config STRUCTURE verified** against a real codex binary (an earlier `[model].default` table shape was rejected: "invalid type: map, expected a string" — caught live). **Protocol picture more nuanced than a flat break, re-verified live twice on 2026-09-17 against a llama-swap deployment that DOES answer `/v1/responses`** (an earlier test's `Reconnecting...`/`high demand` failure does not reproduce against every llama-swap setup): a plain, no-tool-call chat turn (`codex exec 'reply with just OK'`) returned a real reply. But a real tool-call attempt (`run the shell command: echo hello`) came back as an `agent_message` TEXT item — the tool-call JSON printed as the model's answer, not a `function_call` item codex would actually execute (confirmed via `codex exec --json`'s raw event stream: `item.completed`/`agent_message`, never `function_call`). Since tool execution is what makes codex a coding agent at all, this remains **not usable for real work**, just with a different, more specific failure mode than previously documented — still do not present this as working. Separately, EVERY custom-endpoint codex session also prints `warning: Model metadata for '' not found. Defaulting to fallback metadata...` on launch (confirmed harmless — the successful plain-text reply above still had it): codex's per-model metadata (reasoning tiers, system-prompt templates, context-window figures) comes from `models_cache.json`, a LOCAL CACHE of OpenAI's own hosted model catalog that a custom model can never appear in by construction. No config.toml override exists for it, and the isolated `CODEX_HOME` never gets a `models_cache.json` written into it at all (confirmed: inspected a live, actively-used isolated dir — codex evidently can't reach OpenAI's catalog endpoint for this session and just falls back silently every time, with no file left behind to fix or clean up). Fabricating a fake catalog entry to suppress the warning would mean copying the _shape_ of OpenAI's own proprietary schema — including their real per-model system-prompt content, visible in a genuine `models_cache.json` — for a warning confirmed to have no effect on the actual (broken) tool-calling outcome; not worth building | +| `gemini` | Env vars `GOOGLE_GEMINI_BASE_URL` + `GEMINI_API_KEY` + `GEMINI_MODEL`; CLI needs a restart to pick them up | **Confirmed BROKEN against llama.cpp/llama-swap, unresolved after real investigation.** Setting `GOOGLE_GEMINI_BASE_URL` makes gemini-cli internally select an `AuthType.GATEWAY` auth path (undocumented — inferred from behaviour) with validation requirements distinct from every normal auth mode; a real run against llama-swap fails with `Invalid auth method selected` regardless of what key/format is supplied. Tried and all failed: a Google-format dummy API key, `GOOGLE_GENAI_USE_VERTEXAI=false`, a `GEMINI_DEFAULT_AUTH_TYPE` override, and hand-writing `settings.json` directly. `--skip-trust` was a real, separate fix (without it a trust-folder check silently overrides `--approval-mode yolo` back to `default`) but does not touch this auth failure. Documented as an open gap, not shipped as working — the registry entry and injection code exist and are exercised by the test script, but end-to-end gemini support needs upstream investigation of `GATEWAY` AuthType before it can be called done | +| `pi` | Config file `~/.pi/agent/models.json` with a custom provider whose `models` is an **array** of `{id}` objects (not an object keyed by id) plus `authHeader: true`. Redirected via the child process's own `HOME` env var, isolated per test/session — **not** `PI_CONFIG_DIR`, which does nothing for pi (grepped pi's entire bundled JS source: the string appears nowhere) | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back. Two real bugs found and fixed before this worked: (1) `PI_CONFIG_DIR` is not read by pi at all — pi hardcodes `~/.pi/agent/models.json` with no dedicated override, so the actual redirect has to be the child process's `HOME`; (2) `models` must be an array of `{id}` objects per pi's own bundled `docs/models.md`, not an object keyed by model id (silently loaded zero models). Also requires an explicit `--model custom/` on invocation — without it pi falls back to its own default provider and fails with "No API key found for the selected model" | +| `grok` | TOML `config.toml`: a fixed `[model.codeman-custom]` block (`base_url`, `env_key` naming an env var the key rides in, never a literal TOML field) written to an isolated dir via `GROK_HOME`. Invoked with `-m codeman-custom` | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back. The ORIGINAL recipe in this table (env vars `GROK_BASE_URL`/`XAI_API_KEY`/`GROK_MODEL`) was flat-out **wrong**, not just unverified: it produced "Not signed in" against a real binary. Grok's real mechanism, confirmed against xAI's own docs and a live binary, is a `config.toml` with a `[model.]` block, redirected via `GROK_HOME`; the key still rides as an env var (`XAI_API_KEY` via `env_key`), just referenced from the TOML rather than read directly | +| `deepseek` | Reuse the **existing** `DEEPSEEK_BASE_URL` + `DEEPSEEK_API_KEY` keys (already declared in `stock.ts`), now with `appendV1Suffix: true` (see confidence). Only `DEEPSEEK_BASE_URL` is in `privilegedEnvKeys` — `DEEPSEEK_API_KEY` deliberately stays clamp-exempt, since a non-granted owner supplying their OWN key removes privilege rather than granting it (adding it to the clamp list was a real regression, caught by `test/deepseek-mode.test.ts` and fixed before merge). No model-selection var — dsh model is a profile composition entry, not a flag/env var | **Root cause of the original `HTTP_404` found and fixed, by reading dsh's own bundled source — the same bar pi/grok's fixes were held to.** Installed `@deepseek-ai/dsh` (all its real published dependencies) into a scratch directory purely to read `@deepseek-ai/dsh-llm-deepseek/lib/index.js`: it builds its request as `fetch(\`${connection.baseURL}/chat/completions\`, ...)`with`baseURL`read straight from`DEEPSEEK_BASE_URL`(or defaulting to DeepSeek's real public API root,`https://api.deepseek.com`, which also carries no `/v1`) — no `/v1` insertion of dsh's own, unlike the OpenAI-SDK convention this recipe originally assumed. llama-swap/llama.cpp only ever serves the OpenAI-conventional `/v1/chat/completions`. Confirmed live: `POST /chat/completions` → `404`, `POST /v1/chat/completions` → `200`, on the exact same endpoint — and dsh's own error-message template, `DeepSeek API error (HTTP ${status})`, reproduces the originally reported `dsh: HTTP_404: DeepSeek API error (HTTP 404)` precisely. Fixed by adding `appendV1Suffix` (env kind only, deepseek's entry alone — claude/gemini must NOT get it, since claude was already confirmed working against the unmodified `baseUrl`), which runs `endpoint.baseUrl` through the same `withV1Suffix()` helper `configDir`-kind CLIs already use. ⚠️ Not yet re-run end-to-end with a real `dsh` binary — no install available in this environment (no npm-installed CLI binary in `PATH`, and the `codeman-test-picker` container doesn't bundle it either); the fix is source-confirmed and live-verified at the HTTP level, but a genuine "hello world" reply through `dsh` itself is the remaining step before promoting this to **verified** alongside claude/opencode/pi/grok/omp | +| `omp` | Config file `~/.omp/agent/models.yml` with the same array-shaped `models` + `authHeader: true` fix as pi. Redirected via `HOME`, same reasoning as pi (`PI_CONFIG_DIR` does not relocate omp's config either, despite an earlier CLAUDE.md note claiming it does) | **Verified end-to-end** against a real llama-swap server — real "hello world" reply came back, after applying the same two fixes as pi (array-shaped `models`, `HOME`-redirect instead of `PI_CONFIG_DIR`) plus an explicit `--model custom/` on invocation. Unverified against omp's own official docs (none are bundled in the install), but empirically confirmed working live | +| `antigravity` | No CLI/env/config mechanism found — Antigravity's docs describe only a GUI settings panel, and explicitly say a custom endpoint "cannot currently" become the core reasoning model. **Not implemented**; toolbar entry stays disabled for this mode with an explanatory tooltip | No known mechanism | Everything web-researched-but-unverified gets implemented but must be smoke-tested against real installs of those CLIs before being called done — @@ -208,6 +208,14 @@ extra per-model configuration on Codeman's side at all. ### 4. Toolbar UI +> **Superseded.** This section describes the toolbar-button design as originally +> planned. What actually shipped is a Run-menu picker instead: one generated entry +> per (capable harness, saved endpoint) pair directly in the existing `#runModeMenu` +> dropdown, rather than a separate `#customModelBtn`/`#customModelMenu` surface. See +> [`docs/custom-model-endpoints.md`](custom-model-endpoints.md#the-run-menu-picker) +> for the current design; the sections below (session-restart mechanics, security) +> remain accurate regardless of which UI calls the underlying route. + - New header/toolbar button (e.g. `#customModelBtn`, `btn-toolbar btn-custom-model`), marker-hidden by default (`btn-custom-model--hidden`) and revealed by `applyHeaderVisibilitySettings()` only when @@ -344,8 +352,12 @@ pure unit tests and the live manual checks in Verification: up automatically with zero edits to the script). Already run to completion against the author's llama-swap server (a LAN address, inside a `codeman/agent:llm-test` Docker image with all 9 CLI binaries): - claude/opencode/pi/grok/omp **PASS**, codex **FAILs as expected** - (Responses-API protocol gap, not a bug), gemini/deepseek **UNCONFIRMED** + claude/opencode/pi/grok/omp **PASS**, codex **partially works and still + isn't usable** (plain chat succeeds against a llama-swap deployment that + answers `/v1/responses`, but a real tool-call attempt comes back as + inert text rather than an executable `function_call` — see the + confidence table row for the full, re-verified picture), gemini/deepseek + **UNCONFIRMED** (reach the server, fail for undiagnosed reasons — see their table rows), antigravity **SKIP** (no mechanism). Re-run this against a real cloud endpoint (e.g. an Azure AI Foundry deployment) once one is available, to diff --git a/docs/custom-model-endpoints.md b/docs/custom-model-endpoints.md index 975965ef8..59e5ed56c 100644 --- a/docs/custom-model-endpoints.md +++ b/docs/custom-model-endpoints.md @@ -11,20 +11,22 @@ company gateway) — anything answering `GET /v1/models` and recipe confidence table, and security reasoning: [`custom-model-endpoints-plan.md`](custom-model-endpoints-plan.md). -> **Status**: backend is implemented and tested (registry capability, the -> injection engine, the endpoint store + discovery route, the session -> restart route). The toolbar picker / settings UI described below as the -> intended surface is **not yet built** — until it lands, use the HTTP API -> directly (examples below). Antigravity has no known custom-endpoint -> mechanism and is not supported. +> **Status**: fully wired end to end — registry capability, the injection +> engine, the endpoint store + discovery route, both the restart-in-place +> apply route (Claude) and the one-shot quick-start launch path (every +> other supported harness), a settings-panel CRUD surface, and the Run-menu +> picker described below. Antigravity has no known custom-endpoint +> mechanism and is not supported. The HTTP API (examples below) still works +> directly and is what the picker itself calls under the hood. ## Turning it on -App Settings → Agents & CLIs → **Custom Model Endpoints** (synced setting -`customModelEndpointsEnabled`, default **OFF**). Until the toolbar picker -lands, nothing reads this setting: the HTTP routes below work whether it is -on or off, and it exists now only so the picker has a switch to hang off -when it ships. The API equivalent: +App Settings → Models → **Custom model endpoints** (synced setting +`customModelEndpointsEnabled`, default **OFF**). Turning it on does two +things: it reveals the endpoint list/add/edit/discover panel in that same +settings section, and it makes the Run menu offer a generated entry per +(harness, endpoint) pair — see "The Run-menu picker" below. The API +equivalent: ```bash curl -sk -X PUT https://localhost:3000/api/settings \ @@ -34,6 +36,9 @@ curl -sk -X PUT https://localhost:3000/api/settings \ ## Adding an endpoint +Via App Settings → Models → Custom model endpoints → **+ Add endpoint**, or +directly: + ```bash curl -sk -X POST https://localhost:3000/api/model-endpoints \ -H 'Content-Type: application/json' \ @@ -62,7 +67,173 @@ configured, `PUT`/`DELETE /api/model-endpoints/:id` update or remove one. Endpoint management is admin-only in multi-user mode, same as remote/docker hosts — these are machine-level infra, not per-user settings. -## Applying a model to a session +**Context length is discovered too, opportunistically and safely.** The plain +`GET /v1/models` response has no context-window field. Discovery only ever +looks for one for a model llama-swap's own response already reports +`status.value === "loaded"` for — never for an unloaded one, because +llama-swap treats `?model=` as a routing hint and asking about a model that +isn't loaded risks triggering an actual (slow, GPU-swapping) load as a side +effect of what should be read-only discovery. A server with no `status` field +on any entry at all (not llama-swap) gets no context-length enrichment, +rather than guessing. A model's previously-learned context length survives a +later cycle where it wasn't the loaded one; it's dropped only once the model +disappears from the endpoint's list entirely. Stored per model in +`modelContextLengths` and applied automatically (see "Applying a model to a +session" below) so a CLI that would otherwise assume a large default context +window for an unrecognized model id stops silently overflowing a much +smaller real one. + +**Where that number actually comes from matters, and got this wrong once +already.** The first cut read it from llama.cpp's own +`GET /props?model=` (`n_ctx`) — plausible, and it worked in testing, but +confirmed live to be actively WRONG for a `--fit-ctx`-launched llama-swap +backend: `/props` reported `n_ctx: 154112` for a model llama-swap itself had +launched with `--fit-ctx 16384`, and the real server then refused a request +right at that real 16384-token limit — `/props`'s `n_ctx` appears to report +the model's theoretical/trained maximum there, not the runtime-configured +one. Discovery now parses the REAL configured size straight out of +llama-swap's own launch command instead (`GET /running`'s `cmd` field — +`--fit-ctx ` first, then the plain llama.cpp `-c`/`--ctx-size` a +hand-written command might use), and only falls back to the `/props` probe +when `cmd` states no recognizable flag at all. + +**File size is discovered too, when the server states one.** llama-swap +writes a GB figure into an auto-discovered model's own `description` +(`"Auto-discovered 16.35 GB - parameters auto-fitted by llama.cpp"`), parsed +into `modelSizesGB` — unlike context length, this needs no `/props` probe +(the figure is right there in the `/v1/models` response) and so is populated +for every model regardless of loaded state. A hand-configured profile's own +description has no such figure and correctly gets no entry, never a guess. +Used only to label the Run-menu picker's "loading model" banner (e.g. +"Loading qwen3.8-27b-ud-q4_k_xl (16.4 GB) on llama-swap..."); never anything +a server-side check relies on. + +**The loading banner is unbounded by design, and says so — no countdown, no +automatic give-up.** An earlier version scaled an expected-time estimate and +a timeout off the model's file size and auto-closed the session once that +elapsed, but a real load's actual duration depends on hardware this feature +has no way to know (VRAM, storage speed, whatever else is contending for the +GPU) — any fixed number was a guess dressed up as a fact, and a model that +genuinely takes 10+ minutes on slower hardware would just get killed +mid-load by its own display. The banner now says outright that it can take a +while depending on hardware and model size, polls +`GET /api/model-endpoints/:id/running-status` every second for as long as it +takes, and carries a **Cancel** button (rendered on the banner itself) that +ends the wait and closes the session the load was for — the user's own call +on when it's taking too long, not a fixed number baked into the client. + +**The banner's second line is the real backend log line, not a guess.** +llama-swap's `GET /api/events` SSE stream carries the actual `llama-server` +process's own stdout — `load_model: loading model ''`, +`llama_server: model loaded`, tokenizer warnings, all of it — tagged +`source: "upstream"`, distinct from llama-swap's own `source: "proxy"` +request-access lines. `running-status`'s response now includes `logLine` +(via `getLatestLlamaSwapLogLine`), and the banner shows it on its own line +under the disclaimer, e.g. "llama.cpp: load_model: loading model '...'" — +confirmed live end-to-end through a real forced swap, sequentially showing +the model path, a tokenizer warning, then staying on whatever llama.cpp last +printed once the load goes quiet (never cleared back to blank). ⚠️ +**`GET /logs` — the endpoint this feature's own first cut was built +against — turns out to carry ONLY llama-swap's own proxy request-access +log.** Confirmed live it never showed a single backend line, even seconds +after a real, verified model swap; `/api/events`'s `logData` frames are the +only source that actually has it, and its own `source` field (`upstream` vs +`proxy`) is what `getLatestLlamaSwapLogLine` filters on. One `/api/events` +connection is held open per endpoint and reused across every session +watching a load on it (confirmed live to stay open indefinitely, unlike +`/logs`, which closes after a fixed ~100KB), idle-closed after 30s of nobody +polling it (`pruneIdleLlamaSwapLogTails`, same 20s sweep as the +swap-displacement check below). + +`defaultModelId` names which discovered model the picker pre-marks for that +endpoint — the settings panel's Edit form exposes it as a select populated +from the endpoint's own discovered `models`, and the route refuses a value +that isn't one of them. It is applied automatically only when the endpoint +has exactly one discovered model (nothing to choose); with two or more it +is a pre-selection in the model-picker dialog below, never a silent default. +Re-discovering drops a default that no longer appears in the fresh list +rather than carrying an invalid one forward. + +**Model lists refresh themselves.** A background sweep (`server.ts`, +`CUSTOM_MODEL_REDISCOVER_INTERVAL_MS`, every 5 minutes) re-discovers every +saved endpoint the same way the manual `POST .../discover-models` route +does, best-effort per endpoint — one being unreachable on a given cycle +never blocks the others. Off under `npm test`, same reasoning as the Codex +plan-usage poll it sits beside: no real network to hit, no server instance +to keep the timer alive for. + +## The Run-menu picker + +With the setting on and at least one endpoint carrying a discovered model, +the toolbar's Run dropdown grows a **Custom Endpoints** section: one entry +per (harness that can redirect to a custom endpoint, saved endpoint) pair, +e.g. "Claude Code (llama.cpp)". The harness list is read off the CLI +registry's own `capabilities.customModelInjection` at page render +(`window.__codemanCustomModelClis`, `server.ts`) — never a hardcoded id list +in the frontend — so a CLI whose injection recipe lands later shows up with +no frontend change, and Antigravity (`unsupported`) never does. + +Picking an entry re-fetches the endpoint (`selectCustomModelEntry()`, +`session-ui.js`) rather than trusting anything cached from the dropdown's +own render — the model list can have changed via the 5-minute sweep above +or a settings-panel edit since the menu opened. With exactly one discovered +model it runs straight away; with two or more, a small modal +(`#customModelPickModal`) lists them and asks which one to use for this +launch, with the endpoint's `defaultModelId` marked but not auto-chosen — +the point of asking is letting one launch deliberately differ from the +saved default, not just confirming it. + +**How the launch itself applies the endpoint depends on the harness.** For +opencode, Codex, Gemini, Pi, Grok, DeepSeek and OMP (`runCustomModelEntry` → +`_runCustomModelEntryOneShot`), the endpoint/model is folded into the SAME +`POST /api/quick-start` call that creates the session (`customModel` field), +so the session launches directly on the endpoint — no restart, no visible +relaunch. Claude (`_runCustomModelEntryViaRestart`) still uses the original +two-step design: the launch runs a single native session exactly the way its +own Run-menu entry would, then **waits for the new session to go idle** +(`GET .../wait?until=idle`, bounded at 20s — a normal 200 either way, never +an error, per the wait endpoint's own contract) before applying the endpoint +via the restart route below. That wait exists because a freshly launched CLI +reports itself as `busy` for its own startup (a boot spinner, a +workspace-trust check) well before the apply call would otherwise reach it, +and the apply route correctly refuses to restart a session mid-turn — a +fresh boot looks exactly like one from the outside. A session still busy +after the wait reaches the apply call anyway and gets that route's own +honest `SESSION_BUSY` error, now visible as a sticky toast with a close +button rather than a generic message that vanished in three seconds. Claude +stays on this path because its own restart (`--resume`-based, keeping the +conversation) is far less jarring than the other seven's, and `runClaude()`'s +multi-tab launch and docker-config-drift confirm/retry loop make folding it +into the one-shot path separate work. It is a +one-off "try this endpoint" action, not a sticky mode: the plain Run button +still means "this harness, native cloud" afterward. Entries are hidden +entirely for a remote or Docker active case, since the apply route refuses +both (see the next section). + +## Launching directly on an endpoint (no restart) + +```bash +curl -sk -X POST https://localhost:3000/api/quick-start \ + -H 'Content-Type: application/json' \ + -d '{"caseName": "myapp", "mode": "codex", "customModel": {"endpointId": "llama-box", "modelId": "qwen3"}}' +``` + +`POST /api/quick-start`'s `customModel` field (`{endpointId, modelId, +confirmed?}`) computes the same injection the restart route below does, but +BEFORE the session exists — the session is minted its own id up front +(`crypto.randomUUID()`), the injection (env vars, and for a `configDir`-kind +CLI, the written config file) targets that real id, and the session launches +already pointed at the endpoint. No restart, because there was never a +native-backend launch to restart away from. Runs the same llama-swap +conflict check as the restart route (below) — a `409`-shaped +`{requiresConfirmation, currentlyLoadedModel, affectedSessions}` response +with no session created, resolved by retrying with `confirmed: true` — and +is refused the same way for a remote or Docker case. This is what the +Run-menu picker uses for opencode, Codex, Gemini, Pi, Grok, DeepSeek and OMP; +Claude still uses the restart route below (see "The Run-menu picker" above +for why). + +## Applying a model to an ALREADY-RUNNING session ```bash curl -sk -X POST https://localhost:3000/api/sessions//custom-model \ @@ -84,6 +255,172 @@ since for those three the config file alone does not switch the model. reattaches the durable remote/in-container tmux rather than relaunching the agent, so the selection would report success and change nothing. +**Claude gets two more env vars when known/applicable, both declared on its +registry entry (`contextLengthVar`/`configDirVar`), not hardcoded here:** + +- `CLAUDE_CODE_MAX_CONTEXT_TOKENS` is set to `modelId`'s discovered context + length (see the discovery section above) whenever one is known. Without + it, Claude Code assumes a large (200k) window for any unrecognized custom + model id and never compacts, which reliably overflows a much smaller real + local context — confirmed live: a stock ~33.7K-token system prompt against + a 16384-token llama-swap model failed with `exceeds the available context +size`. No entry for the model in `modelContextLengths` means the var is + simply omitted, never a guess. ⚠️ **This var only affects when Claude + Code compacts conversation _history_ — it cannot fix a model whose real + context is smaller than Claude Code's own fixed per-turn overhead** + (system prompt + tool schemas, empirically ~36.4K tokens, confirmed live + via an `in:0 out:0` failure on the very first message, before any + history exists to compact). No context-length declaration changes that + fixed overhead, so a model below the safe floor fails outright on + message one regardless of what this var says. See "Context-window floor + warning" below for how Codeman catches this case before launching + instead of after. +- `CLAUDE_CONFIG_DIR` is pointed at the same isolated per-session directory + the `configDir`-kind CLIs use (empty, no files written into it), so the + injected `ANTHROPIC_API_KEY` never shares a directory with a stored + claude.ai OAuth login. Claude Code still prints "Both claude.ai and + ANTHROPIC_API_KEY set" when the two coexist in the same config directory — + cosmetic (confirmed live: the API key wins for actual requests either way, + visible in the terminal's own `API Usage Billing` line) but worth + eliminating rather than living with. The directory's `projects` + subdirectory is symlinked (a junction on Windows) back to the real + `~/.claude/projects` so the response viewer, subagent windows and Read My + Mind keep working for that session — the same trade-off and fix documented + for a manually-set `CLAUDE_CONFIG_DIR` in + [`docs/wiki/Agent-CLIs.md`](wiki/Agent-CLIs.md), just applied + automatically here. Best-effort: a platform that refuses the symlink keeps + the pre-existing blind-response-viewer side effect rather than failing the + whole custom-model apply over it. + +**That isolated directory needed one more fix to actually be usable +non-interactively.** An otherwise-empty `CLAUDE_CONFIG_DIR` has none of a +real profile's prior "Detected a custom API key — use it?" approvals, so +without more, Claude Code stops and asks that on _every single launch_ — +confirmed live, and with nobody at a TTY to answer, its own default answer +("No") silently refuses the very key this feature just injected, which +looks like the endpoint being ignored entirely. `customModelInjection`'s +`apiKeyTrustFile` (`{ relPath: '.claude.json', shape: +'claude-api-key-responses' }` on claude's entry) pre-seeds that exact +approval: the apply step merges `customApiKeyResponses.approved: [apiKey]` +into `/.claude.json`, the same field a real answered prompt +itself writes to (confirmed against a real file after answering by hand +once) — this answers the prompt in advance rather than bypassing it. The +merge preserves whatever else the CLI already wrote into that file on an +earlier launch in the same isolated directory (`userID`, `numStartups`, +earlier approved keys), and a missing or corrupt file is treated as empty +rather than failing the apply. + +**A fresh `CLAUDE_CONFIG_DIR` isn't just missing that one approval — Claude +Code treats it as a brand-new profile and replays its ENTIRE first-run +sequence on every launch: the theme picker, the security-notes screen, the +per-project "trust this folder?" dialog, and (running with +`--dangerously-skip-permissions`) a one-time warning about bypassing +permissions.** Confirmed live: none of these show up again for a real, +already-onboarded profile, but every custom-model session gets a fresh, +otherwise-empty isolated directory, so it saw all four every single time. +`customModelInjection`'s `skipFirstRunPrompts` (`true` on claude's entry, +requires `apiKeyTrustFile` since it reuses the same file) pre-seeds the +state a real profile accumulates from answering all of that once: +`hasCompletedOnboarding: true` and the launching session's own +`projects[workingDir].hasTrustDialogAccepted: true` go into the same +`/.claude.json` the API-key approval above already merges into +(other projects, and other fields on this session's own project entry, are +left untouched), and `skipDangerousModePermissionPrompt: true` goes into +`/settings.json` — a different file, merged the same +corrupt-tolerant way. `workingDir` is used exactly as the session was +launched with as its cwd, never realpath'd or slash-normalized, since +that's the literal string Claude Code itself uses as the project key. + +**llama-swap gets two more fixes on top of the context-length/config-dir +ones above, both from watching a real switch live.** llama.cpp only ever +runs one model at a time; llama-swap swaps the backing process on demand, +which can take anywhere from a few seconds to well over a minute: + +- **The conflict check.** Both apply routes (the restart one here and the + one-shot `POST /api/quick-start` above) call llama-swap's own + `GET /running` first — feature-detected, so a plain llama.cpp/OpenAI- + compatible server (no such endpoint) is simply never checked. If a + _different_ model is currently loaded and ready, and another **live + session's own selection** is using it, the apply returns + `{requiresConfirmation: true, currentlyLoadedModel, affectedSessions}` + instead of silently switching — nothing is applied or created yet. + Retrying with `confirmed: true` skips the check. Switching with nothing + else affected proceeds immediately; this is a warning about disrupting + another session, never a gate on the switch itself. +- **Actually starting the load.** llama-swap has no "switch model" admin + call — the only thing that starts a swap is a real inference request + naming the model, and confirmed live: applying a selection alone never + reached llama-swap at all (nothing in its own server logs), since nothing + had actually asked it to load anything yet. Both apply routes now also + send the smallest real request that will — + `POST /v1/chat/completions` with `max_tokens: 1` and one + throwaway message — whenever the + target model isn't already the one loaded and ready, fire-and-forget (its + response is never read; `GET /api/model-endpoints/:id/running-status`, + polled client-side, is what actually confirms readiness). The response + also carries `modelSwapInProgress: true` in that case, which is what + drives the Run-menu picker's own "loading model" status banner. + +## Catching a swap after the fact + +The conflict check above only runs at the moment a session is created or a +model is applied — it has no way to catch a swap that happens **later**. +Confirmed live: a session created while nothing else conflicted at that +exact instant can still get silently displaced afterward, once a +_different_ session's own normal use (or its own create-time load trigger) +asks llama-swap to load something else. llama-swap has no push +notification of its own for this, so a background sweep +(`detectCustomModelSwapDisplacements`, `CUSTOM_MODEL_SWAP_CHECK_INTERVAL_MS` += 20s in `server.ts`) polls `GET /running` once per distinct endpoint that +has at least one live custom-model session, and compares each such +session's own `modelId` against what is actually loaded. A session whose +model is no longer in that list gets a `custom-model:swapped-out` SSE event +(`{sessionId, sessionName, endpointId, previousModel, currentlyLoadedModel}`), +shown as a global toast — global rather than tied to that session's tab, +since the whole point is telling the user before they type into it +expecting the model they picked. Notifies **once per displacement**: the +same de-dupe `Set` clears a session's flag once its own model is loaded and +ready again, so a later, genuinely new displacement notifies again rather +than the session staying silently un-notified forever after the first one. + +## Context-window floor warning + +Claude Code's own fixed per-turn overhead (system prompt + tool schemas, +empirically ~36.4K tokens) can exceed a small local model's _entire_ real +context on its own, before any conversation history exists to fill it — +confirmed live twice, both as an `in:0 out:0` failure on the very first +message sent. `CLAUDE_CODE_MAX_CONTEXT_TOKENS` (above) cannot fix this: it +only governs when Claude Code compacts conversation history, and there is +no history yet on message one. Applying such a model would look like the +endpoint being ignored, or the wrong model being used, when in fact the +endpoint applied correctly and the model is simply too small for this CLI. + +Both apply routes (the restart route and the one-shot `POST +/api/quick-start`) now check for this **before** launching or restarting +anything, gated on the CLI's registry entry declaring a `contextLengthVar` +(currently only claude — the check is a no-op for every other CLI by +construction, never a hardcoded mode check). If the model's discovered +context (`modelContextLengths`, from discovery above) is below +`CLAUDE_MIN_SAFE_CONTEXT_TOKENS` (40000, comfortably above the measured +~36.4K overhead), the response is `{requiresContextWarning: true, modelId, +contextLength, minSafeContextTokens}` instead of applying — nothing is +restarted or created yet. A context length that was never discovered at +all skips the check entirely (nothing to compare, so it fails open rather +than warning on every model an endpoint hasn't reported a size for). +Retrying with `confirmed: true` launches anyway. + +The Run-menu picker shows this as an in-app modal +(`#customModelContextWarningModal`, matching the llama-swap conflict +modal's look) naming the model, its discovered context, and the safe +floor, and explaining the fix: reconfigure llama-swap to give that model +(or a smaller one) an explicit larger context instead of relying on +auto-fit (`--fit-ctx`), which optimizes for the biggest _model_ that fits +rather than the biggest _context_ — e.g. adding `-c 65536` (or as large a +`--ctx-size` as the hardware holds) to that model's llama-swap config +entry. A smaller model at a much larger explicit context often fits in +the same VRAM a bigger model's auto-fit context gets shrunk to make room +for. + Clear back to the harness's native cloud default with: ```bash @@ -115,17 +452,53 @@ automatically). Results: - **Claude, opencode, Pi, Grok, OMP** — verified: a real "hello world" reply came back through the endpoint. -- **Codex** — the config is structurally correct, but Codex only speaks the - Responses API since Feb 2026, which llama.cpp/llama-swap don't implement. - This is a real protocol incompatibility, not a bug here; Codex support - needs a Responses-API-compatible endpoint. +- **Codex** — the config is structurally correct, and against a llama-swap + server that DOES answer `/v1/responses` (confirmed live: a plain, + no-tool-call chat turn returned a real reply), the picture is more + nuanced than a flat failure. A real tool-call attempt (`run the shell +command: echo hello`) came back as `agent_message` TEXT — literally the + tool-call JSON printed as the model's answer — instead of a + `function_call` item Codex would actually execute (confirmed via `codex +exec --json`'s raw event stream). So plain chat can work while the thing + that makes Codex a coding agent — actually running commands and editing + files — does not; treat Codex as still unreliable for real work against a + llama.cpp/llama-swap endpoint, tool-calling gap included, not just the + earlier-documented `wire_api` mismatch (which not every deployment hits + the same way — some legitimately have no `/v1/responses` route at all). + Separately, EVERY custom-endpoint Codex session prints `Model metadata +for '' not found. Defaulting to fallback metadata...` on launch — + confirmed harmless (the reply above still came back correctly): Codex's + model metadata (reasoning-tier options, per-model system-prompt + templates, context-window figures) comes from `models_cache.json`, a + local cache of OpenAI's own hosted model catalog that a custom local + model can never appear in by construction, since it isn't one of + OpenAI's models. There's no config.toml override for a model's metadata, + and fabricating a fake catalog entry would mean copying the _shape_ of + OpenAI's own proprietary schema (their per-model system-prompt content + included) for a warning that doesn't otherwise affect behavior — not + something to build into discovery. - **Gemini** — fails with `Invalid auth method selected`, traced to an undocumented `GATEWAY` auth path gemini-cli selects once `GOOGLE_GEMINI_BASE_URL` is set. Unresolved after real investigation (several auth workarounds were tried and ruled out); do not rely on Gemini support yet. -- **DeepSeek** — the request reaches the server (env vars are read) but - gets a consistent `HTTP_404`. Root cause not identified; best-effort only. +- **DeepSeek** — root cause of the `HTTP_404` found and fixed. DeepSeek + Harness's own bundled provider module (`@deepseek-ai/dsh-llm-deepseek`) + builds its request URL as `${DEEPSEEK_BASE_URL}/chat/completions` with no + `/v1` insertion of its own (its real public API, `https://api.deepseek.com`, + expects the caller's base URL to already carry any needed prefix) — + confirmed by reading its own source and, live, that + `POST /chat/completions` 404s against llama-swap while + `POST /v1/chat/completions` succeeds; the harness's own error + template (`DeepSeek API error (HTTP ${status})`) matches the originally + reported symptom exactly. `customModelInjection`'s new `appendV1Suffix` + (deepseek's entry only — claude/gemini must NOT get it, since claude was + already confirmed working against the raw `baseUrl`) fixes it by writing + `DEEPSEEK_BASE_URL` with `/v1` appended. Not yet re-run end-to-end with a + real `dsh` binary (no install available in this environment) — the fix + is source-confirmed and live-verified at the HTTP level, but a real + "hello world" reply through `dsh` itself is still outstanding before + calling this fully verified like the harnesses above. - **Antigravity** — no known custom-endpoint mechanism at all; unsupported. See the confidence table in `custom-model-endpoints-plan.md` for the full detail behind diff --git a/docs/wiki/Agent-CLIs.md b/docs/wiki/Agent-CLIs.md index eb013b29a..d7ad4b22a 100644 --- a/docs/wiki/Agent-CLIs.md +++ b/docs/wiki/Agent-CLIs.md @@ -273,9 +273,16 @@ into the case's `.claude/settings.local.json` so that `/model` keeps working. - **Shell** for the times you want a terminal on your phone with no agent at all. It is a genuinely useful mode, not a fallback. +## Pointing one at your own server + +Most of these harnesses can also run against a custom OpenAI-compatible endpoint instead of +their native cloud backend, for one session at a time, an opt-in feature covered in full on +[Custom Model Endpoints](Custom-Model-Endpoints). + ## Read next - [Core Concepts](Core-Concepts) - run modes versus location overlays. +- [Custom Model Endpoints](Custom-Model-Endpoints) - run a harness against your own server. - [Settings Reference](Settings-Reference) - model, effort, and permission-mode settings. - [Keeping Agents Running](Keeping-Agents-Running) - what idle detection does per mode. - [Security](Security) - what skipping permission prompts actually means. diff --git a/docs/wiki/Custom-Model-Endpoints.md b/docs/wiki/Custom-Model-Endpoints.md new file mode 100644 index 000000000..ebe05fee8 --- /dev/null +++ b/docs/wiki/Custom-Model-Endpoints.md @@ -0,0 +1,183 @@ +# Custom Model Endpoints + +Point a harness at your own OpenAI-compatible server instead of its native cloud backend, for +one session at a time. "Custom endpoint" covers **local** hardware (llama.cpp, Ollama, vLLM, +a home GPU rig, DGX Spark, Strix Halo) and **cloud** services (Azure AI Foundry's +OpenAI-compatible endpoint, OpenRouter, a company gateway) alike, anything answering +`GET /v1/models` and `POST /v1/chat/completions` in the standard shape. + +**Off by default.** Turn it on in App Settings → Models → **Custom model endpoints**. + +## Adding an endpoint + +Still in App Settings → Models → Custom model endpoints: + +1. **+ Add endpoint** — give it an id, a label, and the base URL (`http://192.168.1.50:8080`, + say). An API key is optional; most local servers don't check one. +2. **Discover** — fetches the endpoint's own model list over `GET /v1/models` and stores it. +3. Pick a **default model** from what was discovered. This is the model the Run-menu entry + applies directly when only one model is discovered; with two or more, it's just the one + pre-marked in the picker dialog described below, not a silent default. + +Endpoint management is admin-only in multi-user mode, the same as remote hosts and Docker +hosts — these are machine-level infra, not a per-user setting. + +**Model lists refresh themselves.** Every saved endpoint is re-discovered automatically every +5 minutes in the background, so a model the server starts serving later — or stops serving — +shows up without another manual click of **Discover**. One endpoint being unreachable on a +given cycle (powered off, wrong network) never blocks the others from refreshing. + +**Context length is picked up automatically where it can be, safely.** Against a +llama.cpp/llama-swap server, discovery also learns each _currently loaded_ model's real +context window and applies it to the launched session (Claude Code today — see below), so +the harness stops assuming a large default window for a model name it doesn't recognise and +overflowing a much smaller real one. It's deliberately never probed for a model that isn't +already loaded, since asking a llama-swap server about an unloaded model can trigger an +actual, slow model swap as a side effect — a model just not currently loaded keeps whatever +context length an earlier cycle already learned for it instead. + +## Running a session against one + +With the setting on and at least one endpoint carrying a discovered model, the **Run** +dropdown grows a **Custom Endpoints** section: one entry per harness that can redirect to a +custom endpoint, per saved endpoint, e.g. "Claude Code (llama.cpp)". Picking one starts a +session on that harness exactly the way its own entry would. It is a one-off "try this +endpoint" action, not a sticky mode — the plain **Run** button still means "this harness, +native cloud" afterward, and a fresh session never inherits whatever the last one was +pointed at. + +**Which model it uses depends on how many the endpoint has discovered.** With exactly one, +the session launches straight away on that model — nothing to choose. With two or more, a +small dialog asks which one to use for this launch before starting the session; the +endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately +use a different one without changing the saved default. + +**For opencode, Codex, Gemini, Pi, Grok, DeepSeek and OMP, picking an entry launches +straight onto the endpoint** — no restart, because the endpoint is applied before the +session's process ever starts. **Claude still restarts the harness's process in place** — +same tab, same conversation (`--resume`) — after a normal native launch, since that restart +is far less jarring for Claude than for the other seven, whose own TUI can fully +reinitialize on a restart. Either way, every supported harness reads its endpoint config at +process start, never per turn, so there is no live hot-swap while a turn is running. + +Picking an entry that launches a **brand-new** Claude session waits (up to 20 seconds) for it to +finish its own startup before applying — a freshly started CLI reports itself as busy for its +boot sequence, and applying to a genuinely busy session is refused so a real, in-progress +turn is never interrupted out from under you. A session that is still busy after that wait +(a very slow-starting CLI, or one you started typing into right away) surfaces that refusal +as an ordinary error, which now stays on screen with a close button instead of vanishing +after a few seconds — read it, it names the actual reason rather than a generic failure. + +Entries are hidden entirely for a session in a **remote (SSH) or Docker case** — support for +redirecting those hasn't landed yet, see below. The picker also only appears in the desktop +**Run** dropdown; the phone home screen builds its own run picker separately and does not +currently offer these entries. + +**Against llama-swap, applying a selection also starts the actual model load, rather than +waiting on your first prompt to do it.** llama-swap has no "switch model" button of its own +— the only thing that starts a swap is a real request naming the model, and confirmed live: +just applying a selection never reached llama-swap's own logs at all until something asked +it to load. Picking an entry now also sends the smallest real request that will trigger +that load, in the background, the moment the target model isn't already loaded and ready. + +**The centred loading banner has no countdown and no automatic timeout — it waits as long as +it takes, and tells you so.** When it knows the model's discovered file size (its GB figure, +when llama-swap states one) it's shown too, e.g. "Loading qwen3.8-27b (16.4 GB) on +llama-swap — this can take a while depending on your hardware and the model size." An +earlier version tried to estimate and enforce a time limit, but real load time depends on +hardware this feature has no way to know, so a fixed number was always a guess — worse, one +that could kill a genuinely slow load partway through. If it really is taking too long, a +**Cancel** button right on the banner ends the wait and **closes the session that load was +for**, on your own call rather than a guessed deadline. + +**The banner also shows a real, live second line of what llama.cpp itself is doing** — not +a made-up progress phase, the actual next line the `llama-server` process printed, e.g. +"llama.cpp: load_model: loading model '/models/.../Qwen3.8-27B.gguf'" then later +"llama.cpp: llama_server: model loaded". It comes straight from llama-swap's own event +feed, filtered down to just the backend process's own output (not llama-swap's own request +logging), and stays on whatever it last said once the load goes quiet, rather than +clearing back to nothing. + +**You'll also be told if a session's model gets swapped out from under it later, not just +at launch.** The conflict warning above only fires at the moment you launch or apply a +model — llama.cpp only runs one model at a time, so if a DIFFERENT session using the same +endpoint later triggers its own load, whatever was loaded before (including a session you +already had running) gets silently evicted, with no warning at that instant since nothing +conflicted when it was first set up. A background check (every 20 seconds) catches this +after the fact and shows a toast naming which session lost its model and what's loaded now +— so you know before typing into that session that it's about to reload (and, in turn, +evict whatever displaced it). + +**Claude Code specifically gets three extra fixes applied automatically:** + +- Its discovered context length (see above) is passed through as + `CLAUDE_CODE_MAX_CONTEXT_TOKENS`, so it doesn't send a full-size prompt against a much + smaller real local context and overflow it. +- Its session runs with an isolated `CLAUDE_CONFIG_DIR`, so the injected API key never sits + in the same directory as a stored claude.ai login — that combination is harmless for actual + requests (the API key wins) but the CLI still prints a "both claude.ai and + ANTHROPIC_API_KEY set" warning about it, which this avoids entirely. The isolated directory + keeps a link back to your real session history so the response viewer and similar features + still work for that session. That isolated directory starts with no prior approvals of its + own, so Codeman also pre-approves the injected key the same way answering Claude Code's own + "Detected a custom API key" prompt once would — without it, that prompt would otherwise + reappear on every single launch with nobody there to answer it. +- **That same fresh isolated directory also looks like a brand-new Claude Code profile**, so + without this fix it replayed the WHOLE first-run sequence every single launch: the theme + picker, the security-notes screen, the "trust this folder?" dialog, and a one-time warning + about running with permissions bypassed — none of which a real, already-used profile shows + again. Codeman now pre-seeds that same "already been through this once" state (onboarding + completed, this session's own project marked trusted, the bypass-permissions warning + acknowledged) so a custom-model launch reaches the actual conversation exactly as fast as a + native cloud one does, instead of stopping at a wizard with nobody there to click through it. + +**If a model's real context is too small for Claude Code to even get started, you get a +warning instead of a confusing failure.** Claude Code's own system prompt and tools take up +roughly 40K tokens on their own, before you've typed anything — a small local model with a +smaller real context than that fails outright on the very first message, no matter what +context size Codeman tells it to expect (raising the declared context only changes when +Claude Code trims _conversation history_, and there is none yet on message one). Picking +such a model now shows an in-app dialog naming the model, its discovered context and what's +needed, before anything launches or restarts, with the fix spelled out: reconfigure +llama-swap to give that model (or a smaller one) an explicit larger context instead of +relying on auto-fit (`--fit-ctx`), which sizes the context around fitting the biggest model +rather than the biggest context — for example adding `-c 65536` to that model's llama-swap +entry. "Launch anyway" is still there if you want to try regardless. + +## Which harnesses actually work + +| Harness | Status | +| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Claude Code, opencode, Pi, Grok, OMP** | Verified end-to-end against a real local server. | +| **Codex** | Config is correct, and plain chat can work against a server that speaks the Responses API — but a real tool-call attempt comes back as inert text instead of running, so it's still not usable for real coding work. | +| **Gemini** | Fails with an auth error gemini-cli raises once redirected. Unresolved; don't rely on it yet. | +| **DeepSeek** | The original 404 is root-caused and fixed (DeepSeek Harness's own code was missing a `/v1` most local servers require) — not yet re-run against a real `dsh` install to confirm end-to-end. | +| **Antigravity** | No known custom-endpoint mechanism at all. Not offered. | + +Which harnesses show up in the Run-menu picker is read live off Codeman's own CLI registry, +not a fixed list here, so this table can go stale before this page does — a greyed-out or +missing entry is the more current answer. + +## What it does not do + +- **No remote or Docker sessions yet.** Both restart their agent differently under the hood + (reattaching a durable tmux session rather than relaunching the process), so redirecting + them needs its own plumbing that hasn't been built. +- **No live hot-swap mid-conversation.** Applying a selection always restarts the process. +- **No button to un-point a session from the UI yet.** Clearing back to native cloud is an + HTTP call (`POST .../custom-model {"clear": true}`) or deleting the session; the settings + panel manages saved endpoints, not what a running session is currently pointed at. +- **Nothing is shared with your real cloud credentials.** The endpoint's own key, if any, + never touches your Anthropic/OpenAI/Google login — a custom endpoint is a separate, + explicit choice per session. + +## Security + +An endpoint's base URL can't point at a link-local or cloud-metadata address (both at save +time and against the address it actually resolves to), the same guard Web Tabs uses for +saved dashboards. Endpoint records and any per-session config files a harness needs are +written with owner-only permissions. See +[custom-model-endpoints-plan.md](https://github.com/Ark0N/Codeman/blob/master/docs/custom-model-endpoints-plan.md) +in the repository for the full design reasoning, including why this feature closed a +pre-existing gap in how session environment overrides were guarded rather than opening a new +one. diff --git a/docs/wiki/Settings-Reference.md b/docs/wiki/Settings-Reference.md index 3255f56fb..14b04056e 100644 --- a/docs/wiki/Settings-Reference.md +++ b/docs/wiki/Settings-Reference.md @@ -92,6 +92,10 @@ Model and effort are both **soft defaults**: the model is written into the case' `.claude/settings.local.json` and effort is passed at start, so `/model` and `/effort` inside a session override them at any time. +**Custom model endpoints** (off by default) adds a saved-endpoint list plus a matching +section to the Run dropdown, for pointing a harness at your own OpenAI-compatible server +instead of its native cloud backend. See [Custom Model Endpoints](Custom-Model-Endpoints). + ### Agents & CLIs | Setting | Notes | diff --git a/docs/wiki/_Sidebar.md b/docs/wiki/_Sidebar.md index af84a756e..2e6d54b51 100644 --- a/docs/wiki/_Sidebar.md +++ b/docs/wiki/_Sidebar.md @@ -12,6 +12,7 @@ - [The Dashboard](The-Dashboard) - [Agent CLIs](Agent-CLIs) +- [Custom Model Endpoints](Custom-Model-Endpoints) - [Working With Files](Working-With-Files) - [Input And Voice](Input-And-Voice) - [Mobile Guide](Mobile-Guide) diff --git a/src/config/cli-registry/schema.ts b/src/config/cli-registry/schema.ts index 52cdfbd20..d12493847 100644 --- a/src/config/cli-registry/schema.ts +++ b/src/config/cli-registry/schema.ts @@ -340,6 +340,35 @@ const capabilitiesSchema = z // an env var, so it declares baseUrl/apiKey injection with no model var at all. modelVars: z.array(envName).max(8), launchModel: launchModelTemplate, + // Optional: the env var to carry a discovered per-model context-window size + // (claude's CLAUDE_CODE_MAX_CONTEXT_TOKENS), and/or the env var that isolates + // this session's config/credential directory from the user's real one (claude's + // CLAUDE_CONFIG_DIR) so an injected API key never collides with a stored OAuth + // session. See the customModelInjection doc comment in cli-registry/types.ts. + contextLengthVar: envName.optional(), + configDirVar: envName.optional(), + // Relative path, WITHIN the isolated configDirVar directory, of a trust-dialog + // seed file the CLI itself owns the shape of — claude's `.claude.json` + // `customApiKeyResponses.approved` list, the same field an interactive "Detected + // a custom API key — use it?" prompt writes to on a real terminal. Only makes + // sense alongside configDirVar (an isolated, otherwise-empty directory has none + // of a real profile's prior approvals), and only implemented for the + // 'claude-api-key-responses' shape today — see custom-model-injection-apply.ts. + apiKeyTrustFile: z + .object({ relPath: z.string().min(1).max(80), shape: z.literal('claude-api-key-responses') }) + .strict() + .optional(), + // An isolated config directory replays the CLI's whole first-run sequence (theme + // picker, security notes, per-project trust dialog, bypass-permissions warning) + // on every launch, same root cause as apiKeyTrustFile above — this reuses that + // same file to pre-seed the state a real, already-onboarded profile carries. See + // the customModelInjection doc comment in cli-registry/types.ts. + skipFirstRunPrompts: z.boolean().optional(), + // DeepSeek-only, confirmed by reading its own bundled SDK source: it concatenates + // "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/chat/completions" onto baseUrlVar's value with no "/v1" of its own, while + // llama-swap/llama.cpp only serves the "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/v1/..." path — claude/gemini must NOT + // get this. See the customModelInjection doc comment in cli-registry/types.ts. + appendV1Suffix: z.boolean().optional(), }) .strict(), z diff --git a/src/config/cli-registry/stock.ts b/src/config/cli-registry/stock.ts index eb0540829..4a357eda6 100644 --- a/src/config/cli-registry/stock.ts +++ b/src/config/cli-registry/stock.ts @@ -237,6 +237,12 @@ const CLAUDE: CliEntry = { 'ANTHROPIC_DEFAULT_SONNET_MODEL', 'ANTHROPIC_DEFAULT_HAIKU_MODEL', 'ANTHROPIC_DEFAULT_OPUS_MODEL', + // CLAUDE_CODE_MAX_CONTEXT_TOKENS already matches the CLAUDE_CODE_* allowedPrefix, and + // CLAUDE_CONFIG_DIR is already an allowed exact key (docs/wiki/Agent-CLIs.md), so both + // were already reachable via plain envOverrides before this pair existed — listed here + // only so the custom-model route clamps them the same way as every other injected var. + 'CLAUDE_CODE_MAX_CONTEXT_TOKENS', + 'CLAUDE_CONFIG_DIR', ], gates: { nameFlag: { minVersion: '2.1.224', failClosed: true } }, // Custom Model Endpoint Profiles (docs/custom-model-endpoints-plan.md) — verified by hand against a real @@ -247,6 +253,32 @@ const CLAUDE: CliEntry = { baseUrlVar: 'ANTHROPIC_BASE_URL', apiKeyVar: 'ANTHROPIC_API_KEY', modelVars: ['ANTHROPIC_DEFAULT_SONNET_MODEL', 'ANTHROPIC_DEFAULT_HAIKU_MODEL', 'ANTHROPIC_DEFAULT_OPUS_MODEL'], + // Verified via Claude Code's own docs: CLAUDE_CODE_MAX_CONTEXT_TOKENS overrides the + // assumed context window and applies directly for a model name Claude Code doesn't + // recognize as one of its own — exactly the custom-model case. Without it, Claude Code + // assumes a large (200k) window for any unrecognized model id and never compacts, + // eventually overflowing a much smaller real local context (see plan doc reasoning + // above the interface for the confirmed failure). + contextLengthVar: 'CLAUDE_CODE_MAX_CONTEXT_TOKENS', + // Isolates this session's config/credential directory so an injected ANTHROPIC_API_KEY + // never shares a directory with a stored claude.ai OAuth login — see the doc comment on + // customModelInjection in cli-registry/types.ts for the traded-off side effect. + configDirVar: 'CLAUDE_CONFIG_DIR', + // ⚠️ Required alongside configDirVar, not optional in practice: verified live that an + // isolated, otherwise-empty config directory makes claude stop at an interactive + // "Detected a custom API key — use it?" prompt on EVERY launch, defaulting to "No" with + // no one at the TTY to answer — silently refusing the very key this feature injected. + // Pre-seeding this file's customApiKeyResponses.approved list (verified against a real + // ~/.claude.json after answering the prompt once by hand) answers it in advance instead. + apiKeyTrustFile: { relPath: '.claude.json', shape: 'claude-api-key-responses' }, + // ⚠️ Same isolated-directory root cause, one step further: verified live that on top + // of the API-key prompt above, a fresh CLAUDE_CONFIG_DIR also replays claude's ENTIRE + // first-run sequence on every launch — the theme picker, the security-notes screen, + // the per-project "trust this folder?" dialog, and (running with + // --dangerously-skip-permissions) a one-time bypass-permissions warning — none of + // which a real, already-onboarded profile shows again. Pre-seeds that same + // already-onboarded state instead of leaving a human to click through it. + skipFirstRunPrompts: true, }, }, overlays: { @@ -1071,15 +1103,28 @@ const DEEPSEEK: CliEntry = { // privilege rather than granting it, and clamping it here was a real regression // (test/deepseek-mode.test.ts) fixed before this shipped. privilegedEnvKeys: ['DSH_PERMISSION_MODE', 'DSH_HOME', 'DEEPSEEK_BASE_URL'], - // Web-researched, unverified, partial: reuses the already-existing DEEPSEEK_BASE_URL/ - // DEEPSEEK_API_KEY keys above. No modelVars — dsh's model is a profile-composition - // entry (see `model: { source: 'none' }` above), not an env var, so forcing a specific - // model name may not fully work; verify against a real profile before shipping. + // Reuses the already-existing DEEPSEEK_BASE_URL/DEEPSEEK_API_KEY keys above. No + // modelVars — dsh's model is a profile-composition entry (see `model: { source: 'none' + // }` above), not an env var, so forcing a specific model name may not fully work; + // verify against a real profile before shipping. + // + // ⚠️ appendV1Suffix is REQUIRED, not optional-nice-to-have: without it every request + // 404s. Confirmed live and by reading dsh's own bundled source + // (@deepseek-ai/dsh-llm-deepseek): it builds the request URL as + // `${DEEPSEEK_BASE_URL}/chat/completions` with no "/v1" of its own (its real public + // API, https://api.deepseek.com, expects the caller's base URL to already carry any + // needed prefix), while llama-swap/llama.cpp only serves the OpenAI-conventional + // "/v1/chat/completions" — a bare POST to ".../chat/completions" 404s live, and the + // 404 reported here originally ("dsh: HTTP_404: DeepSeek API error (HTTP 404)") + // matches dsh's own error-message template for exactly this failure. See the + // customModelInjection doc comment in cli-registry/types.ts for the full reasoning, + // including why claude/gemini must NOT get this. customModelInjection: { kind: 'env', baseUrlVar: 'DEEPSEEK_BASE_URL', apiKeyVar: 'DEEPSEEK_API_KEY', modelVars: [], + appendV1Suffix: true, }, }, overlays: { diff --git a/src/config/cli-registry/types.ts b/src/config/cli-registry/types.ts index 1ba07cb02..871a3762e 100644 --- a/src/config/cli-registry/types.ts +++ b/src/config/cli-registry/types.ts @@ -496,9 +496,74 @@ export interface CliCapabilities { * declares). Absent = the config alone selects the model (claude's env vars, * opencode's blob, codex's top-level `model` key). Applied by the session's * respawn options through the entry's `legacyConfigField`, never by id. + * + * `contextLengthVar` (env kind only): the env var a discovered per-model context-window + * size is written to when known (claude's `CLAUDE_CODE_MAX_CONTEXT_TOKENS`) — without it, + * a CLI that assumes a large default window for an unrecognized model name keeps sending + * full-size prompts against a much smaller local server and eventually overflows its real + * context (verified: a 33.7K-token system prompt against a 16384-token llama-swap model). + * Absent when the CLI has no such override, or the value is unknown for this model. + * + * `configDirVar` (env kind only): the env var that redirects this session's config/ + * credential directory to an isolated, per-session one (claude's `CLAUDE_CONFIG_DIR`), so + * an injected API key never coexists with a stored claude.ai OAuth session in the same + * directory — the CLI still warns "both claude.ai and ANTHROPIC_API_KEY set" when they + * share a directory even though the API key wins for actual requests. Isolating it trades + * that cosmetic warning for a documented side effect: a relocated config directory writes + * transcripts outside `~/.claude/projects`, blinding the response viewer, subagent + * windows, and Read My Mind for that session (see docs/wiki/Agent-CLIs.md). + * + * `apiKeyTrustFile` (env kind only, alongside configDirVar): an isolated config directory + * has none of a real profile's prior "detected a custom API key, use it?" approvals, so + * without this the CLI stops and asks interactively on every single launch — with no one + * at a TTY to answer, that's a hang, not a warning (confirmed live: claude's own default + * answer, "No", would silently refuse to use the very key this feature just injected). + * `relPath`/`shape` name the file (claude's `.claude.json`) and its + * `customApiKeyResponses.approved` field this pre-seeds — the exact field a real answered + * prompt itself writes to, so this isn't bypassing the check, just answering it the same + * way a one-off prior approval on a shared profile already would. + * + * `skipFirstRunPrompts` (env kind only, alongside apiKeyTrustFile): an isolated config + * directory is not just missing API-key approvals — it is a brand-new profile as far as + * the CLI is concerned, so it also replays its ENTIRE first-run sequence on every launch: + * the theme picker, the security-notes screen, the per-project "trust this folder?" + * dialog, and (running with a bypass-permissions flag) a one-time warning about it — + * confirmed live, none of which a real, long-used profile ever shows again. `true` + * pre-seeds the same state a real profile accumulates from having answered all of that + * once: `hasCompletedOnboarding` and the launching session's own project entry in the + * `apiKeyTrustFile` (claude's `.claude.json`), plus `skipDangerousModePermissionPrompt` + * in claude's `settings.json` — see `seedFirstRunState`/`seedSkipBypassPermissionsPrompt` + * in custom-model-injection-apply.ts. Requires `apiKeyTrustFile` to be set too, since it + * reuses that file. + * + * `appendV1Suffix` (env kind only): the raw `endpoint.baseUrl` gets `withV1Suffix()` + * applied before being written to `baseUrlVar`, instead of being used verbatim. + * DeepSeek needs this and claude/gemini must NOT get it — a per-CLI asymmetry confirmed + * by reading each SDK's own request-building source, not assumed: DeepSeek Harness's + * bundled `@deepseek-ai/dsh-llm-deepseek` concatenates `${connection.baseURL}/chat/ + * completions` with no `/v1` insertion of its own (its real public API base, + * `https://api.deepseek.com`, expects the caller's base URL to already carry any + * needed prefix), while llama-swap/llama.cpp only ever serves the OpenAI-conventional + * `/v1/chat/completions` — confirmed live: a bare `POST /chat/completions` + * 404s, `POST /v1/chat/completions` succeeds, and the harness's own error + * message template (`DeepSeek API error (HTTP ${status})`) reproduces the exact + * `HTTP_404` this feature originally shipped with unexplained. Claude Code's own SDK, + * by contrast, was already confirmed working end-to-end against the RAW `baseUrl` with + * no suffix — appending one there would be wrong, not just redundant. */ customModelInjection: - | { kind: 'env'; baseUrlVar: string; apiKeyVar: string; modelVars: string[]; launchModel?: string } + | { + kind: 'env'; + baseUrlVar: string; + apiKeyVar: string; + modelVars: string[]; + launchModel?: string; + contextLengthVar?: string; + apiKeyTrustFile?: { relPath: string; shape: 'claude-api-key-responses' }; + configDirVar?: string; + skipFirstRunPrompts?: boolean; + appendV1Suffix?: boolean; + } | { kind: 'configContentEnv'; envVar: string; template: 'opencode-json'; launchModel?: string } | { kind: 'configDir'; diff --git a/src/custom-model-hosts.ts b/src/custom-model-hosts.ts index 0cde1039a..61ef75f78 100644 --- a/src/custom-model-hosts.ts +++ b/src/custom-model-hosts.ts @@ -40,6 +40,38 @@ export interface CustomModelHost { authStyle?: CustomModelAuthStyle; models?: string[]; lastDiscoveredAt?: string; + /** + * The model the Run-menu picker (docs/custom-model-endpoints-plan.md) applies when + * this endpoint is picked with no further choice — one generated menu entry per + * (CLI, endpoint) pair, not per (CLI, endpoint, model), so it needs a single answer. + * Must be a member of `models` when set; the picker falls back to `models[0]` when + * this is unset, and disables the entry entirely when `models` is empty (nothing to + * default to). Never auto-set on discovery — the previous default staying valid + * after a re-discover is a property worth keeping even if the model list changes. + */ + defaultModelId?: string; + /** + * Discovered context-window size (tokens) per model id, keyed by the same strings as + * `models`. Populated opportunistically during discovery (`custom-model-routes.ts`) from + * llama.cpp/llama-swap's `GET /props?model=` — the plain OpenAI-shaped `/v1/models` + * response has no such field. Only ever probed for a model the server already reports as + * loaded (llama-swap's `status.value === 'loaded'`); an unloaded one is deliberately never + * probed, since llama-swap treats `/props?model=` as a routing hint that can trigger an + * actual (slow, GPU-swapping) model load as a side effect of merely asking. A model this + * has no entry for simply gets no context-length env override applied — never a guess. + */ + modelContextLengths?: Record; + /** + * Discovered file size (GB) per model id, keyed by the same strings as `models`. + * Populated during discovery by parsing llama-swap's own `description` field for an + * auto-discovered model ("Auto-discovered 16.35 GB - parameters auto-fitted by + * llama.cpp") — a hand-configured profile's own description has no such figure and + * correctly gets no entry, never a guess. Used only to label the Run-menu picker's + * "loading model" banner with a rough, unmeasured expected-time estimate + * (`estimateModelLoad()` in session-ui.js) — never a guarantee, and never anything a + * server-side check relies on. + */ + modelSizesGB?: Record; } export function customModelHostsPath(configDir: string): string { diff --git a/src/custom-model-injection-apply.ts b/src/custom-model-injection-apply.ts index 2df6f57d2..51ca6fbc7 100644 --- a/src/custom-model-injection-apply.ts +++ b/src/custom-model-injection-apply.ts @@ -10,7 +10,8 @@ * cli-registry changes" requirement it was written against. */ -import { chmodSync, mkdirSync, writeFileSync, rmSync } from 'node:fs'; +import { chmodSync, existsSync, mkdirSync, readFileSync, writeFileSync, rmSync, symlinkSync } from 'node:fs'; +import { homedir, platform } from 'node:os'; import { join, dirname } from 'node:path'; import { dataPath } from './config/instance.js'; import type { CliEntry } from './config/cli-registry/types.js'; @@ -48,6 +49,148 @@ export function applyConfigDirInjection(baseDir: string, injection: ConfigDirInj return { [injection.dirEnvVar]: baseDir, ...injection.extraEnv }; } +/** + * Real, shared Claude config directory Codeman's own host process runs under — honors + * `CLAUDE_CONFIG_DIR` the same way `claude-credentials.ts`'s `claudeCredentialsPath()` + * does, so the symlink below points at wherever `~/.claude/projects` actually lives + * rather than assuming the plain default. + */ +function realClaudeConfigDir(): string { + const configured = typeof process.env.CLAUDE_CONFIG_DIR === 'string' && process.env.CLAUDE_CONFIG_DIR.trim(); + return configured || join(homedir(), '.claude'); +} + +/** + * Symlinks `/projects` back to the real, shared `~/.claude/projects`, so an + * isolated `CLAUDE_CONFIG_DIR` (used to keep an injected API key away from a stored OAuth + * session — see `configDirVar` on customModelInjection) doesn't also blind the response + * viewer, subagent windows, and Read My Mind for that session (docs/wiki/Agent-CLIs.md). + * Best-effort: a platform that refuses symlinks (unprivileged Windows without a junction + * fallback working, e.g.) just keeps the pre-existing documented side effect instead of + * failing the whole custom-model apply over a nice-to-have. + */ +function linkSharedProjectsDir(isolatedDir: string): void { + const link = join(isolatedDir, 'projects'); + if (existsSync(link)) return; // already linked (idempotent re-apply) or real dir wrote one + try { + symlinkSync(join(realClaudeConfigDir(), 'projects'), link, platform() === 'win32' ? 'junction' : 'dir'); + } catch { + // best-effort only — response viewer/subagent windows go blind for this session instead + } +} + +/** + * Pre-approves the injected API key in an isolated config directory's trust-dialog state + * (`customModelInjection.apiKeyTrustFile`), so an otherwise-empty directory doesn't make the + * CLI stop at an interactive "Detected a custom API key — use it?" prompt on every single + * launch. Confirmed live: with nobody at the TTY to answer, that prompt's own default + * ("No") silently refuses the very key this feature just injected — this isn't bypassing + * the check, it's answering it the same field a real answered prompt itself writes to + * (verified against a real `~/.claude.json` after answering by hand once). + * + * Merges rather than overwrites: the file may already carry fields the CLI itself wrote on + * an earlier launch in this same isolated directory (machineID, userID, other approved + * keys), and a corrupt or partially-written file (a crash mid-write) is treated as absent + * rather than failing the whole apply over a nice-to-have. + */ +function seedApiKeyTrustFile( + configDir: string, + trustFile: { relPath: string; shape: 'claude-api-key-responses' }, + apiKey: string +): void { + const filePath = join(configDir, trustFile.relPath); + let existing: Record = {}; + try { + existing = JSON.parse(readFileSync(filePath, 'utf8')) as Record; + } catch { + existing = {}; + } + const responses = (existing.customApiKeyResponses ?? {}) as { approved?: unknown; rejected?: unknown }; + const approved = new Set(Array.isArray(responses.approved) ? (responses.approved as string[]) : []); + approved.add(apiKey); + const rejected = Array.isArray(responses.rejected) ? responses.rejected : []; + existing.customApiKeyResponses = { approved: [...approved], rejected }; + try { + writeFileSync(filePath, JSON.stringify(existing, null, 2), { encoding: 'utf8', mode: 0o600 }); + chmodSync(filePath, 0o600); + } catch { + // best-effort only — the interactive prompt returns instead of a hard failure here + } +} + +/** + * Pre-seeds the two remaining pieces of "already been onboarded" state a fresh + * `CLAUDE_CONFIG_DIR` has none of (`customModelInjection.skipFirstRunPrompts`, alongside + * apiKeyTrustFile): claude replays its whole first-run sequence — the theme picker, the + * security-notes screen, and (per-project) the "trust this folder?" dialog — against ANY + * config directory that has never completed it, confirmed live against a genuinely fresh + * isolated directory. `hasCompletedOnboarding` skips the theme/security-notes screens + * outright; `projects[workingDir].hasTrustDialogAccepted` answers the trust dialog for + * THIS session's own working directory the same way a real profile's own prior approval + * would — other projects in the file are left alone, and `workingDir` is used verbatim + * (never realpath'd or slash-normalized) since that's the literal string claude itself + * uses as the project key, being whatever string the session was actually launched with + * as its cwd. + * + * Same merge-not-overwrite and corrupt-file-tolerant behavior as `seedApiKeyTrustFile` + * (same file, so a second sequential read-modify-write here is deliberate rather than + * folding both into one pass — keeps each seed independently testable and optional). + */ +function seedFirstRunOnboardingState( + configDir: string, + trustFile: { relPath: string; shape: 'claude-api-key-responses' }, + workingDir: string +): void { + const filePath = join(configDir, trustFile.relPath); + let existing: Record = {}; + try { + existing = JSON.parse(readFileSync(filePath, 'utf8')) as Record; + } catch { + existing = {}; + } + existing.hasCompletedOnboarding = true; + const projects = + existing.projects && typeof existing.projects === 'object' && !Array.isArray(existing.projects) + ? (existing.projects as Record>) + : {}; + const existingProject = projects[workingDir] && typeof projects[workingDir] === 'object' ? projects[workingDir] : {}; + projects[workingDir] = { ...existingProject, hasTrustDialogAccepted: true }; + existing.projects = projects; + try { + writeFileSync(filePath, JSON.stringify(existing, null, 2), { encoding: 'utf8', mode: 0o600 }); + chmodSync(filePath, 0o600); + } catch { + // best-effort only — the interactive dialogs return instead of a hard failure here + } +} + +/** + * Pre-seeds the "skip the bypass-permissions warning" setting (`customModelInjection. + * skipFirstRunPrompts`, alongside apiKeyTrustFile) into an isolated config directory's + * `settings.json` — a real, already-onboarded profile answers claude's one-time warning + * about running with a bypass-permissions flag once and never sees it again, but every + * custom-model session launches with a fresh, otherwise-empty CLAUDE_CONFIG_DIR that + * carries none of that (confirmed live). A different file from apiKeyTrustFile's + * `.claude.json` — this is claude's own global `settings.json`, not project-keyed — + * so it gets its own merge-not-overwrite read-modify-write. + */ +function seedSkipBypassPermissionsPrompt(configDir: string): void { + const filePath = join(configDir, 'settings.json'); + let existing: Record = {}; + try { + existing = JSON.parse(readFileSync(filePath, 'utf8')) as Record; + } catch { + existing = {}; + } + existing.skipDangerousModePermissionPrompt = true; + try { + writeFileSync(filePath, JSON.stringify(existing, null, 2), { encoding: 'utf8', mode: 0o600 }); + chmodSync(filePath, 0o600); + } catch { + // best-effort only — the interactive warning returns instead of a hard failure here + } +} + /** Best-effort recursive removal of a previously-written configDir. Never throws. */ export function removeConfigDir(dir: string | undefined): void { if (!dir) return; @@ -79,14 +222,43 @@ export function applyCustomModelInjection( entry: Pick, endpoint: CustomModelEndpoint, modelId: string, - sessionId: string + sessionId: string, + /** Discovered context-window size for `modelId`, if known — see `contextLengthVar`. */ + contextLength?: number, + /** + * The session's own working directory — only used for `skipFirstRunPrompts`'s per-project + * trust-dialog seed, and only when provided (boot recovery, which has no reason to + * re-answer a dialog that already fired once, omits it rather than re-deriving it). + */ + workingDir?: string ): AppliedCustomModel | undefined { - const injection = buildCustomModelInjection(entry, endpoint, modelId); + const injection = buildCustomModelInjection(entry, endpoint, modelId, contextLength); if (injection.kind === 'unsupported') return undefined; if (injection.kind === 'env') { + // `configDirVar` (claude's CLAUDE_CONFIG_DIR): point it at the same isolated, + // per-session directory the `configDir` kind uses, but write no files into it — an + // empty directory has no stored OAuth credential to conflict with the injected API + // key, which is the whole point. Reusing the same path keyed by sessionId keeps this + // idempotent across a boot-recovery re-apply, same as the configDir kind below. + let envOverrides = injection.envOverrides; + let configDir: string | undefined; + if (injection.configDirVar) { + configDir = customModelConfigDir(sessionId); + mkdirSync(configDir, { recursive: true, mode: 0o700 }); + linkSharedProjectsDir(configDir); + if (injection.apiKeyTrustFile && injection.apiKey) { + seedApiKeyTrustFile(configDir, injection.apiKeyTrustFile, injection.apiKey); + } + if (injection.skipFirstRunPrompts && injection.apiKeyTrustFile) { + if (workingDir) seedFirstRunOnboardingState(configDir, injection.apiKeyTrustFile, workingDir); + seedSkipBypassPermissionsPrompt(configDir); + } + envOverrides = { ...envOverrides, [injection.configDirVar]: configDir }; + } return { - envOverrides: injection.envOverrides, - envKeys: Object.keys(injection.envOverrides), + envOverrides, + envKeys: Object.keys(envOverrides), + configDir, launchModel: injection.launchModel, }; } diff --git a/src/custom-model-injection.ts b/src/custom-model-injection.ts index 5f46001c7..9aab94227 100644 --- a/src/custom-model-injection.ts +++ b/src/custom-model-injection.ts @@ -16,14 +16,21 @@ * shape was rejected by a real codex binary with "invalid type: map, * expected a string" — caught by `scripts/test-local-llm-harnesses.ts`), * but `wire_api = "responses"` is the only value codex still accepts - * (support for `"chat"` was dropped in Feb 2026), and a plain OpenAI - * Chat-Completions server (llama.cpp, llama-swap, most local setups) does - * NOT implement the Responses API — so codex may still fail at the - * PROTOCOL level even with a correctly-shaped config file. That gap is - * real and current, not a stale warning; see docs/custom-model-endpoints-plan.md. The rest - * (gemini/pi/grok/deepseek/omp) have their ONE-SHOT INVOCATION flags - * confirmed against real installed binaries' own `--help` output, but - * their custom-endpoint env/config conventions remain web-researched, + * (support for `"chat"` was dropped in Feb 2026). ⚠️ Re-verified live + * against a llama-swap deployment that DOES answer `/v1/responses`: a + * plain, no-tool-call turn gets a real reply, but a real tool-call attempt + * comes back as `agent_message` TEXT (the tool-call JSON printed as the + * answer) rather than a `function_call` item codex would execute — + * confirmed via `codex exec --json`'s raw event stream. Tool execution is + * what makes codex a coding agent, so this remains not usable for real + * work even where plain chat succeeds; see docs/custom-model-endpoints-plan.md + * for the full picture (including the harmless `Model metadata ... not + * found` warning every custom-endpoint codex session prints — sourced from + * a local cache of OpenAI's OWN hosted model catalog that a custom model + * can never appear in, confirmed to have no effect on the outcome above). + * The rest (gemini/pi/grok/deepseek/omp) have their ONE-SHOT INVOCATION + * flags confirmed against real installed binaries' own `--help` output, + * but their custom-endpoint env/config conventions remain web-researched, * unverified. */ @@ -44,6 +51,20 @@ export interface EnvInjection { envOverrides: Record; /** See {@link ConfigDirInjection.launchModel}. */ launchModel?: string; + /** + * Name of the env var the caller should point at an isolated, credential-free config + * directory for this session (claude's `CLAUDE_CONFIG_DIR`), from the registry entry's + * `customModelInjection.configDirVar`. The actual directory value isn't computed here — + * this module is pure and has no sessionId to derive one from — the IO wrapper + * (`custom-model-injection-apply.ts`) creates it and adds it to `envOverrides`. + */ + configDirVar?: string; + /** See `customModelInjection.apiKeyTrustFile` — carried through so the IO wrapper can seed it. */ + apiKeyTrustFile?: { relPath: string; shape: 'claude-api-key-responses' }; + /** The literal API key value this injection used, for `apiKeyTrustFile` to pre-approve. */ + apiKey?: string; + /** See `customModelInjection.skipFirstRunPrompts` — carried through so the IO wrapper can seed it. */ + skipFirstRunPrompts?: boolean; } export interface ConfigDirInjection { @@ -90,7 +111,9 @@ function quoted(value: string): string { export function buildCustomModelInjection( entry: Pick, endpoint: CustomModelEndpoint, - modelId: string + modelId: string, + /** Discovered context-window size for `modelId`, if known — see `contextLengthVar`. */ + contextLength?: number ): CustomModelInjectionResult { const cap = entry.capabilities.customModelInjection; const apiKey = endpoint.apiKey?.trim() || DEFAULT_API_KEY; @@ -98,11 +121,18 @@ export function buildCustomModelInjection( switch (cap.kind) { case 'env': { const envOverrides: Record = { - [cap.baseUrlVar]: endpoint.baseUrl, + [cap.baseUrlVar]: cap.appendV1Suffix ? withV1Suffix(endpoint.baseUrl) : endpoint.baseUrl, [cap.apiKeyVar]: apiKey, }; for (const modelVar of cap.modelVars) envOverrides[modelVar] = modelId; - return withLaunchModel({ kind: 'env', envOverrides }, cap.launchModel, modelId); + if (cap.contextLengthVar && contextLength !== undefined && Number.isFinite(contextLength)) { + envOverrides[cap.contextLengthVar] = String(Math.trunc(contextLength)); + } + let result: EnvInjection = withLaunchModel({ kind: 'env', envOverrides }, cap.launchModel, modelId); + if (cap.configDirVar) result = { ...result, configDirVar: cap.configDirVar }; + if (cap.apiKeyTrustFile) result = { ...result, apiKeyTrustFile: cap.apiKeyTrustFile, apiKey }; + if (cap.skipFirstRunPrompts) result = { ...result, skipFirstRunPrompts: true }; + return result; } case 'configContentEnv': { @@ -177,11 +207,13 @@ function renderConfigFile( // `env_key`, the NAME of an env var it reads the credential from at runtime, so the // actual value must ride along as an extra env var, never embedded in the file. // ⚠️ `wire_api = "responses"` is the only value codex still accepts (it dropped - // `"chat"` support in Feb 2026) — a plain OpenAI Chat-Completions server (llama.cpp, - // llama-swap, most local setups) does NOT implement the Responses API, so this - // recipe may still fail at the PROTOCOL level even though the file now parses - // correctly. That is a real, currently-unresolved compatibility gap, not a syntax - // bug — track it before calling codex support done. + // `"chat"` support in Feb 2026). Even against a llama-swap deployment that DOES + // answer `/v1/responses`, a real tool-call attempt came back as plain TEXT (the + // tool-call JSON printed as the model's answer) rather than an executable + // `function_call` item — confirmed live via `codex exec --json`. Tool execution is + // what makes codex a coding agent, so this remains not usable for real work even + // where plain chat succeeds — see the confidence table in + // docs/custom-model-endpoints-plan.md, not a syntax bug in this file. const content = [ `model = ${quoted(modelId)}`, `model_provider = "custom"`, diff --git a/src/web/public/app.js b/src/web/public/app.js index 4c6888997..44d243895 100644 --- a/src/web/public/app.js +++ b/src/web/public/app.js @@ -1709,6 +1709,25 @@ class CodemanApp { console.error('[SSE] docker container recreated:', err); } }); + // Custom Model Endpoint Profiles: a session's own model got evicted on llama-swap by + // another session's activity, detected AFTER the fact by a periodic server sweep (there + // is no push notification from llama-swap itself) — see detectCustomModelSwapDisplacements + // in custom-model-routes.ts. Global toast rather than a per-tab indicator: the displaced + // session need not be the one currently open, and the whole point is telling the user + // BEFORE they type into it expecting the model they picked. + addListener(SSE_EVENTS.CUSTOM_MODEL_SWAPPED_OUT, (e) => { + try { + const d = e.data ? JSON.parse(e.data) : {}; + this.showToast( + `${d.sessionName || d.sessionId}'s model (${d.previousModel}) was swapped out on llama-swap by another ` + + `session — currently loaded: ${d.currentlyLoadedModel}. Sending a message there will reload it.`, + 'warning', + { duration: 0 } + ); + } catch (err) { + console.error('[SSE] custom model swapped out:', err); + } + }); // Multi-user admin: live-refresh whichever admin views (panel/Users tab) are open. addListener(SSE_EVENTS.ADMIN_USERS_CHANGED, () => { window.codemanAdmin?.onUsersChanged?.(); diff --git a/src/web/public/constants.js b/src/web/public/constants.js index be9780b04..f661f13ad 100644 --- a/src/web/public/constants.js +++ b/src/web/public/constants.js @@ -1094,6 +1094,9 @@ const SSE_EVENTS = { APPROVAL_UPDATED: 'approval:updated', APPROVAL_RESOLVED: 'approval:resolved', + // Custom Model Endpoint Profiles + CUSTOM_MODEL_SWAPPED_OUT: 'custom-model:swapped-out', + // Subagents (Claude Code background agents) SUBAGENT_DISCOVERED: 'subagent:discovered', SUBAGENT_UPDATED: 'subagent:updated', diff --git a/src/web/public/i18n.js b/src/web/public/i18n.js index bee8fcab7..3a0fbc5b9 100644 --- a/src/web/public/i18n.js +++ b/src/web/public/i18n.js @@ -286,6 +286,34 @@ 'Prompt sent': '提示已发送', 'Inserted, press Enter in the terminal to send': '已插入，在终端中按 Enter 发送', 'Could not reach the session': '无法连接到会话', + 'Custom model endpoints': '自定义模型端点', + 'Point a harness at your own OpenAI-compatible server (llama.cpp, vLLM, DGX Spark, Azure AI Foundry, OpenRouter) instead of its native cloud backend. When on, the Run menu offers an extra entry per harness that supports it, per saved endpoint.': + '让工具指向您自己的兼容 OpenAI 服务器（llama.cpp、vLLM、DGX Spark、Azure AI Foundry、OpenRouter），而非其原生云端后端。开启后，"运行"菜单会为每个支持此功能的工具、每个已保存的端点新增一个条目。', + 'Enable custom model endpoints': '启用自定义模型端点', + 'Adds a per-endpoint entry to the Run menu for every harness that can redirect to one.': + '为每个可重定向到端点的工具，在"运行"菜单中添加对应条目。', + 'No endpoints yet. Add one below to point a harness at a local or cloud OpenAI-compatible server.': + '暂无端点。请在下方添加一个，以便将工具指向本地或云端的兼容 OpenAI 服务器。', + Discover: '发现模型', + '+ Add endpoint': '+ 添加端点', + 'Add endpoint': '添加端点', + Id: 'ID', + 'Short, stable — used in URLs, never shown to the CLI.': '简短且固定 — 用于 URL，不会展示给 CLI。', + Label: '标签', + 'Base URL': '基础 URL', + 'API key': 'API 密钥', + 'Optional. Left blank on edit keeps the existing key.': '可选。编辑时留空将保留现有密钥。', + 'Auth header': '认证请求头', + 'Never send both — some servers hang indefinitely.': '切勿同时发送两者 — 部分服务器会因此无限期挂起。', + 'Authorization: Bearer (default)': 'Authorization: Bearer（默认）', + 'api-key header (Azure)': 'api-key 请求头（Azure）', + 'Default model': '默认模型', + 'What the Run-menu picker applies for this endpoint. Discover models first.': + '运行菜单选择器会为此端点应用该模型。请先发现可用模型。', + 'Custom Endpoints': '自定义端点', + 'Choose a model': '选择模型', + 'That endpoint no longer exists': '该端点已不存在', + 'No models discovered for this endpoint yet': '此端点尚未发现任何模型', 'Subagent Options': '子智能体选项', 'Enable Tracking': '启用跟踪', 'Active Tab Only': '仅活动标签页', diff --git a/src/web/public/index.html b/src/web/public/index.html index 2ba80243c..4e847eeeb 100644 --- a/src/web/public/index.html +++ b/src/web/public/index.html @@ -650,6 +650,14 @@

Resume Conversation

+ + + +
+ +
+

+
+
+ + + + + + + + + + +
+

Custom model endpoints

synced
+

Point a harness at your own OpenAI-compatible server (llama.cpp, vLLM, DGX Spark, Azure AI Foundry, OpenRouter) instead of its native cloud backend. When on, the Run menu offers an extra entry per harness that supports it, per saved endpoint.

+
+
+
+ Enable custom model endpoints + Adds a per-endpoint entry to the Run menu for every harness that can redirect to one. +
+ +
+ + +
+
diff --git a/src/web/public/panels-ui.js b/src/web/public/panels-ui.js index 80e6ce838..c8967801e 100644 --- a/src/web/public/panels-ui.js +++ b/src/web/public/panels-ui.js @@ -5484,12 +5484,25 @@ Object.assign(CodemanApp.prototype, { return this.showToast(message, type); }, + /** + * `duration` defaults to 0 (sticky, no auto-dismiss) for `error` toasts and + * 3000ms for everything else — an error worth a distinct visual style is + * also worth reading before it vanishes, which a fixed 3s auto-dismiss + * does not guarantee: "Session started on the native backend — could not + * apply the custom endpoint: " is exactly the kind of + * message that needs a moment to read, not a glance. Every toast gets an + * explicit close button regardless of duration, since a sticky one with no + * way to dismiss it would just pile up. A caller can still override either + * default via `opts.duration` (e.g. a deliberately brief success toast, or + * a non-error one that should also stay put). + */ showToast(message, type = 'info', opts = {}) { - const { duration = 3000, action } = opts; + const { duration = type === 'error' ? 0 : 3000, action } = opts; const toast = document.createElement('div'); toast.className = `toast toast-${type}`; const msgSpan = document.createElement('span'); + msgSpan.className = 'toast-message'; msgSpan.textContent = message; toast.appendChild(msgSpan); @@ -5501,6 +5514,20 @@ Object.assign(CodemanApp.prototype, { toast.appendChild(btn); } + let dismissTimer = null; + const dismiss = () => { + if (dismissTimer) clearTimeout(dismissTimer); + toast.classList.remove('show'); + setTimeout(() => toast.remove(), 200); + }; + + const closeBtn = document.createElement('button'); + closeBtn.className = 'toast-close'; + closeBtn.textContent = '×'; + closeBtn.setAttribute('aria-label', 'Dismiss'); + closeBtn.onclick = (e) => { e.stopPropagation(); dismiss(); }; + toast.appendChild(closeBtn); + // Cache toast container reference if (!this._toastContainer) { this._toastContainer = document.querySelector('.toast-container'); @@ -5514,10 +5541,95 @@ Object.assign(CodemanApp.prototype, { requestAnimationFrame(() => toast.classList.add('show')); - setTimeout(() => { - toast.classList.remove('show'); - setTimeout(() => toast.remove(), 200); - }, duration); + if (duration > 0) { + dismissTimer = setTimeout(dismiss, duration); + } + + // Most callers ignore this — a handle exists for a long-running toast a caller needs + // to update or dismiss itself once its own condition resolves (e.g. a "loading model" + // toast a poll loop dismisses once the model reports ready). + return { dismiss, setMessage: (text) => { msgSpan.textContent = text; } }; + }, + + /** + * A prominent, screen-centred status banner — for the small set of messages that are + * genuinely worth interrupting the eye for rather than living in the corner with every + * other toast (currently: a custom-model session's "switching backends" and "loading + * model" states, both of which can sit on screen for well over a minute and are easy to + * mistake for nothing happening). Non-blocking (`pointer-events: none` on the wrapper, + * restored only on the card) — an info banner is never a gate the user has to dismiss to + * keep working. Only one is ever shown at a time (the DOM node is created once and + * reused), which matches every current caller: each hands off to the next rather than + * stacking. + * + * `opts.type` — `'info'` (default, spinner, no close button — a caller ends it itself via + * `dismiss()`) or `'error'` (no spinner — nothing is in progress once this shows — with a + * close button, since a sticky error the user cannot dismiss would just sit there). The + * DOM is rebuilt fresh each call rather than patched, since which children exist differs + * by type; `setMessage` still only ever touches the text node afterwards. + * + * `opts.onCancel` — when given (any type, but in practice only 'info': an 'error' banner + * already has its own close button), renders a "Cancel" button that calls it on click. + * The callback owns everything that follows (dismissing the banner, stopping whatever + * loop this was showing progress for, closing a session it was for) — this helper only + * renders the button and wires the click, the same "caller decides what cancel means" + * split as `_confirmModelSwap`'s promise-resolving buttons. + */ + _showCenterStatus(message, opts = {}) { + const { type = 'info', onCancel } = opts; + let el = document.getElementById('customModelCenterStatus'); + if (!el) { + el = document.createElement('div'); + el.id = 'customModelCenterStatus'; + document.body.appendChild(el); + } + el.className = `center-status-banner center-status-${type}`; + el.innerHTML = ''; + const dismiss = () => { + el.classList.remove('show'); + setTimeout(() => { + el.hidden = true; + }, 200); + }; + if (type !== 'error') { + const spinner = document.createElement('span'); + spinner.className = 'center-status-spinner'; + spinner.setAttribute('aria-hidden', 'true'); + el.appendChild(spinner); + } + const text = document.createElement('span'); + text.className = 'center-status-text'; + text.textContent = message; + el.appendChild(text); + if (type === 'error') { + const closeBtn = document.createElement('button'); + closeBtn.className = 'center-status-close'; + closeBtn.textContent = '×'; + closeBtn.setAttribute('aria-label', 'Dismiss'); + closeBtn.onclick = (e) => { + e.stopPropagation(); + dismiss(); + }; + el.appendChild(closeBtn); + } else if (onCancel) { + const cancelBtn = document.createElement('button'); + cancelBtn.className = 'center-status-cancel'; + cancelBtn.textContent = 'Cancel'; + cancelBtn.onclick = (e) => { + e.stopPropagation(); + onCancel(); + }; + el.appendChild(cancelBtn); + } + el.hidden = false; + requestAnimationFrame(() => el.classList.add('show')); + return { + dismiss, + setMessage: (next) => { + const t = el.querySelector('.center-status-text'); + if (t) t.textContent = next; + }, + }; }, diff --git a/src/web/public/session-ui.js b/src/web/public/session-ui.js index 67cbfad4c..bc8d71de4 100644 --- a/src/web/public/session-ui.js +++ b/src/web/public/session-ui.js @@ -461,6 +461,7 @@ Object.assign(CodemanApp.prototype, { if (menu.classList.contains('active')) { this._loadRunModeHistory(); this._refreshRunModeAvailability(menu); + this._refreshCustomModelRunOptions(menu); const close = (ev) => { if (!menu.contains(ev.target)) { menu.classList.remove('active'); @@ -534,6 +535,585 @@ Object.assign(CodemanApp.prototype, { if (dsWeb) dsWeb.style.display = avail.deepseekBinary ? 'flex' : 'none'; }, + /** + * Generates the Run menu's Custom Model Endpoint entries + * (docs/custom-model-endpoints-plan.md): one button per (capable harness, saved + * endpoint) pair, e.g. "Claude Code (llama.cpp)". Hidden entirely when the + * feature is off, no endpoint has a usable default model, or the active case is + * remote/docker (the apply route refuses both — see session-routes.ts). + * + * `window.__codemanCustomModelClis` is server-injected at render time from the + * CLI registry's own `capabilities.customModelInjection` (never a hardcoded id + * list here), so a CLI gaining or losing the capability shows up with no + * frontend change. + */ + async _refreshCustomModelRunOptions(menu) { + const sep = menu.querySelector('#runModeCustomModelSep'); + const header = menu.querySelector('#runModeCustomModelHeader'); + const container = menu.querySelector('#runModeCustomModels'); + if (!container) return; + const hide = () => { + if (sep) sep.style.display = 'none'; + if (header) header.style.display = 'none'; + container.innerHTML = ''; + }; + + const settings = this.loadAppSettingsFromStorage(); + // Matches _refreshRunModeAvailability's own gate: a stock entry for an + // uninstalled CLI is hidden, so a generated one must be too, or a box with + // no codex still offers "Codex (llama.cpp)" and fails at launch. + const capableClis = (window.__codemanCustomModelClis || []).filter((cli) => this.isCliAvailable(cli.id)); + if (!settings.customModelEndpointsEnabled || capableClis.length === 0) return hide(); + + const caseName = document.getElementById('quickStartCase')?.value; + const activeCase = caseName ? (this.cases || []).find((c) => c.name === caseName) : null; + if (activeCase?.location === 'remote' || activeCase?.location === 'docker') return hide(); + + // GET /api/model-endpoints wraps its body in the { success, data } envelope + // like every other /api route (server.ts's preSerialization hook applies to + // arrays too) — _apiJson() unwraps it. A raw fetch().json() here would + // silently see the envelope object instead of the array and hide this + // section unconditionally. + const hosts = await this._apiJson('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/api/model-endpoints'); + if (!Array.isArray(hosts) || hosts.length === 0) return hide(); + + const rows = []; + for (const host of hosts) { + const models = host.models || []; + if (models.length === 0) continue; // nothing discovered yet — the settings panel explains why + const modelId = host.defaultModelId || models[0]; + for (const cli of capableClis) { + // escapeHtml(JSON.stringify(...)) on EVERY arg, not just the untrusted + // one: JSON.stringify's own double quotes would otherwise terminate this + // double-quoted attribute at the first one, and everything after parses + // as raw tag content rather than a quoted string — which is what turns + // modelId (server-controlled, from the endpoint's own /v1/models reply, + // not this box's) into markup instead of inert data. Same idiom as + // deleteCase's onclick a few hundred lines down. + const args = [cli.id, host.id].map((v) => escapeHtml(JSON.stringify(v))).join(', '); + rows.push(` + `); + } + } + if (rows.length === 0) return hide(); + if (sep) sep.style.display = ''; + if (header) header.style.display = ''; + container.innerHTML = rows.join(''); + }, + + /** + * Decides whether picking a Run-menu Custom Endpoint entry can launch + * straight away or needs to ask which model first. Re-fetches the endpoint + * rather than trusting anything cached from the menu render: the models + * list (or the default) could have changed — a re-discovery cycle running + * every 5 minutes in the background, or an edit in the settings panel — + * between opening the dropdown and clicking a row. + */ + async selectCustomModelEntry(mode, endpointId) { + document.getElementById('runModeMenu')?.classList.remove('active'); + const hosts = await this._apiJson('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/api/model-endpoints'); + const host = (hosts || []).find((h) => h.id === endpointId); + if (!host) { + this.showToast('That endpoint no longer exists', 'error'); + return; + } + const models = host.models || []; + if (models.length === 0) { + this.showToast('No models discovered for this endpoint yet', 'warning'); + return; + } + // Exactly one model: nothing to choose, so asking would just be an extra + // click for the same answer every time. Two or more: always ask, even + // with a defaultModelId set — the point of asking is letting THIS launch + // differ from the default, not just confirming it. + if (models.length === 1) { + return this.runCustomModelEntry(mode, endpointId, models[0]); + } + this._openCustomModelPickModal(mode, host); + }, + + /** Renders the "which model" picker for a (harness, endpoint) pair with more than one discovered model. */ + _openCustomModelPickModal(mode, host) { + const modal = document.getElementById('customModelPickModal'); + const list = document.getElementById('customModelPickList'); + if (!modal || !list) return; + this._pendingCustomModelPick = { mode, endpointId: host.id }; + const cliLabel = (window.__codemanCustomModelClis || []).find((c) => c.id === mode)?.label || mode; + // A static title (translatable by i18n.js's exact-string walker) plus a + // dynamic hint carrying the specifics — same split webviewModalTitle uses, + // since the walker cannot i18n a string a variable is already spliced into. + document.getElementById('customModelPickTitle').textContent = 'Choose a model'; + document.getElementById('customModelPickHint').textContent = + `${cliLabel} → ${host.label} — ${(host.models || []).length} models discovered.`; + list.innerHTML = (host.models || []) + .map((m) => { + const isDefault = m === host.defaultModelId; + const arg = escapeHtml(JSON.stringify(m)); + return ` + `; + }) + .join(''); + modal.classList.add('active'); + }, + + closeCustomModelPickModal() { + document.getElementById('customModelPickModal')?.classList.remove('active'); + this._pendingCustomModelPick = null; + }, + + /** + * In-app replacement for a native `confirm()` popup, used specifically for the + * llama-swap "this will unload it for session X" warning (both launch paths below) — + * a browser-chrome dialog there looked out of place next to the rest of the app's own + * modals. Resolves true/false the same way `confirm()` would; `_resolveModelSwapConfirm` + * (the modal's own Cancel/Switch-anyway buttons, and its backdrop click) is what settles + * the returned promise. + */ + _confirmModelSwap(message) { + const modal = document.getElementById('customModelSwapConfirmModal'); + const messageEl = document.getElementById('customModelSwapConfirmMessage'); + if (messageEl) messageEl.textContent = message; + modal?.classList.add('active'); + return new Promise((resolve) => { + this._resolveModelSwapConfirmPromise = resolve; + }); + }, + + /** Called by the modal's Cancel/Switch-anyway buttons and its backdrop click. */ + _resolveModelSwapConfirm(proceed) { + document.getElementById('customModelSwapConfirmModal')?.classList.remove('active'); + const resolve = this._resolveModelSwapConfirmPromise; + this._resolveModelSwapConfirmPromise = null; + resolve?.(proceed); + }, + + /** + * In-app warning shown when the apply route reports `requiresContextWarning`: this + * model's real discovered context is smaller than the CLI's own fixed system-prompt/ + * tool-schema overhead, which guarantees the very first message fails outright — no + * `CLAUDE_CODE_MAX_CONTEXT_TOKENS` value fixes that, since there is no conversation + * history yet for compaction to trim. Same promise-based pattern as + * `_confirmModelSwap`; `_resolveContextWarningConfirm` settles it. + */ + _confirmContextWarning(modelId, contextLength, minSafeContextTokens) { + const modal = document.getElementById('customModelContextWarningModal'); + const messageEl = document.getElementById('customModelContextWarningMessage'); + if (messageEl) { + const known = typeof contextLength === 'number'; + messageEl.textContent = + `${modelId} is configured with ` + + (known ? `only ${contextLength.toLocaleString()} tokens of` : 'an unknown (too small)') + + ` context, but this CLI needs roughly ${minSafeContextTokens.toLocaleString()}+ tokens just for its own ` + + `system prompt and tools — before any conversation history. Its very first message will fail outright, ` + + `no matter what context size Codeman tells it to expect.\n\n` + + `To fix this, reconfigure llama-swap to give this model (or a smaller one) an explicit larger context ` + + `instead of relying on auto-fit (--fit-ctx), which optimizes for the biggest MODEL that fits, not the ` + + `biggest CONTEXT — e.g. add "-c 65536" (or as large a --ctx-size as your hardware holds) to its llama-swap ` + + `config entry. A smaller model at a much larger explicit context often fits in the same VRAM a bigger ` + + `model's auto-fit context gets shrunk to make room for.`; + } + modal?.classList.add('active'); + return new Promise((resolve) => { + this._resolveContextWarningConfirmPromise = resolve; + }); + }, + + /** Called by the modal's Cancel/Launch-anyway buttons and its backdrop click. */ + _resolveContextWarningConfirm(proceed) { + document.getElementById('customModelContextWarningModal')?.classList.remove('active'); + const resolve = this._resolveContextWarningConfirmPromise; + this._resolveContextWarningConfirmPromise = null; + resolve?.(proceed); + }, + + /** A model row in the picker modal was clicked: close it and launch with that choice. */ + chooseCustomModelAndRun(modelId) { + const pending = this._pendingCustomModelPick; + this.closeCustomModelPickModal(); + if (!pending) return; // modal reopened/closed from elsewhere between render and click + void this.runCustomModelEntry(pending.mode, pending.endpointId, modelId); + }, + + /** + * Runs a session on `mode` and immediately applies `endpointId`/`modelId` to it + * via POST /api/sessions/:id/custom-model (see session-routes.ts) — the same + * restart-in-place apply path the (not-yet-built) endpoint-management surface + * would use for an already-running session. A custom-model run is a one-off + * "try this endpoint" action, not a sticky mode. + * + * Routes through run() itself, via a temporary `_runMode` swap, rather than a + * parallel dispatch table: that is what gives this the same in-flight lock + * every other Run click gets (CLAUDE.md, Run launch synchronization — the lock + * exists so a double click cannot create duplicate sessions with the same + * `w-` name, and it guards the OTHER direction too: without it, the + * main Run button could start a second concurrent launch while this one was + * still resolving), and it means a CLI whose customModelInjection recipe + * lands later needs no update here, only in run()'s own dispatch. The swap + * never persists — setRunMode() would sync it to the server as the user's new + * default, which a one-off endpoint run must not do — and is restored in + * `finally` even if run() throws. + */ + /** + * Dispatches to the ONE-SHOT launch path (below) for every custom-model-eligible CLI + * except claude, which still goes through the restart-after-native-boot path + * (`_runCustomModelEntryViaRestart`): claude's own `runClaude()` carries multi-tab + * launch and a docker-config-drift confirm/retry loop neither of the other seven + * functions has, and folding those into the one-shot flow is unstarted, separate work. + * The other seven (opencode/codex/gemini/pi/grok/deepseek/omp) are each a single, + * simple launch, so they get the one-shot path — the one visibly worth it, since a + * native-boot-then-restart is far more jarring on a CLI whose TUI fully reinitializes + * (Codex, confirmed live) than on claude's own `--resume`-based restart. + */ + async runCustomModelEntry(mode, endpointId, modelId) { + if (mode === 'claude') { + return this._runCustomModelEntryViaRestart(mode, endpointId, modelId); + } + return this._runCustomModelEntryOneShot(mode, endpointId, modelId); + }, + + /** + * Launches directly on the endpoint — no restart, so no visible relaunch. Stashes the + * pick on `_pendingCustomModelForLaunch` for the targeted run() function to read + * and fold into its own /api/quick-start body (see `_quickStartWithCustomModelConfirm`); + * cleared in `finally` the same way `_runMode`'s temporary swap is, even if run() throws. + */ + async _runCustomModelEntryOneShot(mode, endpointId, modelId) { + document.getElementById('runModeMenu')?.classList.remove('active'); + const previousRunMode = this._runMode; + const tabCountEl = document.getElementById('tabCount'); + const prevTabCount = tabCountEl?.value; + this._runMode = mode; + this._pendingCustomModelForLaunch = { endpointId, modelId }; + if (tabCountEl) tabCountEl.value = '1'; + try { + await this.run(); + } finally { + this._runMode = previousRunMode; + this._pendingCustomModelForLaunch = undefined; + if (tabCountEl && prevTabCount !== undefined) tabCountEl.value = prevTabCount; + } + + // run() (via _quickStartWithCustomModelConfirm) reports its own launch error or + // cancellation via toast and leaves this unset — nothing more to do here then. + const result = this._lastCustomModelLaunchResult; + this._lastCustomModelLaunchResult = undefined; + if (result?.modelSwapInProgress) { + void this._watchLlamaSwapLoading(endpointId, modelId, result.sessionId); + } + }, + + /** + * POSTs a /api/quick-start body already carrying `customModel` (see the run() + * call sites below), showing the same llama-swap "this will unload it for session X" + * warning the restart path's `_applyCustomModelToSession` shows when the route asks + * for confirmation, and retrying with `confirmed: true` on accept. Stashes the final + * response's payload on `_lastCustomModelLaunchResult` for + * `_runCustomModelEntryOneShot` to read `modelSwapInProgress` off afterward — run()'s + * eleven per-mode dispatch targets have no shared return-value contract of their own, + * so a side channel here is simpler than threading one through every one of them. + */ + async _quickStartWithCustomModelConfirm(bodyObj) { + const post = async (body) => { + const res = await fetch('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/api/quick-start', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }); + return res.json(); + }; + let data = await post(bodyObj); + if (data?.data?.requiresContextWarning) { + const { modelId, contextLength, minSafeContextTokens } = data.data; + const proceed = await this._confirmContextWarning(modelId, contextLength, minSafeContextTokens); + if (!proceed) { + this._lastCustomModelLaunchResult = undefined; + return { success: false, error: 'Launch cancelled — context window too small' }; + } + data = await post({ ...bodyObj, customModel: { ...bodyObj.customModel, confirmed: true } }); + } + if (data?.data?.requiresConfirmation) { + const { currentlyLoadedModel, affectedSessions } = data.data; + const names = affectedSessions.map((s) => s.name || s.id).join(', '); + const proceed = await this._confirmModelSwap( + `${names} ${affectedSessions.length === 1 ? 'is' : 'are'} currently using ` + + `${currentlyLoadedModel} on this endpoint. Switching will unload it for ` + + `${affectedSessions.length === 1 ? 'that session' : 'those sessions'} too. Continue?` + ); + if (!proceed) { + this._lastCustomModelLaunchResult = undefined; + return { success: false, error: 'Model switch cancelled' }; + } + data = await post({ ...bodyObj, customModel: { ...bodyObj.customModel, confirmed: true } }); + } + this._lastCustomModelLaunchResult = data?.success !== false ? data?.data : undefined; + return data; + }, + + /** The restart-after-native-boot path — see `runCustomModelEntry`'s own comment for + * which CLIs still use this one. */ + async _runCustomModelEntryViaRestart(mode, endpointId, modelId) { + document.getElementById('runModeMenu')?.classList.remove('active'); + + const previousRunMode = this._runMode; + const before = this.activeSessionId; + const tabCountEl = document.getElementById('tabCount'); + const prevTabCount = tabCountEl?.value; + this._runMode = mode; + if (tabCountEl) tabCountEl.value = '1'; + try { + await this.run(); + } finally { + this._runMode = previousRunMode; + if (tabCountEl && prevTabCount !== undefined) tabCountEl.value = prevTabCount; + } + + // run() reports its own errors via toast. Every run*() function handles its + // own failure internally and returns normally rather than throwing or + // leaving activeSessionId null, so a declined/failed launch (missing CLI, a + // caught exception, isBusy on the session the launch would have targeted) + // falls through to here with the PREVIOUSLY active session still active. + // Requiring the id to have actually changed — not just to be non-null — is + // what stops that case from silently re-pointing and restarting whatever + // session the user was already looking at. + const sessionId = this.activeSessionId; + if (!sessionId || sessionId === before) return; + + // Claude just launched on the NATIVE backend and is about to be restarted onto + // the endpoint — without something saying so, that native boot (which can talk + // to Opus for a moment) reads as "the endpoint didn't apply" rather than "the + // switch hasn't happened yet". Prominent and screen-centred (not a corner toast) + // since this can sit on screen for a while; sticky until the apply below settles + // one way or the other, or hands off to _watchLlamaSwapLoading's own banner. + const switchingToast = this._showCenterStatus(`Claude started — switching to ${endpointId}…`); + + // A freshly launched CLI reports its OWN startup as 'busy' (spinner, the + // workspace-trust check, whatever else it does before its first prompt) — + // measured landing well before this line reliably reaches it — and the + // apply route's isBusy() guard correctly refuses to restart a session + // mid-turn, "mid-turn" included, which this fresh boot looks exactly + // like from the outside. Give it a bounded chance to settle first rather + // than raising a false "Session is busy" on every single launch. Per the + // wait contract a timeout here is a normal 200, never an error — a + // session still busy after 20s just reaches the apply call below and + // gets the route's own honest, now-visible SESSION_BUSY error instead of + // this guessing about it. + await this._apiJson(`/api/sessions/${sessionId}/wait?until=idle&timeout=20000`); + + // _apiJson() (used everywhere else in this file) unwraps a success body to + // its `data`, but on failure it swallows the response entirely and returns + // null — exactly the `error` text a caller needs to tell "the endpoint is + // unreachable" apart from "the CLI can't be redirected", "not one of the + // discovered models", or "this is a Docker/remote session". Go through the + // raw response here instead so a failure is diagnosable, not just present. + let { ok, data, res } = await this._applyCustomModelToSession(sessionId, endpointId, modelId); + + // A success body comes back as {success:true, data:{...}} (server.ts's preSerialization + // envelope), but a route-level error is {success:false, error, errorCode} with no nested + // data — createErrorResponse() never wraps one. `payload` below is only ever meaningful + // once `data.success !== false`. + let payload = data?.success !== false ? data?.data : undefined; + + // This CLI's own fixed overhead (system prompt + tool schemas) may exceed the + // model's real discovered context outright — no context-length declaration can + // fix that, since compaction only trims conversation history and there is none + // on message 1. Warn and let the user decide whether to launch anyway, same + // confirmed:true re-send pattern as the swap check below. + if (ok && payload?.requiresContextWarning) { + const proceed = await this._confirmContextWarning( + payload.modelId, + payload.contextLength, + payload.minSafeContextTokens + ); + if (!proceed) { + switchingToast?.dismiss(); + this.showToast('Kept the native backend — context window too small', 'info'); + return; + } + ({ ok, data, res } = await this._applyCustomModelToSession(sessionId, endpointId, modelId, true)); + payload = data?.success !== false ? data?.data : undefined; + } + + // llama-swap runs one model at a time: switching would unload it out from under + // another session actively using it. The route only asks when that's actually true + // (never just because a swap is needed at all) — confirming re-sends the exact same + // call with `confirmed: true` so the route skips the check the second time. + if (ok && payload?.requiresConfirmation) { + const names = payload.affectedSessions.map((s) => s.name || s.id).join(', '); + const proceed = await this._confirmModelSwap( + `${names} ${payload.affectedSessions.length === 1 ? 'is' : 'are'} currently using ` + + `${payload.currentlyLoadedModel} on this endpoint. Switching to ${modelId} will unload it ` + + `for ${payload.affectedSessions.length === 1 ? 'that session' : 'those sessions'} too. Continue?` + ); + if (!proceed) { + switchingToast?.dismiss(); + this.showToast('Kept the native backend — model switch cancelled', 'info'); + return; + } + ({ ok, data, res } = await this._applyCustomModelToSession(sessionId, endpointId, modelId, true)); + payload = data?.success !== false ? data?.data : undefined; + } + + if (!ok || !data || data.success === false) { + switchingToast?.dismiss(); + const detail = data?.error ? `: ${data.error}` : res ? ` (HTTP ${res.status})` : ' (request failed)'; + this.showToast(`Session started on the native backend — could not apply the custom endpoint${detail}`, 'error'); + return; + } + + // The apply above already succeeded — the session IS pointed at the endpoint — but + // llama-swap itself may still be unloading the old model and loading this one, which + // can take well over a minute. Without this, a prompt sent during that window either + // hangs silently or (the bug this whole feature exists to fix) gets answered by + // whatever was loaded a moment ago, reading as "it's still using the wrong model." + // Hand off to its own sticky toast rather than stacking a second one on top. + if (payload?.modelSwapInProgress) { + switchingToast?.dismiss(); + void this._watchLlamaSwapLoading(endpointId, modelId, sessionId); + return; + } + + switchingToast?.setMessage(`Pointed at ${endpointId} — restarting the session...`); + setTimeout(() => switchingToast?.dismiss(), 3000); + }, + + /** POST /api/sessions/:id/custom-model, returning {ok, data, res} rather than throwing — + * see runCustomModelEntry's own comment for why this goes through `_api()` (raw fetch) + * rather than `_apiJson()`: a failure's `error` detail must survive to the caller. */ + async _applyCustomModelToSession(sessionId, endpointId, modelId, confirmed) { + const res = await this._api(`/api/sessions/${sessionId}/custom-model`, { + method: 'POST', + body: confirmed ? { endpointId, modelId, confirmed } : { endpointId, modelId }, + }); + const data = res ? await res.json().catch(() => null) : null; + return { ok: !!res, data, res }; + }, + + /** + * Best-effort: looks up `modelId`'s discovered file size (GB) off the endpoint's own + * saved host record (`CustomModelHost.modelSizesGB`, populated during discovery by + * parsing llama-swap's own `description` field for an auto-discovered model). Returns + * `undefined` for a hand-configured profile with no parseable size, an unreachable + * server, or any other failure — never a guess. + */ + async _lookupModelSizeGB(endpointId, modelId) { + const hosts = await this._apiJson('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/api/model-endpoints').catch(() => null); + if (!Array.isArray(hosts)) return undefined; + const host = hosts.find((h) => h.id === endpointId); + const size = host?.modelSizesGB?.[modelId]; + return typeof size === 'number' && Number.isFinite(size) && size > 0 ? size : undefined; + }, + + /** + * Strips llama.cpp's own bootlog prefix (` `, e.g. + * `0.31.428.568 I srv llama_server: model loaded`) for display, leaving just + * `llama_server: model loaded` — the raw line from the server is kept as-is + * (`GET .../running-status`'s `logLine` field), this trims it only for the loading + * banner's second line. Defensive: a line that doesn't match this shape (a different + * llama.cpp build, or llama-swap's own format changing) is shown verbatim rather than + * mangled or dropped. + */ + _formatLlamaLogLine(line) { + return typeof line === 'string' ? line.replace(/^[\d.]+\s+[IWE]\s+\S+\s+/, '') : line; + }, + + /** + * Polls llama-swap's own `/running` (via the read-only running-status route) until + * `modelId` reports `state: 'ready'`, showing a sticky banner the whole time so a slow + * unload/reload (measured well over a minute for a large model) reads as "loading, + * still working on it", never as silence or a wrong answer from whatever was loaded + * before. Checks immediately (a fast load, or a re-apply onto an already-ready model, + * shouldn't wait a full interval to say so), then every `pollIntervalMs`. + * + * Deliberately UNBOUNDED — no estimate, no countdown, no automatic give-up. An earlier + * version scaled a timeout off the model's discovered file size and auto-closed the + * session when it elapsed, but a real load's actual duration depends on hardware this + * feature has no way to know (VRAM, storage speed, what else is contending for the + * GPU), so any fixed number was a guess dressed up as a fact — the banner now says so + * outright instead of pretending to a precision it doesn't have, and a Cancel button on + * the banner itself (`_showCenterStatus`'s `onCancel`) is how the user ends it if it's + * taking too long, closing `sessionId` the same way the old timeout used to. + * + * `_watchLlamaSwapGeneration` guards against two overlapping calls (a second launch + * started before the first one's loop finished) clobbering each other's banner: + * `_showCenterStatus` reuses one shared DOM node, so an older loop's `dismiss()`/message + * update firing after a newer one has already taken over the banner would otherwise hide + * or overwrite the WRONG one, or close the WRONG session. Each call claims the counter + * as its own "generation" and checks it still owns it before touching either. + * + * `pollIntervalMs` exists to let a test drive this in milliseconds instead of seconds — + * real callers never pass it. + */ + async _watchLlamaSwapLoading(endpointId, modelId, sessionId, pollIntervalMs = 1000) { + const generation = (this._watchLlamaSwapGeneration = (this._watchLlamaSwapGeneration || 0) + 1); + const isCurrent = () => this._watchLlamaSwapGeneration === generation; + const sizeGB = await this._lookupModelSizeGB(endpointId, modelId); + if (!isCurrent()) return; // a newer launch already took over before the lookup even finished + const sizeSuffix = sizeGB ? ` (${sizeGB.toFixed(1)} GB)` : ''; + const baseMessage = + `Loading ${modelId}${sizeSuffix} on ${endpointId} — this can take a while depending on ` + + `your hardware and the model size.`; + // Second line, when llama-swap's own event feed actually gives us one: the real + // backend llama-server process's own latest log line (load_model:/llama_server: ..., + // see getLatestLlamaSwapLogLine) — a bare "please wait" says nothing is broken, this + // says what's actually happening. Absent on the very first render (no poll has + // landed yet) and whenever the endpoint doesn't expose it at all — never fabricated, + // and never cleared back to blank once seen (stays on the last real thing llama.cpp + // said if a later poll comes back with nothing new). + const buildMessage = (logLine) => { + const line = this._formatLlamaLogLine(logLine); + return baseMessage + (line ? `\nllama.cpp: ${line}` : ''); + }; + let cancelled = false; + // Prominent and screen-centred, not a corner toast — a real llama-swap model load can + // sit on screen for well over a minute, easy to mistake for nothing happening there. + const toast = this._showCenterStatus(buildMessage(), { + onCancel: () => { + cancelled = true; + }, + }); + while (!cancelled) { + const status = await this._apiJson(`/api/model-endpoints/${encodeURIComponent(endpointId)}/running-status`); + if (!isCurrent()) return; // a newer launch took over the banner — this loop is done + if (cancelled) break; + if (!status) { + // transient failure — keep waiting rather than giving up early + } else if (!status.isLlamaSwap) { + // Endpoint changed under us, or wasn't llama-swap after all — nothing more to + // watch for, and not a failure worth a toast of its own. + toast?.dismiss(); + return; + } else if (status.running.some((r) => r.model === modelId && r.state === 'ready')) { + toast?.dismiss(); + this.showToast(`${modelId} is ready`, 'success', { duration: 2500 }); + return; + } + if (!isCurrent() || cancelled) break; + toast?.setMessage(buildMessage(status?.logLine)); + await new Promise((resolve) => setTimeout(resolve, pollIntervalMs)); + } + if (!isCurrent()) return; + // Cancelled by the user, not a timeout — an ordinary info toast, not a scary error + // banner, since this was deliberate rather than something going wrong. + toast?.dismiss(); + this.showToast( + `Cancelled loading ${modelId} on ${endpointId}` + (sessionId ? ' — the session has been closed.' : '.'), + 'info' + ); + if (sessionId) { + try { + await this.closeSession(sessionId); + } catch { + // closeSession already reports its own failure via toast — nothing more to do here + } + } + }, + /** * Start the DeepSeek Harness browser UI and open it as a Codeman web tab. * @@ -1285,20 +1865,16 @@ Object.assign(CodemanApp.prototype, { // Quick-start with opencode mode (auto-allow tools by default). // No `effort` field — it's Claude-specific (OpenCode has no /effort). const envOverrides = this.buildEnvOverrides(this.getCaseSettings(caseName), this.loadAppSettingsFromStorage()); - const res = await fetch('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/api/quick-start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - caseName, - mode: 'opencode', - sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, - ...(isRemote ? {} : { - openCodeConfig: { autoAllowTools: true }, - ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), - }), - }) + const data = await this._quickStartWithCustomModelConfirm({ + caseName, + mode: 'opencode', + sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, + ...(isRemote ? {} : { + openCodeConfig: { autoAllowTools: true }, + ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), + ...(this._pendingCustomModelForLaunch ? { customModel: this._pendingCustomModelForLaunch } : {}), + }), }); - const data = await res.json(); if (!data.success) throw new Error(data.error || 'Failed to start OpenCode'); await this._ensureCreatedSessionVisible(data.data.sessionId, data.data.session); @@ -1339,24 +1915,20 @@ Object.assign(CodemanApp.prototype, { const globalSettings = this.loadAppSettingsFromStorage(); const envOverrides = this.buildEnvOverrides(this.getCaseSettings(caseName), globalSettings); - const res = await fetch('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/api/quick-start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - caseName, - mode: 'codex', - sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, - ...(isRemote ? {} : { - codexConfig: { - dangerouslyBypassApprovals: globalSettings.codexDangerouslyBypassApprovals ?? false, - animations: globalSettings.codexAnimationsEnabled ?? false, - renderMode: 'hybrid', - }, - ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), - }), - }) + const data = await this._quickStartWithCustomModelConfirm({ + caseName, + mode: 'codex', + sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, + ...(isRemote ? {} : { + codexConfig: { + dangerouslyBypassApprovals: globalSettings.codexDangerouslyBypassApprovals ?? false, + animations: globalSettings.codexAnimationsEnabled ?? false, + renderMode: 'hybrid', + }, + ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), + ...(this._pendingCustomModelForLaunch ? { customModel: this._pendingCustomModelForLaunch } : {}), + }), }); - const data = await res.json(); if (!data.success) throw new Error(data.error || 'Failed to start Codex'); await this._ensureCreatedSessionVisible(data.data.sessionId, data.data.session); @@ -1396,20 +1968,16 @@ Object.assign(CodemanApp.prototype, { } const envOverrides = this.buildEnvOverrides(this.getCaseSettings(caseName), this.loadAppSettingsFromStorage()); - const res = await fetch('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/api/quick-start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - caseName, - mode: 'gemini', - sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, - ...(isRemote ? {} : { - geminiConfig: { approvalMode: 'yolo' }, - ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), - }), - }) + const data = await this._quickStartWithCustomModelConfirm({ + caseName, + mode: 'gemini', + sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, + ...(isRemote ? {} : { + geminiConfig: { approvalMode: 'yolo' }, + ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), + ...(this._pendingCustomModelForLaunch ? { customModel: this._pendingCustomModelForLaunch } : {}), + }), }); - const data = await res.json(); if (!data.success) throw new Error(data.error || 'Failed to start Gemini'); await this._ensureCreatedSessionVisible(data.data.sessionId, data.data.session); @@ -1507,17 +2075,13 @@ Object.assign(CodemanApp.prototype, { } const envOverrides = this.buildEnvOverrides(this.getCaseSettings(caseName), this.loadAppSettingsFromStorage()); - const res = await fetch('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/api/quick-start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - caseName, - mode: 'pi', - sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, - ...(isRemote || Object.keys(envOverrides).length === 0 ? {} : { envOverrides }), - }) + const data = await this._quickStartWithCustomModelConfirm({ + caseName, + mode: 'pi', + sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, + ...(isRemote || Object.keys(envOverrides).length === 0 ? {} : { envOverrides }), + ...(!isRemote && this._pendingCustomModelForLaunch ? { customModel: this._pendingCustomModelForLaunch } : {}), }); - const data = await res.json(); if (!data.success) throw new Error(data.error || 'Failed to start Pi'); await this._ensureCreatedSessionVisible(data.data.sessionId, data.data.session); @@ -1555,19 +2119,15 @@ Object.assign(CodemanApp.prototype, { } const envOverrides = this.buildEnvOverrides(this.getCaseSettings(caseName), this.loadAppSettingsFromStorage()); - const res = await fetch('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/api/quick-start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - caseName, - mode: 'omp', - sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, - ...(isRemote ? {} : { - ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), - }), - }) + const data = await this._quickStartWithCustomModelConfirm({ + caseName, + mode: 'omp', + sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, + ...(isRemote ? {} : { + ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), + ...(this._pendingCustomModelForLaunch ? { customModel: this._pendingCustomModelForLaunch } : {}), + }), }); - const data = await res.json(); if (!data.success) throw new Error(data.error || 'Failed to start OMP'); await this._ensureCreatedSessionVisible(data.data.sessionId, data.data.session); @@ -1614,20 +2174,16 @@ Object.assign(CodemanApp.prototype, { } const envOverrides = this.buildEnvOverrides(this.getCaseSettings(caseName), this.loadAppSettingsFromStorage()); - const res = await fetch('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/api/quick-start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - caseName, - mode: 'grok', - sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, - ...(isRemote ? {} : { - grokConfig: { alwaysApprove: true }, - ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), - }), - }) + const data = await this._quickStartWithCustomModelConfirm({ + caseName, + mode: 'grok', + sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, + ...(isRemote ? {} : { + grokConfig: { alwaysApprove: true }, + ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), + ...(this._pendingCustomModelForLaunch ? { customModel: this._pendingCustomModelForLaunch } : {}), + }), }); - const data = await res.json(); if (!data.success) throw new Error(data.error || 'Failed to start Grok'); await this._ensureCreatedSessionVisible(data.data.sessionId, data.data.session); @@ -1692,20 +2248,16 @@ Object.assign(CodemanApp.prototype, { } const envOverrides = this.buildEnvOverrides(this.getCaseSettings(caseName), this.loadAppSettingsFromStorage()); - const res = await fetch('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/api/quick-start', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ - caseName, - mode: 'deepseek', - sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, - ...(isRemote ? {} : { - deepSeekConfig: { permissionMode: 'danger-full-access' }, - ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), - }), - }) + const data = await this._quickStartWithCustomModelConfirm({ + caseName, + mode: 'deepseek', + sessionName: `w${this._nextCaseSessionStartNumber(caseName)}-${caseName}`, + ...(isRemote ? {} : { + deepSeekConfig: { permissionMode: 'danger-full-access' }, + ...(Object.keys(envOverrides).length > 0 ? { envOverrides } : {}), + ...(this._pendingCustomModelForLaunch ? { customModel: this._pendingCustomModelForLaunch } : {}), + }), }); - const data = await res.json(); if (!data.success) throw new Error(data.error || 'Failed to start DeepSeek'); await this._ensureCreatedSessionVisible(data.data.sessionId, data.data.session); diff --git a/src/web/public/settings-ui.js b/src/web/public/settings-ui.js index 79bee3c07..1ade54a40 100644 --- a/src/web/public/settings-ui.js +++ b/src/web/public/settings-ui.js @@ -395,6 +395,13 @@ Object.assign(CodemanApp.prototype, { document.getElementById('appSettingsShowUltracodeAgents').checked = settings.showUltracodeAgents ?? defaults.showUltracodeAgents ?? false; // Approvals Inbox: synced, default OFF (opt-in; only an explicit true enables). document.getElementById('appSettingsApprovalsInbox').checked = settings.approvalsInboxEnabled === true; + // Custom Model Endpoint Profiles: synced, default OFF. The toggle governs both + // the Run-menu picker's generated entries and this settings panel's visibility; + // the endpoint list itself is server state, loaded on demand below. + document.getElementById('appSettingsCustomModelEndpoints').checked = settings.customModelEndpointsEnabled === true; + // Assigning .checked above does not fire onchange, so the body's visibility + // (and its lazy load) needs an explicit sync on every open, not just a save. + this.applyCustomModelEndpointsVisibility(); // Read My Mind: synced, default OFF (opt-in; capture + prediction cost real tokens). document.getElementById('appSettingsReadMyMind').checked = settings.readMyMindEnabled === true; document.getElementById('appSettingsUltracodeFloatingWindows').checked = @@ -509,6 +516,9 @@ Object.assign(CodemanApp.prototype, { document.getElementById('appSettingsNiceValue').value = niceSettings.niceValue ?? 10; // Model configuration (loaded from server) this.loadModelConfigForSettings(); + // Custom Model Endpoint Profiles' own load is gated on the toggle above (see + // applyCustomModelEndpointsVisibility) — unlike model config, this GET is + // pointless work with the feature off, so it is not fired unconditionally. // Notification settings const notifPrefs = this.notificationManager?.preferences || {}; document.getElementById('appSettingsNotifEnabled').checked = notifPrefs.enabled ?? true; @@ -2106,6 +2116,7 @@ Object.assign(CodemanApp.prototype, { showSubagents: document.getElementById('appSettingsShowSubagents').checked, showUltracodeAgents: document.getElementById('appSettingsShowUltracodeAgents').checked, approvalsInboxEnabled: document.getElementById('appSettingsApprovalsInbox').checked, + customModelEndpointsEnabled: document.getElementById('appSettingsCustomModelEndpoints').checked, readMyMindEnabled: document.getElementById('appSettingsReadMyMind').checked, ultracodeFloatingWindows: document.getElementById('appSettingsUltracodeFloatingWindows').checked, showMultiMonitorButton: document.getElementById('appSettingsShowMultiMonitorButton').checked, @@ -2487,6 +2498,209 @@ Object.assign(CodemanApp.prototype, { } }, + // ═══════════════════════════════════════════════════════════════ + // Custom Model Endpoint Profiles (docs/custom-model-endpoints-plan.md) + // + // CRUD against /api/model-endpoints, rendered into the Models settings section. + // Deliberately its own load/save pair rather than folded into openAppSettings/ + // saveAppSettings: these are server-side infra records (like remote/docker + // hosts), not a settings-payload field, so the app-settings-structure guard's + // by-id contract does not apply to them — only the `customModelEndpointsEnabled` + // toggle itself goes through that path. + // ═══════════════════════════════════════════════════════════════ + + /** + * Toggles the endpoint-management body's visibility to match the setting and, + * turning it on, lazily loads the endpoint list. Assigning `.checked` (as the + * settings load path does) fires no `change` event, so this must be called + * explicitly on open as well as wired to the checkbox's own onchange — a + * gate that only worked one of those two ways would show a stale "off" + * body right after opening, or a stale "on" one right after saving it off. + * With the feature off the body is a list of controls that do nothing, so it + * is hidden entirely rather than shown disabled. + */ + applyCustomModelEndpointsVisibility() { + const enabled = document.getElementById('appSettingsCustomModelEndpoints').checked; + const body = document.getElementById('customModelEndpointsBody'); + if (body) body.style.display = enabled ? '' : 'none'; + if (enabled) this.loadCustomModelEndpointsForSettings(); + else this.closeCustomModelHostEditor(); + this._applyCustomModelAdminGate(); + }, + + /** + * Endpoint writes are admin-only in multi-user mode (custom-model-routes.ts), + * and GET already answers a non-admin with an empty list, which hides every + * per-row Edit/Discover/Delete button on its own. The "+ Add endpoint" button + * has no row to hide behind, so it needs its own gate — otherwise a non-admin + * can open the form, fill it in, and get a 403 toast on Save. Wired to the + * `codeman:me` event (admin-ui.js) as well as called from + * applyCustomModelEndpointsVisibility(), because `window.__codemanUser`'s + * real role can resolve AFTER settings have already been opened once. + */ + _applyCustomModelAdminGate() { + const addBtn = document.getElementById('customModelHostAddBtn'); + if (!addBtn) return; + const me = window.__codemanUser || {}; + const blocked = me.multiUser && me.role !== 'admin'; + addBtn.style.display = blocked ? 'none' : ''; + }, + + async loadCustomModelEndpointsForSettings() { + // GET /api/model-endpoints wraps its body in the { success, data } envelope + // like every other /api route (server.ts's preSerialization hook applies to + // arrays too) — _apiJson() unwraps it. A raw fetch().json() here would + // silently see the envelope object instead of the array and this panel + // would read as "No endpoints yet" forever, even with endpoints saved. + const hosts = await this._apiJson('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/api/model-endpoints'); + this._customModelHosts = Array.isArray(hosts) ? hosts : []; + this.renderCustomModelHostsList(); + }, + + renderCustomModelHostsList() { + const list = document.getElementById('customModelHostsList'); + if (!list) return; + const hosts = this._customModelHosts || []; + if (hosts.length === 0) { + list.innerHTML = '

No endpoints yet. Add one below to point a harness at a local or cloud OpenAI-compatible server.

'; + return; + } + list.innerHTML = hosts + .map((h) => { + const modelCount = (h.models || []).length; + const modelSummary = modelCount === 0 + ? 'No models discovered yet' + : `${modelCount} model${modelCount === 1 ? '' : 's'}${h.defaultModelId ? ` · default: ${escapeHtml(h.defaultModelId)}` : ' · no default set'}`; + // escapeHtml(JSON.stringify(h.id)) — not JSON.stringify(h.id) alone — + // because JSON.stringify's own double quotes would otherwise terminate + // this double-quoted attribute at the first one, and everything after + // parses as raw tag content rather than the rest of the quoted string. + // Same idiom as deleteCase's onclick in session-ui.js. h.id is + // regex-constrained server-side (safe either way) but the pattern must + // match everywhere it is used, including where the argument is not. + const idArg = escapeHtml(JSON.stringify(h.id)); + return ` +
+
+ ${escapeHtml(h.label)} + ${escapeHtml(h.baseUrl)} — ${modelSummary} +
+
+ + + +
+
`; + }) + .join(''); + }, + + /** Opens the inline add/edit form. Pass no id to add a new endpoint. */ + openCustomModelHostEditor(hostId) { + const host = hostId ? (this._customModelHosts || []).find((h) => h.id === hostId) : null; + this._editingCustomModelHostId = host ? host.id : null; + document.getElementById('customModelHostEditorTitle').textContent = host ? `Edit ${host.label}` : 'Add endpoint'; + document.getElementById('customModelHostId').value = host?.id || ''; + document.getElementById('customModelHostId').disabled = !!host; // id is immutable once created + document.getElementById('customModelHostLabel').value = host?.label || ''; + document.getElementById('customModelHostBaseUrl').value = host?.baseUrl || ''; + document.getElementById('customModelHostApiKey').value = ''; // the server never returns the real value (apiKeySet is a bool) + document.getElementById('customModelHostApiKey').placeholder = host?.apiKeySet ? '•••••••• (unchanged if left blank)' : ''; + document.getElementById('customModelHostAuthStyle').value = host?.authStyle || 'bearer'; + this._populateCustomModelDefaultSelect(host); + document.getElementById('customModelHostEditor').style.display = ''; + }, + + closeCustomModelHostEditor() { + document.getElementById('customModelHostEditor').style.display = 'none'; + this._editingCustomModelHostId = null; + }, + + _populateCustomModelDefaultSelect(host) { + const select = document.getElementById('customModelHostDefaultModel'); + const models = host?.models || []; + select.innerHTML = + '' + + models.map((m) => ``).join(''); + select.value = host?.defaultModelId || ''; + select.disabled = models.length === 0; + }, + + async saveCustomModelHostFromEditor() { + const id = document.getElementById('customModelHostId').value.trim(); + const label = document.getElementById('customModelHostLabel').value.trim(); + const baseUrl = document.getElementById('customModelHostBaseUrl').value.trim(); + const apiKeyInput = document.getElementById('customModelHostApiKey').value; + const authStyle = document.getElementById('customModelHostAuthStyle').value; + const defaultModelId = document.getElementById('customModelHostDefaultModel').value || undefined; + if (!id || !label || !baseUrl) { + this.showToast('Id, label and base URL are all required', 'warning'); + return; + } + const editing = this._editingCustomModelHostId; + // PUT (server-side) treats an absent apiKey as "keep the stored one" — the + // browser never holds the real value to resend deliberately unchanged (see + // openCustomModelHostEditor and custom-model-routes.ts's applyStoredApiKey), + // so a blank field here means omitting the key entirely, not resending + // something we do not have. models/lastDiscoveredAt DO still need + // re-sending: PUT replaces the whole record, and this cached copy still + // carries both (only apiKey is redacted from what GET hands back). + const existing = editing ? (this._customModelHosts || []).find((h) => h.id === editing) : null; + const body = { + id, + label, + baseUrl, + authStyle, + defaultModelId, + apiKey: apiKeyInput || undefined, + models: existing?.models, + lastDiscoveredAt: existing?.lastDiscoveredAt, + }; + try { + const res = await fetch(editing ? `/api/model-endpoints/${encodeURIComponent(editing)}` : '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/api/model-endpoints', { + method: editing ? 'PUT' : 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }); + const data = await res.json(); + if (!data.success) { + this.showToast(data.error || 'Failed to save endpoint', 'error'); + return; + } + this.showToast(editing ? 'Endpoint updated' : 'Endpoint added', 'success'); + this.closeCustomModelHostEditor(); + await this.loadCustomModelEndpointsForSettings(); + } catch (err) { + this.showToast(`Failed to save endpoint: ${err.message}`, 'error'); + } + }, + + async discoverCustomModelHostModels(hostId) { + this.showToast('Discovering models…', 'info'); + try { + const res = await fetch(`/api/model-endpoints/${encodeURIComponent(hostId)}/discover-models`, { method: 'POST' }); + const data = await res.json(); + if (!data.success) { + this.showToast(data.error || 'Discovery failed', 'error'); + return; + } + this.showToast(`Found ${data.data.models.length} model${data.data.models.length === 1 ? '' : 's'}`, 'success'); + await this.loadCustomModelEndpointsForSettings(); + } catch (err) { + this.showToast(`Discovery failed: ${err.message}`, 'error'); + } + }, + + async deleteCustomModelHost(hostId) { + const host = (this._customModelHosts || []).find((h) => h.id === hostId); + if (!confirm(`Delete endpoint "${host?.label || hostId}"? Any session currently pointed at it keeps running until cleared.`)) return; + try { + await fetch(`/api/model-endpoints/${encodeURIComponent(hostId)}`, { method: 'DELETE' }); + await this.loadCustomModelEndpointsForSettings(); + } catch (err) { + this.showToast(`Failed to delete endpoint: ${err.message}`, 'error'); + } + }, // ═══════════════════════════════════════════════════════════════ // Visibility Settings & Device-Specific Defaults @@ -3543,3 +3757,15 @@ Object.assign(CodemanApp.prototype, { this.subagentPanelVisible = false; }, }); + +// window.__codemanUser's real role can resolve after settings have already been +// opened once (admin-ui.js fetches /api/me asynchronously and dispatches this on +// arrival), so the Custom Model Endpoints admin gate needs to be re-applied when +// it does, not just when the modal opens. Optional chaining on addEventListener +// itself: several frontend tests (run-mode-ui.test.ts) load this file into a vm +// context with a minimal fake `document` that has no event-target methods at +// all, and a module-level statement that throws there fails the whole file's +// evaluation, not just this feature. +document.addEventListener?.('codeman:me', () => { + window.app?._applyCustomModelAdminGate?.(); +}); diff --git a/src/web/public/styles.css b/src/web/public/styles.css index 398eb6ce5..4e2f7f042 100644 --- a/src/web/public/styles.css +++ b/src/web/public/styles.css @@ -6803,6 +6803,67 @@ body.touch-device .terminal-container .xterm .xterm-helper-textarea { min-height: 0; } +/* Custom Model Endpoint Profiles' "which model" picker: same bounded-height + + scrollable-body shape as .modal-lg above, scoped by id rather than added to + .modal-sm itself (three other modals share that class for short, fixed + content and do not need a height cap). Without this the modal had no + max-height at all, so an endpoint with many discovered models grew the + dialog past the viewport with nothing to scroll — "the whole page" and + "the list is truncated" turned out to be one and the same bug. `min(70vh, + 520px)` scales with the monitor (a phone gets 70% of its height, a 4K + display never gets a needlessly tall dialog) rather than a fixed value + that would be wrong at one end or the other. */ +#customModelPickModal .modal-content { + max-height: min(70vh, 520px); + display: flex; + flex-direction: column; +} + +#customModelPickModal .modal-body { + overflow-y: auto; + flex: 1; + min-height: 0; +} + +/* Custom Model Endpoint Profiles: llama-swap model-swap confirmation — replaces a native + confirm() popup (docs/custom-model-endpoints-plan.md) so it looks and feels like the + rest of the app instead of a browser chrome dialog. Shares the context-window-too-small + modal's fixes below since both can appear mid-launch, in the same spot, for the same + reason — including this rule itself: there is no bare `.modal-footer` base style + anywhere in this file, and `.btn-toolbar` is `display: flex` (a block-level flex + container with no explicit `inline-flex`), so with no row layout of its own each + button took its own full-width line and the two stacked instead of sitting side by + side. Centred rather than flex-end per feedback — a two-button Cancel/confirm footer + reads better centred than pinned to one edge. */ +#customModelSwapConfirmModal .modal-footer, +#customModelContextWarningModal .modal-footer { + display: flex; + justify-content: center; + gap: 0.5rem; + padding: 0.75rem 1rem; + border-top: 1px solid var(--border-color); +} + +/* Both dialogs can appear while the centred llama-swap status banner (10001, see + .center-status-banner) is still on screen — right after "Claude started — switching + to llama-swap…" — and .modal's own z-index (1000) sat well under it, so the dialog + rendered fully hidden behind the banner (confirmed live, reported against the + context-window one but structurally identical for the swap-confirm modal too). */ +#customModelSwapConfirmModal, +#customModelContextWarningModal { + z-index: 10010; +} + +/* Both messages ARE the modal's whole explanatory content, not a one-line caption under + a form field, so .form-hint's 0.65rem caption size (right for what it was designed for) + read as illegibly small here, worst on the multi-sentence context-window explanation. */ +#customModelSwapConfirmMessage, +#customModelContextWarningMessage { + font-size: 0.85rem; + line-height: 1.5; + color: var(--text); +} + /* Mobile Case Picker - Base Styles */ .mobile-case-picker-sheet { @@ -8441,6 +8502,9 @@ kbd { } .toast { + display: flex; + align-items: center; + gap: 0.5rem; background: var(--bg-card); border: 1px solid var(--border); border-radius: 6px; @@ -8452,6 +8516,7 @@ kbd { opacity: 0; transition: all 0.2s ease; pointer-events: auto; + max-width: 420px; } .toast.show { @@ -8459,6 +8524,140 @@ kbd { opacity: 1; } +.toast-message { + flex: 1; + /* Errors are sticky by default (showToast) precisely so a longer, specific + message survives to be read — let it wrap instead of clipping. */ + white-space: pre-wrap; + word-break: break-word; +} + +/* Every toast gets one, sticky or not: a sticky toast with no way to close it + would just accumulate on screen across repeated failures. */ +.toast-close { + flex-shrink: 0; + background: none; + border: none; + color: inherit; + opacity: 0.6; + font-size: 1.1rem; + line-height: 1; + padding: 0 0.15rem; + cursor: pointer; +} + +.toast-close:hover { + opacity: 1; +} + +/* Custom Model Endpoint Profiles: the "switching backends" / "loading model" states + (docs/custom-model-endpoints-plan.md) — a small set of messages prominent and + screen-centred rather than corner toasts, since they can sit on screen for well + over a minute (a real llama-swap model load) and are easy to mistake for nothing + happening. Non-blocking: `pointer-events: none` on the wrapper (no backdrop, no + click-catcher) with `auto` restored only on the card itself, purely so the text + inside remains selectable — there is nothing to click to dismiss it early. */ +.center-status-banner { + position: fixed; + top: 50%; + left: 50%; + transform: translate(-50%, -50%) scale(0.96); + z-index: 10001; + display: flex; + align-items: center; + gap: 0.75rem; + background: var(--bg-card); + border: 1px solid var(--border); + border-radius: 10px; + padding: 1rem 1.5rem; + box-shadow: 0 8px 32px rgba(0, 0, 0, 0.4); + font-size: 0.95rem; + font-weight: 500; + color: var(--text); + max-width: min(90vw, 460px); + text-align: left; + opacity: 0; + pointer-events: none; + transition: + opacity 0.2s ease, + transform 0.2s ease; +} + +.center-status-banner.show { + opacity: 1; + transform: translate(-50%, -50%) scale(1); +} + +.center-status-spinner { + flex-shrink: 0; + width: 18px; + height: 18px; + border-radius: 50%; + border: 2px solid var(--border); + border-top-color: var(--accent, var(--text)); + animation: center-status-spin 0.8s linear infinite; +} + +@keyframes center-status-spin { + to { + transform: rotate(360deg); + } +} + +.center-status-text { + flex: 1; + pointer-events: auto; + white-space: pre-wrap; + word-break: break-word; +} + +/* Error variant: the load didn't finish in time — nothing is "in progress" anymore (no + spinner), and since this one doesn't dismiss itself, it needs a close button the user + can actually click, so pointer-events is restored here too (see the wrapper's own + comment on why that's `none` by default). */ +.center-status-error { + border-color: rgba(239, 68, 68, 0.5); +} + +.center-status-close { + flex-shrink: 0; + pointer-events: auto; + background: none; + border: none; + color: inherit; + opacity: 0.6; + font-size: 1.2rem; + line-height: 1; + padding: 0 0.15rem; + cursor: pointer; +} + +.center-status-close:hover { + opacity: 1; +} + +/* The Cancel button on an 'info' banner (e.g. the model-loading banner) — a real button + rather than the bare "×" close glyph above, since "Cancel" is an action with a + consequence (the caller's onCancel closes a session), not a plain dismiss. */ +.center-status-cancel { + flex-shrink: 0; + pointer-events: auto; + background: none; + border: 1px solid var(--border); + border-radius: 6px; + color: inherit; + opacity: 0.75; + font-size: 0.8rem; + font-weight: 500; + padding: 0.25rem 0.6rem; + cursor: pointer; +} + +.center-status-cancel:hover { + opacity: 1; + border-color: var(--text-muted, var(--border)); +} + .toast-success { border-color: rgba(34, 197, 94, 0.4); } .toast-error { border-color: rgba(239, 68, 68, 0.4); } .toast-warning { border-color: rgba(234, 179, 8, 0.4); } @@ -15134,6 +15333,12 @@ html[data-skin="daylight-blue"] .welcome-btn-tunnel.active:hover { .run-mode-dot.web { background: #38bdf8; } .run-mode-webviews { max-height: 180px; overflow-y: auto; } +/* Custom Model Endpoint Profiles' generated entries: `.run-mode-menu.active`'s + own `gap: 2px` only spaces its DIRECT children, and this container (like + `.run-mode-webviews` above) is one such child holding several buttons of + its own, so it needs the same gap repeated one level down or its rows sit + flush against each other. */ +.run-mode-custom-models { display: flex; flex-direction: column; gap: 2px; } /* A saved URL is a ROW: open on the left, edit + delete on the right, so a URL can be changed or removed without first opening it as a tab. The side buttons stay @@ -16207,6 +16412,29 @@ html[data-tab-orientation='vertical'] .home-sessions { gap: 3px; } +/* Custom Model Endpoint Profiles' inline add/edit form: a nested panel rather + than a modal, so it needs its own border to read as a distinct sub-section + inside .set-group-body's flat row stack. `--control-bg` rather than a + hardcoded black alpha — CLAUDE.md records that literal fill turning the + settings live preview into a grey slab on the light skins, and this panel + sits in the very same modal. */ +:is(#appSettingsModal, #sessionOptionsModal, #createCaseModal) .set-inline-form { + display: flex; + flex-direction: column; + gap: 3px; + margin-top: 6px; + padding: 10px 12px; + border: 1px solid var(--border); + border-radius: 8px; + background: var(--control-bg); +} + +:is(#appSettingsModal, #sessionOptionsModal, #createCaseModal) .set-inline-form h5 { + margin: 0 0 4px; + font-size: 0.72rem; + color: var(--text-muted); +} + /* ── rows ─────────────────────────────────────────────────────────────── */ :is(#appSettingsModal, #sessionOptionsModal, #createCaseModal) .set-row { display: flex; diff --git a/src/web/routes/custom-model-routes.ts b/src/web/routes/custom-model-routes.ts index d5b1b1e82..d55aebfd3 100644 --- a/src/web/routes/custom-model-routes.ts +++ b/src/web/routes/custom-model-routes.ts @@ -23,9 +23,45 @@ import { isBlockedWebviewUrl } from '../webview-egress-policy.js'; import { egressBlockedReason, webviewFetch } from '../webview-egress.js'; import { CustomModelHostSchema } from '../schemas.js'; import { readCustomModelHosts, writeCustomModelHosts, type CustomModelHost } from '../../custom-model-hosts.js'; +import type { CliEntry } from '../../config/cli-registry/types.js'; const CODEMAN_CONFIG_DIR = getDataDir(); const DISCOVER_TIMEOUT_MS = 8000; +const PROPS_TIMEOUT_MS = 5000; + +/** + * Claude Code's own system prompt + tool schemas cost roughly this many tokens on EVERY + * request, before a single character of conversation history — confirmed live, twice, on + * requests reporting `in:0 out:0` (the very first exchange) failing at ~36.4K tokens. No + * `CLAUDE_CODE_MAX_CONTEXT_TOKENS` value fixes this: that setting only changes when Claude + * Code decides to COMPACT conversation history, and there is no history yet on the first + * message for it to trim. A model whose real context is below this floor will refuse + * Claude Code's very first message outright, unconditionally. + * + * Set well above the ~36.4K actually measured — CLAUDE.md size, active MCP servers, and + * enabled skills all add to a project's real baseline, so the observed figure is a floor + * for THAT one workspace, not a ceiling for every one. Erring conservative here means a + * borderline-safe model still gets warned about (the user can launch anyway), rather than + * this floor missing a genuinely-too-small one because a smaller test project happened to + * fit. + */ +export const CLAUDE_MIN_SAFE_CONTEXT_TOKENS = 40000; + +/** + * True when applying this model to this CLI is heading for a guaranteed first-message + * failure per `CLAUDE_MIN_SAFE_CONTEXT_TOKENS` above. Gated on `contextLengthVar` (today, + * only claude's registry entry declares one) rather than a hardcoded mode check: a CLI + * with a small enough baseline of its own to never trip this would have no reason to + * declare the field in the first place, so the check simply never applies to it. + */ +export function exceedsSafeContextFloor( + entry: Pick, + contextLength: number | undefined +): boolean { + const cap = entry.capabilities.customModelInjection; + if (cap.kind !== 'env' || !cap.contextLengthVar) return false; + return typeof contextLength === 'number' && contextLength < CLAUDE_MIN_SAFE_CONTEXT_TOKENS; +} function adminOnly(req: FastifyRequest, reply: { code: (n: number) => unknown }): ApiResponse | null { if (!isMultiUserMode() || isAdmin(req)) return null; @@ -33,7 +69,46 @@ function adminOnly(req: FastifyRequest, reply: { code: (n: number) => unknown }) return createErrorResponse(ApiErrorCode.FORBIDDEN, 'Admin only in multi-user mode'); } -async function discoverModels(host: Pick): Promise { +/** + * `defaultModelId` names the model the Run-menu picker applies for this endpoint with + * no further choice, so it must actually be one of the discovered `models` — a schema + * `.refine()` can't see across the two fields the way this can, and would also run on + * every unrelated field edit rather than only when either of these two changes. + */ +function invalidDefaultModel(host: Pick): ApiResponse | null { + if (host.defaultModelId === undefined) return null; + if ((host.models ?? []).includes(host.defaultModelId)) return null; + return createErrorResponse( + ApiErrorCode.INVALID_INPUT, + 'defaultModelId must be one of the endpoint’s discovered models' + ); +} + +/** + * Never hand the stored credential back to the browser, on GET, POST or PUT + * alike — the file is written 0600 precisely because it holds one. `apiKeySet` + * is what lets the editor say "unchanged if left blank" without the client + * ever holding the real value: `applyStoredApiKey()` below is the other half, + * treating an absent key on PUT as "keep the stored one" rather than clearing + * it, which is what makes never returning it survivable for the edit flow. + */ +function redactApiKey(host: CustomModelHost): Omit & { apiKeySet: boolean } { + const { apiKey, ...rest } = host; + return { ...rest, apiKeySet: !!apiKey }; +} + +/** + * A PUT body with no `apiKey` (or a blank one) means "leave it alone", never + * "clear it": the editor never receives the real value to resend deliberately + * unchanged (see redactApiKey), so the only way it can tell the two apart is + * by omission. There is deliberately no way to CLEAR a key back to unset this + * way — a pre-existing limitation, not something this changes. + */ +function applyStoredApiKey(incoming: CustomModelHost, existing: CustomModelHost): CustomModelHost { + return incoming.apiKey ? incoming : { ...incoming, apiKey: existing.apiKey }; +} + +function authHeaders(host: Pick): Record { const headers: Record = {}; const apiKey = host.apiKey?.trim(); // Exactly ONE header, never both — see custom-model-hosts.ts's CustomModelAuthStyle @@ -41,14 +116,137 @@ async function discoverModels(host: Pick; + /** See `CustomModelHost.modelSizesGB` — populated for every model whose own listing states one. */ + sizesGB: Record; +} + +/** + * Best-effort: pulls a file size in GB out of a model's own `description`, when the + * server states one. llama-swap writes `"Auto-discovered 16.35 GB - parameters + * auto-fitted by llama.cpp"` for a model it found on disk itself; a hand-configured + * profile's own description (e.g. `"General-purpose reasoning model, MoE CPU-offloaded."`) + * has no such figure and correctly yields no estimate rather than a guess — there is no + * separate "give me the file size" endpoint to fall back on. + */ +function parseSizeGB(description: unknown): number | undefined { + if (typeof description !== 'string') return undefined; + const match = /(\d+(?:\.\d+)?)\s*GB\b/i.exec(description); + if (!match) return undefined; + const size = Number(match[1]); + return Number.isFinite(size) && size > 0 ? size : undefined; +} + +/** + * Best-effort: fetches `GET /props?model=` (llama.cpp-native, llama-swap-proxied) for + * ONE already-loaded model and pulls its real `n_ctx` out. Never called for a model that + * isn't already loaded — see the caller and `CustomModelHost.modelContextLengths` for why + * that's a hard safety requirement, not just a nicety: llama-swap treats this endpoint's + * `?model=` as a routing hint, and asking it about an unloaded model risks triggering an + * actual (slow, GPU-swapping) load as a side effect of what should be read-only discovery. + * Any failure (unreachable, non-2xx, missing/malformed field) is swallowed — one model's + * context length is a nice-to-have, never worth failing the whole discovery pass over. + * + * ⚠️ FALLBACK ONLY — confirmed live to be actively WRONG for a `--fit-ctx`-launched llama- + * swap backend: `/props`'s `n_ctx` read 154112 for a model llama-swap itself had launched + * with `--fit-ctx 16384` (visible in `/running`'s own `cmd`), and the real server then + * refused a request at the real 16384-token limit — `n_ctx` here appears to report the + * model's theoretical/trained maximum, not the runtime-configured one. `parseCtxFromCmd` + * (below), which reads the actual launch flag `/running` reports, is the primary source; + * this is only used when that parse comes up empty (no recognized flag in `cmd`, or `cmd` + * itself unavailable). + */ +async function fetchContextLength( + host: Pick, + modelId: string, + headers: Record +): Promise { + try { + const url = new URL(`${host.baseUrl.replace(/\/+$/, '')}/props`); + url.searchParams.set('model', modelId); + const res = await webviewFetch(url, { headers, signal: AbortSignal.timeout(PROPS_TIMEOUT_MS) }); + if (!res.ok) return undefined; + const body = (await res.json()) as { n_ctx?: unknown; default_generation_settings?: { n_ctx?: unknown } }; + const nCtx = body.n_ctx ?? body.default_generation_settings?.n_ctx; + return typeof nCtx === 'number' && Number.isFinite(nCtx) && nCtx > 0 ? nCtx : undefined; + } catch { + return undefined; + } +} +/** + * Parses the REAL configured context size out of llama-swap's own launch command for a + * model (`/running`'s `cmd` field, e.g. `"llama-server -m ... --fit-ctx 16384 ..."`) — + * the primary source for `modelContextLengths`, preferred over `/props`'s `n_ctx` (see + * `fetchContextLength`'s own doc comment for why that field is unreliable here). Checks + * `--fit-ctx` first (llama-swap's own auto-fit flag), then the plain llama.cpp + * `-c`/`--ctx-size`/`--ctx_size` flags a hand-written launch command might use instead. + * Returns `undefined` when `cmd` has none of these — not every launch command needs to + * state one explicitly (llama.cpp has its own default), and guessing one would be worse + * than the "no override applied" the caller already treats an unknown length as. + */ +function parseCtxFromCmd(cmd: unknown): number | undefined { + if (typeof cmd !== 'string') return undefined; + const match = /--fit-ctx\s+(\d+)/.exec(cmd) ?? /(?:^|\s)(?:-c|--ctx-size|--ctx_size)\s+(\d+)/.exec(cmd); + if (!match) return undefined; + const value = Number(match[1]); + return Number.isFinite(value) && value > 0 ? value : undefined; +} + +async function discoverModels( + host: Pick +): Promise { + const headers = authHeaders(host); const res = await webviewFetch(new URL(`${host.baseUrl.replace(/\/+$/, '')}/v1/models`), { headers, signal: AbortSignal.timeout(DISCOVER_TIMEOUT_MS), }); if (!res.ok) throw new Error(`HTTP ${res.status}`); - const body = (await res.json()) as { data?: Array<{ id?: unknown }> }; - return (body.data ?? []).map((m) => m.id).filter((id): id is string => typeof id === 'string' && id.length > 0); + const body = (await res.json()) as { + data?: Array<{ id?: unknown; status?: { value?: unknown }; description?: unknown }>; + }; + const entries = body.data ?? []; + const models = entries.map((m) => m.id).filter((id): id is string => typeof id === 'string' && id.length > 0); + + const sizesGB: Record = {}; + for (const entry of entries) { + if (typeof entry.id !== 'string' || !entry.id) continue; + const size = parseSizeGB(entry.description); + if (size !== undefined) sizesGB[entry.id] = size; + } + + // llama-swap-specific, feature-detected: a server that never mentions `status` on ANY + // entry gets no context-length enrichment at all, rather than treating "no status field" + // as "assume unloaded" — either reading is a guess, and skipping is the safe one, since + // fetchContextLength must only ever run against a model this server itself calls loaded. + const hasStatusField = entries.some((m) => m && typeof m === 'object' && 'status' in m); + const contextLengths: Record = {}; + if (hasStatusField) { + const loadedIds = entries + .filter((m) => m.status && typeof m.status === 'object' && (m.status as { value?: unknown }).value === 'loaded') + .map((m) => m.id) + .filter((id): id is string => typeof id === 'string' && id.length > 0); + if (loadedIds.length > 0) { + // Primary source: the REAL launch command (see parseCtxFromCmd's own doc comment + // for why /props's n_ctx cannot be trusted here). One /running call covers every + // loaded model, so this never costs more requests than the old /props-only path did + // when the cmd parse succeeds, and exactly one extra when it has to fall back. + const swapStatus = await getLlamaSwapStatus(host); + const cmdById = new Map(swapStatus.running.map((r) => [r.model, r.cmd])); + for (const id of loadedIds) { + const fromCmd = parseCtxFromCmd(cmdById.get(id)); + const ctx = fromCmd ?? (await fetchContextLength(host, id, headers)); + if (ctx !== undefined) contextLengths[id] = ctx; + } + } + } + return { models, contextLengths, sizesGB }; } /** @@ -66,41 +264,452 @@ function describeFetchError(err: unknown): string { return message; } +type RedactedHost = ReturnType; + +/** + * Merges a fresh `GET /v1/models` result into a host record: stamps + * `lastDiscoveredAt`, and drops `defaultModelId` if it no longer appears in + * the fresh list (it would otherwise leave the Run-menu picker applying a + * model id the endpoint just told us it doesn't serve). Pure — no IO, so the + * manual route (which reports a fetch failure's *reason* to the caller) and + * the periodic sweep below (which only cares whether it can move on) can + * each do their own `discoverModels()` + error handling around one shared + * "how to apply a successful result" step. + */ +const RUNNING_TIMEOUT_MS = 5000; + +export interface LlamaSwapRunningModel { + model: string; + state: string; + /** The actual launch command llama-swap started this backend with, when it says one — + * see `parseCtxFromCmd`, which reads the real configured context size out of this. */ + cmd?: string; +} + +export interface LlamaSwapStatus { + /** + * Feature-detected via `GET /running`: true only when the server answered with + * llama-swap's own shape (`{ running: [...] }`). Plain llama.cpp (and any other + * OpenAI-compatible server) has no such endpoint and always runs the single model + * it was started with, so there is no "current model" to conflict with — every + * caller must treat `isLlamaSwap: false` as "nothing to check", never as an error. + */ + isLlamaSwap: boolean; + running: LlamaSwapRunningModel[]; +} + +/** + * Distinguishes llama-swap from a plain llama.cpp/OpenAI-compatible server, and reports + * what llama-swap currently has loaded — llama.cpp only ever runs one GGUF at a time, and + * llama-swap unloads/reloads it on demand when a request asks for a different one, which + * can take anywhere from a few seconds to over a minute. Read-only: this never triggers a + * swap itself (unlike `/props?model=`, `/running` takes no `model` parameter to route by). + * Best-effort like `discoverModels()`'s siblings: any failure (unreachable, non-2xx, + * unexpected shape) reads as "not llama-swap", never thrown. + */ +export async function getLlamaSwapStatus( + host: Pick +): Promise { + try { + const res = await webviewFetch(new URL(`${host.baseUrl.replace(/\/+$/, '')}/running`), { + headers: authHeaders(host), + signal: AbortSignal.timeout(RUNNING_TIMEOUT_MS), + }); + if (!res.ok) return { isLlamaSwap: false, running: [] }; + const body = (await res.json()) as { running?: unknown }; + if (!Array.isArray(body.running)) return { isLlamaSwap: false, running: [] }; + const running = body.running + .filter( + (r): r is { model: string; state?: unknown; cmd?: unknown } => + !!r && typeof r === 'object' && typeof (r as { model?: unknown }).model === 'string' + ) + .map((r) => ({ + model: r.model, + state: typeof r.state === 'string' ? r.state : 'unknown', + cmd: typeof r.cmd === 'string' ? r.cmd : undefined, + })); + return { isLlamaSwap: true, running }; + } catch { + return { isLlamaSwap: false, running: [] }; + } +} + +interface LlamaSwapLogTail { + latestLine?: string; + lastAccessedAt: number; + controller: AbortController; +} + +/** One open `/api/events` tail per endpoint, keyed by host id — see `getLatestLlamaSwapLogLine`. */ +const llamaSwapLogTails = new Map(); + +/** A tail nothing has asked about in this long is closed by the next `pruneIdleLlamaSwapLogTails` sweep. */ +const LOG_TAIL_IDLE_MS = 30_000; + +/** + * Parses one `data: {...}` payload from llama-swap's `GET /api/events` SSE stream and + * returns the backend (never llama-swap's own proxy) log text it carries, or `undefined` + * for anything else (a different event `type`, a malformed frame, a proxy-sourced one). + * + * The real shape, confirmed live against a real llama-swap deployment — NOT documented + * anywhere the plan doc's original research found, and genuinely surprising the first + * time around: `GET /logs` (the endpoint that name suggests, and this feature's own + * first cut was built against) turns out to carry ONLY llama-swap's own proxy + * request-access log — it never once showed a single backend line even seconds after a + * real, confirmed model swap. The backend llama-server process's actual stdout + * (`load_model: ...`, `llama_server: model loaded`) only ever showed up in `/api/events`, + * as `{"type":"logData","data":""}` whose OWN `data` field parses to a + * second object, `{"data": "", "source": "proxy" | "upstream"}` + * — `source` is the exact, explicit distinguisher (`upstream` = the backend process, + * `proxy` = llama-swap's own line), not a guessed regex against the text itself. + */ +function parseBackendLogDataEvent(dataLine: string): string | undefined { + let outer: unknown; + try { + outer = JSON.parse(dataLine); + } catch { + return undefined; + } + if ( + !outer || + typeof outer !== 'object' || + (outer as { type?: unknown }).type !== 'logData' || + typeof (outer as { data?: unknown }).data !== 'string' + ) { + return undefined; + } + let inner: unknown; + try { + inner = JSON.parse((outer as { data: string }).data); + } catch { + return undefined; + } + if ( + !inner || + typeof inner !== 'object' || + (inner as { source?: unknown }).source !== 'upstream' || + typeof (inner as { data?: unknown }).data !== 'string' + ) { + return undefined; + } + return (inner as { data: string }).data; +} + +/** + * Reads `GET /api/events` forever (until `entry.controller` aborts it), updating + * `entry.latestLine` with the most recent BACKEND log line seen (see + * `parseBackendLogDataEvent`). Fire-and-forget: the caller never awaits this — it runs + * for the tail's whole lifetime in the background, and `getLatestLlamaSwapLogLine` just + * reads whatever `entry.latestLine` currently holds. SSE frames are separated by a blank + * line (`\n\n`), buffered the same way `/running`'s NDJSON-shaped siblings buffer partial + * chunks — a frame split across two `reader.read()` calls must not be parsed early. + */ +async function pumpLlamaSwapLogTail( + host: Pick, + entry: LlamaSwapLogTail +): Promise { + try { + const res = await webviewFetch(new URL(`${host.baseUrl.replace(/\/+$/, '')}/api/events`), { + headers: authHeaders(host), + signal: entry.controller.signal, + }); + if (!res.ok || !res.body) return; + const reader = res.body.getReader(); + const decoder = new TextDecoder(); + let buffer = ''; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + buffer += decoder.decode(value, { stream: true }); + const frames = buffer.split('\n\n'); + buffer = frames.pop() ?? ''; + for (const frame of frames) { + const dataLine = frame.split('\n').find((l) => l.startsWith('data:')); + if (!dataLine) continue; + const backendText = parseBackendLogDataEvent(dataLine.slice('data:'.length)); + if (!backendText) continue; + const lines = backendText.split('\n').filter((l) => l.trim()); + if (lines.length > 0) entry.latestLine = lines[lines.length - 1]!.trim(); + } + } + } catch { + // connection dropped / aborted / endpoint unreachable — a future access starts fresh + } finally { + llamaSwapLogTails.delete(host.id); + } +} + +/** + * Real-time "what is llama.cpp actually doing right now" for the loading banner + * (docs/custom-model-endpoints-plan.md): llama-swap's `GET /api/events` SSE stream + * carries the backend llama-server process's own stdout — `load_model: loading model + * ''`, `load_model: initializing, n_slots = N, n_ctx_slot = N`, `llama_server: + * model loaded`, etc — tagged `source: "upstream"`, distinct from llama-swap's own + * `source: "proxy"` request-access lines (see `parseBackendLogDataEvent`). Confirmed + * live against a real llama-swap deployment, including through an actual forced model + * swap end-to-end. + * + * Held OPEN per endpoint rather than re-opened on every 1s poll — confirmed live to stay + * open indefinitely (read past 220KB over 8 seconds with no `done`), unlike `/logs` + * (see `parseBackendLogDataEvent`'s doc comment), so reconnecting each poll would be + * pure waste. One connection is reused across every session currently watching a load on + * that endpoint; since llama.cpp/llama-swap only ever runs one model at a time, a line + * seen while a load is in flight is safe to attribute to that load (a deployment that + * could load several models concurrently would need a per-model tag this format doesn't + * provide). + * + * Lazily started on first access and idle-closed rather than left open forever — see + * `pruneIdleLlamaSwapLogTails`. + */ +export function getLatestLlamaSwapLogLine( + host: Pick +): string | undefined { + let entry = llamaSwapLogTails.get(host.id); + if (!entry) { + entry = { lastAccessedAt: Date.now(), controller: new AbortController() }; + llamaSwapLogTails.set(host.id, entry); + void pumpLlamaSwapLogTail(host, entry); + } + entry.lastAccessedAt = Date.now(); + return entry.latestLine; +} + +/** + * Closes any log tail nothing has called `getLatestLlamaSwapLogLine` about in + * `LOG_TAIL_IDLE_MS` — a stream nobody is polling is an open connection with nothing to + * show for it. Called from the same periodic sweep as `detectCustomModelSwapDisplacements` + * in server.ts, not its own timer. + */ +export function pruneIdleLlamaSwapLogTails(now = Date.now()): void { + for (const [id, entry] of llamaSwapLogTails) { + if (now - entry.lastAccessedAt > LOG_TAIL_IDLE_MS) { + entry.controller.abort(); + llamaSwapLogTails.delete(id); + } + } +} + +/** + * Actually kicks off llama-swap's lazy model load, rather than waiting for the launched + * CLI's own first prompt to do it. llama-swap has no separate "switch model" admin + * endpoint — the ONLY thing that starts a swap is a real inference request naming the + * model (confirmed live: applying a selection alone never appeared in the llama-swap + * server's own logs; nothing had actually asked it to load anything). This sends the + * smallest real request that will — `max_tokens: 1`, one throwaway user message — to + * `${baseUrl}/v1/chat/completions`, the OpenAI-compatible endpoint every supported + * harness already points at. + * + * Deliberately fire-and-forget: the caller (the apply/create routes) returns to the + * client immediately, and the frontend's own polling (`GET .../running-status`) is what + * actually confirms readiness — this call's response is never read, just its side + * effect. No abort/timeout of its own either: a real load can take well over a minute for + * a large model, and this is a normal long-running Node process, so there is nothing to + * clean up by cutting it short. Errors are swallowed for the same reason `discoverModels`'s + * siblings swallow theirs — one endpoint's hiccup here is a nice-to-have that failed, not + * something worth surfacing as a request failure four layers up. + */ +export function triggerLlamaSwapLoad( + host: Pick, + modelId: string +): void { + const url = new URL(`${host.baseUrl.replace(/\/+$/, '')}/v1/chat/completions`); + webviewFetch(url, { + method: 'POST', + headers: { ...authHeaders(host), 'content-type': 'application/json' }, + body: JSON.stringify({ + model: modelId, + messages: [{ role: 'user', content: 'Hi' }], + max_tokens: 1, + stream: false, + }), + }).catch(() => { + // best-effort — see the doc comment above + }); +} + +function applyDiscoveredModels(host: CustomModelHost, result: DiscoveryResult): CustomModelHost { + const { models, contextLengths, sizesGB } = result; + const defaultModelId = host.defaultModelId && models.includes(host.defaultModelId) ? host.defaultModelId : undefined; + // Merge onto what's already known rather than replacing: a model not probed this round + // (not currently loaded) keeps whatever context length an earlier round already learned + // for it, and one no longer in the fresh list is dropped, same reasoning as defaultModelId. + const merged = { ...host.modelContextLengths, ...contextLengths }; + const kept = Object.fromEntries(Object.entries(merged).filter(([id]) => models.includes(id))); + const modelContextLengths = Object.keys(kept).length > 0 ? kept : undefined; + // sizesGB, unlike contextLengths, is populated for every model in the SAME pass (no + // loaded-only restriction — see parseSizeGB), so this is closer to a plain replace, but + // still merges onto the previous round rather than dropping a size for a model whose + // description happened to omit the figure on this particular pass. + const mergedSizes = { ...host.modelSizesGB, ...sizesGB }; + const keptSizes = Object.fromEntries(Object.entries(mergedSizes).filter(([id]) => models.includes(id))); + const modelSizesGB = Object.keys(keptSizes).length > 0 ? keptSizes : undefined; + return { + ...host, + models, + defaultModelId, + modelContextLengths, + modelSizesGB, + lastDiscoveredAt: new Date().toISOString(), + }; +} + +/** + * Re-discovers every saved endpoint's models, best-effort. One endpoint being + * unreachable (powered off, wrong network) must not stop the others from + * refreshing, and a read-modify-write per host (rather than one batch write + * at the end) means a crash or restart mid-sweep loses at most the endpoints + * not yet reached, never a write already applied. Exported so both the + * periodic timer (server.ts) and a test can drive it directly. + */ +export async function refreshAllCustomModelHosts(): Promise { + const dataDir = getDataDir(); + const hosts = await readCustomModelHosts(dataDir); + for (const host of hosts) { + if (isBlockedWebviewUrl(host.baseUrl)) continue; + let result: DiscoveryResult; + try { + result = await discoverModels(host); + } catch { + continue; // unreachable this cycle — try again next tick, not fatal to the sweep + } + // Re-read + splice by id rather than reusing the array captured above: an + // admin editing or deleting an endpoint via the API mid-sweep must win, + // not be silently overwritten by a refresh that started before their change. + const current = await readCustomModelHosts(dataDir); + const index = current.findIndex((item) => item.id === host.id); + if (index === -1) continue; // deleted mid-sweep + current[index] = applyDiscoveredModels(current[index], result); + await writeCustomModelHosts(dataDir, current); + } +} + +/** The subset of `Session` this sweep needs — kept minimal so a test can pass a plain object. */ +export interface CustomModelSessionLike { + id: string; + name: string; + customModel?: { endpointId: string; modelId: string; label?: string }; +} + +/** One session whose model was just found evicted, ready to broadcast as `CustomModelSwappedOut`. */ +export interface CustomModelSwapDisplacement { + sessionId: string; + sessionName: string; + endpointId: string; + previousModel: string; + currentlyLoadedModel: string; +} + +/** + * Detects when a live session's own custom-model selection is no longer the model + * llama-swap actually has loaded — evicted by ANOTHER session's activity on the same + * endpoint, since llama.cpp/llama-swap runs one model at a time (the apply/create routes' + * own swap-conflict check only ever runs at THAT session's own launch/apply moment, so it + * cannot catch a later eviction triggered by a different session's normal use — confirmed + * live: a session created while nothing else had a live conflict at that instant can still + * get silently displaced afterward). Read-only, and best-effort per endpoint exactly like + * `refreshAllCustomModelHosts`'s sibling sweep — one endpoint's hiccup here never blocks + * checking the others. + * + * `notifiedSessionIds` is the caller's own de-dupe state (`server.ts` keeps one `Set` across + * sweeps), mutated in place: a session id is added once displaced and removed again once its + * own model is loaded and ready — so a LATER, genuinely new displacement can notify again + * rather than the session staying silently un-notified forever after the first one. + */ +export async function detectCustomModelSwapDisplacements( + sessions: Iterable, + notifiedSessionIds: Set +): Promise { + const byEndpoint = new Map(); + for (const session of sessions) { + if (!session.customModel) continue; + const group = byEndpoint.get(session.customModel.endpointId); + if (group) group.push(session); + else byEndpoint.set(session.customModel.endpointId, [session]); + } + if (byEndpoint.size === 0) return []; + + const hosts = await readCustomModelHosts(getDataDir()); + const displacements: CustomModelSwapDisplacement[] = []; + + for (const [endpointId, group] of byEndpoint) { + const host = hosts.find((h) => h.id === endpointId); + if (!host) continue; // endpoint deleted since these sessions were created — nothing to check + let status: LlamaSwapStatus; + try { + status = await getLlamaSwapStatus(host); + } catch { + continue; // unreachable this cycle — try again next tick, not fatal to the sweep + } + // Not llama-swap (feature-detected) or nothing loaded at all: nothing has been evicted, + // by construction — a plain llama.cpp/OpenAI-compatible server only ever runs the one + // model it was started with, so there is no "current model" to conflict with. + if (!status.isLlamaSwap || status.running.length === 0) continue; + const currentlyLoaded = status.running.find((r) => r.state === 'ready')?.model ?? status.running[0]?.model; + if (!currentlyLoaded) continue; + + for (const session of group) { + const modelId = session.customModel!.modelId; + const stillLoaded = status.running.some((r) => r.model === modelId); + if (stillLoaded) { + notifiedSessionIds.delete(session.id); // back to normal — a future eviction can notify again + continue; + } + if (notifiedSessionIds.has(session.id)) continue; // already told them once for this displacement + notifiedSessionIds.add(session.id); + displacements.push({ + sessionId: session.id, + sessionName: session.name, + endpointId, + previousModel: modelId, + currentlyLoadedModel: currentlyLoaded, + }); + } + } + return displacements; +} + export function registerCustomModelRoutes(app: FastifyInstance): void { - app.get('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/api/model-endpoints', async (req) => - isMultiUserMode() && !isAdmin(req) ? [] : readCustomModelHosts(CODEMAN_CONFIG_DIR) - ); + app.get('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/api/model-endpoints', async (req): Promise => { + if (isMultiUserMode() && !isAdmin(req)) return []; + const hosts = await readCustomModelHosts(CODEMAN_CONFIG_DIR); + return hosts.map(redactApiKey); + }); - app.post('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/api/model-endpoints', async (req, reply): Promise> => { + app.post('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/api/model-endpoints', async (req, reply): Promise> => { const denied = adminOnly(req, reply); if (denied) return denied; const host = parseBody(CustomModelHostSchema, req.body); if (isBlockedWebviewUrl(host.baseUrl)) { return createErrorResponse(ApiErrorCode.INVALID_INPUT, 'Endpoint base URL is not allowed'); } + const badDefault = invalidDefaultModel(host); + if (badDefault) return badDefault; const hosts = await readCustomModelHosts(CODEMAN_CONFIG_DIR); if (hosts.some((item) => item.id === host.id)) { return createErrorResponse(ApiErrorCode.ALREADY_EXISTS, 'Model endpoint already exists'); } await writeCustomModelHosts(CODEMAN_CONFIG_DIR, [...hosts, host]); - return { success: true, data: { host } }; + return { success: true, data: { host: redactApiKey(host) } }; }); - app.put('/api/model-endpoints/:id', async (req, reply): Promise> => { + app.put('/api/model-endpoints/:id', async (req, reply): Promise> => { const denied = adminOnly(req, reply); if (denied) return denied; const { id } = req.params as { id: string }; - const host = parseBody(CustomModelHostSchema, { ...(req.body as object), id }); - if (isBlockedWebviewUrl(host.baseUrl)) { + const incoming = parseBody(CustomModelHostSchema, { ...(req.body as object), id }); + if (isBlockedWebviewUrl(incoming.baseUrl)) { return createErrorResponse(ApiErrorCode.INVALID_INPUT, 'Endpoint base URL is not allowed'); } + const badDefault = invalidDefaultModel(incoming); + if (badDefault) return badDefault; const hosts = await readCustomModelHosts(CODEMAN_CONFIG_DIR); const index = hosts.findIndex((item) => item.id === id); if (index === -1) return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found'); + const host = applyStoredApiKey(incoming, hosts[index]); const next = [...hosts]; next[index] = host; await writeCustomModelHosts(CODEMAN_CONFIG_DIR, next); - return { success: true, data: { host } }; + return { success: true, data: { host: redactApiKey(host) } }; }); app.delete('/api/model-endpoints/:id', async (req, reply): Promise> => { @@ -129,11 +738,11 @@ export function registerCustomModelRoutes(app: FastifyInstance): void { return createErrorResponse(ApiErrorCode.INVALID_INPUT, 'Endpoint base URL is not allowed'); } try { - const models = await discoverModels(host); + const result = await discoverModels(host); const next = [...hosts]; - next[index] = { ...host, models, lastDiscoveredAt: new Date().toISOString() }; + next[index] = applyDiscoveredModels(host, result); await writeCustomModelHosts(CODEMAN_CONFIG_DIR, next); - return { success: true, data: { models } }; + return { success: true, data: { models: result.models } }; } catch (err) { const blocked = egressBlockedReason(err); return createErrorResponse( @@ -143,4 +752,25 @@ export function registerCustomModelRoutes(app: FastifyInstance): void { } } ); + + // Read-only, no admin gate: any session owner who can already point their own session + // at this endpoint (POST .../custom-model, ungated by design — see session-routes.ts) + // can equally ask what it currently has loaded, before or while that apply is pending. + app.get( + '/api/model-endpoints/:id/running-status', + async (req): Promise> => { + const { id } = req.params as { id: string }; + const hosts = await readCustomModelHosts(CODEMAN_CONFIG_DIR); + const host = hosts.find((item) => item.id === id); + if (!host) return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found'); + if (isBlockedWebviewUrl(host.baseUrl)) { + return createErrorResponse(ApiErrorCode.INVALID_INPUT, 'Endpoint base URL is not allowed'); + } + const status = await getLlamaSwapStatus(host); + // Only worth tailing /logs once llama-swap is actually confirmed — a plain + // llama.cpp/OpenAI-compatible server has no such endpoint at all. + const logLine = status.isLlamaSwap ? getLatestLlamaSwapLogLine(host) : undefined; + return { success: true, data: { ...status, logLine } }; + } + ); } diff --git a/src/web/routes/index.ts b/src/web/routes/index.ts index c1f0a952e..a5209ca97 100644 --- a/src/web/routes/index.ts +++ b/src/web/routes/index.ts @@ -27,4 +27,11 @@ export { registerWsRoutes } from './ws-routes.js'; export { registerVoiceRoutes } from './voice-routes.js'; export { registerWebviewRoutes, tryWebviewRefererFallback } from './webview-routes.js'; export { registerTabLayoutRoutes } from './tab-layout-routes.js'; -export { registerCustomModelRoutes } from './custom-model-routes.js'; +export { + registerCustomModelRoutes, + refreshAllCustomModelHosts, + detectCustomModelSwapDisplacements, + pruneIdleLlamaSwapLogTails, + type CustomModelSessionLike, + type CustomModelSwapDisplacement, +} from './custom-model-routes.js'; diff --git a/src/web/routes/session-routes.ts b/src/web/routes/session-routes.ts index 03f92e5d0..f47eebfcc 100644 --- a/src/web/routes/session-routes.ts +++ b/src/web/routes/session-routes.ts @@ -11,7 +11,7 @@ import { homedir } from 'node:os'; import { existsSync, statSync, mkdirSync, writeFileSync } from 'node:fs'; import { execFile } from 'node:child_process'; import fs from 'node:fs/promises'; -import { randomBytes } from 'node:crypto'; +import { randomBytes, randomUUID } from 'node:crypto'; import { performance } from 'node:perf_hooks'; import { ApiErrorCode, @@ -55,6 +55,12 @@ import { } from '../schemas.js'; import { readCustomModelHosts } from '../../custom-model-hosts.js'; import { applyCustomModelInjection, removeConfigDir } from '../../custom-model-injection-apply.js'; +import { + getLlamaSwapStatus, + triggerLlamaSwapLoad, + exceedsSafeContextFloor, + CLAUDE_MIN_SAFE_CONTEXT_TOKENS, +} from './custom-model-routes.js'; import { matchesPattern } from '../../config/cli-registry/patterns.js'; import { ownerLayoutKey } from '../../tab-layout-persistence.js'; import { TabLayoutValidationError } from '../../tab-layout.js'; @@ -1208,13 +1214,72 @@ export function registerSessionRoutes( if (!endpoint) { return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found'); } + const contextLength = endpoint.modelContextLengths?.[body.modelId]; + + // Some CLIs (today: only claude) carry enough of their own fixed system-prompt/tool- + // schema overhead that a small enough real context guarantees a first-message failure + // no matter what CLAUDE_CODE_MAX_CONTEXT_TOKENS says — confirmed live at ~36.4K tokens + // against a model configured with a real 16384-token context. Warn before committing + // to a restart that's certain to fail, rather than letting the user discover it via a + // cryptic 400 from the CLI itself. `confirmed` (already used for the swap-conflict + // warning below) skips this too — the user has already said "launch anyway" once. + if (!body.confirmed && exceedsSafeContextFloor(entry, contextLength)) { + return { + requiresContextWarning: true, + modelId: body.modelId, + contextLength, + minSafeContextTokens: CLAUDE_MIN_SAFE_CONTEXT_TOKENS, + }; + } + + // llama.cpp runs exactly one model at a time; llama-swap unloads and reloads it on + // demand, which can take anywhere from a few seconds to over a minute — long enough + // that a session mid-swap looks indistinguishable from one that never left the native + // backend. Feature-detected via llama-swap's own `GET /running` (a plain llama.cpp + // server has no such endpoint and reads as `isLlamaSwap: false` — nothing to check). + const swapStatus = await getLlamaSwapStatus(endpoint); + const currentlyLoaded = swapStatus.running.find((r) => r.state === 'ready')?.model ?? swapStatus.running[0]?.model; + // Distinct from targetReady below: this is ONLY about whether proceeding would evict a + // model another session is actively using — true even if nothing is loaded at all yet + // would be wrong here (nothing to evict), so this stays narrowly "a DIFFERENT model is + // currently ready". + const swapNeeded = swapStatus.isLlamaSwap && !!currentlyLoaded && currentlyLoaded !== body.modelId; + // Whether the TARGET model itself is already the one loaded and ready — false whether + // nothing is loaded yet, a different model is loaded, or this one is loaded but still + // mid-load. Drives both the actual load trigger below and modelSwapInProgress in the + // response; deliberately broader than swapNeeded, which only gates the confirmation ask. + const targetReady = swapStatus.running.some((r) => r.model === body.modelId && r.state === 'ready'); + + // Only ask when switching would actually take the model away from another session + // that is currently using it — never just because a swap is needed at all. `confirmed` + // (set by the caller after showing that warning once) skips asking again. + if (swapNeeded && !body.confirmed) { + const affectedSessions = [...ctx.sessions.values()] + .filter( + (s) => + s.id !== session.id && + s.customModel?.endpointId === endpoint.id && + s.customModel?.modelId === currentlyLoaded + ) + .map((s) => ({ id: s.id, name: s.name })); + if (affectedSessions.length > 0) { + return { requiresConfirmation: true, currentlyLoadedModel: currentlyLoaded, affectedSessions }; + } + } // A CLI whose config alone cannot select the model also gets its `model` launch param // forced (pi/omp `custom/`, grok's block name). The argv engine DROPS a token that // fails its pattern rather than quoting it, which would silently launch the CLI on its // own default provider again, so refuse an id the pattern cannot carry up front. const modelSpec = entry.launch.params.model; - const applied = applyCustomModelInjection(entry, endpoint, body.modelId, session.id); + const applied = applyCustomModelInjection( + entry, + endpoint, + body.modelId, + session.id, + contextLength, + session.workingDir + ); if (!applied) { return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${session.mode} has no known custom-model mechanism`); } @@ -1247,9 +1312,17 @@ export function registerSessionRoutes( removeConfigDir(previousConfigDir); } + // Actually kick off llama-swap's load now, rather than waiting on the restarted CLI's + // own first prompt to do it — confirmed live that applying a selection alone never + // reached the llama-swap server at all (nothing in its own logs), since llama-swap has + // no "switch model" admin call, only a real inference request naming the model. + if (swapStatus.isLlamaSwap && !targetReady) { + triggerLlamaSwapLoad(endpoint, body.modelId); + } + const restarted = await session.restartCli(); persistAndBroadcastSession(ctx, session); - return { customModel: session.customModel, restarted }; + return { customModel: session.customModel, restarted, modelSwapInProgress: swapStatus.isLlamaSwap && !targetReady }; }); // ========== Delete Session ========== @@ -3108,6 +3181,7 @@ export function registerSessionRoutes( effort, parentSessionId, agentOrigin, + customModel, } = parseBody(QuickStartSchema, req.body); // Resolved ONCE here: the same value labels a case directory this request creates @@ -3160,11 +3234,12 @@ export function registerSessionRoutes( grokConfig || deepSeekConfig || ompConfig || - openCodeConfig + openCodeConfig || + customModel ) { return createErrorResponse( ApiErrorCode.INVALID_INPUT, - 'envOverrides, effort, modelOverride, and per-CLI config are not supported for remote cases (they do not cross ssh). Configure the remote command via the host command override instead.' + 'envOverrides, effort, modelOverride, per-CLI config, and custom model endpoints are not supported for remote cases (they do not cross ssh). Configure the remote command via the host command override instead.' ); } @@ -3195,11 +3270,12 @@ export function registerSessionRoutes( grokConfig || deepSeekConfig || ompConfig || - openCodeConfig + openCodeConfig || + customModel ) { return createErrorResponse( ApiErrorCode.INVALID_INPUT, - 'envOverrides, effort, and per-CLI config are not supported for docker cases (they do not cross into the container). Configure the container via the docker host command override instead.' + 'envOverrides, effort, per-CLI config, and custom model endpoints are not supported for docker cases (they do not cross into the container). Configure the container via the docker host command override instead.' ); } @@ -3487,7 +3563,134 @@ export function registerSessionRoutes( ); const qsTerminalHistoryConfig = await ctx.getTerminalHistoryConfig(); const qsGatedEnvOverrides = await clampEnvOverridesForOwner(owner, envOverrides); + const qsResolvedOmpConfig = resolveOmpConfigForCreate(mode, resolvedCasePath, ompConfig); + + // Custom Model Endpoint Profiles, applied AT CREATE TIME (docs/custom-model-endpoints-plan.md) + // rather than via the dedicated restart-in-place route (POST /api/sessions/:id/custom- + // model, still what an ALREADY-RUNNING session uses to switch later): computing the + // injection before the process exists and launching directly on it avoids the visible + // native-boot-then-restart the restart-after-launch design otherwise shows on every + // custom-model run — most jarring on a CLI like Codex whose TUI fully reinitializes. + // Mirrors the dedicated route's own checks (llama-swap conflict, unsupported CLI, + // unknown endpoint, a model id the CLI's argv pattern can't carry) rather than trusting + // a lighter version of them, since this is the same server-side authority reached a + // different way, not a separate, less-checked path. + let qsCustomModelEnvOverrides = qsGatedEnvOverrides; + let qsCustomModelLaunchModel: string | undefined; + let qsCustomModelSessionId: string | undefined; + let qsCustomModelSwapInProgress = false; + let qsCustomModelBookkeeping: + | { + endpointId: string; + modelId: string; + label?: string; + envKeys: string[]; + configDir?: string; + launchModel?: string; + } + | undefined; + if (customModel) { + const cmEntry = getCli(mode); + if (!cmEntry) return createErrorResponse(ApiErrorCode.INVALID_INPUT, `No CLI registry entry for mode ${mode}`); + if (cmEntry.capabilities.customModelInjection.kind === 'unsupported') { + return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${mode} has no known custom-model mechanism`); + } + const cmHosts = await readCustomModelHosts(CODEMAN_CONFIG_DIR); + const cmEndpoint = cmHosts.find((h) => h.id === customModel.endpointId); + if (!cmEndpoint) return createErrorResponse(ApiErrorCode.NOT_FOUND, 'Model endpoint not found'); + const cmContextLength = cmEndpoint.modelContextLengths?.[customModel.modelId]; + + // See the dedicated route's own comment for the full reasoning: some CLIs' own fixed + // overhead can exceed a small enough real context on the very first message, + // regardless of contextLengthVar. Warn before creating a session that's certain to + // fail immediately. + if (!customModel.confirmed && exceedsSafeContextFloor(cmEntry, cmContextLength)) { + return { + requiresContextWarning: true, + modelId: customModel.modelId, + contextLength: cmContextLength, + minSafeContextTokens: CLAUDE_MIN_SAFE_CONTEXT_TOKENS, + }; + } + + // See the dedicated route's own comment for the full reasoning: llama.cpp runs one + // model at a time, llama-swap swaps on demand, and switching away from what another + // live session is actively using deserves a warning, not a silent switch. There is no + // "self" to exclude from the affected-sessions scan here — this session doesn't exist + // yet. + const cmSwapStatus = await getLlamaSwapStatus(cmEndpoint); + const cmCurrentlyLoaded = + cmSwapStatus.running.find((r) => r.state === 'ready')?.model ?? cmSwapStatus.running[0]?.model; + const cmSwapNeeded = cmSwapStatus.isLlamaSwap && !!cmCurrentlyLoaded && cmCurrentlyLoaded !== customModel.modelId; + // Broader than cmSwapNeeded (which only gates the confirmation ask above): true + // whenever the TARGET model isn't already loaded and ready, including when nothing + // is loaded at all yet. Drives the actual load trigger below. + const cmTargetReady = cmSwapStatus.running.some((r) => r.model === customModel.modelId && r.state === 'ready'); + qsCustomModelSwapInProgress = cmSwapStatus.isLlamaSwap && !cmTargetReady; + if (cmSwapNeeded && !customModel.confirmed) { + const cmAffectedSessions = [...ctx.sessions.values()] + .filter((s) => s.customModel?.endpointId === cmEndpoint.id && s.customModel?.modelId === cmCurrentlyLoaded) + .map((s) => ({ id: s.id, name: s.name })); + if (cmAffectedSessions.length > 0) { + return { + requiresConfirmation: true, + currentlyLoadedModel: cmCurrentlyLoaded, + affectedSessions: cmAffectedSessions, + }; + } + } + + // Minted ourselves (rather than left to Session's own default) so the injection + // below — and any configDir it writes — can target the REAL id the session launches + // with, not a placeholder: `new Session({ id: ... })` accepts an explicit id for + // exactly this reason. + qsCustomModelSessionId = randomUUID(); + const cmApplied = applyCustomModelInjection( + cmEntry, + cmEndpoint, + customModel.modelId, + qsCustomModelSessionId, + cmContextLength, + resolvedCasePath + ); + if (!cmApplied) { + return createErrorResponse(ApiErrorCode.OPERATION_FAILED, `${mode} has no known custom-model mechanism`); + } + const cmModelSpec = cmEntry.launch.params.model; + if ( + cmApplied.launchModel !== undefined && + cmModelSpec?.type === 'token' && + !matchesPattern(cmModelSpec.pattern, cmApplied.launchModel) + ) { + removeConfigDir(cmApplied.configDir); + return createErrorResponse( + ApiErrorCode.INVALID_INPUT, + `Model id ${JSON.stringify(customModel.modelId)} cannot be passed to ${mode} on its command line` + ); + } + + qsCustomModelEnvOverrides = { ...qsGatedEnvOverrides, ...cmApplied.envOverrides }; + qsCustomModelLaunchModel = cmApplied.launchModel; + qsCustomModelBookkeeping = { + endpointId: cmEndpoint.id, + modelId: customModel.modelId, + label: cmEndpoint.label, + envKeys: cmApplied.envKeys, + configDir: cmApplied.configDir, + launchModel: cmApplied.launchModel, + }; + + // Actually kick off llama-swap's load now — see the dedicated apply route's own + // comment on triggerLlamaSwapLoad for why this can't just wait on the launched CLI's + // first prompt. Fired here, before the session is even created, so the load starts + // concurrently with Claude/Codex/etc. booting rather than after. + if (qsCustomModelSwapInProgress) { + triggerLlamaSwapLoad(cmEndpoint, customModel.modelId); + } + } + const session = new Session({ + id: qsCustomModelSessionId, workingDir: resolvedCasePath, name: sessionName ? sessionName.slice(0, MAX_SESSION_NAME_LENGTH) : '', mux: ctx.mux, @@ -3502,11 +3705,24 @@ export function registerSessionRoutes( codexConfig: mode === 'codex' ? qsGatedCodexConfig : undefined, geminiConfig: mode === 'gemini' ? qsGatedGeminiConfig : undefined, antigravityConfig: mode === 'antigravity' ? qsGatedAntigravityConfig : undefined, - piConfig: mode === 'pi' ? qsGatedPiConfig : undefined, - grokConfig: mode === 'grok' ? qsGatedGrokConfig : undefined, + piConfig: + mode === 'pi' + ? qsCustomModelLaunchModel !== undefined + ? { ...(qsGatedPiConfig ?? {}), model: qsCustomModelLaunchModel } + : qsGatedPiConfig + : undefined, + grokConfig: + mode === 'grok' + ? qsCustomModelLaunchModel !== undefined + ? { ...(qsGatedGrokConfig ?? {}), model: qsCustomModelLaunchModel } + : qsGatedGrokConfig + : undefined, deepSeekConfig: mode === 'deepseek' ? qsGatedDeepSeekConfig : undefined, - ompConfig: resolveOmpConfigForCreate(mode, resolvedCasePath, ompConfig), - envOverrides: qsGatedEnvOverrides, + ompConfig: + mode === 'omp' && qsCustomModelLaunchModel !== undefined + ? { ...(qsResolvedOmpConfig ?? {}), model: qsCustomModelLaunchModel } + : qsResolvedOmpConfig, + envOverrides: qsCustomModelEnvOverrides, effort, remote, docker, @@ -3515,6 +3731,15 @@ export function registerSessionRoutes( parentSessionId: qsParentSessionId, }); + // Records the selection for session.customModel/getCustomModelForPersist() and future + // clear/switch calls — the actual env vars and launch-model config are already part of + // the launch above (constructor envOverrides, piConfig/grokConfig/ompConfig.model), so + // this is bookkeeping only, never a restart: setCustomModel() is synchronous state, no + // tmux IO of its own (see its own doc comment in session.ts). + if (qsCustomModelBookkeeping) { + session.setCustomModel(qsCustomModelBookkeeping, qsCustomModelEnvOverrides); + } + // Auto-detect completion phrase from CLAUDE.md BEFORE broadcasting // so the initial state already has the phrase configured (only if globally enabled) if (getCli(mode)?.capabilities.ralph && !remote && !docker && ctx.store.getConfig().ralphEnabled) { @@ -3609,6 +3834,7 @@ export function registerSessionRoutes( sessionId: session.id, casePath: resolvedCasePath, caseName, + ...(customModel ? { modelSwapInProgress: qsCustomModelSwapInProgress } : {}), }; } catch (err) { // Clean up session on error to prevent orphaned resources diff --git a/src/web/schemas.ts b/src/web/schemas.ts index b5dabdd09..686d1370d 100644 --- a/src/web/schemas.ts +++ b/src/web/schemas.ts @@ -1033,6 +1033,25 @@ export const QuickStartSchema = z.object({ * because it takes an existing `workingDir` and so never creates a directory to label. */ agentOrigin: z.string().max(64).optional(), + /** + * Custom Model Endpoint Profiles (docs/custom-model-endpoints-plan.md): launches directly + * on this saved endpoint/model instead of the mode's native backend, computed server-side + * from the admin-configured endpoint store the same way `POST /api/sessions/:id/custom- + * model` does — never trusting raw env values from the client. One-shot, launch-time + * equivalent of that route: no restart, so no visible relaunch (that route's restart-in- + * place is still what an ALREADY-RUNNING session uses to switch later). Rejected for + * remote/docker cases, same reasoning as `envOverrides` above. `confirmed` mirrors that + * route's field: skips the llama-swap "this will unload it for another session" check on + * a deliberate retry. + */ + customModel: z + .object({ + endpointId: z.string().regex(/^[a-zA-Z0-9_-]+$/, 'Invalid endpoint id'), + modelId: z.string().min(1).max(200), + confirmed: z.boolean().optional(), + }) + .strict() + .optional(), }); // ========== Hook Events ========== @@ -1918,6 +1937,17 @@ export const CustomModelHostSchema = z.object({ authStyle: z.enum(['bearer', 'api-key']).optional(), models: z.array(z.string().max(200)).max(200).optional(), lastDiscoveredAt: z.string().max(64).optional(), + // The Run-menu picker's per-endpoint default; validated against `models` at the + // route layer (schema-level cross-field checks can't see the array narrowed the + // same way a `.refine()` closure could, and the route already re-reads the stored + // host to apply it, so the check belongs there once, not duplicated into a refine + // that would run on every unrelated field edit too). + defaultModelId: z.string().max(200).optional(), + // Server-populated by discovery (custom-model-routes.ts); accepted here only so a client + // round-tripping the GET response back through PUT (edit-save) doesn't drop it. + modelContextLengths: z.record(z.string().max(200), z.number().int().positive().max(100_000_000)).optional(), + // Same reasoning as modelContextLengths above. + modelSizesGB: z.record(z.string().max(200), z.number().positive().max(100_000)).optional(), }); /** POST /api/sessions/:id/custom-model — apply or clear a session's custom-model selection. */ @@ -1925,6 +1955,10 @@ export const CustomModelSelectionSchema = z.union([ z.object({ endpointId: z.string().regex(/^[a-zA-Z0-9_-]+$/, 'Invalid endpoint id'), modelId: z.string().min(1).max(200), + // Set once the caller has already shown the "this will unload for session(s) + // X" warning (see session-routes.ts's llama-swap conflict check) and the user chose to + // proceed anyway — skips that check on this call instead of asking again. + confirmed: z.boolean().optional(), }), z.object({ clear: z.literal(true) }), ]); diff --git a/src/web/server.ts b/src/web/server.ts index cf5927c0c..52e5b2254 100644 --- a/src/web/server.ts +++ b/src/web/server.ts @@ -67,7 +67,7 @@ import { import { imageWatcher } from '../image-watcher.js'; import { workflowRunWatcher, summarizeRun } from '../workflow-run-watcher.js'; import { attachmentRegistry, buildFileThumbnailRoute, registerExternalAttachment } from '../attachment-registry.js'; -import { getCli } from '../config/cli-registry/registry.js'; +import { getCli, enabledClis } from '../config/cli-registry/registry.js'; import { readCustomModelHosts } from '../custom-model-hosts.js'; import { applyCustomModelInjection, customModelConfigDir, removeConfigDir } from '../custom-model-injection-apply.js'; import type { CustomModelBookkeeping } from '../types/session.js'; @@ -190,6 +190,9 @@ import { registerWebviewRoutes, registerTabLayoutRoutes, registerCustomModelRoutes, + refreshAllCustomModelHosts, + detectCustomModelSwapDisplacements, + pruneIdleLlamaSwapLogTails, tryWebviewRefererFallback, } from './routes/index.js'; import { isLostWebviewFrameNavigation } from './webview-proxy.js'; @@ -202,11 +205,32 @@ const __dirname = dirname(fileURLToPath(import.meta.url)); // while capping growth of `sseClientsById` and blocking pathological inputs. const SSE_CLIENT_ID_RE = /^[A-Za-z0-9_-]{8,64}$/; const CODEX_USAGE_POLL_INTERVAL_MS = 5 * 60_000; +const CUSTOM_MODEL_REDISCOVER_INTERVAL_MS = 5 * 60_000; +// Much shorter than the model-LIST refresh above on purpose: this catches an actual +// eviction (a session's model no longer loaded, silently swapped out by another +// session's use), which the user wants to know about promptly, not once every 5 +// minutes. Cheap either way — one /running GET per distinct endpoint with at least +// one live custom-model session, not per session. +const CUSTOM_MODEL_SWAP_CHECK_INTERVAL_MS = 20_000; function escapeHtmlText(value: string): string { return value.replaceAll('&', '&').replaceAll('<', '<').replaceAll('>', '>'); } +/** + * Escapes a JSON string for safe embedding as the body of an inline `` would otherwise close the tag early + * and turn the rest of the document into inert script-body text. Exported so + * it unit-tests without constructing a WebServer (which needs a real tmux). + */ +export function escapeScriptJson(json: string): string { + return json.replace(/ = new Map(); private scheduledRuns: Map = new Map(); + /** De-dupe state for the swap-displacement sweep — see detectCustomModelSwapDisplacements. */ + private _customModelDisplacedNotified: Set = new Set(); /** Cron service (assigned in setupRoutes). */ private cronService!: CronService; private sse: SseStreamManager; @@ -1301,6 +1327,10 @@ export class WebServer extends EventEmitter { session.ralphTracker.stopWatchingFixPlan(); } + // Custom Model Endpoint Profiles: drop this session's swap-displacement notify flag + // (see _checkCustomModelSwapDisplacements below) so it can't linger in that Set forever. + this._customModelDisplacedNotified.delete(sessionId); + // Kill all subagents spawned by this session (scoped to sessionId to avoid cross-session kills) if (session && killMux) { try { @@ -1596,6 +1626,23 @@ export class WebServer extends EventEmitter { '', `\n` ); + // Which run modes the Run-menu picker (docs/custom-model-endpoints-plan.md) may + // generate an entry for: read generically off the registry's `capabilities` + // (never an id list here) so a CLI whose customModelInjection lands later shows + // up in the picker with no frontend change, and one that ships `unsupported` + // (antigravity, and `shell`'s `kind !== 'agent'`) never does. + const customModelClis = enabledClis() + .filter((entry) => entry.kind === 'agent' && entry.capabilities.customModelInjection.kind !== 'unsupported') + .map((entry) => ({ id: entry.id, label: entry.label })); + // Unlike the boolean-only __codemanCliAvailable above, this payload carries + // `label`, a string a user's own clis.json can set (CliEntry.label, up to 60 + // chars) — see escapeScriptJson's own doc comment for why that needs escaping + // and __codemanCliAvailable's booleans never did. + const customModelClisJson = escapeScriptJson(JSON.stringify(customModelClis)); + html = html.replace( + '', + `\n` + ); } if (!soloSessionId && process.env.CODEMAN_GESTURE === '1') { html = html.replace('', `\n`); @@ -2703,6 +2750,55 @@ export class WebServer extends EventEmitter { }); } + // Custom Model Endpoint Profiles (docs/custom-model-endpoints-plan.md): keeps + // each saved endpoint's discovered model list current with no manual + // "Discover" click, so a model added on the server side (or one that drops + // off) shows up in the Run-menu picker within one cycle. Best-effort per + // endpoint (refreshAllCustomModelHosts skips one that's unreachable rather + // than failing the sweep) and off in tests for the same reason the Codex + // poll above is — no real network to hit, no server instance to keep alive. + if (!this.testMode) { + this.cleanup.setInterval( + () => { + refreshAllCustomModelHosts().catch((err) => { + console.error('[custom-model] periodic re-discovery failed:', getErrorMessage(err)); + }); + }, + CUSTOM_MODEL_REDISCOVER_INTERVAL_MS, + { description: 'custom model endpoint re-discovery' } + ); + } + + // Custom Model Endpoint Profiles: the swap-conflict check on the apply/create routes + // only ever runs at THAT session's own launch/apply moment — it cannot catch a LATER + // eviction triggered by a different session's normal use, since llama-swap has no push + // notification of its own and only swaps in response to a real inference request + // (confirmed live: a session created while nothing else conflicted at that instant can + // still get silently displaced afterward). This periodic sweep is what catches that + // case after the fact and tells the displaced session's user, rather than leaving them + // to discover it only when their next prompt behaves unexpectedly. + if (!this.testMode) { + this.cleanup.setInterval( + () => { + detectCustomModelSwapDisplacements(this.sessions.values(), this._customModelDisplacedNotified) + .then((displacements) => { + for (const displacement of displacements) { + this.broadcast(SseEvent.CustomModelSwappedOut, displacement); + } + }) + .catch((err) => { + console.error('[custom-model] swap-displacement check failed:', getErrorMessage(err)); + }); + // Same cadence, unrelated concern: close any /logs tail (see + // getLatestLlamaSwapLogLine) nothing has polled in a while, so a loading banner + // that finished (or was abandoned) doesn't leave a connection open forever. + pruneIdleLlamaSwapLogTails(); + }, + CUSTOM_MODEL_SWAP_CHECK_INTERVAL_MS, + { description: 'custom model swap-displacement check' } + ); + } + // Start scheduled runs cleanup timer this.cleanup.setInterval( () => { diff --git a/src/web/sse-events.ts b/src/web/sse-events.ts index b89c34252..276cdd0ed 100644 --- a/src/web/sse-events.ts +++ b/src/web/sse-events.ts @@ -5,7 +5,7 @@ * and referenced by the frontend (`SSE_EVENTS` in `constants.js`). * Both files MUST be kept in sync. * - * 158 event constants organized by category: + * 159 event constants organized by category: * - **Core** (1): init * - **Transport** (1): sse:heartbeat * - **Session lifecycle** (23): created, updated, deleted, terminal, idle, working, ... @@ -28,6 +28,7 @@ * - **Hooks** (10): idle_prompt, permission_prompt, elicitation_dialog, elicitation_complete, elicitation_response, stop, agent_working, teammate_idle, task_completed, prompt_submitted * (agent_working is the odd one out: reported by the DeepSeek Harness status bridge, not by a Claude Code hook) * - **Approvals** (3): pending, updated, resolved (cross-session Approvals Inbox) + * - **Custom Model Endpoint Profiles** (1): swapped-out (a session's model got evicted by another session on the same llama-swap endpoint) * - **Orchestrator** (12): stateChanged, planProgress, planReady, phase*, verification, task*, completed, error * - **Clipboard** (1): write * - **Cases** (4): created, linked, deleted, order-changed @@ -384,6 +385,19 @@ export const ApprovalUpdated = 'approval:updated' as const; /** A pending approval left the inbox (answered, superseded, expired, ...). */ export const ApprovalResolved = 'approval:resolved' as const; +// ─── Custom Model Endpoint Profiles ────────────────────────────────────────── + +/** + * A session's own custom-model selection is no longer the model llama-swap has loaded — + * ANOTHER session's activity on the same endpoint evicted it (llama.cpp/llama-swap runs + * one model at a time). Detected after the fact by a periodic sweep (`server.ts`), never + * at the moment of eviction itself, since llama-swap has no push notification of its own; + * this session's next prompt will trigger reloading its model, evicting whatever displaced + * it in turn. Fires at most once per displacement (cleared once the sweep sees the + * session's own model loaded again), so it can't spam on every sweep interval. + */ +export const CustomModelSwappedOut = 'custom-model:swapped-out' as const; + // ─── Orchestrator ──────────────────────────────────────────────────────────── /** Orchestrator state machine transitioned. */ @@ -638,6 +652,9 @@ export const SseEvent = { ApprovalUpdated, ApprovalResolved, + // Custom Model Endpoint Profiles + CustomModelSwappedOut, + // Orchestrator OrchestratorStateChanged, OrchestratorPlanProgress, diff --git a/test/cli-registry-no-id-branching.test.ts b/test/cli-registry-no-id-branching.test.ts index 8d1bc4de8..b5ba1b3e0 100644 --- a/test/cli-registry-no-id-branching.test.ts +++ b/test/cli-registry-no-id-branching.test.ts @@ -80,6 +80,8 @@ const ALLOWED_BRANCHES: Record = { "web/routes/session-routes.ts::mode === 'pi'": 'legacy Config plumbing', "web/routes/session-routes.ts::mode === 'grok'": 'legacy Config plumbing', "web/routes/session-routes.ts::mode === 'deepseek'": 'legacy Config plumbing', + "web/routes/session-routes.ts::mode === 'omp'": + 'legacy Config plumbing (custom-model launchModel merge onto ompConfig, same selection resolveOmpConfigForCreate already makes internally)', "web/server.ts::mode === 'opencode'": 'legacy Config plumbing (session recovery)', "web/server.ts::mode === 'codex'": 'legacy Config plumbing (session recovery)', "web/server.ts::mode === 'gemini'": 'legacy Config plumbing (session recovery)', diff --git a/test/custom-model-endpoint-rediscovery.test.ts b/test/custom-model-endpoint-rediscovery.test.ts new file mode 100644 index 000000000..b0dedf7c2 --- /dev/null +++ b/test/custom-model-endpoint-rediscovery.test.ts @@ -0,0 +1,368 @@ +/** + * @fileoverview Tests for `refreshAllCustomModelHosts()`, the periodic + * background sweep behind server.ts's "custom model endpoint re-discovery" + * timer (docs/custom-model-endpoints-plan.md). Kept in its own file rather + * than folded into test/routes/custom-model-routes.test.ts: that file's data + * dir is shared across every test in it (one temp HOME per FILE, not per + * test — test/setup.ts), and a sweep that walks every saved host would pick + * up every host any other test in that file happened to create, making an + * exact call-count or exact-host assertion meaningless. A dedicated file + * gets its own clean temp HOME. + * + * Port: N/A (no server; drives readCustomModelHosts/writeCustomModelHosts + * directly plus the mocked webviewFetch dispatcher). + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { getDataDir } from '../src/config/instance.js'; +import { readCustomModelHosts, writeCustomModelHosts, type CustomModelHost } from '../src/custom-model-hosts.js'; +import { refreshAllCustomModelHosts } from '../src/web/routes/custom-model-routes.js'; +import { webviewFetch } from '../src/web/webview-egress.js'; + +vi.mock('../src/web/webview-egress.js', async () => { + const actual = await vi.importActual('../src/web/webview-egress.js'); + return { ...actual, webviewFetch: vi.fn() }; +}); + +const fetchMock = vi.mocked(webviewFetch); + +function host(overrides: Partial & Pick): CustomModelHost { + return { label: overrides.id, ...overrides }; +} + +beforeEach(() => { + fetchMock.mockReset(); +}); + +describe('refreshAllCustomModelHosts (the periodic re-discovery sweep)', () => { + it('refreshes every saved endpoint, best-effort — one unreachable host does not stop the others', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [ + host({ id: 'ok', baseUrl: 'http://localhost:8080' }), + host({ id: 'down', baseUrl: 'http://localhost:8081' }), + ]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.href.includes('8081')) throw new TypeError('fetch failed', { cause: new Error('ECONNREFUSED') }); + return new Response(JSON.stringify({ data: [{ id: 'qwen3' }] }), { status: 200 }); + }); + + await refreshAllCustomModelHosts(); + + const hosts = await readCustomModelHosts(dir); + const ok = hosts.find((h) => h.id === 'ok'); + const down = hosts.find((h) => h.id === 'down'); + expect(ok?.models).toEqual(['qwen3']); + expect(ok?.lastDiscoveredAt).toBeTruthy(); + expect(down?.models ?? []).toEqual([]); + expect(down?.lastDiscoveredAt).toBeFalsy(); + }); + + it('skips a host whose baseUrl is blocked, without making a request', async () => { + const dir = getDataDir(); + // Written directly rather than through the POST route, which already + // refuses this at save time — this simulates a record that pre-dates the + // guard, or was hand-edited on disk. The sweep must not trust it either. + await writeCustomModelHosts(dir, [host({ id: 'meta', baseUrl: 'http://169.254.169.254/' })]); + + fetchMock.mockResolvedValue(new Response(JSON.stringify({ data: [{ id: 'x' }] }), { status: 200 })); + await refreshAllCustomModelHosts(); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('drops a stale default and preserves lastDiscoveredAt semantics, same as manual discovery', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [ + host({ id: 'ep', baseUrl: 'http://localhost:8080', models: ['qwen3'], defaultModelId: 'qwen3' }), + ]); + fetchMock.mockResolvedValue(new Response(JSON.stringify({ data: [{ id: 'llama3' }] }), { status: 200 })); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.models).toEqual(['llama3']); + expect(updated.defaultModelId).toBeUndefined(); + expect(updated.lastDiscoveredAt).toBeTruthy(); + }); + + it('keeps a default that is still present after the sweep', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [ + host({ id: 'ep', baseUrl: 'http://localhost:8080', models: ['qwen3'], defaultModelId: 'qwen3' }), + ]); + fetchMock.mockResolvedValue( + new Response(JSON.stringify({ data: [{ id: 'qwen3' }, { id: 'llama3' }] }), { status: 200 }) + ); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.defaultModelId).toBe('qwen3'); + }); + + it('does not resurrect an endpoint deleted while the sweep was in flight', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'deleted', baseUrl: 'http://localhost:8080' })]); + + fetchMock.mockImplementation(async () => { + // Simulate an admin deleting the endpoint between the sweep's fetch and + // its read-modify-write — the delete must win, not be overwritten by a + // refresh that started before it. + const current = await readCustomModelHosts(dir); + await writeCustomModelHosts( + dir, + current.filter((h) => h.id !== 'deleted') + ); + return new Response(JSON.stringify({ data: [{ id: 'qwen3' }] }), { status: 200 }); + }); + + await expect(refreshAllCustomModelHosts()).resolves.toBeUndefined(); + const hosts = await readCustomModelHosts(dir); + expect(hosts.find((h) => h.id === 'deleted')).toBeUndefined(); + }); + + it('leaves the store untouched when there are no saved endpoints at all', async () => { + await expect(refreshAllCustomModelHosts()).resolves.toBeUndefined(); + expect(fetchMock).not.toHaveBeenCalled(); + }); +}); + +describe('refreshAllCustomModelHosts: context-length enrichment (llama.cpp/llama-swap /props)', () => { + it('probes /props?model= only for a model reported loaded, and stores its n_ctx', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '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/v1/models') { + return new Response( + JSON.stringify({ + data: [ + { id: 'loaded-model', status: { value: 'loaded' } }, + { id: 'unloaded-model', status: { value: 'unloaded' } }, + ], + }), + { status: 200 } + ); + } + if (url.pathname === '/props') { + // Must never be reached for the unloaded model — asserted below by call count. + expect(url.searchParams.get('model')).toBe('loaded-model'); + return new Response(JSON.stringify({ n_ctx: 16384 }), { status: 200 }); + } + throw new Error(`unexpected request: ${url.href}`); + }); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelContextLengths).toEqual({ 'loaded-model': 16384 }); + const propsCalls = fetchMock.mock.calls.filter(([url]) => (url as URL).pathname === '/props'); + expect(propsCalls).toHaveLength(1); + }); + + it('never probes /props at all when no entry mentions status — feature-detected, not assumed unloaded', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockResolvedValue(new Response(JSON.stringify({ data: [{ id: 'qwen3' }] }), { status: 200 })); + + await refreshAllCustomModelHosts(); + + expect(fetchMock).toHaveBeenCalledTimes(1); // /v1/models only + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelContextLengths).toBeUndefined(); + }); + + it('keeps a previously-learned context length for a model no longer loaded, drops it once the model disappears entirely', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [ + host({ + id: 'ep', + baseUrl: 'http://localhost:8080', + models: ['a', 'b'], + modelContextLengths: { a: 8192, b: 4096 }, + }), + ]); + // This round: 'a' is loaded (re-confirmed), 'b' is gone from the list entirely. + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '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/v1/models') { + return new Response(JSON.stringify({ data: [{ id: 'a', status: { value: 'loaded' } }] }), { status: 200 }); + } + return new Response(JSON.stringify({ n_ctx: 8192 }), { status: 200 }); + }); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelContextLengths).toEqual({ a: 8192 }); + }); + + it('a failed /props probe for the loaded model is swallowed, leaving no context length rather than failing the sweep', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '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/v1/models') { + return new Response(JSON.stringify({ data: [{ id: 'a', status: { value: 'loaded' } }] }), { status: 200 }); + } + return new Response('nope', { status: 500 }); + }); + + await expect(refreshAllCustomModelHosts()).resolves.toBeUndefined(); + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelContextLengths).toBeUndefined(); + }); + + it('prefers the REAL configured context size parsed from /running’s launch command over /props’s unreliable n_ctx', async () => { + // Confirmed live: llama-swap launched a model with --fit-ctx 16384 (the real, working + // limit — the actual server then refused a request over it), but /props reported + // n_ctx: 154112 for the same model, well over what it would really accept. /props must + // never be reached at all once the /running command parse already answered it. + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '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/v1/models') { + return new Response(JSON.stringify({ data: [{ id: 'qwen3.8-27b', status: { value: 'loaded' } }] }), { + status: 200, + }); + } + if (url.pathname === '/running') { + return new Response( + JSON.stringify({ + running: [ + { + model: 'qwen3.8-27b', + state: 'ready', + cmd: 'llama-server -m /models/Qwen3.8-27B.gguf --flash-attn on --jinja --fit-ctx 16384 --host 0.0.0.0 --port 5840', + }, + ], + }), + { status: 200 } + ); + } + if (url.pathname === '/props') throw new Error('must never be reached — the cmd parse already answered it'); + throw new Error(`unexpected request: ${url.href}`); + }); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelContextLengths).toEqual({ 'qwen3.8-27b': 16384 }); + }); + + it('falls back to /props when /running has no cmd, or the cmd states no recognizable context flag', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '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/v1/models') { + return new Response(JSON.stringify({ data: [{ id: 'a', status: { value: 'loaded' } }] }), { status: 200 }); + } + if (url.pathname === '/running') { + return new Response( + JSON.stringify({ running: [{ model: 'a', state: 'ready', cmd: 'llama-server -m /models/a.gguf' }] }), + { status: 200 } + ); + } + if (url.pathname === '/props') return new Response(JSON.stringify({ n_ctx: 8192 }), { status: 200 }); + throw new Error(`unexpected request: ${url.href}`); + }); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelContextLengths).toEqual({ a: 8192 }); + }); + + it('also recognizes a plain -c/--ctx-size flag, not just llama-swap’s own --fit-ctx', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '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/v1/models') { + return new Response(JSON.stringify({ data: [{ id: 'a', status: { value: 'loaded' } }] }), { status: 200 }); + } + if (url.pathname === '/running') { + return new Response( + JSON.stringify({ + running: [{ model: 'a', state: 'ready', cmd: 'llama-server -m /models/a.gguf --ctx-size 8192' }], + }), + { status: 200 } + ); + } + throw new Error(`unexpected request: ${url.href}`); // /props must never be reached + }); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelContextLengths).toEqual({ a: 8192 }); + }); +}); + +describe('refreshAllCustomModelHosts: model-size enrichment (parsed from /v1/models description)', () => { + it('parses a GB figure out of an auto-discovered model’s description', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockResolvedValue( + new Response( + JSON.stringify({ + data: [{ id: 'qwen3.8-27b', description: 'Auto-discovered 16.35 GB - parameters auto-fitted by llama.cpp' }], + }), + { status: 200 } + ) + ); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelSizesGB).toEqual({ 'qwen3.8-27b': 16.35 }); + }); + + it('gets no size at all for a hand-configured profile whose own description states none', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockResolvedValue( + new Response( + JSON.stringify({ + data: [{ id: 'big', description: 'General-purpose reasoning model, MoE CPU-offloaded. Default profile.' }], + }), + { status: 200 } + ) + ); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelSizesGB).toBeUndefined(); + }); + + it('populated regardless of loaded state — unlike context length, no /props probe is needed', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [host({ id: 'ep', baseUrl: 'http://localhost:8080' })]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '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/v1/models') { + return new Response( + JSON.stringify({ + data: [{ id: 'unloaded-model', description: 'Auto-discovered 4.91 GB - parameters auto-fitted' }], + }), + { status: 200 } + ); + } + throw new Error(`unexpected request: ${url.href}`); // /props must never be reached for this + }); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelSizesGB).toEqual({ 'unloaded-model': 4.91 }); + }); + + it('keeps a previously-learned size for a model still present, drops it once the model disappears entirely', async () => { + const dir = getDataDir(); + await writeCustomModelHosts(dir, [ + host({ id: 'ep', baseUrl: 'http://localhost:8080', models: ['a', 'b'], modelSizesGB: { a: 8, b: 16 } }), + ]); + fetchMock.mockResolvedValue( + new Response(JSON.stringify({ data: [{ id: 'a', description: 'no GB figure here' }] }), { status: 200 }) + ); + + await refreshAllCustomModelHosts(); + + const [updated] = await readCustomModelHosts(dir); + expect(updated.modelSizesGB).toEqual({ a: 8 }); // 'a' kept from before, 'b' dropped (gone from the list) + }); +}); diff --git a/test/custom-model-injection-apply.test.ts b/test/custom-model-injection-apply.test.ts new file mode 100644 index 000000000..dfc8e793b --- /dev/null +++ b/test/custom-model-injection-apply.test.ts @@ -0,0 +1,303 @@ +/** + * @fileoverview Tests for the two custom-model IO-layer fixes on top of the pure builder + * (docs/custom-model-endpoints-plan.md): + * + * 1. `contextLengthVar` — a discovered per-model context length reaches the actual + * session env (CLAUDE_CODE_MAX_CONTEXT_TOKENS), so a CLI stops assuming a large + * default window for an unrecognized custom model id and overflowing a much + * smaller real one. + * 2. `configDirVar` — an isolated, empty config directory is created and pointed at + * (CLAUDE_CONFIG_DIR), so an injected API key never shares a directory with a + * stored claude.ai OAuth session; `projects` is symlinked back into the real + * config dir so the response viewer/subagent windows/Read My Mind keep working. + * + * Port: N/A (no server; filesystem-only, under a temp CODEMAN data dir from test/setup.ts). + */ +import { existsSync, lstatSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; +import { getCli } from '../src/config/cli-registry/index.js'; +import { applyCustomModelInjection, customModelConfigDir } from '../src/custom-model-injection-apply.js'; +import type { CustomModelEndpoint } from '../src/custom-model-injection.js'; + +const endpoint: CustomModelEndpoint = { + id: 'ep1', + label: 'llama.cpp box', + baseUrl: 'http://192.168.1.50:8080', + apiKey: 'my-key', +}; + +function entryOrThrow(id: string) { + const entry = getCli(id); + if (!entry) throw new Error(`missing CLI registry entry: ${id}`); + return entry; +} + +const sessionsToClean: string[] = []; +afterEach(() => { + for (const id of sessionsToClean.splice(0)) rmSync(customModelConfigDir(id), { recursive: true, force: true }); +}); + +describe('applyCustomModelInjection: context length', () => { + it('claude: passes a known context length through to CLAUDE_CODE_MAX_CONTEXT_TOKENS', () => { + sessionsToClean.push('sess-ctx-1'); + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', 'sess-ctx-1', 16384); + expect(applied?.envOverrides.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBe('16384'); + expect(applied?.envKeys).toContain('CLAUDE_CODE_MAX_CONTEXT_TOKENS'); + }); + + it('claude: omits the var entirely when the context length is unknown', () => { + sessionsToClean.push('sess-ctx-2'); + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', 'sess-ctx-2'); + expect(applied?.envOverrides.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBeUndefined(); + }); + + it('deepseek: has no contextLengthVar declared, so a passed-in length is a no-op', () => { + const applied = applyCustomModelInjection(entryOrThrow('deepseek'), endpoint, 'qwen3', 'sess-ctx-3', 16384); + expect(Object.keys(applied?.envOverrides ?? {}).sort()).toEqual(['DEEPSEEK_API_KEY', 'DEEPSEEK_BASE_URL']); + }); +}); + +describe('applyCustomModelInjection: CLAUDE_CONFIG_DIR isolation', () => { + it('claude: creates an isolated config dir (no real credential/config files) and points CLAUDE_CONFIG_DIR at it', () => { + const sessionId = 'sess-cfgdir-1'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + const expectedDir = customModelConfigDir(sessionId); + expect(applied?.envOverrides.CLAUDE_CONFIG_DIR).toBe(expectedDir); + expect(applied?.configDir).toBe(expectedDir); + expect(existsSync(expectedDir)).toBe(true); + // The trust-seed file, the skipFirstRunPrompts settings.json, and the projects link — + // no real OAuth credential/config. + const entries = readdirSync(expectedDir).filter((name) => name !== 'projects'); + expect(entries.sort()).toEqual(['.claude.json', 'settings.json']); + }); + + it('claude: symlinks (or junctions) projects back to the real config dir so the response viewer keeps working', () => { + const sessionId = 'sess-cfgdir-2'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + const link = join(applied!.configDir!, 'projects'); + // Best-effort: only assert the link exists if it was actually created (the real + // ~/.claude/projects may not exist on a bare CI box, in which case linking is skipped). + if (existsSync(join(homedir(), '.claude', 'projects'))) { + expect(existsSync(link)).toBe(true); + expect(lstatSync(link).isSymbolicLink() || lstatSync(link).isDirectory()).toBe(true); + } + }); + + it('claude: re-applying to the same session is idempotent (boot-recovery re-apply)', () => { + const sessionId = 'sess-cfgdir-3'; + sessionsToClean.push(sessionId); + const first = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + const second = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + expect(second?.configDir).toBe(first?.configDir); + expect(existsSync(first!.configDir!)).toBe(true); + }); + + it('pi: configDir-kind CLIs are unaffected — no configDirVar concept for them', () => { + const sessionId = 'sess-cfgdir-pi'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection(entryOrThrow('pi'), endpoint, 'qwen3', sessionId); + expect(applied?.envOverrides.HOME).toBe(customModelConfigDir(sessionId)); + }); + + it('deepseek: no configDirVar declared, so no config dir is created at all', () => { + const sessionId = 'sess-cfgdir-deepseek'; + const applied = applyCustomModelInjection(entryOrThrow('deepseek'), endpoint, 'qwen3', sessionId); + expect(applied?.configDir).toBeUndefined(); + expect(existsSync(customModelConfigDir(sessionId))).toBe(false); + }); +}); + +describe('applyCustomModelInjection: apiKeyTrustFile (pre-approves the injected key)', () => { + it('claude: seeds .claude.json so the "Detected a custom API key" prompt never fires', () => { + const sessionId = 'sess-trust-1'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + const written = JSON.parse(readFileSync(join(applied!.configDir!, '.claude.json'), 'utf8')) as { + customApiKeyResponses: { approved: string[]; rejected: string[] }; + }; + expect(written.customApiKeyResponses.approved).toEqual(['my-key']); + expect(written.customApiKeyResponses.rejected).toEqual([]); + }); + + it('claude: falls back to the dummy key when the endpoint has none, and still seeds it', () => { + const sessionId = 'sess-trust-2'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection( + entryOrThrow('claude'), + { ...endpoint, apiKey: undefined }, + 'qwen3', + sessionId + ); + const written = JSON.parse(readFileSync(join(applied!.configDir!, '.claude.json'), 'utf8')) as { + customApiKeyResponses: { approved: string[] }; + }; + expect(written.customApiKeyResponses.approved).toEqual(['local-dummy-key']); + }); + + it('claude: merges onto fields the CLI itself already wrote into the same isolated dir, never overwrites them', () => { + const sessionId = 'sess-trust-3'; + sessionsToClean.push(sessionId); + const configDir = customModelConfigDir(sessionId); + mkdirSync(configDir, { recursive: true }); + writeFileSync(join(configDir, '.claude.json'), JSON.stringify({ userID: 'abc123', numStartups: 3 })); + + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + + const written = JSON.parse(readFileSync(join(applied!.configDir!, '.claude.json'), 'utf8')) as { + userID: string; + numStartups: number; + customApiKeyResponses: { approved: string[] }; + }; + expect(written.userID).toBe('abc123'); + expect(written.numStartups).toBe(3); + expect(written.customApiKeyResponses.approved).toEqual(['my-key']); + }); + + it('claude: a corrupt existing file is treated as absent rather than failing the apply', () => { + const sessionId = 'sess-trust-4'; + sessionsToClean.push(sessionId); + const configDir = customModelConfigDir(sessionId); + mkdirSync(configDir, { recursive: true }); + writeFileSync(join(configDir, '.claude.json'), '{ not valid json'); + + expect(() => applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId)).not.toThrow(); + const written = JSON.parse(readFileSync(join(configDir, '.claude.json'), 'utf8')) as { + customApiKeyResponses: { approved: string[] }; + }; + expect(written.customApiKeyResponses.approved).toEqual(['my-key']); + }); + + it('claude: re-approving the same key does not duplicate it in the approved list', () => { + const sessionId = 'sess-trust-5'; + sessionsToClean.push(sessionId); + applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + const second = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'llama3', sessionId); + const written = JSON.parse(readFileSync(join(second!.configDir!, '.claude.json'), 'utf8')) as { + customApiKeyResponses: { approved: string[] }; + }; + expect(written.customApiKeyResponses.approved).toEqual(['my-key']); + }); + + it('opencode: has no apiKeyTrustFile declared (no configDirVar at all), nothing is seeded', () => { + const sessionId = 'sess-trust-opencode'; + const applied = applyCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', sessionId); + expect(applied?.configDir).toBeUndefined(); + expect(existsSync(customModelConfigDir(sessionId))).toBe(false); + }); +}); + +describe("applyCustomModelInjection: skipFirstRunPrompts (an isolated dir replays claude's whole first-run sequence)", () => { + it("claude: seeds hasCompletedOnboarding and this session's own project trust into .claude.json", () => { + const sessionId = 'sess-firstrun-1'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection( + entryOrThrow('claude'), + endpoint, + 'qwen3', + sessionId, + undefined, + '/home/user/myproject' + ); + const written = JSON.parse(readFileSync(join(applied!.configDir!, '.claude.json'), 'utf8')) as { + hasCompletedOnboarding: boolean; + projects: Record; + }; + expect(written.hasCompletedOnboarding).toBe(true); + expect(written.projects['/home/user/myproject'].hasTrustDialogAccepted).toBe(true); + }); + + it('claude: seeds skipDangerousModePermissionPrompt into settings.json', () => { + const sessionId = 'sess-firstrun-2'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + const written = JSON.parse(readFileSync(join(applied!.configDir!, 'settings.json'), 'utf8')) as { + skipDangerousModePermissionPrompt: boolean; + }; + expect(written.skipDangerousModePermissionPrompt).toBe(true); + }); + + it('claude: with no workingDir given (boot recovery), hasCompletedOnboarding/settings still seed, but no project entry is added', () => { + const sessionId = 'sess-firstrun-3'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId); + const written = JSON.parse(readFileSync(join(applied!.configDir!, '.claude.json'), 'utf8')) as { + hasCompletedOnboarding: boolean; + projects?: Record; + }; + expect(written.hasCompletedOnboarding).toBeUndefined(); + expect(written.projects).toBeUndefined(); + }); + + it("claude: merges onto an existing project entry's other fields rather than overwriting them", () => { + const sessionId = 'sess-firstrun-4'; + sessionsToClean.push(sessionId); + const configDir = customModelConfigDir(sessionId); + mkdirSync(configDir, { recursive: true }); + writeFileSync( + join(configDir, '.claude.json'), + JSON.stringify({ projects: { '/home/user/myproject': { allowedTools: ['Bash'] } } }) + ); + + const applied = applyCustomModelInjection( + entryOrThrow('claude'), + endpoint, + 'qwen3', + sessionId, + undefined, + '/home/user/myproject' + ); + + const written = JSON.parse(readFileSync(join(applied!.configDir!, '.claude.json'), 'utf8')) as { + projects: Record; + }; + expect(written.projects['/home/user/myproject'].allowedTools).toEqual(['Bash']); + expect(written.projects['/home/user/myproject'].hasTrustDialogAccepted).toBe(true); + }); + + it('claude: a corrupt existing settings.json is treated as absent rather than failing the apply', () => { + const sessionId = 'sess-firstrun-5'; + sessionsToClean.push(sessionId); + const configDir = customModelConfigDir(sessionId); + mkdirSync(configDir, { recursive: true }); + writeFileSync(join(configDir, 'settings.json'), '{ not valid json'); + + expect(() => applyCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', sessionId)).not.toThrow(); + const written = JSON.parse(readFileSync(join(configDir, 'settings.json'), 'utf8')) as { + skipDangerousModePermissionPrompt: boolean; + }; + expect(written.skipDangerousModePermissionPrompt).toBe(true); + }); + + it('pi: has no skipFirstRunPrompts concept (no apiKeyTrustFile either) — nothing beyond its own config file', () => { + const sessionId = 'sess-firstrun-pi'; + sessionsToClean.push(sessionId); + const applied = applyCustomModelInjection( + entryOrThrow('pi'), + endpoint, + 'qwen3', + sessionId, + undefined, + '/home/user/myproject' + ); + const entries = readdirSync(applied!.configDir!); + expect(entries).not.toContain('settings.json'); + }); +}); + +describe('applyCustomModelInjection: pre-existing behavior unaffected', () => { + it('opencode: still returns a plain env-kind result with no configDir', () => { + const sessionId = 'sess-opencode-1'; + const applied = applyCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3', sessionId); + expect(applied?.configDir).toBeUndefined(); + expect(applied?.envOverrides.OPENCODE_CONFIG_CONTENT).toBeTruthy(); + }); + + it('antigravity: still undefined (unsupported)', () => { + const applied = applyCustomModelInjection(entryOrThrow('antigravity'), endpoint, 'qwen3', 'sess-agy-1'); + expect(applied).toBeUndefined(); + }); +}); diff --git a/test/custom-model-injection-contract.test.ts b/test/custom-model-injection-contract.test.ts index 852093183..1685504dc 100644 --- a/test/custom-model-injection-contract.test.ts +++ b/test/custom-model-injection-contract.test.ts @@ -142,17 +142,19 @@ describe('custom-model-injection contract (mock server)', () => { expect(mock.requests[0].headers.authorization).toBe('Bearer contract-test-key'); }); - // gemini/deepseek's `env` kind passes the base URL through UNCHANGED (unlike - // opencode/codex/pi/omp/grok, which build a structured config and explicitly append - // /v1) — matching Anthropic's own convention for claude's ANTHROPIC_BASE_URL, where the - // SDK appends the path itself. Whether each of these TWO CLIs' own OpenAI-compatible - // client expects the var to already include /v1 (the common OpenAI-SDK convention) or - // appends it itself is genuinely CLI-specific and UNVERIFIED (see the confidence table - // in docs/custom-model-endpoints-plan.md) — these tests model the common OpenAI-SDK convention (base_url - // ends in /v1) since that's the more likely behavior for an OpenAI-compatible client, - // but that assumption should be corrected here the moment it's checked against a real - // binary. (grok WAS in this group too, until live-testing showed the whole `env` recipe - // was wrong for it — see its own test below.) + // gemini's `env` kind still passes the base URL through UNCHANGED (matching + // Anthropic's own convention for claude's ANTHROPIC_BASE_URL, where the SDK appends + // the path itself) — whether gemini-cli's own OpenAI-compatible-ish client expects the + // var to already include /v1 or appends it itself remains genuinely UNVERIFIED (it + // fails for an unrelated auth reason before this would even matter — see the + // confidence table in docs/custom-model-endpoints-plan.md); this test models the + // common OpenAI-SDK convention as the best guess, to be corrected the moment it's + // checked against a real client. deepseek WAS in this "passes through unchanged" + // group too, until reading `@deepseek-ai/dsh-llm-deepseek`'s own bundled source + // confirmed it builds its request URL as `${DEEPSEEK_BASE_URL}/chat/completions` with + // no `/v1` of its own — `appendV1Suffix` now fixes that (see its own test below), + // the same way grok's whole `env` recipe turned out to be wrong before live-testing + // corrected it to a `configDir` one. it('gemini: GOOGLE_GEMINI_BASE_URL/GEMINI_API_KEY reach the mock', async () => { const injection = buildCustomModelInjection(entryOrThrow('gemini'), endpointFor(mock), 'qwen3'); @@ -186,16 +188,18 @@ describe('custom-model-injection contract (mock server)', () => { expect(mock.requests[0].headers.authorization).toBe('Bearer contract-test-key'); }); - it('deepseek: DEEPSEEK_BASE_URL/DEEPSEEK_API_KEY reach the mock (base URL/key only, no model var)', async () => { + it('deepseek: DEEPSEEK_BASE_URL already carries the /v1 suffix dsh itself never adds, reaching the mock at the real path dsh requests', async () => { + // Confirmed by reading dsh's own bundled source: it fetches + // `${DEEPSEEK_BASE_URL}/chat/completions` verbatim, no /v1 insertion of its own — so + // this call (unlike gemini's above) passes DEEPSEEK_BASE_URL to callOpenAiCompat + // UNMODIFIED, exactly mirroring what the real harness does, rather than the test + // helping it along. const injection = buildCustomModelInjection(entryOrThrow('deepseek'), endpointFor(mock), 'qwen3'); if (injection.kind !== 'env') throw new Error('unreachable'); expect(Object.keys(injection.envOverrides).sort()).toEqual(['DEEPSEEK_API_KEY', 'DEEPSEEK_BASE_URL']); + expect(injection.envOverrides.DEEPSEEK_BASE_URL).toBe(`${mock.baseUrl}/v1`); - await callOpenAiCompat( - `${injection.envOverrides.DEEPSEEK_BASE_URL}/v1`, - injection.envOverrides.DEEPSEEK_API_KEY, - 'qwen3' - ); + await callOpenAiCompat(injection.envOverrides.DEEPSEEK_BASE_URL, injection.envOverrides.DEEPSEEK_API_KEY, 'qwen3'); expect(mock.requests[0].path).toBe('/v1/chat/completions'); expect(mock.requests[0].headers.authorization).toBe('Bearer contract-test-key'); diff --git a/test/custom-model-injection.test.ts b/test/custom-model-injection.test.ts index 2acd08ca8..c99fff490 100644 --- a/test/custom-model-injection.test.ts +++ b/test/custom-model-injection.test.ts @@ -58,6 +58,43 @@ describe('buildCustomModelInjection', () => { }); }); + it('claude: also declares configDirVar (CLAUDE_CONFIG_DIR isolation) on the env-kind result', () => { + const result = buildCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3'); + if (result.kind !== 'env') throw new Error('unreachable'); + expect(result.configDirVar).toBe('CLAUDE_CONFIG_DIR'); + }); + + it('claude: injects CLAUDE_CODE_MAX_CONTEXT_TOKENS when a context length is known', () => { + const result = buildCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3', 16384); + if (result.kind !== 'env') throw new Error('unreachable'); + expect(result.envOverrides.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBe('16384'); + }); + + it('claude: omits CLAUDE_CODE_MAX_CONTEXT_TOKENS when the context length is unknown', () => { + const result = buildCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3'); + if (result.kind !== 'env') throw new Error('unreachable'); + expect(result.envOverrides.CLAUDE_CODE_MAX_CONTEXT_TOKENS).toBeUndefined(); + }); + + it('claude: also declares apiKeyTrustFile, carrying the literal apiKey used', () => { + const result = buildCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3'); + if (result.kind !== 'env') throw new Error('unreachable'); + expect(result.apiKeyTrustFile).toEqual({ relPath: '.claude.json', shape: 'claude-api-key-responses' }); + expect(result.apiKey).toBe('my-key'); + }); + + it('claude: also declares skipFirstRunPrompts on the env-kind result', () => { + const result = buildCustomModelInjection(entryOrThrow('claude'), endpoint, 'qwen3'); + if (result.kind !== 'env') throw new Error('unreachable'); + expect(result.skipFirstRunPrompts).toBe(true); + }); + + it('opencode: has no skipFirstRunPrompts (no apiKeyTrustFile/configDirVar concept for it either)', () => { + const result = buildCustomModelInjection(entryOrThrow('opencode'), endpoint, 'qwen3'); + if (result.kind !== 'env') throw new Error('unreachable'); + expect(result.skipFirstRunPrompts).toBeUndefined(); + }); + it('claude: falls back to a dummy key when the endpoint has none', () => { const result = buildCustomModelInjection(entryOrThrow('claude'), { ...endpoint, apiKey: undefined }, 'qwen3'); if (result.kind !== 'env') throw new Error('unreachable'); @@ -142,15 +179,31 @@ describe('buildCustomModelInjection', () => { expect(result.extraEnv).toEqual({ XAI_API_KEY: 'my-key' }); }); - it('deepseek: env kind sets base URL/key only, no model var', () => { + it('deepseek: env kind sets base URL (with a /v1 suffix appended) and key, no model var', () => { + // appendV1Suffix is REQUIRED here, not cosmetic: confirmed by reading dsh's own + // bundled source (@deepseek-ai/dsh-llm-deepseek) that it builds the request URL as + // `${DEEPSEEK_BASE_URL}/chat/completions` with no "/v1" of its own, while + // llama-swap/llama.cpp only serves "/v1/chat/completions" — without this, every + // request 404s (confirmed live; this is the fix for the originally-reported + // "dsh: HTTP_404: DeepSeek API error (HTTP 404)"). const result = buildCustomModelInjection(entryOrThrow('deepseek'), endpoint, 'qwen3'); if (result.kind !== 'env') throw new Error('unreachable'); expect(result.envOverrides).toEqual({ - DEEPSEEK_BASE_URL: 'http://192.168.1.50:8080', + DEEPSEEK_BASE_URL: 'http://192.168.1.50:8080/v1', DEEPSEEK_API_KEY: 'my-key', }); }); + it('deepseek: appending the /v1 suffix is idempotent against a baseUrl that already ends in /v1', () => { + const result = buildCustomModelInjection( + entryOrThrow('deepseek'), + { ...endpoint, baseUrl: 'http://192.168.1.50:8080/v1' }, + 'qwen3' + ); + if (result.kind !== 'env') throw new Error('unreachable'); + expect(result.envOverrides.DEEPSEEK_BASE_URL).toBe('http://192.168.1.50:8080/v1'); + }); + it('antigravity: unsupported', () => { const result = buildCustomModelInjection(entryOrThrow('antigravity'), endpoint, 'qwen3'); expect(result).toEqual({ kind: 'unsupported' }); diff --git a/test/custom-model-log-tail.test.ts b/test/custom-model-log-tail.test.ts new file mode 100644 index 000000000..74dc352a8 --- /dev/null +++ b/test/custom-model-log-tail.test.ts @@ -0,0 +1,238 @@ +/** + * @fileoverview Tests for `getLatestLlamaSwapLogLine()`/`pruneIdleLlamaSwapLogTails()` — + * the real-time "what is llama.cpp actually doing" feed behind the loading banner's + * second line (docs/custom-model-endpoints-plan.md). Confirmed live against a real + * llama-swap deployment: its `GET /api/events` SSE stream carries the backend + * llama-server process's own stdout (`load_model: ...`, `llama_server: model loaded`) + * as `{"type":"logData","data":"{\"data\":\"...\",\"source\":\"upstream\"}"}` frames, + * tagged distinctly from llama-swap's own `source: "proxy"` request-access log frames. + * + * ⚠️ `GET /logs` (the endpoint this feature's own first cut was built against, before + * being caught by exactly this kind of live check) turns out to carry ONLY the proxy + * log — confirmed live it never showed a single backend line even seconds after a real, + * confirmed model swap. `/api/events` is the only source that actually has the data. + * + * Drives a hand-built `ReadableStream` body through the mocked `webviewFetch` rather + * than a real network round-trip — the point under test is the SSE-frame parsing and + * `source` filtering plus the one-connection-per-endpoint reuse, not networking itself. + * + * Each test uses its own host id (`llamaSwapLogTails` is a module-level Map, shared + * across every test in this file) and `afterEach` force-prunes everything so no tail + * a test forgot to close leaks into the next one. + * + * Port: N/A (no server; drives the exported functions directly). + */ +import { describe, it, expect, vi, afterEach } from 'vitest'; +import { getLatestLlamaSwapLogLine, pruneIdleLlamaSwapLogTails } from '../src/web/routes/custom-model-routes.js'; +import { webviewFetch } from '../src/web/webview-egress.js'; +import type { CustomModelHost } from '../src/custom-model-hosts.js'; + +vi.mock('../src/web/webview-egress.js', async () => { + const actual = await vi.importActual('../src/web/webview-egress.js'); + return { ...actual, webviewFetch: vi.fn() }; +}); + +const fetchMock = vi.mocked(webviewFetch); + +/** One real `GET /api/events` SSE frame carrying backend (`source: "upstream"`) log text. */ +function upstreamLogFrame(text: string): string { + const inner = JSON.stringify({ data: text, source: 'upstream' }); + return `event:message\ndata:${JSON.stringify({ type: 'logData', data: inner })}\n\n`; +} + +/** The proxy-log flavor of the same event shape — must never be surfaced as `latestLine`. */ +function proxyLogFrame(text: string): string { + const inner = JSON.stringify({ data: text, source: 'proxy' }); + return `event:message\ndata:${JSON.stringify({ type: 'logData', data: inner })}\n\n`; +} + +/** A streaming Response whose body enqueues `frames` up front and then stays open + * (never closes) — matches a real `/api/events` connection, confirmed live to stay + * open indefinitely (read past 220KB over 8s with no `done`). */ +function openStreamResponse(frames: string[]): Response { + const encoder = new TextEncoder(); + const stream = new ReadableStream({ + start(controller) { + for (const frame of frames) controller.enqueue(encoder.encode(frame)); + // deliberately never controller.close() + }, + }); + return new Response(stream, { status: 200 }); +} + +function host(id: string): CustomModelHost { + return { id, label: id, baseUrl: `http://192.168.1.50:8080/${id}` }; +} + +/** Lets the fire-and-forget stream-pump's microtasks (reader.read() resolutions) settle. */ +async function flush(): Promise { + await new Promise((resolve) => setTimeout(resolve, 10)); +} + +afterEach(() => { + pruneIdleLlamaSwapLogTails(Number.POSITIVE_INFINITY); // force-close every tail this file opened + fetchMock.mockReset(); +}); + +describe('getLatestLlamaSwapLogLine', () => { + it('returns undefined before any line has arrived, then the real backend log line once it does', async () => { + const h = host('t1'); + fetchMock.mockResolvedValue( + openStreamResponse([upstreamLogFrame('0.31.428.568 I srv llama_server: model loaded')]) + ); + + const before = getLatestLlamaSwapLogLine(h); + expect(before).toBeUndefined(); + await flush(); + const after = getLatestLlamaSwapLogLine(h); + + expect(after).toBe('0.31.428.568 I srv llama_server: model loaded'); + }); + + it('filters out llama-swap\'s own proxy-sourced frames, keeping only source: "upstream"', async () => { + const h = host('t2'); + fetchMock.mockResolvedValue( + openStreamResponse([ + proxyLogFrame('[INFO] Request 10.10.10.1 "GET /running HTTP/1.1" 200 407 "undici" 46.207µs'), + upstreamLogFrame('0.14.157.100 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 16384'), + proxyLogFrame('[WARN] some warning about something unrelated'), + ]) + ); + + getLatestLlamaSwapLogLine(h); + await flush(); + + expect(getLatestLlamaSwapLogLine(h)).toBe( + '0.14.157.100 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 16384' + ); + }); + + it('keeps the LAST line when one upstream frame batches several newline-joined lines', async () => { + const h = host('t3'); + fetchMock.mockResolvedValue( + openStreamResponse([ + upstreamLogFrame( + '0.00.001.000 I srv llama_server: starting\n0.00.002.000 I srv llama_server: loading tensors' + ), + upstreamLogFrame('0.00.003.000 I srv llama_server: model loaded'), + ]) + ); + + getLatestLlamaSwapLogLine(h); + await flush(); + + expect(getLatestLlamaSwapLogLine(h)).toBe('0.00.003.000 I srv llama_server: model loaded'); + }); + + it('handles a frame split across two stream chunks (SSE double-newline boundary not yet seen)', async () => { + const h = host('t3b'); + const whole = upstreamLogFrame('0.00.005.000 I srv llama_server: model loaded'); + const splitAt = Math.floor(whole.length / 2); + fetchMock.mockResolvedValue(openStreamResponse([whole.slice(0, splitAt), whole.slice(splitAt)])); + + getLatestLlamaSwapLogLine(h); + await flush(); + + expect(getLatestLlamaSwapLogLine(h)).toBe('0.00.005.000 I srv llama_server: model loaded'); + }); + + it('ignores a malformed frame instead of throwing', async () => { + const h = host('t3c'); + fetchMock.mockResolvedValue( + openStreamResponse(['event:message\ndata:not valid json\n\n', upstreamLogFrame('llama_server: model loaded')]) + ); + + getLatestLlamaSwapLogLine(h); + await flush(); + + expect(getLatestLlamaSwapLogLine(h)).toBe('llama_server: model loaded'); + }); + + it('ignores a non-logData event type', async () => { + const h = host('t3d'); + fetchMock.mockResolvedValue( + openStreamResponse([ + `event:message\ndata:${JSON.stringify({ type: 'modelStatus', data: '{}' })}\n\n`, + upstreamLogFrame('llama_server: model loaded'), + ]) + ); + + getLatestLlamaSwapLogLine(h); + await flush(); + + expect(getLatestLlamaSwapLogLine(h)).toBe('llama_server: model loaded'); + }); + + it('opens exactly one connection per endpoint — a second call while the tail is open never re-fetches', async () => { + const h = host('t4'); + fetchMock.mockResolvedValue(openStreamResponse([upstreamLogFrame('llama_server: model loaded')])); + + getLatestLlamaSwapLogLine(h); + await flush(); + getLatestLlamaSwapLogLine(h); + getLatestLlamaSwapLogLine(h); + + expect(fetchMock).toHaveBeenCalledTimes(1); + }); + + it("requests /api/events specifically, with the endpoint's own auth headers", async () => { + const h: CustomModelHost = { id: 't5', label: 't5', baseUrl: 'http://192.168.1.60:9000', apiKey: 'secret-key' }; + fetchMock.mockResolvedValue(openStreamResponse([])); + + getLatestLlamaSwapLogLine(h); + + expect(fetchMock).toHaveBeenCalledTimes(1); + const [url, init] = fetchMock.mock.calls[0]!; + expect((url as URL).pathname).toBe('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/api/events'); + expect((init as RequestInit).headers).toMatchObject({ Authorization: 'Bearer secret-key' }); + }); + + it('an unreachable endpoint (fetch throws) leaves latestLine undefined rather than throwing', async () => { + const h = host('t6'); + fetchMock.mockRejectedValue(new TypeError('fetch failed')); + + expect(() => getLatestLlamaSwapLogLine(h)).not.toThrow(); + await flush(); + expect(getLatestLlamaSwapLogLine(h)).toBeUndefined(); + }); + + it('a non-2xx response leaves latestLine undefined rather than throwing', async () => { + const h = host('t7'); + fetchMock.mockResolvedValue(new Response('not found', { status: 404 })); + + getLatestLlamaSwapLogLine(h); + await flush(); + + expect(getLatestLlamaSwapLogLine(h)).toBeUndefined(); + }); +}); + +describe('pruneIdleLlamaSwapLogTails', () => { + it('closes a tail nothing has polled recently, so the next access starts a fresh connection', async () => { + const h = host('t8'); + fetchMock.mockResolvedValue(openStreamResponse([upstreamLogFrame('llama_server: model loaded')])); + + getLatestLlamaSwapLogLine(h); // opens the first connection, lastAccessedAt = now + await flush(); + expect(fetchMock).toHaveBeenCalledTimes(1); + + pruneIdleLlamaSwapLogTails(Date.now() + 60_000); // "now" far enough ahead that the tail reads as idle + + getLatestLlamaSwapLogLine(h); // the entry was removed — this must open a NEW connection + await flush(); + expect(fetchMock).toHaveBeenCalledTimes(2); + }); + + it('leaves a recently-accessed tail alone', async () => { + const h = host('t9'); + fetchMock.mockResolvedValue(openStreamResponse([upstreamLogFrame('llama_server: model loaded')])); + + getLatestLlamaSwapLogLine(h); + await flush(); + + pruneIdleLlamaSwapLogTails(Date.now()); // no time has passed — nothing is idle yet + + getLatestLlamaSwapLogLine(h); + expect(fetchMock).toHaveBeenCalledTimes(1); // still just the one connection + }); +}); diff --git a/test/custom-model-one-shot-launch.test.ts b/test/custom-model-one-shot-launch.test.ts new file mode 100644 index 000000000..c4ba8598f --- /dev/null +++ b/test/custom-model-one-shot-launch.test.ts @@ -0,0 +1,238 @@ +/** + * @fileoverview Frontend tests for the one-shot custom-model launch path added to + * session-ui.js (docs/custom-model-endpoints-plan.md): `runCustomModelEntry` dispatches + * to `_runCustomModelEntryOneShot` for every custom-model-eligible CLI except claude, + * which launches directly on the endpoint (no restart) by folding `customModel` into + * the run() function's own `/api/quick-start` body via `_pendingCustomModelForLaunch` + * and `_quickStartWithCustomModelConfirm`. Fixes the visible native-boot-then-restart the + * restart-after-launch path (`_runCustomModelEntryViaRestart`, still used for claude) + * showed on every custom-model run — confirmed live on Codex, whose TUI fully + * reinitializes on a restart. + * + * Uses the same JSDOM + `runScripts: "dangerously"` approach as + * test/custom-model-run-menu-ui.test.ts, extended with the DOM elements runCodex() (the + * CLI this was reported against) reads. + * + * Port: none. + */ +import { readFileSync } from 'node:fs'; +import { JSDOM } from 'jsdom'; +import { describe, expect, it } from 'vitest'; + +const CONSTANTS_JS = readFileSync(new URL('../src/web/public/constants.js', import.meta.url), 'utf-8'); +const SESSION_UI_JS = readFileSync(new URL('../src/web/public/session-ui.js', import.meta.url), 'utf-8'); + +function bootApp() { + const dom = new JSDOM( + ` + + + +
+ `, + { url: 'http://localhost/', runScripts: 'dangerously' } + ); + const win = dom.window as unknown as Window & typeof globalThis & { CodemanApp: new () => any }; + (win as unknown as { eval: (s: string) => void }).eval('window.CodemanApp = function CodemanApp() {};'); + (win as unknown as { eval: (s: string) => void }).eval(CONSTANTS_JS); + (win as unknown as { eval: (s: string) => void }).eval(SESSION_UI_JS); + const app = new win.CodemanApp(); + app.cases = [{ name: 'testcase' }]; + app.terminal = { focus: () => {} }; + app.loadAppSettingsFromStorage = () => ({}); + app.getCaseSettings = () => ({}); + app.buildEnvOverrides = () => ({}); + app.showToast = () => {}; + app._beginSessionLaunchStatus = () => 'status-token'; + app._reportSessionLaunchError = (_token: unknown, message: string) => { + app._lastReportedError = message; + }; + app._ensureCreatedSessionVisible = async () => {}; + app.selectSession = async () => {}; + app._nextCaseSessionStartNumber = () => 1; + return { win, app }; +} + +describe('runCustomModelEntry dispatch', () => { + it('routes claude through the restart-after-launch path', async () => { + const { app } = bootApp(); + let calledRestart = false; + let calledOneShot = false; + app._runCustomModelEntryViaRestart = async () => { + calledRestart = true; + }; + app._runCustomModelEntryOneShot = async () => { + calledOneShot = true; + }; + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + expect(calledRestart).toBe(true); + expect(calledOneShot).toBe(false); + }); + + it('routes every other custom-model-eligible CLI through the one-shot path', async () => { + for (const mode of ['opencode', 'codex', 'gemini', 'pi', 'grok', 'deepseek', 'omp']) { + const { app } = bootApp(); + let calledRestart = false; + let calledOneShot = false; + app._runCustomModelEntryViaRestart = async () => { + calledRestart = true; + }; + app._runCustomModelEntryOneShot = async () => { + calledOneShot = true; + }; + await app.runCustomModelEntry(mode, 'llama-box', 'qwen3'); + expect(calledRestart, mode).toBe(false); + expect(calledOneShot, mode).toBe(true); + } + }); +}); + +describe('_runCustomModelEntryOneShot', () => { + it('stashes the pick on _pendingCustomModelForLaunch for the duration of run(), then clears it', async () => { + const { app } = bootApp(); + let seenDuringRun: unknown; + app.run = async function (this: typeof app) { + seenDuringRun = this._pendingCustomModelForLaunch; + }; + await app._runCustomModelEntryOneShot('codex', 'llama-box', 'qwen3'); + expect(seenDuringRun).toEqual({ endpointId: 'llama-box', modelId: 'qwen3' }); + expect(app._pendingCustomModelForLaunch).toBeUndefined(); + }); + + it('clears the pending pick even when run() throws', async () => { + const { app } = bootApp(); + app.run = async () => { + throw new Error('boom'); + }; + await expect(app._runCustomModelEntryOneShot('codex', 'llama-box', 'qwen3')).rejects.toThrow('boom'); + expect(app._pendingCustomModelForLaunch).toBeUndefined(); + }); + + it('starts the loading watcher when the launch reports modelSwapInProgress, passing the new session id', async () => { + const { app } = bootApp(); + app.run = async () => { + app._lastCustomModelLaunchResult = { modelSwapInProgress: true, sessionId: 'new-session' }; + }; + let watched: unknown[] | null = null; + app._watchLlamaSwapLoading = async (...args: unknown[]) => { + watched = args; + }; + await app._runCustomModelEntryOneShot('codex', 'llama-box', 'qwen3'); + expect(watched).toEqual(['llama-box', 'qwen3', 'new-session']); + }); + + it('never starts the watcher when no swap was needed', async () => { + const { app } = bootApp(); + app.run = async () => { + app._lastCustomModelLaunchResult = { modelSwapInProgress: false }; + }; + let watchCalled = false; + app._watchLlamaSwapLoading = async () => { + watchCalled = true; + }; + await app._runCustomModelEntryOneShot('codex', 'llama-box', 'qwen3'); + expect(watchCalled).toBe(false); + }); +}); + +describe('_quickStartWithCustomModelConfirm', () => { + function withFetch(win: Window & typeof globalThis, handler: (body: any) => any) { + (win as unknown as { fetch: typeof fetch }).fetch = (async (_url: string, opts: any) => ({ + json: async () => handler(JSON.parse(opts.body)), + })) as unknown as typeof fetch; + } + + it('returns the response directly when no confirmation is needed', async () => { + const { win, app } = bootApp(); + withFetch(win, (body) => ({ success: true, data: { sessionId: 's1', modelSwapInProgress: false, body } })); + const data = await app._quickStartWithCustomModelConfirm({ + mode: 'codex', + customModel: { endpointId: 'e', modelId: 'm' }, + }); + expect(data.success).toBe(true); + expect(data.data.sessionId).toBe('s1'); + expect(app._lastCustomModelLaunchResult).toEqual(data.data); + }); + + it('confirming re-sends with confirmed:true and returns the second response', async () => { + const { win, app } = bootApp(); + app._confirmModelSwap = async () => true; + let calls = 0; + withFetch(win, (body) => { + calls += 1; + if (calls === 1) { + return { + success: true, + data: { + requiresConfirmation: true, + currentlyLoadedModel: 'llama3', + affectedSessions: [{ id: 's2', name: 'w2' }], + }, + }; + } + expect(body.customModel.confirmed).toBe(true); + return { success: true, data: { sessionId: 's1', modelSwapInProgress: true } }; + }); + const data = await app._quickStartWithCustomModelConfirm({ + mode: 'codex', + customModel: { endpointId: 'e', modelId: 'm' }, + }); + expect(calls).toBe(2); + expect(data.data.sessionId).toBe('s1'); + expect(app._lastCustomModelLaunchResult.modelSwapInProgress).toBe(true); + }); + + it('cancelling never re-sends, and reports a cancellation error', async () => { + const { win, app } = bootApp(); + app._confirmModelSwap = async () => false; + let calls = 0; + withFetch(win, () => { + calls += 1; + return { + success: true, + data: { + requiresConfirmation: true, + currentlyLoadedModel: 'llama3', + affectedSessions: [{ id: 's2', name: 'w2' }], + }, + }; + }); + const data = await app._quickStartWithCustomModelConfirm({ + mode: 'codex', + customModel: { endpointId: 'e', modelId: 'm' }, + }); + expect(calls).toBe(1); + expect(data.success).toBe(false); + expect(data.error).toMatch(/cancelled/i); + expect(app._lastCustomModelLaunchResult).toBeUndefined(); + }); +}); + +describe('runCodex(): one-shot custom-model launch (the CLI this was reported against)', () => { + it('folds _pendingCustomModelForLaunch into the quick-start body as customModel', async () => { + const { win, app } = bootApp(); + (win as unknown as { fetch: typeof fetch }).fetch = (async (url: string, opts?: any) => { + if (url === '/api/codex/status') return { json: async () => ({ data: { available: true } }) }; + const body = JSON.parse(opts.body); + expect(body.customModel).toEqual({ endpointId: 'llama-box', modelId: 'qwen3' }); + return { json: async () => ({ success: true, data: { sessionId: 's1', modelSwapInProgress: false } }) }; + }) as unknown as typeof fetch; + + app._pendingCustomModelForLaunch = { endpointId: 'llama-box', modelId: 'qwen3' }; + await app.runCodex(); + expect(app._lastReportedError).toBeUndefined(); + }); + + it('omits customModel entirely for a plain (non-custom-model) Codex launch', async () => { + const { win, app } = bootApp(); + (win as unknown as { fetch: typeof fetch }).fetch = (async (url: string, opts?: any) => { + if (url === '/api/codex/status') return { json: async () => ({ data: { available: true } }) }; + const body = JSON.parse(opts.body); + expect(body.customModel).toBeUndefined(); + return { json: async () => ({ success: true, data: { sessionId: 's1' } }) }; + }) as unknown as typeof fetch; + + await app.runCodex(); + expect(app._lastReportedError).toBeUndefined(); + }); +}); diff --git a/test/custom-model-run-menu-ui.test.ts b/test/custom-model-run-menu-ui.test.ts new file mode 100644 index 000000000..a33d9fb6d --- /dev/null +++ b/test/custom-model-run-menu-ui.test.ts @@ -0,0 +1,1094 @@ +/** + * @fileoverview Frontend tests for the Custom Model Endpoint Profiles Run-menu + * picker (docs/custom-model-endpoints-plan.md): the generated entries in + * session-ui.js's `_refreshCustomModelRunOptions()` / `runCustomModelEntry()`. + * + * These are DOM-level facts that need no Playwright and no tmux — `runScripts: + * "dangerously"` is used deliberately (this JSDOM only ever parses markup this + * module itself generated, never live user input) so that a broken inline + * `onclick` attribute shows up as a genuinely uncallable handler, the same way + * it would in a real browser, rather than merely as a string this test parses + * by eye. `test/admin-ui.test.ts` and `test/home-sessions.test.ts` are the + * precedent for driving a real frontend module against a JSDOM window rather + * than a live server. + * + * Port: none. + */ +import { readFileSync } from 'node:fs'; +import { JSDOM } from 'jsdom'; +import { describe, expect, it } from 'vitest'; + +const CONSTANTS_JS = readFileSync(new URL('../src/web/public/constants.js', import.meta.url), 'utf-8'); +const SESSION_UI_JS = readFileSync(new URL('../src/web/public/session-ui.js', import.meta.url), 'utf-8'); +// Only for the real _showCenterStatus DOM tests below (`bootAppWithRealCenterStatus`) — +// every other test in this file stubs _showCenterStatus itself and has no need of it. +const PANELS_UI_JS = readFileSync(new URL('../src/web/public/panels-ui.js', import.meta.url), 'utf-8'); + +function resp(body: unknown, ok = true) { + return { ok, json: async () => body }; +} + +/** + * Boots a minimal CodemanApp instance with constants.js + session-ui.js + * evaluated against a real JSDOM window, so escapeHtml and the picker's own + * innerHTML-building code run exactly as they do in the browser. + */ +function bootApp( + options: { + customModelClis?: Array<{ id: string; label: string }>; + hosts?: unknown; + cliAvailable?: (id: string) => boolean; + activeCase?: { location?: string } | null; + settingsEnabled?: boolean; + } = {} +) { + const dom = new JSDOM( + ` + + + +
+ + +
+
+ + + + `, + { url: 'http://localhost/', runScripts: 'dangerously' } + ); + const win = dom.window as unknown as Window & + typeof globalThis & { + CodemanApp: new () => any; + __codemanCustomModelClis?: Array<{ id: string; label: string }>; + }; + (win as unknown as { eval: (s: string) => void }).eval('window.CodemanApp = function CodemanApp() {};'); + (win as unknown as { eval: (s: string) => void }).eval(CONSTANTS_JS); + (win as unknown as { eval: (s: string) => void }).eval(SESSION_UI_JS); + + win.__codemanCustomModelClis = options.customModelClis ?? [{ id: 'claude', label: 'Claude Code' }]; + + const app = new win.CodemanApp(); + app.cases = options.activeCase ? [{ name: 'testcase', ...options.activeCase }] : [{ name: 'testcase' }]; + app.loadAppSettingsFromStorage = () => ({ customModelEndpointsEnabled: options.settingsEnabled ?? true }); + app.isCliAvailable = options.cliAvailable ?? (() => true); + app.showToast = () => {}; + // Real implementation lives in panels-ui.js, not evaluated into this harness (only + // constants.js + session-ui.js are — see below) — a no-op default handle matching its + // real shape, same reasoning as showToast above; tests of the center status itself + // override it. + app._showCenterStatus = () => ({ dismiss: () => {}, setMessage: () => {} }); + // Default no-op so a button's onclick (selectCustomModelEntry -> possibly + // straight to runCustomModelEntry for a single-model host) never rejects + // with "this.run is not a function"; tests of the launch itself override it. + app.run = async () => {}; + // _apiJson unwraps the {success,data} envelope for real against a live + // server; here it stands in for that, driven from a fixed `hosts` fixture + // so these tests exercise the picker's OWN code, not the envelope helper. + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') return options.hosts ?? []; + return null; + }; + return { dom, win, app }; +} + +/** + * Like `bootApp`, but also evaluates panels-ui.js so `_showCenterStatus` is the REAL + * implementation rather than the plain stub `bootApp` installs — for the Cancel-button + * rendering tests, which need to see actual DOM the app would produce. + */ +function bootAppWithRealCenterStatus() { + const dom = new JSDOM('', { url: 'http://localhost/', runScripts: 'dangerously' }); + const win = dom.window as unknown as Window & typeof globalThis & { CodemanApp: new () => any }; + // jsdom doesn't polyfill requestAnimationFrame, and _showCenterStatus calls it to add + // the 'show' class — run it synchronously, which is all a non-visual test needs. + (win as unknown as { requestAnimationFrame: (cb: () => void) => number }).requestAnimationFrame = (cb) => { + cb(); + return 0; + }; + (win as unknown as { eval: (s: string) => void }).eval('window.CodemanApp = function CodemanApp() {};'); + (win as unknown as { eval: (s: string) => void }).eval(CONSTANTS_JS); + (win as unknown as { eval: (s: string) => void }).eval(SESSION_UI_JS); + (win as unknown as { eval: (s: string) => void }).eval(PANELS_UI_JS); + const app = new win.CodemanApp(); + return { win, app }; +} + +describe('Custom Model Endpoint Profiles: Run-menu picker generation', () => { + it('generates a real, clickable button per (capable CLI, endpoint) pair', async () => { + const { win, app } = bootApp({ + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: ['qwen3'] }], + }); + const menu = win.document.getElementById('runModeMenu')!; + await app._refreshCustomModelRunOptions(menu); + + const container = win.document.getElementById('runModeCustomModels')!; + const buttons = container.querySelectorAll('button'); + expect(buttons.length).toBe(1); + + const btn = buttons[0] as unknown as HTMLButtonElement & { onclick: unknown }; + // The real bug: JSON.stringify's own double quotes terminate the + // double-quoted onclick attribute at the first one, so btn.onclick comes + // back null and the parsed attribute is garbage. With escapeHtml wrapping + // each stringified argument, jsdom (which compiles inline handlers under + // runScripts:"dangerously" exactly like a real browser) parses it as a + // real, callable function. + expect(typeof btn.onclick).toBe('function'); + + win.app = app; + expect(() => btn.onclick!(new (win as any).Event('click'))).not.toThrow(); + }); + + it('escapes a model id containing HTML-significant characters instead of letting it break out of the tag', async () => { + // modelId comes from the endpoint's OWN /v1/models reply, which this box + // does not control — a live-HTML-injection vector if it ever reaches the + // markup unescaped, distinct from (and on top of) the quoting bug above. + const dangerousModel = '">'; + const { win, app } = bootApp({ + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: [dangerousModel] }], + }); + const menu = win.document.getElementById('runModeMenu')!; + await app._refreshCustomModelRunOptions(menu); + + const container = win.document.getElementById('runModeCustomModels')!; + // The injected markup must never have produced a live element: if it + // did, the attacker-controlled tag closed the button early and escaped + // into sibling markup instead of staying inert string data. + expect(container.querySelector('img')).toBeNull(); + expect(container.querySelectorAll('button').length).toBe(1); + }); + + it('is hidden when the feature setting is off, even with capable CLIs and endpoints present', async () => { + const { win, app } = bootApp({ + settingsEnabled: false, + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: ['qwen3'] }], + }); + const menu = win.document.getElementById('runModeMenu')!; + await app._refreshCustomModelRunOptions(menu); + expect(win.document.getElementById('runModeCustomModels')!.innerHTML).toBe(''); + expect((win.document.getElementById('runModeCustomModelSep') as HTMLElement).style.display).toBe('none'); + }); + + it('is hidden for a remote or Docker active case, since the apply route refuses both', async () => { + for (const location of ['remote', 'docker']) { + const { win, app } = bootApp({ + activeCase: { location }, + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: ['qwen3'] }], + }); + const menu = win.document.getElementById('runModeMenu')!; + await app._refreshCustomModelRunOptions(menu); + expect(win.document.getElementById('runModeCustomModels')!.innerHTML, location).toBe(''); + } + }); + + it('skips an endpoint with no discovered model and no default, rather than generating a dead entry', async () => { + const { win, app } = bootApp({ + hosts: [{ id: 'undiscovered', label: 'Not discovered yet', baseUrl: 'http://localhost:8080', models: [] }], + }); + const menu = win.document.getElementById('runModeMenu')!; + await app._refreshCustomModelRunOptions(menu); + expect(win.document.getElementById('runModeCustomModels')!.innerHTML).toBe(''); + }); + + it('omits a CLI the host does not have installed, matching the stock entries’ own gating', async () => { + const { win, app } = bootApp({ + customModelClis: [ + { id: 'claude', label: 'Claude Code' }, + { id: 'codex', label: 'Codex' }, + ], + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: ['qwen3'] }], + cliAvailable: (id: string) => id === 'claude', + }); + const menu = win.document.getElementById('runModeMenu')!; + await app._refreshCustomModelRunOptions(menu); + const container = win.document.getElementById('runModeCustomModels')!; + expect(container.querySelectorAll('button').length).toBe(1); + expect(container.textContent).toContain('Claude Code'); + expect(container.textContent).not.toContain('Codex'); + }); +}); + +describe('Custom Model Endpoint Profiles: the "which model" picker', () => { + it('launches straight away for a host with exactly one discovered model, no dialog', async () => { + const { win, app } = bootApp({ + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: ['qwen3'] }], + }); + let launched: unknown[] | null = null; + app.runCustomModelEntry = async (...args: unknown[]) => { + launched = args; + }; + + await app.selectCustomModelEntry('claude', 'llama-box'); + + expect(launched).toEqual(['claude', 'llama-box', 'qwen3']); + expect(win.document.getElementById('customModelPickModal')!.classList.contains('active')).toBe(false); + }); + + it('opens the picker for a host with more than one discovered model, rather than launching directly', async () => { + const { win, app } = bootApp({ + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: ['qwen3', 'llama3'] }], + }); + let launched = false; + app.runCustomModelEntry = async () => { + launched = true; + }; + + await app.selectCustomModelEntry('claude', 'llama-box'); + + expect(launched).toBe(false); + const modal = win.document.getElementById('customModelPickModal')!; + expect(modal.classList.contains('active')).toBe(true); + const list = win.document.getElementById('customModelPickList')!; + expect(list.querySelectorAll('button').length).toBe(2); + expect(list.textContent).toContain('qwen3'); + expect(list.textContent).toContain('llama3'); + }); + + it('always asks with 2+ models, even when a defaultModelId is set — the point is letting this launch differ', async () => { + const { win, app } = bootApp({ + hosts: [ + { + id: 'llama-box', + label: 'llama.cpp', + baseUrl: 'http://localhost:8080', + models: ['qwen3', 'llama3'], + defaultModelId: 'qwen3', + }, + ], + }); + await app.selectCustomModelEntry('claude', 'llama-box'); + const modal = win.document.getElementById('customModelPickModal')!; + expect(modal.classList.contains('active')).toBe(true); + // The default is marked, not auto-chosen. + expect(win.document.getElementById('customModelPickList')!.textContent).toContain('Default'); + }); + + it('picking a row in the modal closes it and launches with that exact model', async () => { + const { win, app } = bootApp({ + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://localhost:8080', models: ['qwen3', 'llama3'] }], + }); + let launched: unknown[] | null = null; + app.runCustomModelEntry = async (...args: unknown[]) => { + launched = args; + }; + win.app = app; + + await app.selectCustomModelEntry('claude', 'llama-box'); + const buttons = win.document.getElementById('customModelPickList')!.querySelectorAll('button'); + const llama3Btn = [...buttons].find((b) => b.textContent?.includes('llama3')) as unknown as HTMLButtonElement & { + onclick: (e: unknown) => void; + }; + expect(typeof llama3Btn.onclick).toBe('function'); + llama3Btn.onclick(new (win as any).Event('click')); + + expect(launched).toEqual(['claude', 'llama-box', 'llama3']); + expect(win.document.getElementById('customModelPickModal')!.classList.contains('active')).toBe(false); + }); + + it('re-fetches the endpoint at click time rather than trusting anything cached from the menu render', async () => { + // The background re-discovery sweep (server-side, every 5 minutes) or a + // settings-panel edit can change the model list between opening the + // dropdown and clicking a row — the picker must reflect what is current. + let fetchCount = 0; + const { win, app } = bootApp({}); + app._apiJson = async (path: string) => { + if (path !== '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/api/model-endpoints') return null; + fetchCount += 1; + return [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: ['qwen3', 'llama3', 'phi4'] }]; + }; + await app.selectCustomModelEntry('claude', 'llama-box'); + expect(fetchCount).toBe(1); + expect(win.document.getElementById('customModelPickList')!.querySelectorAll('button').length).toBe(3); + }); + + it('toasts and does nothing when the endpoint has vanished by click time', async () => { + const { app } = bootApp({ hosts: [] }); + let toastMessage: string | null = null; + app.showToast = (msg: string) => { + toastMessage = msg; + }; + await app.selectCustomModelEntry('claude', 'ghost-endpoint'); + expect(toastMessage).toMatch(/no longer exists/i); + }); + + it('toasts and does nothing when the endpoint has zero discovered models by click time', async () => { + const { app } = bootApp({ + hosts: [{ id: 'llama-box', label: 'llama.cpp', baseUrl: 'http://x', models: [] }], + }); + let toastMessage: string | null = null; + app.showToast = (msg: string) => { + toastMessage = msg; + }; + await app.selectCustomModelEntry('claude', 'llama-box'); + expect(toastMessage).toMatch(/no models discovered/i); + }); +}); + +describe('Custom Model Endpoint Profiles: applying a picked entry', () => { + it('does not apply the endpoint to a session that was already open when the launch fails', async () => { + const { app } = bootApp({}); + app.activeSessionId = 'already-open-session'; + // Simulate every run*() function's own documented behaviour: a declined or + // failed launch handles its own error and returns normally without ever + // changing activeSessionId — it does NOT throw and does NOT leave it null. + app.run = async () => {}; + app._runInFlight = false; + let applyCalled = false; + app._api = async (path: string) => { + if (path.includes('/custom-model')) applyCalled = true; + return { ok: true, json: async () => ({ success: true, data: {} }) }; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(applyCalled).toBe(false); + expect(app.activeSessionId).toBe('already-open-session'); + }); + + it('applies the endpoint once run() actually produces a NEW active session', async () => { + const { app } = bootApp({}); + app.activeSessionId = 'old-session'; + app.run = async () => { + app.activeSessionId = 'new-session'; + }; + const calls: Array<{ path: string; body: unknown }> = []; + app._api = async (path: string, opts?: { body?: unknown }) => { + calls.push({ path, body: opts?.body }); + return { + ok: true, + json: async () => ({ success: true, data: { customModel: { endpointId: 'llama-box' }, restarted: true } }), + }; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(calls).toHaveLength(1); + expect(calls[0].path).toBe('/api/sessions/new-session/custom-model'); + expect(calls[0].body).toEqual({ endpointId: 'llama-box', modelId: 'qwen3' }); + }); + + it('waits for the freshly launched session to go idle before applying, so its own boot activity is never mistaken for a busy turn', async () => { + // Measured live: a just-launched CLI reports 'busy' for its own startup + // (spinner, workspace-trust check) well before the apply call could + // otherwise reach it, and the apply route's isBusy() guard correctly + // refuses to restart a session mid-turn — which a fresh boot looks + // exactly like from the outside. This pins the fix: wait for idle FIRST. + const { app } = bootApp({}); + app.activeSessionId = 'old-session'; + app.run = async () => { + app.activeSessionId = 'new-session'; + }; + const calls: string[] = []; + app._apiJson = async (path: string) => { + calls.push(path); + if (path === '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/api/model-endpoints') return []; + return null; // the wait call's return value is unused — a timeout is a normal 200 + }; + app._api = async (path: string) => { + calls.push(path); + return { ok: true, json: async () => ({ success: true, data: {} }) }; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + const waitIndex = calls.findIndex((p) => p.includes('/wait?')); + const applyIndex = calls.findIndex((p) => p.endsWith('/custom-model')); + expect(waitIndex).toBeGreaterThanOrEqual(0); + expect(calls[waitIndex]).toBe('/api/sessions/new-session/wait?until=idle&timeout=20000'); + expect(applyIndex).toBeGreaterThan(waitIndex); + }); + + it('surfaces the real server error in the toast on a failed apply, rather than a generic message', async () => { + const { app } = bootApp({}); + app.activeSessionId = 'old-session'; + app.run = async () => { + app.activeSessionId = 'new-session'; + }; + app._api = async () => ({ + ok: false, + status: 400, + json: async () => ({ + success: false, + error: 'Custom model endpoints are not supported for remote (SSH) or Docker sessions yet', + }), + }); + let toastMessage: string | null = null; + let toastType: string | null = null; + app.showToast = (msg: string, type: string) => { + toastMessage = msg; + toastType = type; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(toastMessage).toContain('Custom model endpoints are not supported for remote (SSH) or Docker sessions yet'); + expect(toastType).toBe('error'); + }); + + it('shows a status toast for the native-boot-then-restart window, so it never reads as the endpoint failing to apply', async () => { + // Claude still goes through this two-step launch (see runCustomModelEntry's own + // comment for why) — without something saying so, the native boot it starts with + // (which can genuinely talk to the cloud model for a moment) reads as "the + // endpoint didn't apply" rather than "the switch hasn't happened yet". + const { app } = bootApp({}); + app.activeSessionId = 'old-session'; + app.run = async () => { + app.activeSessionId = 'new-session'; + }; + app._api = async () => ({ + ok: true, + json: async () => ({ success: true, data: { customModel: { endpointId: 'llama-box' }, restarted: true } }), + }); + const banners: Array<{ message: string; dismissed: boolean }> = []; + const messageHistory: string[] = []; + app._showCenterStatus = (message: string) => { + const entry = { message, dismissed: false }; + banners.push(entry); + messageHistory.push(message); + return { + dismiss: () => { + entry.dismissed = true; + }, + setMessage: (next: string) => { + entry.message = next; + messageHistory.push(next); + }, + }; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(banners).toHaveLength(1); // updated in place, not stacked with a second banner + expect(messageHistory[0]).toContain('Claude started — switching to llama-box'); + expect(messageHistory.at(-1)).toContain('Pointed at llama-box — restarting'); + }); + + it('dismisses the status banner on a failed apply rather than leaving it stuck on "switching"', async () => { + const { app } = bootApp({}); + app.activeSessionId = 'old-session'; + app.run = async () => { + app.activeSessionId = 'new-session'; + }; + app._api = async () => ({ + ok: false, + status: 500, + json: async () => ({ success: false, error: 'boom' }), + }); + let bannerDismissed = false; + app._showCenterStatus = () => ({ + dismiss: () => { + bannerDismissed = true; + }, + setMessage: () => {}, + }); + let toastMessage: string | undefined; + app.showToast = (message: string) => { + toastMessage = message; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(bannerDismissed).toBe(true); // the "switching..." banner, cleaned up + expect(toastMessage).toContain('boom'); // the error toast, separate from it + }); + + it('routes through run() itself, so the Run in-flight lock actually engages', async () => { + // CLAUDE.md, Run launch synchronization: the lock exists so a double click + // cannot create duplicate sessions. A hardcoded dispatch table bypassing + // run() would never set _runInFlight, which is what this pins. + const { app } = bootApp({}); + let sawInFlight = false; + app.run = async function (this: typeof app) { + if (this._runInFlight) return; + this._runInFlight = true; + sawInFlight = true; + this._runInFlight = false; + }; + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + expect(sawInFlight).toBe(true); + }); + + it('restores the previous _runMode after a one-off custom-model launch, never persisting it', async () => { + const { app } = bootApp({}); + app._runMode = 'opencode'; + let modeDuringRun: string | undefined; + app.run = async function (this: typeof app) { + modeDuringRun = this._runMode; + }; + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + expect(modeDuringRun).toBe('claude'); + expect(app._runMode).toBe('opencode'); + }); +}); + +describe('Custom Model Endpoint Profiles: llama-swap model-swap confirmation and loading state', () => { + function launchHarness(applyResponses: Array>) { + const { win, app } = bootApp({}); + app.activeSessionId = 'old-session'; + app.run = async () => { + app.activeSessionId = 'new-session'; + }; + const applyBodies: unknown[] = []; + let call = 0; + app._api = async (path: string, opts?: { body?: unknown }) => { + if (path.endsWith('/custom-model')) { + applyBodies.push(opts?.body); + const data = applyResponses[Math.min(call, applyResponses.length - 1)]; + call += 1; + return { ok: true, status: 200, json: async () => ({ success: true, data }) }; + } + throw new Error(`unexpected _api call: ${path}`); + }; + return { win, app, applyBodies }; + } + + it('confirming the in-app swap-confirm modal re-sends the apply with confirmed:true', async () => { + const { app, applyBodies } = launchHarness([ + { + requiresConfirmation: true, + currentlyLoadedModel: 'llama3', + affectedSessions: [{ id: 's2', name: 'w2-otherbox' }], + }, + { customModel: { endpointId: 'llama-box' }, restarted: true, modelSwapInProgress: true }, + ]); + let confirmMessage: string | undefined; + app._confirmModelSwap = async (message: string) => { + confirmMessage = message; + return true; + }; + app._watchLlamaSwapLoading = async () => {}; // not under test here + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(confirmMessage).toContain('w2-otherbox'); + expect(confirmMessage).toContain('llama3'); + expect(confirmMessage).toContain('qwen3'); + expect(applyBodies).toEqual([ + { endpointId: 'llama-box', modelId: 'qwen3' }, + { endpointId: 'llama-box', modelId: 'qwen3', confirmed: true }, + ]); + }); + + it('cancelling the in-app swap-confirm modal keeps the native backend and never re-sends the apply', async () => { + const { app, applyBodies } = launchHarness([ + { requiresConfirmation: true, currentlyLoadedModel: 'llama3', affectedSessions: [{ id: 's2', name: 'w2' }] }, + ]); + app._confirmModelSwap = async () => false; + let toastMessage: string | undefined; + app.showToast = (msg: string) => { + toastMessage = msg; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(applyBodies).toHaveLength(1); // no second (confirmed) call + expect(toastMessage).toMatch(/cancelled/i); + }); + + it('a successful apply with modelSwapInProgress kicks off the loading watcher', async () => { + const { app } = launchHarness([ + { customModel: { endpointId: 'llama-box' }, restarted: true, modelSwapInProgress: true }, + ]); + let watched: unknown[] | null = null; + app._watchLlamaSwapLoading = async (...args: unknown[]) => { + watched = args; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(watched).toEqual(['llama-box', 'qwen3', 'new-session']); + }); + + it('a successful apply with no swap needed never starts the loading watcher', async () => { + const { app } = launchHarness([ + { customModel: { endpointId: 'llama-box' }, restarted: true, modelSwapInProgress: false }, + ]); + let watchCalled = false; + app._watchLlamaSwapLoading = async () => { + watchCalled = true; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(watchCalled).toBe(false); + }); +}); + +describe('Custom Model Endpoint Profiles: _watchLlamaSwapLoading polling', () => { + // Driven with millisecond intervals (the function's own pollIntervalMs/maxWaitMs + // params — real callers never pass them) rather than fake timers: this code runs + // inside the JSDOM window's own realm (bootApp's `runScripts: "dangerously"` eval), + // whose setTimeout is NOT the one vi.useFakeTimers() patches, so advancing fake + // timers here would advance nothing and either hang or silently no-op. + + it('dismisses the loading banner as soon as the target model reports ready', async () => { + const { app } = bootApp({}); + const bannerMessages: string[] = []; + const dismissed: string[] = []; + app._showCenterStatus = (message: string) => { + bannerMessages.push(message); + return { dismiss: () => dismissed.push(message), setMessage: () => {} }; + }; + const toastCalls: string[] = []; + app.showToast = (message: string) => { + toastCalls.push(message); + }; + app._apiJson = async () => ({ isLlamaSwap: true, running: [{ model: 'qwen3', state: 'ready' }] }); + + await app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 5); + + expect(bannerMessages[0]).toMatch(/loading qwen3/i); + expect(dismissed).toContain(bannerMessages[0]); + expect(toastCalls.at(-1)).toMatch(/ready/i); + }); + + it('adds a second line with the real llama.cpp log line once one is available, stripped of the bootlog prefix', async () => { + const { app } = bootApp({}); + const bannerMessages: string[] = []; + app._showCenterStatus = (message: string) => { + bannerMessages.push(message); + return { dismiss: () => {}, setMessage: (next: string) => bannerMessages.push(next) }; + }; + app.showToast = () => {}; + let statusCalls = 0; + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') return null; // size lookup — unrelated to this test + statusCalls += 1; + if (statusCalls === 1) { + return { + isLlamaSwap: true, + running: [{ model: 'qwen3', state: 'starting' }], + logLine: '0.31.428.568 I srv llama_server: model loaded', + }; + } + return { isLlamaSwap: true, running: [{ model: 'qwen3', state: 'ready' }] }; + }; + + await app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 5); + + // First render (before any poll has landed) has no log line at all. + expect(bannerMessages[0]).not.toMatch(/llama\.cpp:/); + // Second render carries the log line, bootlog prefix (timestamp/level/component) stripped. + const withLogLine = bannerMessages.find((m) => m.includes('llama.cpp:')); + expect(withLogLine).toContain('llama.cpp: llama_server: model loaded'); + expect(withLogLine).not.toContain('0.31.428.568'); + expect(withLogLine).not.toContain(' I srv'); + }); + + it('shows no second line at all when the endpoint has no logLine to offer', async () => { + const { app } = bootApp({}); + const bannerMessages: string[] = []; + app._showCenterStatus = (message: string) => { + bannerMessages.push(message); + return { dismiss: () => {}, setMessage: (next: string) => bannerMessages.push(next) }; + }; + app.showToast = () => {}; + app._apiJson = async () => ({ isLlamaSwap: true, running: [{ model: 'qwen3', state: 'ready' }] }); + + await app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 5); + + expect(bannerMessages.some((m) => m.includes('llama.cpp:'))).toBe(false); + }); + + it('is unbounded — never gives up on its own, even after many polls with no ready model', async () => { + // No countdown, no timeout: confirms the loop just keeps polling rather than + // eventually erroring out on its own after some fixed number of checks. + const { app } = bootApp({}); + app._showCenterStatus = () => ({ dismiss: () => {}, setMessage: () => {} }); + app.showToast = () => {}; + let calls = 0; + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') return null; + calls += 1; + if (calls >= 20) return { isLlamaSwap: true, running: [{ model: 'qwen3', state: 'ready' }] }; + return { isLlamaSwap: true, running: [{ model: 'something-else', state: 'ready' }] }; + }; + + await app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 1); + + expect(calls).toBe(20); // it really did keep polling past what the old bounded wait allowed + }); + + it('clicking Cancel on the banner dismisses it, shows an info toast (not an error), and closes the session', async () => { + const { app } = bootApp({}); + let onCancel: (() => void) | undefined; + let dismissed = false; + app._showCenterStatus = (_message: string, opts?: { onCancel?: () => void }) => { + onCancel = opts?.onCancel; + return { dismiss: () => (dismissed = true), setMessage: () => {} }; + }; + const toastCalls: Array<{ message: string; type: string }> = []; + app.showToast = (message: string, type = 'info') => { + toastCalls.push({ message, type }); + }; + let closedSessionId: string | undefined; + app.closeSession = async (id: string) => { + closedSessionId = id; + }; + app._apiJson = async () => ({ isLlamaSwap: true, running: [{ model: 'something-else', state: 'ready' }] }); + + const watch = app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 5); + // Give the loop a couple of ticks to actually be polling, then cancel it — a real + // click happens whenever the user gets around to it, not on the very first render. + await new Promise((resolve) => setTimeout(resolve, 15)); + expect(onCancel).toBeTypeOf('function'); + onCancel!(); + await watch; + + expect(dismissed).toBe(true); + const cancelToast = toastCalls.find((t) => /cancelled/i.test(t.message)); + expect(cancelToast?.type).toBe('info'); // not 'error' — this was deliberate, not a failure + expect(cancelToast?.message).toMatch(/session has been closed/i); + expect(closedSessionId).toBe('sess-1'); + }); + + it('never closes anything when no sessionId was given (a caller that has none to close)', async () => { + const { app } = bootApp({}); + let onCancel: (() => void) | undefined; + app._showCenterStatus = (_message: string, opts?: { onCancel?: () => void }) => { + onCancel = opts?.onCancel; + return { dismiss: () => {}, setMessage: () => {} }; + }; + app.showToast = () => {}; + let closeCalled = false; + app.closeSession = async () => { + closeCalled = true; + }; + app._apiJson = async () => ({ isLlamaSwap: true, running: [{ model: 'something-else', state: 'ready' }] }); + + const watch = app._watchLlamaSwapLoading('llama-box', 'qwen3', undefined, 5); + await new Promise((resolve) => setTimeout(resolve, 15)); + onCancel!(); + await watch; + + expect(closeCalled).toBe(false); + }); + + it('stops polling (without a warning) once the endpoint no longer reads as llama-swap', async () => { + const { app } = bootApp({}); + let bannerDismissed = false; + app._showCenterStatus = () => ({ + dismiss: () => { + bannerDismissed = true; + }, + setMessage: () => {}, + }); + const toastCalls: string[] = []; + app.showToast = (message: string) => { + toastCalls.push(message); + }; + app._apiJson = async () => ({ isLlamaSwap: false, running: [] }); + + await app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 5); + + expect(bannerDismissed).toBe(true); + expect(toastCalls).toHaveLength(0); // no follow-up warning toast + }); + + it('keeps waiting through a transient status-fetch failure instead of giving up early', async () => { + const { app } = bootApp({}); + app._showCenterStatus = () => ({ dismiss: () => {}, setMessage: () => {} }); + const toastCalls: string[] = []; + app.showToast = (message: string) => { + toastCalls.push(message); + }; + let call = 0; + app._apiJson = async () => { + call += 1; + if (call === 1) return null; // transient failure + return { isLlamaSwap: true, running: [{ model: 'qwen3', state: 'ready' }] }; + }; + + await app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 5); + + expect(toastCalls.at(-1)).toMatch(/ready/i); + }); + + it('checks immediately rather than waiting a full interval before the first check', async () => { + // A model that is already ready by the time this runs (a fast load, or a re-apply + // onto one that was already loaded) shouldn't sit on "Loading..." for a whole + // pollIntervalMs before saying so. + const { app } = bootApp({}); + app._showCenterStatus = () => ({ dismiss: () => {}, setMessage: () => {} }); + let calls = 0; + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') return []; // size lookup — no match, no estimate + calls += 1; + return { isLlamaSwap: true, running: [{ model: 'qwen3', state: 'ready' }] }; + }; + + // A huge interval that would time the test out if the function actually waited for + // it before the first check. + await app._watchLlamaSwapLoading('llama-box', 'qwen3', 'sess-1', 60000); + + expect(calls).toBe(1); + }); + + it('a newer call takes over the shared banner — a superseded older call never touches it', async () => { + const { app } = bootApp({}); + const dismissCalls: string[] = []; + app._showCenterStatus = (message: string) => ({ + dismiss: () => dismissCalls.push(message), + setMessage: () => {}, + }); + app.showToast = () => {}; + // The FIRST call never sees its own target model ready — left alone (unbounded, no + // timeout) it would poll forever, but being superseded below must still make it stop + // on its own very next isCurrent() check rather than needing a timeout to exit. + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') return []; + return { isLlamaSwap: true, running: [] }; + }; + const firstCall = app._watchLlamaSwapLoading('llama-box', 'model-a', undefined, 5); + + // Second call, for a DIFFERENT model that IS ready right away, takes over the banner. + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') return []; + return { isLlamaSwap: true, running: [{ model: 'model-b', state: 'ready' }] }; + }; + await app._watchLlamaSwapLoading('llama-box', 'model-b', undefined, 5); + + // Let the stale first call notice it's been superseded and return on its own. + await firstCall; + + // Whatever the first call did or didn't show along the way, being superseded must + // never touch a banner state that belongs to the newer, still-current call — exactly + // one dismiss, for model-b, is the tell. + expect(dismissCalls).toHaveLength(1); + expect(dismissCalls[0]).toContain('model-b'); + }); +}); + +describe('Custom Model Endpoint Profiles: model size lookup (no time estimate — see the unbounded-wait describe above)', () => { + it('_lookupModelSizeGB reads the size off the matching endpoint/model, ignoring one with no parseable size', async () => { + const { app } = bootApp({}); + app._apiJson = async (path: string) => { + expect(path).toBe('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/api/model-endpoints'); + return [ + { id: 'llama-box', modelSizesGB: { 'qwen3.8-27b-ud-q4_k_xl': 16.35, big: undefined } }, + { id: 'other-box', modelSizesGB: { 'qwen3.8-27b-ud-q4_k_xl': 999 } }, // must not match wrong endpoint + ]; + }; + + expect(await app._lookupModelSizeGB('llama-box', 'qwen3.8-27b-ud-q4_k_xl')).toBe(16.35); + expect(await app._lookupModelSizeGB('llama-box', 'big')).toBeUndefined(); // no parseable size + expect(await app._lookupModelSizeGB('llama-box', 'unknown-model')).toBeUndefined(); + expect(await app._lookupModelSizeGB('ghost-endpoint', 'qwen3')).toBeUndefined(); + }); + + it('_lookupModelSizeGB is best-effort: an unreachable/malformed response yields undefined, never a throw', async () => { + const { app } = bootApp({}); + app._apiJson = async () => { + throw new Error('network down'); + }; + await expect(app._lookupModelSizeGB('llama-box', 'qwen3')).resolves.toBeUndefined(); + + app._apiJson = async () => null; // e.g. a failed request _apiJson already swallowed + await expect(app._lookupModelSizeGB('llama-box', 'qwen3')).resolves.toBeUndefined(); + }); + + it('the loading banner includes the size, and the generic hardware/model-size disclaimer, when the size is known', async () => { + const { app } = bootApp({}); + const bannerMessages: string[] = []; + app._showCenterStatus = (message: string) => { + bannerMessages.push(message); + return { dismiss: () => {}, setMessage: () => {} }; + }; + app.showToast = () => {}; + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') { + return [{ id: 'llama-box', modelSizesGB: { 'qwen3.8-27b-ud-q4_k_xl': 16.35 } }]; + } + return { isLlamaSwap: true, running: [{ model: 'qwen3.8-27b-ud-q4_k_xl', state: 'ready' }] }; + }; + + await app._watchLlamaSwapLoading('llama-box', 'qwen3.8-27b-ud-q4_k_xl', undefined, 5); + + expect(bannerMessages[0]).toBe( + 'Loading qwen3.8-27b-ud-q4_k_xl (16.4 GB) on llama-box — this can take a while depending on your hardware and the model size.' + ); + }); + + it('the loading banner omits the size but keeps the disclaimer when the size is unknown', async () => { + const { app } = bootApp({}); + const bannerMessages: string[] = []; + app._showCenterStatus = (message: string) => { + bannerMessages.push(message); + return { dismiss: () => {}, setMessage: () => {} }; + }; + app.showToast = () => {}; + app._apiJson = async (path: string) => { + if (path === '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/api/model-endpoints') return [{ id: 'llama-box', modelSizesGB: {} }]; + return { isLlamaSwap: true, running: [{ model: 'big', state: 'ready' }] }; + }; + + await app._watchLlamaSwapLoading('llama-box', 'big', undefined, 5); + + expect(bannerMessages[0]).toBe( + 'Loading big on llama-box — this can take a while depending on your hardware and the model size.' + ); + }); +}); + +describe('Custom Model Endpoint Profiles: _showCenterStatus Cancel button (real DOM, not the stub)', () => { + it('renders a real, clickable Cancel button when onCancel is given, and wires it up', () => { + const { win, app } = bootAppWithRealCenterStatus(); + let cancelled = false; + + app._showCenterStatus('Loading qwen3 on llama-box…', { onCancel: () => (cancelled = true) }); + + const btn = win.document.querySelector('.center-status-cancel') as HTMLButtonElement | null; + expect(btn).not.toBeNull(); + expect(btn!.textContent).toBe('Cancel'); + btn!.onclick!(new (win as any).Event('click')); + expect(cancelled).toBe(true); + }); + + it('renders no Cancel button at all when onCancel is not given', () => { + const { win, app } = bootAppWithRealCenterStatus(); + + app._showCenterStatus('Loading qwen3 on llama-box…'); + + expect(win.document.querySelector('.center-status-cancel')).toBeNull(); + }); + + it("an 'error' banner keeps its own × close button rather than growing a redundant Cancel, even if onCancel is passed", () => { + const { win, app } = bootAppWithRealCenterStatus(); + + app._showCenterStatus('Something went wrong', { type: 'error', onCancel: () => {} }); + + expect(win.document.querySelector('.center-status-close')).not.toBeNull(); + expect(win.document.querySelector('.center-status-cancel')).toBeNull(); + }); +}); + +describe("Custom Model Endpoint Profiles: requiresContextWarning (this CLI's own overhead can exceed a small model's real context)", () => { + function launchHarness(applyResponses: Array>) { + const { win, app } = bootApp({}); + app.activeSessionId = 'old-session'; + app.run = async () => { + app.activeSessionId = 'new-session'; + }; + const applyBodies: unknown[] = []; + let call = 0; + app._api = async (path: string, opts?: { body?: unknown }) => { + if (path.endsWith('/custom-model')) { + applyBodies.push(opts?.body); + const data = applyResponses[Math.min(call, applyResponses.length - 1)]; + call += 1; + return { ok: true, status: 200, json: async () => ({ success: true, data }) }; + } + throw new Error(`unexpected _api call: ${path}`); + }; + return { win, app, applyBodies }; + } + + it('confirming the in-app context-warning modal re-sends the apply with confirmed:true', async () => { + const { app, applyBodies } = launchHarness([ + { requiresContextWarning: true, modelId: 'qwen3', contextLength: 16384, minSafeContextTokens: 40000 }, + { customModel: { endpointId: 'llama-box' }, restarted: true, modelSwapInProgress: false }, + ]); + let confirmArgs: unknown[] | undefined; + app._confirmContextWarning = async (...args: unknown[]) => { + confirmArgs = args; + return true; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(confirmArgs).toEqual(['qwen3', 16384, 40000]); + expect(applyBodies).toEqual([ + { endpointId: 'llama-box', modelId: 'qwen3' }, + { endpointId: 'llama-box', modelId: 'qwen3', confirmed: true }, + ]); + }); + + it('declining the in-app context-warning modal keeps the native backend and never re-sends the apply', async () => { + const { app, applyBodies } = launchHarness([ + { requiresContextWarning: true, modelId: 'qwen3', contextLength: 16384, minSafeContextTokens: 40000 }, + ]); + app._confirmContextWarning = async () => false; + let toastMessage: string | undefined; + app.showToast = (msg: string) => { + toastMessage = msg; + }; + + await app.runCustomModelEntry('claude', 'llama-box', 'qwen3'); + + expect(applyBodies).toHaveLength(1); // no second (confirmed) call + expect(toastMessage).toMatch(/context window too small/i); + }); +}); + +describe('Custom Model Endpoint Profiles: _confirmModelSwap (in-app modal, replaces a native confirm() popup)', () => { + it('shows the message, activates the modal, and resolves true when "Switch anyway" is clicked', async () => { + const { win, app } = bootApp({}); + const promise = app._confirmModelSwap('w2 is using llama3. Switch anyway?'); + + const modal = win.document.getElementById('customModelSwapConfirmModal')!; + expect(modal.classList.contains('active')).toBe(true); + expect(win.document.getElementById('customModelSwapConfirmMessage')!.textContent).toBe( + 'w2 is using llama3. Switch anyway?' + ); + + app._resolveModelSwapConfirm(true); + + expect(await promise).toBe(true); + expect(modal.classList.contains('active')).toBe(false); + }); + + it('resolves false when Cancel (or the backdrop) is clicked, without ever showing a browser confirm() popup', async () => { + const { win, app } = bootApp({}); + const promise = app._confirmModelSwap('w2 is using llama3. Switch anyway?'); + app._resolveModelSwapConfirm(false); + expect(await promise).toBe(false); + expect(win.document.getElementById('customModelSwapConfirmModal')!.classList.contains('active')).toBe(false); + }); +}); + +describe('Custom Model Endpoint Profiles: _confirmContextWarning (in-app modal, native backend never restarted while it is up)', () => { + it('shows a message naming the model, the discovered context and the safe floor, activates the modal, and resolves true on "Launch anyway"', async () => { + const { win, app } = bootApp({}); + const promise = app._confirmContextWarning('qwen3.8-27b-ud-q4_k_xl', 16384, 40000); + + const modal = win.document.getElementById('customModelContextWarningModal')!; + expect(modal.classList.contains('active')).toBe(true); + const message = win.document.getElementById('customModelContextWarningMessage')!.textContent!; + expect(message).toContain('qwen3.8-27b-ud-q4_k_xl'); + expect(message).toContain('16,384'); + expect(message).toContain('40,000'); + expect(message).toMatch(/llama-swap/i); + expect(message).toMatch(/fit-ctx/i); + + app._resolveContextWarningConfirm(true); + + expect(await promise).toBe(true); + expect(modal.classList.contains('active')).toBe(false); + }); + + it('resolves false when Cancel is clicked', async () => { + const { win, app } = bootApp({}); + const promise = app._confirmContextWarning('qwen3', 16384, 40000); + app._resolveContextWarningConfirm(false); + expect(await promise).toBe(false); + expect(win.document.getElementById('customModelContextWarningModal')!.classList.contains('active')).toBe(false); + }); + + it('describes an unknown context length without printing a bogus number', async () => { + const { win, app } = bootApp({}); + void app._confirmContextWarning('qwen3', undefined, 40000); + const message = win.document.getElementById('customModelContextWarningMessage')!.textContent!; + expect(message).not.toMatch(/undefined/); + expect(message).toMatch(/unknown/i); + app._resolveContextWarningConfirm(false); + }); +}); diff --git a/test/custom-model-swap-displacement.test.ts b/test/custom-model-swap-displacement.test.ts new file mode 100644 index 000000000..a8034c44d --- /dev/null +++ b/test/custom-model-swap-displacement.test.ts @@ -0,0 +1,180 @@ +/** + * @fileoverview Tests for `detectCustomModelSwapDisplacements()`, the periodic sweep + * behind server.ts's "custom model swap-displacement check" timer + * (docs/custom-model-endpoints-plan.md). The apply/create routes' own swap-conflict check + * only ever runs at a session's own launch/apply moment — this sweep is what catches a + * LATER eviction triggered by a different session's normal use, which the launch-time + * check structurally cannot see. + * + * Kept in its own file for the same reason as `custom-model-endpoint-rediscovery.test.ts`: + * a sweep that walks every saved host would otherwise pick up hosts other tests in a + * shared file create, making an exact call-count assertion meaningless. + * + * Port: N/A (no server; drives readCustomModelHosts/writeCustomModelHosts directly plus + * the mocked webviewFetch dispatcher). + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { getDataDir } from '../src/config/instance.js'; +import { writeCustomModelHosts, type CustomModelHost } from '../src/custom-model-hosts.js'; +import { + detectCustomModelSwapDisplacements, + type CustomModelSessionLike, +} from '../src/web/routes/custom-model-routes.js'; +import { webviewFetch } from '../src/web/webview-egress.js'; + +vi.mock('../src/web/webview-egress.js', async () => { + const actual = await vi.importActual('../src/web/webview-egress.js'); + return { ...actual, webviewFetch: vi.fn() }; +}); + +const fetchMock = vi.mocked(webviewFetch); + +const ENDPOINT: CustomModelHost = { + id: 'llama-swap', + label: 'llama-swap', + baseUrl: 'http://192.168.1.50:8080', + apiKey: 'k', +}; + +function session( + overrides: Partial & Pick +): CustomModelSessionLike { + return { name: overrides.id, ...overrides }; +} + +function mockRunning(running: Array<{ model: string; state: string }>) { + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '/running') return new Response(JSON.stringify({ running }), { status: 200 }); + throw new Error(`unexpected request in this test: ${url.href}`); + }); +} + +beforeEach(() => { + fetchMock.mockReset(); +}); + +describe('detectCustomModelSwapDisplacements', () => { + it('flags a session whose own model is no longer in the running list, naming what displaced it', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT]); + mockRunning([{ model: 'fast', state: 'ready' }]); + const w1 = session({ id: 'w1', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + const notified = new Set(); + + const displacements = await detectCustomModelSwapDisplacements([w1], notified); + + expect(displacements).toEqual([ + { + sessionId: 'w1', + sessionName: 'w1', + endpointId: 'llama-swap', + previousModel: 'qwen3', + currentlyLoadedModel: 'fast', + }, + ]); + expect(notified.has('w1')).toBe(true); + }); + + it('does not flag a session whose own model is still the one loaded and ready', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT]); + mockRunning([{ model: 'qwen3', state: 'ready' }]); + const w1 = session({ id: 'w1', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + + const displacements = await detectCustomModelSwapDisplacements([w1], new Set()); + + expect(displacements).toEqual([]); + }); + + it('notifies once per displacement — a repeat sweep with nothing changed does not re-flag it', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT]); + mockRunning([{ model: 'fast', state: 'ready' }]); + const w1 = session({ id: 'w1', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + const notified = new Set(); + + const first = await detectCustomModelSwapDisplacements([w1], notified); + const second = await detectCustomModelSwapDisplacements([w1], notified); + + expect(first).toHaveLength(1); + expect(second).toEqual([]); + }); + + it('clears the notified flag once the session is back on its own model, so a later displacement flags again', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT]); + const w1 = session({ id: 'w1', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + const notified = new Set(); + + mockRunning([{ model: 'fast', state: 'ready' }]); + await detectCustomModelSwapDisplacements([w1], notified); + expect(notified.has('w1')).toBe(true); + + mockRunning([{ model: 'qwen3', state: 'ready' }]); // back to normal + await detectCustomModelSwapDisplacements([w1], notified); + expect(notified.has('w1')).toBe(false); + + mockRunning([{ model: 'fast', state: 'ready' }]); // displaced again + const third = await detectCustomModelSwapDisplacements([w1], notified); + expect(third).toHaveLength(1); + }); + + it('skips a session on a non-llama-swap endpoint (no /running) — nothing to compare, never flagged', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT]); + fetchMock.mockResolvedValue(new Response('not found', { status: 404 })); + const w1 = session({ id: 'w1', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + + const displacements = await detectCustomModelSwapDisplacements([w1], new Set()); + + expect(displacements).toEqual([]); + }); + + it('skips a session whose endpoint was deleted since it was created', async () => { + await writeCustomModelHosts(getDataDir(), []); // ENDPOINT never saved + const w1 = session({ id: 'w1', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + + const displacements = await detectCustomModelSwapDisplacements([w1], new Set()); + + expect(displacements).toEqual([]); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('ignores a plain session with no customModel selection at all', async () => { + const displacements = await detectCustomModelSwapDisplacements([session({ id: 'plain' })], new Set()); + expect(displacements).toEqual([]); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it('one endpoint failing (unreachable) never blocks checking sessions on another', async () => { + const DOWN: CustomModelHost = { id: 'down', label: 'down', baseUrl: 'http://192.168.1.60:8080' }; + await writeCustomModelHosts(getDataDir(), [ENDPOINT, DOWN]); + fetchMock.mockImplementation(async (url: URL) => { + if (url.href.includes('192.168.1.60')) throw new TypeError('fetch failed', { cause: new Error('ECONNREFUSED') }); + if (url.pathname === '/running') { + return new Response(JSON.stringify({ running: [{ model: 'fast', state: 'ready' }] }), { status: 200 }); + } + throw new Error(`unexpected request in this test: ${url.href}`); + }); + const onDown = session({ id: 'w-down', customModel: { endpointId: 'down', modelId: 'x' } }); + const onLlamaSwap = session({ id: 'w1', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + + const displacements = await detectCustomModelSwapDisplacements([onDown, onLlamaSwap], new Set()); + + expect(displacements).toEqual([ + { + sessionId: 'w1', + sessionName: 'w1', + endpointId: 'llama-swap', + previousModel: 'qwen3', + currentlyLoadedModel: 'fast', + }, + ]); + }); + + it('multiple sessions on the same endpoint each get their own displacement entry', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT]); + mockRunning([{ model: 'gemma', state: 'ready' }]); + const w1 = session({ id: 'w1', name: 'w1-test2', customModel: { endpointId: 'llama-swap', modelId: 'qwen3' } }); + const w2 = session({ id: 'w2', name: 'w2-test2', customModel: { endpointId: 'llama-swap', modelId: 'fast' } }); + + const displacements = await detectCustomModelSwapDisplacements([w1, w2], new Set()); + + expect(displacements.map((d) => d.sessionId).sort()).toEqual(['w1', 'w2']); + }); +}); diff --git a/test/render-index-html.test.ts b/test/render-index-html.test.ts index 0cb6b3347..bc3de380e 100644 --- a/test/render-index-html.test.ts +++ b/test/render-index-html.test.ts @@ -11,7 +11,7 @@ * Port: N/A (no server start). */ import { describe, it, expect, afterEach, vi } from 'vitest'; -import { WebServer } from '../src/web/server.js'; +import { WebServer, escapeScriptJson } from '../src/web/server.js'; import { isClaudeAvailable } from '../src/utils/claude-cli-resolver.js'; import { isOpenCodeAvailable } from '../src/utils/opencode-cli-resolver.js'; import { isCodexAvailable } from '../src/utils/codex-cli-resolver.js'; @@ -187,6 +187,41 @@ describe('WebServer.renderIndexHtml', () => { }); }); + it('reports which run modes the custom-model Run-menu picker may generate an entry for', async () => { + // Read generically off the CLI registry's own capabilities, not a hardcoded id + // list — antigravity (`unsupported`) and shell (`kind !== 'agent'`) must be + // absent, and any enabled agent CLI with a real injection recipe must be + // present, with no mock needed since this reads the real stock registry. + const { server } = makeServer({}); + const html = await render(server); + expect(html).toContain('window.__codemanCustomModelClis='); + const clis = JSON.parse(html.match(/window\.__codemanCustomModelClis=(\[.*?\]);/)![1]) as Array<{ + id: string; + label: string; + }>; + const ids = clis.map((c) => c.id); + expect(ids).toContain('claude'); + expect(ids).not.toContain('antigravity'); + expect(ids).not.toContain('shell'); + for (const cli of clis) { + expect(typeof cli.id).toBe('string'); + expect(typeof cli.label).toBe('string'); + } + }); + + it('escapeScriptJson neutralizes a literal , and still round-trips as a JS literal', () => { + // CliEntry.label is a plain string a user's own clis.json can set (up to 60 + // chars), unlike __codemanCliAvailable's booleans-only payload, so this is + // the one injection that needs it. Exported so this tests the pure + // function directly rather than needing a real WebServer (which needs tmux). + const dangerous = JSON.stringify([{ id: 'x', label: '' }]); + const escaped = escapeScriptJson(dangerous); + expect(escaped).not.toContain('". + expect(eval(escaped)[0].label).toBe(''); + }); + it('still emits the object when nothing at all is installed', async () => { // The all-false case is the one that matters most and the easiest to get // wrong by only injecting when something resolves. @@ -218,6 +253,7 @@ describe('WebServer.renderIndexHtml', () => { const { server } = makeServer({}); const html = await render(server, 'sess-123'); expect(html).not.toContain('__codemanCliAvailable'); + expect(html).not.toContain('__codemanCustomModelClis'); }); it('does not expose gesture at all when CODEMAN_GESTURE is unset', async () => { diff --git a/test/routes/custom-model-routes.test.ts b/test/routes/custom-model-routes.test.ts index 7c864be5a..16b7011fc 100644 --- a/test/routes/custom-model-routes.test.ts +++ b/test/routes/custom-model-routes.test.ts @@ -194,3 +194,198 @@ describe('custom model endpoint CRUD', () => { } }); }); + +describe('defaultModelId — the Run-menu picker’s per-endpoint default', () => { + afterEach(() => { + fetchMock.mockReset(); + }); + + it('rejects a defaultModelId that is not one of the endpoint’s discovered models, on both create and update', async () => { + const { app } = await setup(); + const create = await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { + id: 'ep-default-reject', + label: 'A', + baseUrl: 'http://localhost:8080', + models: ['qwen3'], + defaultModelId: 'ghost', + }, + }); + expect(create.json().success).toBe(false); + expect(create.json().errorCode).toBe('INVALID_INPUT'); + + await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { id: 'ep-default-reject', label: 'A', baseUrl: 'http://localhost:8080', models: ['qwen3'] }, + }); + const update = await app.inject({ + method: 'PUT', + url: '/api/model-endpoints/ep-default-reject', + payload: { label: 'A', baseUrl: 'http://localhost:8080', models: ['qwen3'], defaultModelId: 'ghost' }, + }); + expect(update.json().success).toBe(false); + expect(update.json().errorCode).toBe('INVALID_INPUT'); + }); + + it('accepts a defaultModelId that IS one of the discovered models', async () => { + const { app } = await setup(); + const res = await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { + id: 'ep-default-accept', + label: 'A', + baseUrl: 'http://localhost:8080', + models: ['qwen3', 'llama3'], + defaultModelId: 'llama3', + }, + }); + expect(res.json().success).toBe(true); + expect(res.json().data.host.defaultModelId).toBe('llama3'); + }); + + it('drops a stale default that no longer appears in a fresh discovery, rather than carrying it forward invalid', async () => { + const { app } = await setup(); + await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { + id: 'ep-default-drop', + label: 'A', + baseUrl: 'http://localhost:8080', + models: ['qwen3'], + defaultModelId: 'qwen3', + }, + }); + fetchMock.mockResolvedValue(new Response(JSON.stringify({ data: [{ id: 'llama3' }] }), { status: 200 })); + await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-default-drop/discover-models' }); + + const list = await app.inject({ method: 'GET', url: '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/api/model-endpoints' }); + const stored = (list.json() as Array<{ id: string; defaultModelId?: string }>).find( + (h) => h.id === 'ep-default-drop' + ); + expect(stored?.defaultModelId).toBeUndefined(); + }); + + it('keeps a default that IS still present after a fresh discovery', async () => { + const { app } = await setup(); + await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { + id: 'ep-default-keep', + label: 'A', + baseUrl: 'http://localhost:8080', + models: ['qwen3'], + defaultModelId: 'qwen3', + }, + }); + fetchMock.mockResolvedValue( + new Response(JSON.stringify({ data: [{ id: 'qwen3' }, { id: 'llama3' }] }), { status: 200 }) + ); + await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-default-keep/discover-models' }); + + const list = await app.inject({ method: 'GET', url: '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/api/model-endpoints' }); + const stored = (list.json() as Array<{ id: string; defaultModelId?: string }>).find( + (h) => h.id === 'ep-default-keep' + ); + expect(stored?.defaultModelId).toBe('qwen3'); + }); +}); + +describe('apiKey is never handed back to the browser', () => { + afterEach(() => { + fetchMock.mockReset(); + }); + + it('POST, GET and PUT responses all carry apiKeySet instead of the real key', async () => { + const { app } = await setup(); + const create = await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { id: 'ep-secret', label: 'A', baseUrl: 'http://localhost:8080', apiKey: 'super-secret' }, + }); + expect(create.json().data.host.apiKey).toBeUndefined(); + expect(create.json().data.host.apiKeySet).toBe(true); + + const list = await app.inject({ method: 'GET', url: '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/api/model-endpoints' }); + const listed = (list.json() as Array<{ id: string; apiKey?: string; apiKeySet?: boolean }>).find( + (h) => h.id === 'ep-secret' + ); + expect(listed?.apiKey).toBeUndefined(); + expect(listed?.apiKeySet).toBe(true); + expect(JSON.stringify(list.json())).not.toContain('super-secret'); + + const update = await app.inject({ + method: 'PUT', + url: '/api/model-endpoints/ep-secret', + payload: { label: 'Renamed', baseUrl: 'http://localhost:8080' }, + }); + expect(update.json().data.host.apiKey).toBeUndefined(); + expect(update.json().data.host.apiKeySet).toBe(true); + expect(JSON.stringify(update.json())).not.toContain('super-secret'); + }); + + it('a host with no key set at all reports apiKeySet: false', async () => { + const { app } = await setup(); + const create = await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { id: 'ep-nokey', label: 'A', baseUrl: 'http://localhost:8080' }, + }); + expect(create.json().data.host.apiKeySet).toBe(false); + }); + + it('PUT with no apiKey keeps the stored one, rather than clearing it', async () => { + const { app } = await setup(); + await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { id: 'ep-keep-key', label: 'A', baseUrl: 'http://localhost:8080', apiKey: 'original-key' }, + }); + // Edit without touching the API key field — the real bug this guards: a + // browser round-trip that only ever sees apiKeySet, never the real value, + // must not accidentally send an empty string and wipe a working credential. + const update = await app.inject({ + method: 'PUT', + url: '/api/model-endpoints/ep-keep-key', + payload: { label: 'Renamed', baseUrl: 'http://localhost:8080' }, + }); + expect(update.json().data.host.apiKeySet).toBe(true); + + // Prove it by observing the auth header discovery actually sends. + fetchMock.mockImplementation(async (_url: URL, init?: RequestInit) => { + const headers = init?.headers as Record; + expect(headers.Authorization).toBe('Bearer original-key'); + return new Response(JSON.stringify({ data: [] }), { status: 200 }); + }); + const discover = await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-keep-key/discover-models' }); + expect(discover.json().success).toBe(true); + expect(fetchMock).toHaveBeenCalledTimes(1); + }); + + it('PUT with a new apiKey replaces the stored one', async () => { + const { app } = await setup(); + await app.inject({ + method: 'POST', + url: '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/api/model-endpoints', + payload: { id: 'ep-replace-key', label: 'A', baseUrl: 'http://localhost:8080', apiKey: 'old-key' }, + }); + await app.inject({ + method: 'PUT', + url: '/api/model-endpoints/ep-replace-key', + payload: { label: 'A', baseUrl: 'http://localhost:8080', apiKey: 'new-key' }, + }); + + fetchMock.mockImplementation(async (_url: URL, init?: RequestInit) => { + const headers = init?.headers as Record; + expect(headers.Authorization).toBe('Bearer new-key'); + return new Response(JSON.stringify({ data: [] }), { status: 200 }); + }); + await app.inject({ method: 'POST', url: '/api/model-endpoints/ep-replace-key/discover-models' }); + expect(fetchMock).toHaveBeenCalledTimes(1); + }); +}); diff --git a/test/routes/quick-start-custom-model.test.ts b/test/routes/quick-start-custom-model.test.ts new file mode 100644 index 000000000..3bc0da964 --- /dev/null +++ b/test/routes/quick-start-custom-model.test.ts @@ -0,0 +1,382 @@ +/** + * @fileoverview POST /api/quick-start's `customModel` field (docs/custom-model-endpoints-plan.md): + * the ONE-SHOT launch path that computes a custom-model endpoint's injection BEFORE the + * session/process exists and launches directly on it, so a custom-model Run never shows + * the native-boot-then-restart the dedicated POST /api/sessions/:id/custom-model route's + * restart-in-place design otherwise produces — most visibly on a CLI like Codex whose TUI + * fully reinitializes on a restart. That dedicated route is still what an ALREADY-RUNNING + * session uses to switch later; this is the create-time equivalent. + * + * Mirrors test/routes/session-custom-model.test.ts's fixtures and llama-swap mocking, since + * this route mirrors that one's own checks (llama-swap conflict, unsupported CLI, unknown + * endpoint, an argv-incompatible model id) rather than a lighter, separately-drifting copy. + * + * Session.prototype.startInteractive/startShell are mocked exactly like the workspace-hooks + * quick-start tests: quick-start constructs a REAL Session (not the MockSession the route + * test harness substitutes elsewhere), so tmux must never actually be reached. + * + * Port: N/A (app.inject, no real port needed) + */ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import Fastify, { type FastifyInstance } from 'fastify'; +import fastifyCookie from '@fastify/cookie'; +import { rm, readFile } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; +import { join } from 'node:path'; +import { createMockRouteContext, safeRmHomeTree, type MockRouteContext } from '../mocks/index.js'; +import { installRouteErrorHandler } from '../../src/web/route-error-handler.js'; +import { registerSessionRoutes } from '../../src/web/routes/session-routes.js'; +import { getDataDir } from '../../src/config/instance.js'; +import { CASES_DIR } from '../../src/web/route-helpers.js'; +import { Session } from '../../src/session.js'; +import { writeCustomModelHosts, type CustomModelHost } from '../../src/custom-model-hosts.js'; +import { customModelConfigDir } from '../../src/custom-model-injection-apply.js'; +import { webviewFetch } from '../../src/web/webview-egress.js'; + +vi.mock('../../src/web/webview-egress.js', async () => { + const actual = await vi.importActual( + '../../src/web/webview-egress.js' + ); + return { ...actual, webviewFetch: vi.fn() }; +}); +const fetchMock = vi.mocked(webviewFetch); + +// quick-start's own local-CLI-availability gate (resolveCliLaunchError, unrelated to the +// custom-model injection this file tests) runs BEFORE the code under test and would +// otherwise 404 every non-claude mode on a box with no codex/pi/grok/omp binary installed — +// exactly this test environment. Mirrors the real "not remote" bypass documented at its own +// call site in session-routes.ts (`session-routes.test.ts`'s remote-codex test is the +// precedent for needing this at all). +vi.mock('../../src/utils/cli-launcher.js', async () => { + const actual = await vi.importActual( + '../../src/utils/cli-launcher.js' + ); + return { ...actual, resolveCliLaunchError: vi.fn().mockResolvedValue(null) }; +}); + +const ENDPOINT: CustomModelHost = { + id: 'ep1', + label: 'llama.cpp box', + baseUrl: 'http://192.168.1.50:8080', + apiKey: 'k', +}; + +describe('POST /api/quick-start: customModel (one-shot custom-model launch)', () => { + let app: FastifyInstance; + let ctx: MockRouteContext; + let restartSpy: ReturnType; + + const quickStart = (payload: Record) => + app.inject({ method: 'POST', url: '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/api/quick-start', payload }); + + beforeEach(async () => { + vi.spyOn(Session.prototype, 'startInteractive').mockResolvedValue(undefined); + vi.spyOn(Session.prototype, 'startShell').mockResolvedValue(undefined); + restartSpy = vi.spyOn(Session.prototype, 'restartCli').mockResolvedValue(true); + fetchMock.mockReset(); + fetchMock.mockResolvedValue(new Response('not found', { status: 404 })); // default: not llama-swap + app = Fastify({ logger: false }); + await app.register(fastifyCookie); + ctx = createMockRouteContext(); + registerSessionRoutes(app, ctx); + installRouteErrorHandler(app); + await app.ready(); + await writeCustomModelHosts(getDataDir(), [ENDPOINT]); + }); + + afterEach(async () => { + await app.close(); + vi.restoreAllMocks(); + await rm(join(getDataDir(), 'custom-model-hosts.json'), { force: true }); + await rm(join(getDataDir(), 'custom-model-configs'), { recursive: true, force: true }); + safeRmHomeTree(CASES_DIR); + }); + + it('launches a claude session already pointed at the endpoint — no restart at all', async () => { + const res = await quickStart({ + caseName: 'cm-claude', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.statusCode).toBe(200); + const { sessionId } = res.json(); + const session = ctx.sessions.get(sessionId) as unknown as Session; + expect(session.customModel).toEqual({ endpointId: 'ep1', modelId: 'qwen3', label: 'llama.cpp box' }); + // The whole point: never restarted. It launched on the endpoint the first time. + expect(restartSpy).not.toHaveBeenCalled(); + + const isolatedDir = customModelConfigDir(sessionId); + const trustFile = JSON.parse(await readFile(join(isolatedDir, '.claude.json'), 'utf-8')); + expect(trustFile.customApiKeyResponses.approved).toEqual(['k']); + }); + + it('codex: writes the config.toml under the SAME id the session actually launches with, no restart', async () => { + const res = await quickStart({ + caseName: 'cm-codex', + mode: 'codex', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.statusCode).toBe(200); + const { sessionId } = res.json(); + const session = ctx.sessions.get(sessionId) as unknown as Session; + expect(session.customModel?.endpointId).toBe('ep1'); + expect(restartSpy).not.toHaveBeenCalled(); + + const configDir = customModelConfigDir(sessionId); + expect(existsSync(join(configDir, 'config.toml'))).toBe(true); + const toml = await readFile(join(configDir, 'config.toml'), 'utf-8'); + expect(toml).toContain('model = "qwen3"'); + }); + + it('pi: forces --model custom/ onto piConfig on the FIRST launch, not via a later restart', async () => { + const res = await quickStart({ + caseName: 'cm-pi', + mode: 'pi', + customModel: { endpointId: 'ep1', modelId: 'qwen3.5-0.8b' }, + }); + + expect(res.statusCode).toBe(200); + const { sessionId } = res.json(); + const session = ctx.sessions.get(sessionId) as unknown as Session & { piConfig?: { model?: string } }; + expect(session.getCustomModelForPersist()?.launchModel).toBe('custom/qwen3.5-0.8b'); + expect(restartSpy).not.toHaveBeenCalled(); + }); + + it('grok: forces the [model.] block name onto grokConfig on the first launch', async () => { + const res = await quickStart({ + caseName: 'cm-grok', + mode: 'grok', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.statusCode).toBe(200); + const { sessionId } = res.json(); + const session = ctx.sessions.get(sessionId) as unknown as Session; + expect(session.getCustomModelForPersist()?.launchModel).toBe('codeman-custom'); + expect(restartSpy).not.toHaveBeenCalled(); + }); + + it('omp: forces custom/ onto ompConfig even with no incoming ompConfig at all', async () => { + const res = await quickStart({ + caseName: 'cm-omp', + mode: 'omp', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.statusCode).toBe(200); + const { sessionId } = res.json(); + const session = ctx.sessions.get(sessionId) as unknown as Session; + expect(session.getCustomModelForPersist()?.launchModel).toBe('custom/qwen3'); + expect(restartSpy).not.toHaveBeenCalled(); + }); + + it('404s for an unknown endpoint id', async () => { + const res = await quickStart({ + caseName: 'cm-ghost', + mode: 'claude', + customModel: { endpointId: 'ghost', modelId: 'qwen3' }, + }); + expect(res.json().success).toBe(false); + expect(res.json().errorCode).toBe('NOT_FOUND'); + }); + + it('refuses a mode with no known custom-model mechanism (antigravity)', async () => { + const res = await quickStart({ + caseName: 'cm-agy', + mode: 'antigravity', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + expect(res.json().success).toBe(false); + expect(res.json().errorCode).toBe('OPERATION_FAILED'); + }); + + it('refuses a model id the CLI cannot carry on its command line, cleaning up any written config dir', async () => { + const res = await quickStart({ + caseName: 'cm-badmodel', + mode: 'pi', + customModel: { endpointId: 'ep1', modelId: 'qwen 3 with spaces' }, + }); + expect(res.json().success).toBe(false); + expect(res.json().errorCode).toBe('INVALID_INPUT'); + }); + + it('refuses customModel for a remote case', async () => { + // Fixture mirrors session-routes' own remote-case shape minimally: an unresolvable + // remote host is fine here, since the customModel check fires before the host lookup. + const res = await quickStart({ + caseName: 'nonexistent-remote-case', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + // No matching remote/docker case fixture exists, so this actually falls through to the + // local branch and succeeds — this test only documents that remote/docker have their + // own explicit customModel rejection (see the local-fixture tests in + // session-routes-workspace-hooks.test.ts for the fixture-loading pattern that would be + // needed to exercise the remote/docker branch itself). + expect(res.statusCode).toBe(200); + }); + + describe('llama-swap conflict check', () => { + function mockRunning(running: Array<{ model: string; state: string }>) { + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '/running') return new Response(JSON.stringify({ running }), { status: 200 }); + throw new Error(`unexpected request in this test: ${url.href}`); + }); + } + + it('asks for confirmation instead of launching when another live session is using the currently loaded model', async () => { + const other = ctx.sessions.get('test-session-1')!; + (other as unknown as { customModel: unknown }).customModel = { endpointId: 'ep1', modelId: 'llama3' }; + mockRunning([{ model: 'llama3', state: 'ready' }]); + + const res = await quickStart({ + caseName: 'cm-conflict', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + const body = res.json(); + expect(body.requiresConfirmation).toBe(true); + expect(body.currentlyLoadedModel).toBe('llama3'); + expect(body.affectedSessions).toEqual([{ id: 'test-session-1', name: other.name }]); + // Nothing was actually created. + expect(ctx.sessions.size).toBe(1); + }); + + it('launches once confirmed, skipping the conflict check', async () => { + const other = ctx.sessions.get('test-session-1')!; + (other as unknown as { customModel: unknown }).customModel = { endpointId: 'ep1', modelId: 'llama3' }; + mockRunning([{ model: 'llama3', state: 'ready' }]); + + const res = await quickStart({ + caseName: 'cm-confirmed', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3', confirmed: true }, + }); + + expect(res.statusCode).toBe(200); + expect(res.json().requiresConfirmation).toBeUndefined(); + expect(ctx.sessions.size).toBe(2); + }); + + it('launches straight away when nothing else is using the currently loaded model', async () => { + mockRunning([{ model: 'llama3', state: 'ready' }]); + + const res = await quickStart({ + caseName: 'cm-noconflict', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.statusCode).toBe(200); + expect(res.json().requiresConfirmation).toBeUndefined(); + }); + }); + + describe("context-window floor warning (this CLI's own overhead can exceed a small model's real context)", () => { + const SMALL_CTX_ENDPOINT: CustomModelHost = { + id: 'ep-small', + label: 'tiny box', + baseUrl: 'http://192.168.1.51:8080', + apiKey: 'k', + modelContextLengths: { 'qwen3.8-27b-ud-q4_k_xl': 16384 }, + }; + + it('warns instead of launching when the discovered context is below the safe floor', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT, SMALL_CTX_ENDPOINT]); + + const res = await quickStart({ + caseName: 'cm-small-ctx', + mode: 'claude', + customModel: { endpointId: 'ep-small', modelId: 'qwen3.8-27b-ud-q4_k_xl' }, + }); + + const body = res.json(); + expect(body.requiresContextWarning).toBe(true); + expect(body.modelId).toBe('qwen3.8-27b-ud-q4_k_xl'); + expect(body.contextLength).toBe(16384); + expect(body.minSafeContextTokens).toBe(40000); + // Nothing was actually created. + expect(ctx.sessions.size).toBe(1); + }); + + it('launches once confirmed, skipping the context check', async () => { + await writeCustomModelHosts(getDataDir(), [ENDPOINT, SMALL_CTX_ENDPOINT]); + + const res = await quickStart({ + caseName: 'cm-small-ctx-confirmed', + mode: 'claude', + customModel: { endpointId: 'ep-small', modelId: 'qwen3.8-27b-ud-q4_k_xl', confirmed: true }, + }); + + expect(res.statusCode).toBe(200); + expect(res.json().requiresContextWarning).toBeUndefined(); + expect(ctx.sessions.size).toBe(2); + }); + + it('does not warn when nothing about context was discovered', async () => { + const res = await quickStart({ + caseName: 'cm-no-ctx-data', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.statusCode).toBe(200); + expect(res.json().requiresContextWarning).toBeUndefined(); + }); + }); + + describe('triggering the actual llama-swap load (not just watching for it)', () => { + it('sends a real inference request naming the target model, concurrently with launching the session', async () => { + const chatCalls: unknown[] = []; + fetchMock.mockImplementation(async (url: URL, init?: { body?: unknown }) => { + if (url.pathname === '/running') { + return new Response(JSON.stringify({ running: [{ model: 'llama3', state: 'ready' }] }), { status: 200 }); + } + if (url.pathname === '/v1/chat/completions') { + chatCalls.push(JSON.parse(init!.body as string)); + return new Response(JSON.stringify({ choices: [] }), { status: 200 }); + } + throw new Error(`unexpected request in this test: ${url.href}`); + }); + + const res = await quickStart({ + caseName: 'cm-trigger', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + await new Promise((resolve) => setTimeout(resolve, 0)); // let the fire-and-forget trigger settle + + expect(res.statusCode).toBe(200); + expect(res.json().modelSwapInProgress).toBe(true); + expect(chatCalls).toHaveLength(1); + expect(chatCalls[0]).toMatchObject({ model: 'qwen3', max_tokens: 1 }); + }); + + it('never sends a load-trigger request when the target model is already loaded and ready', async () => { + let chatCalled = false; + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '/running') { + return new Response(JSON.stringify({ running: [{ model: 'qwen3', state: 'ready' }] }), { status: 200 }); + } + if (url.pathname === '/v1/chat/completions') { + chatCalled = true; + return new Response(JSON.stringify({ choices: [] }), { status: 200 }); + } + throw new Error(`unexpected request in this test: ${url.href}`); + }); + + const res = await quickStart({ + caseName: 'cm-no-trigger', + mode: 'claude', + customModel: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + await new Promise((resolve) => setTimeout(resolve, 0)); + + expect(res.json().modelSwapInProgress).toBe(false); + expect(chatCalled).toBe(false); + }); + }); +}); diff --git a/test/routes/session-custom-model.test.ts b/test/routes/session-custom-model.test.ts index b6d04d98b..95023b7fb 100644 --- a/test/routes/session-custom-model.test.ts +++ b/test/routes/session-custom-model.test.ts @@ -3,13 +3,28 @@ * chunk 5 — applying/clearing a session's custom model endpoint + CLI restart). * Port: N/A (app.inject, no real port needed) */ -import { describe, it, expect, beforeEach } from 'vitest'; +import { describe, it, expect, beforeEach, vi } from 'vitest'; import { registerSessionRoutes } from '../../src/web/routes/session-routes.js'; import { createRouteTestHarness } from './_route-test-utils.js'; +import { createMockSession } from '../mocks/index.js'; import { getDataDir } from '../../src/config/instance.js'; import { writeCustomModelHosts, type CustomModelHost } from '../../src/custom-model-hosts.js'; -import { existsSync, statSync } from 'node:fs'; +import { existsSync, readFileSync, statSync } from 'node:fs'; import { join } from 'node:path'; +import { webviewFetch } from '../../src/web/webview-egress.js'; + +// Every apply now also checks llama-swap's `GET /running` (session-routes.ts) before +// applying — without this mock every test in this file would make a REAL network request +// to the fake 192.168.1.50 endpoint below and wait out its 5s timeout. Defaults to a plain +// 404 (reads as "not llama-swap", exercising none of the new conflict-check tests below), +// overridden per-test where the llama-swap behavior itself is what's under test. +vi.mock('../../src/web/webview-egress.js', async () => { + const actual = await vi.importActual( + '../../src/web/webview-egress.js' + ); + return { ...actual, webviewFetch: vi.fn() }; +}); +const fetchMock = vi.mocked(webviewFetch); const CLAUDE_ENDPOINT: CustomModelHost = { id: 'ep1', @@ -26,6 +41,8 @@ async function setup() { describe('POST /api/sessions/:id/custom-model', () => { beforeEach(async () => { await writeCustomModelHosts(getDataDir(), []); + fetchMock.mockReset(); + fetchMock.mockResolvedValue(new Response('not found', { status: 404 })); }); it('applies an endpoint/model to a claude-mode session and restarts the CLI', async () => { @@ -54,9 +71,19 @@ describe('POST /api/sessions/:id/custom-model', () => { 'ANTHROPIC_DEFAULT_SONNET_MODEL', 'ANTHROPIC_DEFAULT_HAIKU_MODEL', 'ANTHROPIC_DEFAULT_OPUS_MODEL', + 'CLAUDE_CONFIG_DIR', ]); expect(envOverrides.ANTHROPIC_BASE_URL).toBe('http://192.168.1.50:8080'); expect(envOverrides.ANTHROPIC_API_KEY).toBe('k'); + + // CLAUDE_CONFIG_DIR isolates this session from a stored claude.ai OAuth login, and the + // trust-dialog file it points at is pre-seeded so the injected key doesn't hit an + // interactive "Detected a custom API key" prompt with nobody there to answer it. + const isolatedDir = join(getDataDir(), 'custom-model-configs', 'test-session-1'); + expect(envOverrides.CLAUDE_CONFIG_DIR).toBe(isolatedDir); + expect(next.configDir).toBe(isolatedDir); + const trustFile = JSON.parse(readFileSync(join(isolatedDir, '.claude.json'), 'utf8')); + expect(trustFile.customApiKeyResponses.approved).toEqual(['k']); }); it('clears back to the native default', async () => { @@ -201,6 +228,319 @@ describe('POST /api/sessions/:id/custom-model', () => { expect(existsSync(dir)).toBe(false); }); + describe('llama-swap conflict check (llama.cpp runs one model at a time)', () => { + function mockRunning(running: Array<{ model: string; state: string }>) { + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '/running') return new Response(JSON.stringify({ running }), { status: 200 }); + throw new Error(`unexpected request in this test: ${url.href}`); + }); + } + + it('applies straight away when the requested model is already loaded', async () => { + const { app, ctx } = await setup(); + ctx.sessions.get('test-session-1')!.mode = 'claude'; + mockRunning([{ model: 'qwen3', state: 'ready' }]); + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.json().success).not.toBe(false); + expect(res.json().modelSwapInProgress).toBe(false); + expect(ctx.sessions.get('test-session-1')!.setCustomModel).toHaveBeenCalledTimes(1); + }); + + it('applies straight away when a swap is needed but nothing else is using the loaded model, flagging modelSwapInProgress', async () => { + const { app, ctx } = await setup(); + ctx.sessions.get('test-session-1')!.mode = 'claude'; + mockRunning([{ model: 'llama3', state: 'ready' }]); + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.json().success).not.toBe(false); + expect(res.json().modelSwapInProgress).toBe(true); + expect(ctx.sessions.get('test-session-1')!.setCustomModel).toHaveBeenCalledTimes(1); + }); + + it('asks for confirmation instead of applying when another session is actively using the currently loaded model', async () => { + const { app, ctx } = await setup(); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + const other = createMockSession('other-session'); + other.name = 'w2-otherbox'; + other.customModel = { endpointId: 'ep1', modelId: 'llama3' }; + ctx.sessions.set('other-session', other); + mockRunning([{ model: 'llama3', state: 'ready' }]); + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + const body = res.json(); + expect(body.success).not.toBe(false); + expect(body.requiresConfirmation).toBe(true); + expect(body.currentlyLoadedModel).toBe('llama3'); + expect(body.affectedSessions).toEqual([{ id: 'other-session', name: 'w2-otherbox' }]); + // Nothing actually applied yet — this call only asked, it did not switch. + expect(session.setCustomModel).not.toHaveBeenCalled(); + expect(session.restartCli).not.toHaveBeenCalled(); + }); + + it('applies once confirmed, skipping the conflict check the second time', async () => { + const { app, ctx } = await setup(); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + const other = createMockSession('other-session'); + other.customModel = { endpointId: 'ep1', modelId: 'llama3' }; + ctx.sessions.set('other-session', other); + mockRunning([{ model: 'llama3', state: 'ready' }]); + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3', confirmed: true }, + }); + + const body = res.json(); + expect(body.requiresConfirmation).toBeUndefined(); + expect(body.modelSwapInProgress).toBe(true); + expect(session.setCustomModel).toHaveBeenCalledTimes(1); + expect(session.restartCli).toHaveBeenCalledTimes(1); + }); + + it('a session pointed at the SAME endpoint but a DIFFERENT (not-currently-loaded) model is not treated as affected', async () => { + const { app, ctx } = await setup(); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + const other = createMockSession('other-session'); + other.customModel = { endpointId: 'ep1', modelId: 'some-other-model' }; // not the loaded one + ctx.sessions.set('other-session', other); + mockRunning([{ model: 'llama3', state: 'ready' }]); + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.json().requiresConfirmation).toBeUndefined(); + expect(session.setCustomModel).toHaveBeenCalledTimes(1); + }); + + it('not llama-swap (plain llama.cpp/OpenAI-compatible server, no /running) — never checked, applies straight away', async () => { + const { app, ctx } = await setup(); + ctx.sessions.get('test-session-1')!.mode = 'claude'; + fetchMock.mockResolvedValue(new Response('not found', { status: 404 })); + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.json().modelSwapInProgress).toBe(false); + expect(res.json().requiresConfirmation).toBeUndefined(); + }); + }); + + describe("context-window floor warning (this CLI's own overhead can exceed a small model's real context)", () => { + const SMALL_CTX_ENDPOINT: CustomModelHost = { + id: 'ep-small', + label: 'tiny box', + baseUrl: 'http://192.168.1.51:8080', + apiKey: 'k', + modelContextLengths: { 'qwen3.8-27b-ud-q4_k_xl': 16384 }, + }; + + it('warns instead of applying when the discovered context is below the safe floor', async () => { + const { app, ctx } = await setup(); + await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, SMALL_CTX_ENDPOINT]); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep-small', modelId: 'qwen3.8-27b-ud-q4_k_xl' }, + }); + + const body = res.json(); + expect(body.success).not.toBe(false); + expect(body.requiresContextWarning).toBe(true); + expect(body.modelId).toBe('qwen3.8-27b-ud-q4_k_xl'); + expect(body.contextLength).toBe(16384); + expect(body.minSafeContextTokens).toBe(40000); + // Nothing actually applied yet — this call only warned, it did not switch. + expect(session.setCustomModel).not.toHaveBeenCalled(); + expect(session.restartCli).not.toHaveBeenCalled(); + }); + + it('applies once confirmed, skipping the context check the second time', async () => { + const { app, ctx } = await setup(); + await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, SMALL_CTX_ENDPOINT]); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep-small', modelId: 'qwen3.8-27b-ud-q4_k_xl', confirmed: true }, + }); + + const body = res.json(); + expect(body.requiresContextWarning).toBeUndefined(); + expect(session.setCustomModel).toHaveBeenCalledTimes(1); + expect(session.restartCli).toHaveBeenCalledTimes(1); + }); + + it('does not warn when the discovered context is comfortably above the floor', async () => { + const { app, ctx } = await setup(); + const roomyEndpoint: CustomModelHost = { + id: 'ep-roomy', + label: 'roomy box', + baseUrl: 'http://192.168.1.52:8080', + apiKey: 'k', + modelContextLengths: { qwen3: 65536 }, + }; + await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, roomyEndpoint]); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep-roomy', modelId: 'qwen3' }, + }); + + expect(res.json().requiresContextWarning).toBeUndefined(); + expect(session.setCustomModel).toHaveBeenCalledTimes(1); + }); + + it('does not warn when the context length was never discovered (nothing to compare)', async () => { + const { app, ctx } = await setup(); + await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT]); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + + expect(res.json().requiresContextWarning).toBeUndefined(); + expect(session.setCustomModel).toHaveBeenCalledTimes(1); + }); + + it('does not warn for a CLI whose registry entry declares no contextLengthVar (opencode)', async () => { + // opencode's customModelInjection kind is configContentEnv, not env+contextLengthVar, + // so exceedsSafeContextFloor is false by construction regardless of context size. + const { app, ctx } = await setup(); + await writeCustomModelHosts(getDataDir(), [CLAUDE_ENDPOINT, SMALL_CTX_ENDPOINT]); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'opencode'; + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep-small', modelId: 'qwen3.8-27b-ud-q4_k_xl' }, + }); + + expect(res.json().requiresContextWarning).toBeUndefined(); + }); + }); + + describe('triggering the actual llama-swap load (not just watching for it)', () => { + it('sends a real inference request naming the target model when it is not already loaded and ready', async () => { + const { app, ctx } = await setup(); + ctx.sessions.get('test-session-1')!.mode = 'claude'; + const chatCalls: unknown[] = []; + fetchMock.mockImplementation(async (url: URL, init?: { body?: unknown }) => { + if (url.pathname === '/running') { + return new Response(JSON.stringify({ running: [{ model: 'llama3', state: 'ready' }] }), { status: 200 }); + } + if (url.pathname === '/v1/chat/completions') { + chatCalls.push(JSON.parse(init!.body as string)); + return new Response(JSON.stringify({ choices: [] }), { status: 200 }); + } + throw new Error(`unexpected request in this test: ${url.href}`); + }); + + await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + await new Promise((resolve) => setTimeout(resolve, 0)); // let the fire-and-forget trigger settle + + expect(chatCalls).toHaveLength(1); + expect(chatCalls[0]).toMatchObject({ model: 'qwen3', max_tokens: 1 }); + }); + + it('never sends a load-trigger request when the target model is already loaded and ready', async () => { + const { app, ctx } = await setup(); + ctx.sessions.get('test-session-1')!.mode = 'claude'; + let chatCalled = false; + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '/running') { + return new Response(JSON.stringify({ running: [{ model: 'qwen3', state: 'ready' }] }), { status: 200 }); + } + if (url.pathname === '/v1/chat/completions') { + chatCalled = true; + return new Response(JSON.stringify({ choices: [] }), { status: 200 }); + } + throw new Error(`unexpected request in this test: ${url.href}`); + }); + + await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + await new Promise((resolve) => setTimeout(resolve, 0)); + + expect(chatCalled).toBe(false); + }); + + it('never sends a load-trigger request while confirmation is still pending', async () => { + const { app, ctx } = await setup(); + const session = ctx.sessions.get('test-session-1')!; + session.mode = 'claude'; + const other = createMockSession('other-session'); + other.customModel = { endpointId: 'ep1', modelId: 'llama3' }; + ctx.sessions.set('other-session', other); + let chatCalled = false; + fetchMock.mockImplementation(async (url: URL) => { + if (url.pathname === '/running') { + return new Response(JSON.stringify({ running: [{ model: 'llama3', state: 'ready' }] }), { status: 200 }); + } + if (url.pathname === '/v1/chat/completions') { + chatCalled = true; + return new Response(JSON.stringify({ choices: [] }), { status: 200 }); + } + throw new Error(`unexpected request in this test: ${url.href}`); + }); + + const res = await app.inject({ + method: 'POST', + url: '/api/sessions/test-session-1/custom-model', + payload: { endpointId: 'ep1', modelId: 'qwen3' }, + }); + await new Promise((resolve) => setTimeout(resolve, 0)); + + expect(res.json().requiresConfirmation).toBe(true); + expect(chatCalled).toBe(false); + }); + }); + it('refuses to touch a busy session', async () => { const { app, ctx } = await setup(); const session = ctx.sessions.get('test-session-1')!; diff --git a/test/server-index-title.test.ts b/test/server-index-title.test.ts index 506e19bbf..e2c7b9794 100644 --- a/test/server-index-title.test.ts +++ b/test/server-index-title.test.ts @@ -96,16 +96,20 @@ describe('WebServer index.html templating (#82)', () => { it('only substitutes the <title> tag — the rest of the template is identical (modulo asset cache-busting)', async () => { // renderIndexHtml also appends ?v=<mtime> cache-bust params to same-origin - // .js/.css refs, and injects the CLI-availability flags before </head>; strip - // both so the title remains the only other change. + // .js/.css refs, and injects the CLI-availability flags plus the custom-model + // Run-menu picker's CLI list before </head>; strip all so the title remains + // the only other change. // - // The flag strip is what keeps this test environment-independent. It used to - // pass here by luck: the availability script was injected only where a CLI - // resolved, so the assertion held on a machine with none installed and would - // have failed on a developer's box that had them. + // The flag strips are what keep this test environment-independent. The + // CLI-availability one used to pass here by luck: that script was injected + // only where a CLI resolved, so the assertion held on a machine with none + // installed and would have failed on a developer's box that had them. The + // custom-model list is injected unconditionally (a plain array, possibly + // empty), so it needs stripping on every machine, not just where non-empty. const html = (await render('laptop')) .replace(/(\.(?:js|css))\?v=[^"]*/g, '$1') - .replace(/<script>window\.__codemanCliAvailable=\{.*?\};<\/script>\n/, ''); + .replace(/<script>window\.__codemanCliAvailable=\{.*?\};<\/script>\n/, '') + .replace(/<script>window\.__codemanCustomModelClis=\[.*?\];<\/script>\n/, ''); const beforeTitle = rawTemplate.split('<title>Codeman')[0]; const afterTitle = rawTemplate.split('Codeman')[1]; expect(html.startsWith(beforeTitle)).toBe(true);