Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
25fae9a
feat(custom-model): generate Run-menu entries from saved endpoint pro…
opticon454 Sep 15, 2026
98d26e1
docs(wiki): document Custom Model Endpoints and the Run-menu picker
opticon454 Sep 15, 2026
fed6582
fix(test): strip the custom-model Run-menu picker's injected script too
opticon454 Sep 15, 2026
60e1bd5
fix(custom-model): act on the draft review — unparseable onclick, unw…
opticon454 Sep 15, 2026
5a9ff07
feat(custom-model): ask which model on launch when an endpoint has mo…
opticon454 Sep 16, 2026
9a9e542
fix(custom-model): bound the model-picker dialog's height and make it…
opticon454 Sep 16, 2026
409a6e6
fix(custom-model,toast): surface the real apply error, and make error…
opticon454 Sep 16, 2026
5c25a52
fix(custom-model): wait for a freshly launched session to go idle bef…
opticon454 Sep 16, 2026
0e8b198
fix(custom-model): isolate Claude config dir and inject real context …
opticon454 Sep 16, 2026
97464bf
fix(custom-model): pre-approve the injected API key in the isolated C…
opticon454 Sep 16, 2026
25f22b9
test(custom-model): update session-custom-model route test for CLAUDE…
opticon454 Sep 16, 2026
bcebc81
feat(custom-model): detect llama-swap model conflicts before switching
opticon454 Sep 16, 2026
fbee1b2
docs(changeset): add changeset for the Run-menu custom-model picker PR
opticon454 Sep 16, 2026
f865f74
feat(custom-model): launch directly on the endpoint, no restart, for …
opticon454 Sep 16, 2026
83033b4
fix(custom-model): show a status toast during Claude's native-boot-th…
opticon454 Sep 16, 2026
2936ba6
fix(custom-model): replace the native confirm() popup with an in-app …
opticon454 Sep 16, 2026
01b32ee
fix(custom-model): move the switching/loading status to a centred banner
opticon454 Sep 16, 2026
0929694
fix(custom-model): actually trigger the llama-swap load, not just wat…
opticon454 Sep 16, 2026
0af233c
fix(custom-model): poll llama-swap readiness every 1s, check immediat…
opticon454 Sep 16, 2026
55dae31
feat(custom-model): estimate model load time from its discovered size
opticon454 Sep 16, 2026
7bbe408
feat(custom-model): live countdown on the loading banner; timeout is …
opticon454 Sep 16, 2026
9937102
fix(custom-model): stop trusting /props's n_ctx, parse the real conte…
opticon454 Sep 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .changeset/run-menu-custom-model-picker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
"aicodeman": minor
---

**Custom model endpoints: Run-menu picker, and hardening from real llama-swap validation** (#430, follow-up to #393's HTTP-API-only cut). With **Custom model endpoints** on (App Settings → Models) and at least one saved endpoint carrying a discovered model, the Run dropdown grows a **Custom Endpoints** section generated live off the CLI registry's own `capabilities.customModelInjection` — one entry per (harness that can redirect to a custom endpoint, saved endpoint). Picking one launches that harness and applies the endpoint to it; with two or more discovered models a small, scrollable dialog asks which one first, the endpoint's `defaultModelId` marked but never auto-chosen. Endpoints also now re-discover themselves automatically every 5 minutes in the background, one unreachable endpoint never blocking the others.

Everything below was found and fixed against a **real llama-swap server**, not just unit tests:

- **Session-busy false refusal.** A freshly launched CLI reports itself `busy` for its own startup (spinner, workspace-trust check) well before the apply call would reach it, and the apply route correctly refuses to restart a session mid-turn — indistinguishable from a fresh boot. The picker now waits for the new session to go idle (bounded at 20s, never an error on timeout) before applying.
- **Errors and confirmations you can actually read.** Toasts now default to sticky with a close button (errors always were meant to stay, but a fixed 3s timer silently hid them); a failed apply's real server-side reason (not a generic message) reaches the toast.
- **"Both claude.ai and ANTHROPIC_API_KEY set" warning.** A custom-model Claude session now runs with an isolated `CLAUDE_CONFIG_DIR` (empty, no real credentials in it) so the injected API key never coexists with a stored OAuth login — `projects` is symlinked back to the real config dir so the response viewer/subagent windows/Read My Mind keep working. That isolated, otherwise-empty directory has none of a real profile's prior "Detected a custom API key — use it?" approvals either, which would otherwise re-ask on *every* launch with nobody at a TTY to answer (and silently refuse the key on its own default); the apply step now pre-seeds that exact approval field the same way answering the prompt once by hand would.
- **Context-window overflow.** Claude Code assumes a large default context window for a model id it doesn't recognize and never compacts, so a real local model's much smaller context silently overflowed (confirmed live: a stock ~33.7K-token system prompt against a 16384-token model). Discovery now also learns each model's real context length from llama.cpp/llama-swap's `GET /props?model=`, but **only** for a model llama-swap's own `/v1/models` response already reports loaded — never an unloaded one, since asking about one risks triggering an actual, slow, GPU-swapping load as a side effect of read-only discovery — and applies it as `CLAUDE_CODE_MAX_CONTEXT_TOKENS`.
- **The real root cause of "it still says opus, not my model."** llama.cpp runs exactly one model at a time; llama-swap unloads and reloads it on demand, which can take anywhere from a few seconds to well over a minute — long enough that a session mid-swap is indistinguishable from one that never left the native backend. Applying a selection now checks llama-swap's own `GET /running` first (feature-detected; a plain llama.cpp/OpenAI-compatible server has no such endpoint and is never checked); if switching would unload a model **another live session is actively using**, the apply is refused with a warning naming that session instead of silently switching, and a confirmation retry proceeds anyway. Either way, a sticky "loading model…" toast now covers the actual swap window until llama-swap reports the target model ready, so a prompt sent mid-swap reads as "loading," never as silence or an answer from whatever was loaded a moment before.

Remote (SSH) and Docker sessions are refused for now (400) — their restart reattaches the durable remote/in-container tmux rather than relaunching the agent.

**One more, from watching it launch live: opencode, Codex, Gemini, Pi, Grok, DeepSeek and OMP now launch directly on the endpoint, with no restart at all.** Picking one of these seven from the Run-menu picker used to launch natively first, wait for it to settle, then restart it in place with the endpoint applied — a deliberate two-step design, but visibly a native boot immediately followed by a second one, worst on a CLI whose TUI fully reinitializes on a restart (confirmed live on Codex). `POST /api/quick-start` now accepts a `customModel` field and computes the same injection *before* the session exists, launching straight onto the endpoint the first time — no visible relaunch, and it also runs the same llama-swap conflict check (warns before unloading a model another live session is using) at create time. Claude still uses the original launch-then-restart path for now (its own `--resume`-based restart is far less jarring, and `runClaude()`'s multi-tab and docker-config-drift-retry logic make folding it into the one-shot path separate work).
2 changes: 1 addition & 1 deletion CLAUDE.md

Large diffs are not rendered by default.

51 changes: 51 additions & 0 deletions docs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,57 @@ All four enforce session ownership in multi-user mode; a foreign session id
answers `404 NOT_FOUND` (no existence leak), and profiles of two owners of the
same directory are distinct by construction.

## Custom Model Endpoints

Points a session's harness at a user-configured OpenAI-compatible endpoint —
local (llama.cpp, vLLM, DGX Spark) or cloud (Azure AI Foundry, OpenRouter) —
instead of its native cloud backend, gated by the opt-in
`customModelEndpointsEnabled` setting (default OFF). Endpoints are
machine-level infra, like remote/docker hosts: writes are admin-only in
multi-user mode. Design: [`custom-model-endpoints-plan.md`](custom-model-endpoints-plan.md);
user guide: [`custom-model-endpoints.md`](custom-model-endpoints.md).

- `GET /api/v1/model-endpoints` -> `CustomModelHost[]`, an unwrapped bare
array like every other list route (still riding the standard `{success,
data}` envelope on the wire — unwrap it the same way). Answers `[]` for a
non-admin in multi-user mode. `apiKey` is never returned; `apiKeySet:
boolean` reports whether one is stored, so a client can render "unchanged
if left blank" without ever holding the real value.
- `POST /api/v1/model-endpoints` with `{ id, label, baseUrl, apiKey?,
authStyle?, defaultModelId? }` creates one. `id` must match
`^[a-zA-Z0-9_-]+$`; `authStyle` is `bearer` (default) or `api-key`, never
both (a real server hung indefinitely when sent both headers on one
request); `baseUrl` must be `http(s)`, carry no embedded credentials, and
is refused if it points at (or resolves to) a link-local or
cloud-metadata address. `409 ALREADY_EXISTS` on a duplicate id.
- `PUT /api/v1/model-endpoints/:id` updates one. An **absent** `apiKey`
keeps the stored one rather than clearing it — the client never receives
the real value to resend deliberately unchanged, so omission is the only
way to say "leave it alone"; there is no way to clear a key back to unset
this way. `defaultModelId`, when set, must be one of that endpoint's own
`models` (`400 INVALID_INPUT` otherwise).
- `DELETE /api/v1/model-endpoints/:id` removes one.
- `POST /api/v1/model-endpoints/:id/discover-models` fetches the endpoint's
own `GET /v1/models` and stores the result as `models`, updating
`lastDiscoveredAt`. A `defaultModelId` that no longer appears in the fresh
list is dropped rather than carried forward invalid. Failures answer
`502 OPERATION_FAILED` with the underlying connection error, or a named
egress refusal if the resolved address turned out to be blocked. The same
refresh also runs automatically for every saved endpoint every 5 minutes
in the background (`refreshAllCustomModelHosts()`, `custom-model-routes.ts`,
started from `server.ts`), so there is no route for triggering "refresh
all" — one endpoint being unreachable on a cycle never blocks the others.
- `POST /api/v1/sessions/:id/custom-model` with `{ endpointId, modelId } |
{ clear: true }` applies (or clears) the session's selection and
**restarts the session's CLI process in place** — every supported harness
reads its endpoint config at process start, never per turn, so there is
no live hot-swap. A Claude session resumes its existing conversation
across the restart; pi/omp/grok additionally get a forced `--model`/`-m`
value, since for those three the config file alone does not select it.
`400 INVALID_INPUT` for a remote (SSH) or Docker session — both restart
their agent differently under the hood, and applying to one would report
success while changing nothing.

## Voice dictation

Browser dictation transcribed through this server's Claude Code login, i.e. the
Expand Down
8 changes: 8 additions & 0 deletions docs/custom-model-endpoints-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,14 @@ extra per-model configuration on Codeman's side at all.

### 4. Toolbar UI

> **Superseded.** This section describes the toolbar-button design as originally
> planned. What actually shipped is a Run-menu picker instead: one generated entry
> per (capable harness, saved endpoint) pair directly in the existing `#runModeMenu`
> dropdown, rather than a separate `#customModelBtn`/`#customModelMenu` surface. See
> [`docs/custom-model-endpoints.md`](custom-model-endpoints.md#the-run-menu-picker)
> for the current design; the sections below (session-restart mechanics, security)
> remain accurate regardless of which UI calls the underlying route.

- New header/toolbar button (e.g. `#customModelBtn`, `btn-toolbar
btn-custom-model`), marker-hidden by default (`btn-custom-model--hidden`)
and revealed by `applyHeaderVisibilitySettings()` only when
Expand Down
Loading