Show and edit where a local model's layers run

[androidand]

Show and edit where a local model's layers run

Labels. This repo is public, so fleet hosts and installed models are named by
capability and shape rather than by hostname or model id; the mapping lives in the
private companion repo (docs-skein/fleet-labels.md). Host A is a 24 GB RDNA3
workstation. M1–M6 are that fleet's installed models. All measurements are
verbatim.

Why

An agent session on host A ran at 1.2 tok/s for its whole duration
(2026-08-12). The model was serving 26 of its 66 layers from the CPU because its
cmd pinned --n-gpu-layers 40. Correcting it gave 32.4 tok/s — 27×.

Nothing in opencode said so. The sidebar showed 1.2 t/s, a context bar, a VRAM
bar, and gfx1100 · default. The operator's only signal that anything was wrong
was the speed itself, and the only available reading of that was "local models are
slow." Diagnosing it took reading the GGUF layer count on the host by hand.

The host already knew and already said so. GET /api/fit/{model} returned
run_mode: "cpu_offload" with host_resident_mb: 7165 — the whole diagnosis, in
a field opencode fetches and drops. opencode polls /api/fit for max_safe_ctx
and fit_level; run_mode and host_resident_mb go unread.

The editing surface was scoped out on a false premise. add-gpu-tuning-ui is
complete, and its Non-goals say:

Per-model flag editing beyond what /api/config/models/:id PATCH already
supports (ctx size / offload live there already).

Two problems. The route is /api/models/config/{id}, not
/api/config/models/:id — the note is stale. And "live there already" describes
the server API, not any client surface: ctx_size did get one
(DialogModelCtx, reachable by clicking Context), n_gpu_layers never did. So the
tuning dialog offers auto-tune, flash attention, parallel slots, and MTP — four
host-wide knobs — and no way to change the per-model setting that cost 27×. Its own
footer even warns "explicit cmd flags always win", so no amount of correct tuning
can rescue a bad -ngl.

This is a repeating pattern rather than an oversight. llama-skein's
add-auto-hybrid-placement records it: offload auto-application "was deliberately
deferred to clients (add-model-offload-tuning tasks 9–10, both [D]), and no
client ever shipped it." The server computes the right answer, the client surface is
deferred, and the deferral is never picked up.

It is also not one bad model. Read the same evening, three of five remaining host A
models were run_mode: cpu_offload with VRAM to spare, and the worst of them
graded fit_level: "perfect" — see llama-skein flag-under-offloaded-models,
which fixes the server-side grading and adds the under_offloaded flag this change
displays.

A routing guard this repo already ships is silently disabled by the same
cause. isHostPaced() (packages/opencode/src/local/placement.ts:104-107)
applies a 200,000-point HOST_PACED_PENALTY to keep sub-agents off host-paced
models, keyed on placement.perf_class being cpu-bound-hybrid or cpu-only.
llama-skein returns perf_class: "native-gpu" unconditionally for any model with
pinned placement flags (internal/placement/placement.go:153). Verified on host A:
native-gpu reported alongside run_mode: "cpu_offload", 7165 MB host-resident,
1.2 tok/s. So the penalty never fires for pinned models — the guard is defeated by
exactly the configuration that produces the models it guards against, and the
sub-agent routing this repo's own comment warns about ("a subagent silently lands
on a model ~90x slower") happens anyway. The fix belongs in llama-skein
(flag-under-offloaded-models task 16), not in a client-side workaround on top of
the existing one.

What Changes

• Surface run_mode in the sidebar. When a local model is not fully
  GPU-resident, say so where the speed is already shown, with the host-resident
  amount. A cpu_offload verdict on a model whose weights would fit VRAM is a
  mechanical red flag and needs no new server work to detect — run_mode and
  host_resident_mb are already on /api/fit.
• Consume under_offloaded when the host offers it. Prefer the server's flag
  over a client-side inference, and degrade to the run_mode +
  host_resident_mb heuristic against hosts that predate it.
• A per-model placement control, modelled on the existing DialogModelCtx
  rather than a new pattern: view the current placement and edit
  n_gpu_layers via PATCH /api/models/config/{id}, which already accepts it.
• Offer "remove the pin" as the primary action. Where the host reports that
  the counterfactual plan is full GPU residency, clearing the flag so llama-skein
  computes placement is the durable fix; a raised number goes stale on the next
  model or card. Setting an explicit value stays available for deliberate pins.
• Correct the stale Non-goal in add-gpu-tuning-ui so the wrong route and
  the "offload lives there already" claim stop being cited as settled.

Capabilities

Modified Capabilities

• local-providers: placement is visible in the sidebar and editable per model.

Non-Goals

• Not client-side placement maths. llama-skein owns the decision and the
  counterfactual; opencode displays what /api/fit reports and writes back what
  the operator chooses. Reimplementing capacity arithmetic client-side is the
  failure add-model-offload-tuning already warned against.
• Not automatic correction. opencode SHALL NOT silently rewrite a placement
  flag on a working model. This mirrors llama-skein's non-goal.
• Not MoE offload tuning (--n-cpu-moe / --cpu-moe). The contract accepts
  those, but the host A failure is dense-layer placement; MoE controls are a
  follow-up once this pattern is proven.
• Not a change to placement-based routing. ctx-aware-subagent-placement
  owns the scorer; note that its fit_level×1000 term is affected by the
  llama-skein grading fix and needs its own follow-up.

Open Questions

• Where the control lives. Folding placement into DialogTuning mixes
  host-wide tuning with per-model settings, and that conflation is part of why
  this was missed. Extending DialogModelCtx into a per-model dialog covering
  both ctx and placement is likelier right — but "Context" is a poor name for it,
  and renaming a surface operators know has its own cost.
• How loud the warning should be. A genuinely hybrid model on a small card is
  correct and must not nag. The signal should key on under_offloaded
  (avoidable) rather than run_mode (which is also true for correct hybrids).
• Restart cost. A placement patch reloads the model and drops the session's
  loaded state. DialogModelCtx already faces this; whatever it does should be
  matched, and the cost stated before the write, not after.

Impact

• packages/tui/src/local/llama-skein/gen/, packages/opencode/src/local/llama-skein/gen/
  — regenerated for under_offloaded.
• packages/tui/src/feature-plugins/sidebar/context.tsx — placement indicator.
• packages/tui/src/component/dialog-model-ctx.tsx or a sibling — the control.
• openspec/changes/add-gpu-tuning-ui/proposal.md — corrected Non-goal.
• Depends on llama-skein flag-under-offloaded-models for under_offloaded;
  the run_mode fallback ships without it.

Discoveries

• HOST_PACED_PENALTY (placement.ts:104-107) never fires for pinned models because llama-skein returns perf_class=native-gpu unconditionally for ModeCustom. The guard against routing subagents to CPU-bound models is disabled by exactly the config that creates them. Fix belongs in llama-skein, not a second client workaround.
• The read-modify-write cycle this change depends on is currently unsafe: GET /api/models/config/{id} returns --port resolved (5803) where config stores ${PORT}. And there is no way to remove n_gpu_layers via the patch contract. Both are llama-skein tasks 17-18.

Tasks

Tasks: per-model-placement-controls

The sidebar indicator (tasks 2–4) ships without any llama-skein change, using
run_mode + vram_required_mb. Only the under_offloaded preference and the
"remove the pin" wording depend on llama-skein flag-under-offloaded-models.

• 1. Resolve the "Where the control lives" Open Question. Decided 2026-08-14,
  see design.md. D1: extend DialogModelCtx; do not touch DialogTuning.
  The signatures decide it — DialogTuning(props: {providerID}) has no
  modelID and patches the host-wide /api/tuning, so a per-model write there
  is a category error, not a missing parameter. DialogModelCtx(props: {providerID, modelID}) already reads getModelFit (line 60, whose response
  already carries run_mode/host_resident_mb/placement), already patches
  with an abort/stale guard (line 142), and already faces the same reload cost.
  Retitle from Context — ${modelName()} to ${modelName()} with "Context"
  and "Placement" as sections. D2: the indicator keys on under_offloaded,
  never run_mode alone. D3: reuse the existing reload-cost convention.

• 2. Read run_mode, host_resident_mb, and vram_required_mb from the
  existing /api/fit poll in packages/tui/src/feature-plugins/sidebar/context.tsx.
  Fold into the current hardware/tuning poll — no new loop — reusing the
  existing cancelled-flag/abort guard.
  Validation: cd packages/tui && bun run typecheck

• 3. Render the placement indicator per the spec: fire only on avoidable
  offload, name the host-resident amount, stay silent for genuine hybrids and
  fully-resident models. Also per D1: retitle DialogModelCtx from
  Context — ${modelName()} to ${modelName()} with "Context" and "Placement"
  sections, and make the sidebar VRAM/placement area a second click target
  opening the same dialog (the Context label keeps working, so nobody
  relearns anything).
  Validation: cd packages/tui && bun run typecheck; manual check against
  host A M2 (must warn) and M4
  (must stay silent).

• 4. Verify the indicator against all five host A models recorded in
  llama-skein flag-under-offloaded-models: warn on
  M2 and M3;
  stay silent on M4, M5, and
  M6 — the last two are pinned at -ngl 40 yet fully resident,
  so a check keyed on the pinned number rather than the outcome fails here.
  Validation: recorded per-model verdict; two warn, three silent.

• 5. Build the placement control on the surface chosen in task 1: show current
  placement, offer "remove the pin" as primary where the host reports full
  residency is achievable, allow an explicit value, and state the reload cost
  before the write. Reuse DialogModelCtx's patch + abort/stale handling
  rather than a new pattern.
  Blocked on two llama-skein API defects found 2026-08-12 while fixing host A
  by hand — do not work around either client-side:
  (a) there is no way to remove n_gpu_layers via
  ConfigModelPatchRequest; every value overwrites, and 0 writes
  --n-gpu-layers 0 (all layers on CPU). So the primary action is not
  expressible yet. llama-skein flag-under-offloaded-models task 17.
  (b) GET /api/models/config/{id} returns --port resolved (5803)
  where the config stores ${PORT}, so the read-modify-write cycle this
  control depends on would hardcode a dynamically allocated port and silently
  break the model later. llama-skein task 18. Patching the full cmd string is
  the only current removal route and is unsafe for this reason.
  Validation: cd packages/tui && bun run typecheck; a round-trip test proving
  ${PORT} survives an edit.

• 6. Confirm the patch path end to end against a live host: clearing the pin
  removes --n-gpu-layers from the model's cmd and the model returns
  run_mode: "gpu" on reload. PATCH /api/models/config/{id} scopes to a
  single model — verify no sibling entry changed, by diffing the host's
  config-history snapshot.
  Validation: config-history diff shows exactly one changed line.

• 7. Regenerate the llama-skein TS client once under_offloaded is in the
  published spec, and prefer it over the fallback heuristic. Commit the regen
  separately. Generated clients are never hand-edited.
  Validation: grep -r "under_offloaded" packages/opencode/src/local/llama-skein/gen packages/tui/src/local/llama-skein/gen

• 8. Correct the stale Non-goal in openspec/changes/add-gpu-tuning-ui/proposal.md:
  the route is /api/models/config/{id}, and "ctx size / offload live there
  already" described the server API, not a client surface — ctx_size got one,
  n_gpu_layers did not. Leave the completed tasks alone; annotate the Non-goal
  as superseded by this change.
  Validation: the Non-goal names the correct route and points here.

• 9. Open a follow-up for ctx-aware-subagent-placement: its scorer's dominant
  fit_level×1000 term (packages/opencode/src/local/placement.ts:81-86,
  :189) changes meaning once llama-skein stops grading wasted VRAM
  favourably. Do not change the scorer here.
  Validation: specsync note -change ctx-aware-subagent-placement "<finding>"

• 10. Record the HOST_PACED_PENALTY interaction, which this investigation
  exposed as a live routing bug rather than a future concern.
  isHostPaced() (placement.ts:104-107) keys on
  placement.perf_class ∈ {cpu-bound-hybrid, cpu-only}, but llama-skein
  returns native-gpu unconditionally for any pinned-placement model
  (internal/placement/placement.go:153). Verified on host A:
  perf_class: "native-gpu" alongside run_mode: "cpu_offload", 7165 MB
  host-resident, 1.2 tok/s. The 200,000-point penalty never fires for pinned
  models — the guard is disabled by exactly the configuration that produces the
  models it guards against. The fix is llama-skein
  flag-under-offloaded-models task 16; this repo must not paper over it
  client-side.
  Validation: once that lands, a pinned host-paced model on host A is penalised;
  confirm no double penalty against the placement-aware fit_level (its task 17).

• 11. Repo validation: bun run typecheck in packages/opencode and
  packages/tui; cd packages/opencode && bun test test/local.
  Validation: both typechecks and the local test suite pass.

• 12. Low-confidence follow-up, investigate before speccing: the sidebar VRAM
  readout was observed reading 0.1 / 24.0 GB against a 91% bar. The
  /api/hardware contract is intact (vram.used_mb is present and matched
  rocm-smi exactly), so the likely cause is a torn render across the 30 s
  poll — MemBreakdown taking the modelMb == 0 branch while the bar kept a
  stale percent. Reproduce before deciding whether it is a real defect.
  Validation: reproduced or ruled out, with the finding recorded.

Plan changes

1 done

Blocked by

• androidand/llama-skein#24
• androidand/llama-skein#23