[stevbev]

Description

This branch adds Custom — a fourth engine kind that fronts user-configured
external OpenAI-compatible endpoints (vLLM, llama.cpp, mtplx, strata, hosted APIs)
— and makes the cluster-status surfaces report real engine state.

Before this, PAIR only knew about the three installable managed engines. The
menu-bar / tray cluster-status window always rendered "Engines (n/3)": it
fabricated a placeholder for every manifest engine type, so a node with
nothing installed read "0/3", a node running a custom endpoint read "0/3" as
well, and custom engines were invisible everywhere. The custom proxy plane,
its configuration storage, relay, probe, and UI layers are all new in this
branch; the tray line and the TUI node-detail screen now count and list what
a node actually has. It also carries two macOS packaging fixes needed for
multi-node use: the Local Network permission declaration and a monochrome
menu-bar icon.

Release intent

Changelog title

Custom inference engines; truthful engine counts in the cluster-status window

Changelog body

You can now add external OpenAI-compatible endpoints (vLLM, llama.cpp servers, hosted APIs) as Custom engines on any node: add, edit, and remove them from the desktop app, set the shared custom-proxy port, and see health and model chips per endpoint. The TUI lists Custom endpoints on the node detail screen. A Custom endpoint's API key stays on the node that owns it.

• The menu-bar / tray cluster-status window counts the engines a node actually has — installed managed engines plus configured custom endpoints, online over total — instead of always showing "Engines (n/3)" and ignoring custom engines.
• macOS 14+ builds now declare Local Network usage, so peer discovery works instead of silently failing; the menu-bar icon renders as a proper monochrome template image.

Bumps

• services: minor
• nvpair-cluster-manager: none
• nvpair-engine-manager: minor
• nvpair-errors: none
• nvpair-job-scheduler: patch
• nvpair-manual-nodes: minor
• nvpair-node-info: none
• nvpair-node-scanner: patch
• nvpair-node-settings: none
• nvpair-proxy: minor
• nvpair-tui: minor
• nvpair-ui-broker: minor
• nvpair-workload-manager: none

Bump rationale: broker, proxy, engine-manager, manual-nodes, and tui each ship
new user-visible IPC/HTTP behavior (custom:list/add/remove/refresh,
custom/get-port/set-port, the custom facade and its routing, the custom model
union, the probe source, the TUI section). job-scheduler and node-scanner have
no source change beyond tests; they carry patch because their compiled engines
table / TXT parse set now includes the custom kind (scheduler:get-status
gains a custom entry; node-record TXT gains the cu= key).

Resolves

#182
#165
#77
#39
#24
#2

Scope

Included: the Custom engine kind in the shared engine registry and mDNS TXT
key set; the custom-proxy facade (terminal forwarding, per-model routing,
key injection on the owning node's last hop); the broker-owned endpoint
config store (0600 file), the peer relay, and the configurable facade port;
engine-manager's custom model union and carrier; manual-nodes as the probe
cache; the desktop service bridge, management UI, and the tray/TUI count and
listing fixes; generated service docs, specs, and user docs.

Excluded: model operations and install lifecycle for Custom (unmanaged by
design); any push feed for custom health (the tray polls at 30s while
visible); changes to managed-engine behavior beyond skipping unmanaged kinds
in launcher-shaped consumers; services/versions.json / CHANGELOG.md
(automation-owned).

Validation

Environment: 3 macOS arm64 machines; branch tip 6a41a9e+style commit, based
on develop (039f9fe). 1 Windows 11 amd64 machine running 1.0 release.

From desktop/:

• npm run typecheck — passed.
• npm run lint — passed, 0 errors / 0 warnings after applying.
• npm run test:unit — 51 files / 384 tests passed.
• npm run dead-code:check — passed.
• npm run service-contracts:check — passed (doc fresh).
• node scripts/spdx-headers.mjs — passed.

From services/, go test ./... in the affected modules:
shared/engines, shared/noderec, nvpair-ui-broker, nvpair-proxy,
nvpair-manual-nodes, nvpair-node-scanner, nvpair-tui,
nvpair-job-scheduler — all passed.
nvpair-engine-manager — one failure, TestUninstall_TerminatesRunningInstance,
which reproduces identically on the develop base commit on this machine; it
is a pre-existing environment-dependent executor test, not introduced here.
The new cross-process suites in services/tests (advertise, interop, peer
ingress over mTLS, remote edit) require a paired multi-node harness and are
left to CI.

Manual UAT on macOS with one custom endpoint and no managed engines: the
tray window showed "Engines (1/1)". During a manual LM Studio
install/start/stop/uninstall cycle the same node went 1/2 → 2/2 → 1/2 → 1/1,
confirming installing/stopped states count in the denominator and custom
endpoints persist correctly across it.

Risk

• Wire compatibility: the node-record TXT gains the cu= service key and
  a new emit position. Old peers parse unknown keys through the
  forward-compat path; mixed-version behavior is pinned by the new
  advertise/interop suites rather than manual reasoning.
• Security: endpoint API keys ride only an add request and never return
  in replies (views carry keySet) or in logs (reply entries are rebuilt from
  known fields; redact-log gained apiKey/api_key). Keys live only in the
  owner's 0600 store, and forwarding injects the key only on the owning
  node's last hop with the caller's Authorization stripped. Adding an endpoint
  that would forward to one of the node's own facade addresses is refused, so
  the facade cannot loop on itself.
• Ports: the Custom facade defaults to 8000; users proxying something else
  there set the per-node custom port instead, which is reserved away from
  engines, sibling proxies, and the node's own endpoints.
• Packaging: macOS builds now trigger the Local Network permission prompt
  (documented in getting-started and troubleshooting, including the
  unsigned-build workaround). A denied grant means no peers, not a crash.
• Tray polling: the custom:list pulls pause while the surface is hidden,
  run one-at-a-time, and keep the previous snapshot on a failed read
  (unknown ≠ "no custom engines").

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• I declared version bumps in the release-intent block above. services/versions.json is written by automation — do not edit it by hand.

[Noah-Tervalon-Nvidia]

I think this idea is something we do want to do in the future, we are thinking of how exactly we want to engineer it though. We'd love effectively a universal adapter and need to take some time to design that internally. When we have ideas about the exact execution we would love to work with you to implement it.