[Bug]: PAIR cross-node routing fails: requests to a peer return "upstream returned HTTP 404" and the peer's model inventory does not update #164

PAIR version or commit

nvpair 0.1.1-463 from NVPAIR-Setup-0.1.1-arm64.deb

Affected component

Services / broker

Environment

Summary:
In a two-node PAIR cluster of Jetson Orin systems, requests routed from one node to a model that only the other node has fail with "upstream returned HTTP 404". The receiving node's own Ollama engine is healthy and serves the model directly, and the receiving node's Ollama log shows no request for the failing calls. The requesting node's model inventory never shows the peer's new model, even after the peer's nvpair was restarted.

Secondary (unconfirmed): on 2026-10-10 the nvpair TUI on one node panicked. It may be related to a terminal resize, but a resize test on 2026-10-11 did not reproduce it (see Issue 2).

Environment:

• PAIR version: Debian package nvpair 0.1.1-463 (arm64), installed from the release asset NVPAIR-Setup-0.1.1-arm64.deb. The GitHub release v0.1.1 (published 2026-08-28) points at commit 13b6811. The installed package doesn't name a commit, so the match to that tag is inferred from the version and build date, not confirmed.
• Installed build manifest (AGX): product 0.91.7, built 2026-08-28T21:53:38Z, source fingerprint a36564f8782c03a91727309eec67a6b616a1cd26ff3cdde24c68c8c6ee7dce21.
• Component versions (from the AGX's build manifest and binaries): nvpair-tui 0.7.2, nvpair-ui-broker 0.40.2, nvpair-engine-manager 0.17.4, ollama-proxy 0.26.2, lmstudio-proxy 0.16.2, nvpair-cluster-manager 1.1.4, nvpair-node-scanner 0.20.3, nvpair-node-info 0.13.3, nvpair-workload-manager 0.13.3, nvpair-manual-nodes 0.11.1, nvpair-errors 0.7.4, nvpair-node-settings 1.0.4.
• Hardware and OS: two Jetson Orin systems, AGX Orin Dev Kit and Orin Nano Super, both Ubuntu 24.04.5 with JetPack R39 (nvpair installed on both from the same package).
• Ollama 0.40.2, installed as a systemd service, listening on 127.0.0.1 on a non-default port (11435 on the AGX, 11436 on the Nano), matching each node's engines/ollama.json.
• nvpair was run over SSH from a terminal inside herdr (a terminal multiplexer, default configuration, not customized) on a separate workstation. herdr does not run on the Jetsons.

Ollama configuration (non-default):

• Ollama runs as a systemd service with a drop-in override at /etc/systemd/system/ollama.service.d/override.conf.
• The default Ollama port (11434) is taken by PAIR's ollama-proxy, so Ollama binds elsewhere. On the AGX the override sets OLLAMA_HOST=127.0.0.1:11435, which matches the AGX's engines/ollama.json. On the Nano the override sets 127.0.0.1:11436, which matches the Nano's engines/ollama.json. PAIR wrote the Nano's port when it first started, and Ollama was moved to match.
• On the AGX the override also sets OLLAMA_MODELS=/mnt/nvme/dave/models, which puts the model store on a separate NVMe drive. The Nano uses Ollama's default store.
• The Ollama installer printed "Unsupported JetPack version detected. GPU may not be supported." GPU inference still works (ollama ps reports 100% GPU on both), but the warning is worth knowing about.
• Checks against Ollama directly must use the engine port, not 11434 (PAIR's proxy).

Issue 1 (primary): cross-node routing returns HTTP 404; peer model inventory does not update

• The two nodes trust each other, and each node's workload history shows the other's workloads.
• Requests the Nano makes to itself for qwen2.5:0.5b complete.
• Requests from the AGX for that model, routed to the Nano, fail with "upstream returned HTTP 404". The AGX's model inventory lists only its own llama3.2:1b.
• Reproduced with the TUIs started normally over SSH, after the Nano's nvpair was restarted, and after the Nano's Ollama service was restarted. The Nano's own proxy lists qwen2.5:0.5b.
• Not related to any terminal resize: it has happened with TUIs that were never resized.

Evidence:

• Nano workloads-history.json: IDs 7 and 14 (scheduledOn the Nano's node UUID, originatedFrom the AGX's) failed with "upstream returned HTTP 404". ID 2, originated and scheduled on the Nano, completed.
• Nano Ollama journal: no POST entries during the failing windows (about 19:04 and 19:12 CDT on October 10, 2026). The only POST in that period is a completed local request at 19:13.
• Node UUIDs: AGX e1cecfcd-a32e-4f3c-9b61-251ba2cbd3d9, Nano 16870ce7-8ada-4d49-a26b-5c15ab0b50bd.

Issue 2 (secondary, unconfirmed): nvpair TUI panic on 2026-10-10

• Running nvpair on the Nano crashed with: "runtime error: slice bounds out of range [2:1]" in viewport.Model.visibleLines (bubbles v1.0.0), reached from nvpair-tui/ui.(*clusterView).SetSize (cluster.go:140) via ui.(*Model).resizeViews (model.go:175).
• The call chain is SetSize -> table.SetColumns -> UpdateViewport -> viewport.SetContent -> GotoBottom -> visibleLines. The viewport height in the panic was -1 and the width was 157 columns.
• At the time, the session was in a herdr pane that the user was resizing. This is a possible trigger, not a confirmed one.
• Resize test on 2026-10-11: the user resized herdr panes running nvpair SSH sessions to the Nano and the AGX, narrower and shorter. Neither crashed or showed other visible problems, and the TUI processes started earlier kept running. The trigger has not been reproduced.
• The panic code path (negative height in the viewport) is still a real defect if it can occur, so it's worth reporting as a secondary issue.

Notes on test hygiene:

• Before the failures in Issue 1, the AGX and Nano each had a detached TUI started from a script wrapper. I stopped those with pkill and started PAIR again. The Nano's Ollama service was restarted with systemctl. Issue 1 has been reproduced since then with normally started instances.
• One engine state was seen as "failed" in the Nano's TUI after those manual stops and restarts (cause unknown, not seen since). It is not part of the report's main reproduction.

Steps to reproduce

1. Two nodes with nvpair and Ollama running. Invite and accept.
2. Pull a model that only one node has (for example, qwen2.5:0.5b on the Nano only).
3. Send POST /api/generate for that model to the other node's PAIR port (11434).
4. Result: {"error": "model 'qwen2.5:0.5b' not found"}. The workload history shows error "upstream returned HTTP 404" on the Nano. The Nano's Ollama log has no request at that time.

Expected behavior

Expected:

• A request routed to a peer for a model it has should reach that peer's engine.
• The peer's model inventory should update within a reasonable time after a pull, without restarting nvpair.
• A terminal resize, including to very small sizes, should not panic the TUI.

Actual behavior

Routing fails between nodes

Sanitized logs or screenshots

Confirmations

• I searched existing issues for duplicates.
• This is not a security vulnerability.
• I agree to follow the Code of Conduct.