Repository navigation
Conversation
Lifecycle fixes needed by an engine that takes a minute or more to start and forks helper processes, such as vLLM. None of them is engine-specific. - engine:status and engine:get-installed no longer wait for a running lifecycle operation, so a toggle during a slow start does not time out. - Shutdown caps the stop grace at 3s so the force kill lands inside the parent's window, and a SIGTERM no longer blocks on an idle stdin. - A stop also kills helper processes left in the engine's process group, so a crash mid-load does not orphan a process holding GPU memory. - An orphaned engine run by an interpreter (a Python package's script) is recognised by its command line, so the next launch can stop it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…ache Add an hf_cache manifest action that lists the models in a Hugging Face hub cache directory. It reads the disk, so it works while the engine is stopped, which lets an engine that serves one model per process (vLLM) show every model it could be switched to. A repository is listed only when its snapshot is a complete, text-model-shaped download: vision-only and GGUF-only repositories, interrupted downloads and links whose blob is gone are left out. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a launch_model manifest action for an engine that has no load call because
the model is a launch argument (vLLM). It takes {"model": "<id>"}, writes the
model into the saved launch settings under the declared flag, and restarts a
running engine on it. If the new model does not come up, the previous one is
restored and the error says so. A stopped engine only has its saved launch
changed, and asking for the model already in the launch arguments restarts
nothing.
The action requires editable launch settings and a readiness probe on every
platform, and refuses model ids that could be read as a flag, a path or a
shell fragment.
Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Register vllm alongside Ollama, LM Studio and llama.cpp: facade port 8000, engine port 8001, discovery key "vl". The proxy and broker treat it like any other managed engine, and the terminal interface gets its model wire. The engine-manager manifest (Linux amd64/arm64) is detect-first: an existing vllm is adopted, otherwise a sha256-pinned uv installs vllm 0.32.0 into a private venv after checking for an NVIDIA driver with CUDA 13+ and enough disk. vLLM serves one model per process, so its list_models is an hf_cache action, load_model is a launch_model action, and a new manifest field, advertised_models: loaded, advertises only the served model to peers and the proxy, since a request for any other cached model would fail at the engine. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Check the bundled vLLM manifest's install, launch and model actions, that the install refuses drivers without CUDA 13, and, across real binaries, that the broker starts a vLLM proxy and the facade serves the model list and routes exact model ids. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Show vLLM next to the other engines: type, display name, links, welcome-screen default (Linux only), proxy identity and port state, icon and downloaded-model matcher. The icon is an original lettermark, not the vLLM project logo. vLLM serves one model per process and has no pull, eject or delete call, so a new hasPullModel capability hides Add model for it. Its model list is the Hugging Face cache and Load switches the served model by restarting vLLM; that load is observed so a failure is attributed to the model row, and the row stays locked for up to 20 minutes while the engine restarts. Document the engine across the desktop and top-level docs, tick the roadmap item, and add the vLLM model store to the data-reset notes. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Models with a 256K default context need more KV cache than the default 0.5 GPU memory share provides and fail to start. Pass --max-model-len 32768 by default and document it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…ts it vLLM serves one model per process, so loading another restarts the engine. Show 'starting' instead of 'stopped', keep the model list, and await the load call so the switch does not look like the engine was disabled. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…lling per model vLLM refuses a context above the model's own limit, reserves 92% of the GPU by default, and rejects tool_choice auto without a tool-call parser. Agent clients such as Hermes need a 64K window and tool calling. Add a manifest auto_launch (hf_context) step that reads the served model's config.json and the free GPU memory at every start and appends --max-model-len (64K target, shrunk to fit), YaRN --hf-overrides when the model's window is shorter, --gpu-memory-utilization sized to the weights and KV cache, and --enable-auto-tool-choice with the hermes parser for Qwen families. Arguments the user saved always win, nothing is written back to saved settings, and tensor/pipeline-parallel launches are left alone. Drop the fixed 0.5 GPU share and 32K cap from the default launch. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…for vLLM vLLM has no pull API, so a model not already cached could only be used by naming it in the launch arguments. Add an hf_pull action kind for pull_model: it reads the repo's file list and sizes from Hugging Face, selects the files vLLM reads, and runs the hf tool from the engine's virtualenv, reporting progress from the bytes landing in the hub cache. It needs no running engine and stops when cancelled or stalled. Add a vLLM engine:catalog source: open, text-generating safetensors repos of a few publishers (text-generation and image-text-to-text), with parameter count and size, plus a searchable query like llama.cpp's. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Enable hasPullModel for vLLM, give it an engine hub, let the model hub search the upstream catalog for any engine that reports it (a new hubSearch capability, set for llama.cpp and vLLM), and warm the vLLM catalog with the others. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM had no delete, so a cached model could only be removed by hand. Add an hf_cache delete variant for delete_model: it removes a model's folder and download locks from the hub cache, needs no running engine, refuses a path outside the cache, and refuses the model the engine is set to serve (running or not), failing closed when that cannot be determined. Enable hasDeleteModel for vLLM in the desktop, with a new deleteBlockedWhileLoaded capability that disables Delete on the loaded model. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM cannot drop its model while running, so Eject (unload_model) is a new stop_engine action that stops the engine and frees all of its GPU memory. Load (launch_model) on a stopped engine now saves the choice and starts it, restoring the previous choice if it does not start. Enable hasEject for vLLM and keep its model list, Add model and Load available while stopped, since the list is read from disk. Delete removed only a model's folder, but recent Hugging Face tooling keeps large files in a shared hub/blobs store linked from each repo, so the weights stayed on disk. Delete now also removes shared files no other model uses, with their sidecars, and reports the bytes freed. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add an optional runtime front: PAIR listens on the engine's public port and runs the engine itself on a loopback port at a fixed offset, forwarding requests unchanged, streaming responses, cancelling the engine's request when the client leaves, and answering 503 while the engine is not up. Only the launch that is spawned is repointed; the launch users see and validate keeps the public port. A stale engine on the internal port from a crashed run is reclaimed at the next start. This is the base for idle unload and wake on request. Only vllm.json declares a front, and the hooks in shared lifecycle code do nothing without one; a test checks that every other engine's launch is unchanged. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
An engine behind a gateway now tracks inference activity at the gateway (not GET/HEAD/OPTIONS, so health checks and the model list do not count, and a request in flight keeps the engine loaded). After front.idle_timeout_s (600 s for vLLM, 0 never) with none, the manager stops the engine's process, freeing the GPU, while the gateway and the ON state stay. The status reports idle, /health still answers, inference is refused with 503, and the model it is set to serve is still advertised with nothing marked loaded. Eject on such an engine now unloads and keeps it ON like the other engines, and Load of the model it is set to serve loads it again; a failed load leaves the engine idle. Waking on an incoming request follows. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
A request that needs the model and arrives while the engine is unloaded now loads it behind the gateway and is answered once it is ready, as Ollama and llama.cpp do, instead of getting a 503. Requests that arrive together share one start, a client that leaves does not abandon the load, and Eject followed by a request reloads it. A failed load answers 503 with the reason and leaves the engine idle so the next request retries. Health checks, the model list and other reads never load the model. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…e proxy The proxy routes by each node's advertised models and builds its own model list from the engine's /v1/models. While the engine was unloading or loading, the inventory briefly lost the model, and the idle gateway refused the model list, so a request arriving then was answered 'no node advertises the requested model' and never reached the wake. Record the served model when the engine goes idle, mark the state idle before the gateway so no reader sees an engine that is neither loaded nor idle, keep the engine marked running and idle while it loads (no stopped flicker), and answer GET /v1/models from the gateway while unloaded or loading. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…or loading The health check kept probing /health while the engine loaded again after an idle unload. The gateway forwards that probe to the engine for the start's readiness check, so it failed for the whole load and raised 'failed its health probe', which cleared itself once the load finished. Skip the probe while the engine is idle or loading, and test that no warning is raised while one stands. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…out settable Start, restart and the restore after PAIR restarts now bring an engine with a front up idle, with nothing loaded, as Ollama and llama.cpp start a server: about a second, the engine ON and healthy, the model loaded by the first request or a Load. The launch is still validated and the ports checked, so a launch that cannot work is refused at Start. Applying launch settings and switching or loading the model still load it at once, so a bad setting is found and rolled back. Add get_idle_timeout and set_idle_timeout (front_idle) actions: the limit is read and changed in seconds, 0 for never, applies at once and is saved in the engine's override file. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
The status of an engine that is on with nothing loaded now carries idle, and the card reads 'Idle · loads on next request'. For vLLM the model row's expiry control becomes the idle limit (1 minute to 1 hour, or stays loaded), read from and sent to the engine manager as seconds through get_idle_timeout and set_idle_timeout; a limit that is not offered, such as one set by editing the override file, shows as the nearest one. A new engine-level list marks which engines use the expiry this way, and a test keeps it and the capability in step. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Unload an idle vLLM after five minutes, not ten, and show five minutes as the limit until the engine manager reports the real one. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a darwin/arm64 platform to the vLLM manifest. The install fetches the sha256-pinned uv, creates a managed Python 3.12 venv and installs the pinned vLLM 0.32.0 core and vLLM-Metal wheels, each checked against a digest. It refuses early on non-arm64, macOS older than 15, or under 10 GB free. Allow the install on macOS in the desktop engine capabilities, update the tests that assumed Linux only, and document the new platform. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM serves one model per process, so a request naming another downloaded model used to be refused: only the loaded model was advertised. The gateway in front of vLLM now reads the request's JSON "model" field and, for a different downloaded model, lets the requests already running finish, unloads the model, saves --model, loads the new one behind the same gateway, and answers the held request. Requests that arrive together share one switch. A model that is not downloaded gets a 404 and nothing changes. A switch that fails is not rolled back: the request gets a 503 naming the model and the reason, and vLLM is left unloaded and ON, so the next request loads again. vLLM now advertises every cached model, including while idle. The gateway's /v1/models names them all, and loaded_models asks for /v1/models?resident=1 so only the loaded one is reported as resident. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
- Re-check for a switch under the gateway lock before admitting a request, so a request cannot start on a model that is being unloaded or load the old one while the switch runs. - Verify the current model was really unloaded before saving the new one, and wait out a load that was only just starting. - Leave a name that is not a downloaded model (such as a --served-model-name alias) to the engine instead of answering 404. - Read the downloaded-model list once per interval for all requests, keep the last good list when a read fails, and bound the model-list body separately from the request peek. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
95d8ec6 to
7c0f178
Compare
Several docs described vLLM as it was before later commits in this series: - vLLM also runs on Apple Silicon macOS, not only Linux. - Every cached model is advertised, and a request for one that is not loaded switches vLLM to it; the docs still said only the served model was advertised and other requests were refused. - The PAIR gateway listens on 8001 with vllm serve behind it on loopback 18001; the docs still put vllm serve on 8001 and said vLLM had no load, unload, pull or delete actions. Also document GET /v1/models?resident=1, and say "vLLM engine" rather than "vLLM backend" in the services parity table. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a probe that asks the installed vLLM which of its tool-call parsers reads a model's tool-call format. It runs with vLLM's own Python, renders the model's chat template with one example tool call, and offers that text to every registered parser; a parser matches when it extracts exactly that call. A tokenizer that ships response_template metadata gets vLLM's hf parser. Nothing is downloaded and no code from the model repository runs. Among parsers that read a format equally, the general-purpose ones are preferred. Results are kept per vLLM binary and snapshot for the life of the service; a failed probe is tried again on the next start. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Tool calling was enabled only for the Qwen families, with the hermes parser, so every other model answered HTTP 400 to agent clients, which send tools with tool_choice "auto" on every request. Replace the family table with the probe: a vLLM launch now gets --enable-auto-tool-choice and the parser that reads the model's own tool-call format, or no tool flags when none does. A parser named in the launch arguments is still used as given. Checked against vLLM 0.32.0: Qwen2.5 and Qwen3 get hermes as before; Qwen3.5 and Granite 4.2 now get qwen3_coder; Nemotron Nano v2 and Qwen2.5-VL get none. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
… parser A model that no vLLM parser reads (Nemotron Nano v2, or a template with no tool support) still failed every chat from an agent client: vLLM rejects tool_choice "auto" without a parser, and such clients offer their tools on every request. The gateway now knows whether the running launch parses tool calls. When it does not, it removes tools, tool_choice and parallel_tool_calls from chat requests that leave tool_choice unset or "auto", so the model answers in plain text, and marks the response with X-PAIR-Tool-Calls. A request with tool_choice "required" or a named function is forwarded unchanged and gets vLLM's own error. Bodies are still never logged. Update the engine-manager README and the engine settings guide for the parser choice and this fallback. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
kjlubick
left a comment
kjlubick
left a comment
There was a problem hiding this comment.
waitsForEngineReadiness in nvpair-engine-manager/remoteclient.go has a branch for ollama that might also be needed for vllm with regards to remote vLLM loads being canceled too early
| settings "nvpair-shared/enginesettings" | ||
| ) | ||
|
|
||
| // Front puts a PAIR-owned HTTP gateway on an engine's public port and runs the |
There was a problem hiding this comment.
Could you explain the architectural decision to add the inference gateway in nvpair-engine-manager/front.go? From my understanding, features like inference proxying should be handled by nvpair-proxy, or at least that's what the engine-manager spec says.
There was a problem hiding this comment.
I want to +1 this question. I don't understand the need for another proxy.
There was a problem hiding this comment.
Good question, and you're both right: this conflicts with the spec. nvpair-engine-manager/spec.md puts inference traffic out of scope ("this service never proxies /api/chat etc."), and the gateway does exactly that. I updated the README but not the spec, and I should have raised the design before building on it.
Why it's there: vLLM serves one model per process, loads it at startup, has no load/unload API, and holds most of the GPU until the process exits. The other engines can load a model on the first request, free memory when idle, and serve any model by name. To give vLLM the same behaviour, something has to see each request and start, stop or relaunch the process. I put that next to the lifecycle code in the engine manager, but in hindsight it's the wrong layer. It also has a practical gap: a request that triggers a large model load can outlast the proxy's 120 s response timeout.
Proposal: take the gateway out of this PR.
- This PR lands vLLM as an engine the user manages directly. Load restarts vLLM with the new model, and Eject stops it and frees the GPU. Hugging Face pull/list/delete, desktop support and tool-call parser selection stay; tool-call parsers are chosen at launch, with no request rewriting.
- Removed from this PR:
front.go, idle unload, wake on request, switch on request, removingtoolsfrom requests, and the idle-limit setting. Only the loaded model is advertised for routing. - On-demand loading gets a separate design discussion. The likely shape: the engine manager detects idleness from vLLM's own
/metrics, with no traffic through it, and the vLLM facade innvpair-proxy, which already sees every request, holds a request and asks the engine manager through the broker to load the model. vLLM's sleep mode is worth evaluating there too.
Does that direction work for you, or would you prefer something else? I'll hold off on the rework until we agree.
engine:get-installed listed every manifest, and reported an engine with no block for the host's OS and CPU, such as vLLM on Windows, as an empty not-installed entry. The desktop and TUI then offered an install that could only fail. Leave such an engine out of the listing, and make engine:status return an error for it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
A node's engine manager now leaves out an engine the node cannot run. Once a node has listed its engines, mark the missing ones unsupported and give them no row in the engine list, tray, endpoints or first-run setup. This hides vLLM on Windows, while a Linux node viewed from Windows still shows it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
11269f5 to
a0eff8f
Compare
Noah-Tervalon-Nvidia
left a comment
Noah-Tervalon-Nvidia
left a comment
There was a problem hiding this comment.
I reviewed everything up through 4e8d2f4 (where you add the front.go). Since I don't really understand the rationale for adding that I skimmed the rest and they all rely on that so going to hold off any further comments.
|
|
||
| // hfShardRe matches the shard names Hugging Face repos use when they carry no | ||
| // index file, such as model-00003-of-00004.safetensors. | ||
| var hfShardRe = regexp.MustCompile(`^.+-(\d+)-of-(\d+)\.(?:safetensors|bin)$`) |
There was a problem hiding this comment.
NIT - my regex optimization is going off here, matching the start of a line and then .+ (or .*) is going to make it match every line that it's run against and then check every character before it eventually finds it doesn't match the rest of the string and give up. You can just start with the -(\d+)
There was a problem hiding this comment.
Agreed. The leading ^.+ is unnecessary, so the pattern is now -(\d+)-of-(\d+)\.(?:safetensors|bin)$. Since it only runs on individual file names and Go's RE2 engine does not backtrack, there was no measurable cost, but the shorter pattern states the intent more clearly.
| var hfShardRe = regexp.MustCompile(`^.+-(\d+)-of-(\d+)\.(?:safetensors|bin)$`) | ||
|
|
||
| // runHFCacheAction lists the models in act.HFCache.Dir. It needs no running | ||
| // engine, so the list is available while the engine is stopped. |
There was a problem hiding this comment.
I haven't seen where this is used yet so this question could be dumb, but is it going to cause issues when a user wants to use a model from vllm even though it's turned off, then we have to go turn it back on and then load the model. I assume there's provisions for that? But in the current version we don't manage turning engines on or off for users so I'd be curious about that decision (if that's indeed what happens)
There was a problem hiding this comment.
Listing the cache while vLLM is stopped serves display only: the desktop shows the downloaded models so they can be reviewed or deleted. It does not make PAIR start the engine. The node's advertised model list includes only running engines (models.go:48), so a stopped vLLM's models are never offered to the cluster and no request is routed to it.
vLLM starts only on an explicit user action. Load on a stopped vLLM saves the selected model and starts the engine, and restores the previous selection if the start fails.
The one place PAIR currently starts and stops the vLLM process on its own is the gateway's idle unload and wake-on-request. As proposed in the front.go thread, that behaviour would be removed from this PR.
| if total == 0 || len(seen) != total { | ||
| return false | ||
| } | ||
| for n := 1; n <= total; n++ { |
There was a problem hiding this comment.
this check seems redundant and prone to causing failures if total is for some reason different than the length we actually found. Could give out of bounds reads etc. We just checked that total is the same as the length of seen and seen gets directly created above so it couldn't possibly be false.
There was a problem hiding this comment.
Thanks for the close look. seen is a map, so a missing shard number reads as false rather than going out of bounds. The loop is the check that matters, though: seen is keyed by the shard number parsed from each file name, so its size can equal total while a shard is still missing (for example -00000-of-00002 and -00001-of-00002, with no -00002). You are right that the two checks overlap, but it is the length comparison that is redundant, so I removed it and kept the loop. I also added that case to the tests to document the intent.
| // arguments already say the same thing, whatever their quoting: the launch text | ||
| // shown to a user is quoted canonically, so comparing text would call an | ||
| // unchanged launch changed. | ||
| func withLaunchFlag(text, flag, value string) (result string, changed bool, err error) { |
There was a problem hiding this comment.
is there a reason that we are not using json to control the launch args similar to how we do it with the other engines? Then this parsing gets a lot easier by just searching the json file for the specific arg and updating it rather than this looping check doing string parsing.
There was a problem hiding this comment.
The launch arguments are stored as JSON, the same as for the other engines: the saved settings hold them as an array ("launch_args": ["--model", "Qwen/Qwen3.5-4B"]). The text form is the settings editor's representation, shared by every engine with editable launch settings. launch_model saves through that same path, so the host, port and CORS checks apply and the saved settings always match what the engine runs.
The loop works on the parsed argument list rather than on raw text. It is needed because command-line arguments are ordered rather than key/value: it has to handle both --model X and --model=X, a flag given more than once, and a -- terminator, while leaving every other argument in place. A JSON object could not represent valueless or repeated flags faithfully.
You are right that it could be simpler, though. I've folded the set and remove cases into one helper that drops every form of the flag, so setting the model is now "remove, then insert at the same position", with no behaviour change.
| `--log-level debug` to see live discovery logs on stderr. | ||
| - **"Bind: address already in use"**: the proxies listen on `11434` (Ollama), | ||
| `1234` (LM Studio), and `8080` (llama.cpp); another instance of the proxy or | ||
| `1234` (LM Studio), `8080` (llama.cpp), and `8000` (vLLM); another instance of the proxy or |
There was a problem hiding this comment.
8000 is a fairly common port (not as common as 8080), and since vllm doesn't have a default can we perhaps set this to any port that is less common so we don't have as good a chance at bumping into something else?
There was a problem hiding this comment.
Based on the vLLM documentation, 8000 is vLLM's default port: "By default, it starts the server at http://localhost:8000" (vLLM quickstart), and vllm serve --help lists --port with "default: 8000". PAIR's proxies deliberately take each engine's standard port, so existing clients work through PAIR unchanged: 11434 for Ollama, 1234 for LM Studio, 8080 for llama.cpp, and 8000 for vLLM. The engine itself then runs on the next port up (8001 here, as with 1235 and 8081). A less common port would break that for vLLM alone. If 8000 is taken, the proxy reports the conflict, as covered in this section, and its port can be changed.
| @@ -0,0 +1,190 @@ | |||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
NIT - I like the catalog specifically for vllm, but this feels like an HF catalog file, perhaps it could be generalized in the future to support any pulls off HF? I'm thinking also that it would be nice to pull out the llama.cpp stuff into it's own catalog file too. Neither of these needs to happen in this PR I think.
There was a problem hiding this comment.
Agreed, and most of the groundwork is already shared: the vLLM catalog uses the same Hugging Face client, row type, cached listings and search cache in catalog.go that the llama.cpp catalog uses. What differs per engine is small: the publishers listed, which files make a repository usable (a Q4_K_M GGUF for llama.cpp, safetensors for vLLM), and the download name. A follow-up could move the shared Hugging Face parts into catalog_hf.go and llama.cpp's into catalog_llamacpp.go, leaving each engine's catalog as a small set of rules. I'll keep it out of this PR to avoid moving unrelated code here, and track it as a follow-up.
|
|
||
| // hfDroppedSuffixes are file types a vLLM-served model never reads. Skipping | ||
| // them keeps a repo that ships several formats from downloading each of them. | ||
| var hfDroppedSuffixes = []string{".gguf", ".onnx", ".h5", ".msgpack", ".ot", ".tflite"} |
There was a problem hiding this comment.
Just asking the question: is this something we want to do? If a user pulls a model they probably expect it to be all there in case they want to go use something else with that model that they assume is all on disk. If they try to run it and parts seem to be missing that might be misleading.
There was a problem hiding this comment.
Yes, this is deliberate. A pull downloads what the engine will load, not the whole repository, which is how every engine PAIR manages already behaves: llama.cpp fetches a single GGUF quantization (repo:Q4_K_M), Ollama pulls a single tag, and lms get downloads one variant. vLLM has no download command of its own, so PAIR makes that choice in hfpull.go, and it is the same choice vLLM makes when it downloads a model on first start: the weight format it loads (safetensors when present, otherwise .bin) plus config and tokenizer, skipping original/ checkpoints and other formats. Fetching everything would often multiply the size; Llama 3.3 70B Instruct, for example, carries about 141 GB of safetensors plus about 140 GB of original/ checkpoints.
On using the model with something else: the files sit in the standard Hugging Face cache, which is per file rather than a copy of the repository, so transformers, hf download or another Hugging Face–based tool reuses what is there and fetches only what it is missing. The catalog shows the size of the weights being downloaded, not of the whole repository. I agree it should not be implicit, though, so the action's description and the README now state that it downloads the files vLLM loads and that other formats in the repository are not fetched.
| if err := os.RemoveAll(repo); err != nil { | ||
| return nil, fmt.Errorf("delete %s: %w", id, err) | ||
| } | ||
| _ = os.RemoveAll(filepath.Join(hubDir, ".locks", name)) |
There was a problem hiding this comment.
is there a reason not to just delete the directory and everything below it (not following sym links)?
There was a problem hiding this comment.
That is what happens for the repository itself: its folder is removed with os.RemoveAll, which deletes symlinks without following them. The additional steps cover files that Hugging Face keeps outside that folder. Download locks live in hub/.locks/<repo>, so they are removed separately. Recent Hugging Face tooling also stores the weight files in a store shared by all repositories (hub/blobs/<xx>/<hash>) and leaves only symlinks in the repository's own blobs directory. Deleting the folder alone would therefore leave the weights on disk, so the delete also removes any shared files that no remaining repository links to. Files that another model still uses are kept.
| arguments and restarts vLLM on it, restoring the previous model if the new one | ||
| does not start (for example it does not fit in GPU memory). | ||
|
|
||
| `delete_model` removes a model's folder from the hub cache (an `hf_cache` action with `delete`); the model vLLM is set to serve, running or not, is refused so a switch has to come first. |
There was a problem hiding this comment.
Just curious why you made the decision to block deletion of a model when the engine is not running but set to serve that model? It seems we should allow users to delete that and then bump the engine to no model and raise an error if they try to restart it without switching models first or something like that.
There was a problem hiding this comment.
The block was there because vLLM has no "no model" state of its own: if the configured model is missing it downloads it again on the next start, and vllm serve without a model falls back to a built-in default and downloads that instead. Refusing the delete was the simplest way to avoid an unrequested multi-gigabyte download.
Your approach is better, so I've adopted it. When vLLM is stopped, deleting the configured model now succeeds and clears --model from the saved launch arguments. Starting an engine whose manifest declares a launch_model flag without that flag set is refused with a clear error ("no model selected; load a model to start it"), so nothing is downloaded, including on an automatic start. Loading a model sets it again and starts the engine. While vLLM is running, deleting the model it serves is still refused; once it is stopped, the delete goes through.
| settings "nvpair-shared/enginesettings" | ||
| ) | ||
|
|
||
| // Front puts a PAIR-owned HTTP gateway on an engine's public port and runs the |
There was a problem hiding this comment.
I want to +1 this question. I don't understand the need for another proxy.
…tore the previous model A remote load used the ordinary 30 s response-header budget for every engine except Ollama. Loading a vLLM model restarts vLLM on it, which can take minutes, so the caller gave up, the peer's handler was cancelled mid start, and the rollback to the previous model ran under the same cancelled context and failed at once, leaving vLLM down with the new model saved. Remote vLLM loads now use the readiness-sized client, and the previous model is restored under a context the caller cannot cancel. A test cancels a slow switch and checks that the previous model is back and running. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Replying to this review point. @kjlubick Thanks, good catch. Confirmed and fixed in 5c27d65. For vLLM, The fix:
Verified across two clustered machines with a real vLLM (RTX 6000 Ada → RTX A6000 over LAN mTLS):
|
167fedb to
13666dc
Compare
(review) Thank you for the detailed review. I've replied to each of your comments in its own thread, and the resulting changes are in the last commit. On |
|
Follow-ups from the review, kept out of this PR To keep this PR focused, these improvements came out of the review but are deferred. I'll open an issue for each once the PR merges.
Pending in this PR: removing the vLLM gateway ( To verify before merge: the Qwen3.5-4B default on Apple Silicon through vLLM-Metal, and launch sizing on a DGX Spark. Let me know you thoughts / comments on this |
- Match Hugging Face shard file names without a leading ^.+ in hfShardRe. - Drop the redundant shard-count comparison; the loop over shard numbers is the real completeness check, and a test now covers shards numbered from zero. - Allow deleting the model a stopped vLLM is set to serve. The delete takes --model out of the saved launch arguments, and an engine with a launch_model action refuses to start while its launch names no model, rather than letting vLLM download a default of its own. Loading a model selects one again. The model a vLLM process serves is still refused; an engine ejected behind its gateway is stopped first, so Eject then Delete works. - Say in hfpull.go that the download is done by Hugging Face's hf CLI, and state in the pull_model description and README that it fetches the files vLLM loads rather than the whole repository. - Fold setting and removing a launch flag into one helper, dropLaunchFlag. - Make Qwen/Qwen3.5-4B the default vLLM model in place of Qwen2.5-0.5B, state that it needs a GPU with at least 16 GB, and raise the install's free-disk check to cover the engine and that model (25 GB on Linux, 20 GB on macOS). - Size vLLM launches on unified-memory machines such as DGX Spark, where nvidia-smi reports [N/A] for GPU memory, from /proc/meminfo, as nvpair-node-info already does. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
13666dc to
74f8e14
Compare
Resolve two conflicts with develop: - services/nvpair-engine-manager/README.md: keep develop's shutdown-sweep paragraphs and this branch's signal and engine:status paragraphs. - services/nvpair-ui-broker/enginesettings_test.go: use develop's reserveSettingsPort for the harness ports, keeping the fifth port this branch added for the vLLM proxy. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Description
Adds vLLM as a supported engine alongside Ollama, LM Studio, and llama.cpp, and lets it serve any model already downloaded in the Hugging Face cache on request.
/v1/modelson the engine lists all cached models, while the desktop marks only the loaded model as running.toolswithtool_choice: "auto"on every request, which vLLM rejects with HTTP 400 unless it was started with a tool-call parser. PAIR now picks the parser from the model itself: it renders the model's chat template with one example tool call and keeps the installed vLLM's parser that reads it back exactly (hfwhen the tokenizer shipsresponse_template). No list of model families is kept, nothing is downloaded, and no code from the model repository runs. When no parser fits, the gateway removestoolsfrom chat requests that leavetool_choiceunset or"auto", so the model answers in plain text instead of failing, and marks the response withX-PAIR-Tool-Calls. A request withtool_choice: "required"or a named function is forwarded unchanged.Other engines keep their current model and request behavior. Shared code changes in these ways:
engine:statusandengine:get-installedno longer wait for a running lifecycle operation.hf_cacheandlaunch_model, and the gateway (front), which only the vLLM manifest uses.engine:get-installedleaves out an engine whose manifest has no block for the host's OS and CPU, where it used to report an empty not-installed entry, andengine:statusreturns an error for it. The desktop marks an engine missing from a node's listing asunsupportedand gives it no row. Today this affects only vLLM.commitLaunchLockedis extracted fromConfigureLaunchwith unchanged logic.Release intent
Changelog title
vLLM engine support with on-request model switching
Changelog body
Bumps
Scope
Included: the vLLM engine (manifest, installer, gateway, auto-sizing, idle handling), request-driven model switching, desktop UI, proxy/broker/TUI wiring for the new engine, docs.
Excluded: loaded-node preference in
nvpair-proxy(cross-engine, separate change); fixing vLLM memory sizing for some multimodal/quantized models.This is a large PR (about 9,300 added lines across 116 files), so it is organised as 32 focused commits that can be reviewed one at a time. The groups below are in commit order:
hf_cachemanifest action, which lists Hugging Face cache models.launch_modelmanifest action, which switches an engine's model through its launch flag.Validation
go test ./...inservices/nvpair-engine-manager,services/nvpair-proxyand the other touched modules; race tests repeated 15 times on the gateway switch tests.npm run lint,typecheck,test:unit,dead-code:check,service-contracts:checkindesktop/.go test ./...inservices/tests(cross-process, real binaries), run with no other PAIR instance on the machine.node scripts/spdx-headers.mjs./v1/modelslists all cached models, UI marks only the loaded one, a request for another cached model swaps in.get_weathercall withqwen3_coder(also when streaming), Qwen3-8B withhermes; Nemotron Nano 9B v2 answers 200 in plain text with the tools removed, and withtool_choice: "required"gets vLLM's 400.--modelcleared, a start is refused with no download, and loading another model starts vLLM and answers a chat request.NVPAIR_LIVE_VLLM_BIN=… go test -run TestToolParserProbe_FindsTheParserWithTheInstalledVLLM).Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.jsonis written by automation — do not edit it by hand.