feat(server)!: add --gpu to choose the GPU on multi-GPU machines - #117
Merged
Conversation
--gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index. It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>. /v1/stats reports the engine's GPU under "gpus". ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name. The daemon's /bench/profile returns the running serve's GPU profile. BREAKING CHANGE: ft checkpoint --device is removed; use --gpu.
…DEVICES BREAKING CHANGE: ft bench bw --device is removed; use --gpu.
Splits the published id into _assigned_physical (UUID) and _assigned_visible (CUDA ordinal). One process runs on one GPU, so e4m3_native() judges that card instead of scanning every visible device.
Collaborator
|
@jason-fxz LGTM. I think we can merge it now |
This was referenced Aug 25, 2026
Samfisheryu
pushed a commit
to ServeBig-project/FreeToken
that referenced
this pull request
Aug 26, 2026
…shML-org#117) * feat(server)!: add --gpu to choose the GPU on multi-GPU machines --gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index. It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>. /v1/stats reports the engine's GPU under "gpus". ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name. The daemon's /bench/profile returns the running serve's GPU profile. BREAKING CHANGE: ft checkpoint --device is removed; use --gpu. * refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES BREAKING CHANGE: ft bench bw --device is removed; use --gpu. * refactor: name the id namespaces and drop the multi-device e4m3 scan Splits the published id into _assigned_physical (UUID) and _assigned_visible (CUDA ordinal). One process runs on one GPU, so e4m3_native() judges that card instead of scanning every visible device. (cherry picked from commit 2757bb5)
ezutfen
pushed a commit
to Zutfen-LLC/FreeToken
that referenced
this pull request
Sep 12, 2026
…L-org#153 maintainer review) Corrected classification: Branch B BACKEND_REQUIRES_MULTI_CHUNK, terminal ISSUE117_ARM_C_REMEDIATION_BLOCKED. - The accepted FlashML-org#137 failing population (prompt_len 65-67, all >64) is multi-chunk by frozen-contract necessity: the R4 wire contract rejects token_count > max_token_count (64), the frozen boundary geometry (prefill_bytes = 64*3840*2, activation staging buffers) and the wire receive buffer are sized for exactly 64 rows. Runtime capacity 256 is session KV capacity, not boundary authority. A 65-67-row single call is therefore inadmissible without changing a frozen semantic/wire contract, which exceeds FlashML-org#153 authority — Branch A (UNNECESSARY_PARTITION_POLICY) was wrongly derived from the C2 53-row control's legality and is withdrawn. - The <=64 single-chunk policy cleanup is RETAINED as useful but does NOT remediate the accepted failing population: 65/66/67 still execute as the unchanged accepted 64+remainder path. No CPU-provable backend/state defect exists in the chunk-2 extend path (source inspection: metadata/causal semantics correct, fixed tiles, no autotune/atomics; instability is execution-level per FlashML-org#137) — branch-B option 1 unavailable without new physical evidence, so BLOCKED. - Timing-unit regression fixed: two_stage prefill accumulator restored to perf_counter_ns on both sides (prefill_ns field is nanoseconds); structural AST regression contract added (no bare perf_counter call survives; every *_ns field is ns-sourced). - Capacity ownership corrected: strategy (frozen constant owner) -> stage_chain/two_stage (chunk policy) and strategy -> last_stage_service (wire bound). The wire service no longer imports the chain runtime (removes the service->coordinator dependency of the reviewed head); no runtime module imports the wire service; sender and receiver derive from the same frozen constant. - 65-67 static/reduction controls added: exact failing population classified multi-chunk, partition equal to the accepted hand-literal loop, request-content equality, C2 53-row retained as causal control only (explicitly not a failing-population substitute). Focused suite: 79 passed. Full research suite: 532 passed, 1 deselected pre-existing base failure, 72 subtests. CPU-only; no GPU, no node access, no h109-*.
ezutfen
added a commit
to Zutfen-LLC/FreeToken
that referenced
this pull request
Sep 12, 2026
Issue FlashML-org#117 Arm-C remediation correction (InferSwarm FlashML-org#153): Branch B — BLOCKED; timing-unit fix; capacity ownership
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ft servealways usedcuda:0. This adds--gputo pick the card:ft bench bwandft checkpointtake the same flag. The Desktop app uses it for its GPU picker.Closes #99
Related #43 (fixed on the Desktop side),
What changed
--gpuis turned intoCUDA_VISIBLE_DEVICESbefore the workers start. The engine itself is unchanged.CUDA_VISIBLE_DEVICESis already set,--gpucan only pick from that list./v1/statsshows the GPU in use undergpus.ft bench bwwrites one profile per card (benchbw/<gpu-uuid>.json); the oldbenchbw.jsonstill works.Breaking change
ft checkpoint --deviceis removed. Use--gpu.Tested on
8x RTX PRO 6000 Blackwell (96 GB), Xeon Platinum 8559C, Amazon Linux 2023, driver 595.71.05, CUDA 13.0, torch 2.11.0+cu130.
Checkpoint: Qwen3.8-27B BF16 as FTW
ft serve --gpu <uuid>,--gpu 6, andCUDA_VISIBLE_DEVICES=2,3 ... --gpu 1: the process lands on the expected card innvidia-smi,/v1/stats.gpusmatches, chat completions work.GPU-deadbeef,99,1,2,"", mixed UUID+index) is rejected with a clear error before loading.ft bench bw --device 6andft daemon start ... -- --gpu <uuid prefix>: the per-card profile is written and/bench/profilereturns the right card.pytest tests/server tests/daemon tests/moe/test_hybrid_fetch.py: 570 passed.Not Tested on Windows or on a box with different GPU models.