Steps to reproduce
Prerequisites:
- GPU nodes are not tainted (
dstack doesn't add tolerations if no GPU requested).
- An image with
NVIDIA_VISIBLE_DEVICES=all baked into, e.g., NVIDIA images or dstackai/base (which is used when image is not set in the run configuration).
type: dev-environment
name: dev-environment
resources:
gpu: 0
dstack applyssh dev-environmentls -1 /dev/nvidia[0-9]nvidia-smi -L
Actual behaviour
All GPUs are passed inside the container via the NVIDIA_VISIBLE_DEVICES env var path.
# ls -1 /dev/nvidia[0-9]
/dev/nvidia0
/dev/nvidia1
/dev/nvidia2
/dev/nvidia3
/dev/nvidia4
/dev/nvidia5
/dev/nvidia6
/dev/nvidia7
# nvidia-smi -L
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-fe56c782-f8db-cc17-fa82-4286037f988b)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-cba9dcf6-8c94-fa2b-1bf0-f1476941a458)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-33ac7880-d9d2-c4b7-8ad1-a1c55d5124d6)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-74a529a1-2939-79e1-7f54-aec2d35756c6)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-cbf70ec6-5ef2-eadc-510c-ed98ee0fc6ed)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-37d22e64-5838-4837-a577-916774e2edf7)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-43d11db3-eb4d-5e26-0b3a-81ed92f37a86)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-56aec030-3f6e-09b3-10ce-0c9834e72182)
Expected behaviour
No GPUs inside the container.
dstack version
0.20.26
Server logs
Additional information
No response
Steps to reproduce
Prerequisites:
dstackdoesn't add tolerations if no GPU requested).NVIDIA_VISIBLE_DEVICES=allbaked into, e.g., NVIDIA images ordstackai/base(which is used whenimageis not set in the run configuration).dstack applyssh dev-environmentls -1 /dev/nvidia[0-9]nvidia-smi -LActual behaviour
All GPUs are passed inside the container via the
NVIDIA_VISIBLE_DEVICESenv var path.Expected behaviour
No GPUs inside the container.
dstack version
0.20.26
Server logs
Additional information
No response