Repository navigation
[Bug]: fabricmanager-dependent GPUs do not work in AWS, GCP, Azure, OCI #2348
Description
Steps to reproduce
Try running this configuration on a fabricmanager-dependent instance (e.g. 8:A100 AWS, 8:H100 AWS, 8:A100 OCI, 8:H100 OCI) using dstack's default OS images.
type: task
commands:
- pip install numpy torch
- python -c 'import torch; print("CUDA available:", torch.cuda.is_available())'
resources:
shm_size: 24GBActual behaviour
/opt/conda/envs/workflow/lib/python3.9/site-packages/torch/cuda/__init__.py:129: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
return torch._C._cuda_getDeviceCount() > 0
CUDA available: False
Expected behaviour
CUDA available: True
dstack version
0.18.42
Server logs
Additional information
> ssh proud-dragonfly-1-host sudo journalctl -u nvidia-fabricmanager
Warning: Permanently added '54.161.199.97' (ED25519) to the list of known hosts.
Feb 24 09:28:18 ip-172-31-92-157 systemd[1]: Starting NVIDIA fabric manager service...
Feb 24 09:28:28 ip-172-31-92-157 nv-fabricmanager[1507]: fabric manager NVIDIA GPU driver interface version 535.183.01 don't match with driver version 535.183.06. Please update with matching NVIDIA driver package.
Feb 24 09:28:31 ip-172-31-92-157 nv-fabricmanager[1507]: fabric manager NVIDIA GPU driver interface version 535.183.01 don't match with driver version 535.183.06. Please update with matching NVIDIA driver package.
Feb 24 09:28:31 ip-172-31-92-157 systemd[1]: nvidia-fabricmanager.service: Control process exited, code=exited, status=1/FAILURE
Feb 24 09:28:31 ip-172-31-92-157 systemd[1]: nvidia-fabricmanager.service: Failed with result 'exit-code'.
Feb 24 09:28:31 ip-172-31-92-157 systemd[1]: Failed to start NVIDIA fabric manager service.
Activity
jvstme commentedon Feb 24, 2025
Introduced in 0.18.10 (August 2024)