You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Kubernetes, Slurm: effective resource requirements are not used #4019
$ dstack fleet -v
NAME NODES RESOURCES SPOT BACKEND PRICE STATUS CREATED ERROR
backend 0.. cpu=2.. mem=2GB.. disk=0GB.. gpu=0.. on-demand * - active 23 mins ago
Note: CPU >= 2, memory >=2
Submit a run:
$ dstack --cpu 1.. --memory 1GB.. -b slurm -b kubernetes
Project main
User dmitry-local-admin
Type dev-environment
Resources cpu=1.. mem=1GB.. disk=1GB.. gpu=0..
Spot policy on-demand
Max price off
Retry policy off
Idle duration 5m
Max duration off
Inactivity duration off
# BACKEND RESOURCES INSTANCE TYPE PRICE
1 kubernetes (nebius) cpu=2 mem=2GB disk=1GB computeinstance-e00ef6jy764y8b2jps $0
2 kubernetes (nebius) cpu=2 mem=2GB disk=1GB computeinstance-e00h474t5jwnkkwg8a $0
3 slurm (vm) cpu=2 mem=2GB disk=1GB worker-1 $0
4 slurm (vm) cpu=2 mem=2GB disk=1GB worker-2 $0
Note, the run requests CPU >=1, memory >= 1, but the offers are adjusted according to the effective (fleet/run combined) requirements (CPU >= 2, memory >=2)
Check resources actually requested
Actual behaviour
$ kubectl get -n dstack-dmitry pod dstack-main-dev-environment-0-0-tjrihv30 -o jsonpath='{.spec.containers[*].resources}'
{"requests":{"cpu":"1","ephemeral-storage":"1Gi","memory":"1Gi"}}
$ scontrol show job 271 | grep ReqTRES
ReqTRES=cpu=1,mem=1G,node=1,billing=1
Activity
Pass effective Requirements to Compute's run_job/run_jobs (dstackai#4020