Skip to content

[Bug]: Kubernetes, Slurm: effective resource requirements are not used #4019

Closed

Description

@un-def

Steps to reproduce

  1. Prepare a fleet:

    $ dstack fleet -v
    
     NAME     NODES  RESOURCES                             SPOT       BACKEND  PRICE  STATUS  CREATED      ERROR
     backend  0..    cpu=2.. mem=2GB.. disk=0GB.. gpu=0..  on-demand  *        -      active  23 mins ago
    

    Note: CPU >= 2, memory >=2

  2. Submit a run:

     $ dstack --cpu 1.. --memory 1GB.. -b slurm -b kubernetes
    
      Project              main
      User                 dmitry-local-admin
      Type                 dev-environment
      Resources            cpu=1.. mem=1GB.. disk=1GB.. gpu=0..
      Spot policy          on-demand
      Max price            off
      Retry policy         off
      Idle duration        5m
      Max duration         off
      Inactivity duration  off
    
      #  BACKEND              RESOURCES               INSTANCE TYPE                       PRICE
      1  kubernetes (nebius)  cpu=2 mem=2GB disk=1GB  computeinstance-e00ef6jy764y8b2jps  $0
      2  kubernetes (nebius)  cpu=2 mem=2GB disk=1GB  computeinstance-e00h474t5jwnkkwg8a  $0
      3  slurm (vm)           cpu=2 mem=2GB disk=1GB  worker-1                            $0
      4  slurm (vm)           cpu=2 mem=2GB disk=1GB  worker-2                            $0
    

    Note, the run requests CPU >=1, memory >= 1, but the offers are adjusted according to the effective (fleet/run combined) requirements (CPU >= 2, memory >=2)

  3. Check resources actually requested

Actual behaviour

$ kubectl get -n dstack-dmitry pod dstack-main-dev-environment-0-0-tjrihv30 -o jsonpath='{.spec.containers[*].resources}'
{"requests":{"cpu":"1","ephemeral-storage":"1Gi","memory":"1Gi"}}
$ scontrol show job 271 | grep ReqTRES
   ReqTRES=cpu=1,mem=1G,node=1,billing=1

Expected behaviour

{"requests":{"cpu":"2","ephemeral-storage":"1Gi","memory":"2Gi"}}
ReqTRES=cpu=2,mem=2G,node=1,billing=1

dstack version

0.20.27

Server logs

Additional information

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions