Skip to content

[Bug]: creation_policy: reuse ignored #4045

Description

@un-def

Steps to reproduce

  1. Create a fleet:

    dstack fleet -v
     NAME     NODES  RESOURCES  SPOT  BACKEND  PRICE  STATUS  CREATED      ERROR
     backend  0..    -          auto  *        -      active  30 mins ago
    
  2. Submit a run with creation_policy: reuse/--reuse:

     Project              main
     User                 dmitry-local-admin
     Type                 dev-environment
     Resources            cpu=1.. mem=1GB.. disk=1GB.. gpu=0..
     Spot policy          on-demand
     Max price            off
     Retry policy         off
     Creation policy      reuse
     Idle duration        5m
     Max duration         off
     Inactivity duration  off
    
    No matching instance offers available. Possible reasons: https://dstack.ai/docs/guides/troubleshooting/#no-offers
    

    Note Creation policy reuse

Actual behaviour

A new instance provisioned

Expected behaviour

The run fails with FAILED_TO_START_DUE_TO_NO_CAPACITY – Could not reuse any instances for this job

dstack version

0.20.28

Server logs

DEBUG    dstack._internal.server.services.backends:492 Requesting instance offers from backends: ['aws', 'runpod', 'gcp']
...
INFO     dstack._internal.server.services.events:205 Emitting event: Job created on run submission. Status: SUBMITTED. Event targets: job(0eaa31)dev-environment-0-0. Actor: system
...
INFO     dstack._internal.server.services.events:205 Emitting event: Instance created for job. Instance status: PENDING. Event targets: instance(2aae42)backend-0, job(0eaa31)dev-environment-0-0. Actor: system
...
DEBUG    dstack._internal.server.background.pipeline_tasks.jobs_submitted:2240 job(0eaa31)dev-environment-0-0: trying t3.micro in aws/us-east-1 for $0.0104 per hour
...
INFO     dstack._internal.server.services.events:205 Emitting event: Job status changed SUBMITTED -> PROVISIONING. Event targets: job(0eaa31)dev-environment-0-0. Actor: system

Additional information

JobSubmittedWorker.process, first pass:

  • _process_assignment -> _select_assignment picks a fleet and returns _NewCapacityAssignment
  • _apply_assignment_result creates a placeholder instance and sets both JobModel.instance_assigned and JobModel.instance

JobSubmittedWorker.process, second pass:

  • _process_provisioning, job_model.instance is set -> _process_new_capacity_provisioning, run.run_spec.merged_profile.creation_policy == CreationPolicy.REUSE check below is unreachable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions