Steps to reproduce
-
Create a fleet:
dstack fleet -v
NAME NODES RESOURCES SPOT BACKEND PRICE STATUS CREATED ERROR
backend 0.. - auto * - active 30 mins ago
-
Submit a run with creation_policy: reuse/--reuse:
Project main
User dmitry-local-admin
Type dev-environment
Resources cpu=1.. mem=1GB.. disk=1GB.. gpu=0..
Spot policy on-demand
Max price off
Retry policy off
Creation policy reuse
Idle duration 5m
Max duration off
Inactivity duration off
No matching instance offers available. Possible reasons: https://dstack.ai/docs/guides/troubleshooting/#no-offers
Note Creation policy reuse
Actual behaviour
A new instance provisioned
Expected behaviour
The run fails with FAILED_TO_START_DUE_TO_NO_CAPACITY – Could not reuse any instances for this job
dstack version
0.20.28
Server logs
DEBUG dstack._internal.server.services.backends:492 Requesting instance offers from backends: ['aws', 'runpod', 'gcp']
...
INFO dstack._internal.server.services.events:205 Emitting event: Job created on run submission. Status: SUBMITTED. Event targets: job(0eaa31)dev-environment-0-0. Actor: system
...
INFO dstack._internal.server.services.events:205 Emitting event: Instance created for job. Instance status: PENDING. Event targets: instance(2aae42)backend-0, job(0eaa31)dev-environment-0-0. Actor: system
...
DEBUG dstack._internal.server.background.pipeline_tasks.jobs_submitted:2240 job(0eaa31)dev-environment-0-0: trying t3.micro in aws/us-east-1 for $0.0104 per hour
...
INFO dstack._internal.server.services.events:205 Emitting event: Job status changed SUBMITTED -> PROVISIONING. Event targets: job(0eaa31)dev-environment-0-0. Actor: system
Additional information
JobSubmittedWorker.process, first pass:
_process_assignment -> _select_assignment picks a fleet and returns _NewCapacityAssignment
_apply_assignment_result creates a placeholder instance and sets both JobModel.instance_assigned and JobModel.instance
JobSubmittedWorker.process, second pass:
_process_provisioning, job_model.instance is set -> _process_new_capacity_provisioning, run.run_spec.merged_profile.creation_policy == CreationPolicy.REUSE check below is unreachable
Steps to reproduce
Create a fleet:
Submit a run with
creation_policy: reuse/--reuse:Note
Creation policy reuseActual behaviour
A new instance provisioned
Expected behaviour
The run fails with
FAILED_TO_START_DUE_TO_NO_CAPACITY–Could not reuse any instances for this jobdstack version
0.20.28
Server logs
Additional information
JobSubmittedWorker.process, first pass:_process_assignment->_select_assignmentpicks a fleet and returns_NewCapacityAssignment_apply_assignment_resultcreates a placeholder instance and sets bothJobModel.instance_assignedandJobModel.instanceJobSubmittedWorker.process, second pass:_process_provisioning,job_model.instanceis set ->_process_new_capacity_provisioning,run.run_spec.merged_profile.creation_policy == CreationPolicy.REUSEcheck below is unreachable