Repository navigation
[Bug]: Service is terminated if scaling fails #1979
Copy link
Copy link
Closed as not planned
Labels
Description
Steps to reproduce
This can be reproduced both with auto-scaling and with manual in-place update. This example uses in-place update.
- Provision a fleet with one instance.
> cat fleets/cloud.dstack.yml
type: fleet
name: cloud
nodes: 1
spot_policy: auto
> dstack apply -f fleets/cloud.dstack.yml -y- Run a single-replica service with
--reuse.
> cat services/.dstack.yml
type: service
name: my-service
commands:
- python -m http.server
port: 8000
replicas: 1
spot_policy: auto
> dstack apply -f services/.dstack.yml --reuse -y- Try scaling the service to two replicas by updating and reapplying the configuration with
--reuse.
> cat services/.dstack.yml
type: service
name: my-service
commands:
- python -m http.server
port: 8000
replicas: 2
spot_policy: auto
> dstack apply -f services/.dstack.yml --reuse -yActual behaviour
The second replica fails because there are no idle instances to reuse. The run is terminated because the second replica failed to start.
# BACKEND REGION INSTANCE RESOURCES SPOT PRICE
1 gcp us-west4 e2-standard-2 2xCPU, 8GB, 100.0GB (disk) yes $0.0095 busy
Active run my-service already exists. Detected configuration changes that can be updated in-place: ['replicas']
my-service provisioning completed (running)
Service is published at http://localhost:3000/proxy/services/ilya/my-service/
Serving HTTP on 0.0.0.0 port 8000 (http://localhost:3000/proxy/services/ilya/my-service/) ...
Run failed with error code TERMINATED_BY_SERVER.
Check CLI, server, and run logs for more detailsExpected behaviour
The second replica fails because there are no idle instances to reuse. The run remains running.
dstack version
0.18.24
Server logs
INFO dstack._internal.server.services.runs:861 run(46bef6)my-service: scaling UP 1 replica(s)
[09:06:24] DEBUG dstack._internal.server.background.tasks.process_submitted_jobs:99 job(c9251a)my-service-0-1: provisioning has started
[09:06:29] DEBUG dstack._internal.server.background.tasks.process_submitted_jobs:99 job(c9251a)my-service-0-1: provisioning has started
DEBUG dstack._internal.server.background.tasks.process_submitted_jobs:222 job(c9251a)my-service-0-1: reuse instance failed
INFO dstack._internal.server.background.tasks.process_runs:330 run(46bef6)my-service: run status has changed RUNNING -> TERMINATING
INFO dstack._internal.server.services.jobs:283 job(c9251a)my-service-0-1: job status is FAILED, reason: FAILED_TO_START_DUE_TO_NO_CAPACITY
DEBUG dstack._internal.server.services.jobs:192 job(c1f616)my-service-0-0: stopping runner 34.16.243.7
DEBUG dstack._internal.server.services.jobs:234 job(c1f616)my-service-0-0: stopping container
[09:07:02] INFO dstack._internal.server.services.jobs:268 job(c1f616)my-service-0-0: instance 'cloud-0' has been released, new status is IDLE
INFO dstack._internal.server.services.jobs:283 job(c1f616)my-service-0-0: job status is TERMINATED, reason: TERMINATED_BY_SERVER
INFO dstack._internal.server.services.runs:848 run(46bef6)my-service: run status has changed TERMINATING -> FAILED, reason: JOB_FAILEDAdditional information
It is important to keep the existing replica running to avoid service downtime. One of the ideas of having multiple replicas is increasing fault tolerance, so failures of one replica should not affect other replicas.
Reactions are currently unavailable
Activity
github-actions commentedon Dec 12, 2024
This issue is stale because it has been open for 30 days with no activity.
github-actions commentedon Dec 27, 2024
This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.
peterschmidt85 commentedon Dec 28, 2024
@jvstme is this issue relevant still?
jvstme commentedon Dec 30, 2024
Yes
github-actions commentedon Jan 31, 2025
This issue is stale because it has been open for 30 days with no activity.
github-actions commentedon Apr 18, 2025
This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.