Skip to content

[Bug]: Service is terminated if scaling fails #1979

Closed as not planned
Closed as not planned

Description

@jvstme

Steps to reproduce

This can be reproduced both with auto-scaling and with manual in-place update. This example uses in-place update.

  1. Provision a fleet with one instance.
> cat fleets/cloud.dstack.yml
type: fleet
name: cloud
nodes: 1
spot_policy: auto

> dstack apply -f fleets/cloud.dstack.yml -y
  1. Run a single-replica service with --reuse.
> cat services/.dstack.yml 
type: service
name: my-service

commands:
  - python -m http.server
port: 8000

replicas: 1

spot_policy: auto

> dstack apply -f services/.dstack.yml --reuse -y
  1. Try scaling the service to two replicas by updating and reapplying the configuration with --reuse.
> cat services/.dstack.yml 
type: service
name: my-service

commands:
  - python -m http.server
port: 8000

replicas: 2

spot_policy: auto

> dstack apply -f services/.dstack.yml --reuse -y

Actual behaviour

The second replica fails because there are no idle instances to reuse. The run is terminated because the second replica failed to start.

 #  BACKEND  REGION    INSTANCE       RESOURCES                   SPOT  PRICE         
 1  gcp      us-west4  e2-standard-2  2xCPU, 8GB, 100.0GB (disk)  yes   $0.0095  busy 

Active run my-service already exists. Detected configuration changes that can be updated in-place: ['replicas']
my-service provisioning completed (running)
Service is published at http://localhost:3000/proxy/services/ilya/my-service/

Serving HTTP on 0.0.0.0 port 8000 (http://localhost:3000/proxy/services/ilya/my-service/) ...
Run failed with error code TERMINATED_BY_SERVER.
Check CLI, server, and run logs for more details

Expected behaviour

The second replica fails because there are no idle instances to reuse. The run remains running.

dstack version

0.18.24

Server logs

INFO     dstack._internal.server.services.runs:861 run(46bef6)my-service: scaling UP 1 replica(s)
[09:06:24] DEBUG    dstack._internal.server.background.tasks.process_submitted_jobs:99 job(c9251a)my-service-0-1: provisioning has started
[09:06:29] DEBUG    dstack._internal.server.background.tasks.process_submitted_jobs:99 job(c9251a)my-service-0-1: provisioning has started
           DEBUG    dstack._internal.server.background.tasks.process_submitted_jobs:222 job(c9251a)my-service-0-1: reuse instance failed
           INFO     dstack._internal.server.background.tasks.process_runs:330 run(46bef6)my-service: run status has changed RUNNING -> TERMINATING
           INFO     dstack._internal.server.services.jobs:283 job(c9251a)my-service-0-1: job status is FAILED, reason: FAILED_TO_START_DUE_TO_NO_CAPACITY
           DEBUG    dstack._internal.server.services.jobs:192 job(c1f616)my-service-0-0: stopping runner 34.16.243.7
           DEBUG    dstack._internal.server.services.jobs:234 job(c1f616)my-service-0-0: stopping container
[09:07:02] INFO     dstack._internal.server.services.jobs:268 job(c1f616)my-service-0-0: instance 'cloud-0' has been released, new status is IDLE
           INFO     dstack._internal.server.services.jobs:283 job(c1f616)my-service-0-0: job status is TERMINATED, reason: TERMINATED_BY_SERVER
           INFO     dstack._internal.server.services.runs:848 run(46bef6)my-service: run status has changed TERMINATING -> FAILED, reason: JOB_FAILED

Additional information

It is important to keep the existing replica running to avoid service downtime. One of the ideas of having multiple replicas is increasing fault tolerance, so failures of one replica should not affect other replicas.

Activity

  1. github-actions commented on Dec 12, 2024

    @github-actions

    This issue is stale because it has been open for 30 days with no activity.

  2. github-actions commented on Dec 27, 2024

    @github-actions

    This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.

  3. peterschmidt85 commented on Dec 28, 2024

    @peterschmidt85
    Contributor

    @jvstme is this issue relevant still?

  4. jvstme commented on Dec 30, 2024

    @jvstme
    CollaboratorAuthor

    Yes

  5. reopened this on Dec 30, 2024
  6. github-actions commented on Jan 31, 2025

    @github-actions

    This issue is stale because it has been open for 30 days with no activity.

  7. github-actions commented on Apr 18, 2025

    @github-actions

    This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstale

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions