[Feature]: Zero-downtime in-place update for services

Problem

Currently, when re-deploying a service (e.g., a newer version), dstack stops the existing run before submitting the new one. This causes a disruption, which is unacceptable for any production-grade service.

Solution

Introduce a revision number for service replicas.
During an in-place update, ensure that older replicas are only stopped once the newer replicas are successfully running.

Workaround

No response

Would you like to help us implement this feature by sending a PR?

Yes

There is a hacky workaround to achieve zero-downtime deployment. It works if there are only changes in the repo and not in the service configuration.

Add at least one extra replica in the replicas configuration property and reapply the configuration. This will upload the new repo revision to dstack and start a new replica/replicas to handle traffic while the other replicas are being redeployed.
Manually stop the old replicas by making your application exit cleanly. For example, attach to a replica with dstack attach --replica <replica num> <run name>, SSH into the container, and send a signal to your application. This will force dstack to redeploy the stopped replicas using the new repo revision.
Remove the redundant replica(s) from the replicas configuration property and reapply the configuration.

This workaround is very limited and inconvenient, so we are interested in supporting proper zero-downtime deployments orchestrated by dstack.

@jvstme

Hi @jvstme

I've tried the workaround, but I ran into some issues with scaling (adding new replicas).

I've tried to scale from 1 replica to 2. When my app was starting, the API became unstable, and ~50% of requests were failing because they were directed to the replica where the app was not ready yet. I got 404 because my routes were not added yet.

It takes a few minutes to start my app because it has to load all the models on the GPU, etc. So I cannot use scaling because it degrades the service.

Maybe we can add a health check in nginx to make sure that the requests are not sent to the broken replica?

This is the issue not only for this workaround but for the scaling in general.

@movchan74

Hi @movchan74

I think Nginx has built-in availability checks enabled by default, but they only consider the replica unavailable if Nginx can't communicate with it at all, e.g. if the app is not bound to the port. So if you could adjust your app to only bind to the port once it's ready, it should work.

But many apps can't be implemented that way, so I agree that we'll need to support health checks. We intend to support them natively in #2181. I think this will be a prerequisite for the current ticket

@jvstme

Hi @jvstme, thanks for the pointers so far!

I’ve managed to get around Nginx routing to unready replicas by proxying 127.0.0.1:8001 → 8000 with socat, so the new replica only appears healthy once it’s fully up. Unfortunately, I still can’t get the zero-downtime update to actually pick up my new code:

Scaling the run (e.g. from 1→2 replicas) keeps using the old repo_hash.
Manually bumping repo_hash in the config run fails with
```
Cannot override active run. Stop the run first.
```

It looks like runs are immutable by design, so I can’t do an in-place code swap. Do you have any suggestions for safely rolling out a new commit with zero downtime, or should we try something else entirely until #2180 lands?

Here’s the script I’m using to scale runs for reference:
https://gist.github.com/movchan74/4a3799dbd5577af0ab98320f1eb7402f

Thanks!

@movchan74

@movchan74, indeed, I've tested it again and found that only the uncommitted diff (repo_code_hash) can be updated without stopping the run. Deploying a new commit (repo_hash) currently requires stopping the run.

So far, I can't suggest any other workarounds. Then we'll try to prioritize #2180 so that it lands sooner

@jvstme

Hi @jvstme, thanks for confirming my suspicion. I’ll hang tight for #2180. I appreciate you prioritizing this!

@movchan74

@movchan74, I believe you mentioned blue-green deployments before. In the future, we can support different deployment strategies, including blue-green. However, we are considering starting with rolling deployments first.

Service replicas will be updated one by one to avoid using too many instances at once. The suggested algorithm is:

Start a new replica.
Wait until the new replica becomes running or healthy (once we implement [Feature]: Allow to configure a service healthcheck #2181).
Stop the old replica.
If there are more out-of-date replicas, go to step 1 for the next replica.

If the service only has one replica, this strategy is similar to blue-green deployments, except there can be a short period of time (a few seconds) when the new and the old replica coexist and both handle requests.

Will this work for you?

@jvstme

Hi @jvstme, that rolling-upgrade approach makes sense. It would already help quite a lot!

My only concern is that we still wouldn’t be able to change any service-level config (environment variables, commands, etc.) without stopping the run entirely. Is that something you’d plan to address later, or is it intended that configuration changes will always force a full redeploy?

@movchan74

@movchan74, we plan to address that at once. Changes to properties such as env and commands will be handled the same way as repo changes. For each replica, dstack will first start a new replica version using the updated properties, then stop the old replica version

Great, that addresses my concern. Happy to test and provide feedback when ready.

@movchan74

@movchan74, our latest 0.19.15 release includes initial support for rolling deployments. Feel free to check it out to see how it works.

However, I assume this version will not work for your use case yet:

So far, it's only possible to update certain configuration properties, but not deploy new repo commits. I will address this next in the same issue.
Without health checks (or readiness probes), dstack stops the old replica as soon as the new replica becomes running. This results in downtime if the new replica takes some time to become ready. We will address this soon in [Feature]: Allow to configure a service healthcheck #2181.

@movchan74

@movchan74, rolling deployments for repo commits are available in 0.19.17. Health checks are on the way.

@jvstme

@jvstme let's close it

Navigation Menu

Repository navigation

[Feature]: Zero-downtime in-place update for services #2180

Problem

Solution

Workaround

Would you like to help us implement this feature by sending a PR?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature]: Zero-downtime in-place update for services #2180

Description

Problem

Solution

Workaround

Would you like to help us implement this feature by sending a PR?

Activity

jvstme commented on Apr 9, 2025

movchan74 commented on Apr 30, 2025

jvstme commented on Apr 30, 2025

movchan74 commented on May 19, 2025

jvstme commented on May 21, 2025

movchan74 commented on May 21, 2025

jvstme commented on Jun 4, 2025

movchan74 commented on Jun 4, 2025

jvstme commented on Jun 4, 2025

movchan74 commented on Jun 4, 2025

jvstme commented on Jun 20, 2025

jvstme commented on Jul 2, 2025

r4victor commented on Jul 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions