Skip to content

[Feature]: Balance and fail over in-run server access across server replicas #4033

Closed as not planned

Description

@peterschmidt85

Problem

PR #4023 exposes the dstack server inside a run through /run/dstack/server.sock.

With multiple dstack server replicas, the current implementation keeps one stable owner for this socket. Other replicas probe and adopt the existing socket instead of replacing it.

This avoids ownership churn, but leaves two limitations:

  1. Load can be distributed unevenly. The first replica that establishes the socket for a job handles all server requests from that job. Depending on job processing and server startup timing, one replica may own the sockets for most or all jobs.
  2. Failover is delayed. If the owning replica stops or its tunnel fails, requests from the job fail until another replica detects the unhealthy socket and replaces it. Detection is currently throttled by the 30-second connection health-check interval and can be delayed further by job processing and locking.

Solution

Allow every server replica to expose its own Unix socket in the job, and place a load-balancing/failover layer in front of those sockets at /run/dstack/server.sock.

The gateway implementation provides a similar model: each server replica creates a uniquely named socket, while HTTPMultiClient selects among available sockets and retries alternatives. The runner could provide equivalent routing inside the job.

The implementation should:

  • distribute requests across healthy server replicas;
  • retry another replica when the selected replica is unavailable;
  • add and remove replica sockets without interrupting the stable /run/dstack/server.sock endpoint;
  • clean up stale replica sockets;
  • preserve support for multiple server processes sharing PostgreSQL.

Activity

  1. github-actions commented on Aug 15, 2026

    @github-actions

    This issue is stale because it has been open for 30 days with no activity.

  2. github-actions commented on Aug 30, 2026

    @github-actions

    This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions