You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature]: Balance and fail over in-run server access across server replicas #4033
PR #4023 exposes the dstack server inside a run through /run/dstack/server.sock.
With multiple dstack server replicas, the current implementation keeps one stable owner for this socket. Other replicas probe and adopt the existing socket instead of replacing it.
This avoids ownership churn, but leaves two limitations:
Load can be distributed unevenly. The first replica that establishes the socket for a job handles all server requests from that job. Depending on job processing and server startup timing, one replica may own the sockets for most or all jobs.
Failover is delayed. If the owning replica stops or its tunnel fails, requests from the job fail until another replica detects the unhealthy socket and replaces it. Detection is currently throttled by the 30-second connection health-check interval and can be delayed further by job processing and locking.
Solution
Allow every server replica to expose its own Unix socket in the job, and place a load-balancing/failover layer in front of those sockets at /run/dstack/server.sock.
The gateway implementation provides a similar model: each server replica creates a uniquely named socket, while HTTPMultiClient selects among available sockets and retries alternatives. The runner could provide equivalent routing inside the job.
The implementation should:
distribute requests across healthy server replicas;
retry another replica when the selected replica is unavailable;
add and remove replica sockets without interrupting the stable /run/dstack/server.sock endpoint;
clean up stale replica sockets;
preserve support for multiple server processes sharing PostgreSQL.
Activity
github-actions commentedon Aug 15, 2026
This issue is stale because it has been open for 30 days with no activity.
github-actions commentedon Aug 30, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.