[Bug]: Rolling deployment of type: sglang router is allowed but broken

Steps to reproduce

type: service
name: smg

env:
  - MODEL_ID=Qwen/Qwen2.5-0.5B-Instruct

groups:

  - replicas: 1
    python: "3.12"
    commands:
      - pip install smg
      - |
        smg launch \
          --host 0.0.0.0 \
          --port 8000 \
          --enable-igw \
          --policy cache_aware \
          --model-path $MODEL_ID
    router:
      type: sglang
    resources:
      gpu: 0

  - replicas: 1
    image: lmsysorg/sglang:v0.5.10.post1
    commands:
      - |
        python3 -m sglang.launch_server \
          --model-path $MODEL_ID \
          --host 0.0.0.0 \
          --port 8000 \
          --context-length 4096
    resources:
      gpu:
        memory: 16GB..
        compute_capability: 8.0

port: 8000

model:
  type: chat
  name: Qwen/Qwen2.5-0.5B-Instruct
  format: openai
  prefix: /v1

#probes:
#  - type: http
#    url: /health_generate
#    interval: 10s

Apply the configuration, wait until the service is running
Change the router configuration, e.g., adjust commands, apply again, confirm in-place update

Actual behaviour

Without `probes`

Without probes specified, dstack checks router's /health, which is always 200 OK — the probe passes as soon as SMG's HTTP server is running:

the new router becomes ready without workers – ServiceRouterWorkerSyncPipeline only syncs the oldest replica (the old router);
the new router is registered on the gateway → the gateway distributes requests using the round-robin method → ≈50% of requests fail (No worker available for model)
the old router is terminated → 100% of requests fail (No worker available for model)
ServiceRouterWorkerSyncPipeline syncs the new router → the new router is ready to serve

From the client's point of view, the outage looks like this:

while :; do curl -s https://smg.dstack.example.com/generate -X POST -H 'Content-Type: application/json' -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","text":"test"}' -s | head -c100; echo; done

# old router only
{"text":"-example-chapter\n\n# Participation in group work discussions the problem beyond Jan 1\n\nT
{"text":" data for slim SEANTS iPSS (aka insulin/ glucagon binding site).\n\nWhich foods lots of peo
{"text":".angularPath = require('angular-path')\r\n\r\ndescribe 'angular-path', ->\n  controller: Rx
{"text":".com/ciphers/101-sims.runners-of-our-selected-ciphers\r\n\n\n  2. List the Americans who sh
{"text":"/sec.hidden\r\nusing System;\r\nusing Xunit;\r\n\r\nnamespace BeadsOfRain.Tests\r\n{\r\n\t[
{"text":".py\r\n************\r\n#\r\n\r\nThis file contains several scaffolding tasks to get started
{"text":" removeDigits (int num) {\nwhile( num > 0 ) {\nif (num % 10)\nreturn \"0\" + removeDigits(n
{"text":" template for software design project\n\nI'm working on a project to design a real-time vid
{"text":"-timing-and-load-for-20esio-architecture-known-algorithms-of-20esio-for-loop-slice-for-loop

# rolling deployment started – both replicas are running, half of requests failed
{"text":"_FEC_QPL2018:\n        MODELS += fedtorch\n        DATA += qpl_2018_large\n        DATA.Tes
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":".error = self.text\n```\n\nThis should work to print the error in the text area, but it ass
{"text":" = int(input())\nfor test in range(test):\n    n = int(input())\n    t = int(input())\n
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":"_FastThermal01(std::vector<double> &params\n, GlobalFrame &g_frame) {\ng_frame.stdin(\"test
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":"=raw_input()\r\nfor i in range(len(test)):\r\n    if ord(test[i])>90:\r\n        print test
{"text":" dog September, 1999\n\ndate I decided to let my dog commit some infidelity against a neigh
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":" #{test_code} >> #{output_file}\nexit 0  # success to generate tests output file\n```vagran
{"text":"2::Test2()\n: base(\"<Test2>\", TimerQueueBase::kPriorityUnspecified, kServicePositionFree)
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":"ParameterPromoFeatures::\n~testParameterPromoFeatures() {\n  std::cout << \"cppcheck testPa
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":".STressAnalyticProbe_Draw ( $this => $this, $domainName =>ref 'dummy domain name', $width =
{"text":"-code-mre9/putty...\n\n```js\nfunction main() {\n    // Get a list of all text files in the
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":"-code-constriktor-line-no-2233\n\n# Specify a remainder processor for a Zope AppContainer\n
{"text":"code = open(\"testcase1.txt\",\"w\")\nnum1 , num2 , num3 = 9 , 8 , 7\nnum1 , num3 = num3 ,
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":"ACSIOManager::ShadowTimeDelegator::~ShadowTimeDelegator() {\n  required_fvp.reset(\n      s
{"text":"-fs=s1;str1=\"s1\"\n\nwhile test-fs==s1; do\n    echo \"now File System is $test-fs\"\n
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"text":" syntax for propagating the multivalue operations. Here is how to do this in your environme
{"text":"构造函数在原始类中能否被访问\n\n在C++中，`constructor()`是构造函数的

# old replica de-registered, all requests failed
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen
{"error":{"type":"Not Found","code":"model_not_found","message":"No worker available for model 'Qwen

# new replica synced, service is back
{"text":" is false for all cacheable file systems. In the version 2.2, cacheable based�
{"text":"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/javascript/example.html [ETA]\n\n  • Write a brief article on the inspiration for the ti
{"text":"category::testcategory(QWidget *parent) :\n    QDialog(parent),\n    ui(new Ui::testcategor
{"text":"1 = (1, 1)\nclass Solution(object):\n    def isValid(self, s):\n        \"\"\"\n        :ty
{"text":" FSLext already included in the package before @package some_very_useful_module\n\ndef rere
{"text":" reveb to V4\n\nRevision BC8869c5d5f6ccf7b0e116827e453e2dc39b5027 authored by Manuel Perez-
{"text":"再发布(2014) - 兰谷子\n\n[问题]在'use outlets'语句中,出头字母为\n'B'的出
{"text":"題_1234560 \r\n\r\nhttps://arrow/mysql/examples/documentation/source/\n\nConsider the foll
{"text":"�んでいるメインルール情報を判別してprintsというタイミングの whe
{"text":"-tags {plevel1} R Documentation\n\n## Formal tags in Power Query Rendering\n\n### Descripti
{"text":"圣三一双精会不会对身体有害?会安装发动机还是高速机组?\n\n神经像�
{"text":"-numo-chef/django-pgbackend\n\n/django-pgbackend/create_aws_instances.py\nimport pymysql\ni
{"text":".random_carry_ports\n\ndef random_port():\n    from random import uniform\n    x = uniform(
{"text":"鬘\nAnna Hijukidoku - An elaborate system of underwear made solely from seaweed, intended
{"text":" = { 'callables': [], 'closer': [], 'functions': [], 'errors': [], 'neighbors': [] } # Init
{"text":"\"\ncompileTown = compileTownMonth(smallTown, 2)\n    \n# Find outdated towns not in Kernsv

With a custom `probes` (uncomment the last section in the configuration)

When a probe depends on workers, the new replica is never ready: the probe fails → not ready → ServiceRouterWorkerSyncPipeline doesn't sync it → no workers → the probe fails — the same kind of deadlock as fixed in #4313

Expected behaviour

No response

dstack version

229f535

Server logs

Additional information

No response

@un-def

@un-def
looks like /health goes green before the SGLang workers are actually ready, then rollout traffic starts failing.

send me the underlying SGLang launch command and i'll give you the Badgr Serve version that waits for real inference.

Sync workers with every running router (dstackai#4320)

Navigation Menu

Repository navigation

[Bug]: Rolling deployment of `type: sglang` router is allowed but broken #4315

Steps to reproduce

Actual behaviour

Without `probes`

With a custom `probes` (uncomment the last section in the configuration)

Expected behaviour

dstack version

Server logs

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Rolling deployment of type: sglang router is allowed but broken #4315

Description

Steps to reproduce

Actual behaviour

Without probes

With a custom probes (uncomment the last section in the configuration)

Expected behaviour

dstack version

Server logs

Additional information

Activity

michaelmanly commented on Sep 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[Bug]: Rolling deployment of `type: sglang` router is allowed but broken #4315

Without `probes`

With a custom `probes` (uncomment the last section in the configuration)