Repository navigation
[Bug]: Can over-allocate instances in fleet past nodes-max when using rest API #3663
Description
Steps to reproduce
-
Spin up dstack server (I've only tested using AWS as backend with a pretty simple layout)
-
Add a fleet with a node max:
fleet.yml
name: test-fleet
nodes: 0..3
resources:
cpu: 1..
- write a script to shove tasks in via the rest API:
import requests
import json
token = "*****"
header = {
"Content-Type": "application/json",
"Accept": "application/json",
"Authorization": f"Bearer {token}"
}
server_url = "http://127.0.0.1:3000"
project_name = "main"
sub_url = "runs/apply"
runs = 10
for i in range(runs):
data = {
"force": True,
"plan": {
"run_spec": {
"configuration": {
"type": "task",
"image": "ubuntu:22.04",
"creation_policy": "reuse-or-create",
"commands": ["date", "sleep 6", "date"],
"fleets": ["test-fleet"]
}
}
}
}
requests.post(f"{server_url}/api/project/{project_name}/{sub_url}",
headers=header, data=json.dumps(data))
- Run it (in my case, on the server)
Actual behaviour
The runs appear as expected:
But extra instances appear, more than the node max:
And these are real instances, not just phantoms, as seen here in my AWS console:
Expected behaviour
Given that the max nodes on the fleet was set to 3, I would've expected only 3 instances to spin up.
dstack version
20.12
Server logs
<Full logs as file below>
Activity
ASmedberg-woolpert commentedon Mar 12, 2026
Forgot to put this in above:
Even working at a semi-human time scale doesn't appear to fix the problem. I put an arbitrary
time.sleep(1)into my for loop and still got similar results, although I did get fewer (4) instances. I had to up that to time.sleep(3) to actually get the expected result (only 3 instances)r4victor commentedon Mar 13, 2026
@ASmedberg-woolpert, over provisioning is currently possible but there is a background logic that terminates all extra instances as soon as they are idle. So the fleet
nodeslimits are soft. This is a temporary limitation because hard limit would require a long fleet lock and make provisioning longer. We're planning to change this soon with #3665 and #3551 – already Work in Progress.github-actions commentedon Apr 13, 2026
This issue is stale because it has been open for 30 days with no activity.