Repository navigation
[Bug]: Fleet and run GPU names are combined case-sensitively, dropping all backend offers #4214
Description
Steps to reproduce
- Apply a fleet whose
resources.gpuname is written in lowercase:
type: fleet
name: my-fleet
nodes: 0..
resources:
gpu: h100:1..- Apply a run configuration requesting the same GPU with a different case:
type: task
name: my-task
commands:
- true
resources:
gpu: H100:1Actual behaviour
The run plan shows no backend offers. Only offers from instances that already exist in the fleet are listed, so a run that has no reusable instance to land on gets No matching instance offers available. Writing h100 instead of H100 in step 2 brings the backend offers back.
Backend offers are collected per candidate fleet, and the fleet's requirements are intersected with the run's by _combine_gpu():
dstack/src/dstack/_internal/server/services/requirements/combine.py
Lines 199 to 207 in db06abf
| def _combine_gpu(value1: GPUSpec, value2: GPUSpec) -> GPUSpec: | |
| return GPUSpec( | |
| vendor=get_single_value_optional(value1.vendor, value2.vendor), | |
| name=_intersect_lists_optional(value1.name, value2.name), | |
| count=_combine_range(value1.count, value2.count), | |
| memory=_combine_range_optional(value1.memory, value2.memory), | |
| total_memory=_combine_range_optional(value1.total_memory, value2.total_memory), | |
| compute_capability=_get_min_optional(value1.compute_capability, value2.compute_capability), | |
| ) |
The GPU name intersection is a plain case-sensitive in test:
dstack/src/dstack/_internal/server/services/requirements/combine.py
Lines 112 to 121 in db06abf
| def _intersect_lists_optional( | |
| list1: Optional[list[_T]], list2: Optional[list[_T]] | |
| ) -> Optional[list[_T]]: | |
| if list1 is None: | |
| if list2 is None: | |
| return None | |
| return list2.copy() | |
| if list2 is None: | |
| return list1.copy() | |
| return [x for x in list1 if x in list2] |
So ["h100"] ∩ ["H100"] yields [], and an empty (non-None) name list matches no GPU at all, which drops every backend offer the fleet could contribute. Offers from existing instances survive because that path matches GPU names case-insensitively (gpu_matches_gpu_spec(), gpuhunt.matches()), which is also why dstack offer --gpu H100:1 still returns the offers - it ignores fleets.
Expected behaviour
GPU names are compared case-insensitively when combining fleet and run requirements, as they are everywhere else in offer matching. dstack offer and dstack apply should agree.
dstack version
master (db06abf)
Additional information
_intersect_lists_optional() is also used for regions, availability_zones, and instance_types when combining fleet and run profiles, and all three are lowercased before offers are filtered in _filter_offers(), so they have the same latent problem. backends is an enum and is unaffected.
Minimal check without a backend:
from dstack._internal.core.models.resources import GPUSpec
from dstack._internal.server.services.requirements.combine import _combine_gpu
fleet = GPUSpec(name=["h100"], count="1..")
print(_combine_gpu(fleet, GPUSpec(name=["h100"], count="1")).name) # ['h100']
print(_combine_gpu(fleet, GPUSpec(name=["H100"], count="1")).name) # []
Activity