Repository navigation
[Bug]: Crusoe offers fail because quotas API returns counters as strings #4318
Description
Steps to reproduce
- Configure a
crusoebackend. - Run
dstack offerordstack applyfor any configuration.
Actual behaviour
The Crusoe backend returns no offers, and the server logs a TypeError when comparing quota availability (see server logs).
Crusoe's GET /projects/{project_id}/quotas now returns max, used, and available as JSON strings, although the API docs still document them as integers:
{'programmatic_name': 'h100-80gb-sxm-ib', 'category': 'GPU Instance', 'max': '16', 'used': '0', 'available': '16', 'usage_tracking': 'QUOTA_USAGE_TRACKING_CENTRAL', ...}
_get_quota_map stores available as is, and get_all_offers_with_availability compares it with 0:
dstack/src/dstack/_internal/core/backends/crusoe/compute.py
Lines 156 to 168 in 229f535
| for offer in offers: | |
| family = _get_instance_family(offer.instance.name) | |
| availability = InstanceAvailability.UNKNOWN | |
| for prog_name, available in quota_map.items(): | |
| if family.startswith(prog_name) or prog_name.startswith(family): | |
| availability = ( | |
| InstanceAvailability.AVAILABLE | |
| if available > 0 | |
| else InstanceAvailability.NO_QUOTA | |
| ) | |
| break | |
| result.append(offer.with_availability(availability=availability)) | |
| return result |
dstack/src/dstack/_internal/core/backends/crusoe/compute.py
Lines 170 to 183 in 229f535
| def _get_quota_map(self) -> dict[str, int]: | |
| try: | |
| quotas = self._client.list_quotas() | |
| except Exception: | |
| logger.warning("Failed to fetch Crusoe quotas, availability will be UNKNOWN") | |
| return {} | |
| result = {} | |
| for q in quotas: | |
| prog_name = q.get("programmatic_name", "") | |
| available = q.get("available", 0) | |
| category = q.get("category", "") | |
| if "Instance" in category: | |
| result[prog_name] = available | |
| return result |
This code hasn't changed since the backend was added, so the change is on Crusoe's side. GET /capacities, used by gpuhunt, still returns quantity as an int, which is why the gpuhunt Crusoe catalog job keeps working.
Expected behaviour
Crusoe offers are returned with quota-based availability. If the quotas response can't be parsed, offers should fall back to UNKNOWN availability, as they already do when the quotas request fails, instead of dropping all Crusoe offers.
Possible fix: parse the counters with int(...) so both forms are accepted, and move the parsing inside the existing try.
dstack version
master (229f535)
Server logs
ERROR dstack._internal.server.services.backends:504 Got exception when requesting offers from backend BackendType.CRUSOE
Traceback (most recent call last):
...
File "src/dstack/_internal/core/backends/base/compute.py", line 285, in _get_all_offers_with_availability_cached
return self.get_all_offers_with_availability(unallocated_resources)
File "src/dstack/_internal/core/backends/crusoe/compute.py", line 164, in get_all_offers_with_availability
if available > 0
TypeError: '>' not supported between instances of 'str' and 'int'Additional information
No response
Activity
Parse Crusoe quota counters returned as strings (dstackai#4323)