[Bug]: AWS cluster fleets fail with NO_OFFERS when using a Capacity Block reservation

Steps to reproduce

Purchase an AWS Capacity Block (e.g. p5.48xlarge) with enough available instances for the fleet.
Apply a fleet that targets it with cluster placement:

type: fleet
name: cb-cluster
backends: [aws]
instance_types: [p5.48xlarge]
nodes: 2
placement: cluster
resources:
  gpu: 8
spot_policy: on-demand
reservation: cr-<capacity-block-id>

Actual behaviour

dstack apply shows a valid offer in the plan, but the master instance is terminated with NO_OFFERS ("All offers failed"), and so is the fleet. The same fleet without placement: cluster provisions successfully, but it can't run multi-node tasks, because those require a cluster fleet.

The master instance's launch request includes the fleet's cluster placement group, which AWS rejects for Capacity Blocks: "Capacity Blocks do not support placement groups. You can't specify a placement group when you launch instances into a Capacity Block" (docs). dstack only skips placement groups for reservations when are_placement_groups_compatible_with_reservations() returns False:

dstack/src/dstack/_internal/server/background/pipeline_tasks/instances/cloud_provisioning.py

Lines 139 to 147 in 90887d0

    
               cluster_context is not None 
        
               and cluster_context.is_current_instance_master 
        
               and instance_offer.backend in BACKENDS_WITH_PLACEMENT_GROUPS_SUPPORT 
        
               and isinstance(compute, ComputeWithPlacementGroupSupport) 
        
               and ( 
        
                   compute.are_placement_groups_compatible_with_reservations(instance_offer.backend) 
        
                   or instance_configuration.reservation is None 
        
               ) 
        
           ):

GCP overrides it, but AWS inherits the default True:

dstack/src/dstack/_internal/core/backends/base/compute.py

Lines 562 to 570 in 90887d0

    
               def are_placement_groups_compatible_with_reservations(self, backend_type: BackendType) -> bool: 
        
                   """ 
        
                   Whether placement groups can be used for instances provisioned in reservations. 
        
                   Arguments: 
        
                       backend_type: matches the backend type of this compute, unless this compute is a proxy 
        
                           for other backends (dstack Sky) 
        
                   """ 
        
                   return True

So the AWS backend sends Placement.GroupName together with MarketType: capacity-block. The ClientError is only logged, and every offer ends up counted as failed:

dstack/src/dstack/_internal/core/backends/aws/compute.py

Lines 377 to 390 in 90887d0

    
                       placement_group_name=placement_group.name if placement_group else None, 
        
                       enable_efa=enable_efa, 
        
                       max_efa_interfaces=max_efa_interfaces, 
        
                       reservation_id=instance_config.reservation, 
        
                       is_capacity_block=is_capacity_block, 
        
                       tenancy=reservation_tenancy, 
        
                   ) 
        
               ) 
        
           except botocore.exceptions.ClientError as e: 
        
               logger.warning("Got botocore.exceptions.ClientError: %s", e) 
        
               if e.response["Error"]["Code"] == "InvalidParameterValue": 
        
                   msg = e.response["Error"].get("Message", "") 
        
                   raise ComputeError(f"Invalid AWS request: {msg}") 
        
               continue

The fleet plan doesn't catch this because planning never creates a placement group or launches an instance. The same check also gates placement groups in jobs_submitted.py, so run-triggered provisioning into a cluster fleet is affected too.

Expected behaviour

A cluster fleet in an AWS Capacity Block provisions all nodes without a placement group. AWS already co-locates Capacity Block instances in an EC2 UltraCluster, and EFA is still enabled. Placement groups should still be used with regular on-demand capacity reservations.

dstack version

master (90887d0)

Additional information

A possible fix is to make are_placement_groups_compatible_with_reservations() reservation-aware (e.g. pass the reservation) and have AWS return False when ReservationType == "capacity-block".

The docs for placement: cluster also don't say what it maps to on each backend (on AWS, an EC2 cluster placement group).

Skip placement groups for AWS Capacity Blocks (#4345)

Navigation Menu

Repository navigation

[Bug]: AWS cluster fleets fail with NO_OFFERS when using a Capacity Block reservation #4344

Steps to reproduce

Actual behaviour

Expected behaviour

dstack version

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	cluster_context is not None
	and cluster_context.is_current_instance_master
	and instance_offer.backend in BACKENDS_WITH_PLACEMENT_GROUPS_SUPPORT
	and isinstance(compute, ComputeWithPlacementGroupSupport)
	and (
	compute.are_placement_groups_compatible_with_reservations(instance_offer.backend)
	or instance_configuration.reservation is None
	)
	):

	def are_placement_groups_compatible_with_reservations(self, backend_type: BackendType) -> bool:
	"""
	Whether placement groups can be used for instances provisioned in reservations.

	Arguments:
	backend_type: matches the backend type of this compute, unless this compute is a proxy
	for other backends (dstack Sky)
	"""
	return True

	placement_group_name=placement_group.name if placement_group else None,
	enable_efa=enable_efa,
	max_efa_interfaces=max_efa_interfaces,
	reservation_id=instance_config.reservation,
	is_capacity_block=is_capacity_block,
	tenancy=reservation_tenancy,
	)
	)
	except botocore.exceptions.ClientError as e:
	logger.warning("Got botocore.exceptions.ClientError: %s", e)
	if e.response["Error"]["Code"] == "InvalidParameterValue":
	msg = e.response["Error"].get("Message", "")
	raise ComputeError(f"Invalid AWS request: {msg}")
	continue

[Bug]: AWS cluster fleets fail with NO_OFFERS when using a Capacity Block reservation #4344

Description

Steps to reproduce

Actual behaviour

Expected behaviour

dstack version

Additional information

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions