Skip to content

[Bug]: CloudRift VM reports ready but SSH host_address is unreachable #4003

Closed

Description

@peterschmidt85

Steps to reproduce

  1. Configure the CloudRift backend.
  2. Run a minimal CloudRift task:
type: task
name: cloudrift-repro

image: nvidia/cuda:12.4.1-base-ubuntu22.04
commands:
  - nvidia-smi

resources:
  gpu: 24GB..:1

backends: [cloudrift]
dstack apply -f cloudrift-repro.dstack.yml -y -d

Actual behaviour

The run stays in provisioning.

CloudRift reports the VM as ready/running and returns a host address, but that host is not reachable on SSH from the dstack server. Because dstack cannot reach the shim over SSH, provisioning does not continue.

Expected behaviour

Once CloudRift reports the VM ready and returns a host address, dstack should be able to reach the shim over SSH. If the host is not reachable, provisioning should fail with a clear error.

dstack version

Development checkout, uv run dstack --version reports 0.0.0.

Additional information

Observed in one repro:

region=ap-east-tw-kn-1
instance_type=rtx49-10c-kn.1
instance_id=70bb964a-77b0-11f1-bc41-ab4db0e59660

CloudRift state for the instance:

status=Active
node_status=Ready
node_mode=VirtualMachine
host_address=211.21.50.85
internal_host_address=10.21.106.99
vm_ready=true
vm_state=Running

dstack used the same host as the SSH target:

hostname=211.21.50.85
username=riftuser
ssh_port=22

Network check from the dstack server host:

TCP connect to 211.21.50.85:22 timed out repeatedly.
SSH to riftuser@211.21.50.85:22 timed out.

This reproduced three times. Each run got the same public host_address=211.21.50.85; internal host addresses differed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions