[Feature]: Store user SSH key on the server

Problem

A user starts a run with dstack apply.
The user switches to another computer.
The user tries to attach to the run using dstack attach.

The user won't be able to attach and will get the Can't connect to the remote host error.

This happens because the SSH key for attaching is generated in CLI and stored on the machine, so different machines will have different keys, even if the user is the same.

Solution

Generate and store users' SSH keys in dstack-server and download them to the client machine when needed.
This ensures that all of a user's machines have the same key, and the only secret the user needs to access their runs from a new machine is the dstack API token.

Workaround 1

Generate a private+public key pair with ssh-keygen and deliver them to all machines where access to the run in desired.
Before running dstack apply, init the repo with dstack init --ssh-identity <path to private key>
To attach to the run, use dstack attach --ssh-identity <path to private key> <run name>

Workaround 2

Manually copy ~/.dstack/ssh/id_rsa and ~/.dstack/ssh/id_rsa.pub from one machine to the rest.

Would you like to help us implement this feature by sending a PR?

Yes

This issue is stale because it has been open for 30 days with no activity.

Implementation plan:

Introduce ssh_private_key and ssh_public_key properties in User in the database
Create an Alembic migration
Generate these keys on user creation (similar to project keys)
Make ssh_key_pub in /runs/get_plan and /runs/apply optional and use new user keys in that case
Let CLI download the user's ssh_private_key and ssh_public_key if they are available - using user token hash as a part of the name; use previous logic if the user key can’t be downloaded - when both submitting the run and attaching to it

Backward-compatibility:

Ensure the new CLI can work with an old server (where User doesn't have ssh_private_key and ssh_public_key)
Ensure the old CLI can work with the new server

@r4victor

@r4victor Please let me know if the plan above is OK to you

@peterschmidt85

@peterschmidt85, 👍

Also preferably:

Ensure the new CLI can attach to runs submitted by the old CLI, regardless of the server version

[Feature]: Store user SSH key on the server #2053

[Feature]: Store user SSH key on the server (#3176)

Navigation Menu

Repository navigation

[Feature]: Store user SSH key on the server #2053

Problem

Solution

Workaround 1

Workaround 2

Would you like to help us implement this feature by sending a PR?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature]: Store user SSH key on the server #2053

Description

Problem

Solution

Workaround 1

Workaround 2

Would you like to help us implement this feature by sending a PR?

Activity

github-actions commented on Jan 3, 2025

peterschmidt85 commented on Oct 8, 2025

peterschmidt85 commented on Oct 8, 2025

jvstme commented on Oct 8, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions