Repository navigation
[Bug]: FAILED_TO_START_DUE_TO_NO_CAPACITY #1994
Description
Steps to reproduce
create a fleet.dstack.yml to provision remote backend from my on_prem server. This was created successfully.
dstack apply -f fleet.dstack.yml
type: fleet
name: model-dev-fleet
placement: any
# The user, private SSH key, and hostnames of the on-prem servers
ssh_config:
user: my_user
identity_file: ~/.ssh/id_rsa
hosts:
- 33.33.48.1create another yml config to provision a development server with the provisioned fleet as the backend
dstack apply -f dev_environment.yml
type: dev-environment
name: model-dev-env
#python: "3.11"
image: dstackai/base:py3.13-0.6-cuda-12.1
ide: vscode
spot_policy: autoActual behaviour
I got the below error
All provisioning attempts failed. This is likely due to cloud providers not having enough capacity. Check CLI and server logs for more details.
Then also tried to see extra details about the error with the below command
dstack ps --verbose
output below
NAME BACKEND REGION INSTANCE RESOURCES SPOT PRICE STATUS SUBMITTED ERROR
model-dev-env failed 54 sec ago JOB_FAILED
(FAILED_TO_START_DUE_TO_NO_CAPACITY)
Expected behaviour
Instance provisioning should be completed successfully with a vscode link to my workspace.
dstack version
0.18.22
Server logs
[12:45:55] INFO dstack._internal.server.services.backends:404 Requesting instance offers from backends: []
[12:45:56] INFO dstack._internal.server.background.tasks.process_runs:330 run(110058)model-dev-env: run status has changed SUBMITTED -> TERMINATING
INFO dstack._internal.server.services.jobs:283 job(0d49c8)model-dev-env-0-0: job status is FAILED, reason: FAILED_TO_START_DUE_TO_NO_CAPACITY
[12:45:58] INFO dstack._internal.server.services.runs:739 run(110058)model-dev-env: run status has changed TERMINATING -> FAILED, reason: JOB_FAILEDAdditional information
No response
Activity
peterschmidt85 commentedon Nov 14, 2024
@judeleonard please provide
dstack fleet listoutput.One of the reasons can be that the fleet instance have GPUs while the dev environment doesn't request any.
judeleonard commentedon Nov 14, 2024
Here is the output
judeleonard commentedon Nov 14, 2024
Yes, I tried to attach a GPU before but I got the same error 'Not having enough capacity' And my remote server actually has both GPU and docker preinstalled
peterschmidt85 commentedon Nov 14, 2024
It's
dstack ps, notdstack fleet listdstack by default offers only instances that match exactly the resources of the fleet
judeleonard commentedon Nov 14, 2024
This is the output. Not much details
peterschmidt85 commentedon Nov 14, 2024
Means the fleet creation wasn't successful.
This will help understand why fleet cound't be created
judeleonard commentedon Nov 14, 2024
This is the dastack webserver after the fleet was created. But I will try again and post the entire output
judeleonard commentedon Nov 14, 2024
I just recreated the fleet now
then my stack server log.
Could this fingerprint be an issue with my ssh user?
peterschmidt85 commentedon Nov 14, 2024
Oh that's a known issue, it will be fixed in the next release but for now please do
pip install paramiko -Uand then restart the server, and try again. The issue must be gonejudeleonard commentedon Nov 14, 2024
Thank you. Will do that
judeleonard commentedon Nov 15, 2024
hi @peterschmidt85 , sorry for updating you at this time. Our remote server was undergoing some updates.
So I later tried it after installing paramiko like you suggested and dstack server logs indeed changed from what I had before. This is my log now. I also updated dstack to the latest v0.18.25.
peterschmidt85 commentedon Nov 15, 2024
@judeleonard Now that you've updated
paramiko, please show the full output of creating the fleet (bothdstack applyanddstack serveroutputs).judeleonard commentedon Nov 15, 2024
dstack apply -f fleet.dstack.yml
dstack apply -f dev_environment.yml
peterschmidt85 commentedon Nov 15, 2024
@judeleonard
But what it showed then? Was it successful?
Running anything before the fleet is created doesn't make sense.
Lets try to understand why the fleet isn't created. Need logs for that.
judeleonard commentedon Nov 15, 2024
peterschmidt85 commentedon Nov 15, 2024
This clearly shows that dstack cannot connect to the instance using the provided key
@judeleonard Can you connect to the same host using the provided key via
ssh -i <key path> jude@1.1.4.1?@un-def Any ideas what could be wrong?
judeleonard commentedon Nov 15, 2024
Yes, I can connect to the same server via ssh from my terminal.
judeleonard commentedon Nov 15, 2024
The current user I am using actually requires a password to successfully connect to the server. Could this be why?
peterschmidt85 commentedon Nov 15, 2024
@judeleonard Yes! This certainly can be a reason.
judeleonard commentedon Nov 15, 2024
Okay, let me work on this and try it again.
peterschmidt85 commentedon Nov 15, 2024
@judeleonard Also ensure the SSH key is added to
~/.ssh/authorized_keyson the host?peterschmidt85 commentedon Nov 15, 2024
Basically,
dstackworks only ifsshworks without a password.Fake45mar commentedon Dec 3, 2024
Hello!
I have an issue going on with dstack. To not create one more issue with the similar title will post it here.
Right now i am trying to create dstack server, fleet locally and test it, let's say with vscode task.
My ssh-fleet config:
.dstack.yaml:
After applying with the command: "dstack apply .dstack.yaml" i have following:
Yes, minor bug, it completely ignored declared resource request with CPU 7, 10GB RAM and 15GB Volume, anyway not the main concern yet.
Fleet is created, sudo is passwordless, docker is on the board: "Docker version 27.3.1, build ce12230".
Dedicated GPU is not presented in my laptop however integrated is available and detected according to this command:
So, seems doable.
Then i ran one of examples i found in your docs section:
jvstme commentedon Dec 3, 2024
Hi @Fake45mar. Looks like your
dev-environmentconfiguration is requesting exactly 2 CPUs, 2 GB RAM, and a 5GB disk.However, the instance in your
fleet1fleet has 8 CPUs, 39 GB RAM, and a 35.9GB disk.This instance does not match the
resourcesin yourdev-environmentconfiguration. Sincedstackcan't find any instances that match, the run fails withFAILED_TO_START_DUE_TO_NO_CAPACITY.Try removing
resourcesfrom yourdev-environmentconfiguration or setting the requirements to open ranges:This notation means "2 CPUs or more", "2 GB or more", etc. You can find more examples of setting
resourcesin the reference.Fake45mar commentedon Dec 3, 2024
@jvstme , hey! Would like to thank you for the quick reply. Yes, it stopped failing with the code i mentioded before. However, it doesn't wake up vscode as it was supposed:
what it makes even more strange, is that :10999 port is not actually busy. Should i continue posting my questions here or it's better to open a new thread?
In any case, thank you
jvstme commentedon Dec 3, 2024
@Fake45mar, yes, could you please open another issue for this question or join our Discord server? We can answer your question there. Happy to help!