[Bug]: Run abortion makes an instance unusable #893

dstack version

master

Python version

3.8

Host OS

macOS

Host Arch

ARM

What happened?

1. Run something with dstack run . -b gcp
2. Wait till running status
3. Press Ctrl+C twice, to abort the run
4. Run something again, to use the same instance
5. Your run is stuck at provisioning

CLI logs

...
 Refreshed run shaggy-rattlesnake-1: JobStatus.PROVISIONING
...

Server logs

DEBUG 2024-02-13T12:08:29.840 dstack._internal.server.services.runner.ssh Cannot connect to 34.172.186.208's API: 409 Client Error: Conflict for url: http://localhost:51158/api/submit

Runner logs

No response

Additional Information

Further investigation showed, that the shim still runs the previous container. Stopping the container (docker stop) solved the issue.

Old abort relied on instance termination. With pools we should rely on terminating the container from the shim.

Termination without abort works fine (but the instance becomes ready before the shim deletes the container and is actually ready)