[Bug]: A rolling deployment whose new replica cannot start terminates the healthy service #4186

Steps to reproduce

1. Create a fleet that cannot grow beyond one node:

type: fleet
name: rolling-repro
nodes: 0..1
resources:
  cpu: 2..

2. Deploy a single-replica service onto it:

type: service
name: rolling-repro-svc
image: python:3.12-slim
port: 8000
auth: false
commands:
  - mkdir -p /srv && cd /srv
  - echo v1 > index.html
  - python -m http.server 8000
resources:
  cpu: 2..

3. Wait until the service is running and serving.
4. Change commands (echo v2 > index.html) and dstack apply again, accepting the in-place update.

Actual behaviour

Rather than leaving the running version alone, dstack terminates the run and the healthy old replica with it, so the service goes down permanently.

Expected behaviour

A rolling deployment whose new replica cannot start should leave the previous deployment serving and report that the rollout failed. Destroying a healthy, known-good deployment because its replacement could not be placed is worse than the update not happening at all.

dstack version

master 1298b421a (source checkout)

Server logs

Additional information

Where it happens. The run-level failure analysis is deployment-blind. Any replica with a failed job and no retry contributes FAILED + JOB_FAILED:

and one such replica then decides the whole run, discarding what every other replica contributed:

Neither reads deployment_num, even though the rollout logic next door is built entirely on it (build_replica_lists, get_group_rollout_state, has_out_of_date_replicas in services/runs/replicas.py). There is also no rollback or "abandon the failed deployment" concept anywhere in the run/service services.

How it got here: #986 introduced "any failed replica fails the whole run", in a form structurally identical to today's code:

#2821 then added deployment_num and the in-place update path on top. It changed process_runs.py substantially, but not the failure/termination analysis — the only deployment_num lines it adds there concern bumping and out-of-date detection.