[Bug]: Dev Environment Fails After ~6 Hours with JOB_FAILED (CONTAINER_EXITED_WITH_ERROR) #2344

Steps to reproduce

I'm experiencing an issue where our dev environment crashes approximately 6 hours after it starts, resulting in a JOB_FAILED (CONTAINER_EXITED_WITH_ERROR) error. This failure occurs even though we haven't specified any idle_duration or max_duration parameter, and we're using spot_policy: on-demand (i.e., it’s not a spot instance issue).

Based on discussions in Discord channel, it seems the environment might be hitting a default maximum duration even though the docs state that max_duration defaults to "off".

Steps to Reproduce:

1. Apply the dev environment configuration:

   dstack apply -f dev.dstack.yaml

2. (Optional) Detach and reattach to the run:

   dstack attach aana-episodic-retrieval-dev

3. Let the environment run for approximately 6 hours.
4. Observe the failure:
   • In dstack Sky, the run shows JOB_FAILED (CONTAINER_EXITED_WITH_ERROR).
   • The dstack attach session remains active and does not immediately error out.

Actual behaviour

• The dev environment terminates around 6 hours after start with the error JOB_FAILED (CONTAINER_EXITED_WITH_ERROR).
• The error message is not descriptive enough, as it doesn’t explicitly state that the termination was due to exceeding a maximum duration.
• The dstack attach session remains running despite the failure.

Expected behaviour

• The dev environment should run indefinitely (or at least without termination after 6 hours) when max_duration is set to off by default or documentation needs to be updated.
• If a maximum duration is enforced, the error message should clearly indicate that the termination was due to exceeding the maximum duration.
• The dstack attach session should exit when run fails

dstack version

0.18.42

Server logs

Additional information

Configuration (dev.dstack.yaml):

type: dev-environment

name: aana-episodic-retrieval-dev

image: nvidia/cuda:12.3.2-cudnn9-devel-ubuntu22.04

init:
  - apt-get install -y libgl1 libglib2.0-0 ffmpeg ssh-client htop python3 python3-dev git sqlite3 nvtop cron
  - curl -sSL https://install.python-poetry.org | python3 -
  - export PATH=$PATH:/root/.local/bin
  - 'echo "PATH=$PATH:/root/.local/bin" >> ~/.bashrc'
  - poetry config virtualenvs.in-project true
  - sh install.sh

ide: vscode

spot_policy: on-demand

max_price: 0.9

volumes:
  - name: test-models-cache
    path: /models_cache

resources:
  gpu: 48GB..:2
  cpu: 9..
  memory: 50GB..
  disk: 100GB..