Skip to content

[Bug]: [Presets] Any failure during resume marks the creation permanently failed #4164

Description

@peterschmidt85

Steps to reproduce

  1. Start a creation from a configuration with any env entry, e.g. env: [MODEL_LABEL=some-value].
  2. Stop it: dstack preset stop <id> -y. It correctly records interrupted and prints Resume it with dstack preset resume <id>.
  3. Resume from a shell where that variable is not set: dstack preset resume <id>. It fails with Environment variable MODEL_LABEL is not set ([Bug]: [Presets] Resume requires literal env values in the local environment #4165).
  4. Resume again, with the variable set: dstack preset resume <id>.

Actual behaviour

Step 4 refuses:

Preset <id> creation failed and cannot be resumed

Step 3 is what marked the session failed. create_preset wraps the whole creation in except BaseException: _close_agent_session(session, "failed") (create.py:437-439), so any exception is terminal: it writes failed and deletes the workspace (create.py:754-756, workspace.py:120-130). Step 3's exception is raised at create.py:415, before _create_preset is entered, so nothing had started and nothing had changed, yet the session became unresumable. load_resumable_session then refuses (session.py:331-332).

Not caused by reconcile: _is_reconcilable requires status running, an existing report, and a dead owner (create.py:252-262), so an interrupted session is never reconciled.

Expected behaviour

Step 3 leaves the session interrupted, and step 4 resumes it. A failure that happens before the agent is spawned is not a failed creation.

Activity

  1. changed the title [-][Presets] preset stop suggests resume but the session becomes unresumable[/-] [+][Bug]: [Presets] preset stop suggests resume but the session becomes unresumable[/+] on Aug 18, 2026
  2. changed the title [-][Bug]: [Presets] preset stop suggests resume but the session becomes unresumable[/-] [+][Bug]: [Presets] Any failure during resume marks the creation permanently failed[/+] on Aug 18, 2026
  3. breken-ai commented on Sep 9, 2026

    @breken-ai

    Reproduced on current HEAD. The env resolution runs inside the try whose BaseException handler marks the session failed, so a missing passthrough variable on resume - before the agent is spawned and before anything changed - burns the session: _close_agent_session(session, "failed") writes the terminal state and deletes the workspace, and load_resumable_session then refuses it forever. This hoists _resolve_preset_env above the session claim and the try: a pre-spawn failure leaves an interrupted session resumable, and a fresh attempt leaves no dead record behind. Regression test resumes an interrupted session with a missing env var: main flips it to failed; with the fix it stays interrupted. PR: #4278

    Built by breken, your AI support engineer - breken.ai finds the bug, proves the fix with unmatched RCA, answers customers in your voice, and keeps your code and support docs clean - this one's on us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions