Skip to content

feat: dl --refresh-stale recreates the stale workspaces that are idle - #676

Open
blooop wants to merge 15 commits into
feat/recreate-keeps-agentfrom
feat/refresh-stale
Open

blooop wants to merge 15 commits into
feat/recreate-keeps-agentfrom
feat/refresh-stale

Conversation

@blooop

@blooop blooop commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Part of #673 — A workspace keeps its old image after a pull, and a recreate ends its agent. This PR is part 3 of 3: dl --refresh-stale recreates each stale workspace whose agents are idle. It can run after the daily pull.

What this changes

  • A new flag, dl --refresh-stale. It takes no workspace, no -y and no --force. It is in the README command table, dl.bash global_opts, the --help examples and docs/cli.md.
  • A new module, flows/refresh_stale.rs, makes the plan. It changes nothing. For each stale workspace from part 1, it gives "refresh", or "skip" with the first rule that fails.
  • A new verb, LaunchVerb::RecreateUnattended, runs the recreate path of part 2 with no attach. Part 2 then starts the agents again in their herdr panes.
  • StaleImage::container() gives the container id that part 1 already reads.
  • The words for each plan line and each skip are in rust/dl/src/render.rs.

Where this fits

dl --refresh-stale
  stale = stale_images(every workspace)                # part 1
  plan  = for each stale workspace: judge(it)          # refresh_stale::plan
  print every plan line                                # "Refreshing <ws>: ..." or "Skipping <ws>: <reason>. ..."
  for each workspace to refresh:
    judge(it) again                                    # an agent can start work during an earlier recreate
    if it is now a skip: print "Skipping <ws> after all"
    else: recreate with no attach                      # part 2 collects and restarts the agents
  print the count; exit 1 if a recreate failed

judge(ws) skips when:
  another dl holds the launch lock
  the container is stopped                             # docker exec: "container <id> is not running"
  the processes cannot be read                         # one docker exec as root, /proc/*/stat and cmdline
  a build runs                                         # bazel, cargo, rustc, cc1plus, make, ninja, cmake, gcc, clang, ld, ...
  an agent process has a shell child                   # a tool call or a background task; MCP servers are not shells
  an agent runs and dl is outside herdr                # dl cannot start it again
  herdr does not answer, a pane has no saved line,
    or an agent is not idle or done
  the container runs more agents than herdr's panes hold

"Agent" means each agent that dl knows by name: claude, codex and gemini, run directly or by node. herdr's done means "idle, and nobody has looked at it yet", so idle and done both let a refresh go ahead.

How I checked it

All tests run on FakeRunner, or on the real binary with a fake docker on PATH. This devcontainer has no herdr and no docker daemon.

  • Core: 22 tests on the plan. They cover each skip rule and each build name, zombie processes, an agent run by node, a process list that ends early, a stat line that does not parse, herdr that does not answer, and several workspaces with different verdicts. I broke three rules on purpose, and five tests failed.
  • End to end (rust/dl/tests/read_side.rs): nothing stale, the plan printed before any recreate, exit 1 with the loop going on after a failed recreate, and a stopped workspace skipped as "it is stopped".
  • The review fixes each have a test that failed first:
    • A working codex agent no longer passes when no Claude process runs.
    • A Claude outside herdr no longer passes because an idle codex pane makes the count match.
    • A workspace is judged again just before its recreate, so a turn that started during an earlier recreate is not ended.
  • cargo test --workspace, clippy with -D warnings, cargo fmt --check: clean.
  • pytest test/ against the debug build: 850 passed.
  • prek run --from-ref origin/main: passed.

I did not run:

  • A real refresh. The in-container script did not run under a real busybox or a real container. No real recreate and restart ran.
  • scripts/public-api-snapshots.sh. The tools are not in the container. I copied the diff from the public-api CI job into the snapshot files.

Merge danger

Door: two-way. The flag is new, and nothing else calls the new verb. The promised API (public-api.api.txt) gains LaunchVerb::RecreateUnattended, so a caller with an exhaustive match on LaunchVerb needs an arm for it.

Blast radius: medium. A false "refresh" ends a working agent. Each rule therefore skips when it is in doubt. These limits remain:

  • An agent can start work between the second check and the recreate. That agent ends, the same as with dl <ws> recreate. docs/cli.md says this.
  • The agent check needs the process name, or one of the first two cmdline words, to end in an agent name. A native install that starts a versioned binary by its own path is not seen as an agent. If it runs outside herdr, a refresh ends it.
  • The shell-child rule needs Claude to start a new shell for each command. If Claude keeps one shell for the whole session, every Claude workspace is skipped. That is safe, but no refresh then happens.

Stack

  1. #674 — feat: dl --ls says which workspaces run an older image
  2. #675 — feat: dl recreate starts the agents it ends again
  3. #676 — feat: dl --refresh-stale recreates the stale workspaces that are idle ← this PR

🤖 Generated with Claude Code

Summary by Sourcery

Add unattended stale-workspace refreshes that safely recreate eligible workspaces and restore their idle agents.

New Features:

  • Add the global dl --refresh-stale command to recreate stale workspaces whose agents are idle and restart those agents without attaching.
  • Add unattended recreation support that restores held agent sessions after rebuilding a workspace.

Enhancements:

  • Add safety checks that skip stale workspaces when launches, builds, active agent work, unreadable processes, or unresumable sessions make recreation unsafe.
  • Recheck each workspace immediately before recreation, report skip reasons and outcomes, and continue after individual failures.

Documentation:

  • Document the new command, usage, eligibility rules, output, and exit behavior in the README, CLI reference, shell completions, and changelog.

Tests:

  • Add unit and end-to-end coverage for refresh planning, safety rules, session handling, rejudging, output, and failure behavior.

Chores:

  • Expose the stale container identifier needed for process inspection.

blooop added 14 commits October 7, 2026 15:08
flows::refresh_stale::plan reads, per stale workspace, its launch lock,
one docker exec of its process table, and the Claude sessions herdr's
panes hold, and says Refresh or Skip with the reason. A build process,
a shell under a Claude process, a busy or unresumable agent, a Claude
session no pane holds, and a herdr that cannot be seen are each a skip.

StaleImage now carries the container it was judged by, which is the
container the plan reads.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
LaunchVerb::RecreateUnattended is recreate's path, agent sessions
included, ending at Launched::Ready instead of a session. It is what
dl --refresh-stale runs for each workspace, with nobody to attach.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
… seen

dl --refresh-stale recreates outside herdr only a container it read no
Claude process in, so the warning that the recreate ends the agents it
cannot see was false there. A recreate typed by hand still warns.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
It prints one line per stale workspace (refresh, or skip and why), then
runs the unattended recreate for each it may refresh, says each launch's
notices as they happen, and ends with a count. A recreate that fails is
counted and the loop goes on; the exit status is 1 when any failed.
Named in dl.bash's global options and in the help's examples.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…review/refresh-stale

# Conflicts:
#	CHANGELOG.md
#	rust/devlaunch-core/src/flows/launch.rs
…sed a Claude outside herdr beside a held codex

The rule knew only Claude processes. A codex running a command, or any
codex at all outside herdr, left no Claude process to find, so the
workspace was refreshed and the codex ended. And the count compared
Claude processes against herdr's panes of every agent kind, so one held
codex covered a Claude started outside herdr, which then ended with no
line to start it again.

An agent is now any agent dl knows by name (herdr::agent_named), found
the way Claude was: by comm, or argv[0] or argv[1] for node. The shell
rule and the count apply to all of them, so both sides of the count are
agents of any kind.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…rdr gives

Seven cases the plan had no test for: a listing that ends halfway through a
process, a stat line that does not parse, a held agent herdr does not answer
for, a saved line in a pane herdr says holds no agent, a launch lock that
cannot be asked, a Claude started under a Claude, and three stale workspaces
whose verdicts must not leak into each other. Each was checked to fail with
its branch broken.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
dl --refresh-stale judged every stale workspace once, printed the plan,
and then recreated them in turn. A recreate can take minutes, so a
workspace that was idle at plan time could have a build or a working
agent by the time its turn came, and the recreate ended it.

Each workspace the plan would refresh is now judged again just before
its recreate. A skip then wins: it is counted as skipped and printed as
"Skipping <ws> after all, judged again before its recreate: <why>".
The plan is still printed whole, first.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
render_refresh_stale had no test. Four now run the binary against the
read-side world with a docker fake that answers the process read:

- nothing stale: the "nothing" line, exit 0, no docker exec.
- one quiet and one with cc1plus: the whole plan is on stdout before
  the first devpod up, one recreate, refreshed 1 skipped 1 failed 0.
- the first recreate fails: the second still runs, 1 failed, exit 1.
- a stopped container: a Skipping line, no up, exit 0.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
pixi was on the build list, so a pixi shell that stays open in a pane
kept that workspace stale for as long as it ran. A build that pixi run
starts runs cargo, a compiler or another program the list already
names, so the list loses nothing by dropping pixi.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
…a docker fault

A stopped stale workspace was skipped as "could not read the processes
in its container (docker exec failed ...)". Skipping is right, because
a recreate would also start it, but the line said docker was broken.
docker exec into a stopped container says "container <id> is not
running", so the plan now skips that as Skip::Stopped, and dl says the
workspace is stopped and that `dl <ws> recreate` starts it on the new
image. (The Skip::Stopped variant and its match arm went in with the
previous commit, which touched the same file.)

Also: herdr's `done` is an idle agent nobody has looked at, so with the
part 2 fix a done agent is held and a refresh may go ahead. The test
that pinned the old skip now expects a refresh, and docs/cli.md says
why pixi is not on the build list.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m
@sourcery-ai

sourcery-ai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Sorry @blooop, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 23 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@sourcery-ai

sourcery-ai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Reviewer's Guide

Introduces dl --refresh-stale, which identifies stale workspaces, conservatively refreshes only those whose containers and agent sessions appear idle and recoverable, rechecks eligibility immediately before each unattended recreate, restarts held agents without attaching, and reports skips and failures.

Sequence diagram for unattended stale workspace refresh

sequenceDiagram
    participant User
    participant CLI as dl CLI
    participant Stale as stale_images
    participant Planner as refresh_stale::plan
    participant Docker as Docker container
    participant Herdr as herdr
    participant Launch as Launch

    User->>CLI: --refresh-stale
    CLI->>Stale: stale_images(workspaces)
    Stale-->>CLI: stale images and container ids
    CLI->>Planner: plan(stale workspaces)
    Planner->>Docker: exec_as_root(read process table)
    Docker-->>Planner: processes or unreadable
    Planner->>Herdr: collect and read agent sessions
    Herdr-->>Planner: pane states and saved lines
    Planner-->>CLI: Refresh or Skip verdicts
    CLI->>User: Print complete plan
    loop Each planned refresh
        CLI->>Planner: plan(workspace) again
        Planner-->>CLI: Refresh or Skip verdict
        alt Still eligible
            CLI->>Launch: render_launch(RecreateUnattended)
            Launch->>Launch: Recreate container
            Launch->>Herdr: Restart held agent panes
            Launch-->>CLI: Success or failure
        else No longer eligible
            CLI->>User: Skipping after all
        end
    end
    CLI->>User: Print tally and exit status
Loading

Flow diagram for stale workspace eligibility checks

flowchart TD
    A[Stale workspace] --> B{Launch lock free?}
    B -- No --> S1[Skip: another dl is launching it]
    B -- Yes --> C[Read processes with docker exec as root]
    C --> D{Container stopped?}
    D -- Yes --> S2[Skip: it is stopped]
    D -- No --> E{Processes readable?}
    E -- No --> S3[Skip: processes unreadable]
    E -- Yes --> F{Build process present?}
    F -- Yes --> S4[Skip: build is running]
    F -- No --> G{Agent has shell child?}
    G -- Yes --> S5[Skip: agent is running a command]
    G -- No --> H{Agent sessions present?}
    H -- No --> R[Refresh]
    H -- Yes --> I{Sessions visible and recoverable?}
    I -- No --> S6[Skip: sessions unavailable or unresumable]
    I -- Yes --> J{Every agent idle or done?}
    J -- No --> S7[Skip: agent is busy or unanswered]
    J -- Yes --> K{Running agents <= herdr panes?}
    K -- No --> S8[Skip: sessions outside panes]
    K -- Yes --> R
Loading

File-Level Changes

Change Details Files
Adds the global --refresh-stale command and documents its unattended behavior.
  • Adds CLI parsing, validation, help text, completion, README, changelog, and detailed CLI documentation.
  • Restricts the command to global use without a workspace, -y, or --force.
  • Defines plan, skip, recreate, output, and exit-status behavior for stale workspaces.
rust/dl/src/cli.rs
rust/devlaunch-core/completions/dl.bash
README.md
docs/cli.md
CHANGELOG.md
Implements conservative stale-workspace eligibility checks before recreating containers.
  • Adds a refresh planning flow that evaluates stale workspaces in rule order and reports the first failure.
  • Reads container process tables via a root Docker exec, detecting stopped/unreadable containers, builds, agent shell children, and zombie processes.
  • Validates herdr visibility, resumable saved commands, agent idle/done state, and matching agent-session counts.
  • Carries the stale container ID through StaleImage for process inspection.
rust/devlaunch-core/src/flows/refresh_stale.rs
rust/devlaunch-core/src/flows/refresh_stale/tests.rs
rust/devlaunch-core/src/flows/stale_images.rs
rust/devlaunch-core/src/flows/mod.rs
Extends the launch flow with an unattended recreate that restarts held agents without attaching a terminal.
  • Adds LaunchVerb::RecreateUnattended using the existing recreate path and agent-session holding/restart behavior.
  • Suppresses unseen-agent warnings for the unattended mode because eligibility checks already exclude unsafe outside-herdr cases.
  • Adds coverage confirming agents restart in herdr panes and no SSH session is opened.
rust/devlaunch-core/src/flows/launch.rs
Integrates planning, revalidation, execution, rendering, and failure aggregation into the CLI command.
  • Prints the complete initial plan before any recreate begins.
  • Rejudges each planned refresh immediately before execution, skips workspaces that became busy, and continues after individual recreate failures.
  • Reports refreshed/skipped/failed counts and returns exit status 1 if any recreate fails.
  • Adds human-readable reasons for every skip and end-to-end coverage for ordering, stopped containers, rejudging, and continued execution.
rust/dl/src/commands.rs
rust/dl/src/render.rs
rust/dl/tests/read_side.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@codecov

codecov Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.21429% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.54%. Comparing base (6b15858) to head (f5551ce).

Files with missing lines Patch % Lines
rust/dl/src/commands.rs 94.52% 4 Missing ⚠️
rust/devlaunch-core/src/flows/refresh_stale.rs 98.67% 2 Missing ⚠️
rust/devlaunch-core/src/flows/launch.rs 98.66% 1 Missing ⚠️
rust/dl/src/render.rs 99.23% 1 Missing ⚠️
Additional details and impacted files
Flag Coverage Δ
python 42.98% <ø> (ø)
rust 94.75% <98.21%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
shipped code (rust) 94.75% <98.21%> (+0.03%) ⬆️
harness and tooling (python) 42.98% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copied from the public-api CI job's diff. public-api.api.txt gains
LaunchVerb::RecreateUnattended, a change to the promised API: a caller
that matches LaunchVerb exhaustively needs an arm for it.

Claude-Session: https://claude.ai/code/session_01Ro7QyyAtHmrh9oxyyK6h9m

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant