dl --prune cannot reclaim agent worktrees inside live workspaces' clones (104 GB measured)

Symptom

A host filled up: 468 GB volume at 100%, 1.5 GB free. The largest consumer was
~/.cache/devlaunch/ at 152 GB, and dl --prune reclaimed 1.3 GB of it.

The space is in agent git worktrees that live inside the workspace clones:

~/.cache/devlaunch/repos/<owner>/<repo>/<workspace-id>/.claude/worktrees/agent-<hash>/

Measured 2026-08-24 on one host, after a manual cleanup had already run:


`agent-*` worktree directories	72
...carrying a full `.pixi/envs/default`	18
Total size	104.5 GB
`repos/` tree total (single `du`, hardlinks deduped)	127 GB
Worktrees as a share of the clone tree	~82%

One clone held 55 GB of them on its own. The envs are real copies, not hardlinks
into the shared pixi cache devlaunch bind-mounts at /var/tmp/devlaunch-pixi:
a sampled 217 MB locale-archive.tmpl had st_nlink=1, and du -shx vs
du -shxl over the whole tree differ by only ~13 GB, so this is not a
measurement artifact.

Why `--prune` cannot see it

dl --prune removes clone directories no workspace references (#159). Every one
of those 72 worktrees is inside a clone belonging to a live devpod workspace:

worktrees inside LIVE workspaces:      72  (104.5 GB)
worktrees inside ORPHANED clones:       0  (0.0 GB)

So the existing rule is not merely missing them — it must never fire here, because
firing would delete a live workspace's checkout. Reclaiming this needs its own rule,
scoped to .claude/worktrees/ within a clone that --prune otherwise leaves alone.

This is the same class as the orphaned-volume problem (#324): something creates
named state, nothing garbage-collects it, and devpod delete was never going to.

What makes it non-trivial

These are registered git worktrees, not stray directories, and they were created
from inside the container, so their registered paths are container paths that do not
resolve on the host:

$ git -C <clone> worktree list
/home/ags/.cache/.../devlaunch-fix-arch-pinivoti                     eb8a29d [fix_arch]
/workspaces/devlaunch-fix-arch-pinivoti/.claude/worktrees/agent-a49a 57955a3 [fix/git-refusal-classification] locked
/workspaces/devlaunch-fix-arch-pinivoti/.claude/worktrees/agent-a8da 5f9d40c [fix/aid-pty-flake] prunable

Across all clones on this host: 6 locked, 33 prunable registered entries against
72 directories on disk — so there are three distinct categories, with different
safety profiles:

Unregistered leftovers (~33 dirs): git has already forgotten them, the directory
remains. Safe to delete outright.
Registered and prunable: git worktree prune will drop the metadata, then the
directory can go.
Registered and locked: git worktree prune deliberately skips these. Claude Code
locks a worktree so it is not collected mid-run, so a lock may mean in use right now
or may be a leftover from a killed session. These must not be removed on a timer.

And the branches matter. In the clone sampled, all 7 worktree branches had 0 unpushed
commits, but 3 of them did not exist on origin at all — so the commits were reachable
only from that worktree's branch ref. A blind rm -rf of a worktree whose branch is
unpushed and unmerged loses work, and it will not look like data loss until much later.

Asks

dl --prune reclaims .claude/worktrees/agent-* inside clones it keeps, for
categories 1 and 2 above.
locked worktrees are never removed implicitly. Require an explicit opt-in flag,
and say how many were skipped and why.
Refuse — or require the same explicit flag — for any worktree whose branch has commits
not reachable from the default branch or any remote ref. Report those separately
rather than silently keeping or silently deleting them.
git worktree prune runs in the clone so git metadata and the directory do not drift
apart, and the container-path registrations get cleaned up too.
dl --ls --size attributes worktree bytes, so this is visible before it is a full disk.
It was invisible in --ls --size here, which is why it reached 100%.
A dry run that prints what would go and what it would free, given the amounts involved.

Notes for whoever picks this up

The container-path registrations (/workspaces/<id>/...) mean git worktree remove
from the host will not resolve the path; either run it inside the container, or
git worktree prune the metadata and remove the directory from the host.
The agent-* naming comes from the agent harness, not from devlaunch, so match on the
.claude/worktrees/ container rather than the agent- prefix.
Worth checking whether the per-worktree pixi install can be pointed at the shared
cache so 18 envs are not 18 independent copies. That is a separate fix and probably a
separate issue, but it is the reason the number is 104 GB rather than 10.

Found while auditing devcontainer compatibility across the fleet; not caused by any
devcontainer change.

Correction to the last bullet under "Notes for whoever picks this up".

I wrote that the per-worktree pixi install could be pointed at the shared cache so
18 envs stop being 18 independent copies. That is wrong, and acting on it would waste
the effort — recording it here so nobody does.

The shared cache devlaunch mounts at /var/tmp/devlaunch-pixi is a download cache
only, and README's "The shared pixi package cache" is explicit that this is deliberate:
environments and their trampolines bake absolute paths, and two containers sharing one
environment tree is pixi#5476. Sharing
it is safe precisely because it is the download cache and nothing else.

The 104 GB is environments (.pixi/envs/default/**), not downloads. The whole shared
cache is ~11 GB by comparison. So pointing the worktree installs at it buys download
time, not disk.

Measured on the same host, which also rules out the easy explanation:

cache:    links=1 dev=66306  .../devlaunch/pixi/pkgs/libsanitizer-16.1.0-hf2715c6_2/lib/libtsan.so.2.0.0
worktree: links=1 dev=66306  .../agent-a150da0b262f59b83/.pixi/envs/default/lib/libcrypto.so.3

Same st_dev, so the cache's extracted pkgs/ tree (6.6 GB) and the worktree envs are on
one filesystem and hardlinking between them is physically possible — yet both sample files
have st_nlink=1, so no linking is happening in either direction today.

That makes env-byte deduplication a separate question about whether rattler links or copies
from pkgs/ into a prefix on the same filesystem, not something a devlaunch mount can fix.
Worth its own issue if anyone wants to chase it. The reclamation this issue asks for stands
on its own and does not depend on it.

breadcrumb: adopted as the only ticket of map #435 (wf-one). No parent before this, so no other map lost a ticket. Labelled wayfinder:build, claimed. Next: wf-tdd in fresh context, red before green, ending at a pushed branch with an open PR linking this issue.

One scope note for the record: of the six Asks, the dry run is largely already there (--prune prints prune_plan_lines and asks before acting, -y skips the question), and dl --ls --size attribution is the one ask that touches a different surface from the rest. Keeping all six on this ticket rather than splitting, since they are one job; if the build finds otherwise it exits with a handoff rather than growing the map.

breadcrumb: wf-tdd started. Building all six Asks on one branch.

What I will build, restated: a sweep of the .claude/worktrees/ directories inside the clones --prune decides to keep, classified three ways as the ticket describes plus a fourth for "git still holds this one". Forgotten and prunable go by default; locked ones and ones whose branch holds commits no remote ref reaches are reported and left, and one new opt-in flag is what carries either past its refusal. git worktree prune runs in the clone after the removals. The plan output that already exists grows the worktree section, so the dry run stays one dry run. And --ls --size says how much of a clone's bytes are worktree bytes.

Seams I intend to test at, all of them ones the surrounding prune flow already tests at:

The classification of one clone's worktree directories, against a real git clone with real registrations rewritten to container paths, which is the host's shape. This is the seam the whole ticket turns on: git's own listing is what says locked and prunable, and a fake spawn answering "clean, no output" would read as "nothing is registered", which is the answer that deletes.
The plan and the acting pass. The sweep has to be in the plan a person says yes to, and the acting pass has to classify again before it removes, the way it already does for clones.
The plan and report rendering, because the dry run is an Ask and the counts and the reasons are the whole of what it delivers.
The flag's parse, so that neither existing flag silently widens to mean this one.
The listing's size cell and its JSON, for the attribution ask.

One design note worth recording because it is a two-run data-loss path and not obvious: git worktree prune is all-or-nothing across a clone, and on the host it drops the registration of every container-registered worktree, including one being kept because its branch holds unreachable commits. Dropping that registration turns it into a forgotten directory, which the next run deletes outright -- so the guard would protect it once and then hand it over. The prune therefore runs only in clones where nothing was held back for a branch reason.

breadcrumb: the guiding principle for this ticket is now stated: a prune leaves nothing accumulating except an actual shared cache. Working the Asks against that turned up edge cases the Asks list does not cover. Recording them here as part of the spec rather than leaving them to be found in review.

The loss the Asks list does not catch

Ask 3 gates on "commits not reachable from the default branch or any remote ref". Commits are not the only work in a worktree.

Uncommitted work. A worktree with a fully-merged branch and 200 lines of unstaged edits reads as safe to delete under the rule as written. Reachability says nothing about the working tree or the index. A worktree must be dirty-checked (git status --porcelain) as well as ref-checked, and a dirty one goes in the same "report, do not remove implicitly" bucket as a locked one.
Detached HEAD. An agent worktree need not be on a branch. Commits are then reachable only from HEAD and its reflog, and a reachability check keyed on a branch name finds no branch and therefore nothing to protect. Detached-and-ahead is the same bucket.
Stashes are safe, and worth saying so. refs/stash lives in the shared repository, not the linked worktree, so it survives the directory going. Only mention it so the checker does not go looking.

The reachability check has a stale-ref trap

The workspace clone is cut from the sibling .bare and then has its remote repointed at the forge, with no fetch in the clone (see the module header on flows/workspace_clone.rs). So the clone's own refs/remotes/origin/* is as of clone time and can be arbitrarily old, or absent.

The .bare next door is the thing that gets fetched (git fetch origin +refs/heads/*:refs/heads/* --tags --prune, on a sweep interval). Ask against the clone alone and pushed-and-merged branches read as unpushed, which keeps garbage forever and makes the flag the only way to reclaim anything. Consult the sibling bare's refs.

And whatever it consults, it is answering as of the last fetch. Say that in the output rather than implying a live answer. Do not add a network fetch to --prune: it would fail offline and turn a local cleanup into a network command.

`locked` is not the same question as `in use`

Ask 2 protects locked worktrees. A lock is set by the agent harness as a courtesy and a killed session leaves one behind, so locked is neither necessary nor sufficient for "in use right now". Nothing on the host proves an unlocked worktree is idle. Two consequences:

The report should say what it is asserting (registered, locked, dirty, unpushed) and never assert idleness.
TOCTOU is real and unclosable from here. prune_plan takes the per-repo lock, but a container running git worktree add is not a participant in that lock. Scan says prunable, container re-registers, the delete lands on a live worktree. Re-check immediately before removing each directory rather than trusting the plan, which is the same reason PrunePlan carries Promotion per candidate instead of a plan-wide force boolean.

Ordering, so an interrupted run self-heals

Remove the directory first, then git worktree prune in the clone. Interrupted between the two, git holds a registration whose directory is gone, which is precisely the prunable state the next run already handles. The other order leaves a registered-and-present worktree with its metadata dropped, which nothing recognises.

Discovery, not path-shape matching

Recurse. An agent session running inside an agent worktree creates .claude/worktrees/ inside that. Scanning only <clone>/.claude/worktrees/ misses nested ones, and nesting is how this reached 55 GB in one clone.
Confirm it is a worktree, by the .git file pointing back at <clone>/.git/worktrees/<name>, rather than trusting the path. A plain directory or a file sitting at that path is not ours to delete.
Skip symlinks, as the existing clone scan does, or a link walks the removal out of the tree.
Match on the .claude/worktrees/ container, not the agent- prefix (the issue body already says this; it follows from the harness owning the name).
Distinguish directory absent because it was a container path that does not resolve here from directory absent because it was deleted. Both are prunable metadata; only the second has host bytes behind it.

Sizes must not double-count

A linked worktree shares the clone's object store, and the clone's objects are themselves hardlinked from .bare (that hardlink is what makes a clone per branch affordable). Attributing object bytes to a worktree counts them two or three times. Attribute the working tree and its .pixi/, which is where the real bytes are, and leave the objects to the clone.

Stay inside `repos_dir`

--prune is scoped by XDG_CACHE_HOME through repos_dir, which is what makes a scratch-cache run harmless. Worktree discovery must stay under repos_dir and must not reach into ~/.claude or into a container path, or that property is gone.

One thing that is not in scope, deliberately

The 18 duplicated .pixi/envs/default copies are the reason this is 104 GB rather than ~10 GB, and they are not fixable by pointing them at the shared cache: docs/workspace-tools.md records that only the pixi download cache is shared, because installed environments bake absolute paths and a shared environment tree is pixi#5476. So the envs are reclaimed by removing the worktree, which is this ticket, and there is no config change hiding here.

breadcrumb: PR #442 is open and links this issue. All six Asks plus the sharpened spec are on it.

What is there: --prune sweeps .claude/worktrees/ inside the clones it keeps, four-way classified (git has forgotten it / git says the registration can go / git is holding it locked / git still holds it), the first two going by default. --force-worktrees is the one flag that carries one past a lock, past uncommitted work, or past commits nothing else reaches, and it is deliberately not --force -- there is a test that --force alone does not reach a worktree. The dry run is the plan that already existed, grown a section. git worktree prune runs after the directory goes, and is held back in a clone where something is kept for what it holds. --ls --size names the worktree share of a clone's figure, in the table and in the JSON.

Four things worth carrying forward, because each one changed a decision:

The dirty check has to go through the clone's admin directory for the worktree. Pointing git at the worktree itself refuses, because its own gitfile names a container path. And .claude/worktrees/ has to be excluded from that check, or a worktree holding a nested worktree reads dirty forever and the bytes inside it are never reclaimed.
The sibling bare cache is asked about reachability before the clone is. The clone is never fetched into, so asking it alone reports pushed-and-merged branches as unpushed, which keeps every byte forever and makes the flag the only way to reclaim anything. The report says the answer is as of the last fetch. Nothing fetches.
The scan recurses only into worktrees it is keeping. A directory that is going takes everything inside it, so descending into one would report the same bytes twice and offer a directory that will not be there.
git worktree prune is a data-loss hazard, not tidiness. It is all-or-nothing across a clone, so in a clone where a worktree is kept for what it holds it would drop that worktree's registration too, turning it into a forgotten directory that the next run removes outright. Held back, the guard keeps working. This is the one design decision here that is not visible from the Asks list.

One finding I did not act on, recorded because someone will meet it: an orphan clone that has agent worktrees in it reads as dirty at the clone level, because .claude/ is untracked, so --prune keeps it until --force answers for it. That guard is right to count them -- removing the clone destroys whatever those worktrees hold -- so widening it would be the dangerous direction. It does mean the reclamation of a stale clone full of agent worktrees still takes --force.

Ready for wf-review.

breadcrumb: build complete, PR #442 (wayfinder/devlaunch-426, 4 commits). Gate verified against the full head SHA b812411f8593cfc70dbdb907881dfccaabab90bb: 15 check runs, every one success except publish-job, which is skipped by design. Not taken on the build report's word.

Moving to wf-review in fresh context, two-axis. Four things in the build carry more risk than the rest and are being pointed at explicitly: the regenerated public-api.rest.txt snapshot (2 rows moved, and the toolchain that regenerated it was installed by hand rather than carried by the devcontainer), the pathspec that excludes .claude/worktrees/ from the dirty check, the decision to hold git worktree prune back in a clone where any worktree is kept, and one failing test whose premise was changed rather than the guard it was testing.

Also flagged for the reviewer: dl/tests/read_side.rs reported 1 unnamed failure on one of ~8 full-suite runs and did not reproduce in 9 further runs. CI is green. Unexplained, not dismissed.

wf-review: Request changes on #442. Standards passes with minors; Spec has one blocker: the git worktree prune holdback is computed from the plan's kept list rather than from what the acting pass withheld, so a worktree the re-check correctly refuses is demoted to Forgotten and deleted with its uncommitted work by the next run. Reproduced end to end with no flag typed. #442 (review)

breadcrumb: wf-review verdict — Standards pass, Spec fail. Not approved. Full report on #442.

S1 is a blocker and it is real data loss, reproduced end to end, not a reading. The git worktree prune holdback is computed from the plan (agent_worktrees.rs:1049 gating on metadata_may_be_pruned(), which reads only keeping at :672), but a candidate the acting pass's re-check demotes lands in report.withheld at :1032, which the holdback never consults. Run 1 withheld a worktree that went dirty while the confirmation was on screen, then pruned the registration anyway; run 2 classified it Forgotten and deleted it. An uncommitted file was lost with no flag typed at either prompt.

Worth naming the shape: the safety re-check is what creates the vulnerable state. Forgotten is the one arm with no probe at all, so a demotion becomes a deletion one run later. Both review axes found it independently.

S2 (major): worktree bytes are chained into freed() (lifecycle.rs:2073, :2386) but rendered in clone sentences (render.rs:1386, :1611), so at host scale 104 GB is attributed to "directories nothing references".

Taking the one retry the lifecycle allows, in fresh context, red before green on S1. A second failure parks this ticket.

Two corrections to the build report, for the record. No existing test had its premise changed: zero deletions in dl/tests/lifecycle.rs, the Insisted premise was in a new test, and the call was right. And the read_side.rs flake is not caused by this diff (the file is untouched, and the one read-path change would fail that JSON pin deterministically rather than one run in eight); the likely mechanism is load from five new binary-boundary tests, which is #401's documented shape. Noted there, not fixed here.

Gate the worktree metadata prune on what the run did

breadcrumb: review findings on the PR are fixed. The blocker was real data loss and worth recording as a rule rather than as a bug: the guard that holds back a repository-wide metadata prune has to be computed from what the acting pass actually did, never from what the plan predicted it would do. The two can differ by design here, because a container writing into a worktree is not a participant in devlaunch's lock, so the plan on screen and the re-check under the lock disagree exactly when it matters. Reading the forecast meant a worktree the re-check correctly refused lost its registration to the prune, and the registration was the only thing telling the next run it was not a directory git had forgotten. That next run deleted it, and the afternoon of work in it, with no flag typed at either run.

So the answer is one rule folded by both passes rather than two predicates that can disagree, which is the same lesson as carrying an insistence per candidate instead of one flag for the run.

Second rule out of the same finding: the arm that deletes without asking git anything is the arm that has to be hardest to reach by accident. It now asks what a directory holds wherever there is anything left to ask through, and only takes git's word where there is genuinely nothing. Defence in depth, because fixing the gate alone left the blind deletion one bug away from returning.

Third, smaller, and generalisable: exclude things by what they are, not by where they sit. A path-shaped exclusion in the dirty check was hiding both a tracked edit and, worse, unreported content that nothing else protected.

breadcrumb: one process note worth carrying, because it cost real time and would cost it again. The branch had drifted into conflict with the default branch, and this repo's CI and prek run on pull_request events. GitHub cannot compute a merge ref for a conflicting pull request, so those workflows did not fire at all for the pushed commit: not pending, not failed, absent. What was left reporting was the security scan and a push-triggered publish job that skips, which renders as two checks, both fine.

So "checks are green" is not a safe reading on its own. Confirm the pull request is mergeable before believing a check list, and if a set of checks looks unexpectedly small, count them rather than reading their colour. Merged the default branch in, and the full set fired.

The only merge conflict was in a generated API snapshot. Resolved by taking the default branch's copy and regenerating, never by hand-merging two generated files.

breadcrumb: the review's findings are all landed and the pull request is green on the full check set, mergeable. Nothing is parked.

Three rules came out of this that outlive the ticket, and they generalise past worktrees:

A guard must be asked of the outcome, not of the forecast. Where a plan is shown to somebody and then re-checked before acting, the two answers can differ by design. Any second guard that reads the plan's answer instead of the re-check's is a guard that protects once and then hands the thing over. The shape to reach for is a value both passes fold their own results into, so there is one rule and no way to read the other pass's prediction.

The branch that deletes without asking anything must be the hardest one to reach by accident. Give it a probe wherever a probe is possible at all, and where none is, say so as a limit with its reason rather than leaving a wildcard that reads as "nothing to check here". Enumerating the cases instead of catching the leftovers found two that could be probed and were not.

Exclude by what a thing is, not by where it sits. A path-shaped exclusion in a safety check will eventually hide something that merely shares the path.

The fourth is smaller and about reading CI rather than about code: a conflicting pull request cannot run pull_request-triggered workflows at all, so a suspiciously short check list is a thing to count rather than to read the colour of.

breadcrumb: retry landed. Gate re-verified against the full head SHA 9b6180b41294805300d5432861b5b815a56a411e: 15 check runs, all success, publish-job skipped, MERGEABLE/CLEAN. git diff origin/main...HEAD -- '*api.txt' is empty, so the frozen tier (#251 §7) has not moved.

The retry also caught something the gate would not have: the branch had gone conflicting with main, and GitHub cannot compute a merge ref for a conflicting PR, so ci and prek were not firing at all while two checks reported. That reads as green to anyone scanning, which is precisely why the gate requires runs to exist for the full head SHA rather than trusting a summary. Merged at 7596524; the only conflict was the generated public-api.rest.txt.

Post-review work is three targeted commits: 913df8c (S1), c631646 (S2 docs), 9b6180b (a defect the retry found itself). Going back through wf-review on the delta rather than calling it approved on the fixer's own report — the fixes are substantial and no independent eye has seen them. The one retry the lifecycle allows is spent, so a second Spec failure parks this ticket with a handoff rather than getting another attempt.

Two claims in the retry that need an independent check, not acceptance. It reports that the previous review was wrong to judge S6 as failing safe: a gitfile tail of .git/worktrees/.. reportedly landed on Forgotten and the directory was removed outright. And public-api.rest.txt now removes ten rows while adding three, after a conflict resolved by taking main's copy and regenerating.

breadcrumb: re-review of #442 at 9b6180b (second pass, fresh context): Request changes -- Standards pass, Spec fail. Every finding from the first review is genuinely closed, including the one the first review got wrong (S6: the fixer was right, .git/worktrees/.. did land on the deleting arm). Two new findings at the same boundary: a worktree nested inside one that is going is removed with it, unreported and with no flag typed, taking uncommitted work, unpushed commits or a lock (blocker, #426 Ask 2); and git worktree prune reaches registrations neither pass enumerated, so a worktree the container registers after the plan is deleted outright by the next run (major). Report: #442 (review)

handoff

Parked here, and re-charted. The design question this ticket kept failing on now has a map of its own: What dl reclaims and what it must leave.

Where we are: PR #442 is open, green, and not merged. S1 is genuinely closed and reproduced. Two findings stand from review 5019431339 — T1 (a worktree nested inside one being removed is protected by neither path, including when locked, which breaks Ask 2) and T2 (the metadata prune reaches past the plan the gate reasons over). The second reviewer's read, which I share: those are one idea, and the unresolved decision is what unit the sweep reasons about.

Why parked rather than fixed again: three rounds, three correct instance fixes, same shape each time. That is a seam in the wrong place, not a defect list getting shorter.

First move on resume: do not touch this ticket. Work #444, whose keystone is what unit the sweep reasons about. This ticket's remaining work is carried by rebuild the sweep on the decided unit, which names #442's fate as part of its scope, and its measurement and sharpened spec stay the record everything else is checked against.

Navigation Menu

Repository navigation

dl --prune cannot reclaim agent worktrees inside live workspaces' clones (104 GB measured) #426

Symptom

Why `--prune` cannot see it

What makes it non-trivial

Asks

Notes for whoever picks this up

The loss the Asks list does not catch

The reachability check has a stale-ref trap

`locked` is not the same question as `in use`

Ordering, so an interrupted run self-heals

Discovery, not path-shape matching

Sizes must not double-count

Stay inside `repos_dir`

One thing that is not in scope, deliberately

handoff

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

dl --prune cannot reclaim agent worktrees inside live workspaces' clones (104 GB measured) #426

Description

Symptom

Why --prune cannot see it

What makes it non-trivial

Asks

Notes for whoever picks this up

Activity

blooop commented on Aug 24, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

The loss the Asks list does not catch

The reachability check has a stale-ref trap

locked is not the same question as in use

Ordering, so an interrupted run self-heals

Discovery, not path-shape matching

Sizes must not double-count

Stay inside repos_dir

One thing that is not in scope, deliberately

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

blooop commented on Aug 25, 2026

handoff

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Why `--prune` cannot see it

`locked` is not the same question as `in use`

Stay inside `repos_dir`