May a standing site's re-fetchable subtrees be reclaimed #468

Activity

1. blooop

   added

   wayfinder:grilling

   on Aug 25, 2026
2. blooop

   added a parent issue on Aug 25, 2026

   • What dl reclaims and what it must leave #444
3. blooop

   marked this as blocking on Aug 25, 2026

   • Write the reclaim contract into docs/cleanup.md #455
4. blooop

   self-assigned this

   on Aug 25, 2026
5. blooop commented on Aug 25, 2026

   

   blooop

   on Aug 25, 2026

   OwnerAuthor

   More actions

   agent-decided (principle 2, no straggler data, on a gate principle 1 supplies): yes, and the thing that names it is not a directory name but a declaration the creating program wrote into the directory. .pixi/envs/<name> carries a CACHEDIR.TAG whose first 43 bytes are the Cache Directory Tagging Specification's signature, written by rattler, and the predicate that admits it never reads the directory's name at all. That is the same kind of answer #450 and #451 gave: read a record, do not match a shape.

   Everything below is measured in this repo's devcontainer, pixi 0.77.0, ext4, and the probe directory was removed. Nothing was written to the repo.

   1. The objection, taken at its strongest, and why it fails

   The objection is real: the site stands because we could not prove it safe, and reaching inside something unproven to delete part of it is a different act from leaving it alone. But it fails on the question of what the standing verdict is a statement about.

   #446's reasons for standing are answers about git's account of the site: Q2 is status --porcelain through the site's own admin directory, Q3 is reachability from a ref. NothingToAskThrough means there is no admin directory, so git's account cannot be obtained. It is silence about the tracked and untracked content git could have accounted for. It is not silence about the whole directory, because a specific set of bytes in there is outside git's account by the creator's own writing: pixi writes .pixi/.gitignore containing

   *
   !config.toml
   

   so nothing under .pixi/envs/ is in any index, in any status output, or reachable from any commit, in any clone, ever. The set of bytes the site's standing verdict is uncertain about and the set of bytes under the tag are disjoint by construction, and the construction is a file the installer wrote. Deleting the tagged subtree is therefore not reaching into the unproven part; it is removing bytes the standing verdict says nothing whatsoever about, and about which a second record says regenerable.

   And the contradiction that settles it. The shipped Q2 already reports nothing about .pixi/envs, because it is gitignored. So a devlaunch that refused this would print "this site holds work that exists nowhere else, 0 bytes" and in the same breath refuse to remove 5 GB of it "because we could not prove it safe" — two incompatible readings of one directory in one report. This map's second precedent is that a plan and the outcome acting on it must not be able to disagree; a report that disagrees with itself is the same defect a level up. The wedge does not open, because the rule is the tagged subtree is not part of what the standing verdict is about, and that is checkable rather than assertable.

   Where principle 1 does bite is section 3, and it bites on the recipe rather than on the site.

   2. What names it, and the name is never read

   The general gate: the Cache Directory Tagging Specification. A file CACHEDIR.TAG whose first line is exactly

   Signature: 8a477f597d28d172789f06886806bc55
   

   43 bytes, from https://bford.info/cachedir/. It is a declaration by the program that created the directory that the contents are regenerable and belong outside a backup. It is cross-vendor, published, and written by the creator, which is what makes reading it a read and not a guess. Measured, in this container:

   directory	written by	tag
   .pixi/envs/default	rattler (pixi 0.77.0)	yes
   rust/target	cargo	yes
   .venv from uv venv (uv 0.12.5)	uv	yes
   .pytest_cache	pytest	yes
   .venv from python -m venv	stdlib venv	no
   node_modules after a real npm install (npm 11.18.0, lodash)	npm	no, and no tag anywhere beneath it

   The two rows that matter are the last three. The same directory name lands on both sides: a .venv is admitted or refused depending on which program made it, which no pattern keyed on .venv can express. And node_modules is refused with nothing to argue about, because npm never made the claim. That is a property, not a member list.

   And pixi drew the unit's boundary itself, at exactly the right level. .pixi/ carries no tag; only .pixi/envs/<name> does. .pixi/config.toml is the one path .pixi/.gitignore un-ignores, and it is the one human-writable file in there. So the tag sits precisely inside the line that separates the installer's output from the human's config. Nothing devlaunch invented put it there.

   The per-file route, and why it is refused. The obvious sharpening is to enumerate the derivable set exactly: union the files array of every conda-meta/*.json and call anything outside it foreign. Measured, it does not hold. On the throwaway probe env it looked perfect (6650 recorded, 6678 walked, delta 28, every one of them CACHEDIR.TAG or a file under conda-meta/). On this repo's real env it is 11002 recorded against 12210 walked, and the 1208-file delta is 1160 real files: the pypi half, installed by uv from pixi.lock's own pypi entries and recorded in .dist-info/RECORD rather than in conda-meta, plus __pycache__ trees that no installer records and every import regenerates. So a path-set proof yields ~10% false positives on a normal env and would stand everything. It is #446's rejected content-based proof arriving in a cheaper disguise. Refused, with the numbers.

   3. Does a lockfile prove derivability: the four cases, measured

   The probe: one pixi.toml depending on python = "3.12.*", installed on the same ext4 device as the shared download cache so hardlinking would be physically possible.

   case	result
   lock present and names the env, --frozen --offline	re-derived, 5507 of 5507 files, 0.52 s, no network, and conda-meta/pixi's environment_lock_file_hash came back byte-identical (cb70a71a2c1df89c)
   lock present, all of HTTP(S)_PROXY/ALL_PROXY pointed at a dead port, --frozen	succeeded, 5507 files. So the no-network claim is not just pixi's --offline promise
   lock absent	pixi install --frozen --offline refuses: the lock file is not up-to-date with requested environment: 'default', 0 files. Not derivable
   lock stale (manifest gained a dep, lock untouched)	--frozen --offline restored 5507 files. A stale lock is not a stale env: the env on disk was itself produced from that lock, so the lock reproduces exactly what was there. Without --frozen the solve fails offline, which is why the recipe read is about the lock and not the manifest
   lock does not name the env	measured as a real population: add an extra environment, install, then drop it from the manifest and reinstall. .pixi/envs/extra survives, the lockfile no longer lists it, and pixi never mentions it again. Not derivable from anything on disk

   So: the lockfile's presence, and its naming of that environment, is what proves derivability, and both are readable. Absent or non-naming, nothing on disk reproduces the directory and it stands — that is principle 1 inside the rule rather than bolted to it. Principle 2 is then served the way #450 served the build cache: by a named figure and a pointer, and the pointer here is real and narrow, pixi clean -e <env>.

   One thing the tag does not promise, said out loud. pixi does not defend its own declaration. I planted my-notes.txt and a hand-written site-packages/mypkg/__init__.py inside the env, ran pixi install --frozen, and both survived and neither was mentioned. So the tag is a claim about the directory's purpose, not a proof about its current contents. That is why the argument in section 1 is the load-bearing one: the case for removal rests on the tagged subtree being outside what the site's verdict is about, corroborated by the creator's declaration and by the recipe being present, and not on a claim that every byte in there was installed.

   4. R3, and how much this actually reaches

   The st_nlink=1 finding reproduces, on the same filesystem, independently of #426. Env at st_dev=66306, /var/tmp/devlaunch-pixi/pkgs at st_dev=66306, and:

   CACHE nlink=1 ino=23351545 size=31231264  .../pkgs/python-3.12.14-h8ab3286_0_cpython/lib/libpython3.12.so.1.0
   ENV   nlink=1 ino=13680076 size=31231264  .../.pixi/envs/default/lib/libpython3.12.so.1.0
   

   Same size, different inodes, one link each. All 5507 regular files in the probe env have st_nlink=1; so do all 10867 in this repo's own 2.3 GB env. rattler copies out of pkgs/ rather than linking into the prefix, even where linking is physically available. Under exclusive_usage that means every byte of a tagged env is billed to its own tree and every byte comes back on removal — there is no hardlinked pool to make the figure a fiction. This is the one artifact on the map where du and exclusive_usage agree, and the reason is the same reason the population is expensive.

   Reach: about 94.5 GB of the measured 104.5 GB, roughly 90%. #426 measured 18 of 72 worktrees carrying a full .pixi/envs/default and stated that this is the difference between 104 GB and roughly 10. Two things worth stating about that figure. It is not contingent on #446's verdict: the env goes whether the site was collectable or standing, which is exactly what makes this the principle-2 keystone rather than a slice of #454. And it leaves ~10 GB across 72 directories standing and named, which is the residual principle 1 buys and which #446 already said out loud.

   5. The unit

   A tagged directory paired with the recipe that re-derives it. Not the directory, which is what makes a bare .pixi/envs a pattern; and not the recipe, which does not say what to remove.

   It is a different unit from #445's site subtree in three specific ways, and each one is why this is a decision:

   • Its edges come from a different read. A site's containment edges come from the filesystem walk (What unit does the agent-worktree sweep reason about #445). A derivative's boundary comes from where the creating tool put its tag, which is a fact in a file rather than a fact about the tree.
   • It nests the other way and stops. The walk does not descend past a tag: the outermost tagged directory is the unit, both because the outer declaration covers everything inside it and because descending would double-count under R3. What proves an agent worktree safe to remove, and what if it cannot be proved #446's two-recursions rule extends unchanged.
   • Its verdict is not the site's verdict. A derivative sits inside a site that is standing, by definition of this ticket. It is decided by its own reads, then modulated by the site's reasons per section 6.

   The domain is still enumerated from cache_dir and from nothing else (R1), and the root is still printed before the question (R2), because the derivative is found by the same walk the sweep already performs. A scratch XDG_CACHE_HOME run finds no tags, so it is harmless by construction.

   6. The fold: which standing reasons reach a subtree, and which cannot

   This is the answer to "the thin end of the wedge" in its sharpest form, and it is derived rather than chosen. Ask of each Reason in #446's Standing: is this a statement about git's account of the site's content, or a statement by a claimant about the directory?

   #446 reason	subject	reaches the derivative?
   Holds { Unpushed }, Holds { Uncommitted }	git's account of content	no. The derivative is outside that account by .pixi/.gitignore
   CouldNotProve(NothingToAskThrough)	git's account, unobtainable	no, same reason
   CouldNotProve(GitWouldNotSay)	git's account, refused	no
   CouldNotProve(NotThisClonesToAccountFor)	git's account, out of scope	no. Named explicitly because a reviewer will ask: another repository's env, tagged, with its own lock beside it, is still derivable, and whose repository it is was never part of the argument
   CouldNotProve(ThirdPartyClaim) (git worktree lock)	a claimant	yes, it pins. Somebody asserted a claim over the directory and made no distinction between its parts, and a lock may mean running right now. #426 Ask 2 holds at the subtree level too, for the same reason it holds at the site
   CouldNotProve(CouldNotTakeTheLock) (#459's run_if_lock_free)	a claimant	yes, it pins. A git operation is in flight, so something is operating in there

   A nested site inside a tagged directory is the pathological case, and it needs no new arm: a site under the tag is a claim on the directory by something the tag does not speak for, so it lands in the claimant column with the child's own reason, and the parent tag is never a candidate. The rule composes with #445's fold instead of sitting beside it.

   7. The type, per /constructive-modeling

   The distinction the ticket turns on is re-derivable versus unproven, and the mistake to avoid is giving "unproven" a value that can be constructed without a read. Same discipline as Proof in #446: Derivative is not the fallthrough of a filter, it is a thing you are handed.

   /// A directory whose creator declared it regenerable and whose recipe is on disk.
   /// Private fields, no `Default`, and the only constructor is a read that answered.
   pub struct Derivative { at: Inside, from: Recipe }
   
   /// What re-derives it. One arm per implemented reader, matched exhaustively
   /// everywhere, so a second reader is a compile error at every site rather than
   /// a branch nobody notices (#450's move).
   enum Recipe { PixiEnvironment { environment: String, lock: Inside } }
   
   /// A tag that was read. There is deliberately no arm for "no tag": an untagged
   /// directory is not a `Tagged` at all, so it cannot be constructed. Positive
   /// space, not a filtered-down negative one.
   pub enum Tagged {
       /// A reader answered, and no claimant's reason reaches it.
       Derivable(Derivative),
       /// A reader could not cost it. Carries its bytes, so principle 2 is served
       /// by visibility where it is not served by reclamation.
       CouldNotCost { at: Inside, bytes: DiskUsage, why: NoRecipe },
       /// A claimant's reason on the containing site reaches its subtrees (§6).
       /// Named with its bytes, never removed.
       Claimed { at: Inside, bytes: DiskUsage, by: Reason },
   }
   
   enum NoRecipe {
       NoReaderRecognisedIt,
       LockfileAbsent,
       LockfileDoesNotNameIt { environment: String },
       CouldNotRead(std::io::ErrorKind),
   }

   Four calls that follow from the measurements:

   • Option<Derivative> is refused outright. Its None would mean both nothing tagged here and tagged but not costable, which is the two-meanings-one-value smell and is holds_unsaved_work can answer about an ancestor repository and report a dirty clone as safe to delete #171's shipped bug one artifact over, the same one Does --purge need an unsaved-work check #459 refused for the purge figure.
   • Removal takes a Derivative and nothing else, so there is no path from found a tag to deleted it that skips a read.
   • conda-meta/pixi's manifest_path is not a field of anything. Measured, it is an absolute path recorded by whoever ran the install, so for every env installed inside a container it is /workspaces/<id>/... and does not resolve on the host: the same trap as a container-path worktree registration. What proves an agent worktree safe to remove, and what if it cannot be proved #446 and What unit does the agent-worktree sweep reason about #445 answer it by never resolving a recorded path, and the constructive form of that is that the field does not exist to be resolved. The recipe is located by walking up from the tag, inside the site, to a pixi.lock. What conda-meta/pixi is read for is environment_name, which is a name and not a path.
   • Bytes are DiskUsage (R3), never a bare integer, so weighed nothing and could not weigh stay different values.
   • Reason::subject() is a match with no wildcard arm, so adding a Blank variant later forces somebody to answer §6's question for it rather than inheriting a default.

   8. Generalisation: the property is general, the evidence is per-tool, and confinement is named

   The admission gate is general and measured across four writers and two refusals (§2). The recipe read is not: cargo's target/ is re-derived by cargo build from Cargo.toml plus Cargo.lock plus the source, a uv .venv by uv sync from uv.lock, and a bare uv venv somebody then uv pip installed into has no recipe at all — uv's tag overclaims there, and principle 2's own exception is re-fetchable content, which requires something present to fetch from.

   So the honest shape is: one general gate, an open set of readers, exactly one implemented, and a tag with no reader stands and is named. That is not a rule admitting one member by name, and the test is that .pixi and .pixi/envs appear nowhere in the predicate: the gate is a 43-byte constant from a published spec, and the reader is selected by whether its recipe read answers, not by what the directory is called. The residual the confinement leaves is small and measured — .pytest_cache is kilobytes, rust/target is not inside an agent worktree — and one reader reaches all ~94.5 GB.

   9. Out of scope, refused, and one piece of fog

   • The clone's own .pixi/envs, at the top of a live workspace's checkout. Not decided here, because the clone root is not a standing site and none of dl --prune cannot reclaim agent worktrees inside live workspaces' clones (104 GB measured) #426's 104.5 GB is there. It is the one obvious extension and it has a readable gate available (devpod's own listing says whether the workspace is running, which is a record and not a guess), so it is fog rather than a refusal. Deleting a running container's env would break a session's tooling until reinstall, which is a cost and not a loss, but it should not arrive as a side effect of this decision.
   • No new flag, deliberately. This is not a force, it is a removal with a proof, so it must not ride on --force-worktrees. Gating it there would make one flag carry two consents, which is precisely what Does --purge need an unsaved-work check #459 refused and what PrunePlan deleted its plan-wide force field to forbid.
   • A Provenance field or a from_tag: bool on the derivative, refused for Orphaned volumes with no surviving record #451's reason: it makes the untagged case representable and then relies on nobody building one.
   • Preventing the duplication rather than reclaiming it. Still nobody's, still correctly fog on the map. §4 sharpens it though: rattler copies out of pkgs/ even when the prefix is on the same filesystem as the cache, so the ceiling on any dedup fix is real and large.
   • Whether principle 2's exception census is a closed list. Docker images and build cache: what does dl reclaim #450 already flagged this for #455. This decision does not add a member to it: a tagged env is not left to accumulate, it is reclaimed. The exception list is untouched.

   10. Build work this implies

   Not a new ticket from me, per the map's instruction. It belongs on #454, because it is the same walk, the same plan, the same acting pass, and the same fold, and splitting it would create the two-passes-that-can-disagree shape this map has punished three times. What it must contain:

   1. Tagged, Derivative, Recipe and NoRecipe as §7 writes them, in the sweep module, with Derivative's fields private and no Default — and a public-api.rest.txt row asserting the absence of any other constructor, the way What unit does the agent-worktree sweep reason about #445 and Rebuild the agent-worktree sweep on the decided unit #454 already assert absences.
   2. The gate: first line of <dir>/CACHEDIR.TAG equals the 43-byte signature. The walk does not descend past a tag, which is also what stops find-shaped waste inside a 12000-file env.
   3. The pixi reader, three reads in order, any silence yielding NoRecipe: the tag; conda-meta/pixi parses and yields environment_name, with manifest_path never stored; a pixi.lock found by walking up inside the site whose environments: map names that environment.
   4. Reason::subject() -> Subject on What proves an agent worktree safe to remove, and what if it cannot be proved #446's Reason, a wildcard-free match, and the fold that turns a claimant reason into Tagged::Claimed. A site nested under a tag routes through the same arm.
   5. Byte accounting through exclusive_usage only, with What proves an agent worktree safe to remove, and what if it cannot be proved #446 §6's two-recursions rule extended: a derivative inside a site that is itself going is not reported separately.
   6. The plan names each derivative and its size before the y/N, and the acting pass re-reads the tag and the recipe before removing. Removal is the tagged directory alone, never .pixi (it holds config.toml) and never anything above it.
   7. Test rows, beyond the fixture the ticket names: lock absent -> stands; lockfile does not name the environment -> stands, with the extra case from §3 as the fixture; tag present with no reader -> stands and is named with bytes; site locked -> Claimed; site dirty -> derivable anyway, which is the row that pins §6; a nested site under a tag -> Claimed; a foreign file planted in the env -> still removed, which is the row that records §3's honesty rather than hiding it; and a node_modules and a stdlib .venv in the same fixture asserting nothing happens to them, which is the row that proves the predicate never reads a name.
   8. docs/cleanup.md gains the row (who reclaims it, and "the tag plus the lockfile" in the proves-removal-safe column). docs/workspace-tools.md's "The shared pixi package cache" gains one cross-reference sentence, and its headings must not move because flows::provision::lending_contract matches on them. No em or en dashes in either.

   11. What this container could not establish

   The four tag writers, the six install and re-derive measurements, the inode comparison and the record-versus-walk diffs are all from this devcontainer, pixi 0.77.0 / uv 0.12.5 / npm 11.18.0 / cargo, ext4, with PIXI_CACHE_DIR pointed at the real shared mount at /var/tmp/devlaunch-pixi. The 104.5 GB and the 18-of-72 split are #426's host numbers and were not re-counted from in here; what §4 re-establishes independently is the st_nlink=1 property those numbers rest on, including the same-filesystem case, which is the part that decides whether the bytes actually come back.
6. blooop

   closed this as completedon Aug 25, 2026