Make clone object sharing deliberate and extend it to LFS #154

Activity

1. blooop

   added

   wayfinder:grilling

   on Aug 9, 2026
2. blooop

   added a parent issue on Aug 9, 2026

   • Launch latency: dl <spec> -- <cmd> to a running command #139
3. blooop

   self-assigned this

   on Aug 9, 2026
4. blooop commented on Aug 9, 2026

   

   blooop

   on Aug 9, 2026

   OwnerAuthor

   More actions

   breadcrumb: grilling started — fresh subagent resolving against the principles.
5. blooop commented on Aug 9, 2026

   

   blooop

   on Aug 9, 2026

   OwnerAuthor

   More actions

   breadcrumb: resumed by a fresh wf-auto run — predecessor was killed just after loading this ticket, nothing recorded. Restarting the grilling with a fresh subagent.
6. blooop commented on Aug 9, 2026

   

   blooop

   on Aug 9, 2026

   OwnerAuthor

   More actions

   agent-decided (simplicity):

   One rule, stated once and pinned by a test: the .bare directory is the repo's object cache, and a workspace clone hardlinks its objects out of it — git objects by git's local transport, LFS objects by git-lfs's local transfer. Both sub-questions collapse into that one sentence, which is why simplicity decided it rather than maintainability: the maintainable options and the simple option turned out to be the same option, and the tie-breaker among the survivors was "one concept beats two that overlap". Nothing new appears in the layout, no new config key, no new reserved directory name, no new lock.

   Everything below was measured on this machine (git 2.55.0, git-lfs 3.7.1, ext4), not reasoned about. Scratch dir only; no real cache touched.

   ---

   1. Ordinary objects — keep the default local transport, add no flag, pin it with a test

   Measured, the accidental sharing is real and large. Bare clone of blooop/devlaunch + one workspace clone:

   	bare + clone, hardlinked	bare + clone, --no-hardlinks
   .git only	2044 KB	3788 KB
   whole trees	3584 KB	5328 KB

   The bare alone is 1864 KB. So an additional workspace costs 180 KB of git objects instead of 1924 KB — hardlinking already recovers ~94% of the per-workspace .git. stat confirms the mechanism directly: <bare>/objects/pack/*.pack and <ws>/.git/objects/pack/*.pack are the same inode with st_nlink == 2.

   Rejected: --local. It is already the default for a local-path source, so it changes nothing; and it does not even fail loudly if the source later became a file:// URL (git warns "--local is ignored" and proceeds over the file transport, unshared). A flag that pins nothing is a flag a reader in a year has to look up in order to learn it means nothing. Rejected on principle 2.

   Rejected: --shared / --reference, on measured evidence, not on the usual hand-wave. I built the exact situation devlaunch produces. Workspace cloned --shared off the bare, local branch with a commit on it; upstream then rewrote main (the ordinary force-push / rebase / squash-merge case); the bare ran devlaunch's own refspec git fetch origin +refs/heads/*:refs/heads/* (repo_manager.py:158 — a force refspec) followed by gc --prune=now, which git also does on its own via gc --auto after a fetch. Result in the workspace:

   error: Could not read 5532d92eb180dbfc9eeb8eddcc96b93b91d8e918
   fatal: Failed to traverse parents of commit 5493d4f0...
   error: refs/heads/main: invalid sha1 pointer 5532d92e...
   

   git log fatals, git fsck is a wall of errors, and the user's own commit is stranded on an unreadable base. The hardlinked clone survives the same sequence untouched, because the hardlink keeps the old pack's inode alive after the bare deletes its own name for it (measured: bare repacks → the clone's pack drops to st_nlink == 1 and becomes its private copy). That degradation is the safety property, not a defect. And the thing alternates would buy over hardlinks is nothing: .git measured 30,364 bytes with alternates vs 32,415 bytes with hardlinks — about 2 KB, because the hardlinks were already sharing the pack. Pay 2 KB for a corruptible workspace: no.

   "What breaks if a user's cache spans filesystems?" — Nothing, and it is unreachable anyway. Measured: cloning the bare (ext4) into /dev/shm (tmpfs) exits 0, silently, with different inodes — git falls back to a full copy on EXDEV. Correctness is never at risk; only the sharing is lost. And devlaunch cannot reach that state through its own API: get_bare_path is <repo_root>/.bare and get_workspace_path is <repo_root>/<workspace_id> (repo_manager.py:41-43, workspace_clone.py:70-84) — bare and clones are siblings in one directory, so they are on one filesystem by construction. Only someone bind-mounting .bare by hand can separate them, and the outcome is a bigger cache, not a broken one. This closes that sub-question rather than designing against it.

   So how does it become deliberate? The only form of "deliberate" that still holds in a year is one that fails when it stops being true (principle 4). A comment decays, a no-op flag misleads. An integration test that asserts Path(bare/objects/pack/x.pack).stat().st_ino == Path(ws/.git/objects/pack/x.pack).stat().st_ino fails the day someone inserts a file://, a --no-hardlinks, an intermediate copy, or a cross-filesystem layout. That test is the decision; the docstring merely explains it and records the two measured erosions above (repack orphans an existing clone's link; EXDEV silently copies).

   2. LFS — the bare becomes the repo's LFS store, and the workspace hardlinks out of it

   The constraint that eliminated the obvious answer

   lfs.storage persisted into the workspace's .git/config is wrong here, and it took reading the container story to see why. The workspace clone directory is bind-mounted into the devcontainer (devpod up <clone path>, dl.py:1136; "a container writes into its bind-mounted clone as its own user", dl.py:558). .bare is a sibling of the clone, so it is not in the container. A persisted lfs.storage = /home/<user>/.cache/devlaunch/repos/o/r/.bare/lfs therefore points, inside the container, at a host path that does not exist — and git-lfs consults it on every smudge, i.e. on every git checkout in an LFS repo. Best case it silently creates a phantom store as root; worst case it is a permission error on every checkout. A relative ../../.bare/lfs is worse, because it resolves to a plausible-looking wrong path (/.bare/lfs) inside the container. Writing a host path into a file that gets mounted into a container is an illegal state, and the fix is to not have a place to write it (principle 3).

   Rejected: command-scoped git -c lfs.storage=<abs> lfs pull — which does avoid persisting anything, and which git-lfs does honour (git lfs env shows the right LfsStorageDir). It is rejected because it is measured to break in exactly the environment the tests need. -c travels as GIT_CONFIG_PARAMETERS, which is inherited by the git-lfs child that resolves the remote side, so with a local-path remote the remote's object store also resolves to the override — which is empty — and the pull dies with remote missing object <oid> for every object. Verified directly:

   $ GIT_CONFIG_PARAMETERS="'lfs.storage'='/tmp/.../store2'" git -C origin lfs env
   LocalMediaDir=/tmp/.../store2/objects      # the *remote's* store, hijacked
   

   In production (https/ssh origin) it would work; with a local fixture remote it cannot. A mechanism that works only when you can't test it fails principle 4 on the spot, and is a trap for whoever debugs it in a year (principle 1).

   The chosen mechanism

   Two commands, both inside _materialize_lfs, both under the per-repo lock the caller already holds:

   1. In the bare: git -c lfs.fetchrecentrefsdays=0 -c lfs.fetchrecentcommitsdays=0 lfs fetch origin <ref> — network, once per repo+ref. (git lfs fetch works fine in a bare repo — measured. The two -cs matter: without them git-lfs also fetches "recent branches within 7 days", which on a multi-GB LFS repo is a large gratuitous cold-path download. With them the output is exactly Fetching reference refs/heads/main. These -cs are safe where the storage one was not — the bare's origin is the real remote, so there is no local-remote child to hijack.)
   2. In the workspace: git lfs pull "file://<bare_path>" — local, no network.

   Measured properties of step 2, with the workspace's origin deliberately pointed at a dead URL (https://127.0.0.1:1/nope.git) so nothing can be sneaking out to the network:

   • exit 0, worktree files fully materialized (big1.bin = 3,000,000 bytes, not a 132-byte pointer) — served entirely from the bare, zero network;
   • <ws>/.git/lfs/objects/** are the same inodes as <bare>/lfs/objects/** — git-lfs's local transfer hardlinks, so the workspace's LFS store costs zero extra disk;
   • nothing persisted: no new remote, no lfs.storage, git config --local --get-regexp '^lfs\.' shows only the lfs.repositoryformatversion 0 git-lfs writes anyway. A file:// URL as the positional remote argument is accepted (a bare path is not — measured, it silently falls through to origin);
   • failure mode stays loud and stays retryable: bare store empty + origin dead → exit 2, pointer files left in place → the existing _has_lfs_pointers recheck raises the existing RuntimeError and the next run retries. The deliberate non-gating on is_new_workspace (workspace_clone.py:350-352) is untouched.

   Keep the existing git lfs pull origin as the second phase, entered only if pointers remain after phase 2 — the same cache-first-then-origin shape as the git-object story, and it covers the one real gap (an existing workspace sitting on a commit the bare has never seen, e.g. the user fetched and checked out directly inside the container).

   Disk and network, honestly

   Per workspace of an LFS repo, payload P:

   	today	after
   workspace .git/lfs/objects	1×P	0 (hardlinked to the bare)
   workspace worktree	1×P	1×P
   shared in .bare/lfs	0	1×P (once per repo)
   N workspaces total	2N·P	(N+1)·P
   network for workspace #2	full pull	none

   Break-even at N=1 (2P either way, one download either way — never a regression), strictly better from N=2 on. Measured on a 5 MB fixture: bare + 2 workspaces occupy one copy of the payload plus the two worktree copies, du -sc counting shared inodes once.

   Concurrency — already solved, nothing to add

   _materialize_lfs is called from _prepare_workspace, which runs inside ensure_workspace's hold_lock(repo_manager.lock_path(owner, repo)) (workspace_clone.py:243-247). Two launches of the same repo are already serialized, so they cannot pull into the same store at once; different repos use different stores. No new lock, no new lock ordering. Belt and braces, I measured the unserialized case anyway: two concurrent git lfs pull into one store both exit 0, both worktrees hash-match origin, the store ends with exactly 2 objects (content-addressed writes with atomic rename), and the only residue is two zero-byte files in lfs/tmp, which git lfs prune sweeps.

   GC / lifetime

   The LFS store's lifetime is exactly the bare's lifetime. It lives at <bare>/lfs, which is where git-lfs puts a repo's store anyway, so: no new entry appears under <repo_root> (this matters — migration.py:93 walks repo_dir.iterdir() and treats every directory except .bare as a workspace clone, so a sibling .lfs/ would be misread as a workspace and the exclusion list would have to grow); remove_repository's rmtree(repo_path) and dl --purge already reclaim it with no new code; remove_workspace correctly leaves it, because the other workspaces' hardlinks depend on it. Nothing devlaunch does prunes it, and nothing should — it is a pure cache whose loss costs a re-download, never data (unlike a git alternate, whose loss costs history, as measured in §1).

   One new fact for the orphaned-disk work (#155), stated and handed over, not solved here: the bare's LFS store now grows monotonically with every ref ever launched, and unlike today's per-workspace stores it does not go away when the workspace does. git lfs prune inside the bare is the ready-made tool. Sizing/reporting/reclamation policy for it belongs to #155's territory, not this ticket's.

   Rejected: "pull LFS into the bare and let the clone smudge from it". It reduces to this answer with an extra failure mode — the clone's smudge reads the clone's store, which is empty, so it only works if you also redirect lfs.storage, which is the option eliminated above. And it would fetch LFS for all refs rather than the one being launched.

   Rejected: status quo + documentation. The map's Destination is explicit: "objects shared where git allows (including LFS)". Documenting a 2× per-workspace cost is not sharing it.

   3. How the outcome is tested

   Seam is the one already in use: a fake subprocess.run recording argv for unit tests (test/test_workspace_clone.py:296-388, test/test_devpod_spawn_counts.py), and @pytest.mark.integration ("real git commands, mocked DevPod") for the facts that only real git can prove. Every one of these fails on main today.

   Integration (real git, real git-lfs, local fixture remote — the whole design is offline-testable, which is precisely why -c lfs.storage lost):

   1. Pack hardlink. Bare-clone a fixture, create a workspace, assert st_ino equal and st_nlink >= 2 for objects/pack/*.pack. Red today only in the sense that nothing asserts it — so land it first, as the pin: flip the implementation to --no-hardlinks locally and watch it go red, then revert.
   2. Two workspaces of an LFS repo share one object store. Create workspace A (fetches LFS from the fixture remote), then break the remote (git remote set-url origin https://127.0.0.1:1/nope.git, or delete the fixture's .git/lfs/objects) and create workspace B. Assert B's LFS files are real content, not pointers — proving zero network — and assert st_ino of B's .git/lfs/objects/** equals the bare's. This is the headline test.
   3. Nothing host-specific is persisted into the clone. Assert git config --local --get lfs.storage is empty and git remote lists only origin after materialization — i.e. the mounted directory contains no host path. This is the test that keeps the container from being broken by a future "simplification".
   4. Retry contract preserved. Bare store empty + remote unreachable → RuntimeError, pointer files still on disk; then repair the remote and re-run → materialized. (Extends the existing test_existing_workspace_retries_unmaterialized_lfs.)

   Unit (mocked argv):

   5. The bare-side fetch is issued in the bare cwd, carries lfs.fetchrecentrefsdays=0, and names the ref — not a bare git lfs fetch (which would pull 7 days of branches).
   6. A repo with no LFS attributes issues no git lfs command at all — the gate from Gate the git-lfs probe: skip the git-lfs fork when the repo has no LFS attributes #142/PR Gate the git-lfs probe: skip the git-lfs fork when the repo has no LFS attributes #148 still holds after this change; assert the new bare-side fetch is behind it too.
   7. git lfs pull origin is issued only when pointers survive the cache phase — the cache-first ordering, pinned.

   CI: git-lfs is not in the pixi environment today, so these integration tests would silently skip — and a skipped test proves nothing. Adding git-lfs to the test feature's dependencies is part of the build ticket, not an afterthought.

   Seams — who owns what

   All the code for this decision lives in devlaunch/worktree/workspace_clone.py::_materialize_lfs plus one extra bare_path argument threaded from _prepare_workspace (which already holds it as bare_repo_path), and one docstring in _prepare_workspace/repo_manager. Deliberately zero changes in repo_manager.py's fetch flow.

   • Gate the git-lfs probe: skip the git-lfs fork when the repo has no LFS attributes #142 / PR Gate the git-lfs probe: skip the git-lfs fork when the repo has no LFS attributes #148 owns _lfs_tracked_files and the attributes gate. No textual overlap with _materialize_lfs, but same file: land this after Gate the git-lfs probe: skip the git-lfs fork when the repo has no LFS attributes #148 merges and inherit its gate rather than re-deriving one.
   • Interval fetch moves into the detached updater (try_hold_lock; sweep in dl --update-cache) #149 owns locks.py + dl.py --update-cache. No overlap.
   • Cold launch fetches exactly the requested ref; delete the workspace-clone fetch #150 owns ensure_branch (:172-206) and deletes _prepare_workspace step 4 (:298-310) — adjacent to the _materialize_lfs call site at :353. Sequence after Cold launch fetches exactly the requested ref; delete the workspace-clone fetch #150 (or rebase deliberately); do not touch ensure_branch, and in particular do not put the bare-side LFS fetch there, which is why it lives in _materialize_lfs.
   • Reclaim orphaned disk: stale clone directories and left-behind images #155 owns orphaned disk. This ticket hands it one new fact (the bare's LFS store outlives its workspaces) and does not touch reclamation.
   • Build: persist the devpod workspace id instead of re-deriving it, and reconcile the orphans #88 owns volumes and devpod records. Untouched.
7. blooop commented on Aug 9, 2026

   

   blooop

   on Aug 9, 2026

   OwnerAuthor

   More actions

   breadcrumb: resolved and indexed on the map. Graduated two build tickets: #162 pin the bare→clone hardlink with a shared-inode test (blocked by #150) and #163 make <bare>/lfs the repo's shared LFS store (blocked by #150 and #142, since it inherits PR #148's attributes gate in the same file).

   One new fact this decision opened rather than closed, and it arrived after #155 had already been resolved and closed, so it has no owner: the bare's LFS store now grows monotonically with every ref ever launched and — unlike today's per-workspace stores — does not go away when the workspace does. git lfs prune in the bare is the ready-made tool. Rather than silently reopen #155, I have recorded this as fresh fog in the map's Not-yet-specified; it graduates once #163 lands and the store actually exists.
8. blooop

   closed this as completedon Aug 9, 2026