Skip to content

The bare cache becomes the repo's LFS store; workspaces hardlink from it - #209

Merged
blooop merged 1 commit into
mainfrom
wayfinder/devlaunch-163
Aug 14, 2026
Merged

blooop merged 1 commit into
mainfrom
wayfinder/devlaunch-163

Conversation

@blooop

@blooop blooop commented Aug 14, 2026 •

Copy link
Copy Markdown
Owner

Closes #163. Decided on #154; the direct extension of #208, which pinned that
git clone <bare> <ws> hardlinks the pack files. This does the same for the
files the pack does not contain.

What was wrong

git-lfs objects are not git objects. The bare cache makes a workspace's history
free, and carried none of the large files — so every workspace of an LFS repo
downloaded the entire payload from the forge and kept a private copy of it in
.git/lfs/objects, on top of the worktree copy. The cache made the cheap part
free and left the expensive part fully duplicated, per workspace, forever.

What it does now

_materialize_lfs gained the bare and the ref, and runs in two phases:

  1. Fill the cache. git -c lfs.fetchrecentrefsdays=0 -c lfs.fetchrecentcommitsdays=0 lfs fetch origin <ref>
    with the bare as cwd — that cwd, and nothing else, is what puts the objects
    in <bare>/lfs, which is why no lfs.storage is needed. The two knobs bound
    it to the launched ref; at their defaults git-lfs also walks recent refs and
    recent commits, so launching one branch of a busy repo would drag several
    branches' payloads down with it.
  2. Materialize from it. git lfs pull "file://<bare>" in the workspace.
  3. Then, only if a pointer survived, today's git lfs pull origin — unchanged.

Measured, before writing any of it

git 2.5x, git-lfs 3.7.1, ext4, against a local fixture remote built the way the
suite builds one:

claim result
a bare clone arrives with LFS objects no — lfs/ is empty, so the bare-side fetch is genuinely what fills it
git lfs pull "file://<bare>" shares the object same (st_dev, st_ino), st_nlink 5 — a hardlink, zero extra disk
... with origin at a URL that does not resolve succeeds — zero network
... leaves lfs.storage in .git/config no — only lfs.repositoryformatversion, which is not host-specific
... leaves an extra remote no — git remote is just origin, still the forge

Failure arms measured too, because they decide the shape: an empty cache store
makes the file:// pull exit 2 (remote missing object), and a dead origin makes
both the bare-side fetch and git lfs pull origin exit 2. So neither cache step
can be fatal
— each degrades to the origin pull, and the RuntimeError-and-retry
contract stays the only thing that raises.

The constraint the design is shaped around

Nothing host-specific may be persisted into the clone. dl hands the clone
directory to devpod up, which bind-mounts it; .bare is a sibling and is not
mounted. An lfs.storage entry or an added remote naming a host path would work
perfectly on the host and break every in-container git checkout of an LFS
repo — a failure that appears nowhere near its cause. So the cache is named as a
bare file:// argument, which is gone when the command is. (-c lfs.storage
is doubly out: #163 records it as measured to break against local-path remotes
outright, since GIT_CONFIG_PARAMETERS is inherited by the remote-side git-lfs
child.)

That property is worth its own two tests because it is otherwise invisible:
mutation M7 below — add a cache remote and pull from it — leaves every disk and
network assertion passing and only that guard goes red.

Tests

New integration file test/integration/test_lfs_object_sharing.py, real git and
real git-lfs, offline, over a local fixture remote with a real LFS payload. Its
fixture scopes GIT_CONFIG_GLOBAL and runs git lfs install --skip-repo, because
git lfs pull fetches objects but silently skips the checkout with no
filter.lfs config — on a machine where nobody ran git lfs install, the
assertions would fail for a reason unrelated to the cache.

  • a second workspace materializes with the remote deleted — the fixture remote
    is rmtree'd between the two launches, which is harder than an unreachable URL:
    nothing can be silently re-fetched, so what the second workspace holds provably
    came from the cache. Asserts real content and (st_dev, st_ino) identity —
    either alone is passable by accident.
  • the first workspace leaves the payload in the cache — pins the direction:
    materializing from origin and leaving the cache empty would still give that
    workspace real content and leave every later one downloading again.
  • no lfs.storage, and exactly one remote, still the forge (URL checked,
    not just the name).

Unit, at the subprocess.run argv seam in test/test_workspace_clone.py:

  • the cache store is filled in the bare for the launched ref — cwd is the bare,
    the ref is named, both knobs present.
  • the workspace pulls from the cache before origin — order, as a list equality;
    a cache pull issued after the origin pull costs the download it exists to avoid
    and every disk assertion still holds.
  • origin is not pulled when the cache materialized everything — the saving,
    stated as a command that does not run. Decided by surviving pointers rather than
    exit codes, since git-lfs exits zero having fetched only some objects.
  • a failing cache phase degrades to origin / a launch that cannot materialize
    at all still says so
    — the two halves of the unchanged contract.
  • strengthened the existing no pointers → no git-lfs fork test: its
    cmd[:2] == ["git","lfs"] check is blind to the new git -c … -c … lfs fetch
    form, so it now looks for the word anywhere in the argv.

Mutation checks — every one caught

# mutation caught by
M1 cache fetch runs one directory up, not in the bare the cwd unit test + both integration sharing tests
M2 fetchrecent knobs dropped the argv unit test
M3 the ref is not named the argv unit test
M4 workspace pulls origin instead of the cache order + no-origin unit tests + remote-deleted integration
M5 origin phase skipped whenever the cache phase ran 4 tests, incl. two pre-existing ones
M6 lfs.storage persisted into the clone the lfs.storage guard (+ both sharing tests)
M7 a cache remote added to the clone only the one-remote guard — everything else stayed green
M8 cache fetch's failure not caught both degradation tests
M9 cache pull's failure not caught both degradation tests
M10 origin failure no longer raises the retry-contract test

The whole cache phase removed is the red this started from: the two sharing tests
failed, the two host-specific guards passed.

Also in here

  • git-lfs >=3.4,<4 added to the pixi test feature. It was absent, so these
    tests would have skipped — the failure mode that looks like a pass.
  • Riders from Pin the bare-to-clone object hardlink with a test and state it as the design #208's review, taken because this PR is at the same seams:
    the clone_manager fixture (duplicated at test_clone_object_sharing.py ≡
    test_repo_manager_real.py, and about to become a third copy) is folded into
    test/fixtures/git_fixtures.py; and test_clone_object_sharing.py now compares
    (st_dev, st_ino) rather than st_ino alone, the disk_usage.py idiom.
  • README (How much disk a workspace costs, and the host-prep row of the timing
    table) and CHANGELOG under ## [Unreleased].

Declared deviation

pixi.lock moved from format v6 to v7 — 440/432 lines. That is not a
dependency change, it is what the pinned pixi (0.76.1) writes; it had already
been warning run pixi lock to upgrade to v7 on every task in this repo. Verified
mechanically: the sorted set of every conda:/pypi: package URL in the lock is
identical before and after apart from the single added
git-lfs-3.7.1-hd3da7ce_3.conda line. setup-pixi@v0.9.6 installs current pixi,
which reads v7.

CI: pixi run ci exits 0 — pylint 10.00/10, 1624 passed.

🤖 Generated with Claude Code

Summary by Sourcery

Make the bare cache the shared git‑LFS object store for each repo so workspaces hardlink LFS objects from it instead of downloading and storing private copies, while preserving the existing origin‑based fallback behaviour.

New Features:

  • Support materializing LFS content from a repository’s bare cache by fetching branch-specific LFS objects into the cache and pulling them into workspaces via a file:// URL.
  • Add integration tests that verify LFS object sharing between cache and workspaces and enforce that clones do not persist host-specific LFS configuration such as extra remotes or lfs.storage overrides.

Enhancements:

  • Refine workspace LFS materialization to run a cache-first, origin-second flow based on whether pointer files remain, keeping the previous retry-on-failure contract intact.
  • Strengthen unit tests around workspace cloning to assert correct cache fetch parameters, pull ordering, and fallback behaviour when cache steps fail or cannot materialize objects.
  • Use shared fixtures to construct a real WorkspaceCloneManager for integration tests, reducing duplication across test modules.
  • Update documentation (README and CHANGELOG) to describe cache-based LFS sharing, its disk and network implications, and timing accounting changes for host preparation.

Build:

  • Include git‑LFS as a dependency of the pixi test feature so cache/LFS integration tests run against a real git‑LFS binary instead of silently skipping.

Tests:

  • Add a new integration test suite for git‑LFS object sharing that validates cache filling, workspace materialization from the cache, and absence of host-specific configuration in clones.
  • Extend existing workspace clone unit tests to cover cache fetch semantics, pull ordering, origin suppression when cache fully materializes content, and degradation paths when cache operations fail.

Chores:

  • Refactor integration test fixtures by centralizing WorkspaceCloneManager construction in a shared git_fixtures module.
  • Regenerate pixi.lock to format v7 while keeping dependencies stable aside from the added git‑LFS entry.

git-lfs objects are not git objects, so the bare cache that makes a
workspace's history free carried none of the large files: every workspace
of an LFS repo downloaded the whole payload from the forge and kept a
private copy of it in .git/lfs/objects, on top of the worktree copy.

Fetch the launched ref's LFS objects once into <bare>/lfs with the bare as
cwd and the fetchrecent knobs zeroed, then materialize each workspace with
`git lfs pull "file://<bare>"`. Measured (git-lfs 3.7.1, ext4): that pull
hardlinks -- same (st_dev, st_ino) as the cache's object, zero extra disk --
and completes with the remote deleted from disk, so the second workspace
touches the network for its large files not at all.

Nothing host-specific is persisted into the clone: no lfs.storage, no added
remote. The clone is bind-mounted into the devcontainer and .bare is not, so
a host path in its config would break every in-container `git checkout` of
an LFS repo while working on the host.

The origin pull remains as a second phase, entered when a pointer survives
the cache phase -- decided by pointer content, not by the cache commands'
exit codes -- so the RuntimeError-and-retry contract is unchanged and
neither cache step can fail a launch.

Closes #163

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @blooop, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@sourcery-ai

sourcery-ai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Reviewer's Guide

Makes the bare cache the authoritative git-lfs object store for each repo and updates workspace cloning to hardlink LFS objects from the bare cache, with new unit/integration tests, docs, and test infrastructure to enforce correct behavior and host-agnostic configuration.

File-Level Changes

Change Details Files
Make bare cache the repo's git-lfs store and materialize workspace LFS content from it before falling back to origin.
  • Refactor LFS materialization into cache-fill and cache-pull helpers that operate on the bare repo and workspace, respectively.
  • Invoke git lfs fetch in the bare repo with lfs.fetchrecent* knobs zeroed and the launch ref named, treating failures as best-effort.
  • Invoke git lfs pull file:// in the workspace to hardlink objects from the cache, then only call git lfs pull origin if LFS pointers remain.
  • Update workspace preparation to pass bare repo path and ref into LFS materialization and maintain the existing RuntimeError-and-retry contract when nothing can be materialized.
devlaunch/worktree/workspace_clone.py
Add unit tests that pin the exact LFS cache fill / cache pull command sequences and failure handling.
  • Mock subprocess.run and git-lfs detection to assert the cache fetch runs in the bare, names the ref, and includes both lfs.fetchrecent* configs.
  • Assert that the workspace first pulls from file:// then from origin, with proper cwd.
  • Assert that when cache materialization fully resolves pointers origin is never pulled.
  • Assert that failures in cache fetch or cache pull degrade to origin instead of failing the launch, but that combined failures still raise RuntimeError.
  • Tighten the no-LFS-pointer test to assert no git lfs invocations at all, including the new git -c ... lfs fetch form.
test/test_workspace_clone.py
Introduce integration tests to verify LFS object sharing via the bare cache, and enforce host-agnostic LFS configuration.
  • Create an LFS-enabled fixture remote and local git config that ensures filter.lfs is installed for tests while remaining isolated from the developer environment.
  • Add tests that a second workspace with the remote deleted still materializes LFS content via the cache and that the first workspace leaves payloads in the cache.
  • Add tests that clones record no lfs.storage override and keep exactly one remote (origin) still pointing at the forge URL.
test/integration/test_lfs_object_sharing.py
Reuse a shared real WorkspaceCloneManager fixture across integration tests.
  • Move the real WorkspaceCloneManager fixture into git_fixtures.clone_manager and reuse it in integration tests.
  • Remove duplicate clone_manager fixtures from individual integration modules and import the shared fixture via conftest.
test/fixtures/git_fixtures.py
test/integration/test_clone_object_sharing.py
test/integration/test_repo_manager_real.py
test/conftest.py
Strengthen existing integration test for packfile sharing to check full file identity.
  • Update the packfile-sharing test to compare (st_dev, st_ino) instead of only st_ino and retain link-count assertions.
test/integration/test_clone_object_sharing.py
Ensure test environment has a real git-lfs binary and document the new LFS sharing behavior.
  • Add git-lfs >=3.4,<4 as a pixi test feature dependency so LFS integration tests run instead of skipping.
  • Document LFS cache-store behavior and constraints in README and CHANGELOG, including host-prep timing row update.
  • Accept pixi.lock format v7 with effectively unchanged dependency set aside from git-lfs, keeping CI green.
pyproject.toml
README.md
CHANGELOG.md
pixi.lock

Assessment against linked issues

Issue Objective Addressed Explanation
#163 Make the bare cache (<bare>/lfs) the shared LFS store by changing _materialize_lfs to: (1) run git -c lfs.fetchrecentrefsdays=0 -c lfs.fetchrecentcommitsdays=0 lfs fetch origin <ref> in the bare as cwd; (2) run git lfs pull "file://<bare_path>" in the workspace to hardlink objects; (3) only then run git lfs pull origin if LFS pointers remain, preserving the existing RuntimeError-and-retry contract and not gating on is_new_workspace. ✅
#163 Ensure the design persists no host-specific configuration into the clone (no lfs.storage, no extra remotes) and add tests (unit and integration) that: verify cache-side fetch runs in the bare with the fetchrecent knobs and named ref, verify cache-first then origin materialization and hardlinking behaviour including offline/broken remote cases, and strengthen the no-LFS repo gate so no git-lfs commands run when there are no pointer files. ✅
#163 Add git-lfs as a dependency of the pixi test feature so the new LFS-related tests run instead of silently skipping. ✅

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@codecov

codecov Bot commented Aug 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.38%. Comparing base (ba14410) to head (4dd28c8).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #209      +/-   ##
==========================================
+ Coverage   92.28%   92.38%   +0.09%     
==========================================
  Files          24       24              
  Lines        3448     3467      +19     
==========================================
+ Hits         3182     3203      +21     
+ Misses        266      264       -2     
Files with missing lines Coverage Δ
devlaunch/worktree/workspace_clone.py 93.62% <100.00%> (+1.73%) ⬆️

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@blooop blooop left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during review.

Two-axis review of #209 against ticket #163 at merge-base ba14410. Preflight: all checks green (ci ×5, e2e, prek, gate, codecov patch+project, GitGuardian), non-empty diff (11 files, +1120/−472), MERGEABLE.

Standards

Nothing blocking. The change matches this repo's house style (constraint-carrying docstrings, static helpers on WorkspaceCloneManager, timing.span per subprocess, module-level pytestmark, fixtures re-exported through test/conftest.py's import + __all__), and the lockfile claim verifies.

  • Lockfile claim verified mechanically (observation). Parsed both pixi.lock revisions: across all five environments (default, py310–py313, linux-64) the only membership change is +git-lfs-3.7.1-hd3da7ce_3.conda; zero removals, zero version/build changes, no environment added or dropped. The 872-line churn is the v6→v7 format rewrite alone. One residual: v6 recorded options: pypi-prerelease-mode: if-necessary-or-explicit in each of the 5 envs and v7 records no options block anywhere. Package set is identical so resolution is unaffected, but it is the one line of the rewrite that isn't pure reshaping. Also nothing pins a pixi version (setup-pixi@v0.9.6, no pixi-version), so contributors on a pre-v7 pixi now can't read the lock.
  • _materialize_lfs shape is defensible but asymmetric (nit). It is now gate → cache-fill → cache-pull → gate → inline origin pull. Two of three steps are extracted helpers with logger.warning-and-continue semantics; the third is inlined with raise semantics. Extracting _pull_lfs_from_origin would make the body five lines that read as the phase diagram the 30-line docstring describes. Not a god function — it's an orchestrator with one inlined leaf.
  • Rationale prose is duplicated ~5× (nit, Shotgun Surgery in comments). The "file:// not -c lfs.storage, because GIT_CONFIG_PARAMETERS reaches the remote-side child" argument and the "measured, git-lfs 3.7.1, ext4, hardlinks" claim each appear in _pull_lfs_from_cache, _materialize_lfs, the test_lfs_object_sharing.py module docstring, its class docstring, README and CHANGELOG. When the constraint changes, five places drift.
  • Stale prose the rider left behind (nit). devlaunch/worktree/workspace_clone.py:590 still says the integration test asserts "same st_ino" — the rider changed exactly that test to (st_dev, st_ino).
  • Fixture name now means two things (nit). The folded clone_manager (real cache, real git) shares its name with local fixtures in test/test_workspace_clone.py:128 (mock-based) and test/test_repo_lock_cycles.py:234 (tmp_path). Shadowing resolves it, but the fold's own argument — one name, one wiring — points at real_clone_manager, matching the sibling real_managers.
  • Sibling test files re-derive the identity check (nit). identity() in test_lfs_object_sharing.py:131; test_clone_object_sharing.py:91 inlines the same tuple pair, in a file the new module names as its sibling.

Spec

Verified locally: git-lfs 3.7.1 present; all 4 new integration tests pass; full suite 1624 passed (matches PR claim); mutation _fill_cache_lfs_store/_pull_lfs_from_cache removed → exactly the 2 sharing tests go red, the 2 host-specific guards stay green.

Nothing blocks.

Spec conformance (all satisfied):

  • "Run git -c lfs.fetchrecentrefsdays=0 -c lfs.fetchrecentcommitsdays=0 lfs fetch origin <ref> with the bare as cwd" — workspace_clone.py:339-361, argv and cwd=bare_path exactly as written, ref is workspace.ref.
  • "Then git lfs pull "file://<bare_path>" in the workspace" — :376-380, cwd=ws_path.
  • "Keep today's git lfs pull origin only as a second phase, entered when _has_lfs_pointers still holds" — :422-435: the gate is the content predicate re-run, not exit codes. Cache-phase failures are caught and only warned. Confirmed empirically: the remote-deleted integration test would raise RuntimeError if a pointer survived, and it passes.
  • "Keep the existing RuntimeError-and-retry contract and the deliberate non-gating on is_new_workspace" — origin except clause byte-identical; _materialize_lfs call at :708 is at method-body indent, outside the is_new_workspace block.
  • "no lfs.storage in .git/config, no extra remote" — both integration guards pass; URL checked, not just name.
  • "Add git-lfs to the pixi test-feature dependencies" — pyproject.toml:36-38; all five environments include test, so it is genuinely present.
  • "Touch nothing in repo_manager.py's fetch flow" — untouched.
  • Amendment: "Any test this ticket adds against the union behaviour must use real git." No new unit test asserts union behaviour; the union path is exercised only by real git in the new integration file. Compliant. The amendment's "re-examine the predicate under this ticket's own design" is satisfied only implicitly (the remote-deleted test would fail if it didn't hold) and is never stated as such in the PR body — non-blocking.

Non-blocking findings:

  • Stale comment at workspace_clone.py:625-628: "the clone source is the local bare cache, which has no LFS objects". After this PR the bare does hold LFS objects for previously-fetched refs, so the stated rationale for GIT_LFS_SKIP_SMUDGE=1 is now false. Behaviour is still correct; the comment contradicts the feature it sits next to. The same wording was correctly deleted from _materialize_lfs's docstring and missed here.
  • Every new workspace of an already-full cache still pays a network round trip for the bare-side fetch. Spec-mandated ("1. Run … 2. Then …"), so not a defect — just noting the saving is the payload, not the trip.

Declared deviations:

  1. pixi.lock v6→v7 — sound. Verified mechanically by both axes independently: the sorted set of conda:/pypi: URLs differs by exactly one line, git-lfs-3.7.1-hd3da7ce_3.conda.
  2. Two of four integration tests green-before — sound. The ticket itself asks for them; they are regression guards for a constraint the ticket calls binding, and M7 is caught by nothing else. The reviewer's own red/green run reproduces exactly the split the PR declares.
  3. #208 riders folded in — sound. Test-only, at the same seams, no production behaviour; the clone_manager move is shadowed correctly by the two local fixtures that still define their own.

Verdict

Approve — no blocking findings on either axis; the Spec axis independently reproduced the mutation split and the union-path behavior with real git. Non-blocking riders for the next toucher of these seams: fix the now-false comment at workspace_clone.py:625-628 (bare "has no LFS objects") and the stale "same st_ino" prose at :590; consider real_clone_manager as the shared fixture's name; consider extracting _pull_lfs_from_origin. GitHub refuses --approve from the PR author's account, so this is posted as a comment; this written verdict governs. Merging with a merge commit per repo rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The bare cache becomes the repo's LFS store; workspaces hardlink from it

1 participant