Skip to content

fix(server): bulk worktree launches no longer fail on Git races - #17197

Open
jakeleventhal wants to merge 4 commits into
pingdotgg:mainfrom
jakeleventhal:t3/fix-concurrent-worktree-creation
Open

jakeleventhal wants to merge 4 commits into
pingdotgg:mainfrom
jakeleventhal:t3/fix-concurrent-worktree-creation

Conversation

@jakeleventhal

@jakeleventhal jakeleventhal commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Concurrent launches into one repository can fail on fetch ref locks, occupied branch names or paths, and shared Git config writes. Retry can hit the same collision again.

This keeps launches independent: concurrent fetches of the same ref share their in-flight result, branch/path collisions retry with a bounded suffix (temporary branches get a fresh hash), and branch creation, merge-base config writes, and renames share a lock keyed by the Git common directory. Checkout stays parallel. A launch never retries once it has claimed a worktree path, and retrying a failed thread keeps its existing run and message.

Maintainer takeover of @jakeleventhal’s #17197, preserving the contributor’s commits and merging current main. The implementation also locks upstream config written during branch creation. PR-added test cases were reduced from 13 to 4 across the existing launch and Git driver test files.

Verification on headless Linux:

  • Real Git service integration: six simultaneous launches preserve an occupied branch, its uncommitted file, and an unrelated occupied directory. Each launch gets a separate worktree, branch, merge base, and upstream tracking.
  • The five focused behavior checks pass: bulk launches, bounded collision retries recovering the same run, concurrent rename/config preservation, shared fetches, and claimed-worktree cleanup.
  • Both complete touched suites: 246 passed, 1 failed. The existing review diff previews > preserves renames, unusual paths, modes, and binary statistics failure also reproduces with main’s production code.
  • All four retained PR-added regression cases fail with main’s production implementation restored; they pass with this fix.
  • Server typecheck, targeted lint, formatting, and diff checks pass.
  • Two independent full-diff review rounds using Claude Opus 5.5 and Codex GPT-6 Sol. The first-round unused-branch concern was reproduced on main as well; it is disclosed below rather than adding branch deletion to this fix.

Service-level tests exercise the server and real temporary Git repositories; screenshots do not demonstrate this server concurrency behavior. No browser, Electron desktop, macOS, Windows, iOS, or Android verification was performed. Network fetch coalescing is verified with a controlled process fixture, not a live remote race.

Coordination is local to one Git driver instance; external Git processes do not use its locks. As with Git’s existing worktree add -b, an attempt that fails before claiming a worktree can leave an unused branch. Collision retries skip it without deleting existing work.

Implementation: GPT-6.1 Sol (high) through the Codex harness in T3 Code. Tests: GPT-6 Astra (high), Codex harness. Independent reviews: Claude Opus 5.5 and GPT-6 Sol.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Oct 8, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 8, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at bcedc86

Macroscope's review found this PR approvable — This is a localized server-side concurrency bug fix that makes existing bulk worktree launches resilient to Git ref, path, config, and fetch races. The production changes are confined to the orchestration and VCS layers, with focused tests covering the new retry and synchronization behavior.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 54d11896-684e-468f-b5de-52da6842890a
📥 Commits

Reviewing files that changed from the base of the PR and between ea5c623 and 34c6bf5.

📒 Files selected for processing: 4
  • apps/server/src/orchestration-v2/ThreadLaunchService.test.ts
  • apps/server/src/orchestration-v2/ThreadLaunchService.ts
  • apps/server/src/vcs/GitVcsDriverCore.test.ts
  • apps/server/src/vcs/GitVcsDriverCore.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

📝 Walkthrough

GitVcsDriverCore classifies branch-ref collisions, serializes selected repository updates, and shares concurrent matching fetches. ThreadLaunchService retries eligible worktree branch or path collisions. Tests cover concurrent launches, retry limits, failure handling, recovery, and cleanup.

Changes: Git Concurrency and Worktree Provisioning

Layer / File(s) Summary
Classify collisions and serialize Git updates
apps/server/src/vcs/GitVcsDriverCore.ts, apps/server/src/vcs/GitVcsDriverCore.test.ts
The driver classifies existing refs/heads/... collisions as branch_already_exists and serializes worktree base-ref configuration and branch renames by repository. Tests cover ref collision classification and concurrent renames.
Coalesce matching remote fetches
apps/server/src/vcs/GitVcsDriverCore.ts, apps/server/src/vcs/GitVcsDriverCore.test.ts
Concurrent fetches with the same working directory, remote, and optional ref share an in-flight operation. A later fetch starts a new operation after completion.
Retry eligible worktree collisions
apps/server/src/orchestration-v2/ThreadLaunchService.ts, apps/server/src/orchestration-v2/ThreadLaunchService.test.ts
Worktree creation retries branch or path collisions before a path is claimed, up to 100 retries. Tests cover concurrent launches, temporary names, retry limits, recovery, and partial-checkout cleanup.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: bil0000

Merge Risk: ⚪ Minimal · up to 34c6b

The investigated concurrency and retry paths do not show a remaining obstacle to merging after normal checks.

Security architecture risk: 🔵 Low · up to 34c6b

The changes reduce Git races while preserving the repository and base reference used by each launch. No introduced security issue was established. Remaining uncertainty concerns partial Git failures before a worktree is reported as created, rather than a demonstrated expansion of access or privileges.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The changed retry flow operates within the selected project's repository and configured worktree location. It expands the number of candidate branch/path attempts, not the repository or base-reference authority passed to Git. The supplied evidence does not establish deployment-wide or tenant-wide exposure.

Trust Boundaries and Controls

  • observed — The new atomic-ref collision classification is restricted to refs/heads. Only classified branch/path collisions enter the provisioning retry loop; authentication, host-key, and other failures do not satisfy its retry predicate. Git diagnostic classification excludes remote-prefixed output.

Resilience and Maintainability Implications

  • observed — A reported worktree claim prevents collision retries. Failure before workspace recording, or cancellation before run release, attempts removal of the claimed path; thread metadata is cleared only after successful removal. Ordinary failure after recording preserves the workspace for recovery.
  • observed — Ownership reporting occurs after successful git worktree add, and worktree removal contains no branch-ref deletion. Both behaviors existed at the PR base. Inspected evidence does not establish Git's partial-failure rollback before ownership reporting, so orphaned state or its amplification by retries is unresolved, not a verified introduced issue.

Hardening Proposals

  • proposed — Add fault-injection coverage around real worktree creation, including interruption before ownership reporting and failure after ref creation. Establish which directory and ref state survives, and ensure recovery never deletes state owned by another launch.
Pre-merge checks: Passed 1 · Failed 1 · Inconclusive 2

Failed Failed checks: 1 warning · 2 inconclusive

Check name Status Explanation Resolution
Description check Warning The description covers the problem, implementation, scope, verification results, limitations, and agent attribution. However, it omits the required section headings and does not include an explicit ma… Reformat the description with the required Problem, Change, Scope and approval, and Verification headings. Add a link to the explicit maintainer approval comment or explain why this focused bug fix qualifies for the approval exemption.
Linked Issues check Inconclusive The linked issue assessment could not be completed. Retry the assessment when the required review evidence is available.
Out of Scope Changes check Inconclusive The linked issue assessment could not be completed. Retry the assessment when the required review evidence is available.
Passed Passed checks: 1 passed
Check name Status Explanation
Title check Passed The title clearly identifies the server fix for bulk worktree launch failures caused by Git races. It is concise and specific.
Full details: Description check

Explanation

The description covers the problem, implementation, scope, verification results, limitations, and agent attribution. However, it omits the required section headings and does not include an explicit maintainer approval comment or link as required by the Scope and approval section.

Resolution

Reformat the description with the required Problem, Change, Scope and approval, and Verification headings. Add a link to the explicit maintainer approval comment or explain why this focused bug fix qualifies for the approval exemption.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

aliceisjustplaying added a commit to aliceisjustplaying/t3code that referenced this pull request Oct 8, 2026
Keep upstream migration IDs 59 and 60; move YSK to 61 and repair historical fork ledgers atomically after applying missing schema changes.

Adapt selected upstream fixes for Pi approvals, MCP images, workspace discovery, provider worker starvation, concurrent worktree launches, primary runtime ownership, session refresh, DPoP URLs and embedded-render scrolling. Preserve fork lifecycle and notice behavior, and close the approval and authentication gaps found in review.

Adapted from pingdotgg#16854, pingdotgg#15879, pingdotgg#17190, pingdotgg#16790, pingdotgg#17197, pingdotgg#17172, pingdotgg#17075, pingdotgg#17037, pingdotgg#17065.

Co-authored-by: Adamulek123 <adam.bogucki2018@gmail.com>
Co-authored-by: anntnzrb <anntnzrb@proton.me>
Co-authored-by: Wout Stiens <71498452+StiensWout@users.noreply.github.com>
Co-authored-by: Lakshmi Tanmay <lakshmi@voltcrash.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Leventhal <jakeleventhal@me.com>
Co-authored-by: Malte Sussdorff <malte.sussdorff@cognovis.de>
Co-authored-by: sheehanmunim <sheehanmunim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Dara Adedeji <daraadedeji07@gmail.com>
Co-authored-by: Joseph Vidal <josephv4000@gmail.com>
@jakeleventhal
jakeleventhal force-pushed the t3/fix-concurrent-worktree-creation branch 2 times, most recently from 06b39da to b6bd69f Compare October 9, 2026 21:34
@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 11, 2026 — with ChatGPT Codex Connector
@macroscopeapp
macroscopeapp Bot dismissed their stale review October 11, 2026 06:51

Dismissing prior approval to re-evaluate b6bd69f

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 11, 2026
@jakeleventhal
jakeleventhal force-pushed the t3/fix-concurrent-worktree-creation branch 2 times, most recently from 32ab2f6 to ea5c623 Compare October 12, 2026 16:29
@jakeleventhal
jakeleventhal force-pushed the t3/fix-concurrent-worktree-creation branch from ea5c623 to 34c6bf5 Compare October 14, 2026 14:27
@macroscopeapp
macroscopeapp Bot dismissed their stale review October 14, 2026 14:28

Dismissing prior approval to re-evaluate 34c6bf5

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 14, 2026
@jakeleventhal
jakeleventhal force-pushed the t3/fix-concurrent-worktree-creation branch 3 times, most recently from e7aeb01 to b044dc4 Compare October 18, 2026 20:59
@macroscopeapp
macroscopeapp Bot dismissed their stale review October 18, 2026 21:00

Dismissing prior approval to re-evaluate b044dc4

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 18, 2026
@juliusmarminge

Copy link
Copy Markdown
Member

Taking over this one as maintainer for batch worktree-git-16. I will preserve author credit, merge current main, reduce the tests, rebuild the Git race fix, and run independent adversarial reviews. No merge is planned.

@juliusmarminge juliusmarminge self-assigned this Oct 19, 2026
@macroscopeapp
macroscopeapp Bot dismissed their stale review October 19, 2026 06:48

Dismissing prior approval to re-evaluate 6cb55dd

@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Oct 19, 2026
Launching many threads into new worktrees at once failed most of them:
concurrent fetches of the base lost the ref lock, branch and path names
collided, and concurrent writes to the shared Git config failed on its
lock. Retry hit the same collisions again.

Concurrent fetches of one ref now share a single running fetch. A taken
branch or path gets the next `-1`, `-2` suffix, the same style renames
use, while Git stays the arbiter. Config writes and branch renames take a
per-repository lock, so checkouts stay concurrent and concurrent renames
pick distinct names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jakeleventhal
jakeleventhal force-pushed the t3/fix-concurrent-worktree-creation branch from 6cb55dd to bcedc86 Compare October 19, 2026 21:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:L 100-499 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants