Skip to content

Parallel worktree creation hits the fixed 5-minute git timeout and leaves orphan worktrees that block retry #15541

Open
#15764
Open
Parallel worktree creation hits the fixed 5-minute git timeout and leaves orphan worktrees that block retry#15541
#15764
Labels
bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

Description

@areidyOTH
areidyOTH
opened on Oct 4, 2026

What happened

Seven new-worktree threads were launched together against one project. Six of them failed with "Workspace preparation failed during provision worktree: Git command failed in GitVcsDriver.createWorktree (…): Git command timed out." The user saw them as failed threads. Only one of the seven threads started.

Diagnosis

  • All seven launches ran git worktree add -b <branch> <path> <ref> against the same repository at the same time (traces show starts between 06:58:07 and 06:59:04 UTC).
  • The repository is fairly large: 122,629 tracked files and a 1.28 GiB pack. It lives on a fuseblk (NTFS) mount, and worktrees are created on ext4 under ~/.t3/worktrees.
  • With seven checkouts competing for I/O, each one took about 5 minutes. Six hit the fixed WORKTREE_ADD_TIMEOUT_MS = 300_000 (apps/server/src/vcs/GitVcsDriverCore.ts:50; runGitCommand was interrupted at 300000 ms). The seventh succeeded at 293 s.
  • gitProcesses = Semaphore.makeUnsafe(8) lets all seven worktree adds run concurrently, so starting more launches makes every one of them slower. The timeout is per command and can't be configured.
  • The failed checkouts still produced valid worktrees. Afterwards all seven worktree paths and t3/… branches exist, and each is complete and clean (git ls-files = 122,629 files, git status --porcelain empty). T3 marks the thread as failed but leaves the worktree and branch behind with no thread attached.
  • That leftover state probably also blocks the new "retry workspace preparation" (feat: retry a failed workspace preparation #15326). Retry re-runs git worktree add -b <same branch> <same path>, and both the branch and the path already exist. (This comes from reading createWorktree; not tested.)

Possible directions:

  • Serialize, or cap concurrency for, worktree add per repository.
  • Measure the timeout as time without checkout progress instead of total runtime (progress lines are already streamed via GIT_PROGRESS_DELAY=0).
  • On timeout, kill git and clean up the worktree and branch, or adopt the finished worktree.
  • Make retry tolerate an existing branch and worktree that match the requested ref.

Steps to reproduce

  1. Use a repository with about 100k+ tracked files on a slow filesystem (an NTFS/fuseblk mount reproduces it; a network mount would probably too).
  2. Create about 7 threads in "new worktree" mode on that project at the same time. Here they were launched through the HTTP/RPC API, within a minute of each other.
  3. Most launches fail with "Git command timed out" after 5 minutes, although git worktree list later shows every worktree created successfully.
  4. Press Retry on one of the failed threads. Expected (not verified): it fails because the branch already exists.

Version

0.0.46-nightly.20261004.2644 (desktop AppImage, commit 7379933)

Environment

Linux x64 6.x/7.0.0-38-generic, Node v26.8.2, git 2.53.0, claude 2.1.289; repo on fuseblk (NTFS), worktrees on ext4

Evidence

# server.trace.ndjson (start time UTC, span, durationMs, exit)
06:58:07 GitVcsDriver.createWorktree 394393 Failure  git.cwd=/mnt/<ntfs-volume>/<repo>
06:58:07 runGitCommand               300001 Interrupted
06:58:11 GitVcsDriver.createWorktree 303205 Failure
06:58:28 GitVcsDriver.createWorktree 303465 Failure
06:58:36 GitVcsDriver.createWorktree 301932 Failure
06:58:46 GitVcsDriver.createWorktree 303199 Failure
06:59:04 GitVcsDriver.createWorktree 307751 Failure
06:59:15 GitVcsDriver.createWorktree 292871 Success
ThreadLaunchError: Thread launch <id> failed during provision-worktree.
GitCommandError: Git command failed in GitVcsDriver.createWorktree (/mnt/<ntfs-volume>/<repo>): Git command timed out.

# thread item shown to the user
Workspace preparation failed during provision worktree: Git command failed in GitVcsDriver.createWorktree (/mnt/<ntfs-volume>/<repo>): Git command timed out.

# afterwards
$ git worktree list
/mnt/<ntfs-volume>/<repo>                         38b7dd756 [master]
~/.t3/worktrees/<repo>/t3-<w1-branch>             38b7dd756 [t3/<w1-branch>]
... (all 7 present; each ls-files=122629, status clean)

Related issues

#11735 (worktree creation fails at configureBaseRef and leaves an orphan worktree and branch): it also leaves an orphan, but there the cause is a locked .git/config, not a timeout under concurrency. #11099 (attach to an existing branch) would also cover the retry case.

Fix applied or workaround

The user's own orchestration recreated the worktrees one at a time and relaunched the six threads with --existing-worktree on the leftover paths. No T3 changes were made.

Filed by

Claude Code (claude-opus-5-5) via t3 triage

Activity

  1. juliusmarminge commented on Oct 4, 2026

    @juliusmarminge
    juliusmarminge
    on Oct 4, 2026
    Member

    Note

    Grok responding on behalf of Julius.

    Triage

    Thanks for the traces and for digging into the leftover checkouts, @areidyOTH. I checked this against current main, which has the same worktree code as nightly 73799330, including #15326. The timeout and the leftover worktrees are real. Two details differ from the report: the semaphore isn't what let all seven adds run at once, and Retry only fails the way you describe when the launch named the branch.

    What I found

    The 5-minute limit is fixed, and nothing caps these adds. GitVcsDriver.createWorktree always passes WORKTREE_ADD_TIMEOUT_MS (300_000) to runGitCommand, which matches the trace: runGitCommand is interrupted at 300001 ms. No setting changes that limit.

    gitProcesses is a global semaphore of 8, but only commands with a timeout of 30 seconds or less take a slot. Longer or unlimited timeouts skip it on purpose so long checkouts don't starve short commands like status (#11405). git worktree add uses 300 seconds, so it never takes a slot. All seven adds ran together because nothing limits them, not because the limit is 8.

    A timed-out add is missed by the cleanup from #15326. onWorktreeClaimed runs only after executeGit returns, and the launch service removes a worktree only after that callback has stored its path. A timeout ends inside executeGit, so the path is never stored and git worktree remove never runs, even though git has usually registered the worktree by then. Your trees were complete because they were killed near the end of a roughly 5-minute checkout. A kill earlier in the checkout could leave a partial tree that's orphaned the same way.

    Retry reuses a worktree only if the thread recorded one. retryPreparation switches to existing_worktree only when both worktreePath and branch are set. Otherwise it runs the original strategy again.

    • If the launch named the branch, as your HTTP launches did (t3/…), the retry runs git worktree add -b with the same branch and a path derived from it. Both already exist, so the retry fails. I confirmed that from the code, not in the app.
    • If the launch didn't name a branch, the server picks a new t3code/<8 hex> name on each attempt, so Retry creates a second worktree and the first orphan stays behind.

    Your --existing-worktree relaunch worked because that strategy never runs git worktree add.

    Related

    #11735 also leaves a worktree behind, but there the failure happens after git worktree add has returned, so the path is recorded and the #15326 cleanup can remove it. #11099 asks to attach a thread to an existing branch, which wouldn't stop concurrent adds from timing out.

    Likely fix area

    • Concurrency: one option is to limit or queue git worktree add per repository, separate from the short-command semaphore. Another is to time out on lack of checkout progress instead of total runtime.
    • Timeout cleanup: either adopt a finished checkout that matches the requested branch and ref, or remove the worktree and branch git already registered.
    • Retry: handle an existing branch and worktree when the failed attempt never recorded a path.

    A maintainer will decide on the fix direction.

  2. juliusmarminge
    added
    bugSomething is broken or behaving incorrectly.
    via-triageFiled through npx t3 triage
    on Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.via-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions