Skip to content

fix: concurrent callers each spawned their own git rev-parse - #253

Merged
pftg merged 1 commit into
masterfrom
fix/vcs-cache-thundering-herd
Aug 24, 2026
Merged

fix: concurrent callers each spawned their own git rev-parse#253
pftg merged 1 commit into
masterfrom
fix/vcs-cache-thundering-herd

Conversation

@pftg

@pftg pftg commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #250, found while auditing parallel test execution.

#250 caches git rev-parse --show-toplevel per root, removing ~200 process spawns from a 200-screenshot suite. Under threads it removed none.

MRI releases the GVL for the duration of Open3.capture3, so concurrent callers all miss key? before any of them writes. Measured on master, 8 threads asking about one root:

git spawns for ONE root, 8 concurrent callers: 8

That is the cost the cache exists to remove, still fully paid — in parallelize(with: :threads), which is the default parallel mode on JRuby. JRuby also has no GVL, so the unsynchronized Hash#[]= is unsafe there as well as wasteful.

The fix synchronizes the lookup rather than the write, so the spawn happens once. Holding the lock across the subprocess is deliberate: callers almost always share a single root, so the remaining threads wait once and then read the cache.

After: git spawns for ONE root, 8 concurrent callers: 1

Guard: test/unit/vcs_test.rb drives 8 concurrent checkout_vcs calls through a stubbed capture3 that sleeps (standing in for the GVL-releasing subprocess) and asserts exactly one lookup. Mutation-checked — replacing the mutex with a lock that merely yields reds the test; restored, green.

588 unit runs / 0 failures, standardrb clean.

Summary by Sourcery

Synchronize Git-root caching so concurrent VCS operations share a single repository lookup.

Bug Fixes:

  • Prevent concurrent callers from spawning duplicate git rev-parse processes when resolving the same repository root.
  • Make cached Git-root lookups safe for parallel execution, including runtimes without a global VM lock.

Tests:

  • Add coverage verifying that eight concurrent VCS lookups perform exactly one Git lookup.

The per-root cache added in #250 removed 200 process spawns from a
200-screenshot serial suite, but bought nothing under threads. MRI
releases the GVL for the whole of `Open3.capture3`, so every thread
misses `key?` before any thread writes: measured 8 spawns for 8
threads asking about a single root.

That is the default parallel mode on JRuby, which additionally has no
GVL to make the unsynchronized Hash write safe.

Synchronize the lookup itself, not just the write. Holding the lock
across the spawn is deliberate -- callers almost always share one root,
so the other threads wait once and then read the cache.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @pftg, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 44 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 32d7aa51-4597-4b2f-ade1-438aadad519c

📥 Commits

Reviewing files that changed from the base of the PR and between d5b7a9e and e347e74.

📒 Files selected for processing (2)
  • lib/snap_diff/vcs.rb
  • test/unit/vcs_test.rb

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai

sourcery-ai Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Synchronizes git root caching in SnapDiff::Vcs so concurrent thread callers share a single git rev-parse invocation, and adds a unit test that exercises and mutation-checks the concurrent behavior via a stubbed capture3.

Sequence diagram for synchronized git root lookup

sequenceDiagram
    participant T1 as Thread 1
    participant T2 as Thread 2
    participant Vcs as SnapDiff::Vcs
    participant Git as Open3.capture3

    par Concurrent callers
        T1->>Vcs: git_root_for(root_path)
        T2->>Vcs: git_root_for(root_path)
    end
    Vcs->>Vcs: @git_roots_lock.synchronize
    Vcs->>Git: Open3.capture3(git, -C, root_path, rev-parse, --show-toplevel)
    Git-->>Vcs: git_root, status
    Vcs->>Vcs: @git_roots[root_path] = status.success? && git_root.chomp
    Vcs-->>T1: cached git root
    Vcs-->>T2: cached git root
    Vcs->>Vcs: @git_roots_lock.synchronize
    Vcs-->>Vcs: @git_roots.key?(root_path)
Loading

Flow diagram for concurrent git root cache behavior

flowchart TD
    A[Concurrent git_root_for calls] --> B{Cache lookup under mutex}
    B -->|miss| C[Open3.capture3 git rev-parse]
    C --> D[Store result in @git_roots]
    D --> E[Return cached git root]
    B -->|hit| E
Loading

File-Level Changes

Change Details Files
Make git root lookup and caching thread-safe so concurrent callers only spawn one git rev-parse subprocess and share the cached result.
  • Introduce a class-level Mutex to guard access to the git root cache
  • Wrap git_root_for cache lookup and potential git rev-parse call in a synchronized block
  • Return cached git root immediately when present within the synchronized section, otherwise perform git rev-parse and write the result back into the cache
lib/snap_diff/vcs.rb
Add a concurrency-focused unit test that verifies multiple threads share a single git root lookup via the cache.
  • Create a new test that spawns multiple threads calling checkout_vcs against the same root directory
  • Stub Open3.capture3 with a counting function that sleeps to simulate GVL-releasing subprocess behavior
  • Use a Mutex-protected counter to track how many times the git rev-parse stub is invoked and assert it is exactly one
test/unit/vcs_test.rb

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@pftg
pftg merged commit 4bd7416 into master Aug 24, 2026
8 checks passed
@pftg
pftg deleted the fix/vcs-cache-thundering-herd branch August 24, 2026 06:13
pftg added a commit that referenced this pull request Aug 24, 2026
The v2.0.0 section was written before #250, #253, #254, #255, #256, #261,
#263, #264, #266 and #267 landed, and three of its claims had gone false:

- "Known limitations: fork-based parallel tests produce no HTML report ...
  Fixed in 2.1" -- fixed in 2.0 by #266. Reproduced both sides here:
  1.15.1 + `parallelize(workers: 2, threshold: 0)` writes NO report and
  prints no summary line; master writes one merged report and
  `4 verified, 4 changed, 0 new`.
- "a suite whose only contact with the v1 API is
  `require \"capybara_screenshot_diff/minitest\"` + `include ...Assertions`
  still prints nothing" -- #263 made the require doors warn. That exact
  setup now prints the migration notice; verified in a scratch project.
- "Two removals 2.0 cannot warn about ... `driver:` as a setting" -- #263
  made both the setting writer and the per-screenshot key warn. Verified:
  `Capybara::Screenshot::Diff.driver = :vips` prints the removal line with
  a call site.

And the silent-by-design constant list repeated the shape of the beta2
`defined?` mistake: it listed "Os, Region" inside a run of
`Capybara::Screenshot::Diff::` names. Probed on master --
`defined?(Capybara::Screenshot::Diff::Os)` and
`defined?(Capybara::Screenshot::Diff::Region)` are both nil. The real
names are `Capybara::Screenshot::Os` and the top-level `Region`, neither
of which existed under `::Diff` in 1.15.1 either. Fully qualified now, and
`::Comparison` added to match docs/UPGRADING.md.

New material, every claim checked against the code or a live run:

- a "why upgrade" section for the four green-suite-testing-nothing bugs
  (#255, #256, #254, #266), plus the unfollowable CI message (#267) and
  the fail_if_new precedence change
- before/after transcripts of the failure message (#264), taken from the
  same page rendered on 1.15.1 and on master
- the summary line (#261), with the fact that it comes from the HTML
  reporter and needs its one-line require -- an omission that would have
  read as a missing feature
- the #250 / #253 perf table, attributed to its harness, with columns
  labelled before/after rather than 1.x/2.0
- the libvips fix is stated as guarded on libvips 8.15+, so a reader on an
  older libvips knows the bug is still theirs

Install snippets stay pinned to 2.0.0.beta3 on purpose: `~> 2.0` resolves
to nothing on rubygems today. docs/RELEASE_PREP.md already carries a
precise step to swap all five (its grep finds exactly those five), and
gains one line so the record-modes placeholder in the entry cannot ship
unfilled.

`rake test:unit` 651 runs / 0 failures, `standardrb lib test` clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant