feat: the summary line counts what was verified, not just what was compared

[pftg]

Why

Three separate causes produced a green suite that compared nothing this week:

1. no committed baseline (now loud),
2. rake test running zero system tests in a Rails app,
3. an inherited GIT_DIR redirecting every baseline lookup.

(2) and (3) are invisible to any per-assertion rule — no assertion runs, so nothing can warn. The only signal that catches all three is a count of what was actually verified.

[snap_diff] N screenshots compared, no failures. counted only what it compared, so it was silent about exactly the screenshots it did not — and silent altogether when it compared nothing.

What

[snap_diff] 14 verified, 0 changed, 1 new (not verified). Report: /abs/path.html

• verified — a committed baseline existed and was compared
• changed — of those, the ones that differed
• new — captured but NOT compared, for want of a committed baseline: neither a pass nor a failure (Argos calls this "orphan")

Printed on every run, passing or failing — the way Minitest prints Run options: --seed N: the information is worthless if it only appears once you already know you need it.

The zero case

Same line shape every run (so 0 verified is always the first number the eye lands on), with an explicit sentence appended rather than a separate line — a distinct extra line is easy to lose in a failure dump, and re-wording the whole line would break the shape people learn to scan:

[snap_diff] 0 verified, 0 changed, 0 new (not verified). NOTHING WAS VERIFIED -- no screenshot was compared to a committed baseline.

Evidence — real runs, real git baselines

A throwaway git repo with committed baselines, a user's minitest file, one subprocess per scenario:

scenario	printed	exit
all verified	[snap_diff] 1 verified, 0 changed, 0 new (not verified).	0
one changed	[snap_diff] 2 verified, 1 changed, 0 new (not verified). Report: /private/var/.../screenshots/snap_diff_report.html	1
one new	[snap_diff] 1 verified, 0 changed, 1 new (not verified).	0
zero tests ran	[snap_diff] 0 verified, 0 changed, 0 new (not verified). NOTHING WAS VERIFIED -- no screenshot was compared to a committed baseline.	0

Exit codes are unchanged in every case — this PR touches nothing but a string. Whether a new screenshot fails is #259's decision and is not smuggled in here.

Guards

test/integration/summary_line_test.rb runs test/fixtures/summary_line_case.rb — the user's require, the user's assert_matches_screenshot, real Minitest, real committed baselines — and asserts on what the finished process printed. An in-process assertion on reporter.summary would stay green through a regression that stops the line from being printed at all, which is how this class of bug got through before.

Six mutations, each reverted with a targeted edit and re-verified green:

mutation	reds
verified counter never increments	5 unit, 2 integration
changed count hard-coded to 0	2 unit, 1 integration
new count hard-coded to 0	2 unit, 1 integration
drop the NOTHING WAS VERIFIED clause	1 unit, 1 integration
restore return if total.zero? (silent again)	2 unit, 1 integration
drop the report path	2 unit, 1 integration

rake test:unit (611 runs), rake test:canonical (506 runs), standardrb lib test all green.

Note for review

Four existing #summary expectations in test/unit/reporters/html_reporter_test.rb pinned the old wording ("2 screenshots", "1 failure", "no failures", assert_nil at zero). They are the specification of the line this PR was asked to replace, so they were rewritten rather than preserved — the only edits to existing test expectations in this branch. Nothing else changed behaviour.

🤖 Generated with Claude Code

https://claude.ai/code/session_014BQJX6eWzBj2UTm5zQsjEs

Summary by Sourcery

Report what snapshot runs actually verified, changed, and left unverified, including an unmistakable warning when no screenshot was compared to a committed baseline.

New Features:

• Update the end-of-run snapshot summary to report verified, changed, and unverified new screenshots on every run, including an explicit zero-verification warning.
• Expose the number of screenshots captured without committed baselines for summary reporting.

Bug Fixes:

• Prevent runs that verify no screenshots from appearing silently successful by always printing a meaningful summary line.

Enhancements:

• Preserve report links for changed screenshots while replacing the previous comparison-only wording with verification-focused counts.

Documentation:

• Update reporter documentation to reflect the new summary-line format under forked parallelism.

Tests:

• Add end-to-end subprocess coverage using real Minitest runs and committed git baselines for verified, changed, new, and zero-verification scenarios.
• Revise reporter unit tests to cover verification, change, missing-baseline, report-path, and zero-verification output.

[coderabbitai]

Warning

Review limit reached

Next included review available in 24 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ddde6413-7536-46ba-ae61-1cfd30a28706

📥 Commits

Reviewing files that changed from the base of the PR and between 17a74a6 and e6e222d.

📒 Files selected for processing (6)

• docs/reporters.md
• lib/snap_diff/reporters/html.rb
• lib/snap_diff/reporting.rb
• test/fixtures/summary_line_case.rb
• test/integration/summary_line_test.rb
• test/unit/reporters/html_reporter_test.rb

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Sorry @pftg, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[sourcery-ai]

Reviewer's Guide

Update the HTML reporter summary line to always report what was actually verified (verified/changed/new counts) and add integration coverage that asserts on the real end-of-run output from a separate process with committed baselines, plus minor docs and unit test updates to match the new behavior.

Sequence diagram for the HTML reporter summary output

sequenceDiagram
    participant TestRun
    participant HTMLReporter
    participant Reporting
    participant Output

    TestRun->>HTMLReporter: summary()
    HTMLReporter->>Reporting: missing_baselines_count()
    Reporting-->>HTMLReporter: new count
    HTMLReporter->>HTMLReporter: calculate total and failures
    alt no screenshots verified
        HTMLReporter-->>Output: print 0 verified, 0 changed, 0 new + NOTHING WAS VERIFIED
    else screenshots verified and no failures
        HTMLReporter-->>Output: print verified, changed, and new counts
    else changed screenshots exist
        HTMLReporter->>HTMLReporter: output_path()
        HTMLReporter-->>Output: print counts and report path
    end

Loading

Flow diagram for screenshot verification outcomes

flowchart TD
    A[Capture screenshot] --> B{Committed baseline exists?}
    B -->|Yes| C[Compare screenshot]
    C --> D{Different?}
    D -->|No| E[Increment verified]
    D -->|Yes| F[Increment verified and changed]
    B -->|No| G[Record missing baseline]
    G --> H[Increment new not verified]
    E --> I[Print summary on every run]
    F --> I
    H --> I
    I --> J{Verified count is zero?}
    J -->|Yes| K[Append NOTHING WAS VERIFIED]
    J -->|No| L[Print counts and report path when failures exist]

Loading

File-Level Changes

Change	Details	Files
HTML reporter summary string now reports verified, changed, and new (unverified) screenshot counts, always prints, and includes the report path only when applicable.	Replaced the old conditional summary wording logic with a single base line containing verified, changed, and new counts. Ensured the summary line is never nil and adds a prominent NOTHING WAS VERIFIED clause when total verified is zero. Kept the report path only when there are failures, appending it to the summary line. Documented the semantics of verified/changed/new directly above the summary method.	lib/snap_diff/reporters/html.rb
Reporting now exposes a live count of missing baselines for use in the summary line and clarifies the missing baselines summary documentation.	Added a thread-safe missing_baselines_count accessor that returns the number of screenshots captured without a committed baseline. Updated comments to describe that the reporters’ summary line carries the count of newly captured, unverified screenshots and that missing_baselines_summary names them for git add. Left existing reset_missing_baselines! behavior intact to preserve per-test isolation.	lib/snap_diff/reporting.rb
Added an integration test that runs a real user-style Minitest file in a subprocess against a temporary git repo with committed baselines to assert the printed summary line for several scenarios.	Introduced a fixture test file that uses SnapDiff with a stubbed screenshoter to produce verified, changed, and new screenshots based on environment-controlled cases. Configured SnapDiff in the fixture to use a real git repository with committed baselines and non-failing behavior for new screenshots. Implemented an integration test harness that builds a temporary git repo with baselines, runs the fixture via Open3, and asserts on the final output for passing, mixed, and zero-verified runs.	test/fixtures/summary_line_case.rb test/integration/summary_line_test.rb
Updated existing unit tests for the HTML reporter summary to match the new semantics and add coverage for edge cases including zero verified and non-zero new counts.	Rewrote expectations to assert exact new summary strings and counts instead of previous ‘screenshots compared / failures’ wording. Added unit coverage for scenarios with multiple failures, all passing assertions, zero screenshots recorded, and non-zero missing baselines even when nothing was verified. Ensured the report path appears only when a report is written and omitted for purely passing runs.	test/unit/reporters/html_reporter_test.rb
Adjusted documentation to reflect the new summary line wording under Rails’ forking parallelization.	Updated reporters documentation to mention the new [snap_diff] N verified, N changed, N new … summary line phrasing in the parallelize(workers: N) row. Maintained the explanation that the summary line is not printed under Rails’ forking parallelization.	docs/reporters.md

Possibly linked issues

• #unknown: The PR adds the issue’s requested third summary state, counting verified, changed, and new unverified screenshots.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.