Skip to content

[2.x] feat: fail integration tests that run N+1 queries (and fix the first one it found) - #4871

Merged
imorland merged 6 commits into
2.xfrom
im/testing-n1-detector
Jul 31, 2026
Merged

[2.x] feat: fail integration tests that run N+1 queries (and fix the first one it found)#4871
imorland merged 6 commits into
2.xfrom
im/testing-n1-detector

Conversation

@imorland

@imorland imorland commented Jul 31, 2026

Copy link
Copy Markdown
Member

Changes proposed in this pull request

Every request sent through Flarum\Testing\integration\TestCase::send() is now inspected for N+1 query patterns, and the test fails when one is found.

The motivation is concrete: while profiling discussion-view performance this week, one extension turned out to be issuing a query per post on every page of every discussion, and another declared default includes that were never eager-loaded. Both were found by hand-profiling a live forum — no test noticed, because no test could. Extension authors should get that feedback while writing the feature.

And it immediately proved the point. On this PR's first CI run the detector failed two tests in flarum/flags — a bundled extension nobody had profiled, whose suite was green. GET /api/flags declares post.discussion and post.user as default includes but never eager-loaded what those nested resources need: core's DiscussionResource eager-loads the actor's discussion state on its own endpoints, and that doesn't carry over when a discussion is included by another resource. So every flag on the moderation page read discussion_user on its own, and the flag authors' groups were re-fetched per flag. That fix is included here (FlagResource), because a detector that lands with a known finding allow-listed on day one teaches the wrong lesson.

How it detects. An N+1 is one query shape executed once per record. RepeatedQueryDetector groups a request's query log by normalised SQL (IN lists collapsed so batched loads of differing sizes count as one shape, numeric and quoted literals replaced, transaction/savepoint noise skipped) and fails when a shape repeats past a threshold (default 5).

Two tiers, decided by whether the work grows with the data. Bindings are counted rather than folded into the shape, and that count is what separates a defect from mere waste:

Signature Meaning Result
distinct bindings ≈ executions (10x, 10 distinct) one query per record — add rows, add queries fails
a few values repeated (8x, 1 distinct) wasteful but bounded; five queries for two users stays five at two million warns

The warning is a E_USER_WARNING, which PHPUnit attributes to the triggering test — the author sees it without the build going red.

Making the warnings actually visible. As first written the warning tier was invisible three times over: the trigger_error call was @-silenced so PHPUnit never saw it, only one of eighteen phpunit configs in this repo enabled displayDetailsOnTestsThatTriggerWarnings, and a developer who relies on CI would have had to read the middle of a job log regardless. Fixed here: the @ is gone, every integration config displays warning details, and the detector appends findings to FLARUM_REPEATED_QUERY_LOG when set. The reusable backend workflow points that at a temp file and turns it into GitHub annotations (errors for N+1s, warnings for non-scaling repetition) plus a table in the run summary — so findings land on the pull request itself, where the CI-only audience is looking.

This calibration came from running the detector across every bundled extension: beyond the flags N+1 it produced 22 further findings, and all of them were the second kind (5–10 executions, 1–4 distinct values), mostly on post-save paths where a formatter resolves each mention. Failing on those would have meant rewriting formatter and write-path code for no scaling benefit — the threshold was what was wrong, not the extensions. mentions (19 findings) and subscriptions (1) now pass with warnings; the flags N+1 still fails.

On by default, with graduated escapes:

Escape Scope When
allowedRepeatedQueries() one query shape Preferred — rest of the request stays covered
detectsRepeatedQueries() one test case A test that legitimately can't satisfy it
FLARUM_DETECT_REPEATED_QUERIES=0 whole run Bisecting an unrelated failure

Verification

  • 11 unit tests on the detector: the real N+1 shape, batched in (…) loads of differing sizes (must not flag), sub-threshold loops, same-bindings repetition, transaction noise, inlined vs bound literals, ordering, and the message format.
  • Against a real N+1: reintroducing the per-post loading in fof/moderator-warnings fails with 10x (10 distinct bindings): select * from warningswherewarnings.post_id = ?. With the fix restored, that extension's whole suite passes with detection on.
  • Against core: tests/integration/api gives identical results with detection on and off — 353 tests, the same pre-existing failures, zero findings.
  • flags: 16 tests green with detection on, after the FlagResource fix (was 2 failures).
  • Bundled extensions: flags, mentions, subscriptions and approval all pass with detection on. tags (5) and likes (1) have pre-existing failures unrelated to queries — identical with FLARUM_DETECT_REPEATED_QUERIES=0.

Reviewers should focus on

  • The threshold (5) and whether the normalisation is too aggressive or not aggressive enough for query shapes I haven't seen.
  • The distinctBindings >= count - 1 rule that decides fail vs warn (one duplicate is tolerated, since batch loaders often re-read a single value while still doing per-record work), and whether that boundary holds for shapes I haven't seen.
  • Implementation note worth knowing: processIsolation="true" makes PHPUnit parse a subprocess's entire output as its result protocol, so printing findings — STDOUT, STDERR, /dev/tty, shutdown functions — all surface as test errors. Failing the assertion is the only channel PHPUnit sanctions here.

Developer documentation: flarum/docs#568.

Confirmed

  • Backend changes: tests are green (run composer test).

Every request sent through the integration TestCase is now inspected for
N+1 query patterns, and the test fails when it finds one. Extension
authors get the feedback while writing the feature rather than when a
forum grows: this session alone, one extension was issuing a query per
post on every page of every discussion, found only by hand-profiling a
live forum.

An N+1 is one query shape executed once per record. The detector groups
a request's query log by normalised SQL — IN lists collapsed, literals
replaced — and fails when a shape repeats past a threshold. Bindings are
counted separately rather than folded into the shape: the same SQL run
for four different users is not the same defect as one query per row,
and conflating them produces false positives (it fooled me on one
extension before this distinction existed).

On by default. A single legitimate shape can be exempted with
allowedRepeatedQueries(); a test case can override
detectsRepeatedQueries(); FLARUM_DETECT_REPEATED_QUERIES=0 disables it
for a whole run.

Verified against core's api suite: identical results with detection on
and off (353 tests, same pre-existing failures, no findings), and
against a real N+1 reintroduced in an extension, where it fails with
'10x (10 distinct bindings)' naming the offending query.
@imorland
imorland requested a review from a team as a code owner July 31, 2026 16:47
The flags index declares post.discussion and post.user as default
includes, but never eager loaded what those nested resources need. Core's
DiscussionResource eager loads the actor's discussion state on its own
endpoints; that doesn't carry over when a discussion is included by
another resource. So every flag on the moderation page read
discussion_user on its own, and the flag authors' groups were re-fetched
per flag.

Caught by the N+1 detection added in this branch, on its first CI run.
@imorland imorland changed the title [2.x] feat: fail integration tests that run N+1 queries [2.x] feat: fail integration tests that run N+1 queries (and fix the first one it found) Jul 31, 2026
Running the detector across the bundled extensions turned up 22 findings
beyond the flags N+1, and they were all the same shape: a query repeated
5-10 times for only 1-4 distinct values. That is not an N+1 — five
queries for two users stays five queries whether the forum has two users
or two million. Failing on it would have meant rewriting formatter and
write-path code for no scaling benefit, so the threshold was the thing
that was wrong.

The two cases are now distinguished by the data already being collected.
Roughly as many distinct bindings as executions means one query per
record: that fails, because the work grows with the forum. A handful of
values repeated is wasteful but bounded: that raises a PHP warning, which
PHPUnit attributes to the test without failing the run.

mentions (19 findings) and subscriptions (1) now pass; the flags N+1
still fails when its fix is reverted.
imorland added a commit to flarum/docs that referenced this pull request Jul 31, 2026
Follows the two-tier behaviour in flarum/framework#4871: one query per
record fails the test, while a query repeated for the same few values
raises a warning instead.
The warning tier was invisible in three separate ways. The trigger_error
call was prefixed with @, so PHPUnit never saw it at all — nothing
appeared even with --display-warnings. Only one of eighteen phpunit
configs in this repo set displayDetailsOnTestsThatTriggerWarnings, so
even a working warning printed no detail. And a developer who relies on
CI rather than local runs would have to read the middle of a job log to
find either.

So: the @ is gone, every integration config displays warning details, and
the detector appends findings to FLARUM_REPEATED_QUERY_LOG when it is
set. The reusable backend workflow points that at a temp file and turns
it into GitHub annotations — errors for N+1s, warnings for non-scaling
repetition — plus a table in the run summary. Annotations attach to the
pull request, which is where the audience that most needs them is
looking.
imorland added a commit to flarum/docs that referenced this pull request Jul 31, 2026
Follows flarum/framework#4871: the example phpunit config now displays
warning details, and findings appear as pull request annotations and a run
summary in CI via FLARUM_REPEATED_QUERY_LOG.
Without displayDetailsOnTestsThatTriggerWarnings PHPUnit prints only a
count — 'Warnings: 11' — which is visible but not actionable. The first
warning of a run now carries the pointer to --display-warnings and the
config setting.

Once per run, not per finding: tests run with processIsolation, so each
test is a separate process and a static flag cannot track 'first'. A
marker file keyed to the project and the hour serves as the shared
signal.
@imorland
imorland merged commit a9b5d25 into 2.x Jul 31, 2026
25 checks passed
@imorland
imorland deleted the im/testing-n1-detector branch July 31, 2026 21:00
imorland added a commit to flarum/docs that referenced this pull request Jul 31, 2026
* Document N+1 query detection in integration tests

Companion to flarum/framework#4871, which fails integration tests whose
requests run N+1 queries. Explains how to read a failure, what the
binding count means, and how to exempt a legitimately repeated query
shape.

* Distinguish N+1 failures from non-scaling query warnings

Follows the two-tier behaviour in flarum/framework#4871: one query per
record fails the test, while a query repeated for the same few values
raises a warning instead.

* Document where query findings surface

Follows flarum/framework#4871: the example phpunit config now displays
warning details, and findings appear as pull request annotations and a run
summary in CI via FLARUM_REPEATED_QUERY_LOG.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants