Optimize CI by categorizing tests and parallelizing builds even more

[johnsimons]

Why

CI took ~14.3 minutes wall clock. The assumption was that Windows runners are just slow, so pre-compiling on Linux would fix it. Measuring run 31189671891 showed the real cause was redundant work:

Waste	Evidence
Every job compiled the whole solution	dotnet build src built all 76 projects in all 25 jobs: 180-281s on Windows
Every job ran every test assembly	All 31 assemblies started in every job; the category filter is a runtime Assert.Ignore, so irrelevant ones still spin up. Windows-AzureServiceBus burned 73s of its 372s on this
Setup ran serially with the build	Build (281s) → Azure login (27s) → ASB setup (55s), back to back
Every job downloaded a RavenDB server it never used	7-18s × 25 jobs; only ServiceControlInstaller.Packaging needs it

But the dependency closures are tiny: a transport test project needs 6 of 76 projects.

A cross-OS binary artifact was considered and rejected. It makes the build a serial stage (est. ~12.4m, barely better) and risks the net10.0-windows WPF/WinUI projects, Linux ELF apphosts, and absolute Linux NuGet paths in runtimeconfig.dev.json.

Result

	Before	After
Wall clock	14.3m	8.6m
Runner minutes	213m	149m
Jobs	35	48

Windows-AzureServiceBus, the old bottleneck, went 14.3m → 7.0m.

What changed

Scoped builds. Each test project declares <TestCategory>. tools/select-test-projects.ps1 scans for it and generates a tests.proj traversal file (using Microsoft.Build.NoTargets, already pinned in global.json), so a job builds only its category's closure. tools/run-tests.ps1 then runs only that category's assemblies.

Overlapped build and infrastructure, using the new parallel steps feature. The build is background: true and the job joins it at wait: build after the infra chain.
background rather than a parallel: group because the infra steps are an ordered chain (WSL before the database actions) that must run concurrently with the build, which a group can't express.

Split the two fat categories. Default → DefaultCore / DefaultAudit / DefaultMonitoring, and RabbitMQ → one per routing topology. Both were single jobs running several slow assemblies back to back.

Containerised the Windows infrastructure. Local setup-rabbitmq and setup-ibmmq actions run their Linux containers in WSL2 on Windows, replacing Azure Container Instances for RabbitMQ and enabling IBM MQ on Windows for the first time. Setup WSL is now gated to the categories that need it, and Azure login dropped to just the two categories that actually provision cloud resources.

Added a compile job building all 76 projects on both OSes. Scoped builds mean projects outside every test closure (HealthCheckApp, Particular.PlatformSample.ServiceControl, LegacyArtifacts) would otherwise stop being compiled at all.

Bugs found along the way

PostgreSQL tests had never run. The matrix used PostgreSQL/PostgreSQLPersistence; the attributes declare PostgreSql/PostgreSqlPersistence; the comparison is ordinal. 310 persistence + 23 transport tests were silently skipped, and four jobs burned ~21 minutes testing nothing. They now run.

Notifications settings could read stale after a write. NotificationsManager.LoadSettings read through AggressivelyCacheForAsync(5 minutes), which invalidates asynchronously via the Changes API, so a read straight after a save could return the pre-save document. This is user-visible: save SMTP settings, re-read, see the old values. Removed.

MultiInstance tests depended on assembly ordering. They host a primary and an audit instance but only pre-created the ServiceControl event source, relying on Audit.AcceptanceTests.RavenDB running earlier in the same job to create the audit one. Now explicit.

⚠️ Requires a branch protection change before merge

Six required status checks point at job names this PR renames, so they will never report:

Stale check	Now
Linux-Default / Windows-Default	-DefaultCore, -DefaultAudit, -DefaultMonitoring
Linux-PostgreSQL / Windows-PostgreSQL	-PostgreSql (casing)
Linux-RabbitMQ / Windows-RabbitMQ	-RabbitMQ{ClassicConventional,ClassicDirect,QuorumConventional,QuorumDirect}

Worth also requiring the new Linux-Compile / Windows-Compile jobs, since they are now the only thing compiling projects outside the test closures.

Notes for reviewers

• tools/run-tests.ps1 replaces Particular/run-tests-action, which has no way to be told which projects to run. It should fold back into that action once it grows a projects input.
• The local setup-rabbitmq / setup-ibmmq actions are deliberately self-contained so they can move to their own Particular/setup-*-action repos. They have no teardown because composite actions can't declare post: steps; hosted-runner VMs are destroyed anyway.
• 48 jobs is ~68% of the org's 60-concurrent-job Team-plan cap, and no workflow sets concurrency, so two overlapping runs will queue. Adding cancel-in-progress on PR builds would likely free more capacity than this costs.
• Remaining floor is a single 268s test assembly plus a 225s scoped build on the critical path. Going below ~8m needs NUnit-level sharding, not more CI plumbing.

[danielmarbach]

Will be ready soon. I think this is total overkill to introduce this now

[johnsimons]

this is just temp till is ready

[danielmarbach]

Have you considered building once, upload artifacts and download artifacts in the test jobs?

[danielmarbach]

Might be worth a spike: build once per OS in a dedicated job, upload the test outputs, and have each test job download its category and run. needs: would give us the no-resources-on-failed-build guarantee structurally instead of by ordering, and the runner minute count would drop well below 149m: two builds plus a download per test job instead of a scoped build per job. The catch is the size of the artifacts. The Release output adds up to roughly 10 GB per OS, with each RavenDB test project carrying its own copy of the RavenDB server for both net8.0 and net10.0, and the org's Team plan allows 2 GB of artifact storage per repository. Artifacts would only fit once the RavenDB server moves out of the test output (a shared download, tools/download-ravendb-server.ps1 already exists) and test runs go single-TFM

[johnsimons]

I am spiking it, but according the Claude, that is not going to buy much time back

[johnsimons]

#5718

[johnsimons]

So I looked at this, and the only way to make it come close to the 8 mins mark is to compile it all in Linux, but that is not possible because we need .exe.
So compiling per OS would mean everything is waiting on the Windows compiler, which can take up to 5mins!

[danielmarbach]

One thing I want to raise before this merges: backgrounding the build changes when we provision Azure resources. On master the build runs before any setup step, so a broken commit fails the job before Azure login and nothing is ever created. With background: true the foreground chain keeps running while the build compiles, and the job only fails at wait: build. The docs are explicit on this: a failed background step fails the job at the next wait that includes it,
not before. So the ASB and ASQ jobs create a Service Bus namespace and a storage account even for a build that cannot compile, then tear them down again at job end.

If the runner is killed or times out, that teardown does not happen at all. This is exactly the case we deliberately avoided. We would rather create no resources for a build that cannot run tests than create them and clean them up afterwards.

The fix is small. Move wait: build from the end of the chain to right before Azure login:

      - name: Setup WSL                                                                                                                                         
        # ...SQL, PostgreSQL, RabbitMQ, IBM MQ, azure-cli install stay here, still overlapping the build...                                                     
      - name: Wait for build                                                                                                                                    
        wait: build                                                                                                                                             
      - name: Azure login                                                                                                                                       
        if: matrix.test-category == 'AzureServiceBus' || matrix.test-category == 'AzureStorageQueues'                                                           
        # ...Setup ASB / Setup ASQ...                                                                                                                           
      - name: Run tests                                                                                                                                         

A failed build then fails the wait step and the job stops before login. The container categories keep the full overlap, since their infrastructure is local and never touches Azure. Only ASB and ASQ (4 jobs) lose it, roughly a minute each by my estimate, which puts it back around 9.5m instead of 8.6m. I think that is a fair price for keeping the invariant.

[johnsimons]

backgrounding the build changes when we provision Azure resources.

@danielmarbach this is now also fixed, and it seems we are still under 9mins