Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions .github/actions/setup-ibmmq/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: Setup IBM MQ
description: >
Runs IBM MQ in a Linux container for the duration of the job. On Linux runners the container runs
directly through Docker. On Windows runners it runs inside the WSL2 Docker host provisioned by
Particular/setup-wsl-action, which must run first, because the IBM MQ image is Linux-only and Windows
runners can otherwise only run Windows containers.

inputs:
connection-string-name:
description: The name of the environment variable to fill with the IBM MQ connection string.
required: true
queue-manager:
description: The name of the queue manager to create.
required: false
default: QM1
admin-password:
description: The password for the admin user.
required: false
default: passw0rd
image:
description: The IBM MQ container image to run.
required: false
default: icr.io/ibm-messaging/mq:latest

runs:
using: composite
steps:
- name: Run IBM MQ container
shell: pwsh
env:
CONNECTION_STRING_NAME: ${{ inputs.connection-string-name }}
QUEUE_MANAGER: ${{ inputs.queue-manager }}
ADMIN_PASSWORD: ${{ inputs.admin-password }}
IMAGE: ${{ inputs.image }}
run: |
$ErrorActionPreference = 'Stop'

$onWindows = $Env:RUNNER_OS -eq 'Windows'

if ($onWindows) {
if (-not $Env:WSL_TOOLS_MODULE_PATH) {
throw "WSL_TOOLS_MODULE_PATH is not set. Particular/setup-wsl-action must run before this action on Windows."
}
Import-Module $Env:WSL_TOOLS_MODULE_PATH
}

function Invoke-Docker([string]$Arguments) {
if ($onWindows) { Invoke-Wsl -CheckExitCode -Command "docker $Arguments" }
else { Invoke-Expression "docker $Arguments" }
}

$containerName = 'ibmmq'

Write-Output "Starting $($Env:IMAGE) as $containerName"
Invoke-Docker ("run --name $containerName --detach --publish 1414:1414 --publish 9443:9443 " +
"--health-cmd dspmq --health-interval 10s --health-timeout 5s --health-retries 10 --health-start-period 30s " +
"-e LICENSE=accept -e MQ_QMGR_NAME=$($Env:QUEUE_MANAGER) -e MQ_ADMIN_PASSWORD=$($Env:ADMIN_PASSWORD) " +
$Env:IMAGE)

# On Windows the queue manager listens inside the WSL VM, reachable on its gateway address.
$mqHost = if ($onWindows) { $Env:WSL_IP } else { 'localhost' }

# Docker's health check runs dspmq inside the container, which reports the queue manager
# process rather than whether the listener is reachable through the WSL port forward. Probe
# the endpoint the tests actually connect to instead, the way setup-postgres-action does.
function Test-Endpoint([string]$TargetHost, [int]$Port) {
$client = [System.Net.Sockets.TcpClient]::new()
try { return $client.ConnectAsync($TargetHost, $Port).Wait(5000) -and $client.Connected }
catch { return $false }
finally { $client.Dispose() }
}

Write-Output "::group::Waiting for IBM MQ to accept connections on ${mqHost}:1414"

$deadline = (Get-Date).AddMinutes(5)
while (-not (Test-Endpoint $mqHost 1414)) {
if ((Get-Date) -gt $deadline) {
# Out-String because the WSL and native paths differ in whether they hand back a string
# or a collection of output lines.
$status = (Invoke-Docker "inspect --format '{{.State.Status}} {{.State.Health.Status}}' $containerName" | Out-String).Trim()
Write-Output "Container state: $status"
Invoke-Docker "ps --all --filter name=$containerName"
Invoke-Docker "logs $containerName"
throw "IBM MQ did not accept connections on ${mqHost}:1414 within 5 minutes."
}
Start-Sleep -Seconds 2
}

Write-Output "::endgroup::"
$connectionString = "mq://admin:$($Env:ADMIN_PASSWORD)@${mqHost}:1414/$($Env:QUEUE_MANAGER)?channel=DEV.ADMIN.SVRCONN&topicprefix=DEV"

Write-Output "IBM MQ is healthy on $mqHost"
Write-Output "$($Env:CONNECTION_STRING_NAME)=$connectionString" | Out-File -FilePath $Env:GITHUB_ENV -Encoding utf-8 -Append
164 changes: 123 additions & 41 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,87 @@ defaults:
run:
shell: pwsh
jobs:
# Compiles the whole solution once, on Linux because it is the faster runner, and publishes the
# output for every test job to reuse. Only bin/ is cached: obj/ holds absolute paths into the Linux
# NuGet cache, so each test job regenerates it with a local `dotnet restore` instead.
compile:
name: Compile
runs-on: ubuntu-latest
outputs:
cache-key: ${{ steps.key.outputs.value }}
steps:
- name: Check for secrets
env:
SECRETS_AVAILABLE: ${{ secrets.SECRETS_AVAILABLE }}
run: exit $(If ($env:SECRETS_AVAILABLE -eq 'true') { 0 } Else { 1 })
- name: Checkout
uses: actions/checkout@v7.0.1
with:
fetch-depth: 0
- name: Setup .NET SDK
uses: actions/setup-dotnet@v6.0.0
with:
global-json-file: global.json
- name: Compute cache key
id: key
run: echo "value=build-output-${{ runner.os }}-${{ github.sha }}" | Out-File -FilePath $Env:GITHUB_OUTPUT -Encoding utf-8 -Append
# ServiceControlInstaller.Packaging's zip targets fail without a self-contained server in deploy/,
# so the full-solution build needs this and every test job waits behind it. The server is pinned
# to the RavenDB.Embedded version, so cache it rather than re-downloading ~18s per run.
# Keyed on exactly what determines the download: the RavenDB.Embedded version and the OS (the
# script fetches a different archive per platform), plus the script itself so a change to how it
# extracts invalidates the entry. Hashing Directory.Packages.props instead would re-download on
# every unrelated package bump.
- name: Read RavenDB version
id: raven-version
run: |
$version = (Select-Xml -Path src/Directory.Packages.props -XPath "/Project/ItemGroup/PackageVersion[@Include='RavenDB.Embedded']/@Version").Node.Value
if (-not $version) { throw "Could not read the RavenDB.Embedded version from src/Directory.Packages.props" }
Write-Output "RavenDB.Embedded version: $version"
echo "value=$version" | Out-File -FilePath $Env:GITHUB_OUTPUT -Encoding utf-8 -Append
- name: Cache RavenDB Server
id: raven
uses: actions/cache@v6.1.0
with:
path: deploy/RavenDBServer
key: ravendb-server-${{ runner.os }}-${{ steps.raven-version.outputs.value }}-${{ hashFiles('tools/download-ravendb-server.ps1') }}
- name: Download RavenDB Server
if: steps.raven.outputs.cache-hit != 'true'
run: ./tools/download-ravendb-server.ps1
# EnableWindowsTargeting lets the net*-windows projects compile off Windows by restoring the
# targeting packs from NuGet.
- name: Build
run: dotnet build src --configuration Release -graph --property:EnableWindowsTargeting=true
# The headline number for this experiment: everything below is paid once here and then again,
# as a download, in every test job.
- name: Report build output size
run: |
# Resolve the bin directories first. `Get-ChildItem -Path src/*/bin -Recurse` does not work:
# with a wildcard in the path, PowerShell treats the leaf as a filter and looks for items
# *named* bin during recursion, which matches nothing.
$binDirs = Get-ChildItem -Path src -Directory | ForEach-Object { Join-Path $_.FullName 'bin' } | Where-Object { Test-Path $_ }
$files = Get-ChildItem -Path $binDirs -Recurse -File -ErrorAction SilentlyContinue
$gb = [math]::Round(($files | Measure-Object -Property Length -Sum).Sum / 1GB, 2)
Write-Output "Build output to cache: $gb GB uncompressed across $($files.Count) files"
- name: Save build output to cache
uses: actions/cache/save@v6.1.0
with:
path: src/*/bin
key: ${{ steps.key.outputs.value }}
# Required for the Windows test jobs to restore a cache written on Linux. Without it the
# archive is platform-specific and every Windows restore misses.
enableCrossOsArchive: true

build:
name: ${{ matrix.os-name }}-${{ matrix.test-category }}
needs: compile
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [windows-latest, ubuntu-latest]
test-category: [ Default, SqlServer, SqlServerPersistence, AzureServiceBus, RabbitMQ, AzureStorageQueues, MSMQ, SQS, PrimaryRavenAcceptance, PrimaryRavenPersistence, PostgreSQL, PostgreSQLPersistence, IBMMQ ]
# Categories are declared by the <TestCategory> property in each test project. Default and
# RabbitMQ are split because each was a single job running several slow assemblies back to back.
test-category: [ DefaultCore, DefaultAudit, DefaultMonitoring, SqlServer, SqlServerPersistence, AzureServiceBus, RabbitMQClassicConventional, RabbitMQClassicDirect, RabbitMQQuorumConventional, RabbitMQQuorumDirect, AzureStorageQueues, MSMQ, SQS, PrimaryRavenAcceptance, PrimaryRavenPersistence, PostgreSql, PostgreSqlPersistence, IBMMQ ]
include:
- os: windows-latest
os-name: Windows
Expand All @@ -27,8 +101,6 @@ jobs:
exclude:
- os: ubuntu-latest
test-category: MSMQ
- os: windows-latest
test-category: IBMMQ
fail-fast: false
steps:
- name: Check for secrets
Expand All @@ -43,30 +115,42 @@ jobs:
uses: actions/setup-dotnet@v6.0.0
with:
global-json-file: global.json
- parallel:
- name: Setup WSL
uses: Particular/setup-wsl-action@v1.1.0
- name: Download RavenDB Server
run: ./tools/download-ravendb-server.ps1
- name: Build
run: dotnet build src --configuration Release -graph
- name: Zip PowerShell module
- name: Select test projects
id: select
run: ./tools/select-test-projects.ps1 -Category ${{ matrix.test-category }}
# Both are backgrounded so they overlap the infrastructure steps below, which are an ordered
# chain (Setup WSL provisions the Docker host the database actions rely on). They are independent
# of each other: the cache supplies bin/, the restore regenerates obj/. Separate steps so the
# cache download time is attributable on its own.
- name: Download compiled output from cache
id: cache
background: true
uses: actions/cache/restore@v6.1.0
with:
path: src/*/bin
key: ${{ needs.compile.outputs.cache-key }}
fail-on-cache-miss: true
# Must match the save side: the cache is written on Linux and restored on both OSes.
enableCrossOsArchive: true
# obj/ is deliberately not cached: it embeds absolute paths into the Linux runner's NuGet cache,
# so it has to be regenerated for this OS. dotnet test --no-build then uses the cached assemblies
# without recompiling.
- name: Restore NuGet packages
id: build
background: true
run: |
New-Item assets\PowerShellModules -ItemType Directory
Compress-Archive -Path deploy\PowerShellModules\Particular.ServiceControl.Management\* -DestinationPath assets\PowerShellModules\Particular.ServiceControl.Management.zip
- name: Upload assets
uses: actions/upload-artifact@v7.0.1
if: matrix.test-category == 'Default'
with:
name: ${{ matrix.os-name }}-assets
path: |
nugets/
zip/
assets/
retention-days: 1
- name: Smoke test PowerShell module import
if: matrix.os-name == 'Windows'
run: Import-Module ./deploy/PowerShellModules/Particular.ServiceControl.Management
dotnet restore src
if ($LASTEXITCODE -ne 0) { throw "dotnet restore failed with exit code $LASTEXITCODE" }
# Provisions the WSL2 Docker host that the SQL Server, PostgreSQL, RabbitMQ and IBM MQ containers
# run in on Windows. A no-op on Linux, but still run there so those actions see the same
# environment variables on both runners.
- name: Setup WSL
if: startsWith(matrix.test-category, 'RabbitMQ') || contains(fromJSON('["SqlServer", "SqlServerPersistence", "PostgreSql", "PostgreSqlPersistence", "IBMMQ"]'), matrix.test-category)
uses: Particular/setup-wsl-action@v1.1.0
with:
# The action defaults to 4GB. The runner has 16GB and the build runs concurrently with the
# container starting up, so give the VM real headroom.
memory: 8GB

# there is an issue with az cli and python 3.14, so for now we need to pin it
# once the issue is resolved it should be able to be re-floated
Expand All @@ -84,9 +168,11 @@ jobs:
python -m pip install --user "azure-cli==2.64.0"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

# Only the Azure Service Bus and Azure Storage Queues actions provision cloud resources. RabbitMQ,
# PostgreSQL and SQL Server all run as containers now, so they no longer need an Azure session.
- name: Azure login
uses: azure/login@v3.0.1
if: matrix.test-category == 'AzureServiceBus' || matrix.test-category == 'AzureStorageQueues' || matrix.test-category == 'RabbitMQ' || matrix.test-category == 'PostgreSQL' || matrix.test-category == 'PostgreSQLPersistence'
if: matrix.test-category == 'AzureServiceBus' || matrix.test-category == 'AzureStorageQueues'
with:
creds: ${{ secrets.AZURE_ACI_CREDENTIALS }}
- name: Setup SQL Server
Expand All @@ -104,21 +190,21 @@ jobs:
enable-full-text-search: true
- name: Setup PostgreSQL
uses: Particular/setup-postgres-action@v3.0.0
if: matrix.test-category == 'PostgreSQL'
if: matrix.test-category == 'PostgreSql'
with:
connection-string-name: ServiceControl_TransportTests_PostgreSQL_ConnectionString
registry-username: ${{ secrets.DOCKERHUB_USERNAME }}
registry-password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Setup PostgreSQL persistence
uses: Particular/setup-postgres-action@v3.0.0
if: matrix.test-category == 'PostgreSQLPersistence'
if: matrix.test-category == 'PostgreSqlPersistence'
with:
connection-string-name: ServiceControl_Persistence_PostgreSql_ConnectionString
registry-username: ${{ secrets.DOCKERHUB_USERNAME }}
registry-password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Setup RabbitMQ
uses: Particular/setup-rabbitmq-action@v2.0.0
if: matrix.test-category == 'RabbitMQ'
if: startsWith(matrix.test-category, 'RabbitMQ')
with:
connection-string-name: ServiceControl_TransportTests_RabbitMQ_ConnectionString
registry-username: ${{ secrets.DOCKERHUB_USERNAME }}
Expand All @@ -138,17 +224,10 @@ jobs:
azure-credentials: ${{ secrets.AZURE_ACI_CREDENTIALS }}
tag: ServiceControl
- name: Setup IBM MQ
uses: ./.github/actions/setup-ibmmq
if: matrix.test-category == 'IBMMQ'
run: |
docker run --name ibmmq -d -p 1414:1414 -p 9443:9443 `
--health-cmd "dspmq" --health-interval 10s --health-timeout 5s --health-retries 10 --health-start-period 30s `
-e LICENSE=accept -e MQ_QMGR_NAME=QM1 -e MQ_ADMIN_PASSWORD=passw0rd `
icr.io/ibm-messaging/mq:latest
# Wait for container health check to pass
while ((docker inspect --format '{{.State.Health.Status}}' ibmmq) -ne 'healthy') {
Start-Sleep -Seconds 2
}
echo "ServiceControl_TransportTests_IBMMQ_ConnectionString=mq://admin:passw0rd@localhost:1414/QM1?channel=DEV.ADMIN.SVRCONN&topicprefix=DEV" | Out-File -FilePath $Env:GITHUB_ENV -Encoding utf-8 -Append
with:
connection-string-name: ServiceControl_TransportTests_IBMMQ_ConnectionString
- name: Setup SQS environment variables
if: matrix.test-category == 'SQS'
run: |
Expand All @@ -159,9 +238,12 @@ jobs:
# Cleanup of queues starting with `GHA-` handled by https://github.com/Particular/NServiceBus.AmazonSQS/blob/master/.github/workflows/tests-cleanup.yml
$connectString = "AccessKeyId=${{ secrets.AWS_ACCESS_KEY_ID }};SecretAccessKey=${{ secrets.AWS_SECRET_ACCESS_KEY }};Region=${{ secrets.AWS_REGION }};QueueNamePrefix=GHA-${{ github.run_id }}"
echo "ServiceControl_TransportTests_SQS_ConnectionString=$connectString" | Out-File -FilePath $Env:GITHUB_ENV -Encoding utf-8 -Append
- name: Wait for compiled output and packages
wait: [cache, build]
- name: Run tests
uses: Particular/run-tests-action@v1.7.0
run: ./tools/run-tests.ps1 -Projects $Env:TEST_PROJECTS
env:
TEST_PROJECTS: ${{ steps.select.outputs.test-projects }}
ServiceControl_TESTS_FILTER: ${{ matrix.test-category }}
PARTICULARSOFTWARE_LICENSE: ${{ secrets.LICENSETEXT }}
AZURE_ACI_CREDENTIALS: ${{ secrets.AZURE_ACI_CREDENTIALS }}
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
/binaries
/deploy
/nugets
# Generated by tools/select-test-projects.ps1
/tests.proj
build32
*.vshost.*
.nu
Expand Down
27 changes: 23 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,18 +46,37 @@ Testing using the [CI workflow](/.github/workflows/ci.yml) depends on the follow

Running all tests all the times takes a lot of resources. Tests are filtered based on the `ServiceControl_TESTS_FILTER` environment variable. To run only a subset, e.g., SQS transport tests, define the variable as `ServiceControl_TESTS_FILTER=SQS`. The following list contains all the possible `ServiceControl_TESTS_FILTER` values:

- `Default` - runs only non-transport-specific tests
Non-transport-specific:

- `DefaultCore`
- `DefaultAudit`
- `DefaultMonitoring`

Transports:

- `AzureServiceBus`
- `AzureStorageQueues`
- `IBMMQ`
- `MSMQ`
- `RabbitMQ`
- `PostgreSql`
- `RabbitMQClassicConventional`
- `RabbitMQClassicDirect`
- `RabbitMQQuorumConventional`
- `RabbitMQQuorumDirect`
- `SqlServer`
- `SqlServerPersistence`
- `PostgresSqlPersistence`
- `SQS`

Persisters:

- `PostgreSqlPersistence`
- `PrimaryRavenAcceptance`
- `PrimaryRavenPersistence`
- `SqlServerPersistence`

NOTE: If no variable is defined all tests will be executed.

Each category is declared by the `<TestCategory>` property in the test project and by the matching assembly-level `IncludeInTestCategory` attribute. CI uses the property to build and run only that category's projects; the attribute is the runtime safety net. Run `./tools/select-test-projects.ps1 -List` to see every category and the projects it selects.

## Security Configuration

Documentation for configuring security features:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

<PropertyGroup>
<TargetFramework>net10.0</TargetFramework>
<TestCategory>DefaultCore</TestCategory>
</PropertyGroup>

<ItemGroup>
Expand Down
2 changes: 1 addition & 1 deletion src/Particular.LicensingComponent.UnitTests/TestsFilter.cs
Original file line number Diff line number Diff line change
@@ -1 +1 @@
[assembly: IncludeInDefaultTests()]
[assembly: IncludeInTestCategory("DefaultCore")]
Loading
Loading