flake: Test process crash - NATS peer refresh after shutdown #1631

CI failure

• CI run: https://github.com/coder/coder/actions/runs/30355415070
• Failing job: https://github.com/coder/coder/actions/runs/30355415070/job/90265046999 (test-go-pg-17)
• Failure date: 2026-07-28 (panic at 11:52:50 UTC; job completed at 11:56:26 UTC)
• Run attempt: 1
• Commit: coder/coder@bd5d640
• SHA: bd5d640f1e4ed8c607d6d00f7cc8392721ca8679
• Commit author: Jake Howell

The failure occurred in the same time window as the CI alert. The complete 2,037-line failing-job log was available and reviewed.

Failure evidence

The coderd/x/nats test process crashed, causing 11 tests/subtests to be reported as (unknown). These are crash fallout, not 11 independent test failures.

=== FAIL: coderd/x/nats TestPubsub_ClusterTLS/RotationOverlap (unknown)
panic: send on closed channel

goroutine 1439 [running]:
github.com/nats-io/nats-server/v2/server.(*Server).reloadAuthorization(...)
    .../github.com/nats-io/nats-server/v2@v2.14.2/server/reload.go:2298
github.com/nats-io/nats-server/v2/server.(*Server).applyOptions(...)
    .../github.com/nats-io/nats-server/v2@v2.14.2/server/reload.go:2050
github.com/nats-io/nats-server/v2/server.(*Server).ReloadOptions(...)
    .../github.com/nats-io/nats-server/v2@v2.14.2/server/reload.go:1474
github.com/coder/coder/v2/coderd/x/nats.(*Pubsub).setPeerAddresses(...)
    /home/runner/work/coder/coder/coderd/x/nats/cluster.go:131
github.com/coder/coder/v2/coderd/x/nats.(*Pubsub).runPeerRefresh(...)
    /home/runner/work/coder/coder/coderd/x/nats/cluster.go:84
created by github.com/coder/coder/v2/coderd/x/nats.New in goroutine 361
    /home/runner/work/coder/coder/coderd/x/nats/pubsub.go:420

Immediately before the panic, test cleanup logged nats server shut down. Other affected (unknown) results include TestPubsub_ClusterTLS, Mesh, SANMismatch, MixedTLSAndPlaintext, WrongCARejected, TestPubsub_ClusterTLS_CacheSwap, and TestPubsub_ClusterTLS_RealCA.

Error analysis / root cause

Classification: process crash caused by a code-level shutdown race.

New starts go p.runPeerRefresh(). Close cancels p.ctx, closes connections, and calls p.Server.Shutdown() / WaitForShutdown(), but does not join the peer-refresh goroutine or synchronize server shutdown with clusterMu.

setPeerAddresses checks p.ctx.Err() while holding clusterMu, then later calls p.Server.ReloadOptions. A refresh that passes the context check just before cancellation can therefore enter ReloadOptions while Close shuts down the NATS server. NATS reloadAuthorization then sends on a server channel that shutdown has already closed.

Likely fix direction: track/join runPeerRefresh before shutting down the embedded server, or otherwise serialize cancellation/server shutdown against the whole setPeerAddresses/ReloadOptions critical section.

No Go race-detector warning was present (WARNING: DATA RACE / race detected during execution of test). No OOM, signal: killed, allocation failure, disk exhaustion, or file-descriptor exhaustion indicator was present.

Assignment analysis

Blame/history targets:

git blame -L 120,317 coderd/x/nats/tls_internal_test.go
git blame -L 72,135 coderd/x/nats/cluster.go
git blame -L 405,423 coderd/x/nats/pubsub.go
git log --oneline -10 --follow coderd/x/nats/tls_internal_test.go
git log --oneline -20 -- coderd/x/nats/cluster.go
git log --oneline -10 --follow coderd/x/nats/pubsub.go

The failing cluster-TLS test file has one substantive commit: ad29777 (feat: NATS mTLS pubsub implementation, PR #26902), authored by cstyan. That same commit is the most recent substantive modifier of the crashing peer-refresh implementation and Pubsub lifecycle code. Assignment is therefore based on test-function and implementation ownership, not on the author of the failing CI commit.

Assignee: cstyan.

Duplicate search

Searched open and closed coder/internal issues, including the last 30 days, for:

• TestPubsub_ClusterTLS, TestPubsub_ClusterTLS/RotationOverlap, and the test family
• send on closed channel
• reloadAuthorization, ReloadOptions, and runPeerRefresh
• coderd/x/nats, tls_internal_test.go, NATS shutdown, panic, process crash, and unknown failures
• data-race and OOM signatures

No matching issue was found. Related NATS flakes are different failure modes: #1617, #1607, #1612, and #1552.

Reproduction

The race is timing-dependent. Stress the parallel cluster-TLS cleanup path:

go test ./coderd/x/nats -run 'TestPubsub_ClusterTLS($|_)' -count=100

A focused regression test should force a peer refresh to pause after the context check, invoke Close, then release the refresh and verify that shutdown waits without calling ReloadOptions on a stopped server.