Skip to content

flake: TestPubsub_Metrics #1607

Closed
Closed
flake: TestPubsub_Metrics#1607
coder/coder
#26944
 (+1)
Assignees
sreya
Labels
flake

Description

@flake-investigator
flake-investigator
bot
opened on Jun 25, 2026

CI Run Link: https://github.com/coder/coder/actions/runs/28162827287
Failing Job: https://github.com/coder/coder/actions/runs/28162827287/job/83410409640 (test-go-pg (ubuntu-latest))
Failure time: 2026-06-25T10:38:45Z (same window as Slack alert)

Commit Info:

  • SHA: 1ae96fcf8af361448b7fa40732841bda40303e08
  • Author: Danielle Maywood
  • Commit: coder/coder@1ae96fc

Failure evidence:

=== FAIL: coderd/x/nats TestPubsub_Metrics (10.05s)
    metrics_test.go:84:
        Error:       Condition never satisfied
        Test:        TestPubsub_Metrics
    t.go:111: 2026-06-25 10:36:05.425 [info]  embedded nats server started  client_url=nats://127.0.0.1:36355
    t.go:111: 2026-06-25 10:36:15.446 [info]  nats server shut down
DONE 26850 tests, 63 skipped, 1 failure in 333.357s

Error analysis:

  • Single failing test in coderd/x/nats/metrics_test.go.
  • The require.Eventually at metrics_test.go:84 timed out waiting for expected Prometheus metric values (pubsub counters/gauges/latency) after publish/subscribe activity.
  • No panic/OOM indicators observed in the job logs around the failure.

Root cause classification: Flaky test (timing-sensitive metrics collection / eventual consistency in NATS pubsub metrics).

Assignment analysis:

  • Intended blame target:
    • git blame -L 15,118 coderd/x/nats/metrics_test.go (covers TestPubsub_Metrics).
  • File history:
    • git log --oneline -10 --follow coderd/x/nats/metrics_test.go shows the test was introduced in commit 6da322d (PR #26441) by @sreya.
    • No other commits touch this file, so blame falls to the initial test introduction.

Assignee: @sreya

Related issues search (coder/internal):

  • "TestPubsub_Metrics"
  • "coderd/x/nats" "metrics_test.go"
  • "coder_nats_pubsub" / "Condition never satisfied"
    No duplicates found.

Reproduction (best effort):

go test ./coderd/x/nats -run TestPubsub_Metrics -count=1

Activity

  1. flake-investigator
    assigned
    sreya
    on Jun 25, 2026
  2. flake-investigator
    added
    flake
    on Jun 25, 2026

  3. linear-code commented on Jun 25, 2026

    @linear-code
    linear-codeBot
    on Jun 25, 2026 – with Linear Code

    ENG-2962

  4. flake-investigator commented on Jul 1, 2026

    @flake-investigator
    flake-investigatorBot
    on Jul 1, 2026
    Author

    New occurrence on 2026-07-01 in nightly gauntlet.

    CI Run: https://github.com/coder/coder/actions/runs/28496003256
    Failed Job: https://github.com/coder/coder/actions/runs/28496003256/job/84462298128 (test-go-pg (windows-2022))
    Commit: cb1a87b9c0d0d04e4e1079a2f0c1203904c4f27b (Jake Howell) coder/coder@cb1a87b
    Failure time: 2026-07-01T05:52:31Z

    Failure evidence:

    === FAIL: coderd/x/nats TestPubsub_Metrics (10.03s)
    metrics_test.go:48:
        Error:      Condition never satisfied
    t.go:111: 2026-07-01 05:46:55.182 [warn]  failed to measure latency ...
        error= failed to publish:
                   github.com/coder/coder/v2/coderd/database/pubsub.(*LatencyMeasurer).Measure
                     C:/actions-runner/coder/coder/coderd/database/pubsub/latency.go:59
                 - nats pubsub closed:
                   github.com/coder/coder/v2/coderd/x/nats.init
  5. sreya
    closed this as completedin coder#26944on Jul 1, 2026
  6. flake-investigator
    reopened this on Jul 2, 2026

  7. flake-investigator commented on Jul 2, 2026

    @flake-investigator
    flake-investigatorBot
    on Jul 2, 2026
    Author

    New occurrence:

    Failure evidence:

    === FAIL: coderd/x/nats TestPubsub_Metrics
    metrics_test.go:89:
        Error:       Condition never satisfied
        Test:        TestPubsub_Metrics
DONE 27211 tests, 61 skipped, 1 failure in 345.635s

    Notes:

    • No panic/OOM or data race indicators observed in the job logs around the failure.
    • The failure aligns with the second require.Eventually block in coderd/x/nats/metrics_test.go (after the colossal publish/subscribers path).

  8. flake-investigator commented on Jul 5, 2026

    @flake-investigator
    flake-investigatorBot
    on Jul 5, 2026
    Author

    New occurrence:

    Failure evidence:

    === FAIL: coderd/x/nats TestPubsub_Metrics (10.04s)
    metrics_test.go:89:
        Error:       Condition never satisfied
        Test:        TestPubsub_Metrics
    t.go:111: 2026-07-02 14:13:01.468 [info]  embedded nats server started  client_url=nats://127.0.0.1:35509
    t.go:111: 2026-07-02 14:13:11.481 [info]  nats server shut down

    Notes:

    • No panic/OOM/data race indicators found in the job logs around the failure.
    • This matches the prior flaky timing signature for TestPubsub_Metrics in coderd/x/nats/metrics_test.go.
  9. sreya
    closed this as completedin coder#27040on Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

flake

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions