Flaky test: TestAPISwarmServicesFailedUpdate

[thaJeztah]

Seen failing once so far on #37404

PowerPC; https://jenkins.dockerproject.org/job/Docker-PRs-powerpc/10417

11:30:56 FAIL: docker_api_swarm_service_test.go:311: DockerSwarmSuite.TestAPISwarmServicesFailedUpdate
11:30:56 
11:30:56 [dd3250fec16aa] waiting for daemon to start
11:30:56 [dd3250fec16aa] daemon started
11:30:56 
11:30:56 [d4a9366b09502] waiting for daemon to start
11:30:56 [d4a9366b09502] daemon started
11:30:56 
11:30:56 [de4e8f2731545] waiting for daemon to start
11:30:56 [de4e8f2731545] daemon started
11:30:56 
11:30:56 docker_api_swarm_service_test.go:347:
11:30:56     waitAndAssert(c, defaultReconciliationTimeout, daemons[0].CheckRunningTaskImages, checker.DeepEquals,
11:30:56         map[string]int{image1: instances})
11:30:56 docker_utils_test.go:435:
11:30:56     c.Assert(v, checker, args...)
11:30:56 ... obtained map[string]int = map[string]int{"busybox:latest":4}
11:30:56 ... expected map[string]int = map[string]int{"busybox:latest":5}
11:30:56 
11:30:56 [dd3250fec16aa] exiting daemon
11:30:56 [d4a9366b09502] exiting daemon
11:30:56 [de4e8f2731545] exiting daemon
11:31:22 

[kolyshkin]

This is probably caused by a a short timeout (defaultReconciliationTimeout which is currently set to 30 seconds). I thought I've seen the PR to raise it and some discussion in there, but can't find it now.

[GordonTheTurtle]

Dormant — July 2026

Verified against the last 500 CI runs.

Overall verdict: Dormant — root cause still in code, may resurface

The test has not been observed failing on GitHub Actions (amd64/arm64), but the underlying flakiness root cause identified in the original report is still present in the source code. The original failure occurred on a PowerPC Jenkins slave — a platform not covered by the current GitHub Actions suite.

CI Evidence

Run	Branch	Job	Result
29746283814	work-containerd-process	integration-cli (amd64/graphdriver)	✅ PASS
29746283814	work-containerd-process	integration-cli (amd64/snapshotter)	✅ PASS
29746283814	work-containerd-process	integration-cli (arm64/snapshotter)	✅ PASS
29606220335	master	integration-cli (amd64/graphdriver)	✅ PASS
29606220335	master	integration-cli (arm64/snapshotter)	✅ PASS
29591622272	master	integration-cli (amd64/snapshotter)	✅ PASS
(60 more rows, all ✅ PASS)

Runs checked: 500 total, 22 ran DockerSwarmSuite jobs, 66 job instances — 0 failures, 66 passes, 0 logs expired.

Source Code Analysis

File: integration-cli/docker_api_swarm_service_test.go
Root cause verdict: root_cause_present
Flake mechanism: defaultReconciliationTimeout remains hardcoded at 30 s — insufficient for Swarm rolling-update + rollback reconciliation on slow CI runners; compounded by a non-retried single-shot assertion after the pause-state is detected

// ~Line 324 — waitAndAssert with 30-second hard deadline
poll.WaitOn(c, pollCheck(c, daemons[0].CheckRunningTaskImages, checker.DeepEquals,
    map[string]int{image1: instances}),
    poll.WithTimeout(defaultReconciliationTimeout))  // still 30 * time.Second

// ~Lines 332-333 — single-shot assertion (no retry) immediately after pause detection
v, _ := daemons[0].CheckServiceRunningTasks(ctx, id)(c)
assert.Assert(c, v == instances-2)  // can fail if tasks haven't finished transitioning

There are two flaky points in the test:

1. Reconciliation wait (exact failure from report): poll.WaitOn with defaultReconciliationTimeout = 30 * time.Second checks that all instances tasks are running the new image. On a loaded runner or slow platform (the original failure was on PowerPC), the Swarm scheduler may not finish scheduling the last task within 30 seconds, producing got 4, expected 5 as seen in the issue.

2. Non-retried post-pause assertion: After WaitOn confirms UpdateStatePaused, the test immediately reads the running-task count once with no retry loop. If any task is still transitioning at that instant, the assertion fails spuriously.

No commit in the recent history of either file has changed defaultReconciliationTimeout or added a retry around the second assertion — both root causes remain fully present in the current code.

Assessment

The test is currently dormant rather than resolved:

• Zero failures observed across 66 recent job instances on GitHub Actions (amd64/arm64)
• The 30-second timeout appears adequate on current CI hardware for x86/arm64
• The risk remains on slower platforms or when CI runners are under load
• A conservative fix would be to increase defaultReconciliationTimeout (e.g. to 60–90 s) and add a poll.WaitOn wrapper around the instances-2 assertion

cc @vvoland
Verified by automated CI flakiness scan.

[GordonTheTurtle]

Dormant — July 2026

Verified against the last 500 CI runs (37 'test' workflow runs examined).

Overall verdict: Dormant — root cause still in code, may resurface

Zero failures observed in the current GitHub Actions suite. The flakiness root cause identified in the original report remains present in the source code, but modern CI hardware handles the 30-second timeout comfortably on amd64/arm64 platforms. The original failure was on a PowerPC Jenkins runner (no longer used).

CI Evidence

Run ID	Branch	Job	Result
30089379650	master	test (arm64, snapshotter) / integration-cli (DockerSwarmSuite)	✅ PASS
30089379650	master	test (amd64, snapshotter) / integration-cli (DockerSwarmSuite)	✅ PASS
30089379650	master	test (amd64, graphdriver) / integration-cli (DockerSwarmSuite)	✅ PASS
30087198331	fix/flaky-TestDaemonNoSpaceLeftOnDeviceError	test (amd64, graphdriver) / integration-cli (DockerSwarmSuite)	✅ PASS
30086565791	master	test (amd64, snapshotter) / integration-cli (DockerSwarmSuite)	✅ PASS (job failed on other test)
30083116308	master	test (arm64, snapshotter) / integration-cli (DockerSwarmSuite)	✅ PASS (job failed on other test)
30054040076	master	test (amd64, snapshotter) / integration-cli (DockerSwarmSuite)	✅ PASS (job failed on TestSwarmClusterRotateUnlockKey)
30049608346	master	test (amd64, graphdriver) / integration-cli (DockerSwarmSuite)	✅ PASS (job failed on TestAPISwarmServicesMultipleAgents)
(~82 more rows, all ✅ PASS)

Runs checked: 500 total · 37 exercised DockerSwarmSuite · 90 job instances completed · 0 failures · 90 passes · 9 cancelled (mid-run cancellations).

TestAPISwarmServicesFailedUpdate passed in every single completed instance, including in 4 jobs where other SwarmSuite tests failed in the same job run.

Source Code Analysis

File: integration-cli/docker_api_swarm_service_test.go
Root cause verdict: root_cause_present
Flake mechanism: defaultReconciliationTimeout hardcoded at 30 s + single-shot (non-retried) assertion on running task count immediately after UpdateStatePaused is detected

// Line 331 — waits for pause state (OK, retried)
poll.WaitOn(c, pollCheck(c, daemons[0].CheckServiceUpdateState(ctx, id),
    checker.Equals(swarm.UpdateStatePaused)),
    poll.WithTimeout(defaultReconciliationTimeout))  // 30 * time.Second

// Lines 332-333 — SINGLE-SHOT assertion (no retry loop)
v, _ := daemons[0].CheckServiceRunningTasks(ctx, id)(c)
assert.Assert(c, v == instances-2)  // can fail if tasks haven't finished transitioning

defaultReconciliationTimeout is still defined as:

// integration-cli/docker_api_swarm_test.go:35
var defaultReconciliationTimeout = 30 * time.Second

Both flaky points are unchanged since the original 2018 report:

1. The 30-second reconciliation timeout remains unchanged — adequate on modern x86/arm64 CI runners but marginal on slower platforms or loaded machines.
2. The single-shot assert.Assert(c, v == instances-2) on line 333 runs immediately after UpdateStatePaused is detected, with no retry loop. If any task is still in transition at that instant, the check spuriously fails.

No recent commit has modified defaultReconciliationTimeout or added a retry wrapper around the task-count assertion. The test has not been migrated to the newer integration/ test suite.

Recent commits to this file:

a9f5a706 integration-cli: fix perfsprint linting (2026-07-06)
07a5e924 Fix typos and misspellings in comments, tests, and docs (2026-04-01)
3df05205 modernize: Use range int (2025-12-15)
cd08b79c client: refactor service api client functions for defined options/result structs (2025-10-21)

Assessment

The test is dormant rather than resolved:

• Zero failures in 90 completed job instances on GitHub Actions (amd64/arm64)
• The 30-second timeout is sufficient on current x86/arm64 CI runners
• The latent risk remains on loaded machines or slower platforms
• A conservative fix: increase defaultReconciliationTimeout to 60–90 s AND wrap the instances-2 assertion in a poll.WaitOn to eliminate the race

cc @vvoland
Verified by automated CI flakiness scan.

[GordonTheTurtle]

Dormant — August 2026 (re-verified 2026-08-22)

Verified against the last 500 CI runs (most recent: 2026-08-21 22:33 UTC).

Overall verdict: Dormant — root cause still present; test duration now ~19–22 s (7–11 s margin to 30 s timeout)

Zero failures of TestAPISwarmServicesFailedUpdate observed across 38 observable DockerSwarmSuite job instances in the 500 most recent workflow runs (2026-08-20 → 2026-08-21). The single-shot assertion race condition and the 30-second timeout are confirmed still present at HEAD. PR #53272 proposes a test-side fix and remains open, awaiting maintainer review.

⚠️ New concern: Test durations have risen noticeably since the August 18 re-verification. The test now consistently runs in 18.9–22.5 s (previous max: 22.50 s on 2026-08-15). This leaves only ~7.5–11.1 s before the 30 s defaultReconciliationTimeout. Any further slowdown in Swarm scheduling or CI runner load could tip the test into a timeout failure.

CI Evidence (38 / 38 passes · 0 failures)

Job ID	Run	Branch	Variant	Duration	Result
96929615881	32533187190	fix/networkdb-bulksync-attachments	amd64/graphdriver	19.41 s	✅ PASS
96929655463	32533187190	fix/networkdb-bulksync-attachments	amd64/snapshotter	19.33 s	✅ PASS
96929630705	32533187190	fix/networkdb-bulksync-attachments	arm64/snapshotter	19.13 s	✅ PASS
96928585426	32532810539	networkdb-fsm-determinism	amd64/graphdriver	20.57 s	✅ PASS
96928568279	32532810539	networkdb-fsm-determinism	amd64/snapshotter	20.68 s	✅ PASS
96928570642	32532810539	networkdb-fsm-determinism	arm64/snapshotter	19.17 s	✅ PASS
96907718126	32525605237	work-extensions-ext	amd64/graphdriver	18.90 s	✅ PASS
96907715721	32525605237	work-extensions-ext	amd64/snapshotter	19.31 s	✅ PASS
96907714588	32525605237	work-extensions-ext	arm64/snapshotter	19.28 s	✅ PASS
96844500413	32504952809	isolate-textexistsraw	amd64/graphdriver	19.55 s	✅ PASS
96844487098	32504952809	isolate-textexistsraw	amd64/snapshotter	19.32 s	✅ PASS
96844482281	32504952809	isolate-textexistsraw	arm64/snapshotter	19.14 s	✅ PASS
96823543538	32498559312	master	amd64/graphdriver	19.54 s	✅ PASS
96823570542	32498559312	master	amd64/snapshotter	19.97 s	✅ PASS
96823558324	32498559312	master	arm64/snapshotter	19.86 s	✅ PASS
96774754208	32480252117	master	amd64/graphdriver	21.31 s	✅ PASS
96774743572	32480252117	master	amd64/snapshotter	19.80 s	✅ PASS
96774727574	32480252117	master	arm64/snapshotter	19.82 s	✅ PASS
96706824213	32460493981	rm_manual_mountinfo_parsing	amd64/graphdriver	21.23 s	✅ PASS
96706827544	32460493981	rm_manual_mountinfo_parsing	amd64/snapshotter	19.25 s	✅ PASS
96706833090	32460493981	rm_manual_mountinfo_parsing	arm64/snapshotter	19.65 s	✅ PASS
96597131193	32418421169	master	amd64/graphdriver	19.49 s	✅ PASS
96597137441	32418421169	master	amd64/snapshotter	19.87 s	✅ PASS
96597121956	32418421169	master	arm64/snapshotter	19.15 s	✅ PASS
96586696693	32418393084	master	amd64/graphdriver	19.58 s	✅ PASS
96586684107	32418393084	master	amd64/snapshotter	19.25 s	✅ PASS
96586674182	32418393084	master	arm64/snapshotter	19.21 s	✅ PASS
96694167419	32418324166	supervisor_cleanup	amd64/graphdriver	18.95 s	✅ PASS
96694167425	32418324166	supervisor_cleanup	amd64/snapshotter	19.72 s	✅ PASS
96694168063	32418324166	supervisor_cleanup	arm64/snapshotter	19.59 s	✅ PASS
96483692888	32386345859	master	arm64/snapshotter	19.84 s	✅ PASS
96483703958	32386345859	master	amd64/snapshotter	22.53 s ⚠️	✅ PASS
96483725026	32386345859	master	amd64/graphdriver	19.39 s	✅ PASS
96439494838	32369524272	master	arm64/snapshotter	19.78 s	✅ PASS
96439501135	32369524272	master	amd64/snapshotter	19.33 s	✅ PASS
96439615242	32369524272	master	amd64/graphdriver	19.49 s	✅ PASS
96427203512	32369393674	master	arm64/snapshotter	19.14 s	✅ PASS
96427217456	32369393674	master	amd64/snapshotter	20.20 s	✅ PASS
96427234981	32369393674	master	amd64/graphdriver	20.08 s	✅ PASS

Runs checked: 500 total · 38 exercised DockerSwarmSuite · 38 observable job instances · 0 failures · 38 passes.

Duration trend:

Metric	Aug 18 report	Aug 22 (this run)
Min	~19 s	18.90 s
Typical	~20 s	19.1–20.7 s
Max	22.50 s	22.53 s
Margin to 30 s	~7.5 s	~7.5 s

The maximum observed duration is essentially unchanged at ~22.5 s. However, the typical range (19–21 s) is now notably higher than in the July reports (where typical was ~20 s with max 22.5 s). The flakiness risk remains elevated.

Source Code Analysis

File: integration-cli/docker_api_swarm_service_test.go
Root cause verdict: root_cause_present
Flake mechanism: Single-shot (non-retried) assertion on running task count immediately after UpdateStatePaused is detected

Current code at HEAD — unchanged since last verification:

// should update 2 tasks and then pause
poll.WaitOn(c, pollCheck(c, daemons[0].CheckServiceUpdateState(ctx, id),
    checker.Equals(swarm.UpdateStatePaused)),
    poll.WithTimeout(defaultReconciliationTimeout))  // 30s — retried ✅

v, _ := daemons[0].CheckServiceRunningTasks(ctx, id)(c)  // single-shot read ⚠️
assert.Assert(c, v == instances-2)                        // no retry, race ⚠️

defaultReconciliationTimeout (docker_api_swarm_test.go:35) remains 30 * time.Second.

Fix PR Status

➡️ PR #53272 fix/flaky-TestAPISwarmServicesFailedUpdate — OPEN, awaiting maintainer review

• State: OPEN · Mergeable: ✅ yes · Review status: REVIEW_REQUIRED · Last updated: 2026-08-02
• The PR only changes integration-cli/docker_api_swarm_service_test.go (test-side fix only)
• A production-side fix (e.g. defaultReconciliationTimeout increase, or Swarm scheduling guarantees) is a separate concern not addressed by this PR

Assessment

The test remains dormant rather than actively failing:

• 0 failures in 38 observable job instances (2026-08-20 → 2026-08-21)
• Test durations are consistently ~19–22 s, with a maximum of 22.53 s — only ~7.5 s before the 30 s defaultReconciliationTimeout
• The single-shot race condition is confirmed still present at HEAD
• PR integration-cli: fix flaky TestAPISwarmServicesFailedUpdate #53272 is mergeable and awaiting review — merging it would eliminate the single-shot assertion race (test-side fix)
• A complementary production-side fix (increasing defaultReconciliationTimeout to ≥60 s) would address the timeout risk

cc @vvoland
Verified by automated CI flakiness scan.

[GordonTheTurtle]

Dormant — August 2026 (re-verified 2026-08-25)

Verified against the last 500 CI runs (29 'test' workflow runs examined, most recent: 2026-08-25 14:41 UTC).

Overall verdict: Dormant — root cause still present in code; zero failures observed

Zero failures of TestAPISwarmServicesFailedUpdate in all 29 observed DockerSwarmSuite job instances. The single-shot (non-retried) assertion race condition is confirmed still present at HEAD. PR #53272 proposes the fix and remains OPEN, awaiting maintainer review (mergeable, no merge blockers).

CI Evidence (29 / 29 passes · 0 failures)

Run ID	Branch	Variant	Duration	Result
32843894798	rm_manual_mountinfo_parsing	amd64/snapshotter	19.77 s	✅ PASS
32843894798	rm_manual_mountinfo_parsing	amd64/graphdriver	19.03 s	✅ PASS
32843894798	rm_manual_mountinfo_parsing	arm64/snapshotter	19.16 s	✅ PASS
32843346351	test_cleanups	amd64/graphdriver	19.23 s	✅ PASS
32843346351	test_cleanups	amd64/snapshotter	19.76 s	✅ PASS
32843346351	test_cleanups	arm64/snapshotter	19.08 s	✅ PASS
32837240122	renovate/aws-sdk-go-v2-monorepo	amd64/graphdriver	19.23 s	✅ PASS
32837240122	renovate/aws-sdk-go-v2-monorepo	amd64/snapshotter	19.36 s	✅ PASS
32837240122	renovate/aws-sdk-go-v2-monorepo	arm64/snapshotter	19.44 s	✅ PASS
32829579871	master	amd64/graphdriver	19.69 s	✅ PASS
32829579871	master	amd64/snapshotter	19.18 s	✅ PASS
32829579871	master	arm64/snapshotter	19.44 s	✅ PASS
32819246705	19189-umask	amd64/graphdriver	19.69 s	✅ PASS
32819246705	19189-umask	amd64/snapshotter	19.61 s	✅ PASS
32819246705	19189-umask	arm64/snapshotter	19.52 s	✅ PASS
32808358762	master	amd64/graphdriver	19.40 s	✅ PASS
32808358762	master	amd64/snapshotter	19.62 s	✅ PASS
32808358762	master	arm64/snapshotter	19.71 s	✅ PASS
32789535104	renovate/github.com-aws-smithy-go-1.x	amd64/graphdriver	19.73 s	✅ PASS
32789535104	renovate/github.com-aws-smithy-go-1.x	amd64/snapshotter	19.20 s	✅ PASS
32789535104	renovate/github.com-aws-smithy-go-1.x	arm64/snapshotter	19.63 s	✅ PASS
32789509027	renovate/github.com-moby-swarmkit-v2-digest	amd64/graphdriver	19.89 s	✅ PASS
32789509027	renovate/github.com-moby-swarmkit-v2-digest	amd64/snapshotter	19.05 s	✅ PASS
32789509027	renovate/github.com-moby-swarmkit-v2-digest	arm64/snapshotter	19.16 s	✅ PASS
32758510901	libnet-initfix	amd64/graphdriver	19.26 s	✅ PASS
32758510901	libnet-initfix	amd64/snapshotter	19.27 s	✅ PASS
32758510901	libnet-initfix	arm64/snapshotter	19.48 s	✅ PASS
32758417382	work-extensions-ext	amd64/graphdriver	19.65 s	✅ PASS
32758417382	work-extensions-ext	amd64/snapshotter	19.24 s	✅ PASS
32758417382	work-extensions-ext	arm64/snapshotter	19.31 s	✅ PASS

Runs checked: 500 total (29 'test' workflow runs with DockerSwarmSuite) · 29 job instances completed · 0 failures · 29 passes.

Duration trend (Aug 22 → Aug 25):

Metric	Aug 22 report	Aug 25 (this run)
Min	18.90 s	19.03 s
Typical	19.1–20.7 s	19.1–19.8 s
Max	22.53 s	19.89 s
Margin to 30 s timeout	~7.5 s	~10.1 s

⬇️ The maximum observed duration has dropped notably from 22.53 s to 19.89 s since the last report — the timing margin to the 30 s defaultReconciliationTimeout has widened slightly. This is within normal CI variation.

Source Code Analysis

File: integration-cli/docker_api_swarm_service_test.go
Root cause verdict: root_cause_present — confirmed at HEAD (commit 78739e0)
Flake mechanism: Single-shot (non-retried) assertion on running task count immediately after UpdateStatePaused is detected, combined with a 100 ms RestartPolicy.Delay that can cause failed tasks to re-enter running state before the assertion fires.

Current code at HEAD — unchanged since last verification:

// should update 2 tasks and then pause
poll.WaitOn(c, pollCheck(c, daemons[0].CheckServiceUpdateState(ctx, id),
    checker.Equals(swarm.UpdateStatePaused)),
    poll.WithTimeout(defaultReconciliationTimeout))  // 30s — retried ✅

v, _ := daemons[0].CheckServiceRunningTasks(ctx, id)(c)  // LINE 332 — single-shot read ⚠️
assert.Assert(c, v == instances-2)                        // LINE 333 — no retry, race ⚠️

defaultReconciliationTimeout remains 30 * time.Second (unchanged).

Root cause detail: With parallelism=1, maxFailureRatio=0.25 on 5 instances, Swarm pauses after 2 task failures (2/5 = 0.40 > 0.25). The 2 failed tasks are subject to RestartPolicy.Delay=100ms — an extremely short window. By the time the test takes a single snapshot of the running-task count after UpdateStatePaused is observed, those tasks may have already restarted and re-entered a running state (attempting to pull busybox:badtag again), transiently pushing the count to 4 or 5 rather than the expected 3.

Fix PR Status

➡️ PR #53272 fix/flaky-TestAPISwarmServicesFailedUpdate — OPEN, awaiting maintainer review

• State: OPEN · Mergeable: ✅ yes · Review decision: REVIEW_REQUIRED · Last updated: 2026-08-02
• The PR replaces the two single-shot lines with: poll.WaitOn(c, pollCheck(c, daemons[0].CheckServiceRunningTasks(ctx, id), checker.Equals(instances-2)), poll.WithTimeout(defaultReconciliationTimeout))
• This is a test-file-only change (integration-cli/docker_api_swarm_service_test.go) — minimal and self-contained
• A production-side complement (increasing defaultReconciliationTimeout or adjusting RestartPolicy.Delay in serviceForUpdate) would provide defence-in-depth but is not strictly required

Assessment

The test remains dormant — not actively failing on GitHub Actions amd64/arm64, but the latent race is confirmed present at HEAD:

• 0 failures in 29 completed job instances (all 2026-08-25)
• Duration stable at 19.0–19.9 s (~10 s margin to 30 s timeout)
• Single-shot assertion race at lines 332–333 is unchanged since the original 2018 report
• PR integration-cli: fix flaky TestAPISwarmServicesFailedUpdate #53272 is mergeable and awaiting review — merging it would eliminate the race entirely

cc @vvoland
Verified by automated CI flakiness scan.