From dad69dbc65b38314d19f399ad0676b725a37bea4 Mon Sep 17 00:00:00 2001 From: Brandon McAnsh Date: Wed, 9 Sep 2026 17:18:39 -0400 Subject: [PATCH] fix(tests): reset Photos access before the previous app instance lingers `AccessKeyBackupSmokeTests.testAccessKeyBackup_saveToPhotos` fails in the `AllTargets` plan run with `Expected "Allow" Button to be hittable within 30.0s` (`BaseUITestCase.swift:101`) while passing alone. In the plan it runs right after `testAccessKeyBackup_copyToClipboard` in the same runner, so `setUp` resets Photos access while the previous test's app instance can still be running, and the reset API only promises that a running app "might" be terminated during the reset. The plan run also executes UI tests on a throwaway simulator clone, where the Photos alert took 12.5s to become hittable against about 6s on the base device. - `BaseUITestCase.setUp` terminates the app before resetting permissions. - `allowSystemAlertIfNeeded(orUntil:)` waits up to 60s for either the springboard "Allow" button or the app moving on without one, and taps Allow only when it shows. - `testAccessKeyBackup_saveToPhotos` uses it in place of the hard 30s wait. - Reflection added: the base simulator's `TCC.db` says nothing about a plan run that used a clone. --- ...26-09-09-ui-test-tcc-evidence-on-clones.md | 19 +++++++++++++ .claude/reflections/index.md | 1 + .../Smoke/AccessKeyBackupSmokeTests.swift | 12 +++++---- FlipcashUITests/Support/BaseUITestCase.swift | 27 +++++++++++++++++++ 4 files changed, 54 insertions(+), 5 deletions(-) create mode 100644 .claude/reflections/2026-09-09-ui-test-tcc-evidence-on-clones.md diff --git a/.claude/reflections/2026-09-09-ui-test-tcc-evidence-on-clones.md b/.claude/reflections/2026-09-09-ui-test-tcc-evidence-on-clones.md new file mode 100644 index 000000000..b0c5cb6c9 --- /dev/null +++ b/.claude/reflections/2026-09-09-ui-test-tcc-evidence-on-clones.md @@ -0,0 +1,19 @@ +# 2026-09-09 - TCC Evidence Read From the Wrong Simulator + +## What happened + +`AccessKeyBackupSmokeTests.testAccessKeyBackup_saveToPhotos` failed in the full `AllTargets` plan run (`Expected "Allow" Button to be hittable within 30.0s`) but passed alone. The base "iPhone 17" simulator's `TCC.db` still held `kTCCServicePhotosAdd = 2` from the earlier isolated run, unchanged by the plan run, which read as "the reset in `setUp` did not clear the add-only Photos row". Two hypotheses followed: `resetAuthorizationStatus(for: .photos)` skips `kTCCServicePhotosAdd`, or the reset is ignored while the previous test's app instance is still alive. + +## The misstep + +The row was unchanged because the plan run never touched that device. `AllTargets.xctestplan` marks `FlipcashTests` parallelizable, so `xcodebuild test -testPlan AllTargets` boots `Clone 1 of iPhone 17` and runs every target there, UI tests included (`CoreSimulator.log`: "Boot requested: Clone 1 of iPhone 17"). The clone is deleted after the run, taking its `TCC.db` and `tccd` log with it. Both hypotheses were built on evidence from a simulator the failing test never ran on. + +Measured on the base device with a `tccd` log stream and a `TCC.db` poller, the reset deleted the `kTCCServicePhotosAdd` row and the alert appeared in every configuration tried: the test alone, the whole class with the app pre-launched, and the test right after a fresh install where `tccd` logs `bundleRecordWithBundleIdentifier failed ... -10814` and still publishes the delete. The one measurable difference was time-to-alert: about 6s on the base device versus 12.5s on a clone running next to a single parallel unit-test target. + +## Resolution + +Harness-only change: `BaseUITestCase.setUp` terminates the app before resetting permissions, and `allowSystemAlertIfNeeded(orUntil:)` waits up to 60s for either the springboard "Allow" button or the app moving on without one, tapping Allow only when it shows. The exact failing run was not reproduced; the fix covers the slow-alert path that was measured and the stale-grant path that was hypothesized. + +## Lesson + +Before reading simulator state as evidence, confirm which simulator the test ran on. With parallel testing on, look for "Clone N of " in `~/Library/Logs/CoreSimulator/CoreSimulator.log` and the xcodebuild log; if the run used a clone, the base device's `TCC.db`, containers, and logs say nothing about it. Capture from inside the run instead: `xcrun simctl spawn log stream --predicate 'process == "tccd"'` on the base device for `-parallel-testing-enabled NO` runs, and `xcresulttool get test-results activities` for tap timings on any run. diff --git a/.claude/reflections/index.md b/.claude/reflections/index.md index 1d6faf19d..f336f25f0 100644 --- a/.claude/reflections/index.md +++ b/.claude/reflections/index.md @@ -8,3 +8,4 @@ A log of situations where things got out of hand. Each entry documents the issue - [2026-03-02 - SendCashOperation Verified State Regression](2026-03-02-send-cash-verified-state-regression.md): Path 2 (transfer) called `getVerifiedState()` from cache instead of reusing the state already resolved in Path 1 (message send). New currencies weren't in cache → `missingVerifiedState` → bill instantly disappeared. - [2026-05-09 - Camera Isolation Masked by @preconcurrency](2026-05-09-camera-isolation-masked-by-preconcurrency.md): Step 2 of the package restructure was tagged "Risk: low" because stripping `@MainActor` while adding `defaultIsolation` is semantically a no-op. The required tools-version bump to 6.2 enabled a runtime isolation check that caught a years-old bug `@preconcurrency import AVKit` had been hiding — instant crash on camera start. - [2026-07-11 - Regression Test at the Wrong Layer](2026-07-11-regression-test-wrong-layer.md): PR #472's regression test asserted diff shape (the fix's layer) instead of the crash path; the same UIKit crash shipped again two days later as 6a522ee. Crash repros must drive a windowed view, read live cells, and be observed failing first. +- [2026-09-09 - TCC Evidence Read From the Wrong Simulator](2026-09-09-ui-test-tcc-evidence-on-clones.md): Diagnosed a Photos-permission UI test flake from the base simulator's `TCC.db`, but the `AllTargets` plan runs UI tests on a throwaway clone; measured on the device the test actually used, the reset worked and the alert was just slower. diff --git a/FlipcashUITests/Smoke/AccessKeyBackupSmokeTests.swift b/FlipcashUITests/Smoke/AccessKeyBackupSmokeTests.swift index 9525dc1b6..422d948b2 100644 --- a/FlipcashUITests/Smoke/AccessKeyBackupSmokeTests.swift +++ b/FlipcashUITests/Smoke/AccessKeyBackupSmokeTests.swift @@ -87,16 +87,18 @@ final class AccessKeyBackupSmokeTests: BaseUITestCase { dialog.buttons["View Access Key"].tap() // Tap "Save to Photos" - waitAndTap(app.buttons["Save to Photos"]) + let saveButton = app.buttons["Save to Photos"] + waitAndTap(saveButton) - // Allow Photos access via the system permission dialog - let springboard = XCUIApplication(bundleIdentifier: "com.apple.springboard") - waitUntilHittableAndTap(springboard.buttons["Allow"]) + // Allow Photos access via the system permission dialog. The alert can + // take well over ten seconds on a freshly cloned simulator under a full + // plan run, and it never appears at all when the reset in `setUp` left + // access granted, so wait for whichever comes first. + allowSystemAlertIfNeeded { !saveButton.exists } // The button should transition to success state (checkmark). // We verify the button is no longer showing the original title, // indicating it transitioned to .success state. - let saveButton = app.buttons["Save to Photos"] let disappeared = saveButton.waitForNonExistence(timeout: 10) XCTAssertTrue( disappeared, diff --git a/FlipcashUITests/Support/BaseUITestCase.swift b/FlipcashUITests/Support/BaseUITestCase.swift index 832df7c40..cd1c89496 100644 --- a/FlipcashUITests/Support/BaseUITestCase.swift +++ b/FlipcashUITests/Support/BaseUITestCase.swift @@ -37,6 +37,13 @@ class BaseUITestCase: XCTestCase { app.launchArguments.append("--beta-flags=\(enabledBetaFlags.joined(separator: ","))") } + // The previous test's app instance can still be running here. The reset + // API only promises that a running app "might" be terminated during the + // reset, so stop it first and reset against a known-dead process. + if !resetPermissions.isEmpty { + app.terminate() + } + for permission in resetPermissions { app.resetAuthorizationStatus(for: permission) } @@ -268,6 +275,26 @@ class BaseUITestCase: XCTestCase { waitUntilHittableAndTap(springboard.buttons["Allow"]) } + /// Taps the springboard "Allow" alert if it shows up, or returns as soon as + /// `settled` reports the app moved on without one; fails when neither + /// happens within `timeout`. + func allowSystemAlertIfNeeded(timeout: TimeInterval = 60, orUntil settled: () -> Bool) { + let springboard = XCUIApplication(bundleIdentifier: "com.apple.springboard") + let allowButton = springboard.buttons["Allow"] + let deadline = Date().addingTimeInterval(timeout) + + while Date() < deadline { + if allowButton.exists, allowButton.isHittable { + allowButton.tap() + return + } + if settled() { return } + Thread.sleep(forTimeInterval: 0.25) + } + + XCTFail("Neither the system \"Allow\" alert nor the expected app state appeared within \(Int(timeout))s") + } + /// Everything legible on screen, for failure messages. func visibleText() -> String { app.staticTexts.allElementsBoundByIndex