Repository navigation
Keep MeetingPromptDetector tests off the real Mac's Calendar and apps - #1907
Merged
Merged
Conversation
Signal pushes (mic, camera, output, requestEvaluation, EventKit change, workspace activate) each spawn an evaluate() task that awaits an off-main running-apps read. The tests waited a fixed ~100 ms for it, so on a loaded Mac they read the result before evaluation finished and got nil / 0 prompts. The detector now counts evaluations in flight (spawned passes, the first poll pass, title reads that re-evaluate, and timed re-checks once they fire) and exposes waitUntilEvaluationsSettle(). The tests await that instead of sleeping. The one test that waits on a 1 s timed re-check waits for its condition, then settles. Also: the DefaultInputDeviceNotificationLookupDispatcher ordering test used a 250 ms lookup timeout around a 50 ms fake lookup. Under load it timed out, delivered a nil device, and failed the self-write flag. The timeout isn't what that test checks, so it gets a 30 s bound.
The first loaded run (load avg ~280) failed "an unrecognized site waits before prompting" at the "not right away" check: with a 1 s wait counted from the mic push, a first evaluation pass slower than 1 s was already allowed to prompt. The hold-back check now uses an hour-long wait; a separate suite keeps the 1 s wait and checks that the detector's own re-check prompts.
Every detector the suite built used the production defaults, so on a Mac whose terminal has Calendar access each fresh detector opened its own EKEventStore, ran a live EventKit query on its first pass (about 120 ms here) and scored the owner's real meeting-link events next to the test's own. It also read the real running apps, and a few browser-mic tests read real browser window titles through Accessibility. CI has no Calendar access, which is why only the owner's Mac saw it. All 47 detectors now come from makeIsolatedDetector, which fakes Calendar access, the EventKit fetch, running and frontmost apps, browser titles and the shortcut display. The detector only opens an EKEventStore when no fetch is injected, so a test detector never touches EventKit. Production still builds the real reader in init, as before.
This was referenced Sep 29, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
MeetingPromptDetectorTestsfails on and off on Justin's Mac (2 to 13 failures a run, a different set each time, cleanorigin/mainincluded) and never on CI. The owner's hunch was right: the suite reads the real Mac.Every detector in the suite (47 of them) was built with the production defaults:
calendarAccessGrantedasks TCC for real. A binary started from this Mac's terminal inherits full Calendar access; I checked with a small probe. So every fresh detector opened its ownEKEventStoreand ran a live EventKit query on its first evaluation. One query took ~116 ms here, and the old fixed wait was ~100 ms. It also scores the owner's real meeting-link events next to the test's own. Tonight the calendar had none in the 12 h window, so only the timing bit. On a day with a Meet on the calendar, checks like "the candidate ismic:googleMeetwith no title hint" would flip to the calendar prompt.NSWorkspaceread (runningBundleIDsProviderwas never stubbed).showBrowserTabread real browser window titles through Accessibility.UserDefaults.CI has no Calendar access, so its evaluations never waited on EventKit. That's why only the owner's Mac saw it.
What changed
makeIsolatedDetector(...). It takes the same parameters as the init, but the defaults are fakes: no Calendar access, an empty injected fetch, no running or frontmost apps, no browser windows, and a fixed⌥Mshortcut. Tests still override whatever they need. The 47 now-redundantfrontmostBundleIDProvider = { nil }lines are gone.MeetingPromptDetector.init: it only builds the realMeetingPromptCalendarReader(and itsEKEventStore) when no fetch is injected, so a test detector never touches EventKit. Production is unchanged:TranscriptedAppstill uses the default, which still builds the reader in init.Sources/Meeting/CLAUDE.md: one agent note saying the init defaults read the real Mac, and new tests should use the factory.Relationship to #1906
This branch is stacked on #1906 (
claude/deflake-meeting-prompt-detector), so againstmainit also contains #1906's two commits. The two fixes are complementary:waitForPromptEvaluation(detector).Merge #1906 first and this shrinks to one commit, or merge this one and it brings #1906 along. Whichever you prefer.
Checks
check-source-pins.py --changed-only: pass.check-test-shape.py: pass (nothing new).check-telemetry-keys.py: pass.bash run-tests.sh --filter MeetingPromptDetectorTests.swift, 5 back-to-back runs on this branch at load average 265–300 (Deflake MeetingPromptDetector tests with an explicit settle signal #1906'syesload repro was running on the same Mac): 5/5 green, 138/138 each. This terminal has full Calendar access, so this is the setup that used to fail.bash check.sh(11 steps): everything passed except one fast test out of 19,961,TranscriptedConstantsTests.swift:225("detached timeout should not wait for non-cooperative model work to unwind"). This diff doesn't touch that file. It's a separate load flake: rerun alone 3 times at load ~100–200 it failed 1 and passed 2. It's flagged as its own fix-or-bench task, not folded in here.gpt-6-solis rejected for this ChatGPT account. I ran/code-reviewmedium on this commit against Deflake MeetingPromptDetector tests with an explicit settle signal #1906's branch, with no findings. That ran in the same session that wrote the change, so it isn't independent. This still needs an independent pass.start()still installs realNSWorkspaceactivation observers. Only the two calendar-refresh tests and one calendar-prompt test callstart(). Activating Zoom/Teams/Webex/FaceTime during those few seconds could add a runtime signal, but none of their asserts depend on it. Closing that needs an injectable notification center; I left it out.🤖 Generated with Claude Code