Skip to content

Keep MeetingPromptDetector tests off the real Mac's Calendar and apps - #1907

Merged
r3dbars merged 3 commits into
mainfrom
claude/meeting-prompt-detector-isolation
Sep 29, 2026
Merged

r3dbars merged 3 commits into
mainfrom
claude/meeting-prompt-detector-isolation

Conversation

@r3dbars

@r3dbars r3dbars commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Why

MeetingPromptDetectorTests fails on and off on Justin's Mac (2 to 13 failures a run, a different set each time, clean origin/main included) and never on CI. The owner's hunch was right: the suite reads the real Mac.

Every detector in the suite (47 of them) was built with the production defaults:

  • Calendar. calendarAccessGranted asks TCC for real. A binary started from this Mac's terminal inherits full Calendar access; I checked with a small probe. So every fresh detector opened its own EKEventStore and ran a live EventKit query on its first evaluation. One query took ~116 ms here, and the old fixed wait was ~100 ms. It also scores the owner's real meeting-link events next to the test's own. Tonight the calendar had none in the 12 h window, so only the timing bit. On a day with a Meet on the calendar, checks like "the candidate is mic:googleMeet with no title hint" would flip to the calendar prompt.
  • Running apps. Every evaluation did a real NSWorkspace read (runningBundleIDsProvider was never stubbed).
  • Browser titles. Tests that didn't call showBrowserTab read real browser window titles through Accessibility.
  • Shortcut. The prompt detail read the saved meeting shortcut from UserDefaults.

CI has no Calendar access, so its evaluations never waited on EventKit. That's why only the owner's Mac saw it.

What changed

  • Tests: every detector now comes from makeIsolatedDetector(...). It takes the same parameters as the init, but the defaults are fakes: no Calendar access, an empty injected fetch, no running or frontmost apps, no browser windows, and a fixed ⌥M shortcut. Tests still override whatever they need. The 47 now-redundant frontmostBundleIDProvider = { nil } lines are gone.
  • MeetingPromptDetector.init: it only builds the real MeetingPromptCalendarReader (and its EKEventStore) when no fetch is injected, so a test detector never touches EventKit. Production is unchanged: TranscriptedApp still uses the default, which still builds the reader in init.
  • Sources/Meeting/CLAUDE.md: one agent note saying the init defaults read the real Mac, and new tests should use the factory.

Relationship to #1906

This branch is stacked on #1906 (claude/deflake-meeting-prompt-detector), so against main it also contains #1906's two commits. The two fixes are complementary:

Merge #1906 first and this shrinks to one commit, or merge this one and it brings #1906 along. Whichever you prefer.

Checks

  • check-source-pins.py --changed-only: pass. check-test-shape.py: pass (nothing new). check-telemetry-keys.py: pass.
  • bash run-tests.sh --filter MeetingPromptDetectorTests.swift, 5 back-to-back runs on this branch at load average 265–300 (Deflake MeetingPromptDetector tests with an explicit settle signal #1906's yes load repro was running on the same Mac): 5/5 green, 138/138 each. This terminal has full Calendar access, so this is the setup that used to fail.
  • bash check.sh (11 steps): everything passed except one fast test out of 19,961, TranscriptedConstantsTests.swift:225 ("detached timeout should not wait for non-cooperative model work to unwind"). This diff doesn't touch that file. It's a separate load flake: rerun alone 3 times at load ~100–200 it failed 1 and passed 2. It's flagged as its own fix-or-bench task, not folded in here.
  • Review: Codex couldn't run: its configured model gpt-6-sol is rejected for this ChatGPT account. I ran /code-review medium on this commit against Deflake MeetingPromptDetector tests with an explicit settle signal #1906's branch, with no findings. That ran in the same session that wrote the change, so it isn't independent. This still needs an independent pass.
  • Not covered: start() still installs real NSWorkspace activation observers. Only the two calendar-refresh tests and one calendar-prompt test call start(). Activating Zoom/Teams/Webex/FaceTime during those few seconds could add a runtime signal, but none of their asserts depend on it. Closing that needs an injectable notification center; I left it out.

🤖 Generated with Claude Code

Signal pushes (mic, camera, output, requestEvaluation, EventKit change,
workspace activate) each spawn an evaluate() task that awaits an
off-main running-apps read. The tests waited a fixed ~100 ms for it, so
on a loaded Mac they read the result before evaluation finished and got
nil / 0 prompts.

The detector now counts evaluations in flight (spawned passes, the first
poll pass, title reads that re-evaluate, and timed re-checks once they
fire) and exposes waitUntilEvaluationsSettle(). The tests await that
instead of sleeping. The one test that waits on a 1 s timed re-check
waits for its condition, then settles.

Also: the DefaultInputDeviceNotificationLookupDispatcher ordering test
used a 250 ms lookup timeout around a 50 ms fake lookup. Under load it
timed out, delivered a nil device, and failed the self-write flag. The
timeout isn't what that test checks, so it gets a 30 s bound.
The first loaded run (load avg ~280) failed "an unrecognized site waits
before prompting" at the "not right away" check: with a 1 s wait counted
from the mic push, a first evaluation pass slower than 1 s was already
allowed to prompt. The hold-back check now uses an hour-long wait; a
separate suite keeps the 1 s wait and checks that the detector's own
re-check prompts.
Every detector the suite built used the production defaults, so on a Mac
whose terminal has Calendar access each fresh detector opened its own
EKEventStore, ran a live EventKit query on its first pass (about 120 ms
here) and scored the owner's real meeting-link events next to the test's
own. It also read the real running apps, and a few browser-mic tests read
real browser window titles through Accessibility. CI has no Calendar
access, which is why only the owner's Mac saw it.

All 47 detectors now come from makeIsolatedDetector, which fakes Calendar
access, the EventKit fetch, running and frontmost apps, browser titles and
the shortcut display. The detector only opens an EKEventStore when no
fetch is injected, so a test detector never touches EventKit. Production
still builds the real reader in init, as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant