Repository navigation
feat(desktop): Computer Use for macOS, Windows, and Linux - #6537
sheehanmunim wants to merge 90 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
One finding: the new desktop-control resolver hides its ServerSettingsService dependency behind Effect.serviceOption, so the requirement is absent from the type of every adapter that calls it and a missing layer silently resolves to "Computer Use enabled". Everything else in the changed TypeScript (subpath namespace imports for effect/*, Effect.fn helpers acquiring FileSystem/Path/HostProcess* from the environment, new Schema.is predicates that add real logic, static CodexAppServerRequestError.methodNotFound factory, no new ManagedRuntime/runPromise/fake Layer.succeed) matches the conventions.
Posted via Macroscope — Effect Service Conventions
There was a problem hiding this comment.
Pull request overview
Adds cross-platform “Computer Use” to T3 Code Desktop by bundling a local desktop-control MCP server (Swift on macOS, Rust on Windows/Linux) and an agent-owned Chrome tab workflow via a bundled extension, plus end-to-end settings/approvals plumbing across contracts, server adapters, desktop IPC, and the web UI.
Changes:
- Bundle and resolve a
t3-desktopMCP server across macOS/Windows/Linux, staging it into desktop build artifacts and injecting it into supported provider adapters when enabled. - Add Chrome extension bridge + install scripts, plus a new Settings → Computer Use page that surfaces enablement, extension status, and macOS privacy permission readiness.
- Extend contracts and runtime ingestion to support new approval kinds (
tool/permissions) and Codex MCP form-elicitation flows, with accompanying test coverage.
Reviewed changes
Copilot reviewed 60 out of 75 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| vite.config.ts | Adjust staged formatter globs to avoid native/ files breaking commits |
| scripts/build-desktop-artifact.ts | Stage desktop MCP binaries (Swift/Rust) and macOS Info.plist Apple Events usage text |
| scripts/build-desktop-artifact.test.ts | Add tests for MCP executable naming/targets and macOS usage description |
| packages/effect-codex-app-server/src/errors.ts | Extend identifier purpose literals for new approval request types |
| packages/contracts/src/settings.ts | Add desktopControl settings subtree + patch schema |
| packages/contracts/src/providerRuntime.ts | Add canonical request types for tool/permissions approvals |
| packages/contracts/src/orchestration.ts | Extend ProviderRequestKind to include tool/permissions |
| packages/contracts/src/ipc.ts | Add IPC schemas + DesktopBridge methods for Computer Use readiness |
| package.json | Add build script for Swift desktop MCP server |
| native/t3-desktop-mcp/Sources/BrowserBridge.swift | Swift Chrome-extension bridge via unix socket + native messaging framing |
| native/t3-desktop-mcp/Package.swift | SwiftPM package definition for macOS desktop MCP server |
| native/t3-desktop-mcp/.gitignore | Ignore SwiftPM build output |
| native/t3-desktop-mcp-rs/src/tools.rs | Rust tool schema definitions matching macOS tool surface (with browser_* gating) |
| native/t3-desktop-mcp-rs/src/platform/mod.rs | Cross-platform desktop backend trait + shared formatting/types |
| native/t3-desktop-mcp-rs/src/main.rs | Rust MCP server: newline-delimited JSON-RPC over stdio + tool dispatch |
| native/t3-desktop-mcp-rs/src/capture.rs | Shared display/window capture with Wayland/X11 fallbacks and PNG encoding |
| native/t3-desktop-mcp-rs/src/browser.rs | Rust Chrome bridge + native-host relay over local socket |
| native/t3-desktop-mcp-rs/src/apps.rs | App/window enumeration and pid resolution shared across backends |
| native/t3-desktop-mcp-rs/Cargo.toml | New Rust crate definition + deps for Windows/Linux automation/capture |
| native/t3-desktop-mcp-rs/.gitignore | Ignore Rust target output |
| native/t3-chrome-extension/manifest.json | MV3 extension manifest (native messaging + debugger/tabGroups perms) |
| native/t3-chrome-extension/install.sh | macOS/Linux host registration script for native messaging |
| native/t3-chrome-extension/install.ps1 | Windows registry-based host registration script |
| native/t3-chrome-extension/background.js | Chrome service worker implementing agent-owned tab group + DevTools-based automation |
| apps/web/src/session-logic.ts | Teach pending approval derivation about tool/permissions kinds |
| apps/web/src/session-logic.test.ts | Tests for deriving/retaining new approval kinds in UI work log |
| apps/web/src/routeTree.gen.ts | Add generated route for Settings → Computer Use |
| apps/web/src/routes/settings.computer-use.tsx | Route entry for the Computer Use settings page |
| apps/web/src/components/settings/SettingsSidebarNav.tsx | Add sidebar icon mapping for Computer Use section |
| apps/web/src/components/settings/settingsSearch.ts | Add search items/labels for Computer Use settings and permissions |
| apps/web/src/components/settings/ComputerUseSettings.tsx | New settings UI: toggles, extension status, macOS permission readiness + deep links |
| apps/web/src/components/settings/browserBrandIcons.tsx | Brand icons for browser/agent cursor display in settings |
| apps/web/src/components/chat/ComposerPendingApprovalPanel.tsx | UI labels for tool vs permission pending approvals |
| apps/web/src/components/chat/ComposerPendingApprovalPanel.test.tsx | Tests for new pending-approval labels |
| apps/web/src/assets/computer-use/google-chrome.svg | Add Chrome brand asset |
| apps/web/src/assets/computer-use/firefox.svg | Add Firefox brand asset (not supported yet) |
| apps/web/src/assets/computer-use/edge.svg | Add Edge brand asset |
| apps/web/src/assets/computer-use/brave.svg | Add Brave brand asset |
| apps/server/src/provider/Layers/GrokAdapter.ts | Inject resolved desktop MCP server into Grok adapter MCP server list |
| apps/server/src/provider/Layers/CursorAdapter.ts | Inject resolved desktop MCP server into Cursor adapter MCP server list |
| apps/server/src/provider/Layers/CodexSessionRuntime.ts | Add permissions approval handling + MCP elicitation (tool/permissions) request plumbing |
| apps/server/src/provider/Layers/CodexSessionRuntime.test.ts | Unit tests for permission approvals + MCP elicitation mapping/response building |
| apps/server/src/provider/Layers/CodexProvider.ts | Advertise Codex MCP form-elicitation capability |
| apps/server/src/provider/Layers/CodexProvider.test.ts | Test for Codex initialize capability flags |
| apps/server/src/provider/Layers/CodexAdapter.ts | Map new request kinds/types and include elicitation detail extraction |
| apps/server/src/provider/Layers/CodexAdapter.test.ts | Lifecycle tests for tool/permissions canonical approval mapping |
| apps/server/src/provider/Layers/ClaudeAdapter.ts | Add desktop MCP server into Claude mcpServers map when resolvable |
| apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts | Preserve new approval kinds through activity summaries/payload |
| apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.approval.test.ts | Tests ensuring new approval kinds reach clients correctly |
| apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts | Resolve desktop MCP only when enabled; pass env flags for cursor/browser toggles |
| apps/server/src/desktopControl/resolveEnabledDesktopMcp.test.ts | Tests for enable/disable and env flag behavior |
| apps/server/src/desktopControl/desktopMcpBinary.ts | Locate bundled/override MCP binary across platforms and dev layouts |
| apps/server/src/desktopControl/desktopMcpBinary.test.ts | Tests for override resolution and unsupported platform behavior |
| apps/desktop/src/preload.ts | Expose Computer Use IPC methods to renderer via preload bridge |
| apps/desktop/src/ipc/methods/computerUse.ts | IPC handlers for permissions readiness + opening privacy panes |
| apps/desktop/src/ipc/DesktopIpcHandlers.ts | Register new Computer Use IPC handlers |
| apps/desktop/src/ipc/channels.ts | Add IPC channel constants for Computer Use |
| apps/desktop/src/computerUse/permissions.ts | Read macOS TCC statuses + detect Chrome extension/host registration |
| apps/desktop/src/computerUse/permissions.test.ts | Tests for permission status mapping, Settings deep links, and extension detection |
| apps/desktop/scripts/electron-launcher.mjs | Patch macOS Info.plist to include Apple Events usage description |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
ApprovabilityVerdict: Needs human review 13 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR. You can customize Macroscope's approvability policy. Learn more. |
Babysit updateBranch is merge-clean onto current Review fixes pushed (settings fail-closed + per-session resolve without widening session Note for reviewers / Macroscope: the Effect Service Conventions check keeps reporting the old Fork CI workflows may still need org approval; Vercel marketing fails with “Authorization required to deploy” (expected for fork PRs). |
|
Follow-up correctness gaps from review are now fixed on tip:
Windows/Linux desktop MCP packaging was already wired through `stageDesktopMcpRust` in desktop artifact builds — once this lands in Nightly, hand-copying the binary should no longer be needed. |
Note on Macroscope Effect Service ConventionsThis check is still failing with a retained finding against the deleted path `apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts` (`Effect.serviceOption` / `onNone` defaults). That file was renamed to `desktopMcpLaunch.ts`, and the current implementation acquires settings with `yield* ServerSettings.ServerSettingsService` (fail-closed on `getSettings` errors). There is no `serviceOption` / `onNone` path left in the PR. Correctness Check is green; unresolved review threads are at 0. Remaining merge blockers appear to be human review + fork Vercel auth, not code correctness. |
|
Addressed remaining bot findings on
Vercel marketing deploy auth failure is outside this PR’s scope. |
Keep History-only CLI wiring off the mac/computer-use branch. Co-authored-by: Cursor <cursoragent@cursor.com>
…ttings error handling Only unlink bridge sockets this listener owns, stop advertising grim-only displays without capture support, and catch rejected privacy-settings open promises. Co-authored-by: Cursor <cursoragent@cursor.com>
… branch Apply Macroscope fixes for agent cursor lifecycle, browser bridge writes, AX window pairing, app resolution, JSON-RPC parse errors, and Linux cursor init without History-only daemon changes. Co-authored-by: Cursor <cursoragent@cursor.com>
Prevent whitespace-only app arguments from matching every running app via substring fallback. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Also strip accidental computer-history CLI from the Computer Use branch. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep Computer Use History-free and avoid short PID queries binding unrelated apps via substring name matches. Co-authored-by: Cursor <cursoragent@cursor.com>
A leftover closing brace from stripping the History CLI broke the top-level Swift script structure after native-host. Co-authored-by: Cursor <cursoragent@cursor.com>
Nonblocking bridge writes, owner-only agent-cursor sockets, no WAYLAND_DISPLAY mutation, Wayland type_text/X11 cookie checks, create-first bridge binding, and inert browser when control is disabled. Co-authored-by: Cursor <cursoragent@cursor.com>
…MOST Avoid O_NONBLOCK on the shared socket, revalidate the fd under writeLock, propagate Shift release failures, and stop fabricating FRONTMOST markers. Co-authored-by: Cursor <cursoragent@cursor.com>
Pick the advertised press/click/activate action instead of hard-coding index 0. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the blueprint extension icons with the production black T3 mark, and shrink the agent pointer favicons so Chrome's circular tab crop no longer clips the cursor. Co-authored-by: Cursor <cursoragent@cursor.com>
Port shared Macroscope/Bugbot fixes: safer bridge socket ownership, Int-safe coords, AppleScript escaping, scroll/click targeting, bridge write reset, and AT-SPI reconnect. Co-authored-by: Cursor <cursoragent@cursor.com>
Use a transparent circular badge (no black square crop) and rounded-full clipping so the Computer Use settings row matches the pointer favicon. Co-authored-by: Cursor <cursoragent@cursor.com>
Point the Computer Use settings icon at the same soft-glow cursor-112 artwork as the desktop overlay, on a transparent canvas with no rounded crop. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace removed FiberRef usage so Dev can boot, and pad the settings/overlay cursor art so the lavender glow does not square-crop at size-6. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the original pointer art and fade the glow into the badge rim so Settings and Chrome tab favicons no longer show a nested square crop. Co-authored-by: Cursor <cursoragent@cursor.com>
Reject overlapping-window element drags and refuse symlink bridge directories so background drag and Unix socket setup fail closed. Co-authored-by: Cursor <cursoragent@cursor.com>
efc8af8 to
03b1beb
Compare
| if (typeof entry.state === "number") return entry.state === 1; | ||
| if (typeof entry.path === "string" && entry.path.length > 0) return true; | ||
| if (entry.location === 4) return true; | ||
| return true; |
There was a problem hiding this comment.
Extension status false positives
Medium Severity
After handling explicit state, path, and unpacked location === 4, chromeExtensionInstalledInPreferences still falls through to return true for any leftover preference entry. Ghost or incomplete extensions.settings records then report as installed in Settings even when the extension is not actually loadable.
Reviewed by Cursor Bugbot for commit 03b1beb. Configure here.
| ]; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Edge and Brave detection missing
Medium Severity
Computer Use Settings documents Edge and Brave setup, but chromeProfileRoots only scans Chrome/Chromium paths, and the install scripts only register the native host for Chrome/Chromium. An Edge- or Brave-only install never shows as installed and never gets a working native-messaging host registration.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 03b1beb. Configure here.
| // Without this key macOS denies every Apple Event with errAEEventNotPermitted | ||
| // (-1743) and never shows the Automation prompt, so Codex Computer Use and any | ||
| // other MCP server we spawn silently fail to drive other apps. | ||
| setPlistString( |
There was a problem hiding this comment.
🟡 Medium scripts/electron-launcher.mjs:234
Existing .electron-runtime bundles continue to lack NSAppleEventsUsageDescription, so Computer Use Apple Events remain denied for developers with a cached launcher. buildMacLauncher returns the cached bundle before calling patchMainBundleInfoPlist; bump LAUNCHER_VERSION from 15 to invalidate those bundles and apply the new key.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/desktop/scripts/electron-launcher.mjs around line 234:
Existing `.electron-runtime` bundles continue to lack `NSAppleEventsUsageDescription`, so Computer Use Apple Events remain denied for developers with a cached launcher. `buildMacLauncher` returns the cached bundle before calling `patchMainBundleInfoPlist`; bump `LAUNCHER_VERSION` from 15 to invalidate those bundles and apply the new key.
|
|
||
| fn compositing_manager_running(conn: &RustConnection, screen_num: usize) -> bool { | ||
| let name = format!("_NET_WM_CM_S{screen_num}"); | ||
| let Ok(atom) = conn.intern_atom(false, name.as_bytes()) else { |
There was a problem hiding this comment.
🟡 Medium platform/agent_cursor_linux.rs:801
present performs two synchronous X11 request/reply round trips on every rendered frame, so a visible cursor repeatedly blocks the UI thread and can reduce animation to a few FPS on high-latency or remote displays. Cache the _NET_WM_CM_S{screen_num} atom and query compositor ownership outside the render loop.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/agent_cursor_linux.rs around line 801:
`present` performs two synchronous X11 request/reply round trips on every rendered frame, so a visible cursor repeatedly blocks the UI thread and can reduce animation to a few FPS on high-latency or remote displays. Cache the `_NET_WM_CM_S{screen_num}` atom and query compositor ownership outside the render loop.
| // become `connection`, blocking the real overlay. | ||
| _ = chmod(path, 0o600) | ||
|
|
||
| let source = DispatchSource.makeReadSource(fileDescriptor: fd, queue: .main) |
There was a problem hiding this comment.
🟡 Medium Sources/AgentCursor.swift:356
The cursor listener never accepts the overlay connection during normal MCP operation, so queued cursor messages are never flushed and the agent cursor does not appear. startListening schedules acceptConnection() on .main, but the executable keeps that queue blocked in the synchronous runJSONRPCLoop/readLine loop; schedule the source on a dedicated serial queue instead.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp/Sources/AgentCursor.swift around line 356:
The cursor listener never accepts the overlay connection during normal MCP operation, so queued cursor messages are never flushed and the agent cursor does not appear. `startListening` schedules `acceptConnection()` on `.main`, but the executable keeps that queue blocked in the synchronous `runJSONRPCLoop`/`readLine` loop; schedule the source on a dedicated serial queue instead.
| // One shared element budget across every window — recreating the walk | ||
| // buffer per window would let multi-window apps emit | ||
| // windows.len() * max_elements rows. | ||
| let mut element_lines = Vec::new(); |
There was a problem hiding this comment.
🟡 Medium platform/windows.rs:421
When element_from_handle fails for every visible window, get_app_state still returns Ok with no window contents and the misleading no interactive elements found message. Because each conversion error is skipped with continue, callers treat an inaccessible snapshot as a valid empty snapshot; track successful conversions and return an error when none succeed.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/windows.rs around line 421:
When `element_from_handle` fails for every visible window, `get_app_state` still returns `Ok` with no window contents and the misleading `no interactive elements found` message. Because each conversion error is skipped with `continue`, callers treat an inaccessible snapshot as a valid empty snapshot; track successful conversions and return an error when none succeed.
| "description": "Read an app's accessibility tree as an indented outline. Interactive elements are prefixed with an id like [e12] that you pass to click/type_text/scroll. Call this before interacting, and again after the UI changes, since ids are per-snapshot.", | ||
| "inputSchema": { | ||
| "type": "object", | ||
| "properties": { |
There was a problem hiding this comment.
🟡 Medium src/tools.rs:57
get_app_state on Windows/Linux cannot accept the advertised window argument, so requests for a specific window are rejected or read with the wrong scope in multi-window apps. Add the same window input property exposed by the macOS toolDefs, and extend advertises_the_macos_tool_surface to validate input schemas rather than only tool names.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/tools.rs around line 57:
`get_app_state` on Windows/Linux cannot accept the advertised `window` argument, so requests for a specific window are rejected or read with the wrong scope in multi-window apps. Add the same `window` input property exposed by the macOS `toolDefs`, and extend `advertises_the_macos_tool_surface` to validate input schemas rather than only tool names.
| .or_else(|_| std::env::var("USER")) | ||
| .unwrap_or_else(|_| "user".into()); | ||
| // Named-pipe namespace is global; embed the username so sessions do not collide. | ||
| format!("t3-desktop-mcp-bridge-{user}") |
There was a problem hiding this comment.
🟠 High src/browser.rs:127
bridge_pipe_name produces the same pipe name for multiple Windows login/RDP sessions using the same USERNAME, so a host in a second session connects to the first session's MCP listener and routes browser control and replies to the wrong desktop. Include session- and account-unique identifiers, such as the session ID and SID, in the pipe name.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/browser.rs around line 127:
`bridge_pipe_name` produces the same pipe name for multiple Windows login/RDP sessions using the same `USERNAME`, so a host in a second session connects to the first session's MCP listener and routes browser control and replies to the wrong desktop. Include session- and account-unique identifiers, such as the session ID and SID, in the pipe name.
| let mut next_id = 0u32; | ||
| let mut visited = 0usize; | ||
| let mut lines = vec![format!("{name}")]; | ||
| self.walk( |
There was a problem hiding this comment.
🟡 Medium platform/linux.rs:934
get_app_state returns Ok with only the app name and “no interactive elements exposed” when the root accessibility proxy or its AT-SPI calls fail after applications() succeeds. Because walk returns () and silently stops on those errors, callers cannot distinguish an unavailable application from a valid empty snapshot; propagate the traversal error instead.
🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/linux.rs around line 934:
`get_app_state` returns `Ok` with only the app name and “no interactive elements exposed” when the root accessibility proxy or its AT-SPI calls fail after `applications()` succeeds. Because `walk` returns `()` and silently stops on those errors, callers cannot distinguish an unavailable application from a valid empty snapshot; propagate the traversal error instead.
Map press_key to press, clear stale Chrome window cache, escape app-list brackets, and fix Wayland/X11 click and type edge cases flagged after rebase. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 20 total unresolved issues (including 18 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 92d6674. Configure here.
| const SS_NOTIFY: i32 = 0x0001_0000; | ||
| let style = unsafe { GetWindowLongW(hwnd, GWL_STYLE) }; | ||
| return style & SS_NOTIFY != 0; | ||
| } |
There was a problem hiding this comment.
Wrong SS_NOTIFY style constant
High Severity
The Static background-click gate uses 0x0001_0000 as SS_NOTIFY, but the real Win32 value is 0x00000100. That bit is WS_TABSTOP, so clickable notify statics are often rejected while tab-stop labels can be accepted. Posted mouse messages then either never run or report success when the control ignores them.
Reviewed by Cursor Bugbot for commit 92d6674. Configure here.
| } else { | ||
| 0x0100_0000 + code | ||
| } | ||
| } |
There was a problem hiding this comment.
Newline maps to Linefeed keysym
Medium Severity
char_to_keysym maps '\n' to 0xff0a (XK_Linefeed) while press_key / enter uses 0xff0d (XK_Return). type_text with Unix newlines therefore synthesizes Linefeed instead of Return, so many GTK/Qt/terminal fields never insert a line break.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 92d6674. Configure here.
Only fall back from AT-SPI to X11 when there is no match, not when the query matches several apps. Co-authored-by: Cursor <cursoragent@cursor.com>


Summary
t3-desktopMCP (Swift on macOS, Rust on Windows/Linux), including agent cursor overlay and background clicks that do not steal the user mouse.origin/main(ahead of current nightlyv0.0.34-nightly.20260813.1087): contracts/desktop/web typecheck clean for this work; Windows Blade E2E confirmedlist_apps, backgroundclick … in background, andscreenshot.Demo
computer-use-demo.mp4
Settings → Computer Use:
Test plan
t3-desktop-mcp.exe,list_appsworks, coordinate click returnsin backgroundlist_apps/ screenshot on X11 (or expected Wayland messaging)native/t3-chrome-extension, verify Settings shows extension installed andbrowser_*tools work in a labelled tab groupmainMade with Cursor
Note
Add Computer Use desktop control for macOS, Windows, and Linux
/settings/computer-useroute surfaces permission status (Accessibility, Screen Recording, Chrome extension) and links to relevant OS privacy panes.desktopControlsection with flags to independently disable agent cursor and browser control.Macroscope summarized c918e12.