Skip to content

feat(desktop): Computer Use for macOS, Windows, and Linux - #6537

Closed
sheehanmunim wants to merge 90 commits into
pingdotgg:mainfrom
munimtechnologies:mac/computer-use
Closed

sheehanmunim wants to merge 90 commits into
pingdotgg:mainfrom
munimtechnologies:mac/computer-use

Conversation

@sheehanmunim

@sheehanmunim sheehanmunim commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Adds local Computer Use via t3-desktop MCP (Swift on macOS, Rust on Windows/Linux), including agent cursor overlay and background clicks that do not steal the user mouse.
  • Ships agent-owned Chrome tab control through a bundled extension, plus Settings → Computer Use for enablement, extension status, and macOS privacy permission status.
  • Validated against latest origin/main (ahead of current nightly v0.0.34-nightly.20260813.1087): contracts/desktop/web typecheck clean for this work; Windows Blade E2E confirmed list_apps, background click … in background, and screenshot.

Demo

computer-use-demo.mp4

Settings → Computer Use:

Computer Use settings

Test plan

  • macOS: enable Computer Use in Settings, grant Accessibility + Screen Recording, confirm agent cursor overlay and app click/type/screenshot
  • Windows: confirm Nightly/server resolves t3-desktop-mcp.exe, list_apps works, coordinate click returns in background
  • Linux: smoke list_apps / screenshot on X11 (or expected Wayland messaging)
  • Chrome: load unpacked native/t3-chrome-extension, verify Settings shows extension installed and browser_* tools work in a labelled tab group
  • Toggle Computer Use off and confirm providers no longer inject the desktop MCP
  • CI green on this branch against main

Made with Cursor

Note

Add Computer Use desktop control for macOS, Windows, and Linux

  • Introduces a cross-platform desktop MCP server: a Swift implementation for macOS (native/t3-desktop-mcp) and a Rust implementation for Windows/Linux (native/t3-desktop-mcp-rs), both speaking JSON-RPC 2.0 over stdio.
  • The MCP server exposes tools for app listing, screen capture, accessibility-based element interaction, keyboard/mouse input, and browser tab control via a Chrome extension native messaging bridge.
  • An agent cursor overlay (per-platform: Win32 layered window, X11 shaped window, macOS LSUIElement app) visually tracks the agent's pointer position during tool execution and fades when idle.
  • Provider adapters (Claude, Codex, Cursor, Grok) are updated to launch the desktop MCP as a stdio server when enabled in settings and a binary is available.
  • A new /settings/computer-use route surfaces permission status (Accessibility, Screen Recording, Chrome extension) and links to relevant OS privacy panes.
  • Settings gain a desktopControl section with flags to independently disable agent cursor and browser control.
  • Risk: Computer Use is opt-in via settings, but when enabled the desktop process gains broad OS-level control (AX API, synthetic input, screen capture) and the Chrome extension requires debugger and all-URL host permissions.

Macroscope summarized c918e12.

Copilot AI lite review requested due to automatic review settings August 13, 2026 21:37
@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4a9f1b4b-0da0-4215-ba98-b397860b8c9a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 13, 2026

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One finding: the new desktop-control resolver hides its ServerSettingsService dependency behind Effect.serviceOption, so the requirement is absent from the type of every adapter that calls it and a missing layer silently resolves to "Computer Use enabled". Everything else in the changed TypeScript (subpath namespace imports for effect/*, Effect.fn helpers acquiring FileSystem/Path/HostProcess* from the environment, new Schema.is predicates that add real logic, static CodexAppServerRequestError.methodNotFound factory, no new ManagedRuntime/runPromise/fake Layer.succeed) matches the conventions.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts Outdated
Comment thread apps/server/src/provider/Layers/ClaudeAdapter.ts Outdated
Comment thread apps/server/src/provider/Layers/CursorAdapter.ts Outdated
Comment thread native/t3-chrome-extension/background.js

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds cross-platform “Computer Use” to T3 Code Desktop by bundling a local desktop-control MCP server (Swift on macOS, Rust on Windows/Linux) and an agent-owned Chrome tab workflow via a bundled extension, plus end-to-end settings/approvals plumbing across contracts, server adapters, desktop IPC, and the web UI.

Changes:

  • Bundle and resolve a t3-desktop MCP server across macOS/Windows/Linux, staging it into desktop build artifacts and injecting it into supported provider adapters when enabled.
  • Add Chrome extension bridge + install scripts, plus a new Settings → Computer Use page that surfaces enablement, extension status, and macOS privacy permission readiness.
  • Extend contracts and runtime ingestion to support new approval kinds (tool / permissions) and Codex MCP form-elicitation flows, with accompanying test coverage.

Reviewed changes

Copilot reviewed 60 out of 75 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
vite.config.ts Adjust staged formatter globs to avoid native/ files breaking commits
scripts/build-desktop-artifact.ts Stage desktop MCP binaries (Swift/Rust) and macOS Info.plist Apple Events usage text
scripts/build-desktop-artifact.test.ts Add tests for MCP executable naming/targets and macOS usage description
packages/effect-codex-app-server/src/errors.ts Extend identifier purpose literals for new approval request types
packages/contracts/src/settings.ts Add desktopControl settings subtree + patch schema
packages/contracts/src/providerRuntime.ts Add canonical request types for tool/permissions approvals
packages/contracts/src/orchestration.ts Extend ProviderRequestKind to include tool/permissions
packages/contracts/src/ipc.ts Add IPC schemas + DesktopBridge methods for Computer Use readiness
package.json Add build script for Swift desktop MCP server
native/t3-desktop-mcp/Sources/BrowserBridge.swift Swift Chrome-extension bridge via unix socket + native messaging framing
native/t3-desktop-mcp/Package.swift SwiftPM package definition for macOS desktop MCP server
native/t3-desktop-mcp/.gitignore Ignore SwiftPM build output
native/t3-desktop-mcp-rs/src/tools.rs Rust tool schema definitions matching macOS tool surface (with browser_* gating)
native/t3-desktop-mcp-rs/src/platform/mod.rs Cross-platform desktop backend trait + shared formatting/types
native/t3-desktop-mcp-rs/src/main.rs Rust MCP server: newline-delimited JSON-RPC over stdio + tool dispatch
native/t3-desktop-mcp-rs/src/capture.rs Shared display/window capture with Wayland/X11 fallbacks and PNG encoding
native/t3-desktop-mcp-rs/src/browser.rs Rust Chrome bridge + native-host relay over local socket
native/t3-desktop-mcp-rs/src/apps.rs App/window enumeration and pid resolution shared across backends
native/t3-desktop-mcp-rs/Cargo.toml New Rust crate definition + deps for Windows/Linux automation/capture
native/t3-desktop-mcp-rs/.gitignore Ignore Rust target output
native/t3-chrome-extension/manifest.json MV3 extension manifest (native messaging + debugger/tabGroups perms)
native/t3-chrome-extension/install.sh macOS/Linux host registration script for native messaging
native/t3-chrome-extension/install.ps1 Windows registry-based host registration script
native/t3-chrome-extension/background.js Chrome service worker implementing agent-owned tab group + DevTools-based automation
apps/web/src/session-logic.ts Teach pending approval derivation about tool/permissions kinds
apps/web/src/session-logic.test.ts Tests for deriving/retaining new approval kinds in UI work log
apps/web/src/routeTree.gen.ts Add generated route for Settings → Computer Use
apps/web/src/routes/settings.computer-use.tsx Route entry for the Computer Use settings page
apps/web/src/components/settings/SettingsSidebarNav.tsx Add sidebar icon mapping for Computer Use section
apps/web/src/components/settings/settingsSearch.ts Add search items/labels for Computer Use settings and permissions
apps/web/src/components/settings/ComputerUseSettings.tsx New settings UI: toggles, extension status, macOS permission readiness + deep links
apps/web/src/components/settings/browserBrandIcons.tsx Brand icons for browser/agent cursor display in settings
apps/web/src/components/chat/ComposerPendingApprovalPanel.tsx UI labels for tool vs permission pending approvals
apps/web/src/components/chat/ComposerPendingApprovalPanel.test.tsx Tests for new pending-approval labels
apps/web/src/assets/computer-use/google-chrome.svg Add Chrome brand asset
apps/web/src/assets/computer-use/firefox.svg Add Firefox brand asset (not supported yet)
apps/web/src/assets/computer-use/edge.svg Add Edge brand asset
apps/web/src/assets/computer-use/brave.svg Add Brave brand asset
apps/server/src/provider/Layers/GrokAdapter.ts Inject resolved desktop MCP server into Grok adapter MCP server list
apps/server/src/provider/Layers/CursorAdapter.ts Inject resolved desktop MCP server into Cursor adapter MCP server list
apps/server/src/provider/Layers/CodexSessionRuntime.ts Add permissions approval handling + MCP elicitation (tool/permissions) request plumbing
apps/server/src/provider/Layers/CodexSessionRuntime.test.ts Unit tests for permission approvals + MCP elicitation mapping/response building
apps/server/src/provider/Layers/CodexProvider.ts Advertise Codex MCP form-elicitation capability
apps/server/src/provider/Layers/CodexProvider.test.ts Test for Codex initialize capability flags
apps/server/src/provider/Layers/CodexAdapter.ts Map new request kinds/types and include elicitation detail extraction
apps/server/src/provider/Layers/CodexAdapter.test.ts Lifecycle tests for tool/permissions canonical approval mapping
apps/server/src/provider/Layers/ClaudeAdapter.ts Add desktop MCP server into Claude mcpServers map when resolvable
apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts Preserve new approval kinds through activity summaries/payload
apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.approval.test.ts Tests ensuring new approval kinds reach clients correctly
apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts Resolve desktop MCP only when enabled; pass env flags for cursor/browser toggles
apps/server/src/desktopControl/resolveEnabledDesktopMcp.test.ts Tests for enable/disable and env flag behavior
apps/server/src/desktopControl/desktopMcpBinary.ts Locate bundled/override MCP binary across platforms and dev layouts
apps/server/src/desktopControl/desktopMcpBinary.test.ts Tests for override resolution and unsupported platform behavior
apps/desktop/src/preload.ts Expose Computer Use IPC methods to renderer via preload bridge
apps/desktop/src/ipc/methods/computerUse.ts IPC handlers for permissions readiness + opening privacy panes
apps/desktop/src/ipc/DesktopIpcHandlers.ts Register new Computer Use IPC handlers
apps/desktop/src/ipc/channels.ts Add IPC channel constants for Computer Use
apps/desktop/src/computerUse/permissions.ts Read macOS TCC statuses + detect Chrome extension/host registration
apps/desktop/src/computerUse/permissions.test.ts Tests for permission status mapping, Settings deep links, and extension detection
apps/desktop/scripts/electron-launcher.mjs Patch macOS Info.plist to include Apple Events usage description

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread native/t3-desktop-mcp/Sources/BrowserBridge.swift
Comment thread native/t3-desktop-mcp/Sources/BrowserBridge.swift Outdated
Comment thread apps/server/src/provider/Layers/GrokAdapter.ts Outdated
Comment thread apps/server/src/provider/Layers/CursorAdapter.ts Outdated
Comment thread apps/server/src/provider/Layers/ClaudeAdapter.ts Outdated
Comment thread scripts/build-desktop-artifact.ts Outdated
Comment thread apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts Outdated
Comment thread native/t3-desktop-mcp/Sources/AgentCursor.swift Outdated
Comment thread native/t3-desktop-mcp/Sources/BrowserBridge.swift
Comment thread native/t3-desktop-mcp/Sources/AgentCursor.swift Outdated
Comment thread native/t3-desktop-mcp/Sources/AgentCursor.swift Outdated
Comment thread native/t3-desktop-mcp/Sources/AgentCursor.swift Outdated
Comment thread native/t3-desktop-mcp/Sources/BrowserBridge.swift
Comment thread native/t3-desktop-mcp/Sources/AgentCursor.swift Outdated
Comment thread native/t3-desktop-mcp/Sources/BrowserBridge.swift Outdated
Comment thread native/t3-desktop-mcp/Sources/BrowserBridge.swift Outdated
@macroscopeapp

macroscopeapp Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

13 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

You can customize Macroscope's approvability policy. Learn more.

Comment thread apps/server/src/desktopControl/desktopMcpBinary.ts
Comment thread native/t3-desktop-mcp/Sources/AgentCursor.swift
@sheehanmunim

Copy link
Copy Markdown
Contributor Author

Babysit update

Branch is merge-clean onto current origin/main, and the latest nightly tag (v0.0.34-nightly.20260813.1088) points at the same commit as main.

Review fixes pushed (settings fail-closed + per-session resolve without widening session R, platform-specific binary candidates + executable validation, Chrome profile enumeration, Swift bridge/cursor hardening). High/clear threads resolved; 4 remaining are explicitly deferred follow-ups (extension owned-tab persistence, Chrome window-id reuse, bridge bind TOCTOU, SkyLight coord fallback).

Note for reviewers / Macroscope: the Effect Service Conventions check keeps reporting the old Effect.serviceOption finding against resolveEnabledDesktopMcp.ts, but that code path is gone — current tip uses yield* ServerSettings.ServerSettingsService plus makeResolveEnabledDesktopMcp for adapters. Local t3 typecheck is clean.

Fork CI workflows may still need org approval; Vercel marketing fails with “Authorization required to deploy” (expected for fork PRs).

@sheehanmunim

Copy link
Copy Markdown
Contributor Author

Follow-up correctness gaps from review are now fixed on tip:

  • Chrome extension owned-tab/group state survives service-worker restarts via `chrome.storage.session`
  • Agent window id is scoped to the live Chrome pid (no reclaim after Chrome restart)
  • Browser bridge ownership uses an exclusive flock around bind
  • Background SkyLight input requires `setWindowLocation` or falls back

Windows/Linux desktop MCP packaging was already wired through `stageDesktopMcpRust` in desktop artifact builds — once this lands in Nightly, hand-copying the binary should no longer be needed.

Comment thread native/t3-desktop-mcp/Sources/main.swift Outdated
Comment thread native/t3-chrome-extension/background.js
Comment thread native/t3-desktop-mcp/Sources/main.swift
Comment thread native/t3-desktop-mcp/Sources/main.swift
Comment thread apps/server/src/provider/Layers/CodexSessionRuntime.ts Outdated
Comment thread apps/desktop/src/computerUse/permissions.ts
@sheehanmunim

Copy link
Copy Markdown
Contributor Author

Note on Macroscope Effect Service Conventions

This check is still failing with a retained finding against the deleted path `apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts` (`Effect.serviceOption` / `onNone` defaults).

That file was renamed to `desktopMcpLaunch.ts`, and the current implementation acquires settings with `yield* ServerSettings.ServerSettingsService` (fail-closed on `getSettings` errors). There is no `serviceOption` / `onNone` path left in the PR.

Correctness Check is green; unresolved review threads are at 0. Remaining merge blockers appear to be human review + fork Vercel auth, not code correctness.

Comment thread apps/server/src/provider/Layers/CodexAdapter.ts
Comment thread apps/server/src/provider/Layers/CodexAdapter.ts
Comment thread native/t3-desktop-mcp-rs/src/platform/agent_cursor.rs Outdated
Comment thread native/t3-desktop-mcp-rs/src/platform/windows.rs Outdated
Comment thread native/t3-desktop-mcp-rs/src/platform/windows.rs
Comment thread native/t3-chrome-extension/install.ps1 Outdated
Comment thread native/t3-chrome-extension/background.js
Comment thread apps/server/src/provider/Layers/CodexSessionRuntime.ts
Comment thread apps/server/src/provider/Layers/CodexSessionRuntime.ts Outdated
Comment thread native/t3-chrome-extension/background.js Outdated
Comment thread native/t3-chrome-extension/background.js Outdated
Comment thread native/t3-chrome-extension/background.js Outdated
@sheehanmunim

Copy link
Copy Markdown
Contributor Author

Addressed remaining bot findings on cdb9503a5:

  • MCP elicitation correlations use the JSON-RPC request id (fiber-local), so concurrent same-serverName prompts no longer collide
  • Chrome extension: dwell/message-based session teardown, reply only on the originating port, and disconnect cleanup only touches the captured tab set
  • resolveEnabledDesktopMcp.ts restored as a fail-closed re-export of desktopMcpLaunch.ts (acquires ServerSettingsService directly; settings failures disable Computer Use)

Vercel marketing deploy auth failure is outside this PR’s scope.

Comment thread native/t3-chrome-extension/background.js Outdated
Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs Outdated
Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs Outdated
Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs Outdated
Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs Outdated
Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs
Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs
sheehanmunim and others added 18 commits August 16, 2026 13:08
Keep History-only CLI wiring off the mac/computer-use branch.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ttings error handling

Only unlink bridge sockets this listener owns, stop advertising grim-only displays without capture support, and catch rejected privacy-settings open promises.

Co-authored-by: Cursor <cursoragent@cursor.com>
… branch

Apply Macroscope fixes for agent cursor lifecycle, browser bridge writes,
AX window pairing, app resolution, JSON-RPC parse errors, and Linux cursor
init without History-only daemon changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
Prevent whitespace-only app arguments from matching every running app via
substring fallback.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Also strip accidental computer-history CLI from the Computer Use branch.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep Computer Use History-free and avoid short PID queries binding
unrelated apps via substring name matches.

Co-authored-by: Cursor <cursoragent@cursor.com>
A leftover closing brace from stripping the History CLI broke the top-level
Swift script structure after native-host.

Co-authored-by: Cursor <cursoragent@cursor.com>
Nonblocking bridge writes, owner-only agent-cursor sockets, no WAYLAND_DISPLAY
mutation, Wayland type_text/X11 cookie checks, create-first bridge binding, and
inert browser when control is disabled.

Co-authored-by: Cursor <cursoragent@cursor.com>
…MOST

Avoid O_NONBLOCK on the shared socket, revalidate the fd under writeLock,
propagate Shift release failures, and stop fabricating FRONTMOST markers.

Co-authored-by: Cursor <cursoragent@cursor.com>
Pick the advertised press/click/activate action instead of hard-coding index 0.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the blueprint extension icons with the production black T3 mark, and shrink the agent pointer favicons so Chrome's circular tab crop no longer clips the cursor.

Co-authored-by: Cursor <cursoragent@cursor.com>
Port shared Macroscope/Bugbot fixes: safer bridge socket ownership, Int-safe coords, AppleScript escaping, scroll/click targeting, bridge write reset, and AT-SPI reconnect.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use a transparent circular badge (no black square crop) and rounded-full clipping so the Computer Use settings row matches the pointer favicon.

Co-authored-by: Cursor <cursoragent@cursor.com>
Point the Computer Use settings icon at the same soft-glow cursor-112 artwork as the desktop overlay, on a transparent canvas with no rounded crop.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace removed FiberRef usage so Dev can boot, and pad the settings/overlay cursor art so the lavender glow does not square-crop at size-6.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the original pointer art and fade the glow into the badge rim so Settings and Chrome tab favicons no longer show a nested square crop.

Co-authored-by: Cursor <cursoragent@cursor.com>
Reject overlapping-window element drags and refuse symlink bridge directories so background drag and Unix socket setup fail closed.

Co-authored-by: Cursor <cursoragent@cursor.com>
if (typeof entry.state === "number") return entry.state === 1;
if (typeof entry.path === "string" && entry.path.length > 0) return true;
if (entry.location === 4) return true;
return true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extension status false positives

Medium Severity

After handling explicit state, path, and unpacked location === 4, chromeExtensionInstalledInPreferences still falls through to return true for any leftover preference entry. Ghost or incomplete extensions.settings records then report as installed in Settings even when the extension is not actually loadable.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 03b1beb. Configure here.

];
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge and Brave detection missing

Medium Severity

Computer Use Settings documents Edge and Brave setup, but chromeProfileRoots only scans Chrome/Chromium paths, and the install scripts only register the native host for Chrome/Chromium. An Edge- or Brave-only install never shows as installed and never gets a working native-messaging host registration.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 03b1beb. Configure here.

// Without this key macOS denies every Apple Event with errAEEventNotPermitted
// (-1743) and never shows the Automation prompt, so Codex Computer Use and any
// other MCP server we spawn silently fail to drive other apps.
setPlistString(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium scripts/electron-launcher.mjs:234

Existing .electron-runtime bundles continue to lack NSAppleEventsUsageDescription, so Computer Use Apple Events remain denied for developers with a cached launcher. buildMacLauncher returns the cached bundle before calling patchMainBundleInfoPlist; bump LAUNCHER_VERSION from 15 to invalidate those bundles and apply the new key.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/desktop/scripts/electron-launcher.mjs around line 234:

Existing `.electron-runtime` bundles continue to lack `NSAppleEventsUsageDescription`, so Computer Use Apple Events remain denied for developers with a cached launcher. `buildMacLauncher` returns the cached bundle before calling `patchMainBundleInfoPlist`; bump `LAUNCHER_VERSION` from 15 to invalidate those bundles and apply the new key.


fn compositing_manager_running(conn: &RustConnection, screen_num: usize) -> bool {
let name = format!("_NET_WM_CM_S{screen_num}");
let Ok(atom) = conn.intern_atom(false, name.as_bytes()) else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium platform/agent_cursor_linux.rs:801

present performs two synchronous X11 request/reply round trips on every rendered frame, so a visible cursor repeatedly blocks the UI thread and can reduce animation to a few FPS on high-latency or remote displays. Cache the _NET_WM_CM_S{screen_num} atom and query compositor ownership outside the render loop.

🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/agent_cursor_linux.rs around line 801:

`present` performs two synchronous X11 request/reply round trips on every rendered frame, so a visible cursor repeatedly blocks the UI thread and can reduce animation to a few FPS on high-latency or remote displays. Cache the `_NET_WM_CM_S{screen_num}` atom and query compositor ownership outside the render loop.

Comment thread native/t3-desktop-mcp-rs/src/browser.rs
// become `connection`, blocking the real overlay.
_ = chmod(path, 0o600)

let source = DispatchSource.makeReadSource(fileDescriptor: fd, queue: .main)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium Sources/AgentCursor.swift:356

The cursor listener never accepts the overlay connection during normal MCP operation, so queued cursor messages are never flushed and the agent cursor does not appear. startListening schedules acceptConnection() on .main, but the executable keeps that queue blocked in the synchronous runJSONRPCLoop/readLine loop; schedule the source on a dedicated serial queue instead.

🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp/Sources/AgentCursor.swift around line 356:

The cursor listener never accepts the overlay connection during normal MCP operation, so queued cursor messages are never flushed and the agent cursor does not appear. `startListening` schedules `acceptConnection()` on `.main`, but the executable keeps that queue blocked in the synchronous `runJSONRPCLoop`/`readLine` loop; schedule the source on a dedicated serial queue instead.

// One shared element budget across every window — recreating the walk
// buffer per window would let multi-window apps emit
// windows.len() * max_elements rows.
let mut element_lines = Vec::new();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium platform/windows.rs:421

When element_from_handle fails for every visible window, get_app_state still returns Ok with no window contents and the misleading no interactive elements found message. Because each conversion error is skipped with continue, callers treat an inaccessible snapshot as a valid empty snapshot; track successful conversions and return an error when none succeed.

🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/windows.rs around line 421:

When `element_from_handle` fails for every visible window, `get_app_state` still returns `Ok` with no window contents and the misleading `no interactive elements found` message. Because each conversion error is skipped with `continue`, callers treat an inaccessible snapshot as a valid empty snapshot; track successful conversions and return an error when none succeed.

Comment thread native/t3-desktop-mcp-rs/src/browser.rs
Comment thread native/t3-desktop-mcp/Sources/main.swift
"description": "Read an app's accessibility tree as an indented outline. Interactive elements are prefixed with an id like [e12] that you pass to click/type_text/scroll. Call this before interacting, and again after the UI changes, since ids are per-snapshot.",
"inputSchema": {
"type": "object",
"properties": {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium src/tools.rs:57

get_app_state on Windows/Linux cannot accept the advertised window argument, so requests for a specific window are rejected or read with the wrong scope in multi-window apps. Add the same window input property exposed by the macOS toolDefs, and extend advertises_the_macos_tool_surface to validate input schemas rather than only tool names.

🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/tools.rs around line 57:

`get_app_state` on Windows/Linux cannot accept the advertised `window` argument, so requests for a specific window are rejected or read with the wrong scope in multi-window apps. Add the same `window` input property exposed by the macOS `toolDefs`, and extend `advertises_the_macos_tool_surface` to validate input schemas rather than only tool names.

.or_else(|_| std::env::var("USER"))
.unwrap_or_else(|_| "user".into());
// Named-pipe namespace is global; embed the username so sessions do not collide.
format!("t3-desktop-mcp-bridge-{user}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High src/browser.rs:127

bridge_pipe_name produces the same pipe name for multiple Windows login/RDP sessions using the same USERNAME, so a host in a second session connects to the first session's MCP listener and routes browser control and replies to the wrong desktop. Include session- and account-unique identifiers, such as the session ID and SID, in the pipe name.

🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/browser.rs around line 127:

`bridge_pipe_name` produces the same pipe name for multiple Windows login/RDP sessions using the same `USERNAME`, so a host in a second session connects to the first session's MCP listener and routes browser control and replies to the wrong desktop. Include session- and account-unique identifiers, such as the session ID and SID, in the pipe name.

let mut next_id = 0u32;
let mut visited = 0usize;
let mut lines = vec![format!("{name}")];
self.walk(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium platform/linux.rs:934

get_app_state returns Ok with only the app name and “no interactive elements exposed” when the root accessibility proxy or its AT-SPI calls fail after applications() succeeds. Because walk returns () and silently stops on those errors, callers cannot distinguish an unavailable application from a valid empty snapshot; propagate the traversal error instead.

🤖 Copy this AI Prompt to have your agent fix this:
In file @native/t3-desktop-mcp-rs/src/platform/linux.rs around line 934:

`get_app_state` returns `Ok` with only the app name and “no interactive elements exposed” when the root accessibility proxy or its AT-SPI calls fail after `applications()` succeeds. Because `walk` returns `()` and silently stops on those errors, callers cannot distinguish an unavailable application from a valid empty snapshot; propagate the traversal error instead.

Map press_key to press, clear stale Chrome window cache, escape app-list brackets, and fix Wayland/X11 click and type edge cases flagged after rebase.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

There are 20 total unresolved issues (including 18 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 92d6674. Configure here.

const SS_NOTIFY: i32 = 0x0001_0000;
let style = unsafe { GetWindowLongW(hwnd, GWL_STYLE) };
return style & SS_NOTIFY != 0;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong SS_NOTIFY style constant

High Severity

The Static background-click gate uses 0x0001_0000 as SS_NOTIFY, but the real Win32 value is 0x00000100. That bit is WS_TABSTOP, so clickable notify statics are often rejected while tab-stop labels can be accepted. Posted mouse messages then either never run or report success when the control ignores them.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 92d6674. Configure here.

} else {
0x0100_0000 + code
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Newline maps to Linefeed keysym

Medium Severity

char_to_keysym maps '\n' to 0xff0a (XK_Linefeed) while press_key / enter uses 0xff0d (XK_Return). type_text with Unix newlines therefore synthesizes Linefeed instead of Return, so many GTK/Qt/terminal fields never insert a line break.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 92d6674. Configure here.

Only fall back from AT-SPI to X11 when there is no match, not when the query matches several apps.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants