feat(desktop): Computer History for macOS, Windows, and Linux

[sheehanmunim]

Summary

• Adds opt-in Computer History: accessibility interaction capture (not screenshots) → local segment store → summarized memories agents can reference, with optional Codex skysight mirror.
• Desktop owns the recorder daemon (macOS AX via Swift MCP; Windows UI Automation + Linux AT-SPI/EWMH via Rust MCP); server summarizes on an interval and injects recent history into provider context.
• Settings UI under Computer History (enable / pause / mirror / clear / timeline), plus Linux FRONTMOST sampling fixes and a Linux Xvfb+Openbox smoke harness.

Depends on #6537 (mac/computer-use). This branch is stacked on that work — for a Computer History–only diff, compare against mac/computer-use.

Test plan

• macOS Dev — enable persists; daemon running; events collected; summaries + Codex mirror
• Windows Blade (interactive session) — Notepad/Calculator sample.frontmost events; memory written; Dev coexists with Nightly (phase=running, events continuing)
• Linux (Colima Ubuntu aarch64) — build t3-desktop-mcp; Xvfb + Openbox records sample.frontmost for XTerm (reconfirmed after PR open)
• Shared store unit tests — packages/shared/src/computerHistory/store.test.ts (6/6 passing)
• Review Settings → Computer History in desktop Dev and confirm Accessibility / UI Automation prompts as needed
• Confirm disabling stops the daemon and clears injection from new agent turns

Note

Add Computer History recording and Computer Use settings for macOS, Windows, and Linux

• Introduces a new cross-platform desktop MCP binary (t3-desktop-mcp) in Rust for Windows/Linux (alongside the existing Swift binary for macOS) that exposes desktop control tools over stdio JSON-RPC, including a computer-history daemon mode that writes structured JSONL activity segments.
• Adds a ComputerHistoryManager in Electron main that spawns/stops the daemon, syncs a control.json with enable/pause/filter settings, and exposes status, timeline, clear, and reveal operations via new IPC channels and window.desktopBridge methods.
• Adds a Chrome MV3 extension (native/t3-chrome-extension) with a native messaging bridge to relay browser events to the MCP server, plus install scripts for macOS/Linux and Windows.
• Integrates the t3-desktop MCP server into all provider adapters (Claude, Codex, Cursor, Grok) alongside the existing t3-code HTTP server, re-resolving per session so settings changes apply without restart.
• Adds new settings UI routes (/settings/computer-use, /settings/computer-history) with permission status display, privacy pane openers, and exclusion list management.
• Extends Codex session runtime to handle item/permissions/requestApproval and mcpServer/elicitation/request events, mapping them to canonical permissions_approval / tool_approval request kinds with distinct UI icons and labels.
• Risk: the Computer History daemon is off by default (enabled: false), but when enabled it continuously writes activity segments to disk and syncs a control file every minute via a new background server loop.

^{Macroscope summarized e4775dc.}

---

Note

High Risk
Spawns native binaries with desktop automation, continuous local activity recording when enabled, and broad provider/MCP integration across auth-adjacent approval flows.

Overview
Computer Use and Computer History are wired end-to-end: settings pages, desktop IPC, a spawned t3-desktop-mcp recorder daemon, and optional injection of recent history into agent turns.

Computer Use adds settings for enabling desktop control, agent cursor overlay, and Chrome browser control. The desktop app exposes macOS accessibility/screen-recording status and Chrome extension detection via IPC. When enabled, Claude, Codex, Cursor, and Grok sessions attach a stdio t3-desktop MCP server (alongside existing t3-code), resolved per session from desktopControl settings. Codex advertises form elicitation; the runtime maps MCP permission and generic tool approvals into permissions_approval / tool_approval with matching UI in the composer and timeline. macOS Info.plist gains NSAppleEventsUsageDescription so Automation prompts work for spawned MCP tools.

Computer History adds a ComputerHistoryManager that writes control.json, spawns computer-history on the desktop MCP binary at bootstrap (if already enabled) and stops it cleanly on quit. Server-side background work syncs control and summarizes segments every minute; Codex turns can append loaded history to developer instructions. Settings cover enable/pause, Codex mirror, privacy exclusions, timeline clear/reveal/delete, with status polls that do not respawn the daemon after disable.

^{Reviewed by Cursor Bugbot for commit e4775dc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7a71e845-6c16-4e75-876a-ddab0bbefa25

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Reviewed the changed TypeScript for the Effect service conventions. Five findings, all in new server/desktop code: a hidden process.env dependency channel between two layers, a compatibility re-export shim, module-global daemon state consumed from Effect, and a layer export name.

^{Posted via Macroscope — Effect Service Conventions}

[Copilot]

Pull request overview

Adds an opt-in “Computer History” feature (accessibility-driven interaction event capture, local segment storage, periodic summarization into memory markdown, and optional Codex Skysight mirroring) and wires it through desktop IPC, server/provider context injection, and the web Settings UI alongside updated approval kinds for tool/permission gating.

Changes:

• Extend contracts/runtime ingestion + web UI to support new approval kinds (tool, permissions) and surface them consistently.
• Add Computer History shared store/summarization utilities, server-side summarization loop + provider injection, and desktop IPC + daemon management.
• Update desktop build/artifact staging to bundle the desktop MCP server across platforms (Swift on macOS; Rust on Windows/Linux) and include required macOS Info.plist usage strings.

Reviewed changes

Copilot reviewed 86 out of 102 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vite.config.ts	Restricts staged formatter glob to file types vp fmt can handle.
scripts/build-desktop-artifact.ts	Stages desktop MCP binaries (Swift/Rust) into desktop artifacts and adds macOS Apple Events usage description.
scripts/build-desktop-artifact.test.ts	Adds tests for desktop MCP executable naming/targets and macOS Apple Events metadata.
packages/shared/src/computerHistory/events.ts	Defines Computer History event/control/status file shapes and filter helpers.
packages/shared/src/computerHistory/paths.ts	Adds filesystem layout helpers for Computer History + Codex Skysight mirror paths.
packages/shared/src/computerHistory/summarize.ts	Implements summarization + markdown rendering + Skysight instruction text.
packages/shared/src/computerHistory/store.ts	Implements segment reading, summarization passes, mirroring, timeline, deletion, and context rendering.
packages/shared/src/computerHistory/store.test.ts	Unit tests for filters, summarization markdown, timeline/mirroring/clearing, and memory writes.
packages/shared/src/computerHistory/index.ts	Exports Computer History module surface.
packages/shared/package.json	Exposes ./computerHistory subpath export.
packages/contracts/src/computerHistory.ts	Adds contract schemas for status/timeline/clear scope/suggestions.
packages/contracts/src/index.ts	Exports new computerHistory contracts.
packages/contracts/src/settings.ts	Adds desktopControl and computerHistory settings + patch schema support.
packages/contracts/src/providerRuntime.ts	Adds canonical request types for tool/permissions approvals.
packages/contracts/src/orchestration.ts	Extends provider request kinds to include tool and permissions.
packages/contracts/src/ipc.ts	Extends desktop bridge IPC types for Computer Use permissions + Computer History operations.
packages/effect-codex-app-server/src/errors.ts	Adds new identifier purposes for approval request types.
packages/effect-codex-app-server/src/client.ts	Adds fiber-local CurrentServerRequestId for correlating concurrent server requests.
package.json	Adds build:desktop-mcp script for macOS Swift build.
native/t3-desktop-mcp/Package.swift	Defines SwiftPM package for macOS desktop MCP server.
native/t3-desktop-mcp/.gitignore	Ignores SwiftPM build output.
native/t3-desktop-mcp/Sources/ComputerHistory.swift	Adds macOS Computer History daemon recording AX + frontmost sampling to JSONL + status/control files.
native/t3-desktop-mcp/Sources/BrowserBridge.swift	Adds Chrome native messaging bridge via Unix socket + native-host mode.
native/t3-desktop-mcp-rs/Cargo.toml	Adds Rust crate for Windows/Linux desktop MCP server.
native/t3-desktop-mcp-rs/.gitignore	Ignores Rust target/ build output.
native/t3-desktop-mcp-rs/src/platform/mod.rs	Adds platform abstraction/types and shared formatting + tests.
native/t3-desktop-mcp-rs/src/capture.rs	Adds guarded capture + Wayland/X11 handling + PNG scaling with tests.
native/t3-desktop-mcp-rs/src/apps.rs	Adds window-based app discovery + pid resolution with tests.
native/t3-desktop-mcp-rs/linux-ch-smoke/Dockerfile	Adds Linux Xvfb/Openbox-ish smoke harness build image for CH sampling.
native/t3-desktop-mcp-rs/linux-ch-smoke/run.sh	Adds Linux smoke script validating daemon writes status/events.
native/t3-chrome-extension/manifest.json	Adds MV3 extension manifest for desktop control bridge.
native/t3-chrome-extension/wake.js	Adds navigation wake-up ping for MV3 service worker.
native/t3-chrome-extension/install.sh	Adds macOS/Linux native host registration script for Chrome/Chromium profiles.
native/t3-chrome-extension/install.ps1	Adds Windows native host registration (registry + wrapper) for Chrome/Chromium.
apps/server/src/server.ts	Installs Computer History runtime layer into background layer.
apps/server/src/computerHistory/service.ts	Adds server helpers for CH context injection + control sync + periodic summarization loop.
apps/server/src/desktopControl/desktopMcpBinary.ts	Adds resolver for bundled/dev desktop MCP binary path.
apps/server/src/desktopControl/desktopMcpBinary.test.ts	Tests desktop MCP path resolution behavior across platforms.
apps/server/src/desktopControl/desktopMcpLaunch.ts	Adds settings-gated resolver that emits env flags for desktop MCP launch.
apps/server/src/desktopControl/desktopMcpLaunch.test.ts	Tests settings gating + env flags + fail-closed behavior for desktop MCP injection.
apps/server/src/desktopControl/resolveEnabledDesktopMcp.ts	Compatibility re-export for the new desktop MCP resolution API.
apps/server/src/provider/Layers/ClaudeAdapter.ts	Injects desktop MCP server into Claude sessions when enabled/resolved.
apps/server/src/provider/Layers/CursorAdapter.ts	Injects desktop MCP server into Cursor sessions when enabled/resolved.
apps/server/src/provider/Layers/GrokAdapter.ts	Injects desktop MCP server into Grok sessions when enabled/resolved.
apps/server/src/provider/Layers/GrokAdapter.test.ts	Updates test layer wiring to include server settings service.
apps/server/src/provider/Layers/CodexProvider.ts	Advertises Codex form-elicitation capability.
apps/server/src/provider/Layers/CodexProvider.test.ts	Tests Codex initialize capabilities include form elicitation.
apps/server/src/provider/Layers/CodexSessionRuntime.ts	Adds permissions + MCP elicitation approval handling; injects Computer History context into Codex turn start.
apps/server/src/provider/Layers/CodexSessionRuntime.test.ts	Adds tests for permissions response + MCP approval kind detection/mapping.
apps/server/src/provider/Layers/CodexAdapter.ts	Maps new approval kinds; injects desktop MCP into Codex via app server args/env.
apps/server/src/provider/Layers/CodexAdapter.test.ts	Adds lifecycle tests for mapping permissions/tool approvals into canonical events.
apps/server/src/provider/CodexDeveloperInstructions.ts	Allows appending Computer History context into Codex developer instructions.
apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts	Preserves tool/permissions approvals into activities with correct summaries.
apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.approval.test.ts	Adds tests ensuring tool/permissions approvals are preserved for clients.
apps/web/src/session-logic.ts	Extends requestKind derivation/extraction for tool + permissions approvals.
apps/web/src/session-logic.test.ts	Adds tests for deriving/preserving tool + permissions approvals.
apps/web/src/components/chat/ComposerPendingApprovalPanel.tsx	Adds UI labels for tool vs permission approvals.
apps/web/src/components/chat/ComposerPendingApprovalPanel.test.tsx	Tests new approval label behavior.
apps/web/src/routeTree.gen.ts	Adds settings routes for Computer Use and Computer History.
apps/web/src/routes/settings.computer-use.tsx	Adds Settings route entry for Computer Use page.
apps/web/src/routes/settings.computer-history.tsx	Adds Settings route entry for Computer History page.
apps/web/src/components/settings/SettingsSidebarNav.tsx	Adds sidebar icons for Computer Use and Computer History sections.
apps/web/src/components/settings/settingsSearch.ts	Adds settings search entries/labels for Computer Use + Computer History.
apps/web/src/components/settings/ComputerHistorySettings.tsx	Adds Settings UI for enabling/pausing/mirroring CH and viewing/clearing timeline.
apps/web/src/components/settings/browserBrandIcons.tsx	Adds brand icons + agent cursor icon component for Computer Use UI.
apps/web/src/assets/computer-use/google-chrome.svg	Adds Chrome icon asset for settings UI.
apps/web/src/assets/computer-use/edge.svg	Adds Edge icon asset for settings UI.
apps/web/src/assets/computer-use/brave.svg	Adds Brave icon asset for settings UI.
apps/desktop/src/preload.ts	Exposes new IPC methods for Computer Use permissions + Computer History actions to renderer.
apps/desktop/src/ipc/channels.ts	Adds IPC channel constants for Computer Use + Computer History.
apps/desktop/src/ipc/methods/computerUse.ts	Adds IPC handlers for reading permissions and opening macOS privacy panes.
apps/desktop/src/ipc/methods/computerHistory.ts	Adds IPC handlers for status/timeline/patch/clear/reveal/delete for CH.
apps/desktop/src/ipc/DesktopIpcHandlers.ts	Registers new Computer Use + Computer History IPC handlers.
apps/desktop/src/computerUse/permissions.ts	Adds permission and Chrome extension/native host readiness detection + privacy pane open.
apps/desktop/src/computerUse/permissions.test.ts	Adds tests for permissions state computation and privacy pane open behavior.
apps/desktop/src/computerHistory/resolveBinary.ts	Adds sync resolver for desktop MCP binary path in Electron main.
apps/desktop/src/computerHistory/manager.ts	Adds daemon lifecycle management + status/timeline/clear/delete/reveal plumbing for CH.
apps/desktop/scripts/electron-launcher.mjs	Bumps launcher version and adds NSAppleEventsUsageDescription to dev bundle Info.plist.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[macroscopeapp]

Approvability

Verdict: Needs human review

Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

[macroscopeapp]

Effect service conventions review of the Computer History / Computer Use changes. The desktop ComputerHistoryManager service is the main concern: its implementation drives a fresh Effect runtime (Effect.runPromise) from inside the service constructor and exposes an error-free interface that turns real failures into defects. Two smaller items: make is not exported and a dead re-export was left in the new server module.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

One convention issue found in the Computer History bootstrap path. The rest of the new Effect service code (apps/desktop/src/computerHistory/manager.ts, apps/server/src/computerHistory/service.ts, apps/server/src/desktopControl/desktopMcp*.ts) follows the module layout, make/layer, typed-error and environment-acquisition conventions.

^{Posted via Macroscope — Effect Service Conventions}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 7 total unresolved issues (including 6 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d06518e. Configure here.}

[sheehanmunim]

Macroscope UI Consistency is still failing against retained findings from e1f85f3 that no longer match HEAD:

• Exclusion chip remove is now SettingIconAction (same primitive as SettingResetButton) in ComputerHistoryPrivacyControls.tsx
• Privacy dialog Cancel is variant="outline"
• More browsers uses Collapsible + CollapsibleTrigger + CollapsibleContent

Please re-run UI Consistency against current HEAD.

[sheehanmunim]

Temporarily closing to force Macroscope UI Consistency to re-evaluate against current HEAD (retained findings from e1f85f3 no longer match).

[sheehanmunim]

@macroscope-app review

Please fully re-evaluate UI Consistency against current HEAD (ee1495a38), not the retained findings from e1f85f3.

Those three items are already fixed on HEAD:

1. Exclusion-chip remove uses SettingIconAction in ComputerHistoryPrivacyControls.tsx (extracted from ComputerHistorySettings.tsx).
2. Privacy dialog Cancel uses variant="outline" via DialogClose + Button.
3. "More browsers" uses Collapsible + CollapsibleTrigger + CollapsiblePanel with aria-controls in ComputerUseSettings.tsx.

The check has been re-running with a stale summary that still claims only e1f85f3 (native-only) changed since the last UI pass, even after web UI commits 519db1639 and ee1495a38.

[macroscopeapp]

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

[sheehanmunim]

Closing to replace with a fresh PR — Macroscope UI Consistency is stuck retaining findings from e1f85f3 and ignoring subsequent web settings commits (including full privacy UI rewrite). Replacement PR incoming.