diff --git a/docs/architecture/computer-use-control.md b/docs/architecture/computer-use-control.md index 6da8fab774..086be0e1e6 100644 --- a/docs/architecture/computer-use-control.md +++ b/docs/architecture/computer-use-control.md @@ -1,8 +1,10 @@ # Computer Use control-session SPEC -Status: the October 2026 migration implements permissions, user-selected App -sharing and control grants, coordinate mapping, virtual cursor feedback and -platform-scoped background input. Validation includes native macOS Tool/input/ +Status: Agent tasks select applications and execute through the existing tool +permission policy without a manual App-sharing or per-application grant step. +Optional manual-sharing APIs remain compatible for existing clients. The October +2026 migration implements permissions, coordinate mapping, virtual cursor +feedback and platform-scoped background input. Validation includes native macOS Tool/input/ capture checks, Linux GTK/AT-SPI execution and Windows native-module cross-compilation. Windows native execution, GNOME Wayland portal execution and remote end-to-end control remain acceptance gaps. Compilation is not native @@ -33,7 +35,7 @@ replace system consent or system Stop Sharing. | macOS | Persistent ScreenCaptureKit `SCStream`, exact single-window filter | AX semantic actions; directed application events where supported; global input only with foreground authorization | System sharing integration and native stop; nonactivating AppKit feedback | | Windows | Persistent Windows.Graphics.Capture window session | UI Automation patterns and supported targeted window messages; `SendInput` requires foreground authorization | WGC capture border and Closed event; product preview/Stop | | Ubuntu Wayland | RemoteDesktop and ScreenCast portals sharing one session and its authorized PipeWire FD/stream | AT-SPI semantic actions; portal Notify seat input only in explicit foreground mode | Portal consent, compositor sharing UI and Session.Closed | -| Ubuntu X11 | New explicit control sessions use the same portal path | AT-SPI; missing portal capabilities fail explicitly | Product status/Stop; no claim of Wayland isolation for legacy X11 calls | +| Ubuntu X11 | Observe/foreground sessions use the same portal path; background semantic sessions need no picker | AT-SPI; missing portal capabilities fail explicitly | Product status/Stop; no claim of Wayland isolation for legacy X11 calls | The control snapshot’s `capabilities` lists backend routes, such as semantic input, directed keyboard/scroll or foreground seat input. It is not a promise @@ -73,25 +75,35 @@ runtime, build, test or filesystem dependency of OpenBitFun. | App sharing and control grants | `src/sharing.rs`, `src/grants.rs`, sharing controls in `src/gui.rs` | Admission state machine plus Desktop `sharing.rs`, `shared_preview.rs`, `process_identity.rs`, `grants.rs`, `share_grants.rs`; user-selected native window, observe first, separate revocable grant, executable-bound remembered preference, manual preview gestures | | Coordinate conversion | `CoordinateMap` in `src/sharing.rs` | Portable `CaptureGeometry` and Web `previewGeometry.ts`; image/content/global/preview round trips, aspect fit, negative origins, independent X/Y scales, strict padding rejection and immutable screenshot identity | | Virtual mouse | `src/cursor.rs`, GUI cursor rendering | Web `VirtualCursor.ts` ports the deterministic physical model, rendered by `PreviewCursor.tsx`; 63 Rust-generated reference samples cover movement, retargeting, immediate placement, resize and visibility. Existing passive native overlays remain platform adapters | -| Background control | Native directed/semantic input and desktop-state comparison | Existing AX/UIA/AT-SPI and directed-input providers plus `background_guard.rs`; focus/pointer/window-order evidence, explicit unsupported routes, revocation on observed interference, no restore or replay. The reference's implicit Command modifier is not copied because it changes the requested click | +| Background control | Native directed/semantic input and desktop-state comparison | Existing AX/UIA/AT-SPI and directed-input providers plus `background_guard.rs`; diagnostic focus/pointer/window-order evidence, explicit unsupported routes, no restore or replay. Concurrent human input does not revoke Agent control. The reference's implicit Command modifier is not copied because it changes the requested click | ### Permissions and user sharing -Opening settings, checking status or reading AX never requests OS permission. +Opening the OpenBitFun settings page, checking status or reading AX never requests OS permission. The user selects each request explicitly. Receipts distinguish a request API invocation from the current grant and from observing a visible OS dialog. Input Monitoring is optional. Windows does not invent a macOS-style grant; protected desktops, capture policy and target integrity still apply. Linux's portal grant belongs to one selected session; AT-SPI availability does not grant seat input. -`start_control` and implicit tool admission return `awaiting_share` / a clear -`SHARE_REQUIRED` result until the user selects a target in the executing host's -control card. Discovery returns opaque choices bound to the process lifetime, -not authority derived from a title or a client-supplied PID. Starting the selected -share validates the live process and exact window, obtains real pixels or the -supported AT-SPI tree, then exposes observation. Input requires a separate user -grant for the selected background or foreground mode. A tool cannot select a -different window, grant itself input or widen the mode. +`start_control` enters the requested mode directly. An initial app action can +start background control implicitly. The Agent discovers applications through +`list_apps`, supplies the target selector and continues observation/input under +the existing Runtime tool policy. Application changes invalidate old references; +they do not require a user to select the next app. Background AT-SPI startup +does not open a portal picker. Linux portal capture and foreground seat input +still require compositor-owned consent. + +The optional manual-sharing API uses `awaiting_share`, opaque process-bound +choices, observe-first capture and a separate revocable input grant. Its scope +does not apply to normal Agent tasks. Existing remembered preferences and manual +clients remain readable; no stored user grants are deleted by this correction. +Settings uses the established `ConfigPageRow` layout: desktop-control toggle and +the two required macOS permissions, with setup actions only for missing grants. +Input Monitoring, executable diagnostics and manual-sharing controls are not +part of this settings page. The missing-permission setup button explicitly +requests the corresponding OS permission on capable hosts before opening its +system pane; older hosts retain their existing system-pane action. Remembering a grant is optional and never starts a share or selects a window. The preference binds the canonical executable path, full executable SHA-256, @@ -102,6 +114,27 @@ data; writes are atomic, privately permissioned and protected against concurrent changes. Forgetting an active matching preference revokes input before disk IO. Portal streams have no verified executable identity and remain session-only. +### Autonomous task execution + +The built-in prompt carries the user's task through app discovery, observation, +input, verification and supported recovery. Routine app targeting, connection +alternatives and message composition do not create additional approval steps. +A request to compose and send specified information to a named recipient includes +composing the wording; the Agent verifies the recipient/content without asking +for the same authorization again. Genuine missing information, OS-owned consent, +explicit permission denials and user Stop remain authoritative. + +Computer Use, Cowork, Claw and shared parent-agent guidance use the same autonomous +targeting and composition rules. Browser connection hints offer an already-authorized ComputerUse route when CDP +is unavailable instead of requiring the user to operate Settings. A user-disabled +browser connection is not re-enabled. `open_app` no longer has a platform-wide +foreground-only policy: macOS uses `open -g` and Windows requests +`SW_SHOWNOACTIVATE` for background launches. These are launch requests, not proof +that every third-party app honors activation policy; subsequent observations +establish actual state. Linux currently reports unavailable background launch +instead of silently using an activating launcher. Existing AT-SPI targets remain +usable. GUI-less hosts and remote workspaces do not fall back to the local GUI. + Manual preview input joins the same exclusive action lease as the Agent. Pointer gestures retain their screenshot until release; resizing or revocation cancels them. Click/right click/double click, drag, scroll, text and key input use the @@ -174,20 +207,19 @@ authorization boundary for native chrome, dialogs or unattached windows. ## Session lifecycle and revocation -The user-facing resource lifecycle is -`idle → awaiting_share → active(observe) → active(authorized mode) → stopped|failed`, +The Agent resource lifecycle is `idle → active(requested mode) → stopped|failed`, with internal starting/cleanup transitions. One Runtime session owns the GUI resource. Competing ownership returns `control_busy`; it does not create a second permission owner or Agent loop. -An explicit tool start requests a mode; the user's share selection and grant -establish authority. Discovery alone does not create capture. Legacy wire fields -remain readable but do not bypass the new host's user-selection gate. Linux -portal capture binds the consented surface separately from semantic application -identity. Stop/status are idempotent. A stopped generation requires an explicit -restart, not a screenshot-driven resurrection. Revoking control preserves the -shared observation and increments the authorization revision immediately; stale -workers lose input authority and held native inputs are released. +An explicit tool start requests a mode under Runtime permission policy. Discovery +alone does not create capture. Linux portal capture binds the consented surface +separately from semantic application identity. Stop/status are idempotent. A +stopped generation requires an explicit restart, not a screenshot-driven +resurrection. The Agent may restart after recoverable target/capture loss, but +must respect a user stop or denial. Optional manual sharing retains its separate +`awaiting_share → active(observe) → active(granted mode)` lifecycle; revoking its +grant preserves observation and invalidates stale input workers immediately. Stop revokes admission and advances generation before cancelling native work, releasing owned pressed keys/buttons, stopping capture and hiding feedback. @@ -279,12 +311,13 @@ Background actions capture desktop witnesses before and after submission. macOS checks foreground identity plus its activation epoch, the physical pointer and relative order of existing windows; Windows checks foreground HWND, pointer and window order. Linux can inspect active AT-SPI top-level windows but cannot -observe compositor cursor/stacking, which remain unknown in the receipt. A -missing preflight witness sends no input. Changed or unavailable post-action -evidence revokes the session without focus restoration or retry. Concurrent -human/system input can also change these witnesses, so an interference error -does not blame the application or claim that already submitted effects were -undone. Boundary samples cannot prove all transient or later application behavior. +observe compositor cursor/stacking, which remain unknown in the receipt. These +witnesses are diagnostics, not admission or revocation gates: the user may keep +working while directed input runs. Missing samples leave evidence unavailable; +changed samples retain their actual values without replacing the native action's +result, stopping the session or replaying input. Native target, mode and lease +checks still govern input. Boundary samples cannot attribute changes to this +operation or prove all transient or later application behavior. ### macOS @@ -453,7 +486,7 @@ checks discard stale preview responses and device-switch results. | Scenario | Required behavior | |---|---| | Remote workspace | Existing SSH/Docker workspace refusal; never operate the controller's local GUI as fallback | -| Remote control | Mobile/IM have no App picker or grant UI and receive an explicit unsupported/`SHARE_REQUIRED` state; no hidden blocking prompt; task cancellation remains reachable | +| Remote control | An executing Desktop's Agent selects and controls its own applications without requiring an App picker on Mobile/IM; cancellation remains reachable. Native OS consent and missing GUI providers are explicit limitations; no hidden app-selection dialog | | Peer Device | Share/permission/preview commands proxy to the peer only after `computer_use_sharing_v1` negotiation; controller permission is not peer permission; old peers and CLI without GUI reject explicitly | | Detached Dispatch | Headless execution has no implied GUI capability or submitter UI dependency; require an explicitly negotiated target provider | @@ -465,6 +498,52 @@ Local fixtures and unsupported-host unit tests are not remote end-to-end evidenc ## Verification and acceptance +### Autonomous-control correction, 2026-10-13 + +The correction starts from `99acb6f04`. Focused checks passed: 51 Core +ComputerUse tool tests, two browser connection-hint tests, 163 Desktop Computer +Use tests, 33 control/permission contract tests, eight prompt-catalog tests, +64 isolated production-source context tests and 27 frontend settings/control +tests. Nine Desktop native tests and one isolated browser test are ignored in +those unit commands. `cargo build -p openbitfun-desktop`, `check:web`, the Web +production build, changed-page ESLint, generated capabilities, repository hygiene +and core boundaries passed. These suite counts overlap. + +The real macOS Tool roundtrip starts implicitly through application observation, +with no manual sharing API or separate input grant. It passed exact click, +Unicode/Return delivery, OCR verification, stale-reference rejection and Stop. +The strict native fixture starts background control explicitly, resolves a +running bundle, opens its explicit app path without activation, then passes +the covered button/text/scroll and five-action canvas/shortcut/emoji program. +Observer foreground/key/stacking, activation history, input history and the +physical cursor stay unchanged. Optional manual-preview tests run only after +the Agent sequence has passed. Two earlier strict runs exposed bundle-ID and +unregistered-path resolution failures; both were fixed before the final pass, +with no relaxed assertions or input replay. + +The settings page was inspected in a real macOS Tauri window using the newly +built frontend and an isolated test profile. The desktop-control section has +the standard toggle and two permission rows, with no app selector, grant panel, +Input Monitoring or executable diagnostics. The test profile does not exercise +a fresh OS permission denial/approval/restart cycle. + +Windows production modules, including the shell launcher, pass cross-compilation. +Windows and Linux native execution and all four remote scenarios were not rerun +for this correction. Prior platform evidence below does not prove the updated +end-to-end behavior. Cold application startup and arbitrary third-party app +activation behavior are not covered by the macOS already-running fixture. + +A wider Core `--lib computer_use` filter additionally selected the unchanged +`registry_exposes_controlhub_and_computer_use` test, whose product-registry +expectation failed under the minimal feature set: without `product-full`, the +default runtime selects the Agent Runtime baseline tool plan. The owning +`computer_use_tool::tests` suite passes. Full product assembly is covered by CI; +the registry test's narrow-feature mismatch remains unresolved. The older +`direct_desktop_policy_tests` filter matches no current +tests and is not counted as coverage. + +### Historical migration evidence, 2026-10-12 + The 2026-10-12 refactor started from `f5b6d3ab2`. Focused checks passed: 72 Core Computer Use tests, 160 Desktop tests, 33 contract tests, five runtime-evidence tests, eight prompt-catalog tests, 63 isolated production-source context tests, diff --git a/scripts/test-macos-input-controls.mjs b/scripts/test-macos-input-controls.mjs index ff4da167a4..c3255872c3 100644 --- a/scripts/test-macos-input-controls.mjs +++ b/scripts/test-macos-input-controls.mjs @@ -75,6 +75,7 @@ try { const hostReady = join(directory, 'host-ready'); const testEnvironment = { ...process.env, OPENBITFUN_INPUT_FIXTURE_PID: pid, + OPENBITFUN_INPUT_FIXTURE_APP: join(directory, 'OpenBitFun Input Target.app'), OPENBITFUN_INPUT_FIXTURE_RESULT: result, OPENBITFUN_INPUT_OBSERVER_RESULT: `${result}.observer`, OPENBITFUN_INPUT_HOST_READY: hostReady, }; diff --git a/scripts/test-windows-computer-use-native.mjs b/scripts/test-windows-computer-use-native.mjs index 071d29bf83..b770fdaf48 100644 --- a/scripts/test-windows-computer-use-native.mjs +++ b/scripts/test-windows-computer-use-native.mjs @@ -44,7 +44,7 @@ windows = { version = "0.61", features = [ "Win32_System_Com", "Win32_System_Ole", "Win32_System_Variant", "Win32_System_LibraryLoader", "Win32_System_Threading", "Win32_System_WinRT", "Win32_System_WinRT_Direct3D11", "Win32_System_WinRT_Graphics_Capture", "Win32_UI_Accessibility", "Win32_UI_Input_KeyboardAndMouse", - "Win32_UI_WindowsAndMessaging" + "Win32_UI_WindowsAndMessaging", "Win32_UI_Shell" ] } `); await copyFile(join(root, 'Cargo.lock'), join(dir, 'Cargo.lock')); @@ -75,6 +75,7 @@ mod computer_use { mod desktop_host { use crate::util::errors::{OpenBitFunError, OpenBitFunResult}; #[path = ${source('src/apps/desktop/src/computer_use/desktop_host/background_guard.rs')}] mod background_guard; + #[path = ${source('src/apps/desktop/src/computer_use/desktop_host/windows_launch.rs')}] mod windows_launch; } } `); diff --git a/src/apps/desktop/src/computer_use/AGENTS.md b/src/apps/desktop/src/computer_use/AGENTS.md index e76bc8b004..139f44a280 100644 --- a/src/apps/desktop/src/computer_use/AGENTS.md +++ b/src/apps/desktop/src/computer_use/AGENTS.md @@ -180,10 +180,13 @@ legacy payloads, advertise `observation-id-v1`, and reject explicit stale IDs before input. New observations invalidate prior AX/view indices even when the content digest is unchanged. Do not replace namespace identity with a digest. -New native fixture setup must use the production target inventory, selected-share -capture and explicit grant. Tool `start_control` only requests sharing. Do not -bypass that gate to make a native fixture pass. Permission probes must never -prompt, and remembered grants must never select or share a window automatically. +Agent native fixtures must discover and address their target through production +tools without manual share selection or a separate input grant. Cover explicit +`start_control` and implicit first-action startup. Optional manual-share fixtures +exercise target inventory, observe-first capture and grants separately; they must +not hide an Agent regression by performing those UI steps in fixture setup. +Permission probes never prompt. Desktop-state witnesses record concurrent human +activity without revoking Agent control or replaying input. The Web preview's deterministic cursor is verified against Rust-generated `computer-use-rust` samples retained in the repository. The reference source is diff --git a/src/apps/desktop/src/computer_use/control_session.rs b/src/apps/desktop/src/computer_use/control_session.rs index 5b15700834..dafeff47ac 100644 --- a/src/apps/desktop/src/computer_use/control_session.rs +++ b/src/apps/desktop/src/computer_use/control_session.rs @@ -151,6 +151,7 @@ fn projected(r: &Resource) -> ControlSnapshot { s.supported = true; s.capabilities = vec![ "control-session-v1".into(), + "agent-directed-control-v1".into(), "stop-control".into(), "action-feedback".into(), "observation-scope-v1".into(), diff --git a/src/apps/desktop/src/computer_use/control_session/admission.rs b/src/apps/desktop/src/computer_use/control_session/admission.rs index a7a890ba9c..647152c135 100644 --- a/src/apps/desktop/src/computer_use/control_session/admission.rs +++ b/src/apps/desktop/src/computer_use/control_session/admission.rs @@ -86,7 +86,8 @@ impl Admission { .map_or(0, |sharing| sharing.authorization_revision) } - /// A tool can request sharing, but cannot select a window or grant input. + /// Optional manual sharing has its own selection and input grant. Normal + /// Agent sessions use begin_start/admit and never enter this workflow. pub(super) fn request_share(&mut self, owner: &str, mode: ControlMode) -> Result<(), String> { if self.snapshot.owner.as_deref() != Some(owner) || self.busy || self.cleaning { return Err("[CONTROL_BUSY] Another task or action owns desktop control".into()); @@ -401,7 +402,10 @@ impl Admission { } let initial = self.snapshot.state.is_empty() || self.snapshot.state == "idle"; if self.snapshot.state == "awaiting_share" { - return Err("[SHARE_REQUIRED] Select an App in the Computer Use sharing panel on the executing Desktop or its connected peer controller. Mobile/IM/headless surfaces cannot select a native window; this call is not waiting for a hidden dialog.".into()); + if self.snapshot.owner.as_deref() != Some(owner) { + return Err("[CONTROL_BUSY] Desktop control is reserved by another session".into()); + } + return Err("[SHARE_REQUIRED] This optional manual share has no target. Agent tasks use start_control to select applications through their tool requests.".into()); } if !initial && !self.active() { return Err("[CONTROL_STOPPED] Use start_control to begin a new session".into()); @@ -415,7 +419,8 @@ impl Admission { self.snapshot.mode }; if self.snapshot.sharing.is_some() - && (policy.foreground && !policy.target || matches!(action, "clipboard_set")) + && (policy.foreground && !policy.target + || matches!(action, "clipboard_set" | "open_app")) { return Err("[SHARED_TARGET_ONLY] Window sharing does not authorize global clipboard, launching or scripts".into()); } @@ -471,6 +476,24 @@ impl Admission { mod tests { use super::*; #[test] + fn agent_selects_targets_and_operates_without_a_manual_share() { + let mut state = Admission::::default(); + let observe = state.admit("task", "get_app_state", 1).unwrap(); + assert_eq!(state.snapshot.mode, ControlMode::Background); + assert!(state.snapshot.sharing.is_none()); + state.bind("pid:1/window:10".into()).unwrap(); + state.release(observe, true); + let input = state.admit("task", "app_type_text", 1).unwrap(); + assert!(state.check_input(input).is_ok()); + state.release(input, true); + let next = state.admit("task", "get_app_state", 1).unwrap(); + state.bind("pid:2/window:20".into()).unwrap(); + assert_eq!(state.snapshot.target_revision, 2); + state.release(next, true); + let launch = state.admit("task", "open_app", 1).unwrap(); + assert!(state.check_input(launch).is_ok()); + } + #[test] fn sharing_requires_selection_capture_and_a_separate_revocable_grant() { let mut state = running(); let generation = state.snapshot.generation; diff --git a/src/apps/desktop/src/computer_use/desktop_host/background_guard.rs b/src/apps/desktop/src/computer_use/desktop_host/background_guard.rs index 5da92f69a3..e612496f07 100644 --- a/src/apps/desktop/src/computer_use/desktop_host/background_guard.rs +++ b/src/apps/desktop/src/computer_use/desktop_host/background_guard.rs @@ -51,47 +51,34 @@ pub(super) async fn run( return input.await; } let scope = ObservationScope::from_control(&control); - let before = capture().await.map_err(|error| { - OpenBitFunError::tool(format!( - "[BACKGROUND_OBSERVATION_UNAVAILABLE] No input sent: {error}" - )) - })?; - let result = input.await; - // Once an operation may have been submitted, any failure must preserve - // ambiguity. A fallback click or a foreground repair can duplicate effects. - let after = capture().await; - match after { - Ok(after) => { - let observation = evidence(&before, &after, control.sequence); - let changed = !observation.foreground_unchanged - || observation.cursor_unchanged == Some(false) - || observation.window_order_unchanged == Some(false); - crate::computer_use::control_session::record_background_observation( - &scope, - observation, - ); - if !changed { - return result; - } - revoke(&scope, "background_desktop_changed"); - Err(OpenBitFunError::tool(format!("[BACKGROUND_INTERFERENCE] Desktop focus, physical pointer or window ordering changed during the operation; control was revoked. Cause may be application behavior or concurrent user/system input. Effects may already have occurred. No restoration or retry was sent. Operation error: {}", result.err().map(|error| error.to_string()).unwrap_or_else(|| "none".into())))) - } - Err(error) => { - revoke(&scope, "background_observation_failed"); - Err(OpenBitFunError::tool(format!("[BACKGROUND_OBSERVATION_FAILED] Could not verify the desktop after input; control was revoked. Effects may already have occurred; observe before deciding another action. {error}"))) - } + let (result, observation) = observe_operation(input, capture, control.sequence).await; + if let Some(observation) = observation { + crate::computer_use::control_session::record_background_observation(&scope, observation); } + result } -fn revoke(scope: &ObservationScope, reason: &str) { - if let Some(target) = &scope.target { - crate::computer_use::control_session::native_stopped_target( - scope.generation, - scope.target_revision, - target, - reason, - ); - } +/// Global desktop changes cannot be attributed to this action: the user is +/// allowed to keep working while directed input runs. Witnesses are diagnostic +/// evidence, never admission or revocation. Native target/lease checks still own +/// input validity. Preserve the action's exact outcome and never replay it. +async fn observe_operation( + input: impl std::future::Future>, + mut read: F, + sequence: u64, +) -> (OpenBitFunResult, Option) +where + F: FnMut() -> Fut, + Fut: std::future::Future>, +{ + let before = read().await; + let result = input.await; + let after = read().await; + let observation = before + .ok() + .zip(after.ok()) + .map(|(before, after)| evidence(&before, &after, sequence)); + (result, observation) } #[cfg(any(target_os = "macos", target_os = "windows"))] @@ -203,6 +190,49 @@ mod tests { activation_epoch: Some(4), } } + #[tokio::test] + async fn concurrent_human_input_does_not_cancel_or_replay_directed_input() { + let before = witness(&["human", "target"]); + let mut after = witness(&["other", "target", "human"]); + after.foreground = vec!["other".into()]; + after.pointer = Some((900.0, 500.0)); + after.activation_epoch = Some(5); + let mut witnesses = [Ok(before), Ok(after)].into_iter(); + let mut calls = 0; + let (result, evidence) = observe_operation( + async { + calls += 1; + Ok("delivered") + }, + || std::future::ready(witnesses.next().unwrap()), + 7, + ) + .await; + assert_eq!(result.unwrap(), "delivered"); + assert_eq!(calls, 1); + let evidence = evidence.unwrap(); + assert!(!evidence.foreground_unchanged); + assert_eq!(evidence.cursor_unchanged, Some(false)); + } + #[tokio::test] + async fn missing_diagnostics_preserve_the_native_operation_result() { + let mut calls = 0; + let (result, evidence) = observe_operation( + async { + calls += 1; + Err::<(), _>(OpenBitFunError::tool("native target closed")) + }, + || std::future::ready(Err("desktop witness unavailable".into())), + 8, + ) + .await; + assert_eq!( + result.unwrap_err().to_string(), + OpenBitFunError::tool("native target closed").to_string() + ); + assert_eq!(calls, 1); + assert!(evidence.is_none()); + } #[test] fn detects_raise_pointer_and_transient_activation_without_blame_or_repair() { let before = witness(&["human", "target", "other"]); diff --git a/src/apps/desktop/src/computer_use/desktop_host/mod.rs b/src/apps/desktop/src/computer_use/desktop_host/mod.rs index 0785c89516..beb1f8fa24 100644 --- a/src/apps/desktop/src/computer_use/desktop_host/mod.rs +++ b/src/apps/desktop/src/computer_use/desktop_host/mod.rs @@ -7,6 +7,8 @@ mod observation; pub(crate) mod shared_preview; pub(crate) mod sharing; mod state; +#[cfg(target_os = "windows")] +mod windows_launch; use state::ComputerUseSessionMutableState; #[cfg(any(target_os = "macos", target_os = "windows"))] use state::{CachedInteractiveView, CachedVisualMarkView}; @@ -72,7 +74,7 @@ const OSASCRIPT_TIMEOUT_MS: u64 = 10_000; fn run_osascript_bounded(script: &str, timeout_ms: u64) -> std::io::Result { use std::process::Stdio; - let mut child = std::process::Command::new("/usr/bin/osascript") + let mut child = openbitfun_core::util::process_manager::create_command("/usr/bin/osascript") .args(["-e", script]) .stdin(Stdio::null()) .stdout(Stdio::piped()) @@ -447,13 +449,16 @@ impl DesktopComputerUseHost { /// running with none — the usual state for an Electron client the user /// closed earlier. The result was `success: true` with an empty screen. /// - /// So: resolve the bundle id via LaunchServices, activate, poll for a - /// window, and re-open the bundle when the poll comes up empty. + /// Resolve the bundle identity, open with the session's activation policy, + /// poll for a window, and re-open the bundle when the poll comes up empty. #[cfg(target_os = "macos")] fn open_app_macos( name: String, + background: bool, ) -> OpenBitFunResult { - use crate::computer_use::macos_bg_input::running_app_identity_macos; + use crate::computer_use::macos_bg_input::{ + pid_for_bundle_id_macos, running_app_identity_macos, + }; use openbitfun_core::agentic::tools::computer_use_host::OpenAppResult; let failure = |err: String| OpenAppResult { @@ -467,22 +472,66 @@ impl DesktopComputerUseHost { launch_path: None, }; - // `id of application "X"` asks LaunchServices to resolve the name the - // same way `tell application "X"` will, so the bundle id we report is - // guaranteed to describe the app we are about to activate. - let bundle_id = run_osascript_bounded( - &format!("id of application {}", applescript_quote(&name)), - OSASCRIPT_TIMEOUT_MS, - ) - .ok() - .filter(|o| o.status.success()) - .map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()) - .filter(|s| !s.is_empty()); - - let activate = match run_osascript_bounded( - &format!("tell application {} to activate", applescript_quote(&name)), - OSASCRIPT_TIMEOUT_MS, - ) { + // Discovery returns bundle identifiers as well as names. A running + // bundle need not be registered in LaunchServices (for example a + // freshly built app), and an existing window needs no launch at all. + let running_pid = pid_for_bundle_id_macos(&name); + if background { + if let Some(pid) = running_pid { + let window_count = crate::computer_use::macos_ax_ui::window_count_for_pid(pid); + if window_count.is_some_and(|count| count > 0) { + let (process_name, bundle_id) = + running_app_identity_macos(pid).unwrap_or((None, None)); + return Ok(OpenAppResult { + app_name: name, + success: true, + process_id: Some(pid), + error_message: None, + bundle_id, + process_name, + window_count, + launch_path: Some("already_running".into()), + }); + } + } + } + let bundle_id = running_pid.map(|_| name.clone()).or_else(|| { + ["application", "application id"] + .into_iter() + .find_map(|specifier| { + run_osascript_bounded( + &format!("id of {specifier} {}", applescript_quote(&name)), + OSASCRIPT_TIMEOUT_MS, + ) + .ok() + .filter(|o| o.status.success()) + .map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()) + .filter(|s| !s.is_empty()) + }) + }); + + crate::computer_use::control_session::input_allowed().map_err(OpenBitFunError::tool)?; + let opened = if background { + let mut command = + openbitfun_core::util::process_manager::create_command("/usr/bin/open"); + command.arg("-g"); + if std::path::Path::new(&name).is_absolute() { + // A caller-provided app path stays authoritative even before + // LaunchServices has registered that bundle identifier. + command.args(["-a", &name]); + } else if let Some(bundle) = bundle_id.as_deref() { + command.args(["-b", bundle]); + } else { + command.args(["-a", &name]); + } + command.output() + } else { + run_osascript_bounded( + &format!("tell application {} to activate", applescript_quote(&name)), + OSASCRIPT_TIMEOUT_MS, + ) + }; + let activate = match opened { Ok(out) => out, // A timeout is a real outcome, not an internal error: the app is // installed but not answering. Report it as a failed launch the @@ -490,7 +539,7 @@ impl DesktopComputerUseHost { Err(e) if e.kind() == std::io::ErrorKind::TimedOut => { return Ok(failure(format!( "'{}' did not respond to activation within {}s — it may be hung or showing a modal dialog. \ -Check the app directly, or ask the user to bring it up.", +Inspect its process and current application state before choosing another operation.", name, OSASCRIPT_TIMEOUT_MS / 1000 ))); @@ -503,7 +552,11 @@ Check the app directly, or ask the user to bring it up.", )); } - let mut launch_path = "activate"; + let mut launch_path = if background { + "open_background" + } else { + "activate" + }; // Resolving by bundle id beats "whoever is frontmost right now" — // activation is asynchronous, so the frontmost app during the first // poll ticks is often still the previous one. @@ -515,13 +568,24 @@ Check the app directly, or ask the user to bring it up.", // alone never triggers. if window_count == Some(0) { if let Some(bid) = bundle_id.as_deref() { - let reopened = std::process::Command::new("/usr/bin/open") + crate::computer_use::control_session::input_allowed() + .map_err(OpenBitFunError::tool)?; + let mut command = + openbitfun_core::util::process_manager::create_command("/usr/bin/open"); + if background { + command.arg("-g"); + } + let reopened = command .args(["-b", bid]) .output() .map(|o| o.status.success()) .unwrap_or(false); if reopened { - launch_path = "reopen_bundle"; + launch_path = if background { + "reopen_background" + } else { + "reopen_bundle" + }; pid = Self::poll_for_app_pid(Some(bid), OPEN_APP_WINDOW_WAIT_MS).or(pid); window_count = Self::poll_for_window(pid, OPEN_APP_WINDOW_WAIT_MS); } @@ -544,17 +608,15 @@ Check the app directly, or ask the user to bring it up.", }) } - /// Poll until the app owning `bundle_id` is running (or the frontmost app - /// settles, when no bundle id could be resolved). Returns its pid. + /// Poll for the launched bundle. Never report the user's foreground app + /// as the launch target when application identity could not be resolved. #[cfg(target_os = "macos")] fn poll_for_app_pid(bundle_id: Option<&str>, budget_ms: u64) -> Option { - use crate::computer_use::macos_bg_input::{frontmost_pid_macos, pid_for_bundle_id_macos}; + use crate::computer_use::macos_bg_input::pid_for_bundle_id_macos; + let bundle_id = bundle_id?; let deadline = std::time::Instant::now() + std::time::Duration::from_millis(budget_ms); loop { - let found = match bundle_id { - Some(bid) => pid_for_bundle_id_macos(bid), - None => frontmost_pid_macos(), - }; + let found = pid_for_bundle_id_macos(bundle_id); if found.is_some() { return found; } @@ -919,8 +981,30 @@ impl ComputerUseHost for DesktopComputerUseHost { owner: &str, request: openbitfun_core::agentic::tools::computer_use_host::ControlStartRequest, ) -> OpenBitFunResult { - crate::computer_use::control_session::request_share(owner, request.mode) - .map_err(OpenBitFunError::tool) + let snapshot = crate::computer_use::control_session::start(owner, request.mode) + .map_err(OpenBitFunError::tool)?; + let scope = + openbitfun_agent_tools::computer_use_control::ObservationScope::from_control(&snapshot); + drop(self.state_for_scope(&scope)?); + // Background AT-SPI actions need no portal or window picker. The + // compositor's consent is required only for a portal capture/seat session. + #[cfg(target_os = "linux")] + if request.mode != openbitfun_agent_tools::computer_use_control::ControlMode::Background { + if let Err(error) = crate::computer_use::linux_control::start_session( + request.mode + == openbitfun_agent_tools::computer_use_control::ControlMode::Foreground, + ) + .await + { + let _ = crate::computer_use::control_session::stop_checked( + Some(owner), + Some(snapshot.generation), + &error, + ); + return Err(OpenBitFunError::tool(error)); + } + } + Ok(crate::computer_use::control_session::snapshot()) } fn control_snapshot( @@ -964,14 +1048,6 @@ impl ComputerUseHost for DesktopComputerUseHost { if !policy.lease { return Ok(None); } - let current = crate::computer_use::control_session::snapshot(); - if current.state == "idle" { - crate::computer_use::control_session::request_share( - owner, - openbitfun_agent_tools::computer_use_control::ControlMode::Background, - ) - .map_err(OpenBitFunError::tool)?; - } crate::computer_use::control_session::acquire(owner, action) .map(Some) .map_err(OpenBitFunError::tool) @@ -1232,11 +1308,16 @@ impl ComputerUseHost for DesktopComputerUseHost { ) -> OpenBitFunResult { use openbitfun_core::agentic::tools::computer_use_host::OpenAppResult; let name = app_name.to_string(); + crate::computer_use::control_session::input_allowed().map_err(OpenBitFunError::tool)?; + let background = crate::computer_use::control_session::snapshot().mode + == openbitfun_agent_tools::computer_use_control::ControlMode::Background; #[cfg(target_os = "macos")] { let result = crate::computer_use::control_session::spawn_blocking( - move || -> OpenBitFunResult { Self::open_app_macos(name) }, + move || -> OpenBitFunResult { + Self::open_app_macos(name, background) + }, ) .await .map_err(|e| OpenBitFunError::tool(e.to_string()))??; @@ -1247,30 +1328,7 @@ impl ComputerUseHost for DesktopComputerUseHost { { let result = crate::computer_use::control_session::spawn_blocking( move || -> OpenBitFunResult { - let output = openbitfun_core::util::process_manager::create_command("cmd") - .args(["/c", "start", "", &name]) - .output() - .map_err(|e| OpenBitFunError::tool(format!("open_app: {}", e)))?; - Ok(OpenAppResult { - app_name: name, - success: output.status.success(), - process_id: None, - error_message: if output.status.success() { - None - } else { - Some(String::from_utf8_lossy(&output.stderr).trim().to_string()) - }, - // `start` hands off to the shell and returns immediately - // without telling us what it launched, so there is no pid - // to resolve identity or window count from. Left as `None` - // (the "not measured" value) rather than faked — the model - // reads `window_count: Some(0)` as a definite windowless - // app and would act on it. - bundle_id: None, - process_name: None, - window_count: None, - launch_path: Some("shell_start".to_string()), - }) + windows_launch::open(name, background) }, ) .await @@ -1280,12 +1338,17 @@ impl ComputerUseHost for DesktopComputerUseHost { #[cfg(target_os = "linux")] { + if background { + return Err(OpenBitFunError::tool("[BACKGROUND_LAUNCH_UNSUPPORTED] This Linux host cannot request application launch without compositor activation. Already-running applications remain available through list_apps and AT-SPI.")); + } let result = crate::computer_use::control_session::spawn_blocking( move || -> OpenBitFunResult { - let output = std::process::Command::new("xdg-open") + let output = openbitfun_core::util::process_manager::create_command("xdg-open") .arg(&name) .output() - .or_else(|_| std::process::Command::new(&name).output()) + .or_else(|_| { + openbitfun_core::util::process_manager::create_command(&name).output() + }) .map_err(|e| OpenBitFunError::tool(format!("open_app: {}", e)))?; Ok(OpenAppResult { app_name: name, diff --git a/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs b/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs index 6de4dfc527..1d1db62229 100644 --- a/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs +++ b/src/apps/desktop/src/computer_use/desktop_host/native_control_roundtrip_tests.rs @@ -8,7 +8,11 @@ use std::sync::Arc; const OWNER: &str = "native-control-roundtrip-fixture"; async fn select_and_grant_fixture(host: &DesktopComputerUseHost, pid: i32, window: Option) { - let pending = host.control_snapshot(); + let pending = crate::computer_use::control_session::request_share( + OWNER, + openbitfun_agent_tools::computer_use_control::ControlMode::Background, + ) + .unwrap(); assert_eq!(pending.state, "awaiting_share"); assert!(crate::computer_use::control_session::acquire(OWNER, "app_click").is_err()); let target = crate::computer_use::share_targets() @@ -174,14 +178,8 @@ pub(crate) async fn run() { context.primary_model_facts.api_format = "anthropic".into(); let tool = ComputerUseTool::new(); let _cleanup = StopOnExit; - let start = call( - &tool, - &context, - json!({"action":"start_control", "mode":"background"}), - ) - .await; - let generation = start["generation"].as_u64().expect("control generation"); - select_and_grant_fixture(&host, pid, None).await; + // Reproduce ordinary Agent usage: discovery followed by the first app + // observation starts background control without any UI sharing calls. let apps = call(&tool, &context, json!({"action":"list_apps"})).await; assert!( apps.to_string().contains(&pid.to_string()), @@ -193,6 +191,10 @@ pub(crate) async fn run() { json!({"action":"get_app_state", "app":{"pid":pid}}), ) .await; + let control = host.control_snapshot(); + assert_eq!(control.state, "active"); + assert!(control.sharing.is_none()); + let generation = control.generation; let old_observation_id = observation["app_state"]["observation_id"] .as_str() .expect("scoped observation identity") @@ -358,7 +360,35 @@ async fn run_covered_semantic() { json!({"action":"start_control","mode":"background"}), ) .await; - select_and_grant_fixture(&host, pid, initial["window_id"].as_u64()).await; + assert_eq!(host.control_snapshot().state, "active"); + assert!(host.control_snapshot().sharing.is_none()); + let opened = call( + &tool, + &context, + json!({ + "action":"open_app", "app_name":"dev.openbitfun.input-fixture.target" + }), + ) + .await; + assert_eq!(opened["success"], true); + assert_eq!(opened["process_id"], pid); + assert_eq!( + crate::computer_use::macos_bg_input::frontmost_pid_macos(), + foreground + ); + let opened_by_path = call( + &tool, + &context, + json!({"action":"open_app", "app_name":std::env::var("OPENBITFUN_INPUT_FIXTURE_APP").unwrap()}), + ) + .await; + assert_eq!(opened_by_path["success"], true); + assert_eq!(opened_by_path["process_id"], pid); + assert_eq!(opened_by_path["launch_path"], "open_background"); + assert_eq!( + crate::computer_use::macos_bg_input::frontmost_pid_macos(), + foreground + ); let observation = call( &tool, &context, @@ -542,6 +572,9 @@ async fn run_covered_semantic() { ); // Human preview input uses a separate host instance but the same native // sharing resource, lease and background guard as the Agent. + // Exercise that optional API separately, after the entire Agent task has + // completed without a share selection or per-application grant. + select_and_grant_fixture(&host, pid, initial["window_id"].as_u64()).await; use crate::computer_use::shared_preview::{self, SharedInputRequest, SharedPreviewRequest}; let shared = host.control_snapshot(); let scope = || SharedPreviewRequest { diff --git a/src/apps/desktop/src/computer_use/desktop_host/windows_launch.rs b/src/apps/desktop/src/computer_use/desktop_host/windows_launch.rs new file mode 100644 index 0000000000..ba593e3177 --- /dev/null +++ b/src/apps/desktop/src/computer_use/desktop_host/windows_launch.rs @@ -0,0 +1,66 @@ +//! Literal Shell launch with an explicit activation policy; no cmd.exe parsing. +use openbitfun_core::agentic::tools::computer_use_host::OpenAppResult; +use openbitfun_core::util::errors::{OpenBitFunError, OpenBitFunResult}; +use windows::core::{w, HSTRING}; +use windows::Win32::Foundation::RPC_E_CHANGED_MODE; +use windows::Win32::System::Com::{ + CoInitializeEx, CoUninitialize, COINIT_APARTMENTTHREADED, COINIT_DISABLE_OLE1DDE, +}; +use windows::Win32::UI::{ + Shell::ShellExecuteW, + WindowsAndMessaging::{SW_SHOWNOACTIVATE, SW_SHOWNORMAL}, +}; + +pub(super) fn open(name: String, background: bool) -> OpenBitFunResult { + crate::computer_use::control_session::input_allowed().map_err(OpenBitFunError::tool)?; + if name.trim().is_empty() || name.contains('\0') { + return Err(OpenBitFunError::tool( + "[APP_REQUIRED] A valid application name or path is required", + )); + } + // Shell handlers may require COM. A blocking worker can already own an + // apartment; balance only the initialization performed by this call. + let initialized = + unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE) }; + if initialized.is_err() && initialized != RPC_E_CHANGED_MODE { + return Err(OpenBitFunError::tool(format!( + "Windows application launch could not initialize COM: {initialized:?}" + ))); + } + let result = unsafe { + ShellExecuteW( + None, + w!("open"), + &HSTRING::from(&name), + None, + None, + if background { + SW_SHOWNOACTIVATE + } else { + SW_SHOWNORMAL + }, + ) + } + .0 as isize; + if initialized.is_ok() { + unsafe { CoUninitialize() }; + } + Ok(OpenAppResult { + app_name: name, + success: result > 32, + process_id: None, + error_message: (result <= 32) + .then(|| format!("Windows application launch failed: {result}")), + bundle_id: None, + process_name: None, + window_count: None, + launch_path: Some( + if background { + "shell_no_activate" + } else { + "shell_open" + } + .into(), + ), + }) +} diff --git a/src/crates/assembly/agent-content/prompts/agents/claw_mode.md b/src/crates/assembly/agent-content/prompts/agents/claw_mode.md index 970faef188..58afba326b 100644 --- a/src/crates/assembly/agent-content/prompts/agents/claw_mode.md +++ b/src/crates/assembly/agent-content/prompts/agents/claw_mode.md @@ -22,7 +22,7 @@ For browser and web-page work, route in this order: 1. Only opening, showing, previewing, or displaying a URL for the user (no page reading, no interaction): use `ControlHub` with `domain: "browser"`, `action: "open_builtin"`, `params: { url }`. The page renders in OpenBitFun's built-in right-side browser panel. Do not delegate this to a `ComputerUse` sub-agent and do not call `connect`/`navigate` for it. 2. Reading page content that does not require the user's login state: use `WebFetch`. -3. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). On Chrome 144+ and Edge, `connect` requests access to the currently running real profile; for one-time setup, ask the user to click **Enable default CDP** in OpenBitFun Settings > Browser control, enable Remote debugging in the browser-owned page, and approve OpenBitFun. Other supported Chromium browsers reuse a real-profile endpoint when available and otherwise use OpenBitFun's persistent managed profile. +3. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). Attempt the connection yourself. If connection setup is unavailable, continue through `ComputerUse` application observations and input when available instead of requiring the user to visit Settings or enable remote debugging. Respect browser-owned consent and explicit connection denial; if no authorized route remains, report that concrete limitation. 4. Native desktop apps, browser chrome, and OS dialogs in any browser: use the `ComputerUse` tool directly when available. Prefer the browser interface for web content; browser process identity does not prohibit desktop control. Do not use `ControlHub` for local computer, operating-system, or desktop UI work. Desktop and system actions have moved to the dedicated `ComputerUse` tool/agent. This includes screenshots, OCR, mouse, keyboard, app state, app launching, opening local files and non-http(s) URLs through the OS, clipboard access, OS facts, and local scripts. diff --git a/src/crates/assembly/agent-content/prompts/agents/computer_use_mode.md b/src/crates/assembly/agent-content/prompts/agents/computer_use_mode.md index a972e0f4b3..c6f6f2a17d 100644 --- a/src/crates/assembly/agent-content/prompts/agents/computer_use_mode.md +++ b/src/crates/assembly/agent-content/prompts/agents/computer_use_mode.md @@ -25,9 +25,9 @@ Choose the control surface that owns the task: # Control Session And Target -For an app task, call `start_control` with `mode: "background"`. On hosts advertising `app-sharing-v1`, this requests a user selection; it does not grant control. When the result is `awaiting_share` or `SHARE_REQUIRED`, explain that the user must select the App window in the executing Desktop's sharing panel (or its connected peer controller). Stop this attempt until the user completes that choice. Sharing starts in observation mode, and input needs a separate user grant or an exact remembered application grant. Never invoke permission, selection or grant UI commands on the user's behalf. Mobile/IM and headless callers that cannot select a native window must receive an explicit unsupported/selection-required result, not wait for a hidden dialog. +For an app task, call `start_control` with `mode: "background"`, identify the application with `list_apps`, and choose its `app` selector yourself. Starting control enables the Agent to observe and operate applications within the user's task; it does not require a user-selected share, a per-app grant, or a trip to Settings. The optional `app-sharing-v1` interface is for manual sharing clients, not an Agent prerequisite. A first app action may also start the background session automatically. -After sharing is established, use the bound application from `control_status` and the same `app` selector for `get_app_state` and `app_*` actions, for example `{"pid":421}`. Other windows, including windows in the same application, are outside this selected capture and input scope. Keep the session active across observations, actions and verification; do not stop and restart between calls. Use `control_status` to inspect the owner, mode and target, and `stop_control` when the task is finished. A system stop requires a new user-selected share. Older hosts without `app-sharing-v1` retain their advertised control contract; never assume the new grant flow exists remotely. +Use the same `app` selector for `get_app_state` and `app_*` actions, for example `{"pid":421}`. When the task moves to another application, observe that application with its explicit selector; target changes invalidate old observations and coordinates. Keep the session active across observations, actions and verification; do not stop and restart between calls. Use `control_status` to inspect the owner, mode and target, and `stop_control` when the task is finished. After a target closes or the native capture ends, re-discover the intended target and start a new session yourself when the task still requires it. Respect a user stop, task cancellation, or permission denial; do not restart around it. Older execution hosts may retain a manual-sharing contract: report that host incompatibility instead of asking the user to complete the task's UI steps or claiming the updated behavior exists remotely. The controlled target can be behind another application. The human's foreground application, display and cursor are metadata, not a reason to switch the task's target. An observation with no explicit selector follows the bound target when one exists. App-scoped actions require an explicit application identity from the observation. @@ -82,7 +82,7 @@ For websites and web apps, route in this order: 1. Only opening, showing, previewing, or displaying a URL for the user (no page reading, no interaction): use `ControlHub` with `domain: "browser"`, `action: "open_builtin"`, `params: { url }`. The page renders in OpenBitFun's built-in right-side browser panel. Do not call `connect`/`navigate` for this. 2. Reading page content that does not require the user's login state: use `WebFetch` when it is available. -3. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). On Chrome 144+ and Edge, ask the user to click **Enable default CDP** in OpenBitFun Settings > Browser control, enable Remote debugging in the browser-owned page, and approve OpenBitFun if prompted; this preserves the current profile's tabs and login state. Other supported Chromium browsers reuse a real-profile endpoint when available and otherwise use OpenBitFun's persistent managed profile. +3. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). Attempt the available connection action yourself. If the browser requires connection setup or refuses the connection, continue through `ComputerUse` application observations and input when available. Do not make the user visit OpenBitFun Settings, enable remote debugging, or operate the browser as a prerequisite to an ordinary task. Browser-owned consent may still require the user's response if no already-authorized interface can perform the task. 4. Native desktop apps, browser chrome, and OS dialogs in any browser: use `ComputerUse` desktop actions when available. Prefer the browser interface for web content; browser process identity does not prohibit desktop control. If `ControlHub` is unavailable or its browser interface cannot operate the required surface, use available `ComputerUse` capabilities with fresh observations and the authorized control scope. Do not claim browser-domain automation when that interface is unavailable. @@ -91,13 +91,13 @@ Use `ComputerUse` for browser chrome, OS dialogs, permission prompts, file picke # Safety And User Trust -Treat destructive actions, payments, purchases, account changes, sending messages, deleting data, permission changes, and security-sensitive settings as high-risk. Pause for user confirmation before final submission unless the user has explicitly authorized that exact action. +Treat destructive actions, payments, purchases, account changes, sending messages, deleting data, permission changes, and security-sensitive settings as high-risk. Pause for user confirmation before final submission unless the user has explicitly authorized that exact action. A direct request to perform an action authorizes the necessary ordinary UI steps. When the user asks you to compose and send specified information to a named recipient, composing that content is part of the request; do not add a second confirmation solely because you wrote the wording. Resolve the recipient and verify the content yourself. Ask only for genuinely missing information or an action outside the user's authorization. Existing explicit authorization remains valid across observation errors and tool retries for the same recipient, exact content and action. Do not ask the user to confirm the same unchanged submission again. A failed observation after submitted input requires checking delivery, not resending or repeating the confirmation. For chat and messaging apps, verify the recipient or conversation header and the exact message before sending. Clipboard byte equality is not evidence of text in the composer or delivery to the recipient. If Return does not visibly send, inspect the observed send control and current content; do not invent an explanation about the user's keyboard settings. Use app-scoped input such as `app_type_text` for Unicode content, then verify the resulting conversation state through the same target observation. Do not use shell scripts or AppleScript keystrokes to bypass the control session. -If permissions are missing, explain the needed OS permission or capability briefly and stop instead of improvising unsafe alternatives. +Complete the requested task through observation, action and verification without handing routine UI work back to the user. Recover from stale observations, changed targets, unavailable accessibility, and connection errors using the supported interfaces. An absent optional capability is not a reason to stop when another authorized route can finish the task. Request user involvement only for a genuine OS-enforced consent, credentials or other missing information, or a decision outside the request. Never claim an unsupported operation or an unverified submission succeeded. # Communication Style diff --git a/src/crates/assembly/agent-content/prompts/agents/cowork_mode.md b/src/crates/assembly/agent-content/prompts/agents/cowork_mode.md index d6c68dca78..c944a5fbbf 100644 --- a/src/crates/assembly/agent-content/prompts/agents/cowork_mode.md +++ b/src/crates/assembly/agent-content/prompts/agents/cowork_mode.md @@ -63,7 +63,7 @@ For browser and web-page work, route in this order: 1. Only opening, showing, previewing, or displaying a URL for the user (no page reading, no interaction): use `ControlHub` with `domain: "browser"`, `action: "open_builtin"`, `params: { url }`. 2. Reading page content that does not require the user's login state: use `WebFetch`. -3. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). On Chrome 144+ and Edge, ask the user to click **Enable default CDP** in OpenBitFun Settings > Browser control, enable Remote debugging in the browser-owned page, and approve OpenBitFun if prompted; this preserves the current profile's tabs and login state. Other supported Chromium browsers reuse a real-profile endpoint when available and otherwise use OpenBitFun's persistent managed profile. +3. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). Attempt the connection yourself. If connection setup is unavailable, continue through `ComputerUse` application observations and input when available instead of requiring the user to visit Settings or enable remote debugging. Respect browser-owned consent and explicit connection denial; if no authorized route remains, report that concrete limitation. 4. Native desktop apps, browser chrome, and OS dialogs in any browser: use the `ComputerUse` tool directly when available. Prefer the browser interface for web content; browser process identity does not prohibit desktop control. Do not use `ControlHub` for local computer, operating-system, or desktop UI work, and do not substitute a browser-automation skill for it. diff --git a/src/crates/assembly/agent-content/prompts/agents/general_purpose_agent.md b/src/crates/assembly/agent-content/prompts/agents/general_purpose_agent.md index e52ae297cd..cdb98c4b23 100644 --- a/src/crates/assembly/agent-content/prompts/agents/general_purpose_agent.md +++ b/src/crates/assembly/agent-content/prompts/agents/general_purpose_agent.md @@ -33,4 +33,6 @@ You are a general-purpose agent for OpenBitFun, a desktop AI IDE and agent runti - Avoid emojis. -For ComputerUse handoffs, preserve the original user's request and any relevant approval as quotations, separate from your proposed plan. Delegate the desired outcome, target, exact approved content and verification criteria; let the desktop agent select actions from current observations. Default to background app control. Do not add application activation, foreground takeover, global input or clipboard scripts to an ordinary app task. A request such as "control my computer and send a message" does not request foreground takeover. Confirmation of message content does not authorize a change of control mode, even if your preceding narration suggested taking over the mouse and keyboard. An agent-written plan is not evidence of user authorization. +For ComputerUse handoffs, preserve the original user's request and any relevant approval as quotations, separate from your proposed plan. Delegate the desired outcome, target, requested information, any user-supplied exact wording and verification criteria; let the desktop agent select actions from current observations. Default to background app control. Do not add application activation, foreground takeover, global input or clipboard scripts to an ordinary app task. A request such as "control my computer and send a message" does not request foreground takeover. Confirmation of message content does not authorize a change of control mode, even if your preceding narration suggested taking over the mouse and keyboard. An agent-written plan is not evidence of user authorization. + +Carry an authorized desktop task through app discovery, observation, input and verification yourself. Choose application selectors from `list_apps`; do not require the user to select or share an app or grant it separate input control. When asked to compose and send information, compose it, verify the recipient and content, and send without an extra approval solely for your wording. Recover through supported tool routes instead of handing routine UI steps to the user. Genuine OS-owned consent, missing credentials or information, explicit Stop and actions outside the request still require the appropriate user involvement. diff --git a/src/crates/assembly/agent-content/tests/prompt_catalog_contracts.rs b/src/crates/assembly/agent-content/tests/prompt_catalog_contracts.rs index 646919ea6a..cb575dc865 100644 --- a/src/crates/assembly/agent-content/tests/prompt_catalog_contracts.rs +++ b/src/crates/assembly/agent-content/tests/prompt_catalog_contracts.rs @@ -257,6 +257,10 @@ fn computer_use_prompt_preserves_background_observation_and_input_contract() { "\"screenshot_id\":\"capture-1\"", "Do not replace failed GUI observations or input with ad hoc AppleScript", "an unchanged digest does not prove failure", + "choose its `app` selector yourself", + "does not require a user-selected share", + "do not add a second confirmation solely because you wrote the wording", + "continue through `ComputerUse` application observations and input", ] { assert!( prompt.contains(required), @@ -275,6 +279,9 @@ fn computer_use_prompt_preserves_background_observation_and_input_contract() { "Drive hard-to-reach apps directly", "screenshot_unavailable: true", "verify `ax_state_digest` changed", + "Stop this attempt until the user completes that choice", + "A system stop requires a new user-selected share", + "ask the user to click **Enable default CDP**", ] { assert!( !prompt.contains(obsolete), @@ -288,6 +295,10 @@ fn computer_use_delegation_preserves_user_scope_in_parents_and_child() { for name in ["claw_mode", "standard_mode", "cowork_mode", "creative_mode"] { let prompt = agent_prompt(name).unwrap(); assert!(prompt.contains("{COMPUTER_USE_GUIDANCE}"), "{name}"); + assert!( + !prompt.contains("ask the user to click **Enable default CDP**"), + "{name}" + ); } let prompt = agent_prompt("general_purpose_agent").unwrap(); assert!( @@ -298,6 +309,8 @@ fn computer_use_delegation_preserves_user_scope_in_parents_and_child() { prompt.contains("Default to background app control"), "general_purpose_agent" ); + assert!(prompt.contains("do not require the user to select or share an app")); + assert!(prompt.contains("without an extra approval solely for your wording")); assert!( prompt.contains( "Confirmation of message content does not authorize a change of control mode" diff --git a/src/crates/assembly/core/src/agentic/agents/prompt_builder/prompt_builder_impl.rs b/src/crates/assembly/core/src/agentic/agents/prompt_builder/prompt_builder_impl.rs index cb7e0456e7..74b5b48fb4 100644 --- a/src/crates/assembly/core/src/agentic/agents/prompt_builder/prompt_builder_impl.rs +++ b/src/crates/assembly/core/src/agentic/agents/prompt_builder/prompt_builder_impl.rs @@ -42,7 +42,9 @@ const PLACEHOLDER_FILE_REFERENCES: &str = "{FILE_REFERENCES}"; const COMPUTER_USE_GUIDANCE: &str = r#"# Direct desktop work -For ComputerUse handoffs, preserve the original user's request and any relevant approval as quotations, separate from your proposed plan. Delegate the desired outcome, target, exact approved content and verification criteria; let the desktop agent select actions from current observations. Default to background app control. Do not add application activation, foreground takeover, global input or clipboard scripts to an ordinary app task. A request such as "control my computer and send a message" does not request foreground takeover. Confirmation of message content does not authorize a change of control mode, even if your preceding narration suggested taking over the mouse and keyboard. An agent-written plan is not evidence of user authorization. +For ComputerUse handoffs, preserve the original user's request and any relevant approval as quotations, separate from your proposed plan. Delegate the desired outcome, target, requested information, any user-supplied exact wording and verification criteria; let the desktop agent select actions from current observations. Default to background app control. Do not add application activation, foreground takeover, global input or clipboard scripts to an ordinary app task. A request such as "control my computer and send a message" does not request foreground takeover. Confirmation of message content does not authorize a change of control mode, even if your preceding narration suggested taking over the mouse and keyboard. An agent-written plan is not evidence of user authorization. + +Carry an authorized desktop task through app discovery, observation, input and verification yourself. Choose application selectors from `list_apps`; do not require the user to select or share an app or grant it separate input control. When asked to compose and send information, compose it, verify the recipient and content, and send without an extra approval solely for your wording. Recover through supported tool routes instead of handing routine UI steps to the user. Genuine OS-owned consent, missing credentials or information, explicit Stop and actions outside the request still require the appropriate user involvement. Use `ComputerUse` directly for native application and OS UI tasks when it appears in your current tool list. Keep the user's conversation and observations in this agent; a separate ComputerUse subagent is optional for independently delegated work, not a prerequisite for desktop control. If neither the tool nor an available ComputerUse subagent can handle the executing host, report the missing capability without local fallback. Default to background app control. diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs index daaf407739..6560a509b6 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs @@ -641,7 +641,7 @@ If neither accessibility nor built-in OCR exposes the required content, report t json!("Target from list_apps. Omit to keep the bound application."); properties["mode"]["description"] = json!("Default background. Foreground only for a user-requested takeover."); - properties["app_name"]["description"] = json!("Application name for open_app; launching may activate it and requires foreground control."); + properties["app_name"]["description"] = json!("Application name, bundle identifier or path for open_app. Background mode requests a launch without activation where the host supports it; then observe the returned target."); properties["steps"]["description"] = json!("Use app_batch when multiple inputs are already decided. One app, ordered steps, one final observation. Example assumes an observed search field where Return submits the query. Stop before selecting unseen results; batch changes neither input capability nor authorization. Never replay completed steps."); let observed_focus = if vision { json!({"kind":"image_xy","x":120,"y":80,"screenshot_id":"observed-frame"}) @@ -2548,8 +2548,8 @@ impl ComputerUseTool { let next_step = if windowless { Some(format!( "'{}' is running (PID {}) but owns no window, so there is nothing on screen to click. \ -The host already retried via `open -b`. Re-run `open_app`, or ask the user to open the app's main window (e.g. from its Dock icon). \ -Do not fall back to screen-coordinate clicks — there is no window to hit.", +Inspect the application's current state and supported reopen or menu actions, then continue the task. \ +Do not send screen-coordinate clicks without an observed window or report launch as task completion.", result.app_name, result .process_id @@ -2834,6 +2834,11 @@ mod tests { .await; assert!(visual.to_string().len() < legacy.to_string().len() / 2); for schema in [&visual, &textual] { + let launch = schema["properties"]["app_name"]["description"] + .as_str() + .unwrap(); + assert!(launch.contains("without activation")); + assert!(!launch.contains("requires foreground")); let actions = action_enum(schema); for current in [ "app_batch", diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs b/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs index 53a4bb9050..42385f5122 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs @@ -139,7 +139,7 @@ impl ControlHubTool { "Browser Control was disconnected by the user in Settings.", ) .with_hint( - "Ask the user to reconnect it in Settings > Desktop & browser before using external-browser actions again.", + "Respect the disconnected browser connection. If ComputerUse is available and desktop control is authorized for this task, use its app observations and app-scoped input to continue in the existing browser.", ) .with_hint( "Use target='builtin' for OpenBitFun's built-in browser when that surface is sufficient.", @@ -257,20 +257,12 @@ impl ControlHubTool { fn default_browser_connect_hints(kind: &BrowserKind, port: u16) -> Vec { match kind { BrowserKind::Chrome | BrowserKind::Edge => { - let setup_url = if matches!(kind, BrowserKind::Chrome) { - "chrome://inspect/#remote-debugging" - } else { - "edge://inspect/#remote-debugging" - }; vec![ format!( "{} can connect OpenBitFun to the current real profile, preserving its open tabs, cookies, extensions, and login state.", kind ), - format!( - "For one-time setup, ask the user to click Enable default CDP in OpenBitFun Settings > Browser control. OpenBitFun opens {}; enable Remote debugging there (the browser remembers this for normal future starts), then approve OpenBitFun's connection dialog in {}.", - setup_url, kind - ), + "If connection requires browser-owned setup or consent, use ComputerUse application observations and app-scoped input when available. Do not hand ordinary browser work back to the user or require a visit to Settings when an authorized desktop route can finish the task.".to_string(), "After approval, keep using browser.connect / snapshot / click / fill; OpenBitFun retains one guarded browser connection to avoid repeated prompts.".to_string(), ] } @@ -4316,14 +4308,14 @@ mod control_hub_tests { } #[test] - fn default_connect_hints_point_to_guarded_user_profile_not_raw_debug_port() { + fn default_connect_hints_offer_desktop_control_without_manual_setup() { let hints = ControlHubTool::default_browser_connect_hints(&BrowserKind::Chrome, 9222); let joined = hints.join(" | "); assert!( joined.contains("current real profile") - && joined.contains("chrome://inspect/#remote-debugging") - && joined.contains("approve"), - "hints must guide toward Chrome's guarded real-profile connection: {joined}" + && joined.contains("ComputerUse application observations") + && joined.contains("browser-owned setup or consent"), + "hints must preserve browser consent and offer available desktop control: {joined}" ); assert!( !joined.contains("--remote-debugging-port"), @@ -4332,15 +4324,19 @@ mod control_hub_tests { } #[test] - fn edge_connect_hints_use_its_guarded_real_profile_setup() { + fn edge_connect_hints_offer_the_same_autonomous_fallback() { let hints = ControlHubTool::default_browser_connect_hints(&BrowserKind::Edge, 9222); let joined = hints.join(" | "); assert!(joined.contains("current real profile"), "{joined}"); assert!( - joined.contains("edge://inspect/#remote-debugging"), + joined.contains("ComputerUse application observations"), + "{joined}" + ); + assert!( + joined.contains("browser-owned setup or consent"), "{joined}" ); - assert!(joined.contains("approve"), "{joined}"); + assert!(!joined.contains("ask the user to click"), "{joined}"); assert!(!joined.contains("--remote-debugging-port"), "{joined}"); } diff --git a/src/crates/execution/tool-contracts/src/computer_use_control.rs b/src/crates/execution/tool-contracts/src/computer_use_control.rs index d9fdcce927..c725e7f732 100644 --- a/src/crates/execution/tool-contracts/src/computer_use_control.rs +++ b/src/crates/execution/tool-contracts/src/computer_use_control.rs @@ -148,7 +148,8 @@ impl ControlActionPolicy { | "interactive_scroll" | "visual_click" => (false, false, true, true), "clipboard_set" => (false, false, false, true), - "open_app" | "run_apple_script" | "run_script" | "open_url" | "open_file" => { + "open_app" => (false, false, false, true), + "run_apple_script" | "run_script" | "open_url" | "open_file" => { (false, true, false, true) } "click" | "mouse_move" | "pointer_move_rel" | "scroll" | "drag" | "key_chord" @@ -220,11 +221,12 @@ mod tests { } #[test] fn seat_actions_never_claim_background_semantics() { - for action in ["paste", "drag", "click_target", "key_chord", "open_app"] { + for action in ["paste", "drag", "click_target", "key_chord"] { assert!(action_requires_foreground(action)); assert!(!action_is_observation(action)); } assert!(!action_requires_foreground("app_click")); + assert!(!action_requires_foreground("open_app")); assert!(action_is_observation("get_app_state")); } diff --git a/src/web-ui/src/app/scenes/settings/pages/shared/RuntimeSettings.test.tsx b/src/web-ui/src/app/scenes/settings/pages/shared/RuntimeSettings.test.tsx index f50b25a188..13feecdf4f 100644 --- a/src/web-ui/src/app/scenes/settings/pages/shared/RuntimeSettings.test.tsx +++ b/src/web-ui/src/app/scenes/settings/pages/shared/RuntimeSettings.test.tsx @@ -124,6 +124,50 @@ describe('Tool execution concurrency settings', () => { afterEach(async () => { await act(async () => root.unmount()); container.remove(); }); describe('Browser and desktop control settings', () => { + it('keeps detailed host permissions in the standard compact settings rows', async () => { + const original = mocks.invoke.getMockImplementation()!; + mocks.invoke.mockImplementation(async (command: string, ...args: unknown[]) => command === 'computer_use_get_status' + ? { computerUseEnabled: true, accessibilityGranted: true, screenCaptureGranted: true, + platformNote: 'Accessibility is available.', permissions: { process_id: 1234, + executable: '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/private/runtime', permissions: [{ kind: 'input_monitoring', state: 'granted' }] } } + : original(command, ...args)); + await render(); + for (const label of ['computerUse.enable', 'computerUse.accessibility', 'computerUse.screenCapture']) { + expect(Array.from(container.querySelectorAll('[data-openbitfun-part="row"]')) + .some(row => row.textContent?.includes(label))).toBe(true); + } + expect(container.textContent).not.toContain('computerSharing'); + expect(container.textContent).not.toContain('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/private/runtime'); + expect(container.textContent).not.toContain('input_monitoring'); + expect(button('computerUse.openSettings')).toBeUndefined(); + expect(container.textContent).not.toContain('computerUse.platformNotes.macos'); + expect(mocks.invoke.mock.calls.filter(([name]) => String(name).startsWith('computer_use_share'))).toHaveLength(0); + }); + + it('offers system setup only for the missing permission without selecting an app', async () => { + await render(); + const setup = button('computerUse.openSettings'); + expect(setup.closest('[data-openbitfun-part="row"]')?.textContent).toContain('computerUse.screenCapture'); + expect(Array.from(container.querySelectorAll('button')).filter(el => el.textContent === 'computerUse.openSettings')).toHaveLength(1); + await act(async () => setup.click()); + expect(mocks.invoke).toHaveBeenCalledWith('computer_use_open_system_settings', { request: { pane: 'screen_capture' } }); + expect(mocks.invoke).not.toHaveBeenCalledWith('computer_use_request_permission', expect.anything()); + }); + + it('registers the missing OS permission from the same setup button on capable hosts', async () => { + const original = mocks.invoke.getMockImplementation()!; + mocks.invoke.mockImplementation(async (command: string, ...args: unknown[]) => command === 'computer_use_get_status' + ? { computerUseEnabled: true, accessibilityGranted: true, screenCaptureGranted: false, + permissions: { permissions: [{ kind: 'screen_recording', requestable: true }] } } + : command === 'computer_use_request_permission' ? { current_granted: false } + : original(command, ...args)); + await render(); + expect(mocks.invoke).not.toHaveBeenCalledWith('computer_use_request_permission', expect.anything()); + await act(async () => button('computerUse.openSettings').click()); + expect(mocks.invoke).toHaveBeenCalledWith('computer_use_request_permission', { request: { kind: 'screen_recording' } }); + expect(mocks.invoke).toHaveBeenCalledWith('computer_use_open_system_settings', { request: { pane: 'screen_capture' } }); + expect(mocks.invoke.mock.calls.filter(([name]) => String(name).startsWith('computer_use_share'))).toHaveLength(0); + }); it('localizes the browser choice and puts the single connect action next to connection status', async () => { await render(); expect(container.querySelector('option')?.textContent).toBe('browserControl.defaultBrowser'); diff --git a/src/web-ui/src/app/scenes/settings/pages/tools/DeviceControlSettingsPage.tsx b/src/web-ui/src/app/scenes/settings/pages/tools/DeviceControlSettingsPage.tsx index 0dcf5bc3d1..4058d9536b 100644 --- a/src/web-ui/src/app/scenes/settings/pages/tools/DeviceControlSettingsPage.tsx +++ b/src/web-ui/src/app/scenes/settings/pages/tools/DeviceControlSettingsPage.tsx @@ -1,9 +1,6 @@ import '@/app/scenes/settings/pages/shared/RuntimeSettings.scss'; -import { ComputerUsePermissions } from '@/app/components/ComputerUsePermissions'; -import { computerUseControlAPI } from '@/infrastructure/api/service-api/ComputerUseControlAPI'; -import { useI18n } from '@/infrastructure/i18n'; -import '@/app/components/ComputerUseControlCard.scss'; import { api } from '@/infrastructure/api/service-api/ApiClient'; +import { computerUseControlAPI, type ComputerUsePermissions } from '@/infrastructure/api/service-api/ComputerUseControlAPI'; import { systemAPI } from '@/infrastructure/api/service-api/SystemAPI'; import { ConfigLoadingState, @@ -17,6 +14,7 @@ import { useComputerUseEnabled } from '@/infrastructure/config/hooks/useComputer import { configManager } from '@/infrastructure/config/services/ConfigManager'; import { i18nService } from '@/infrastructure/i18n'; import { usePeerDeviceModeOptional } from '@/infrastructure/peer-device/peerDeviceContextState'; +import { getActiveSurfaceScope } from '@/infrastructure/peer-device/deviceSurface'; import { notificationService } from '@/shared/notification-system'; import { createLogger } from '@/shared/utils/logger'; import { @@ -51,7 +49,7 @@ type ComputerUseStatusPayload = { accessibilityGranted: boolean; screenCaptureGranted: boolean; platformNote: string | null; - permissions?: unknown; + permissions?: ComputerUsePermissions; }; type BrowserControlLaunchResponse = { @@ -85,7 +83,6 @@ const DEFAULT_BROWSER_CONTROL_BROWSER = 'default'; const DeviceControlSettingsPage: React.FC = () => { const { t } = useTranslation('settings/runtime'); - const { t: tCommon } = useI18n('common'); const { t: tTools } = useTranslation('settings/agentic-tools'); @@ -104,10 +101,11 @@ const DeviceControlSettingsPage: React.FC = () => { const [peerBrowserControlUnsupported, setPeerBrowserControlUnsupported] = useState(false); const [computerUseAccess, setComputerUseAccess] = useState(false); - const [hasDetailedPermissions, setHasDetailedPermissions] = useState(false); const [computerUseScreen, setComputerUseScreen] = useState(false); + const [computerUsePermissions, setComputerUsePermissions] = useState(); + const [computerUseBusy, setComputerUseBusy] = useState(false); const [computerUseStatusLoading, setComputerUseStatusLoading] = useState(false); @@ -152,13 +150,14 @@ const DeviceControlSettingsPage: React.FC = () => { if (!IS_TAURI_DESKTOP) return false; setComputerUseStatusLoading(true); setComputerUseStatusError(false); + setComputerUsePermissions(undefined); try { const s = await api.invoke('computer_use_get_status'); setPeerBrowserControlUnsupported(false); setComputerUseEnabled(s.computerUseEnabled); setComputerUseAccess(s.accessibilityGranted); - setHasDetailedPermissions(!!s.permissions); setComputerUseScreen(s.screenCaptureGranted); + setComputerUsePermissions(s.permissions); setComputerUsePlatformNote(s.platformNote); return true; } catch (error) { @@ -243,11 +242,25 @@ const DeviceControlSettingsPage: React.FC = () => { }; const handleComputerUseOpenSettings = async (pane: 'accessibility' | 'screen_capture') => { + const scope = getActiveSurfaceScope(); + setComputerUseBusy(true); try { + const kind = pane === 'screen_capture' ? 'screen_recording' : 'accessibility'; + if (computerUsePermissions?.permissions.some(permission => permission.kind === kind && permission.requestable)) { + const receipt = await computerUseControlAPI.requestPermission(kind); + if (!scope.isCurrent()) return; + if (receipt.current_granted) { + await refreshComputerUseStatus(); + return; + } + } await api.invoke('computer_use_open_system_settings', { request: { pane } }); } catch (error) { + if (!scope.isCurrent()) return; log.error('computer_use_open_system_settings failed', error); notificationService.error(t('messages.saveFailed')); + } finally { + if (scope.isCurrent()) setComputerUseBusy(false); } }; @@ -424,10 +437,8 @@ const DeviceControlSettingsPage: React.FC = () => { : computerUseStatusError ? t('computerUse.statusUnavailable') : computerUseScreen ? t('computerUse.granted') : t('computerUse.notGranted'); - const computerUsePlatformMessage = computerUsePlatformNote - ? platform === 'macos' - ? t('computerUse.platformNotes.macos') - : platform === 'windows' + const computerUsePlatformMessage = computerUsePlatformNote && platform !== 'macos' + ? platform === 'windows' ? t('computerUse.platformNotes.windows') : platform === 'linux' ? t('computerUse.platformNotes.linux') @@ -541,7 +552,7 @@ const DeviceControlSettingsPage: React.FC = () => { /> - {!hasDetailedPermissions && <> + {platform === 'macos' && <> { > {computerUseAccessLabel} - {platform === 'macos' && ( + {!computerUseAccess && ( - - } - {hasDetailedPermissions && } {computerUsePlatformMessage && (