Skip to content

feat(cua): migrate explicit app sharing and scoped native control - #3359

Merged
bobleer merged 4 commits into
GCWing:mainfrom
bobleer:bob/cua-app-sharing-migration
Oct 12, 2026
Merged

bobleer merged 4 commits into
GCWing:mainfrom
bobleer:bob/cua-app-sharing-migration

Conversation

@bobleer

@bobleer bobleer commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Computer Use now waits for the user to select an App/window, shares it in observe-only mode, and requires a separate revocable grant before input. This brings the permission, sharing, coordinate, virtual-cursor and background-control behavior from computer-use-rust into OpenBitFun's existing platform providers.

  • Add explicit OS permission probes/requests, process-bound remembered grants, App selection and a controllable preview in the existing CUA settings/card.
  • Fence observation caches, native callbacks and input with generation, target revision and authorization revision. Stop, cancellation, revocation and target changes invalidate stale work.
  • Share strict coordinate conversion across platforms; port the virtual cursor with Rust-generated parity samples. Manual preview input uses the same native host and lease as Agent input.
  • Preserve platform-specific routes: macOS directed input/AX, Windows WGC/UIA/window messages, Linux AT-SPI and explicitly authorized foreground portal input. Unsupported routes fail explicitly.
  • Remove heuristic repeated-action/retry advice that could duplicate valid input, and avoid unnecessary Windows frame encoding and legacy capture fallbacks.

Type and Areas

Type: Feature, refactor, regression protection.

Areas: Desktop/Tauri native CUA, portable tool contracts, Core orchestration, Web UI, peer capability negotiation, native fixtures and architecture documentation.

Motivation / Impact

Starting an Agent tool no longer silently grants control of an arbitrary application. Users can inspect the selected surface, allow or revoke control, optionally remember an executable-bound preference, and stop the session. Remembered grants never choose or share a window automatically. Input cannot reuse stale screenshot/node references after a target or authorization change.

Verification

Local focused verification passed:

  • cargo test --locked -p openbitfun-desktop --lib computer_use: 160 passed, nine opt-in native tests ignored.
  • cargo test --locked -p openbitfun-core --no-default-features --features agent-runtime,git,tools-computer-use --lib computer_use: 72 passed.
  • cargo test --locked -p openbitfun-agent-tools --no-default-features --features computer-use-contract --lib computer_use: 33 passed.
  • node scripts/test-computer-use-context.mjs: 63 passed, one opt-in browser test ignored.
  • Focused runtime evidence, prompt catalog, product-domain registry and Desktop remote-workspace policy suites: 5, 8, 19 and 9 passed respectively.
  • pnpm --dir src/web-ui run test:run src/app/components/ComputerUseControlCard.test.tsx src/app/components/computer-use src/infrastructure/api/adapters/peer-device-adapter.test.ts src/infrastructure/appearance/registry/AppearanceRegistry.test.ts: 82 passed, including 63 reference cursor samples.
  • cargo build --locked -p openbitfun-desktop, pnpm run check:web, pnpm run i18n:audit, pnpm run theme:color-audit:all, pnpm run capabilities:check, pnpm run check:core-boundaries, pnpm run check:repo-hygiene, and the Git object-size check passed.

Native evidence is separate from compilation/unit tests:

  • macOS: test-macos-input-controls.mjs --orchestration, test-macos-control-roundtrip.mjs, test-macos-control-capture.mjs, test-macos-control-pointer.mjs and test-native-ax-context.mjs passed. The strict fixture covers production App sharing/grants, native preview input, stale-input rejection after revocation, exact event counts and preservation of observer foreground/key/stacking/system cursor. Earlier hardware-idle/cursor assertion failures are recorded in the SPEC; an unchanged run during a quiet interval passed without restoring focus or relaxing assertions.
  • Linux: production-module harness passed 65 tests; a separate real GTK/AT-SPI fixture under D-Bus/Xvfb passed activation and UTF-8 insertion while preserving existing text.
  • Windows: node scripts/test-windows-computer-use-native.mjs --check --target x86_64-pc-windows-msvc passed production-module cross-compilation.
  • The sharing card was visually checked in Chromium with real design-system styling at 1000×800 and 390×700; this visual harness mocks transport.

Counts across suites overlap and are not a unique-test total.

CI additionally exercised workspace compilation on Windows/macOS/Linux, all three CLI suites, the complete frontend suite and Linux/macOS Desktop library tests. The Windows Desktop test executable hit the repository's existing Tauri loader limitation (0xc0000139, STATUS_ENTRYPOINT_NOT_FOUND) before tests could run; the unchanged workflow recognizes only that specific pre-test failure. Windows compilation and the other platform contract suites passed. This does not establish native Windows GUI execution.

Reviewer Notes

AI-assisted implementation; testing is substantial but platform acceptance is incomplete. Windows native execution/full Desktop build, live GNOME Wayland Portal/PipeWire, OS permission denial/approval/restart and system Stop Sharing UI, and all four remote scenarios' end-to-end GUI flows were not locally validated. CI builds do not establish interactive native behavior.

Wire fields are additive with conservative defaults; older peers without computer_use_sharing_v1 are rejected explicitly for the new operations. Persisted grants retain unknown fields and are tied to executable identity. Linux arbitrary background coordinates remain unsupported. Background witnesses are before/after checks, not a universal guarantee for every application.

The five-part reference migration matrix, ownership boundaries and platform evidence are documented in docs/architecture/computer-use-control.md.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, and untested platform scenarios are explained.
  • User-facing strings, docs, and locales are updated where applicable.

…-migration

# Conflicts:
#	docs/interactive-capabilities/README.md
#	docs/interactive-capabilities/capabilities.json
#	docs/interactive-capabilities/technical/product-control-open-audit.json
#	docs/interactive-capabilities/technical/tauri-command-map.json
#	src/crates/contracts/product-domains/src/generated/product-control-catalog.json
#	src/crates/contracts/product-domains/src/generated/remote-surface-registry.json
#	src/web-ui/src/app/global-search/generated/interactive-capabilities.json
#	src/web-ui/src/infrastructure/api/generated/productControl.ts
#	src/web-ui/src/infrastructure/api/generated/remoteSurface.ts
@bobleer
bobleer merged commit 99acb6f into GCWing:main Oct 12, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant