Repository navigation
feat(cua): migrate explicit app sharing and scoped native control - #3359
Merged
Merged
Conversation
…-migration # Conflicts: # docs/interactive-capabilities/README.md # docs/interactive-capabilities/capabilities.json # docs/interactive-capabilities/technical/product-control-open-audit.json # docs/interactive-capabilities/technical/tauri-command-map.json # src/crates/contracts/product-domains/src/generated/product-control-catalog.json # src/crates/contracts/product-domains/src/generated/remote-surface-registry.json # src/web-ui/src/app/global-search/generated/interactive-capabilities.json # src/web-ui/src/infrastructure/api/generated/productControl.ts # src/web-ui/src/infrastructure/api/generated/remoteSurface.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Computer Use now waits for the user to select an App/window, shares it in observe-only mode, and requires a separate revocable grant before input. This brings the permission, sharing, coordinate, virtual-cursor and background-control behavior from
computer-use-rustinto OpenBitFun's existing platform providers.Type and Areas
Type: Feature, refactor, regression protection.
Areas: Desktop/Tauri native CUA, portable tool contracts, Core orchestration, Web UI, peer capability negotiation, native fixtures and architecture documentation.
Motivation / Impact
Starting an Agent tool no longer silently grants control of an arbitrary application. Users can inspect the selected surface, allow or revoke control, optionally remember an executable-bound preference, and stop the session. Remembered grants never choose or share a window automatically. Input cannot reuse stale screenshot/node references after a target or authorization change.
Verification
Local focused verification passed:
cargo test --locked -p openbitfun-desktop --lib computer_use: 160 passed, nine opt-in native tests ignored.cargo test --locked -p openbitfun-core --no-default-features --features agent-runtime,git,tools-computer-use --lib computer_use: 72 passed.cargo test --locked -p openbitfun-agent-tools --no-default-features --features computer-use-contract --lib computer_use: 33 passed.node scripts/test-computer-use-context.mjs: 63 passed, one opt-in browser test ignored.pnpm --dir src/web-ui run test:run src/app/components/ComputerUseControlCard.test.tsx src/app/components/computer-use src/infrastructure/api/adapters/peer-device-adapter.test.ts src/infrastructure/appearance/registry/AppearanceRegistry.test.ts: 82 passed, including 63 reference cursor samples.cargo build --locked -p openbitfun-desktop,pnpm run check:web,pnpm run i18n:audit,pnpm run theme:color-audit:all,pnpm run capabilities:check,pnpm run check:core-boundaries,pnpm run check:repo-hygiene, and the Git object-size check passed.Native evidence is separate from compilation/unit tests:
test-macos-input-controls.mjs --orchestration,test-macos-control-roundtrip.mjs,test-macos-control-capture.mjs,test-macos-control-pointer.mjsandtest-native-ax-context.mjspassed. The strict fixture covers production App sharing/grants, native preview input, stale-input rejection after revocation, exact event counts and preservation of observer foreground/key/stacking/system cursor. Earlier hardware-idle/cursor assertion failures are recorded in the SPEC; an unchanged run during a quiet interval passed without restoring focus or relaxing assertions.node scripts/test-windows-computer-use-native.mjs --check --target x86_64-pc-windows-msvcpassed production-module cross-compilation.Counts across suites overlap and are not a unique-test total.
CI additionally exercised workspace compilation on Windows/macOS/Linux, all three CLI suites, the complete frontend suite and Linux/macOS Desktop library tests. The Windows Desktop test executable hit the repository's existing Tauri loader limitation (
0xc0000139,STATUS_ENTRYPOINT_NOT_FOUND) before tests could run; the unchanged workflow recognizes only that specific pre-test failure. Windows compilation and the other platform contract suites passed. This does not establish native Windows GUI execution.Reviewer Notes
AI-assisted implementation; testing is substantial but platform acceptance is incomplete. Windows native execution/full Desktop build, live GNOME Wayland Portal/PipeWire, OS permission denial/approval/restart and system Stop Sharing UI, and all four remote scenarios' end-to-end GUI flows were not locally validated. CI builds do not establish interactive native behavior.
Wire fields are additive with conservative defaults; older peers without
computer_use_sharing_v1are rejected explicitly for the new operations. Persisted grants retain unknown fields and are tied to executable identity. Linux arbitrary background coordinates remain unsupported. Background witnesses are before/after checks, not a universal guarantee for every application.The five-part reference migration matrix, ownership boundaries and platform evidence are documented in
docs/architecture/computer-use-control.md.Checklist