Computer use: OCR/pictureOnly fallback and Windows (UIAutomation) backend #123

Status — DONE (2026-08-25)

The Windows track is implemented and validated on real hardware; nothing tracked here remains except an optional CDP-roots nicety noted below.

Shipped:

• macOS backend — AX/CGEvent (pre-existing).
• Raw-image fallback (PR feat(computer-use): raw-image fallback for text-sparse AX trees #258) — text-sparse AX trees attach a window screenshot, gated by imageMode. (OCR was explicitly rejected: PR feat(computer-use): OCR pictureOnly fallback (macOS Vision) #256 reverted by revert: OCR pictureOnly fallback (#256) #257.)
• Windows (UIAutomation) backend (PR feat(computer-use): Windows UIAutomation backend #268) — implemented over the uiautomation crate (a hand-rolled COM attempt was scrapped in favor of the mature crate). Root enumeration, tree walk, control-pattern actions (Invoke/Value/Toggle/ExpandCollapse/SelectionItem/Scroll/…), keyboard/mouse input, and screenshot capture. Validated on physical Windows 11 via tcode --cu-smoke: real multi-node UIA trees (WindowsTerminal 23, taskbar 25, Program Manager 31 nodes), correct root enumeration and text-sparse classification.

outline/state/tool-surface stay platform-neutral, so improvements apply across every provider.

Deliberately not done (out of scope, low value):

• CDP browser roots — enumerating Chromium DevTools targets as observation roots. A documented v1 deviation; open a fresh issue if it's ever wanted.
• Linux AT-SPI backend — no demand; add if it appears.

Closing: the accessibility-tree gap this issue was filed for (macOS-only, no OCR/pictureOnly path) is closed on both counts — Windows has a real UIA backend, and text-sparse windows fall back to raw images.