Infrastructure: Prove the desktop test driver on real platforms and pass its release gate

[JasonVMo]

Summary

Prove the desktop driver on real macOS and Windows systems, eliminate every
documented false-green path, and satisfy the package's release gate. Complete
the branch's Priority 0 native verification and Priority 1 reporting work,
publish compatibility evidence, and validate an external consumer.

The historical Windows Button crash is already fixed on main; this task must
rerun the unchanged shared suite after integration rather than reopen that
diagnosis.

Goal

Turn @fluentui-react-native/desktop-driver from "implemented and covered by unit
and fake-backend tests" into "verified on real macOS and Windows hardware, incapable
of reporting a false green, and releasable" by executing the branch's own Priority 0
verification, completing its Priority 1 reporting work, and satisfying its
seven-item release gate.

Stage

Stage 2 (beta) for Priority 0 and Priority 1; Stage 3 (production) for the full
release gate, published compatibility ranges, and external-consumer validation.

Observed. The branch's
NEXT-STEPS.md
states plainly: "The safety and false-green fixes are implemented and covered by
focused tests and the fake backend. They still need current evidence from both
native backends." It then enumerates macOS verification, Windows re-verification, a
then-blocking application defect, four priority tiers of remaining work, 14 open
decisions, and a release gate.

This task carries that list to completion, with one item already discharged by
main: the blocking Button crash is fixed by
4777e54f
(PR #4208), so only
a Windows rerun proving the unchanged suite is unblocked remains - see
The Windows Button crash.
It is separated from test-driver.md because landing the package
and proving it are independently reviewable, and because the proof work requires
physical interactive desktop sessions that a merge review does not.

Why it matters

Observed. The fake backend can produce a passing run with no application and no
native driver, and the branch warns that "a configuration that defaults to fake
can return a valid fake-backend pass without touching the real app."

Observed. Several failure surfaces are documented as not yet truthful:
captureArtifacts() returns the cumulative run manifest rather than files captured
by that call; sessionStrategy: 'spec' workers can overwrite a shared run.json
and junit.xml; configuration/capability/ownership/readiness failures are not all
mapped to infrastructureError; startup failures can be masked by a secondary
"endpoint not published" error.

Inferred. Until those are fixed, promoting any desktop job to a required CI
gate would institutionalize a check that can pass while the product is broken. This
task is therefore a hard prerequisite for the required-gate phases of
storybook-e2e.md.

Observed current state

Priority 0 - real-platform proof

Observed. macOS has never been verified. The branch requires recording Mac2
launch and bundle-identity attach behavior, attach leaving the application running,
observed window readiness, focused attribute behavior and active-element fallback,
native wheel scrolling, application/WebDriverAgentMac/xcodebuild/host lifecycle
evidence, bounded cancellation and cleanup, Storybook Run current / Run all /
Cancel behavior, and a generated manifest digest matching the Windows job byte for
byte. It states the package "must not claim a verified macOS compatibility range
until this passes on a clean machine".

Observed. Windows was verified before the process-supervision and lifecycle
changes and must be re-verified: attach leaves the original PID and window running;
launch mode stops only the launched application; cancelling an on-device run exits
the command interpreter, WDIO runner, driver host, and WinAppDriver descendants
within the deadline; forced post-readiness app and driver-host failures cannot
report success; an unlocked WinAppDriver screenshot is inspected for WinAppSDK
Composition content; and the regenerated manifest digest matches macOS.

The Windows Button crash: historical branch observation versus current main

Observed - historical, at branch tip
8f971021.
NEXT-STEPS.md records a "Blocking application defect": "The shared Windows
Storybook suite remains blocked by an agentic Button crash, not a known driver
failure. Clicking the component fail-fast terminates ReactApp.exe roughly three
seconds later with 0xc0000409 in ucrtbase.dll; plain React Native Pressable
controls in the Storybook shell click successfully." The branch directed that a
native crash dump be collected and the defect fixed or filed in the agentic
components package. That text is unchanged on the branch and still reads as
blocking.

Observed - current origin/main
51fab45c.
The defect is fixed. Commit
4777e54f
"Fix windows crash in components"
(PR #4208) is an
ancestor of origin/main. It:

• adds the FocusVisual primitive at
  packages/agentic/components/src/primitives/focus-visual,
  exported from
  packages/agentic/components/src/index.ts;
• migrates Accordion, Button, Card, Checkbox, ListItem, ListboxItem, MenuItem,
  Radio, Switch, Tab, and Tag off outline-based focus styling onto persistent focus
  visuals;
• sets enableFocusRing: false on the Button pressable, with the in-code
  rationale "RNW 0.81 crashes when either outline props or its native focus ring
  creates border visuals after mount"; and
• adds Windows regression coverage to
  apps/storybook/windows-tests/storybook-smoke.test.cjs:
  a Button Overview focus-movement test and ten "focuses ... without terminating
  the app" cases - nine parameterized components plus the interactive Card - that
  click a control, wait three seconds, and assert HasKeyboardFocus.

Its changeset states: "Render persistent single- and dual-ring focus visuals across
agentic components to avoid React Native Windows Fabric crashes."

Inferred. Landing or rebasing the driver branch onto current main integrates
this fix. No new diagnosis, crash-dump collection, or defect-filing work is
required. The remaining obligation is evidentiary: rerun the unchanged shared
Windows suite on top of current main and prove it is unblocked. This task keeps
that rerun and drops the diagnose/fix/file work entirely.

Priority 1 - truthful reporting

Observed items, grouped as the branch groups them:

• Normalize failures and startup diagnosis: map configuration, capability,
  ownership, driver-host, readiness, and runner-spawn failures to
  infrastructureError; preserve framework skips and cancellation instead of
  collapsing them into failed; classify application crashes separately from
  driver/host/monitor/runner failures; ensure startup, session, and readiness
  failures still write lifecycle events, run.json, and JUnit; clear stale endpoint
  environment state before startup and preserve the original launcher failure.
• Report and artifact integrity: captureArtifacts() returns only that call's
  files; bounded event payloads without truncating run.json; merge per-worker
  reports for sessionStrategy: 'spec'; cover reports larger than 100 results; end
  generated JUnit with a newline.
• Storybook execution hardening: one owned WDIO invocation and warm session for
  Run all while preserving per-story progress; a runner deadline in addition to
  cancellation; make the generated exact story filter an explicit config-factory
  contract so Run current cannot silently run every test; resolve Windows launchers
  through PATH/PATHEXT instead of assuming .cmd.
• CLI and backend claims: keep NovaWindows unavailable; make
  desktop-driver start a foreground host or rename it; make stories list
  validate only the connection it uses; report direct window-handle discovery as
  matchedBy: 'windowHandle'; preserve backend/transport failures during window
  discovery instead of rewriting them as "no matching window".

Priority 2 and 3

Observed. Priority 2 is transport and runtime hardening (nested runtime field
validation, deadline-bounded Storybook HTTP requests, valid W3C errors for oversized
bodies without destroying the socket, real cycle detection, robustness coverage for
stream failures and temporary-directory cleanup). Priority 3 is consolidation
(launcher/worker split behind a validated codec, shared loopback mechanics,
extraction of CLI parsing out of import.meta entry modules, deduplication of
loopback allowlists and error mapping, removal of the attach-window delegate,
bounded concurrency for window attribute reads). The branch requires Priority 3
after the behavior work so refactoring cannot hide contract regressions.

Open decisions

Observed. 14 open decisions are recorded with recommendations, including:
whether terminateLaunchedApp stays public; whether Windows should use Job Objects
for descendants; whether app crashes are product failures or infrastructure errors;
whether macOS attach gets platform-specific types; whether the portability digest
includes transitive imports; whether empty manifests are valid; how linked tests are
discovered; whether Run all isolates every story; whether NovaWindows is supported;
whether desktop-driver start is a host or a probe; whether Storybook control may
use a remote host; whether a launcher/worker split changes the public API; whether a
Windows Graphics Capture fallback is required; and the long-term Appium 4 hosting
path.

Security and lifecycle invariants that must survive

Observed. All hosts bind only to allowlisted loopback addresses; attach never
grants termination ownership; cleanup never targets a process by name; backend
capability overrides cannot weaken ownership or routing; Storybook requests cannot
choose commands, paths, or test code; inline plans use a closed schema and testID
selectors only; platform commands capable of arbitrary local execution stay disabled
by default; artifact paths remain inside the run directory with sensitive fields
redacted; protocol and manifest versions are validated at every process boundary.

Observed. The Storybook service's per-boot token "is not a defense against a
local attacker because the channel is also local and unauthenticated"; the real
controls are loopback binding, manifest allowlisting, serialized mutation, and
rejection of arbitrary execution.

Interactive desktop constraints

Observed. Windows tests require a real, interactive, unlocked desktop; React
Native Windows pressables generally expose no UI Automation InvokePattern, so
clicks use synthetic input and fail on a locked workstation - and a locked session
can still return source, attributes, and screenshots, "making the failure look like
an application defect". macOS requires macOS 11.3+, Xcode 13+, Command Line Tools,
Xcode Helper accessibility permission, automation mode, a logged-in GUI session, and
a writable WebDriverAgentMac build cache.

Scope

• Execute and record the macOS Priority 0 verification list on a clean machine with
  documented Xcode, accessibility, automation-mode, signing, and GUI-session setup.
• Execute and record the Windows Priority 0 re-verification list.
• Rerun the unchanged shared Windows Storybook suite on top of current main and
  record that the previously blocking Button crash no longer reproduces, citing
  4777e54f
  as the fix. Only if it still reproduces does this become a defect investigation
  again.
• Complete Priority 1 in full, with tests that fail without the fix.
• Complete Priority 2 hardening.
• Complete Priority 3 consolidation only after Priority 1 and 2 land.
• Resolve every open decision, recording the outcome in DESIGN.md and removing it
  from NEXT-STEPS.md.
• Publish verified compatibility ranges: OS versions, Xcode version, WinAppDriver
  version, Appium driver versions, Node version, WebdriverIO version.
• Validate a clean external consumer with no globally installed Appium driver.

Out of scope

• Landing the package - see test-driver.md.
• Defining CI jobs - see storybook-e2e.md.
• Adding a win32 platform or backend - see
  win32-storybook.md. Inferred: a Win32 backend should
  not be added until macOS and Windows have both passed Priority 0, otherwise a
  third unverified backend compounds the same risk.
• Fixing agentic component defects. The previously blocking Button crash is
  already fixed on main by
  4777e54f;
  any new component defect surfaced by these runs is filed and handed to the
  Components workstream.
• Maintaining or extending the FocusVisual primitive introduced by that fix.

Deliverables

1. A recorded macOS verification report covering every Priority 0 macOS item, with
   artifact paths and the observed manifest digest.
2. A recorded Windows re-verification report covering every Priority 0 Windows item,
   with the same digest.
3. A recorded rerun of the unchanged shared Windows suite on current main showing the
   previously blocking Button crash does not reproduce.
4. Code and tests completing Priority 1, then Priority 2, then Priority 3.
5. An updated DESIGN.md recording every resolved decision, and a NEXT-STEPS.md
   reduced to genuinely open items.
6. A published compatibility and prerequisite matrix in README.md.
7. An external-consumer validation record.

Acceptance criteria

• The same generated story-tests.manifest.json digest is produced on a macOS
  machine and a Windows machine, and both runs execute the same test IDs and
  report the same portable command-matrix version.
• On both platforms: launch mode terminates only the application it launched;
  attach mode leaves the original PID and window running after the run.
• On both platforms: an on-device Cancel exits the runner, driver host, and all
  native driver descendants within the documented deadline, verified by process
  inspection rather than by absence of error.
• A forced post-readiness application crash and a forced driver-host failure each
  produce a non-passing run with a distinct lifecycle reason, and both still
  write run.json, events.ndjson, and junit.xml.
• A run configured for a real platform cannot silently fall back to the fake
  backend; attempting it is an explicit error.
• captureArtifacts() returns only the files captured by that invocation, and a
  sessionStrategy: 'spec' run with multiple workers produces one merged
  run.json and one merged junit.xml.
• "Run current" executes only the current story's tests, proven by the generated
  exact-filter contract test plus one observed real run.
• The unchanged shared Windows Storybook suite is rerun on a build containing
  4777e54f
  and runs to completion: clicking the agentic Button does not terminate the
  application, and the run reports a terminal state with artifacts. The recorded
  result explicitly supersedes the branch's "Blocking application defect" note.
• The existing yarn windows:test focus regression cases added by that commit
  still pass on the same build, so the desktop-driver path does not regress the
  coverage that proved the fix.
• Every open decision in NEXT-STEPS.md is either resolved in DESIGN.md or
  restated with an explicit owner and target stage.
• README.md publishes verified OS, Xcode, WinAppDriver, Appium driver, Node,
  and WebdriverIO ranges, and no prerequisite reported as unknown is described
  as satisfied.
• A consumer outside this repository, with no globally installed Appium driver,
  installs the package and runs one passing spec on each real platform.
• All seven release-gate items are demonstrably satisfied and recorded.

Dependencies and ordering

• Depends on: test-driver.md.
• Blocks: promotion of any desktop CI job to required in
  storybook-e2e.md; the Win32 backend work in
  win32-storybook.md.
• Cross-workstream: the Button crash was an agentic component defect and is
  already fixed on main by
  4777e54f.
  This task now owns only the rerun evidence. Any newly surfaced component defect is
  handed to the Components workstream.
• Internal ordering (Observed, branch-mandated): Priority 0 and Priority 1
  before Priority 3 consolidation.

Risks and open decisions

Item	Evidence	Risk
macOS entirely unverified	Observed: "They still need current evidence from both native backends"; macOS verification list is complete and unexecuted.	The macOS half of the "write once, run on both" claim is currently unproven.
Locked-session false failures on Windows	Observed: a locked session still returns source, attributes, and screenshots while clicks fail.	A CI failure can be misdiagnosed as a product defect. Detecting and reporting session lock state is required for a trustworthy gate.
Previously blocking Button crash	Observed (historical, branch tip 8f971021): 0xc0000409 in ucrtbase.dll, ~3s after click, ReactApp.exe. Observed (current main): fixed by 4777e54f / PR #4208.	Residual risk is only that the fix has not yet been proven against the desktop-driver click path, which differs from the @react-native-windows/automation path the fix was validated with. The rerun is the discharge.
Focus-ring workaround durability	Observed: the fix disables the native focus ring (enableFocusRing: false) and renders persistent focus visuals instead, with the in-code reason "RNW 0.81 crashes when either outline props or its native focus ring creates border visuals after mount".	A future React Native Windows upgrade may change this behavior; the driver suite should keep the focus assertions that would detect a regression.
WinAppDriver screenshot fidelity	Observed: does not reliably capture WinAppSDK Composition content; Graphics Capture fallback is an open decision.	Any screenshot-based evidence or visual gate is unreliable on Windows until resolved.
Appium 4 / driver-author API deprecation	Observed: server export from appium/driver.js marked deprecated for Appium 4.	A future Appium major could break the host; the isolation boundary limits but does not remove the risk.
WinAppDriver longevity	Observed: WinAppDriver 1.1 is installed by CI, 1.2.1 is required by the Storybook Windows harness, and NovaWindows is rejected for now.	Backend continuity is an unresolved production risk.
Artifact privacy	Observed: screenshots, source, and logs "can still contain private content".	Artifact publication in CI needs an explicit retention and review policy.

Evidence and references

Retrieved 2026-08-20.

• NEXT-STEPS.md
  • Priority 0-3, open decisions table, release gate.
• DESIGN.md
  • ownership, lifecycle, platform constraints, security invariants, validation
    strategy.
• README.md
  • prerequisites, artifacts, doctor.
• USAGE.md
  • agent workflow and run-state expectations ("A successful HTTP 202 means the run
    started; it does not mean the tests passed").
• Current repository: apps/storybook/README.md
  (WinAppDriver 1.2.1 requirement, Composition screenshot limitation),
  .github/workflows/pr.yml (WinAppDriver 1.1
  installation in CI).
• Current main fix:
  4777e54f
  "Fix windows crash in components"
  (PR #4208),
  ancestor of origin/main
51fab45c;
  resulting sources on main:
  packages/agentic/components/src/primitives/focus-visual,
  packages/agentic/components/src/components/button/useButton.ts,
  apps/storybook/windows-tests/storybook-smoke.test.cjs.

[JasonVMo]

Completion update - 2026-09-03

PR #4306 closed this issue by replacing the branch's Appium-backed proof path
with source-built, Appium-independent macOS and Windows/Win32 native providers.
The merge includes verified build/cache selection, exact launch/attach leases,
cancellation and input recovery, normalized accessibility, native capture,
Storybook integration on all three desktop endpoints, and explicit CI
diagnostics and capability skips.

The issue body's older Appium-, WinAppDriver-, DESIGN.md-, and
NEXT-STEPS.md-specific acceptance text is superseded by the merged
architecture. Closing this implementation milestone is not a claim that
every production-release qualification is complete.

Remaining authoritative input/TCC/DPI qualification, compatibility, security,
cache, performance, reliability, external-consumer, and publishing work is
recorded in packages/agentic/desktop-driver/PLAN.md and rolled up by #4275;
the accessibility, pipeline, focus-parity, and Windows-capture portions remain
open in #4255, #4256, #4254, and #4260 respectively.