perf(computer-use): batch known inputs and add AX-only observations

docs/architecture/computer-use-control.md

@@ -188,6 +188,30 @@ input returns its delivery result so partial delivery is not hidden by dropping
 a future. Remaining steps are not run after failure or cancellation. Native macOS gestures also release any presses left unfinished by a failed or cancelled action, so a partial chord or drag cannot leave modifiers or mouse buttons held for the next action. An
 observation completing after stop, rebinding or generation change is discarded.
 
+For latency, the model starts with a known application's observation instead of
+unconditionally listing applications, starting control and checking status. The
+first app observation can enter background control; stopped sessions still require
+an explicitly permitted restart. Prompt guidance groups already-decided inputs
+until the next information boundary and reuses the returned observation. It does
+not reduce the user's selected reasoning effort or impose step/retry-count caps.
+
+`get_app_state`, `app_batch` and single app-input aliases accept optional
+`include_image` (default `true`). With `false`, Desktop reads AX and refreshes
+observation/node identities without retrieving or encoding a model screenshot.
+This is suitable for sufficient semantic state or known keyboard flows. Canvas,
+visual verification and selecting image coordinates still need an image-bearing
+observation. Target preparation, control leases, cancellation and native consent
+remain unchanged; AX-only is not a different input route. Legacy port providers
+can retain their old capture path and omit the image from its result. Existing
+callers/persisted tool calls omit the field and retain their previous behavior.
+The option is local tool presentation, not a new remote GUI capability.
+
+Program receipts include `observation_elapsed_ms` alongside per-step and total
+elapsed time. These measure host execution, not model reasoning or end-to-end
+latency. macOS focus-and-type/key operations take one desktop witness pair around
+the complete operation, including an internal native focus click; nested witness
+pairs are unnecessary. No native validation or post-input frame barrier is removed.
+
 Receipts distinguish attempted steps, submitted steps, the failing step and
 observation failure. A native error may follow partial submission. Neither that
 error, a missing screenshot nor an unchanged AX digest authorizes replay.
@@ -498,6 +522,79 @@ Local fixtures and unsupported-host unit tests are not remote end-to-end evidenc
 
 ## Verification and acceptance
 
+### Latency design evidence, 2026-10-13
+
+The installed `openai-codex-electron` 26.1007.30912 artifact (`@oai/cua` 0.2.6,
+`@oai/sky` 0.7.6) informed the interaction policy. Focused REA static analysis
+covered the shipped CUA JavaScript API and macOS Sky client, plus their packaged
+tool documentation. It established these distinctions:
+
+- Persistent JavaScript can await several input operations and return one final
+  observation in one model-facing call. Native input and state reads still use
+  separate IPC requests. This is grouping at the model boundary, not evidence of
+  one native batch RPC.
+- App selection already emits initial state. Native input wrappers do not
+  implicitly emit another tree/image after each input.
+- AX text, image and combined observations have separate public methods.
+  `getAXState` only emits text to the model, but calls the same underlying state
+  RPC; these wrappers do not prove that the native service avoids capture.
+- The macOS client caches its transport, correlates responses by request ID and
+  serializes writes. The exposed AX diff option is visible; the native diff and
+  settling algorithms are outside the inspected JavaScript boundary.
+
+The corresponding REA parent Evidence IDs are
+`ev_2f831c53ee4f06ac0236cdbc8eb701cbd5e584c2e33fa12029006b3fc80512e0`
+and `ev_49bc75d47413c72fd45730106ded6c3377f0651baa70105ec8494e64f7898056`.
+The inventories completed, while dynamic/external semantic relationships remain
+unknown. This is shipped-code evidence, not observed Codex execution or a recovered
+server-side system prompt. No comparative model latency or hidden reasoning
+configuration was established. Third-party code is not a product dependency.
+
+The [public computer-use API guide](https://developers.openai.com/api/docs/guides/tools-computer-use)
+also describes executing ordered actions before returning an observation; this
+is a public integration pattern, not proof of the desktop application's API wire.
+The [programmatic-tool guide](https://developers.openai.com/api/docs/guides/tools-programmatic-tool-calling)
+calls for measuring correctness, evidence, model turns and latency together.
+OpenBitFun applies the interaction policy through its existing typed program and
+host port, retaining its authorization, partial receipts and cancellation rules.
+
+Local macOS acceptance used the production Tool → host → native input/observation
+path, with independent target and foreground-observer applications. The strict
+fixture passed AX-only node input, fresh/stale observation checks, default-image
+recovery, semantic and image targets, Unicode entry, scroll, five-step canvas
+input, manual preview/revocation and Stop. Foreground identity/key state, window
+order, physical cursor and exact native delivery assertions remained enabled.
+
+The latency fixture executed the same five inputs (four Unicode text insertions
+and one known shortcut) in fresh processes for each case. Three repetitions per
+case, using a development build and a 12-node AX tree, produced:
+
+| Observation policy | Input tool calls | Returned observations / images | Median host time | Range |
+|---|---:|---:|---:|---:|
+| Single inputs, default images | 5 | 5 / 5 | 1,749 ms | 1,661–1,755 ms |
+| One batch, default image | 1 | 1 / 1 | 464 ms | 462–464 ms |
+| Single inputs, AX only | 5 | 5 / 0 | 176 ms | 164–181 ms |
+| One batch, AX only | 1 | 1 / 0 | 149 ms | 137–150 ms |
+
+Each timed result had to contain the exact final text in its returned AX state;
+an immediate fixture read confirmed shortcut delivery before recovery reads or
+waits. That read adds 0.073–0.130 ms, reported separately. Setup, the initial
+observation, later screenshot recovery and cleanup are outside the measurement.
+These are strategy comparisons in the same updated build, not a before/after
+model benchmark: batching already existed. No model inference, complex-app tree,
+Codex runtime, Windows/Linux native execution or remote end-to-end latency is
+included. Three samples establish a range/median, not a meaningful p95.
+The measured executable SHA-256 is
+`e1c2e5b3beb4eb608b94d3520905d53f16575aa85c009fe0de8475ad6cdb7d16`.
+
+The 78 focused tool/prompt contracts passed, including legacy provider/image
+defaults, malformed option rejection before input, cancellation/scope changes,
+partial receipts, and SSH-workspace/headless refusal of controller-local GUI.
+Remote workspace, remote control, Peer Device Mode and Detached Dispatch were
+not exercised end to end. Locked or concurrently active test desktops are
+inconclusive; clean runs retain all isolation assertions rather than accepting
+those samples as performance evidence.
+
 ### Autonomous-control correction, 2026-10-13
 
 The correction starts from `99acb6f04`. Focused checks passed: 51 Core

scripts/fixtures/computer-use-input-controls.m

@@ -77,6 +77,12 @@ int main(int argc, const char **argv) { @autoreleasepool {
         const double timeoutSeconds = argc > 2 ? strtod(argv[2], NULL) : 20.0;
         const double started = NSProcessInfo.processInfo.systemUptime;
         do {
+            NSDictionary *session = CFBridgingRelease(CGSessionCopyCurrentDictionary());
+            NSNumber *locked = session[@"CGSSessionScreenIsLocked"];
+            if ([locked isKindOfClass:NSNumber.class] && locked.boolValue) {
+                fprintf(stderr, "INCONCLUSIVE: execution host screen is locked; no observer was activated\n");
+                return 75;
+            }
             double idle = CGEventSourceSecondsSinceLastEventType(kCGEventSourceStateHIDSystemState, kCGAnyInputEventType);
             if (!isfinite(idle) || idle < 0) {
                 fprintf(stderr, "INCONCLUSIVE: hardware input idle state unavailable; no observer was activated\n");
@@ -170,6 +176,7 @@ int main(int argc, const char **argv) { @autoreleasepool {
     NSButton *button = [NSButton buttonWithTitle:@"Count one action" target:actions action:@selector(clicked:)];
     button.frame = NSMakeRect(20,240,240,30);
     field = [[NSTextField alloc] initWithFrame:NSMakeRect(20,180,300,30)];
+    [field setAccessibilityIdentifier:@"fixture-text-field"];
     field.delegate = actions;
     OBFPlainCanvas *canvas = [[OBFPlainCanvas alloc] initWithFrame:NSMakeRect(20,20,300,100)];
     [targetWindow.contentView addSubview:button]; [targetWindow.contentView addSubview:field]; [targetWindow.contentView addSubview:canvas];

scripts/test-macos-input-controls.mjs

@@ -5,18 +5,79 @@ import { join } from 'node:path';
 import { spawn } from 'node:child_process';
 import { fileURLToPath } from 'node:url';
 assert.equal(process.platform, 'darwin', 'directed input fixture requires macOS');
-const orchestration = process.argv.includes('--orchestration') || process.argv.includes('--semantic');
+const latency = process.argv.includes('--latency');
+const orchestration = process.argv.includes('--orchestration') || process.argv.includes('--semantic') || latency;
 const semantic = process.argv.includes('--semantic') || orchestration;
-assert.ok(process.argv.length === 2 || (process.argv.length === 3 && semantic), 'Usage: node scripts/test-macos-input-controls.mjs [--semantic | --orchestration]');
+assert.ok(process.argv.length === 2 || (process.argv.length === 3 && semantic), 'Usage: node scripts/test-macos-input-controls.mjs [--semantic | --orchestration | --latency]');
 const root = fileURLToPath(new URL('../', import.meta.url));
-const directory = await mkdtemp(join(tmpdir(), 'openbitfun-directed-input-'));
 const run = (command, args, env = process.env, onSpawn) => new Promise((resolve, reject) => {
   const child = spawn(command, args, { cwd: root, stdio: 'inherit', windowsHide: true, env });
   onSpawn?.(child);
   const timer = setTimeout(() => child.kill('SIGTERM'), 30_000);
   child.on('error', error => { clearTimeout(timer); reject(error); });
   child.on('close', code => { clearTimeout(timer); code === 0 ? resolve() : reject(new Error(`${command} exited ${code}`)); });
 });
+
+const nativeTestBinary = async () => process.env.OPENBITFUN_TEST_BINARY || new Promise((resolve, reject) => {
+  const build = spawn('cargo', ['test', '--locked', '-p', 'openbitfun-desktop', ...(orchestration ? ['--features', 'devtools', '--test', 'computer_use_native_roundtrip'] : ['--lib']), '--no-run', '--message-format=json'], {
+    cwd: root, stdio: ['ignore', 'pipe', 'inherit'], windowsHide: true,
+  });
+  let pending = '';
+  let executable;
+  build.stdout.on('data', chunk => {
+    pending += String(chunk);
+    const lines = pending.split('\n');
+    pending = lines.pop();
+    for (const line of lines) {
+      const message = JSON.parse(line);
+      if (message.reason === 'compiler-artifact' && message.target.name === (orchestration ? 'computer_use_native_roundtrip' : 'openbitfun_desktop_lib') && message.executable) executable = message.executable;
+      if (message.reason === 'compiler-message' && message.message.rendered) process.stderr.write(message.message.rendered);
+    }
+  });
+  build.once('error', reject);
+  build.once('close', code => code === 0 && executable ? resolve(executable) : reject(new Error(`Test build failed (${code}) or produced no executable`)));
+});
+
+if (latency) {
+  // Compile once, then isolate every case in a fresh target, observer and native
+  // host. Child runners retain the ordinary input-idle gates and final checks.
+  const testBinary = await nativeTestBinary();
+  const measurements = [];
+  for (const fixtureCase of ['singles:default', 'batch:default', 'singles:ax', 'batch:ax']) {
+    await new Promise((resolve, reject) => {
+      const child = spawn(process.execPath, [fileURLToPath(import.meta.url), '--orchestration'], {
+        cwd: root, windowsHide: true, stdio: ['ignore', 'inherit', 'pipe'],
+        env: { ...process.env, OPENBITFUN_TEST_BINARY: testBinary, OPENBITFUN_INPUT_LATENCY_CASE: fixtureCase },
+      });
+      let pending = '';
+      child.stderr.on('data', chunk => {
+        process.stderr.write(chunk);
+        pending += String(chunk);
+        const lines = pending.split('\n');
+        pending = lines.pop();
+        for (const line of lines) {
+          if (line.startsWith('CUA_NATIVE_LATENCY ')) measurements.push(JSON.parse(line.slice('CUA_NATIVE_LATENCY '.length)));
+        }
+      });
+      child.once('error', reject);
+      child.once('close', code => code === 0 ? resolve() : reject(new Error(`Native latency case ${fixtureCase} failed (${code})`)));
+    });
+  }
+  assert.equal(measurements.length, 4, 'every measured case must report after native assertions pass');
+  for (const result of measurements) {
+    const batched = result.case.startsWith('batch:');
+    const images = result.case.endsWith(':default');
+    assert.equal(result.input_steps, 5);
+    assert.equal(result.tool_calls, batched ? 1 : 5);
+    assert.equal(result.returned_observation_count, batched ? 1 : 5);
+    assert.equal(result.returned_image_count, images ? result.tool_calls : 0);
+    assert.ok(result.initial_node_count > 0);
+  }
+  console.log(`CUA_NATIVE_LATENCY_COMPARISON ${JSON.stringify({ measurements, scope: 'Same five native inputs in fresh disposable apps; no model latency or remote behavior measured. Timings are measurements, not pass/fail thresholds.' })}`);
+  process.exit(0);
+}
+
+const directory = await mkdtemp(join(tmpdir(), 'openbitfun-directed-input-'));
 let fixture;
 let observer;
 let harness;
@@ -34,28 +95,7 @@ try {
   await run('xcrun', ['clang', '-fobjc-arc', '-Wall', '-Werror', 'scripts/fixtures/computer-use-input-controls.m', '-framework', 'AppKit', '-o', binary]);
   // Build before opening either fixture window, so compiler latency never
   // leaves the foreground observer sitting above the user's work.
-  let testBinary = process.env.OPENBITFUN_TEST_BINARY;
-  if (!testBinary) {
-    testBinary = await new Promise((resolve, reject) => {
-      const build = spawn('cargo', ['test', '-p', 'openbitfun-desktop', ...(orchestration ? ['--features', 'devtools', '--test', 'computer_use_native_roundtrip'] : ['--lib']), '--no-run', '--message-format=json'], {
-        cwd: root, stdio: ['ignore', 'pipe', 'inherit'], windowsHide: true,
-      });
-      let pending = '';
-      let executable;
-      build.stdout.on('data', chunk => {
-        pending += String(chunk);
-        const lines = pending.split('\n');
-        pending = lines.pop();
-        for (const line of lines) {
-          const message = JSON.parse(line);
-          if (message.reason === 'compiler-artifact' && message.target.name === (orchestration ? 'computer_use_native_roundtrip' : 'openbitfun_desktop_lib') && message.executable) executable = message.executable;
-          if (message.reason === 'compiler-message' && message.message.rendered) process.stderr.write(message.message.rendered);
-        }
-      });
-      build.once('error', reject);
-      build.once('close', code => code === 0 && executable ? resolve(executable) : reject(new Error(`Test build failed (${code}) or produced no executable`)));
-    });
-  }
+  const testBinary = await nativeTestBinary();
   const targetBinary = await bundleExecutable('OpenBitFun Input Target', 'dev.openbitfun.input-fixture.target');
   const observerBinary = await bundleExecutable('OpenBitFun Input Observer', 'dev.openbitfun.input-fixture.observer');
   await run(binary, ['--wait-input-idle', '20']);

src/apps/desktop/src/computer_use/AGENTS.md

@@ -130,6 +130,9 @@ node scripts/test-macos-input-controls.mjs --semantic
 # Target/observer have distinct app bundles; AppKit initializes before the
 # foreground baseline. Activation history catches transient focus changes.
 node scripts/test-macos-input-controls.mjs --orchestration
+# Identical five-input sequence: five tool calls vs one batch, with default
+# images and AX-only observations. Fresh isolated app/host per case.
+node scripts/test-macos-input-controls.mjs --latency
 # Diagnostic raw-pointer contract, including ordinary non-AX first click and
 # unchanged modifiers/z-order. Both this raw runner and the full Tool five-action
 # fixture passed; do not substitute a permissive first-click implementation.
@@ -145,6 +148,16 @@ the rendered JPEG. The AX fixture compiles the Desktop test target and checks
 native AX nodes, text, states, parent indices, cache entries, filtering, and DTO
 round trips. These are local macOS checks, not Windows/Linux or remote evidence.
 
+The latency fixture reports actual returned observations/images, AX node counts
+and native tool wall time including one separately timed fixture JSON check.
+It checks exact Unicode text, immediate shortcut delivery,
+fresh AX-only node identities, recovery of default screenshot observations,
+and the same foreground/cursor invariants as the orchestration fixture. It does
+not invoke a model. Startup and cleanup are outside the reported measurement;
+timings are diagnostic evidence, not machine-dependent pass/fail thresholds.
+The native fixture requires an unlocked desktop; its read-only startup gate
+reports a locked screen as inconclusive without waking or unlocking it.
+
 Observation invariants:
 
 - Rectangle containment does not prove two controls have the same action.

src/apps/desktop/src/computer_use/desktop_host/ax_orchestration.rs

@@ -709,7 +709,9 @@ impl DesktopComputerUseHost {
                         modifier_keys: vec![],
                         wait_ms_after: None,
                     };
-                    self.dispatch_app_click_impl(click).await?;
+                    // The outer focus-and-type action already witnesses the
+                    // complete operation, including this native focus click.
+                    self.dispatch_app_click_native(click).await?;
                     // Pointer targets (including OCR) record the actual submitted
                     // location. Do not infer it from the human's real mouse.
                     let intended_point = verify_pointer_focus
@@ -995,7 +997,8 @@ impl DesktopComputerUseHost {
                         modifier_keys: vec![],
                         wait_ms_after: None,
                     };
-                    self.dispatch_app_click_impl(click).await?;
+                    // The outer chord action already witnesses this focus click.
+                    self.dispatch_app_click_native(click).await?;
                 }
             }
             require_macos_background_input()?;

src/apps/desktop/src/computer_use/desktop_host/mod.rs

@@ -1692,14 +1692,22 @@ impl ComputerUseHost for DesktopComputerUseHost {
         max_depth: u32,
         focus_window_only: bool,
     ) -> OpenBitFunResult<AppStateSnapshot> {
-        // Public path: always auto-attach a focused-window screenshot so the
-        // model is never blind on Canvas / WebView / WebGL surfaces that the
-        // AX tree can't describe (Codex parity — its `get_app_state` is the
-        // single "eyes" of the desktop loop).
+        // Preserve image-bearing observations for existing callers.
         self.get_app_state_inner(app, max_depth, focus_window_only, true)
             .await
     }
 
+    async fn get_app_state_with_options(
+        &self,
+        app: AppSelector,
+        max_depth: u32,
+        focus_window_only: bool,
+        include_image: bool,
+    ) -> OpenBitFunResult<AppStateSnapshot> {
+        self.get_app_state_inner(app, max_depth, focus_window_only, include_image)
+            .await
+    }
+
     fn supports_app_shortcuts(&self) -> bool {
         #[cfg(any(target_os = "macos", target_os = "windows"))]
         {