Repository navigation
[Refactor] Driver architecture for browser_move - #12
Open
sirotaoleg wants to merge 400 commits into
Open
sirotaoleg wants to merge 400 commits into
sirotaoleg wants to merge 400 commits into
Conversation
added 30 commits
October 11, 2026 01:18
browser_get_element_coordinates, browser_drag_and_drop and browser_set_input_files become functions of the tool context under their controller names (CUSTOM_ACTIONS), with the controller's parameters, aliases, error texts and result shapes. Coordinates come from one page script (querySelector and the bounding box, falling back to the element with the visible text). Drag stamps both elements and sends Drag between the stamps, or between two points of the last screenshot. File inputs get UploadFile on the element where the driver accepts a locator for it, otherwise a Click that opens the chooser and UploadFile answering it. The page script values are read as values, so these actions no longer fail on a result envelope. The fallbacks that sent page code to the LLM worker are not ported. Refs: #driver-architecture
Ports the custom-action cases that still apply (fenced coordinate result now read as a script value, empty paths, the default file input) and covers the aliases, offsets, coordinates without a page script, the url opened first, the error texts, drag between stamped elements and between screenshot points, a driver that cannot drag points, and both upload paths. The cases for the code executor and the LLM fallback are dropped with those paths. Live, per driver: a file uploaded and an element dragged onto another on the scene. Refs: #driver-architecture
A StaleRef raised by the driver (after its retry observed afresh) reached CommandTool with the fresh observation on the error, but the tool passed no observation to the renderer and left the page state without it. The tool now records that observation in the page state, so its refs get the current generation, and hands the recorded observation to the renderer. Refs: #driver-architecture
Refs: #driver-architecture
message_metadata reported executed = dispatched, so browser_find over a current observation (no command sent) read as not executed, unlike the old server call that always ran. A READ record now reports executed as its ok; other effects keep dispatched. Refs: #driver-architecture
Refs: #driver-architecture
UploadFile and Drop copy a file from outside the server's working directory to uploads/<12 hex>/ under it, and the copies stayed. The driver now remembers each folder it created and removes those on stop(), then the uploads folder only if nothing else is left in it: the working directory defaults to the process directory, which may hold a user's own uploads folder (decided with Aleh instead of removing uploads/ whole). Refs: #driver-architecture
Refs: #driver-architecture
When managed Chrome died under browser-use, the sidecar answered the
next call with a plain DriverError ("Reconnection failed - CDP still
not connected", from browser-use's watchdogs), so the driver session
never restarted it. Unclassified sidecar errors saying the browser is
not connected, or that the reconnect wait timed out, now become
ConnectionLost (dispatched, as for any unclassified error).
Refs: #driver-architecture
Unit: the disconnect messages map to ConnectionLost without the unclassified-error warning. Live: managed Chrome killed under browser-use is found dead, provisioned again, and the next observe runs on a rebuilt driver, as for Playwright MCP. Refs: #driver-architecture
S_20: the batch tool (validation, one step as one command, the loop with preflight and stamped targets, the result and its record), the custom actions, the stale observation CommandTool records, executed for reads, the browser-use disconnect mapping and the MCP upload cleanup on stop. F_07: new section with the decisions, rejected options, the step to command table, ported and dropped tests, differences from the old path, golden numbers, verification and leftovers; earlier leftovers marked done. Refs: #driver-architecture
The coordinate, drag, set-input-files and batch actions ran Playwright scripts through a code executor; they are now commands in surfaces/browser/helpers (CUSTOM_ACTIONS and the batch tool). The controller keeps ping, echo, browser_task and list_upload_files, and the specs of the three custom actions the runtime registers as handlers. Refs: #driver-architecture
Deleted with the script actions: the batch-interact runtime tests (ported to surfaces/test_browser_batch.py), the set-input-files and script-builder cases of test_file_uploads.py and the coordinate LLM-fallback case of test_controller.py (ported to surfaces/test_browser_custom_actions.py, the LLM fallbacks are not ported), and the status-logging test that read the batch script source. The architecture allowlists lose the emptied controller entries. Refs: #driver-architecture
BrowserAgentRuntime owns a DriverSession started through the composition root at the first invoke, where the MCP server was registered before; BrowserRuntimeRail adds the factory's command tools and extra tools to the agent there and registers the parts' rails once per agent. The rail reads a ToolKind (the command decoded from the arguments, or a helper's flags) before a call and the CommandRecord after it, and copies the record's metadata onto the tool message (O5); the text classifiers, the MCP tool lookups, the old probes, batch and custom-action paths and the tool-name canonicalisation are deleted. capture_browser_state reuses the free observation of the last command (Q14). The custom actions are the surface's CUSTOM_ACTIONS, registered on the controller with the tool context. AgentParts carries the factory's builders. Refs: #driver-architecture
The rail and facade tests run on a FakeDriver that serves the Playwright MCP tool dialect (runtime_fixtures.py); inputs are command records and tool kinds, the test names and behaviours stay. Deleted with the old path: the tool-name canonicalisation cases (tools carry their final names), the MCP text classifier and RPC envelope cases (records carry ok, error and value), the runtime's own probe, batch, code-executor and MCP lookup cases (ported to surfaces/test_browser_helpers.py and surfaces/test_browser_batch.py), the single-step primitive case of the runtime tools, and the AX-link URL extraction case. New: the record metadata copied onto the tool message, a policy denial, the free observation reused by a capture, the session started on demand. The seven golden views that offered server tools with no command are regenerated: unsafe_dev, vision, devtools, config, network, storage and testing lose exactly the C7/C20/C25 tools; core, runner, advanced_code, extended_interaction and pdf are unchanged. The factory golden test compares byte for byte again. The core import of the browser helpers by the facade is allowlisted until Phase 9 moves it behind the profile. Refs: #driver-architecture
A driver that cannot type key by key refuses TypeText(slowly=True) before sending anything (browser-use); the batch then types the value in one go without clearing, as the field was emptied first. Refs: #driver-architecture
Refs: #driver-architecture
S_20: the runtime facade and BrowserRuntimeRail (session start at the first invoke, ToolKind, record-based classifiers, argument normalisation by dialect, captures reusing the free observation), AgentParts.builders, the helpers and record rails now wired into the main agent. F_07: the switch's first step, its decisions, the regenerated goldens and what S2 and S3 still owe. Refs: #driver-architecture
The surface factory takes the instance and an explicit settings override (`factory(environ, instance=None, settings=None)`), so `assemble_driver` stops applying `with_instance` by duck typing and a caller's browser settings reach provisioning, the driver and the tool context. `BrowserSettings.mcp_server` carries a server config given by the caller; the Playwright MCP factory keeps its command line under the factory's own name, id and device environment, so the tool prefix stays `mcp_playwright-official_`. `ToolFlags.worker` says whether an extra tool is offered to a delegated worker too; the recall tool is main-agent only. `capabilities_of` moves next to the catalog so the service can derive the worker's capabilities. Refs: #driver-architecture
Refs: #driver-architecture
`stop()` stops the stack and keeps the device; `start()` starts it again on a live device and provisions a new one first when the device died. `health()` asks the stack. The restart on a lost connection reuses `start()`. Refs: #driver-architecture
Refs: #driver-architecture
`BrowserService` holds the `DriverSession`: `ensure_runtime_ready` opens it through `assemble_driver` with the facade's page state, the instance, the service settings (the given `mcp_cfg` as `mcp_server`, its cwd) and `service.core` (the instance's driver, when set, wins). The last task binding stops the stack and keeps the device; the next task starts it again. Gone: the heartbeat loop, `_check_connection`, the client-patch call, the MCP registration and refresh, the service's own provisioning and profile adoption, `_validate_mcp_command`. The worker is built from `build_agent_parts(..., worker=True)`: the core catalog tools, both probes, the batch and find; rails `ActionPolicy`, `CommandRepeatGuardRail`, `CommandInterceptedHintRail` and `FinalAnswerRecoveryRail`. `build_browser_worker_agent` takes `tools` and no MCP config or allowlist; the old `runtime_tools.py` helper classes go. The facade serves the WebVoyager runner on both drivers: Playwright MCP through the worker as before; browser-use through the main agent built on this facade (`create_browser_agent(model, runtime=self, ...)`), run once and mapped to the runner's answer dict. `runtime_health` reports `session.health()`; `shutdown()` closes the session and the model clients. Refs: #driver-architecture
The service tests run on a fake session (`assemble_driver` patched): one session per service, the stack restarted after a release, the worker from the factory with the rails, the given MCP config and the instance reaching the session. The registry, isolation, allowlist and prompt-surface tests follow the service onto the session; the two strict xfails of the record rails flip. The heartbeat, MCP registration, service provisioning and runtime helper-class cases go. Refs: #driver-architecture
Refs: #driver-architecture
Delete RepeatedCallGuardRail and InterceptedActionHintRail on MCP text
(call-log parsing, Events section masking, evaluate name checks) and
rename CommandRepeatGuardRail and CommandInterceptedHintRail to those
names; the guard no longer subclasses the text rail. The retry sentence
of the intercepted-action hint names browser_snapshot, so it moves from
the core to the browser profile's messages ("retry") and the hint rail
takes it from the profile at assembly.
Refs: #driver-architecture
The three MCP-text rail test files go: every test name lives on in the records twins, which now build the rails under the old names and pass the profile's retry sentence to the hint rail. The fake surface and the build_agent profile carry a retry message; the architecture allowlists lose the entries the deleted text rails held. Refs: #driver-architecture
Refs: #driver-architecture
DriverAssembly.surface_settings duplicated surface.settings; the driver session provisions a replacement device from the surface bundle instead. Refs: #driver-architecture
Refs: #driver-architecture
added 30 commits
October 15, 2026 12:21
Refs: #driver-architecture
Refs: #driver-architecture
…data allowlisted Refs: #driver-architecture
The policy approved a relative upload name under the upload root while browser-use resolved it against the process directory first and the Playwright MCP driver against its output directory. commands.upload_paths is now the one resolution rule: an absolute path as given, a relative name only under the upload root and never leaving it, a ValueError the model can act on otherwise. The policy rule and both drivers call it; the process directory is no longer a root for anything. Refs: #driver-architecture
The reviewer's case (allowed/report.txt against outside/report.txt with the process elsewhere) through upload_paths, the policy rule, the rail on both dialects, the browser-use handler with a mocked transport, the Playwright MCP handler with a fake session and set_input_files; a relative symlink out of the root; no upload root configured. Refs: #driver-architecture
Refs: #driver-architecture
offload_session_dir_name renamed the offload folder of an id with unsafe characters, and OffloadMessageBuffer read the new name only, so sessions stored before the renaming were no longer found on Linux and macOS. The buffer now also looks in the literal <session id>_context/offload folder, after the new one, but only for an id without a path separator that is not . or ..; new writes keep the new name. Refs: #driver-architecture
Refs: #driver-architecture
Refs: #driver-architecture
Setting os.name to nt made pathlib build a WindowsPath on macOS and Linux, which fails; the helper reads the real os.name, so the test now runs on Windows only and patches nothing. Refs: #driver-architecture
…rride Refs: #driver-architecture
The script rule is a text match; an alias such as const c = navigator.clipboard; c.readText() passed it. The resource probes/clipboard_guard.js now runs in every document before the page's scripts and rejects clipboard reads and the paste command with the policy message. Playwright MCP passes it with --init-script when it starts a stdio server; the browser-use driver sends it with connect and the sidecar installs it on every CDP session and on the document already open. Both drivers read it as a package resource, not an import. The regex stays as early feedback. The scene gains the two cases for the live suites. Refs: #driver-architecture
The resource is shipped with the policy message and, under node, rejects the aliased read and refuses paste; Playwright MCP starts its server with --init-script; browser-use sends the script with connect and the sidecar installs it once per session; the live suites run the alias and the paste command through RunScript on both drivers. Refs: #driver-architecture
Refs: #driver-architecture
The driver declared TypeText but refused slowly=True. The sidecar now sends one Input.insertText per character with a short pause when the flag is set; the driver passes it through. The scene gains the slow typing case for the live suites. Refs: #driver-architecture
Refs: #driver-architecture
Refs: #driver-architecture
The browser-use sidecar evaluates the guard through its expression wrapper, and the trailing semicolon made that a SyntaxError at connect, which failed every live browser-use case. The script now ends without it. Refs: #driver-architecture
Refs: #driver-architecture
Sessions stored before ids were made path-safe sit under the literal id; the tool looked only under the path-safe name. It now tries that folder first and the literal one after it, under the same safety condition the message buffer uses, keeping every workspace and symlink check. Refs: #driver-architecture
…lder Refs: #driver-architecture
The driver resolves and authorises upload paths before dispatch, so the sidecar's own resolution against the working directory and the upload root was dead code and read the environment for nothing. Refs: #driver-architecture
Refs: #driver-architecture
…'s absolute paths Refs: #driver-architecture
The 2026-10-15 WebVoyager reruns show browser-use's gap to Playwright MCP comes from the Direct topology's budgets, not the driver: the failures rerun through the worker pass at half the cost per task. BROWSER_TOPOLOGY=direct stays the explicit override. Refs: #driver-architecture
The golden builder derives each driver's runner views from its registered default topology, so browser-use now records runner (the worker) and runner_direct (the main agent) like Playwright MCP; the runner golden gains the worker agent and runner_worker.json goes. Refs: #driver-architecture
Refs: #driver-architecture
After the single restart a second connection loss raised the driver_lost error from the tool, which the agent loop handed back to the model as a tool error; the model retried until the task deadline. DriverSession now remembers the terminal loss (lost), the new core rail ConnectionLostRail ends the run before the next model call in both topologies, the worker loop takes that result as the blocked answer and the browser rail marks the task blocked with browser_connection_lost. Refs: #driver-architecture
A fake driver that loses the connection on every command runs through the facade in both topologies: the worker makes two model calls and the direct main agent one, and both answers carry the lost-connection reason. Unit tests for DriverSession.lost, ConnectionLostRail, the worker loop blocked answer and the direct rail terminal marking; the core-rail lists gain the fourth rail. Refs: #driver-architecture
S_20: the lost property and the invariant on the driver session, the core_rails session lookup, the ConnectionLostRail section, the worker loop and browser rail answer paths. F_07: the decision, the rejected options, the Google Map--40 evidence and the verification. Refs: #driver-architecture
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On
developthe browser agent works with one library only. It calls the Playwright MCP server's tools by name in 57 places across three files, guesses that the page changed from the name of the tool that ran, matches that server's error text in its fixes, and replaces the core's MCP client classes at import. A second library means editing the agent, the prompts, the rails and the configuration at once, and the core cannot change its MCP clients without checking the browser package.How
A typed command contract sits between the agent and the browser library. The agent sends commands (
Click,TypeText,Navigate, ...) and reads typed results; a driver translates commands for one library and declares which commands and which kinds of target it handles. The package becomes five layers underopenjiuwen/harness/tools/browser_move/:contract/drivers/playwright_mcp/,browser_use/with its sidecar process, the middleware that wraps every driverprovisioning/surfaces/browser/runtime/The core and the drivers depend only on the contract. The core finds a driver, a surface or a provisioner by name through a registry, so a new driver is one folder and one
register_driverline. The tool factory offers the model only the tools whose commands the selected driver declares. 24 architecture tests read the source and fail on an import that crosses a layer the wrong way, a driver name or a browser word in the core, a module over 1,500 lines, or an environment read outside the settings.Two drivers run as equals: Playwright MCP (the Node server over stdio) and browser-use (the library in its own Python process over a line-based JSON protocol).
BROWSER_DRIVERpicks one. Both pass the same conformance suite on a real Chrome.Safety rules now run in the core before any driver: a policy rail denies paste key chords and clipboard reads in page scripts; uploads and drops are confined to
BROWSER_UPLOAD_ROOTand the agent workspace; sensitive text and script arguments are masked in traces; a command that may have reached the browser is never retried; a lost connection restarts the driver once per task; an element id from an older screen is refused with a fresh observation attached.Design record:
openjiuwen/harness/docs/specs/S_20_browser-driver-contract.md(contracts and invariants) andopenjiuwen/harness/docs/features/F_07_browser-drivers.md(decisions, rejected options, verification per step).What
357 commits on top of the current
developtip (422dc44df); the merge is a fast-forward. Logic changes are code, test, docs triples. Reading by range: contract and drivers (..d062c3173), surface, topologies and renames (..700dbe288), benchmark fixes (..77091ac9e), review fixes (..8deb58b8e), cleanup (..184904618). Supersedes #7, #8, #9, #10 and #11; their commits are in this branch.Changes outside
browser_move:openjiuwen/core/single_agent/agents/react_agent.pyno longer imports the harness for tool-call logging;openjiuwen/core/common/logging/tool_call.py(new) is a registry the browser package registers its logger in.openjiuwen/core/foundation/llm/model_clients/openai_model_client.py,anthropic_model_client.py: the shared client cache is keyed by the running event loop; clients from closed loops are dropped on a cache miss. Fixes "Event loop is closed" on reuse.openjiuwen/core/context_engine/context/context_utils.py,message_buffer.py,processor/offloader/tool_result_budget_processor.py: the offload folder for a session id with path-unsafe characters gets a readable name plus a digest (WinError 123on Windows for ids containing:).openjiuwen/harness/subagents/browser_agent.py: new import paths; optionalruntime=keyword; the browser tools arrive at the first invoke, once the driver has started; aDecisionPolicyModelin the model slot is used as is.openjiuwen/harness/schema/decision_policy.py(new): theDecisionPolicyModelprotocol;S_18andF_05describe it.openjiuwen/harness/rails/subagent/subagent_rail.py,openjiuwen/harness/tools/subagent/task_tool.py: import paths.pyproject.toml: package data for the conformance scene page and the probe script; alivepytest marker.docs/en,docs/zh: module paths in the API reference.Breaking changes:
playwright_runtimepackage (the old name ofruntime/) and the root moduleoffload_recall.pyare removed. Callers change three imports:playwright_runtime.configtoruntime.config,playwright_runtime.runtimetoruntime.session,playwright_runtime.browser_capabilitiestosurfaces.browser.capabilities.bench/webvoyager/run.py, the imports around lines 132 to 140) uses all three old paths and fails at start until they are updated. The benchmark numbers below were produced before the removal, with the old paths. The runner needs the three-line change in the same release.BrowserPageStatemoved fromruntime/page_state.pytosurfaces/browser/page_state.py;runtime/page_state.pykeeps the surface-neutralPageState.BROWSER_DRIVERnow names a driver (playwright_mcp,browser_use, or any registered name). The old valuesmanaged,remoteandextensionmap toBROWSER_PROVISIONINGwith a deprecation warning.Unchanged: the
BrowserAgentRuntimeconstructor andrun_browser_task(task, session_id, request_id, timeout_s); the signature ofcreate_browser_agent(one optional keyword added);RuntimeSettings;BrowserInstanceConfig.Known limits, listed in
S_20andF_07: the browser runner is browser-only, so a second surface builds its agent through the composition root; one allowlisted core-to-surface import in the direct topology; provisioner names are global; a screenshot scale other than 1 is untested.Verification
2,532 passed, 123 skipped. Test functions in the source: 1,388 (457 on
develop). Recorded tool lists and prompts per driver and capability view are golden files; any change to what the model sees fails a test. A test-only screen surface runs an agent task through the core with no browser code.Live suites, on a local Chrome:
100 passed: the conformance suite on both drivers and the topology suite.
WebVoyager, 528 valid tasks (others excluded as benchmark-side failures: judge errors, delivery region, blocked sites). Sonnet 5.5, temperature 1, 50 steps, 600 s per task, gpt-4o judge.
develop, one pass