Repository navigation
[Fix] Stop the browser worker from looping on no-progress calls - #10
Closed
sirotaoleg wants to merge 2 commits into
Closed
sirotaoleg wants to merge 2 commits into
sirotaoleg wants to merge 2 commits into
Conversation
added 2 commits
September 30, 2026 16:36
When a Playwright MCP action fails because another element sits on top of the target, the worker model got the full Playwright call log: 1.9-5.6 KB of ANSI-coloured retry noise in which the blocker is one line. On WebVoyager (@playwright/mcp 0.0.78, agent-core 18b840c) 12 of 90 tasks hit "element intercepts pointer events" (18 failures, each after a 5 s timeout), and it took 4.5 steps on average (54 in total, worst 8-9) to get from the failed click to the next successful one. Blockers were the OneTrust consent dialog (Coursera, Cambridge, Allrecipes), Amazon's redirect modal, Amazon's radio icon over its own input, and a Google overlay. Add InterceptedActionHintRail, registered on the Playwright worker by BrowserService.ensure_started(). Its after_tool_call parses the call log (target from the last "locator resolved to", blocker and container from the last "... intercepts pointer events" line) and replaces only the model-facing ToolMessage content with a short error, keeping the {"result": ...} envelope and the "### Error" marker: before (2.4 KB, shortened): {"result": "### Error\nTimeoutError: browserBackend.callTool: Timeout 5000ms exceeded.\nCall log:\n \u001b[2m- waiting for locator(...) ... - <div class=\"onetrust-pc-dark-filter ot-fade-in\"></div> from <div ... id=\"onetrust-consent-sdk\">…</div> subtree intercepts pointer events ... - retrying click action ... (x24)"} after (394 chars): ### Error Click not delivered: button.css-bnpmeq[type="button"][data-testid= "filter-and-sort-button"] is covered by div.onetrust-pc-dark-filter (inside div#onetrust-consent-sdk). This is a cookie-consent dialog: close it first (prefer 'Reject all' / 'Close'; if its settings panel is open, confirm or close that panel). Then retry the action (take a fresh browser_snapshot if the page changed). Non-consent blockers get a generic hint (dismiss the overlay, or click the control's visible label when an icon covers its input). Detection is by content, so every action tool (click, hover, type, fill_form, select_option, drag, ...) is covered. Content that does not mention an intercepted action, or whose call log cannot be parsed, is left byte-identical. A rail rather than a wrapper around ability_manager.execute: S_05 (tools contract, invariant 11) makes AFTER_TOOL_CALL the place that rewrites tool_msg, and resolve_tool_message/rendered_result read that message, so what is streamed and what the model reads stay the same text. The structured tool_result is untouched, so tool_result_observer (progress and failure tracking) still receives the raw result. Out of scope: auto-dismissing consent banners, a blocked_by field in the probe tools, the Amazon geo redirect, and porting to the refactor layout (browser_move/runtime/..., legacy MCP path).
On a full WebVoyager run (556 tasks, @playwright/mcp 0.0.78, Claude
Sonnet via OpenRouter, 50 steps and 600 s per task), 107 tasks made 5+
identical consecutive tool calls, and 49 of the 59 max_iterations
failures were exactly this loop. The model usually has the answer
already: in Allrecipes--0 it had the recipe (4.6 stars, 117 ratings,
serves 6) by step 15, then called browser_evaluate("() => 'ok'") 35
times until the step limit. Common loop bodies: () => 1,
() => document.title, () => location.href; later also varying counters
('done19', 'done20', ... or ()=>34, ()=>35, ...). Raising sampling to
temperature 1.0 / top_p 1.0 reduced the loops but did not remove them,
so the runtime needs a guard regardless of model and sampling.
Add RepeatedCallGuardRail, registered on the Playwright worker in
BrowserService.ensure_started(). Per worker run it counts two streaks:
- identical calls with identical results: key = tool name without the
MCP prefix + canonical JSON arguments; fingerprint = sha256 of the
model-facing content without Playwright's volatile parts (### Events,
"New console entries", console counts). For browser_evaluate the
function is compared without whitespace and with digit runs masked;
other tools compare exactly (clicking e12 is not clicking e13);
- no-op evaluates: browser_evaluate functions that only return literals,
document.title, document.URL, location.href or a flat object/array of
those, whatever their arguments. Anything that reads page content is
not a no-op.
From the 3rd call of a streak the tool message gets a short
"### Repeated call" note (inside the {"result": ...} envelope when there
is one, never twice). After the 5th, before_model_call sends the next
request with no tools and one prompt section asking for the final JSON,
the way BrowserRuntimeRail._prepare_terminal_synthesis does. ReActAgent
runs BEFORE_MODEL_CALL rails before building the context window from
ctx.inputs.tools, so the request really carries no tools (asserted in
the test). State lives in ctx.extra and is reset in before_invoke.
The structured tool_result and what tool_result_observer receives are
unchanged. Replaying the rule over 375 logged Fable trajectories gives 0
hits. The registration tests of this rail and InterceptedActionHintRail
search all register_rail calls instead of only the last one.
Out of scope: worker sampling defaults, loops across different tools,
and porting to the refactor layout (browser_move/runtime/...).
Author
|
Superseded by #12 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem. After the worker already has the answer, it sometimes keeps calling a tool that does nothing, until it hits the step limit:
browser_evaluate("() => 1"),() => document.title,() => 'ok';'done19','done20', … or()=>34,()=>35, ….Fix.
RepeatedCallGuardRailcounts, per worker run:browser_evaluate, digits are ignored, so'done19'='done20');From the 3rd call in such a streak, the tool message gets a short note: "this repeats, answer now or do something different". After the 5th, the next model call is sent without tools, and one prompt section asks for the final JSON.
ctx.extra) and is reset inbefore_invoke.Result.
Tests.
test_browser_repeated_call_guard.py:Note: the worker's default sampling (temperature 0.2 / top_p 0.1) is not changed here. Raising it clearly reduced loops for Sonnet, but that is a separate decision that needs a check on other models.