Repository navigation
[Fix] Tell the model which element blocked its click - #9
Closed
sirotaoleg wants to merge 1 commit into
Closed
sirotaoleg wants to merge 1 commit into
sirotaoleg wants to merge 1 commit into
Conversation
When a Playwright MCP action fails because another element sits on top of the target, the worker model got the full Playwright call log: 1.9-5.6 KB of ANSI-coloured retry noise in which the blocker is one line. On WebVoyager (@playwright/mcp 0.0.78, agent-core 18b840c) 12 of 90 tasks hit "element intercepts pointer events" (18 failures, each after a 5 s timeout), and it took 4.5 steps on average (54 in total, worst 8-9) to get from the failed click to the next successful one. Blockers were the OneTrust consent dialog (Coursera, Cambridge, Allrecipes), Amazon's redirect modal, Amazon's radio icon over its own input, and a Google overlay. Add InterceptedActionHintRail, registered on the Playwright worker by BrowserService.ensure_started(). Its after_tool_call parses the call log (target from the last "locator resolved to", blocker and container from the last "... intercepts pointer events" line) and replaces only the model-facing ToolMessage content with a short error, keeping the {"result": ...} envelope and the "### Error" marker: before (2.4 KB, shortened): {"result": "### Error\nTimeoutError: browserBackend.callTool: Timeout 5000ms exceeded.\nCall log:\n \u001b[2m- waiting for locator(...) ... - <div class=\"onetrust-pc-dark-filter ot-fade-in\"></div> from <div ... id=\"onetrust-consent-sdk\">…</div> subtree intercepts pointer events ... - retrying click action ... (x24)"} after (394 chars): ### Error Click not delivered: button.css-bnpmeq[type="button"][data-testid= "filter-and-sort-button"] is covered by div.onetrust-pc-dark-filter (inside div#onetrust-consent-sdk). This is a cookie-consent dialog: close it first (prefer 'Reject all' / 'Close'; if its settings panel is open, confirm or close that panel). Then retry the action (take a fresh browser_snapshot if the page changed). Non-consent blockers get a generic hint (dismiss the overlay, or click the control's visible label when an icon covers its input). Detection is by content, so every action tool (click, hover, type, fill_form, select_option, drag, ...) is covered. Content that does not mention an intercepted action, or whose call log cannot be parsed, is left byte-identical. A rail rather than a wrapper around ability_manager.execute: S_05 (tools contract, invariant 11) makes AFTER_TOOL_CALL the place that rewrites tool_msg, and resolve_tool_message/rendered_result read that message, so what is streamed and what the model reads stay the same text. The structured tool_result is untouched, so tool_result_observer (progress and failure tracking) still receives the raw result. Out of scope: auto-dismissing consent banners, a blocked_by field in the probe tools, the Amazon geo redirect, and porting to the refactor layout (browser_move/runtime/..., legacy MCP path).
Author
|
Superseded by #12 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem. When a click lands on an element covered by something else (a cookie dialog, a modal, an icon over its own input), Playwright MCP returns its full retry log: 1.9–5.6 KB of ANSI-coloured noise, in which the actual blocker is one line.
Fix. A new
InterceptedActionHintRailon the worker (after_tool_call, the documented place for rewritingtool_msg, S_05 invariant 11) turns that log into a short message: what was clicked, what covers it, and what to do. For consent dialogs that means "close it first, prefer Reject all / Close". Example (2.4 KB → 394 chars):Details:
Result. Steps from the failed click to recovery: 3.8 → 2.7 on the same tasks.
Tests.
test_browser_intercepted_action_hint.py:{"result": ...}envelope kept;