Repository navigation
[Fix] Recover the final answer when the worker ends without JSON - #11
Closed
sirotaoleg wants to merge 3 commits into
Closed
sirotaoleg wants to merge 3 commits into
sirotaoleg wants to merge 3 commits into
Conversation
added 3 commits
September 30, 2026 16:36
When a Playwright MCP action fails because another element sits on top of the target, the worker model got the full Playwright call log: 1.9-5.6 KB of ANSI-coloured retry noise in which the blocker is one line. On WebVoyager (@playwright/mcp 0.0.78, agent-core 18b840c) 12 of 90 tasks hit "element intercepts pointer events" (18 failures, each after a 5 s timeout), and it took 4.5 steps on average (54 in total, worst 8-9) to get from the failed click to the next successful one. Blockers were the OneTrust consent dialog (Coursera, Cambridge, Allrecipes), Amazon's redirect modal, Amazon's radio icon over its own input, and a Google overlay. Add InterceptedActionHintRail, registered on the Playwright worker by BrowserService.ensure_started(). Its after_tool_call parses the call log (target from the last "locator resolved to", blocker and container from the last "... intercepts pointer events" line) and replaces only the model-facing ToolMessage content with a short error, keeping the {"result": ...} envelope and the "### Error" marker: before (2.4 KB, shortened): {"result": "### Error\nTimeoutError: browserBackend.callTool: Timeout 5000ms exceeded.\nCall log:\n \u001b[2m- waiting for locator(...) ... - <div class=\"onetrust-pc-dark-filter ot-fade-in\"></div> from <div ... id=\"onetrust-consent-sdk\">…</div> subtree intercepts pointer events ... - retrying click action ... (x24)"} after (394 chars): ### Error Click not delivered: button.css-bnpmeq[type="button"][data-testid= "filter-and-sort-button"] is covered by div.onetrust-pc-dark-filter (inside div#onetrust-consent-sdk). This is a cookie-consent dialog: close it first (prefer 'Reject all' / 'Close'; if its settings panel is open, confirm or close that panel). Then retry the action (take a fresh browser_snapshot if the page changed). Non-consent blockers get a generic hint (dismiss the overlay, or click the control's visible label when an icon covers its input). Detection is by content, so every action tool (click, hover, type, fill_form, select_option, drag, ...) is covered. Content that does not mention an intercepted action, or whose call log cannot be parsed, is left byte-identical. A rail rather than a wrapper around ability_manager.execute: S_05 (tools contract, invariant 11) makes AFTER_TOOL_CALL the place that rewrites tool_msg, and resolve_tool_message/rendered_result read that message, so what is streamed and what the model reads stay the same text. The structured tool_result is untouched, so tool_result_observer (progress and failure tracking) still receives the raw result. Out of scope: auto-dismissing consent banners, a blocked_by field in the probe tools, the Amazon geo redirect, and porting to the refactor layout (browser_move/runtime/..., legacy MCP path).
On a full WebVoyager run (556 tasks, @playwright/mcp 0.0.78, Claude
Sonnet via OpenRouter, 50 steps and 600 s per task), 107 tasks made 5+
identical consecutive tool calls, and 49 of the 59 max_iterations
failures were exactly this loop. The model usually has the answer
already: in Allrecipes--0 it had the recipe (4.6 stars, 117 ratings,
serves 6) by step 15, then called browser_evaluate("() => 'ok'") 35
times until the step limit. Common loop bodies: () => 1,
() => document.title, () => location.href; later also varying counters
('done19', 'done20', ... or ()=>34, ()=>35, ...). Raising sampling to
temperature 1.0 / top_p 1.0 reduced the loops but did not remove them,
so the runtime needs a guard regardless of model and sampling.
Add RepeatedCallGuardRail, registered on the Playwright worker in
BrowserService.ensure_started(). Per worker run it counts two streaks:
- identical calls with identical results: key = tool name without the
MCP prefix + canonical JSON arguments; fingerprint = sha256 of the
model-facing content without Playwright's volatile parts (### Events,
"New console entries", console counts). For browser_evaluate the
function is compared without whitespace and with digit runs masked;
other tools compare exactly (clicking e12 is not clicking e13);
- no-op evaluates: browser_evaluate functions that only return literals,
document.title, document.URL, location.href or a flat object/array of
those, whatever their arguments. Anything that reads page content is
not a no-op.
From the 3rd call of a streak the tool message gets a short
"### Repeated call" note (inside the {"result": ...} envelope when there
is one, never twice). After the 5th, before_model_call sends the next
request with no tools and one prompt section asking for the final JSON,
the way BrowserRuntimeRail._prepare_terminal_synthesis does. ReActAgent
runs BEFORE_MODEL_CALL rails before building the context window from
ctx.inputs.tools, so the request really carries no tools (asserted in
the test). State lives in ctx.extra and is reset in before_invoke.
The structured tool_result and what tool_result_observer receives are
unchanged. Replaying the rule over 375 logged Fable trajectories gives 0
hits. The registration tests of this rail and InterceptedActionHintRail
search all register_rail calls instead of only the last one.
Out of scope: worker sampling defaults, loops across different tools,
and porting to the refactor layout (browser_move/runtime/...).
The Playwright worker must end with a JSON object (ok, final, page,
screenshot, error, ...). When it does not, BrowserService._run_task_once
throws away whatever the model had and returns ok=false. On a full
WebVoyager run (556 tasks, @playwright/mcp 0.0.78, Claude Sonnet via
OpenRouter, 50 steps) 48 runs ended without valid JSON and 59 hit
max_steps, whose final is the literal "Max iterations reached without
completion" - even in Allrecipes--0, where the answer was found by step
15. On a re-run with the repeated-call guard, three runs whose tools had
been withheld got completion_tokens=1 / finish_reason=end_turn with
empty content: the only instruction was a system-prompt section and no
user turn followed the last tool result. Google Search--14 answered in
markdown prose instead of JSON.
Make one recovery model call when the worker output
1. is empty (reason "empty_output"),
2. is non-JSON prose ("non_json_output"), or
3. is the max-iterations message ("max_iterations_reached").
Valid JSON, a raised run, cancellation and timeouts are unchanged; the
task deadline around run_task_once still bounds the recovery.
The recovery is Runner.run_agent on the same worker with the same
worker_conversation_id, so the checkpointed conversation (task, tool
calls, results) is restored, with a short user message as the query:
"Tools are disabled now. Using only what you have already seen in this
conversation, reply with the final JSON object ...", prefixed with "The
step limit has been reached." for case 3. A user turn after the last
tool result is what reliably gets this model to answer. The new invoke
starts its own iteration count, so max_steps does not block it.
FinalAnswerRecoveryRail (registered in BrowserService.ensure_started())
withholds tools in before_model_call only while a ContextVar set around
the recovery invoke is active, so concurrent runs never see each
other's flag. If the model still emits a tool call, after_model_call
force-finishes instead of executing it: exactly one model call.
If the recovery output parses as JSON it is returned through the normal
path with "recovered_from": <reason>, which run_task keeps in its
response; ok is whatever the model said. Otherwise the result is
exactly today's (same ok=false, error and status), with the best text
in final. A recovery that raises is logged and also falls back to
today's result.
On a 47-task WebVoyager re-run the recovery fired 6 times and produced
valid JSON 5 times, including all 3 empty replies after withheld tools.
Out of scope: the worker prompt, MCP args, sampling, step limits and
retry/resume policy; the outer runtime's terminal synthesis; the
WebVoyager judge.
Author
|
Superseded by #12 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem. The worker must end with a JSON object. When it doesn't,
_run_task_oncethrows away what the model found and returnsok=false. Three cases:end_turn).Fix. In those three cases, make one extra model call in the same conversation, with no tools and a short user message: "Tools are disabled now … reply with the final JSON …". A user turn after the last tool result is what reliably gets the model to answer.
FinalAnswerRecoveryRailwithholds tools only while a per-request ContextVar is set, so concurrent runs never interfere.recovered_from = empty_output | non_json_output | max_iterations_reached.okis whatever the model said.final.run_taskkeepsrecovered_fromin its response (it rebuilt the response from fixed keys and dropped it).Result (47-task targeted run):
Tests.
test_browser_final_answer_recovery.py:recovered_fromsurvivesrun_task.