Skip to content

fix(llm): normalize message part order for Vertex AI Gemini tool turns (fixes #156) - #186

Open
VG31OP wants to merge 2 commits into
google:mainfrom
VG31OP:fix/vertex-gemini-turn-part-order
Open

VG31OP wants to merge 2 commits into
google:mainfrom
VG31OP:fix/vertex-gemini-turn-part-order

Conversation

@VG31OP

@VG31OP VG31OP commented Oct 16, 2026

Copy link
Copy Markdown

Fixes #156.

Problem

In ARTEMIS's mobile workflows using Vertex AI Gemini (e.g. gemini-3.8-flash), the first model turn succeeds and executes a UI action. However, when ARTEMIS sends the follow-up request with the tool result and the next screen observation, Vertex AI returns:

400 Requests ending with a model turn are not supported

This occurs because ARTEMIS appends a ToolMessage (tool execution result) followed by a HumanMessage (new screen observation) to its conversation history. LangChain's Vertex serializer (_parse_chat_history_gemini) converts ToolMessage to functionResponse and merges consecutive user-turn messages into a single Content(role="user") object with parts ordered as [functionResponse, text]. Vertex Gemini 3.x rejects this part ordering.

Solution

  1. Outbound Normalization (normalize_vertex_messages):
    • In artemis/services/llm.py, produces a normalized copy of the message list for Vertex AI invocations where HumanMessage observations in a tool-calling turn are positioned before ToolMessage results.
    • Populates ToolMessage.name from the preceding AIMessage.tool_calls map if missing.
    • Preserves the underlying stored conversation history in TranscriptLedger and State without mutation.
  2. Gateway Integration:
    • Updated RobustChatModelWrapper (complete(), ainvoke(), invoke()) and acomplete() to automatically detect Vertex AI endpoints (is_vertex_provider or ChatVertexAI class name) and apply the normalization before request execution.
    • Other providers (Google Developer API, OpenAI, Anthropic, etc.) remain completely untouched.
  3. Provider Helper:
    • Added is_vertex_provider() in artemis.llm.google to cleanly check for Vertex AI provider configurations.

Validation

  • New Regression Tests: Added tests/unit/test_vertex_message_normalizer.py (7 tests) covering:
    • Preserving empty and non-tool conversation histories.
    • Reordering screen observation text before tool responses.
    • Multi-tool bursts and multi-turn workflows.
    • LangChain _parse_chat_history_gemini part ordering verification ([text, function_response]).
    • RobustChatModelWrapper and acomplete gateway integration.
    • Non-Vertex provider isolation.
  • Unit Test Suite: 61/61 tests passed across test_vertex_message_normalizer.py, test_google_provider.py, and test_llm_gateway.py.
  • Linting & Formatting: ruff check and ruff format --check passed cleanly across all changed files.

fixes google#156)

In Vertex AI, multi-turn tool-calling requests fail with HTTP 400
('Requests ending with a model turn are not supported') when a tool result
(ToolMessage) is followed by new screen observation text (HumanMessage).
LangChain's Vertex serializer merges consecutive user-turn messages into
a single Content object with parts ordered as [functionResponse, text],
which Vertex Gemini 3.x rejects.

This normalizes outbound message copies in RobustChatModelWrapper and
acomplete for Vertex AI endpoints by positioning HumanMessage observations
before ToolMessages within tool turns, yielding [text, functionResponse]
part ordering and populating tool names on ToolMessages from preceding
AIMessage tool calls. The underlying stored conversation history remains
intact and unchanged.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces message normalization for Vertex AI Gemini models to prevent HTTP 400 errors caused by incorrect ordering of tool responses and user observations. It adds a normalize_vertex_messages utility, integrates it into the robust chat model wrapper and completion functions, and includes comprehensive unit tests. The review feedback suggests using getattr to safely access tool_call_id on messages to avoid potential AttributeErrors, and extracting the duplicated input preparation logic into a static helper method to improve maintainability.

Comment thread artemis/services/llm.py
Comment thread artemis/services/llm.py
Comment thread artemis/services/llm.py Outdated
…s tool_call_id

Addresses review comments from @gemini-code-assist:
- Extract duplicated input preparation logic into RobustChatModelWrapper._prepare_vertex_inputs static method
- Use getattr(m, 'tool_call_id', None) on ToolMessage to avoid potential AttributeError with mock objects
@VG31OP

VG31OP commented Oct 16, 2026 •

Copy link
Copy Markdown
Author

Thank you for the review! I've addressed the feedback in commit 83b4445:

  1. Extracted Static Helper: Extracted RobustChatModelWrapper._prepare_vertex_inputs to eliminate duplicated input preparation logic between RobustChatModelWrapper._prepare_inputs and acomplete().
  2. Safe Attribute Access: Used getattr(m, 'tool_call_id', None) when inspecting ToolMessages in normalize_vertex_messages to prevent potential AttributeErrors on mock or custom message objects.
  3. Validation: All 61/61 unit tests pass cleanly alongside ruff check and ruff format --check.

@VG31OP

VG31OP commented Oct 18, 2026

Copy link
Copy Markdown
Author

@gemini-code-assist please review the latest updates in commit 83b4445.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vertex Gemini 3.8 Flash fails after the first ARTEMIS tool-calling turn

1 participant