Skip to content

feat(google): map the grounding image parameter to the google_search tool - #588

Merged
Kamilbenkirane merged 3 commits into
mainfrom
feat/google-image-grounding
Nov 9, 2026
Merged

Kamilbenkirane merged 3 commits into
mainfrom
feat/google-image-grounding

Conversation

@Kamilbenkirane

Copy link
Copy Markdown
Member

Google image models never received the unified grounding image parameter added in #513: no Google images mapper wrote it, so grounding=True raised UnsupportedParameterWarning ("Parameter 'grounding' is not supported by model 'gemini-3.1-flash-image' and will be ignored.") and was dropped.

Google documents Search grounding for image generation:

  • Nano Banana image generation (Interactions): "Use the Google Search tool to generate images based on real-time information, such as weather forecasts, stock charts, or recent events", with "tools": [{"type": "google_search"}], and "Grounding with Google Search is not supported by the gemini-3.1-flash-lite-image model."
  • The model pages mark Search grounding supported for gemini-3-pro-image, gemini-3.1-flash-image and gemini-nano-banana-2.1, and not supported for gemini-3.1-flash-lite-image and gemini-2.5-flash-image.
  • Interactions API reference (POST /v1beta/interactions): GoogleSearch, "A tool that can be used by the model to search Google", type: google_search, optional search_types.
  • Grounding with Google Search on Vertex AI lists Gemini Nano Banana 2.1, Gemini 3.1 Flash Image and Gemini 3 Pro Image as supported models; the request carries a googleSearch tool. The GenerateContent reference defines Tool.googleSearch ("If not set, web search is enabled by default").

Fix:

  • src/celeste/providers/google/interactions/parameters.py and src/celeste/providers/google/generate_content/parameters.py: a GoogleSearchMapper per wire. When the value is true it appends the existing WebSearchMapper().map_tool(WebSearch()) output to tools: {"type": "google_search"} on Interactions, {"google_search": {}} on GenerateContent, the same shape the text modality sends for Tool.WEB_SEARCH. None and False write nothing, because Google's default is no search tool.
  • src/celeste/modalities/images/providers/google/parameters.py: InteractionsGroundingMapper and VertexGroundingMapper bind ImageParameter.GROUNDING, appended to GOOGLE_INTERACTIONS_PARAMETER_MAPPERS and GOOGLE_VERTEX_PARAMETER_MAPPERS.
  • src/celeste/modalities/images/providers/google/models.py: ImageParameter.GROUNDING: Bool() on the three documented models, the same constraint flux-3-image uses. The two unsupported models stay unconstrained, so the value passes through for Google to reject, by design.
  • tests/unit_tests/test_parameter_wire_contracts.py: a True and a False wire row for each backend list.
  • CHANGELOG.md: one line under Features.

Not in this PR:

  • True enables web search only. Image Search (search_types: ["web_search", "image_search"]) exists only on 3.1 Flash Image and Nano Banana 2.1, so mapping it would make grounding model-dependent. It stays reachable through extra_body.
  • ImageOutput.grounding stays None. The search steps (google_search_result.search_suggestions) and groundingMetadata.searchEntryPoint are in the response but not surfaced: metadata["raw_response"] drops steps / candidates as content fields. Google requires displaying the search suggestions for grounded results (terms). Parsing them for images, reusing map_grounding_interactions / map_grounding_vertex, is a parsing change that needs your approval first.

Validation:

  • make ci passes (767 unit tests).
  • Baseline on main: _build_request(..., grounding=True) for gemini-3.1-flash-image emits UnsupportedParameterWarning and the body has no tools.
  • Live, Interactions (POST /v1beta/interactions, gemini-3.1-flash-image, quality="512"), through client.generate with the raw response captured at _make_request. Prompt: "Visualize the current weather forecast for the next 5 days in San Francisco as a clean, modern weather chart with the dates and temperatures."
    • grounding=True: request tools: [{"type": "google_search"}], no warning. The response has google_search_call (queries: ["current 5 day weather forecast in San Francisco", ...], search_type: web_search) and google_search_result (with search_suggestions) steps, and usage.grounding_tool_count: [{"type": "google_search", "count": 2}]. The image is dated "Monday, November 9, 2026" (today) through Friday, Nov 13, with Veterans Day on Nov 11.
    • Omitted: no tools, steps model_output/thought only, no grounding_tool_count. The image shows stale dates, "Oct 26 – Oct 30".
  • Live, the Vertex backend body against the GenerateContent wire: GoogleVertexImagesClient with an API key routes to Gemini API POST /v1beta/models/gemini-3.1-flash-image:generateContent, the same wire Vertex serves.
    • grounding=True: request tools: [{"google_search": {}}]. The candidate carries groundingMetadata with webSearchQueries ("current weather forecast San Francisco 5 days", ...), groundingChunks (weather.gov, accuweather.com) and searchEntryPoint. The image is dated Nov 10 – Nov 14, 2026.
    • Omitted: no groundingMetadata.
  • No live Vertex call: the local ADC project lacks aiplatform.endpoints.predict (403 on publishers/google/models/gemini-3.1-flash-image).

@Kamilbenkirane
Kamilbenkirane merged commit f90d9a5 into main Nov 9, 2026
10 checks passed
@Kamilbenkirane
Kamilbenkirane deleted the feat/google-image-grounding branch November 9, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant