[LiteLLM] `_get_content` encodes audio `inline_data` as `audio_url` which is not a valid LiteLLM content type

[benjamin-callonnec]

Summary

When an ADK agent backed by LiteLlm receives a message containing an audio inline_data part, the _get_content function in lite_llm.py encodes it as {"type": "audio_url", "audio_url": {"url": "data:audio/...;base64,..."}}. This content type does not exist in the LiteLLM / OpenAI message schema. As a result, the audio data is silently dropped by the provider and the model receives an empty message.

Environment

• google-adk version: 1.31.0
• litellm version: 1.82.6
• Provider: any LiteLLM-compatible endpoint (tested with a custom OpenAI-compatible gateway)
• Python version: 3.11

Steps to Reproduce

1. Create an Agent backed by LiteLlm:

from google.adk.agents import Agent
from google.adk.models.lite_llm import LiteLlm

agent = Agent(
    model=LiteLlm(model="openai/gpt-4o-audio-preview"),
    instruction="You are a helpful assistant.",
)

2. Send a message containing an audio inline_data part (e.g. via the /run HTTP endpoint or runner.run_async):

message = {
    "role": "user",
    "parts": [
        {"text": "What is said in this audio?"},
        {
            "inline_data": {
                "mime_type": "audio/mp4",
                "data": "<base64-encoded audio bytes>",
            }
        },
    ],
}

3. Observe that the model response indicates no audio was received (e.g. "It looks like you forgot to attach the audio file").

Root Cause

The wrong mapping

In lite_llm.py, a lookup table maps MIME type prefixes to LiteLLM content type strings:

# lite_llm.py ~L102
_MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE = {
    "image": "image_url",
    "video": "video_url",
    "audio": "audio_url",   # ← does not exist in LiteLLM
}

How it is used

In _get_content (~L836), when an inline_data part is encountered, the code builds a data URI and dispatches on this mapping:

base64_string = base64.b64encode(part.inline_data.data).decode("utf-8")
data_uri = f"data:{mime_type};base64,{base64_string}"

url_content_type = _media_url_content_type(mime_type)  # returns "audio_url"
if url_content_type:
    content_objects.append({
        "type": url_content_type,            # "audio_url"
        url_content_type: {"url": data_uri}, # "audio_url": {"url": "data:audio/mp4;base64,..."}
    })

This produces:

{
  "type": "audio_url",
  "audio_url": { "url": "data:audio/mp4;base64,..." }
}

What LiteLLM actually expects

The LiteLLM ChatCompletionUserMessage schema defines the following audio content block (mirroring the OpenAI spec):

{
  "type": "input_audio",
  "input_audio": {
    "data": "<base64-encoded bytes (no data URI prefix)>",
    "format": "mp3" | "wav"
  }
}

There are two mismatches:

	ADK output	LiteLLM expectation
type field	"audio_url"	"input_audio"
Payload key	"audio_url"	"input_audio"
Data format	Full data URI (data:audio/...;base64,…)	Raw base64 string + separate "format" field

audio_url is not a recognised content type in the LiteLLM schema, so providers either reject the block or silently ignore it.

Expected Behaviour

Audio inline_data parts should be serialised as input_audio content blocks:

{
  "type": "input_audio",
  "input_audio": {
    "data": "<raw base64 string>",
    "format": "mp3"
  }
}

Suggested Fix

Replace the "audio" entry in _MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE with dedicated handling in _get_content, similar to how file content is handled separately:

# Remove "audio" from the URL mapping
_MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE = {
    "image": "image_url",
    "video": "video_url",
    # "audio" intentionally omitted — handled separately as input_audio
}

# In _get_content, add a branch before the url_content_type check:
elif mime_type.startswith("audio/"):
    # Extract format from mime type (e.g. "audio/mp3" -> "mp3", "audio/mp4" -> "mp4")
    audio_format = mime_type.split("/", 1)[1].split(";")[0]
    # Normalise: mp4/m4a containers are typically mp3-compatible; adjust as needed
    content_objects.append({
        "type": "input_audio",
        "input_audio": {
            "data": base64_string,   # raw base64, no data URI prefix
            "format": audio_format,
        },
    })

References

• _get_content function: google/adk/models/lite_llm.py
• _MEDIA_URL_CONTENT_TYPE_BY_MAJOR_MIME_TYPE mapping: lite_llm.py ~L102
• LiteLLM ChatCompletionAudioObject schema: input_audio with data (base64) and format fields