Skip to content

feat(google): add gemini-3.5-transcribe to Google audio transcription - #589

Open
Kamilbenkirane wants to merge 3 commits into
mainfrom
feat/google-audio-transcribe
Open

Kamilbenkirane wants to merge 3 commits into
mainfrom
feat/google-audio-transcribe

Conversation

@Kamilbenkirane

Copy link
Copy Markdown
Member

Google audio had no Operation.TRANSCRIBE, so gemini-3.5-transcribe (GA Aug 26, 2026) could not be used through celeste. The transcription guide says "The Gemini API converts speech in audio files into text using the Gemini 3.5 Transcribe model (gemini-3.5-transcribe)". It runs on the same Interactions endpoint (POST /v1beta/interactions) that Google audio already uses for speak and generate. The Interactions reference defines generation_config.transcription_config: "Optional. Configuration for speech recognition (transcription). If present, ASR is enabled". Its language_codes field is "Optional. BCP-47 language codes providing hints about the languages present in the audio. If omitted or empty, defaults to automatic language detection." The guide says "The complete transcript text is returned in response.output_text", which is the text parts of the model_output step.

Fix:

  • src/celeste/modalities/audio/providers/google/client.py, GoogleAudioClient:
    • _transcribe_endpoint is the Interactions endpoint already used by _speak_endpoint and _generate_endpoint. Same wire format, so this is an endpoint ClassVar, not a dispatcher.
    • _init_request follows OpenAIAudioClient: one AudioArtifact, sent as an audio content part through the existing build_content_part helper, with generation_config.transcription_config set.
    • _parse_content returns the joined model_output text when the response carries no audio part, the same comprehension as GoogleInteractionsTextClient. Audio responses parse as before.
  • src/celeste/providers/google/interactions/parameters.py, LanguageMapper: on a request carrying transcription_config, language goes to transcription_config.language_codes; otherwise it goes to speech_config as before. It goes through the same locale_map, so "en" becomes "en-US", a code from Google's transcription language table.
  • src/celeste/modalities/audio/providers/google/models.py: gemini-3.5-transcribe with {AUDIO: {TRANSCRIBE}} and an AudioConstraint listing the eight celeste mime types the Interactions audio part accepts (AAC, AIFF, FLAC, M4A, MP3, OGG, WAV, WEBM).
  • Only language is mapped. It is the only unified audio parameter with a transcription_config field.
    • temperature is not in the reference's GenerationConfig.
    • A text prompt part is ignored live: "Transcribe in uppercase." left the transcript unchanged.
    • custom_vocabulary and mode (smart, diarization, word timestamps) have no unified parameter. They work through extra_body, and the word annotations come back in metadata["text_blocks"].
  • Tests:
    • tests/unit_tests/test_google_audio_transcribe.py (copied from the Mistral one): _init_request payload test.
    • tests/unit_tests/test_parameter_wire_contracts.py: a language → transcription_config.language_codes row.
    • tests/integration_tests/audio/test_transcribe.py: a (GOOGLE, "gemini-3.5-transcribe") case.

Conflict recorded: the Interactions reference's model enum does not list gemini-3.5-transcribe, but the transcription guide uses it and live serving accepts it. Live serving wins.

Validation:

  • make ci passes (765 unit tests).
  • Live, through celeste's client: client.transcribe(test_audio) on the existing tests/integration_tests/text/assets/test_audio.mp3 fixture (10 s):
    • gemini-3.5-transcribe → TextOutput: "This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity". Usage 251 in / 30 out; finish reason completed.
    • Reference: OpenAI whisper-1 on the same file gives the same words: "This week I traveled to Chicago to deliver my final farewell address to the nation following in the tradition of presidents before me. It was an opportunity". Only the punctuation differs.
    • language="en" sends {"transcription_config": {"language_codes": ["en-US"]}} and returns the same transcript.
  • Live wire probes:
    • An unknown transcription_config field is rejected: "Unknown parameter 'bogus_field' at 'generation_config.transcription_config'". So accepting language_codes and an empty transcription_config shows the fields are recognized.
    • The eight constrained mime types each return the transcript (fixture converted with ffmpeg).
  • make integration-test subset: Google test_transcribe, test_speak (gemini-2.5-flash-preview-tts) and both test_generate cases (lyria-3-clip-preview) pass.
  • tests/unit_tests/test_dispatcher_delegation.py is green. Google audio is a single-backend client, so it is not a dispatcher.

GoogleAudioClient sets _transcribe_endpoint to the Interactions endpoint
it already uses for speak and generate. An audio input becomes an audio
content part with generation_config.transcription_config, and the
transcript text is parsed from the model_output text parts. language
maps to transcription_config.language_codes on transcription requests
and to speech_config otherwise.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant