Repository navigation
feat(google): add gemini-3.5-transcribe to Google audio transcription - #589
Open
Kamilbenkirane wants to merge 3 commits into
Open
Kamilbenkirane wants to merge 3 commits into
Kamilbenkirane wants to merge 3 commits into
Conversation
GoogleAudioClient sets _transcribe_endpoint to the Interactions endpoint it already uses for speak and generate. An audio input becomes an audio content part with generation_config.transcription_config, and the transcript text is parsed from the model_output text parts. language maps to transcription_config.language_codes on transcription requests and to speech_config otherwise.
…nscribe # Conflicts: # CHANGELOG.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Google audio had no
Operation.TRANSCRIBE, sogemini-3.5-transcribe(GA Aug 26, 2026) could not be used through celeste. The transcription guide says "The Gemini API converts speech in audio files into text using the Gemini 3.5 Transcribe model (gemini-3.5-transcribe)". It runs on the same Interactions endpoint (POST /v1beta/interactions) that Google audio already uses for speak and generate. The Interactions reference definesgeneration_config.transcription_config: "Optional. Configuration for speech recognition (transcription). If present, ASR is enabled". Itslanguage_codesfield is "Optional. BCP-47 language codes providing hints about the languages present in the audio. If omitted or empty, defaults to automatic language detection." The guide says "The complete transcript text is returned inresponse.output_text", which is thetextparts of themodel_outputstep.Fix:
src/celeste/modalities/audio/providers/google/client.py,GoogleAudioClient:_transcribe_endpointis the Interactions endpoint already used by_speak_endpointand_generate_endpoint. Same wire format, so this is an endpoint ClassVar, not a dispatcher._init_requestfollowsOpenAIAudioClient: oneAudioArtifact, sent as an audio content part through the existingbuild_content_parthelper, withgeneration_config.transcription_configset._parse_contentreturns the joinedmodel_outputtext when the response carries no audio part, the same comprehension asGoogleInteractionsTextClient. Audio responses parse as before.src/celeste/providers/google/interactions/parameters.py,LanguageMapper: on a request carryingtranscription_config,languagegoes totranscription_config.language_codes; otherwise it goes tospeech_configas before. It goes through the samelocale_map, so"en"becomes"en-US", a code from Google's transcription language table.src/celeste/modalities/audio/providers/google/models.py:gemini-3.5-transcribewith{AUDIO: {TRANSCRIBE}}and anAudioConstraintlisting the eight celeste mime types the Interactions audio part accepts (AAC, AIFF, FLAC, M4A, MP3, OGG, WAV, WEBM).languageis mapped. It is the only unified audio parameter with atranscription_configfield.temperatureis not in the reference'sGenerationConfig.promptpart is ignored live: "Transcribe in uppercase." left the transcript unchanged.custom_vocabularyandmode(smart, diarization, word timestamps) have no unified parameter. They work throughextra_body, and the wordannotationscome back inmetadata["text_blocks"].tests/unit_tests/test_google_audio_transcribe.py(copied from the Mistral one):_init_requestpayload test.tests/unit_tests/test_parameter_wire_contracts.py: alanguage→transcription_config.language_codesrow.tests/integration_tests/audio/test_transcribe.py: a(GOOGLE, "gemini-3.5-transcribe")case.Conflict recorded: the Interactions reference's
modelenum does not listgemini-3.5-transcribe, but the transcription guide uses it and live serving accepts it. Live serving wins.Validation:
make cipasses (765 unit tests).client.transcribe(test_audio)on the existingtests/integration_tests/text/assets/test_audio.mp3fixture (10 s):gemini-3.5-transcribe→TextOutput: "This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity". Usage 251 in / 30 out; finish reasoncompleted.whisper-1on the same file gives the same words: "This week I traveled to Chicago to deliver my final farewell address to the nation following in the tradition of presidents before me. It was an opportunity". Only the punctuation differs.language="en"sends{"transcription_config": {"language_codes": ["en-US"]}}and returns the same transcript.transcription_configfield is rejected: "Unknown parameter 'bogus_field' at 'generation_config.transcription_config'". So acceptinglanguage_codesand an emptytranscription_configshows the fields are recognized.make integration-testsubset: Googletest_transcribe,test_speak(gemini-2.5-flash-preview-tts) and bothtest_generatecases (lyria-3-clip-preview) pass.tests/unit_tests/test_dispatcher_delegation.pyis green. Google audio is a single-backend client, so it is not a dispatcher.