feat: add push-to-talk voice dictation with Parakeet - #240
Open
EduCosta85 wants to merge 2 commits into
Open
EduCosta85 wants to merge 2 commits into
EduCosta85 wants to merge 2 commits into
Conversation
Support push-to-talk voice dictation directly in the composer. Speech clips recorded via MediaRecorder are validated, decoded, and forwarded over JSON-RPC to a local speech-to-text server exposing an OpenAI-compatible transcription endpoint (such as NVIDIA Parakeet running on mlx-audio). - protocol: define dictation/status and dictation/transcribe in JSON-RPC schema - config: add user-owned DictationConfig with endpoint, model, timeout, and cap settings - core: implement pure audio container sniffing and OpenAI-compatible client - application: implement DictationService with model egress policy checks - desktop: add useDictation hook and composer push-to-talk microphone button - docs & tests: add complete unit test suites and setup documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds push-to-talk voice input directly to the composer. Spoken text is recorded locally, transcribed via an OpenAI-compatible speech-to-text server (such as NVIDIA Parakeet served by
mlx-audio), and inserted directly into the prompt draft at the caret position.What is included
dictation/statusmethod returning endpoint capability (available,model,maxAudioSeconds). Added toREAD_METHODSfor safe retries.dictation/transcribemethod receiving{ audio, mimeType, language?, projectId? }and returning{ text, model }. Kept out ofREAD_METHODS.DictationConfig):dictationin user configuration (~/.deepcode/deepcode_config.json).endpoint(must be absolute HTTP/HTTPS without userinfo, query, or fragment).webm,ogg,mp4,wav,mp3,flac,aac) matching declared MIME types.SpeechToTextClienttargetingPOST <endpoint>/audio/transcriptionswith multipart form data, no redirect following, and zero response-body leakage in error messages.DictationServicecoordinates egress verification (providers.egress), audio decoding, client dispatch, and error translation.useDictationReact hook managing recording state, platform-compatible MIME negotiation (MediaRecorder), timers, chunked base64 conversion, and cancellation.X), andEscapeshortcut to cancel.docs/guide/dictation.mdcoveringmlx-audiolocal Parakeet setup, config reference, and security boundaries.Verification
pytest tests/test_dictation.py tests/test_dictation_service.py tests/test_config_layering.py tests/contract/test_protocol_schema.py: 105 passed.ruff checkandruff format --check(v0.15.21): Clean.python -m compileall -q app_server cli core tools workflows: Passed.npm run lint: Clean.npm run typecheck: Passed without errors.npm test -- --run(includinguseDictation.test.ts): 40 test files passed, 274 tests passed.npm run check:protocol,check:version,check:tauri: All passed.