Skip to content

feat: Codex voice conversations on web, desktop, and mobile - #13168

Open
jakeleventhal wants to merge 678 commits into
pingdotgg:t3code/codex-turn-mappingfrom
jakeleventhal:t3code/codex-voice-mode
Open

jakeleventhal wants to merge 678 commits into
pingdotgg:t3code/codex-turn-mappingfrom
jakeleventhal:t3code/codex-voice-mode

Conversation

@jakeleventhal

@jakeleventhal jakeleventhal commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor
Screen.Recording.2026-09-22.at.8.13.50.PM.mov
ScreenRecording_09-22-2026.20-15-53_1.MP4

Codex 0.156 shipped voice conversations: you talk to a realtime voice model, it hands coding work to Codex in the thread, and it reads Codex's answer back. T3 Code had no way to do this.

How it works

  • Audio stays off the T3 socket. The client owns the mic, speaker, and WebRTC peer (oai-events data channel), and audio flows directly between the client and OpenAI. The new provider.voice.session stream RPC carries only the SDP handshake, captions, and lifecycle, so voice costs the same locally, over a relay, or through a tunnel.
  • The stream is the conversation. Unsubscribing or disconnecting stops the Codex realtime session, so a dropped client can't leave a billed session running.
  • Server (CodexVoice.ts, CodexAdapterV2). Starts thread/realtime/start with the same V3 WebRTC, client-managed-handoff settings the Codex TUI uses. When a voice turn finishes, its final answer goes back through thread/realtime/appendSpeech; commentary and reasoning are never spoken.
  • Idle-thread handoffs. Codex starts or steers the delegated turn itself, and that can't be turned off. Steering an active run needs nothing special. On an idle thread V2 has no provider-initiated run, so the adapter interrupts Codex's turn and, once it settles, offers a voice continuation. The request then runs as an ordinary user message. The tradeoff: Codex's own history keeps a stub of the interrupted turn. First-class provider-initiated runs would remove this.
  • Clients. A shared VoiceModeController in client-runtime handles negotiation, mute, captions, mic/speaker levels, the connect timeout, and teardown. Browsers and react-native-webrtc share the same structural peer API.
  • Web/desktop. Waveform button in the composer, /voice, F8 (voice.toggle), an unbound voice.toggleMute, command palette actions, and a voice strip with status, meters, mute/stop, and live captions. Captions are ephemeral.
  • Desktop. Grants mic access to the app origin only, asks macOS on first use, and adds NSMicrophoneUsageDescription and the audio-input entitlement.
  • Mobile. Adds react-native-webrtc (native rebuild required), Android RECORD_AUDIO, loudspeaker routing, and the same button and strip. Conversations stop on background or when leaving the thread.
  • Settings and scope. Voice is chosen in Codex provider settings. It's Codex-only (ServerProvider.supportsVoice) and needs a thread whose provider session is running.

Verification

  • Typecheck is clean in contracts, shared, client-runtime, server, web, desktop, and mobile.
  • Focused tests: 188 server (voice module, V2 Codex adapter, continuation service), 7 client-runtime, 88 web, 74 desktop, 69 shared keybindings, and 5 mobile.
  • Codex 0.156's app-server accepts the exact start params.
  • Installed and launched on desktop and iPhone from a personal build. A full spoken round trip has not been verified yet, and no screenshots are attached.

🤖 Generated with Claude Code (Claude Opus 5.5, running in T3 Code)

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Sep 23, 2026
Comment thread apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts
Comment thread apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts Outdated
Comment thread apps/mobile/src/features/threads/ThreadComposer.tsx
Comment thread apps/server/src/provider/Layers/CodexVoice.ts
Comment thread apps/mobile/src/features/voice-mode/voiceMode.ts Outdated
Comment thread apps/server/src/ws.ts
Comment thread apps/mobile/src/features/threads/ThreadComposer.tsx Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a substantial cross-platform Codex voice workflow with WebRTC, microphone permissions, native entitlements, new server orchestration, and an authenticated streaming RPC. It also changes the default F8 shortcut and adds static-analysis suppressions, so the scope and side effects warrant human review.

You can add or adjust custom eligibility rules. Learn more.

Comment thread apps/web/src/components/chat/ChatComposer.tsx Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Ignore my attempted follow-up review submission; it was not posted. The submitted inline comment on ChatComposer.tsx is the only UI consistency finding from this check.

Posted via Macroscope — UI Consistency

Comment thread packages/contracts/src/provider.ts
Comment thread apps/server/src/provider/Layers/CodexVoice.ts Outdated
Comment thread apps/server/src/provider/Layers/CodexVoice.ts
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch from a419382 to f2bdd9f Compare September 23, 2026 00:50
@jakeleventhal
jakeleventhal force-pushed the t3code/codex-voice-mode branch from 2ace4c3 to b2614fe Compare September 23, 2026 00:59
Comment thread apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts Outdated
Comment thread apps/web/src/components/ChatView.tsx Outdated
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 5 times, most recently from 6ca6a24 to 3e4ca4c Compare September 23, 2026 23:24
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 2 times, most recently from 9fc8676 to ae065ff Compare September 24, 2026 21:34
juliusmarminge and others added 19 commits September 24, 2026 16:12
…sts (pingdotgg#13459)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…dotgg#13461)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gg#13462)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ingdotgg#13467)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gg#13476)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#13470)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…g#13471)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…oval (pingdotgg#13486)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…fallback (pingdotgg#13494)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…base (pingdotgg#13502)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…gg#13499)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ingdotgg#13493)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…13509)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pingdotgg#13505)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ingdotgg#13508)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch from 6448cfe to 1563a64 Compare September 24, 2026 23:21
jakeleventhal and others added 5 commits September 24, 2026 19:37
Codex 0.156 voice mode for T3 Code: talk to a realtime voice model that hands
work to Codex in the thread and reads its answers back. Audio runs over WebRTC
between the client and OpenAI; the server relays only SDP and captions over
provider.voice.session, and ending the stream stops the conversation.

On orchestrator V2, Codex's self-started voice turns are interrupted and
resubmitted as ordinary user runs, since V2 has no provider-initiated runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Resubmit a Codex-started voice turn only once it reports interrupted, so a
  turn that finished before the interrupt landed never runs twice.
- Carry each pending resubmit's own provider thread, and keep a replaced
  voice stream from clearing its successor's thread.
- Mobile: gate voice on the provider thread existing, keep the voice strip
  reachable while a question card hides the composer, and keep a newer
  conversation's wake lock.
- Drop unused exports flagged by knip and update the Codex settings fields test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…versation

- Resubmit a Codex-started voice turn that failed, not only one that was
  interrupted, so the spoken request is never silently dropped.
- /voice stops an active conversation even if the provider no longer reports
  voice support.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jakeleventhal
jakeleventhal force-pushed the t3code/codex-voice-mode branch from 46603d5 to 9d8c964 Compare September 24, 2026 23:42
Comment thread packages/shared/src/keybindings.ts
Comment thread apps/server/src/ws.ts
…ional shortcut

Startup default sync only treated identical `when` strings as conflicts, so a
default like `f8 voice.toggle when !terminalFocus` was appended after a user's
bare `f8` binding and won whenever the terminal lacked focus. A default is now
skipped when its shortcut matches a user rule and either side is unconditional.

Also corrects the voice lifecycle comment: screens pushed over the thread keep
a conversation running.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch from fe4f6ad to 87c67bd Compare September 25, 2026 05:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.