Repository navigation
feat: Codex voice conversations on web, desktop, and mobile - #14638
jakeleventhal wants to merge 7 commits into
Conversation
Codex 0.156 voice mode for T3 Code: talk to a realtime voice model that hands work to Codex in the thread and reads its answers back. Audio runs over WebRTC between the client and OpenAI; the server relays only SDP and captions over provider.voice.session, and ending the stream stops the conversation. On orchestrator V2, Codex's self-started voice turns are interrupted and resubmitted as ordinary user runs, since V2 has no provider-initiated runs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Resubmit a Codex-started voice turn only once it reports interrupted, so a turn that finished before the interrupt landed never runs twice. - Carry each pending resubmit's own provider thread, and keep a replaced voice stream from clearing its successor's thread. - Mobile: gate voice on the provider thread existing, keep the voice strip reachable while a question card hides the composer, and keep a newer conversation's wake lock. - Drop unused exports flagged by knip and update the Codex settings fields test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…versation - Resubmit a Codex-started voice turn that failed, not only one that was interrupted, so the spoken request is never silently dropped. - /voice stops an active conversation even if the provider no longer reports voice support. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ional shortcut Startup default sync only treated identical `when` strings as conflicts, so a default like `f8 voice.toggle when !terminalFocus` was appended after a user's bare `f8` binding and won whenever the terminal lacked focus. A default is now skipped when its shortcut matches a user rule and either side is unconditional. Also corrects the voice lifecycle comment: screens pushed over the thread keep a conversation running. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The base now imports IdAllocator, ProviderContinuationRequests, and EnvironmentRegistry as namespaces. Update the three voice call sites that still used the old named imports. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
vouch:trusted
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| }); | ||
| }); | ||
| session.setPermissionCheckHandler((_webContents, permission, requestingOrigin) => | ||
| permission === "media" ? isAppUrl(requestingOrigin, options.appUrl) : true, |
There was a problem hiding this comment.
🟠 High window/MainWindowPermissions.ts:61
Trusted media checks return true, so Electron skips setPermissionRequestHandler and never calls askForMediaAccess("microphone"). On macOS, users with microphoneStatus() set to "not-determined" therefore receive no system consent prompt and voice capture remains unauthorized. Return false for media checks so the request handler can grant or deny the request and trigger the prompt.
- permission === "media" ? isAppUrl(requestingOrigin, options.appUrl) : true,
+ permission === "media" ? false : true,🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/desktop/src/window/MainWindowPermissions.ts around line 61:
Trusted `media` checks return `true`, so Electron skips `setPermissionRequestHandler` and never calls `askForMediaAccess("microphone")`. On macOS, users with `microphoneStatus()` set to `"not-determined"` therefore receive no system consent prompt and voice capture remains unauthorized. Return `false` for `media` checks so the request handler can grant or deny the request and trigger the prompt.
| const expectedIndex = | ||
| text === undefined | ||
| ? -1 | ||
| : active.expectedHandoffs.findIndex((expected) => text.includes(expected)); |
There was a problem hiding this comment.
🟡 Medium Layers/CodexVoice.ts:190
A normal user turn can consume a pending voice handoff and have its reply spoken instead, while the actual continuation is no longer tracked and its answer is not read aloud. expectSpokenTurn stores only the request text, and the fallback at findIndex matches with text.includes(expected), so any queued user message containing the same phrase—such as run the tests—is treated as the handoff. Correlate the expected handoff with the specific continuation/turn identifier rather than matching message text.
Also found in 1 other location(s)
apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts:5367
voice.expectSpokenTurnrecords only the request text, not an identifier for the continuation it is about to enqueue. Before that queued continuation starts, any ordinary user turn on the voice thread whose message contains the same text is consumed byCodexVoiceas the expected handoff. Its reply is spoken instead, while the actual voice continuation is no longer tracked and its answer is not read aloud. This is reachable when a user submits the same/containing text while the continuation worker is queued behind another turn.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/Layers/CodexVoice.ts around line 190:
A normal user turn can consume a pending voice handoff and have its reply spoken instead, while the actual continuation is no longer tracked and its answer is not read aloud. `expectSpokenTurn` stores only the request text, and the fallback at `findIndex` matches with `text.includes(expected)`, so any queued user message containing the same phrase—such as `run the tests`—is treated as the handoff. Correlate the expected handoff with the specific continuation/turn identifier rather than matching message text.
Also found in 1 other location(s):
- apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.ts:5367 -- `voice.expectSpokenTurn` records only the request text, not an identifier for the continuation it is about to enqueue. Before that queued continuation starts, any ordinary user turn on the voice thread whose message contains the same text is consumed by `CodexVoice` as the expected handoff. Its reply is spoken instead, while the actual voice continuation is no longer tracked and its answer is not read aloud. This is reachable when a user submits the same/containing text while the continuation worker is queued behind another turn.
| return resolveVoiceAvailability({ | ||
| provider: providers.find((provider) => provider.instanceId === thread.providerInstanceId), | ||
| hasMessages: thread.latestUserMessageAt !== null, | ||
| hasSession: thread.activeProviderThreadId !== null, |
There was a problem hiding this comment.
🟡 Medium voice/voiceAvailability.ts:44
resolveThreadVoiceAvailability returns ready after the provider session has been torn down, so the UI enables voice and requests microphone access even though the subsequent voice RPC rejects because providerSessionsV2.get(providerSessionId) is null. activeProviderThreadId only proves that a persisted provider-thread record exists; use the live provider-session state when computing hasSession so this path returns unavailable instead.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/web/src/voice/voiceAvailability.ts around line 44:
`resolveThreadVoiceAvailability` returns `ready` after the provider session has been torn down, so the UI enables voice and requests microphone access even though the subsequent voice RPC rejects because `providerSessionsV2.get(providerSessionId)` is `null`. `activeProviderThreadId` only proves that a persisted provider-thread record exists; use the live provider-session state when computing `hasSession` so this path returns `unavailable` instead.
| const previous = yield* Ref.getAndSet(activeRef, active); | ||
| if (previous && !previous.closed) { | ||
| // One conversation per thread: a newer client takes over. | ||
| yield* finish(previous, { type: "closed", reason: "replaced" }); | ||
| yield* client.raw | ||
| .request("thread/realtime/stop", { threadId: previous.providerThreadId }) | ||
| .pipe(Effect.ignore); | ||
| } |
There was a problem hiding this comment.
🟡 Medium Layers/CodexVoice.ts:262
Taking over a thread publishes the replacement active before the previous realtime transport is marked closed and stopped, so delayed thread/realtime/closed or thread/realtime/error notifications from the old transport are routed to and terminate the new session. Mark and stop the previous session before installing the replacement in activeRef.
- const previous = yield* Ref.getAndSet(activeRef, active);
+ const previous = yield* Ref.get(activeRef);
if (previous && !previous.closed) {
// One conversation per thread: a newer client takes over.
yield* finish(previous, { type: "closed", reason: "replaced" });
yield* client.raw
.request("thread/realtime/stop", { threadId: previous.providerThreadId })
.pipe(Effect.ignore);
}
+ yield* Ref.set(activeRef, active);🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/Layers/CodexVoice.ts around lines 262-269:
Taking over a thread publishes the replacement `active` before the previous realtime transport is marked closed and stopped, so delayed `thread/realtime/closed` or `thread/realtime/error` notifications from the old transport are routed to and terminate the new session. Mark and stop the previous session before installing the replacement in `activeRef`.
| if (!offer.sdp) throw new Error("Could not create a voice connection offer."); | ||
|
|
||
| session.cancelSession = this.platform.openSession( | ||
| { ...target, offerSdp: offer.sdp }, |
There was a problem hiding this comment.
🟠 High voice-mode/controller.ts:297
The server receives an SDP offer without the ICE candidates gathered after setLocalDescription, so voice connections time out on normal NATed or remote networks. openSession is called immediately with offer.sdp, and this controller neither waits for ICE gathering to complete nor forwards icecandidate events; wait for gathering before opening the session or add candidate forwarding.
🤖 Copy this AI Prompt to have your agent fix this:
In file @packages/client-runtime/src/voice-mode/controller.ts around line 297:
The server receives an SDP offer without the ICE candidates gathered after `setLocalDescription`, so voice connections time out on normal NATed or remote networks. `openSession` is called immediately with `offer.sdp`, and this controller neither waits for ICE gathering to complete nor forwards `icecandidate` events; wait for gathering before opening the session or add candidate forwarding.
| interruptionMode: "doNotMix", | ||
| shouldPlayInBackground: false, | ||
| shouldRouteThroughEarpiece: false, | ||
| }); |
There was a problem hiding this comment.
🟠 High voice-mode/voiceMode.ts:75
After releaseVoiceAudio disables Expo's global audio subsystem, every subsequent voice session leaves it disabled, so remote audio does not play. routeVoiceToSpeaker only changes the audio mode and never calls setIsAudioActiveAsync(true); re-enable the subsystem when routing audio for a new session.
});
+ await setIsAudioActiveAsync(true);🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/mobile/src/features/voice-mode/voiceMode.ts around line 75:
After `releaseVoiceAudio` disables Expo's global audio subsystem, every subsequent voice session leaves it disabled, so remote audio does not play. `routeVoiceToSpeaker` only changes the audio mode and never calls `setIsAudioActiveAsync(true)`; re-enable the subsystem when routing audio for a new session.
|
|
||
| useEffect(() => { | ||
| const handler = (event: globalThis.KeyboardEvent) => { | ||
| const command = resolveShortcutCommand(event, keybindings, { |
There was a problem hiding this comment.
🟡 Medium chat/ChatComposer.tsx:5657
The voice shortcut handler treats editableFocus as false, so a binding such as voice.toggleMute with when: "!editableFocus" mutes the microphone while the user is typing in a text editor, while bindings requiring editableFocus never match. Pass the actual editableFocus and the relevant ChatView context flags to resolveShortcutCommand, as the other shortcut paths do.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/web/src/components/chat/ChatComposer.tsx around line 5657:
The voice shortcut handler treats `editableFocus` as `false`, so a binding such as `voice.toggleMute` with `when: "!editableFocus"` mutes the microphone while the user is typing in a text editor, while bindings requiring `editableFocus` never match. Pass the actual `editableFocus` and the relevant ChatView context flags to `resolveShortcutCommand`, as the other shortcut paths do.
| const MAX_SPEAKABLE_CHARS = 990 * 4; | ||
|
|
||
| function unescapeXmlText(text: string): string { | ||
| return text.replaceAll("<", "<").replaceAll(">", ">").replaceAll("&", "&"); |
There was a problem hiding this comment.
🟡 Medium Layers/CodexVoice.ts:28
unescapeXmlText leaves valid " entities in the delegation input, so say "hello" is submitted as literal entity text instead of say "hello". It also leaves doubly escaped </> as </> because </> are replaced before &; decode & first and include quote entities.
| return text.replaceAll("<", "<").replaceAll(">", ">").replaceAll("&", "&"); | |
| return text | |
| .replaceAll("&", "&") | |
| .replaceAll("<", "<") | |
| .replaceAll(">", ">") | |
| .replaceAll(""", '"') | |
| .replaceAll("'", "'"); |
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/Layers/CodexVoice.ts around line 28:
`unescapeXmlText` leaves valid `"` entities in the delegation input, so `say "hello"` is submitted as literal entity text instead of `say "hello"`. It also leaves doubly escaped `<`/`>` as `<`/`>` because `<`/`>` are replaced before `&`; decode `&` first and include quote entities.
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR adds a substantial Codex voice capability across server, web, desktop, and mobile, including WebRTC, microphone permissions, native dependencies, and new provider-session behavior. It also changes the default keybindings and adds static-analysis suppressions, while unresolved findings identify concrete risks in connectivity, permissions, audio lifecycle, and voice handoff handling. Not approved because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
2 participants
Replaces #13168, which was closed under the prior-approval rule. The code is the same change rebased onto the current
t3code/codex-turn-mapping, plus one small commit that follows the base's new namespace imports at three call sites. #13168 showed ~1.6k files only because the base was force-pushed underneath it; the real diff is 60 files.Screen.Recording.2026-09-22.at.8.13.50.PM.mov
ScreenRecording_09-22-2026.20-15-53_1.MP4
Codex 0.156 shipped voice conversations: you talk to a realtime voice model, it hands coding work to Codex in the thread, and it reads Codex's answer back. T3 Code had no way to do this.
Scope and approval
How it works
oai-eventsdata channel), and audio flows directly between the client and OpenAI. The newprovider.voice.sessionstream RPC carries only the SDP handshake, captions, and lifecycle, so voice costs the same locally, over a relay, or through a tunnel.CodexVoice.ts,CodexAdapterV2). Startsthread/realtime/startwith the same V3 WebRTC, client-managed-handoff settings the Codex TUI uses. When a voice turn finishes, its final answer goes back throughthread/realtime/appendSpeech; commentary and reasoning are never spoken.voicecontinuation. The request then runs as an ordinary user message. The tradeoff: Codex's own history keeps a stub of the interrupted turn. First-class provider-initiated runs would remove this.VoiceModeControllerinclient-runtimehandles negotiation, mute, captions, mic/speaker levels, the connect timeout, and teardown. Browsers and react-native-webrtc share the same structural peer API./voice, F8 (voice.toggle), an unboundvoice.toggleMute, command palette actions, and a voice strip with status, meters, mute/stop, and live captions. Captions are ephemeral.NSMicrophoneUsageDescriptionand the audio-input entitlement.react-native-webrtc(native rebuild required), AndroidRECORD_AUDIO, loudspeaker routing, and the same button and strip. Conversations stop on background or when leaving the thread.ServerProvider.supportsVoice) and needs a thread whose provider session is running.Verification
Re-run on this branch after the rebase:
tsc --noEmitis clean in contracts, shared, client-runtime, server, web, desktop, and mobile.vp lintandvp fmt --checkon the changed files: clean.Carried over from #13168, not repeated on this branch:
Done by Claude Opus 5.5 in Claude Code, running in T3 Code.
🤖 Generated with Claude Code