Skip to content

Add Realtime Voice plugin - #587

Open
NoHara42 wants to merge 1 commit into
agent0ai:mainfrom
NoHara42:add-realtime-voice
Open

NoHara42 wants to merge 1 commit into
agent0ai:mainfrom
NoHara42:add-realtime-voice

Conversation

@NoHara42

@NoHara42 NoHara42 commented Sep 24, 2026 •

Copy link
Copy Markdown

Summary

Adds realtime_voice to the Agent Zero Plugin Index: continuous, interruptible speech-to-speech conversation using OpenAI's Realtime API, alongside the existing voice mode.

The Realtime model handles conversation, not agent work. A single delegate_to_agent function hands tasks to the normal Agent Zero agent, gives a short spoken acknowledgement while it works, and speaks the result when it returns.

Integration

  • Direct browser-to-OpenAI WebRTC audio; no third-party voice hosting. Session setup runs through Agent Zero and the OpenAI API key stays server-side.
  • Reuses Agent Zero's API key storage, scoped plugin settings, and normal chat/task path. Delegated work follows the agent's existing model and tool configuration.
  • UI controls for starting/stopping calls, mute, interruption, and captions, with theme-aware settings. Existing voice mode remains available.

Validation

  • 42 Python tests and 11 JavaScript tests passed, covering session setup, delegation, input/error handling, and conversation state.
  • Live Chromium testing with synthetic microphone audio covered WebRTC, actual terminal delegation, spoken results, and barge-in. Install, enable/disable, and uninstall were exercised in an isolated instance.
  • Authentication and CSRF checks passed for all three backend endpoints; dark/light UI styling was checked.
  • The official submission validator passed against the published repository and generated index; this PR's validate CI check also passed.

Scope and limitations

This PR adds only plugins/realtime_voice/index.yaml; runtime code stays in the source repository.

The plugin inherits Agent Zero's trusted-instance security model and agent permissions. Audio and conversation data are sent to OpenAI, and usage is billable. Ending a call does not cancel agent work already started. Safari/mobile and remote reverse-proxy deployments have not been verified; setup, manual test steps, and known limitations are documented in the README.

@NoHara42
NoHara42 marked this pull request as ready for review September 24, 2026 12:49

@POWERFULMOVES POWERFULMOVES left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: index-only addition, validate CI green. Upstream repo https://github.com/NoHara42/a0-realtime-voice reachable (HTTP 200). Tags/labels consistent with catalog. Safe index entry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants