agent-conv

A real-time voice appointment assistant built on OpenAI's Realtime API. A caller opens a link, speaks to the assistant, asks questions and books a consultation — and the booking is posted to an external system as soon as it is agreed, mid-conversation.

The current deployment is configured for B2B Accelerator, a growth consulting firm, but the assistant's domain knowledge lives entirely in one prompt builder and can be swapped for another business.

---

Architecture

The server is a relay between the browser and OpenAI. It holds two WebSockets open and pumps audio between them in both directions.

flowchart LR
  A[Browser<br/>mic + speaker] <-->|WebSocket<br/>PCM16 base64| B[FastAPI relay]
  B <-->|WebSocket<br/>Realtime API| C[OpenAI<br/>gpt-4o-realtime]
  B --> D[(MongoDB<br/>client records)]
  B -->|appointment captured| E[External booking API]

Loading

Relaying rather than connecting the browser straight to OpenAI is the central design decision: the API key never leaves the server. A browser-side connection would require handing the key — or a short-lived token endpoint — to the client. Here the browser only ever talks to its own origin.

The two directions run as separate coroutines under a single asyncio.gather:

• forward_client_to_openai — receives mic audio and appends it to the input buffer, and relays explicit commit and cancel control messages.
• forward_openai_to_client — reads Realtime events and forwards audio deltas, transcript deltas and speech-start/stop signals, while intercepting function calls before they reach the browser.

The audio path

Everything is PCM16 at 24 kHz, base64-encoded for JSON transport.

Capture. The browser requests the mic with echo cancellation and noise suppression, then runs each 4096-sample block through a converter that clamps to [-1, 1] and scales to signed 16-bit — asymmetrically, using 0x8000 for negative samples and 0x7fff for positive, since two's complement is not symmetric around zero.

Playback. Audio arrives as a stream of deltas rather than one file, so the client keeps a queue. Each chunk is decoded back to Float32Array, written into an AudioBuffer and played through a source node whose onended handler pulls the next chunk. That keeps playback gapless without needing to know the total length in advance.

Interruption handling

Talking over the assistant has to work, or the conversation feels like a phone tree. Barge-in is handled at three levels within one round trip:

1. Detection — OpenAI's server-side VAD (threshold: 0.3, silence_duration_ms: 200) emits input_audio_buffer.speech_started the moment the user starts talking.
2. Local silence — the browser immediately stops the playing source node and flushes the queue. Stopping playback alone is not enough; every chunk already buffered would otherwise play straight after, so the assistant would keep talking over the interruption.
3. Cancel upstream — the browser sends cancel, the server forwards response.cancel, and OpenAI stops generating audio nobody will hear.

Appointment capture

The assistant is given one tool, capture_appointment, taking tookAppointment and an optional ISO-8601 appointmentDate.

When a response.done event contains a function call, the server posts the booking to EXTERNAL_API_URL, then writes a function_call_output back into the conversation and issues a fresh response.create. That last step matters: without it the model has no turn in which to acknowledge, and the caller is left in silence after agreeing to a time. With it, the assistant confirms out loud, and the booking has already landed in the external system by the time it speaks.

Declines are posted too, with tookAppointment: false, so the outcome is recorded either way.

Per-client context

POST /clients stores a free-text info blob and returns a link containing the new client's id. When someone connects on that link, the WebSocket handler loads the record and splices the blob into the system instructions, so the assistant already knows who it is talking to. Connecting without a client id is supported and simply yields the generic prompt.

---

Setup

python3.10 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
python main.py

Requires a running MongoDB. The server listens on port 8000.

Variable	Purpose
OPENAI_API_KEY	Realtime API access
EXTERNAL_API_URL	Where captured appointments are POSTed
SERVER_URL	Base URL used to build client links (default http://localhost:8000)
MONGODB_URI	Mongo connection string (default mongodb://localhost:27017/)
DATABASE_NAME	Database name (default voice_assistant)

API

Method	Route	Description
POST	/clients	Create a client from an info string; returns clientId and agentLink
GET	/clients/{clientId}	Fetch a stored client record
GET	/{client_id}	Voice UI personalised for that client
GET	/	Voice UI with no client context
WS	/ws/voice/{client_id}	Audio relay

curl -X POST http://localhost:8000/clients \
  -H 'Content-Type: application/json' \
  -d '{"info": "Tech startup looking to scale sales operations"}'

Open the returned agentLink and click Start Conversation.

Layout

main.py        FastAPI app: REST endpoints, prompt builder, WebSocket relay, tool handling
database.py    MongoDB connection and the clients index
index.html     single-page voice UI — capture, playback queue, interruption

Status

A working prototype built for one client. Known gaps:

• The business description is hardcoded in create_instructions() rather than stored per client.
• Audio capture uses ScriptProcessorNode, which is deprecated in favour of AudioWorklet; it runs on the main thread and can glitch under load.
• No authentication on any endpoint, and client ids are sequential and guessable.
• GET /{client_id} is a catch-all at the root, so any new top-level route must be declared before it.
• Conversations are not persisted — only the appointment outcome is, and only to the external API.
• No test suite.