Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 32 additions & 30 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ aai.settings.api_key = "your-key"
- `aai.AsyncSyncTranscriber` — Asyncio counterpart of `SyncTranscriber`. Same input types, config, result, and errors; `transcribe()` and `warm()` are coroutines. Owns an HTTP pool: use `async with` or `await aclose()`, or pass an `aai.AsyncClient` to share one
- `aai.SyncTranscriptionConfig` — Sync options: `model` (default `universal-3-5-pro`), `prompt`, `keyterms_prompt`, `conversation_context`, `language_codes`, `timestamps`, `sample_rate`, `channels`
- `aai.SyncTranscriptResponse` — Sync result: `.text`, `.words` (`SyncWord` with `confidence` always, `start`/`end` only when `timestamps=True`), `.confidence`, `.audio_duration_ms`, `.session_id`, `.request_time_ms`
- `assemblyai.streaming.v3.StreamingClient` — Real-time streaming with event-based API (threaded)
- `assemblyai.streaming.v3.AsyncStreamingClient` — Asyncio-native counterpart; same options/events
- `assemblyai.streaming.v3.RealTimeTranscriber` — Real-time streaming with event-based API (threaded)
- `assemblyai.streaming.v3.AsyncRealTimeTranscriber` — Asyncio-native counterpart; same options/events

## Common patterns

Expand Down Expand Up @@ -280,7 +280,9 @@ open connection. Note `keepalive_expiry` applies to the whole client (sync and a

## Streaming (real-time)

Use `universal-3-5-pro` as the streaming model — it's the flagship and what every example below targets. Two clients with identical option/event/handler surfaces: `StreamingClient` (threaded) and `AsyncStreamingClient` (asyncio).
Use `universal-3-5-pro` as the streaming model — it's the flagship and what every example below targets. Two clients with identical option/event/handler surfaces: `RealTimeTranscriber` (threaded) and `AsyncRealTimeTranscriber` (asyncio).

The former `Streaming*` names (`StreamingClient`, `AsyncStreamingClient`, `StreamingClientOptions`, `StreamingParameters`, `StreamingSessionParameters`, `StreamingEvents`, `StreamingError`, `StreamingErrorCodes`) are aliases bound to the same objects, so existing code keeps working; prefer the `RealTime*` names in new code.

**Handler contract — important**: every handler is called as `handler(client, event)`. Two positional args. Plain functions and `async def` functions both work for the async client; async handlers are awaited inline on the read task, so don't block — use `asyncio.create_task(...)` if you need to fan out work. Exceptions inside handlers are logged and swallowed.

Expand All @@ -290,15 +292,15 @@ Use `universal-3-5-pro` as the streaming model — it's the flagship and what ev
- `Termination` → `TerminationEvent(audio_duration_seconds: int | None, session_duration_seconds: int | None)`
- `SpeechStarted` → `SpeechStartedEvent(timestamp: int)` (ms)
- `Warning` → `WarningEvent(warning_code: int, warning: str)`
- `Error` → `StreamingError` (an `Exception` subclass) with `.code: int | None`; `str(error)` is the message. Server-side errors come through `on_error` rather than being raised, and the payload is a `StreamingError`, **not** the wire `ErrorEvent` class.
- `Error` → `RealTimeError` (an `Exception` subclass) with `.code: int | None`; `str(error)` is the message. Server-side errors come through `on_error` rather than being raised, and the payload is a `RealTimeError`, **not** the wire `ErrorEvent` class.
- `LLMGatewayResponse` → `LLMGatewayResponseEvent(turn_order: int, transcript: str, data: Any)`
- `SpeakerRevision` → `SpeakerRevisionEvent(revisions: list[SpeakerRevisionItem])` — diarization-only. Sent once per offline-recluster resolve. Each `SpeakerRevisionItem(turn_order: int, speaker_label: str | None, words: list[Word])` is an earlier Turn whose labels changed (unchanged turns are omitted). For each item, match by `turn_order` against the original Turn and replace its per-word `speaker` (and the turn-level `speaker_label`) with the revision's values. Text and word timestamps are unchanged.

**Sync streaming:**
```python
from assemblyai.streaming.v3 import (
StreamingClient, StreamingClientOptions, StreamingEvents,
StreamingParameters, TurnEvent,
RealTimeTranscriber, RealTimeTranscriberOptions, RealTimeEvents,
RealTimeParameters, TurnEvent,
)

def on_turn(client, event: TurnEvent):
Expand All @@ -307,10 +309,10 @@ def on_turn(client, event: TurnEvent):
def on_error(client, error):
print(f"Error {error.code}: {error}")

client = StreamingClient(StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"]))
client.on(StreamingEvents.Turn, on_turn)
client.on(StreamingEvents.Error, on_error)
client.connect(StreamingParameters(
client = RealTimeTranscriber(RealTimeTranscriberOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"]))
client.on(RealTimeEvents.Turn, on_turn)
client.on(RealTimeEvents.Error, on_error)
client.connect(RealTimeParameters(
sample_rate=16000, speech_model="universal-3-5-pro",
))
try:
Expand All @@ -323,19 +325,19 @@ finally:
```python
import asyncio
from assemblyai.streaming.v3 import (
AsyncStreamingClient, StreamingClientOptions, StreamingEvents,
StreamingParameters,
AsyncRealTimeTranscriber, RealTimeTranscriberOptions, RealTimeEvents,
RealTimeParameters,
)

async def on_turn(client, event):
print(event.transcript)

async def main():
async with AsyncStreamingClient(
StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"])
async with AsyncRealTimeTranscriber(
RealTimeTranscriberOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"])
) as client:
client.on(StreamingEvents.Turn, on_turn)
await client.connect(StreamingParameters(
client.on(RealTimeEvents.Turn, on_turn)
await client.connect(RealTimeParameters(
sample_rate=16000, speech_model="universal-3-5-pro",
))
await client.stream(audio_async_generator) # async iterable of bytes
Expand All @@ -347,7 +349,7 @@ asyncio.run(main())

**Voice-agent tuning** (knobs that matter most when building a voice agent):
```python
StreamingParameters(
RealTimeParameters(
sample_rate=8000, speech_model="universal-3-5-pro", encoding="pcm_mulaw", # telephony
prompt="Transcribe verbatim with filler words and full punctuation.",
agent_context="What is your account number?", # agent's last reply; biases U3Pro context (U3Pro only)
Expand All @@ -360,19 +362,19 @@ StreamingParameters(
Use `pcm_s16le` (default) + `sample_rate=16000` for microphone capture.

`agent_context` (like `prompt`) can also be updated mid-stream after each agent
turn: `client.set_params(StreamingSessionParameters(agent_context="..."))`.
turn: `client.set_params(RealTimeSessionParameters(agent_context="..."))`.

**Error codes**: `StreamingErrorCodes` is a `dict[int, str]` mapping wire codes to human messages. Use `.get(...)` for lookup — not enum-style attribute access:
**Error codes**: `RealTimeErrorCodes` is a `dict[int, str]` mapping wire codes to human messages. Use `.get(...)` for lookup — not enum-style attribute access:
```python
from assemblyai.streaming.v3 import StreamingErrorCodes
from assemblyai.streaming.v3 import RealTimeErrorCodes

def on_error(client, error):
message = StreamingErrorCodes.get(error.code, str(error))
message = RealTimeErrorCodes.get(error.code, str(error))
print(f"Streaming error {error.code}: {message}")
```
Common codes worth branching on: `4001` Not Authorized, `4002` Insufficient Funds, `4029` Client sent audio too fast, `4031` Session idle for too long.

**Events enum**: `StreamingEvents.{Begin, Turn, Termination, SpeechStarted, Error, Warning, LLMGatewayResponse, SpeakerRevision}`. Register a handler for each you care about; the call is the same shape: `client.on(StreamingEvents.Begin, on_begin)`, `client.on(StreamingEvents.Error, on_error)`, etc.
**Events enum**: `RealTimeEvents.{Begin, Turn, Termination, SpeechStarted, Error, Warning, LLMGatewayResponse, SpeakerRevision}`. Register a handler for each you care about; the call is the same shape: `client.on(RealTimeEvents.Begin, on_begin)`, `client.on(RealTimeEvents.Error, on_error)`, etc.

**Slow async handlers — fan-out pattern**: async handlers are awaited inline on the read task. If `on_turn` calls a slow LLM/TTS, ingestion stalls. Fan out and drain on shutdown:
```python
Expand All @@ -391,22 +393,22 @@ if pending:

**Mint a temporary token without streaming** (typical FastAPI/server use):
```python
async with AsyncStreamingClient(StreamingClientOptions(api_key=MASTER_KEY)) as client:
async with AsyncRealTimeTranscriber(RealTimeTranscriberOptions(api_key=MASTER_KEY)) as client:
return await client.create_temporary_token(expires_in_seconds=60)
```
Use `async with` even when you never call `connect()` — `create_temporary_token` lazily opens an `httpx.AsyncClient` and `__aexit__` closes it. Without the context manager you leak the HTTP pool per request. The sync `StreamingClient.create_temporary_token` doesn't need this (no pool to close).
Use `async with` even when you never call `connect()` — `create_temporary_token` lazily opens an `httpx.AsyncClient` and `__aexit__` closes it. Without the context manager you leak the HTTP pool per request. The sync `RealTimeTranscriber.create_temporary_token` doesn't need this (no pool to close).

**Pass the token to the streaming client** via `StreamingClientOptions(token=...)` — same surface for `StreamingClient` and `AsyncStreamingClient`:
**Pass the token to the streaming client** via `RealTimeTranscriberOptions(token=...)` — same surface for `RealTimeTranscriber` and `AsyncRealTimeTranscriber`:
```python
async with AsyncStreamingClient(StreamingClientOptions(token=token_from_server)) as client:
await client.connect(StreamingParameters(sample_rate=16000, speech_model="universal-3-5-pro"))
async with AsyncRealTimeTranscriber(RealTimeTranscriberOptions(token=token_from_server)) as client:
await client.connect(RealTimeParameters(sample_rate=16000, speech_model="universal-3-5-pro"))
await client.stream(audio)
```

**Other gotchas**:
- Don't feed `AsyncStreamingClient.stream()` from a sync generator that blocks (e.g. `time.sleep` pacing) — it starves the read task. Use an `async def` generator with `await asyncio.sleep(...)` instead.
- `format_turns=True` enables punctuation/casing on confirmed end-of-turns. Toggle mid-session via `client.set_params(StreamingSessionParameters(format_turns=True))`.
- `AsyncStreamingClient` used as `async with` calls `disconnect(terminate=True)` on normal block exit and `disconnect(terminate=False)` on exception — no explicit `disconnect()` needed inside the block.
- Don't feed `AsyncRealTimeTranscriber.stream()` from a sync generator that blocks (e.g. `time.sleep` pacing) — it starves the read task. Use an `async def` generator with `await asyncio.sleep(...)` instead.
- `format_turns=True` enables punctuation/casing on confirmed end-of-turns. Toggle mid-session via `client.set_params(RealTimeSessionParameters(format_turns=True))`.
- `AsyncRealTimeTranscriber` used as `async with` calls `disconnect(terminate=True)` on normal block exit and `disconnect(terminate=False)` on exception — no explicit `disconnect()` needed inside the block.

## Important gotchas

Expand Down
Loading
Loading