Skip to main content
The core classes that define an agent and drive a live call: the Agent you subclass to give it instructions and tools, and the AgentSession that runs it, speaking, listening, and managing the call lifecycle. serve and invoke register an agent and start sessions against it.

Agent

A voice agent: instructions, tools, and lifecycle behavior. Subclass to add behavior: define @function_tool methods for tools and override on_enter/on_exit for lifecycle hooks. The agent carries its Pipeline and is registered for incoming sessions by zrt.serve.

Constructor

instructions
str
required
System prompt defining the agent’s behavior and persona.
name
Optional[str]
Display name. Defaults to agent_id.
pipeline
Any
The Pipeline (STT/LLM/TTS + VAD/turn detection) to run on.
tools
List[FunctionTool]
Extra @function_tool callables. Tools defined as methods on the agent are auto-registered, so use this only for tools defined elsewhere.
agent_id
str
Stable id the agent is registered and routed under. Required.
mcp_servers
list
MCP servers whose tools are exposed to the agent.
inherit_context
bool
default:"False"
Carry chat history over when handed off from another agent.
greeting
Optional[str]
Optional line spoken automatically when the agent starts.
greeting_non_interruptible
bool
default:"False"
Disallow barge-in during the greeting.
voice_suffix
Optional[str]
Text appended to the agent’s voice id (provider-specific).
append_voice_suffix
bool
default:"True"
Whether to apply voice_suffix. Default True.
agents
Optional[List['Agent']]
Alternate agents this one can hand off to.
max_session_duration_seconds
Optional[int]
Hard cap on session length for this agent.

add_server

Register an additional MCP server with the agent. Extension hook for attaching an MCP server at runtime. The base implementation does nothing.
mcp_server
Any
required
The MCP server to add.

capture_frames

Return the most recent captured camera frames (vision).
num_of_frames
int
default:"1"
How many recent frames to return. Default 1.
returns
list
A list of frames, empty when vision is not active.

cleanup

Release the agent’s resources. Extension hook for cleaning up after a session. The base implementation does nothing.

emit

Emit an event, invoking all registered handlers with the given arguments. Handlers are called in registration order. Coroutine handlers are scheduled on the running event loop. If the emitter is closed, the call is a no-op.
event
T
required
The event to emit.

hangup

End the active call/session immediately.

initialize_mcp

Initialize the agent’s MCP servers. Extension hook for setting up configured MCP servers. The base implementation does nothing.

off

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.
event
T
required
The event the handler was registered for.
callback
Callable[..., Any]
required
The handler to remove.

on

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.
event
T
required
The event to listen for.
callback
Callable[..., Any] | None
The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.
returns
Callable[..., Any]
The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

on_enter

Lifecycle hook fired when the agent starts handling a session. Override to greet the caller or run setup, e.g. await self.session.say(...).

on_exit

Lifecycle hook fired when the session is ending. Override to say a closing line or release resources before the session closes.

play_background_audio

Play a background-audio clip under the conversation.
file
str
Audio file URL/path.
volume
float
default:"1.0"
Playback volume. Default 1.0.
looping
bool
default:"False"
Loop the clip. Default False.
override_thinking
bool
default:"True"
Replace any thinking-audio while this plays. Default True.

preload_background_audio

Preload a background-audio clip so later playback starts instantly.
file
str
Audio file URL/path.
volume
float
default:"1.0"
Playback volume. Default 1.0.

register_a2a

Register the agent for agent-to-agent communication. Extension hook for publishing the agent’s A2A card. The base implementation does nothing.
card
Any
required
The agent card describing this agent for A2A.

register_tools

Discover and register tools defined as methods on the agent. Scans the agent’s public attributes and appends any that are @function_tool callables to the agent’s tool set, skipping tools already registered. Called automatically during initialization.

set_thinking_audio

Set an audio clip looped while the agent is thinking.
file
str
Audio file URL/path. Pass a falsy value to clear it.
volume
float
default:"0.3"
Playback volume, 0.0–1.0. Default 0.3.

stop_background_audio

Stop any background audio currently playing.

unregister_a2a

Unregister the agent from agent-to-agent communication. Extension hook for tearing down A2A registration. The base implementation does nothing.

update_tools

Replace the agent’s tool set.
tools
List[FunctionTool]
required
The new list of @function_tool callables.

AgentSession

A running agent session, your handle to interact with the live conversation. Available as self.session inside an agent. Use it to speak (say, reply), steer the turn (interrupt, generate), read or edit the conversation (get_context_history, update_instructions, update_tools), hot-swap components (update_provider), play audio, and end the call (hangup, leave).

Constructor

agent
Any
required
The agent whose instructions, tools and callbacks drive the session.
pipeline
Any
required
The media pipeline (STT/LLM/TTS and hooks) for the session.
wake_up
Optional[int]
Idle timeout in seconds after which the wake-up callback fires.
wake_up_max_attempts
Optional[int]
Maximum number of wake-up callbacks before the session is closed; None means unlimited.
background_audio
Any
Background audio configuration for the session.
dtmf_handler
Any
Handler dispatched when DTMF digits are received.
voice_mail_detector
Any
Detector invoked when voicemail is detected.
recording
Any
Recording configuration applied to the session.

active_agent

Return the agent currently driving the session.
returns
Any

add_shutdown_callback

Register a callback to run when the session is closing. Callbacks are invoked during close; coroutine callbacks are awaited.
callback
Callable[[], Any]
required
A callable taking no arguments; may be sync or async.

call_transfer

Transfer the active phone call to another destination.
token
str
required
Auth token for the transfer.
transfer_to
str
required
Destination number/address to transfer to.

cancel_generation

Cancel the in-flight LLM generation, if any.

close

Close the session and release its resources.
reason
str
default:"sdk_close"
Optional reason recorded for the close.

emit

Emit an event, invoking all registered handlers with the given arguments. Handlers are called in registration order. Coroutine handlers are scheduled on the running event loop. If the emitter is closed, the call is a no-op.
event
T
required
The event to emit.

fetch_context_history

Fetch a fresh copy of the conversation history. The async form of get_context_history: always retrieves the latest messages for the live session.
last_n
int
default:"0"
Return only the most recent last_n messages (0 = all).
include_function_calls
bool
default:"False"
Include tool-call/tool-result messages. Default False.
include_system_messages
bool
default:"False"
Include system messages. Default False.
returns
list[dict]
A list of message dicts.

generate

Feed text to the agent as if the caller had said it, triggering a response.
text
str
required
The user-turn text to inject and generate a reply for.

get_context_history

Read the conversation history. Returns immediately with a cached snapshot, and is also awaitable: await the result to fetch a fresh copy of the live conversation. Each item is a dict with role, content, message_id and related fields.
last_n
int
default:"0"
Return only the most recent last_n messages (0 = all).
include_function_calls
bool
default:"False"
Include tool-call/tool-result messages. Default False.
include_system_messages
bool
default:"False"
Include system messages. Default False.
returns
'_HistoryAwaitable'
An awaitable list of message dicts.

handoffs

Return the list of handoffs that have occurred this session.
returns
list
A copy of the handoff records, each a dict with from, to and reason fields.

hangup

End the call and close the session.
reason
str
default:"manual_hangup"
Reason recorded on the close (defaults to “manual_hangup”).

inject_context

Inject multiple messages into the conversation context.
ctx
Any
required
An object exposing messages() or an iterable of messages, each injected via inject_message.
returns
bool
True if every message was injected successfully, False otherwise.

inject_message

Insert a message into the conversation context without triggering a reply. Accepts either a message object or a dict with fields such as role, content, message_id, images, tool_calls and tool_call_id.
message
Any
required
The message to inject.
position
str
default:""
Optional placement hint for where to insert the message.
returns
bool
True if the message was accepted, False otherwise (e.g. no active session or transport).

interrupt

Stop the agent’s current utterance and cancel any in-flight generation.
force
bool
default:"False"
Interrupt even when the current utterance is non-interruptible.

last_handoff_reason

Return the reason recorded for the most recent handoff, or an empty string.
returns
str

leave

Leave the session gracefully (the agent disconnects).

off

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.
event
T
required
The event the handler was registered for.
callback
Callable[..., Any]
required
The handler to remove.

on

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.
event
T
required
The event to listen for.
callback
Callable[..., Any] | None
The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.
returns
Callable[..., Any]
The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

play_background_audio

Play background audio for the session. The source may be a URL, a local file path, or a config object/dict with file_url (or file_path/url), volume, looping and a playback mode. Local files are read and sent inline. A config whose enabled flag is False is ignored.
config
Any
Audio source as a string path/URL, dict, or config object.
override_thinking
bool
default:"True"
Whether this playback may override active thinking audio.

preload_background_audio

Preload background audio so later playback starts without delay. Accepts the same source forms as play_background_audio. A config whose enabled flag is False is ignored.
config
Any
Audio source as a string path/URL, dict, or config object.

publish_message

Publish a message to a pub/sub topic.
config
Any
required
An object with a topic and message, plus optional options and payload. Non-string messages, options and payloads are JSON-encoded.

push_audio_frame

Push a raw PCM audio frame into the session.
pcm
bytes
required
Raw PCM audio bytes; empty input is ignored.
sample_rate
int
default:"48000"
Sample rate of the PCM data in Hz. Default 48000.

register_handoff_agent

Register an agent so it can be a handoff target during the session.
agent
Any
required
The agent to register; indexed by its agent_id or id.
returns
str
The registered agent’s id, or an empty string if it has none.

remove_message

Remove a single message from the conversation history.
message_id
str
required
Id of the message to remove (from get_context_history).
returns
bool
True if the message was removed.

reply

Have the agent generate and speak a reply from a prompt. Unlike say (which speaks fixed text), this asks the LLM to produce the response from instructions.
instructions
str
required
Prompt telling the agent what to say.
wait_for_playback
bool
default:"True"
Await until playback finishes before returning. Default True.
frames
list | None
Image frames to include with the prompt (vision).
latest_frames
int
default:"0"
Instead of frames, attach this many most-recent camera frames.
interruptible
bool
default:"True"
Allow the caller to barge in. Default True.
returns
UtteranceHandle
An UtteranceHandle you can await for playback completion.

say

Speak a specific line of text (synthesized via TTS).
message
str
required
The exact text to speak.
interruptible
bool
default:"True"
Allow the caller to barge in over it. Default True.
voice
str
default:""
Optional one-off voice override for this utterance.
audio_data
Any
Reserved; pre-synthesized audio injection is not supported yet.
add_to_chat_context
bool
default:"True"
Reserved; transcript handling is automatic.
returns
UtteranceHandle
An UtteranceHandle you can await for playback completion.

send_a2a

Send an agent-to-agent message to another agent over pub/sub. The message is wrapped in an envelope carrying the sending agent’s id and published to the target agent’s A2A topic.
target_agent_id
str
required
Id of the agent to deliver the message to; required.
message_json
str
required
The message body to deliver.
correlation_id
str
default:""
Optional id to correlate request and response messages.

send_image

Send an image to the session. Raw bytes are sent as-is with the given mime_type; other inputs are coerced to JPEG bytes. Empty input is ignored.
data
Any
required
Image data as bytes-like or any object coercible to JPEG.
mime_type
str
default:"image/jpeg"
MIME type to use when data is already bytes. Default "image/jpeg".

send_message_with_frames

Send a text message together with image frames (vision input). Each frame may be a (payload, mime_type) tuple, a dict with data and an optional mime_type, or a raw image payload; non-bytes payloads are coerced to JPEG. The call is skipped when there is no message, no frames, and num_latest_frames is not positive.
message
Optional[str]
required
Text to send alongside the frames; may be None.
frames
list
required
The image frames to attach.
default_mime_type
str
default:"image/jpeg"
MIME type used when a frame does not specify one.
num_latest_frames
int
default:"0"
Number of most-recent camera frames to attach instead of (or in addition to) frames.

set_user_input_enabled

Enable or disable processing of the caller’s audio input.
enabled
bool
required
False to ignore the caller, True to resume.

start

Start the session: connect to the runtime and run the agent. Establishes the session, emits the join events, and invokes the agent’s on_enter. Optionally holds startup until a participant (and audio stream) is present and can block until shutdown.
wait_for_participant
bool
default:"False"
Hold on_enter until a participant joins.
run_until_shutdown
bool
default:"False"
Block until the session is closed, then clean up.
sip_hangup_on_shutdown
bool
default:"False"
Hang up the SIP call when the session shuts down.
wait_for_participant_timeout_ms
int
default:"60000"
How long to wait for a participant, in ms.
wait_for_audio_stream
bool
default:"False"
Also wait for the participant’s audio stream before proceeding; defaults to enabled for SIP calls.
runtime_options
Optional[dict]
Optional runtime configuration overrides.
session_id
Optional[str]
Caller-supplied session identifier to associate with the call.

start_recording

Start recording the session.
config
Any
Recording configuration; falls back to the session-level recording= config if omitted. If neither is set, the call is ignored.

start_thinking_audio

Play the agent’s configured thinking audio. Requires thinking audio to have been configured on the agent (via agent.set_thinking_audio(...)); otherwise the call is ignored.

stop_background_audio

Stop any background audio currently playing.

stop_recording

Stop the active session recording.

stop_thinking_audio

Stop the agent’s thinking audio.

subscribe_a2a

Subscribe to incoming agent-to-agent messages for the active agent. Subscribes to the active agent’s A2A topic and emits an a2a_message event for each well-formed envelope received. Re-subscribing to the same topic is a no-op; switching agents resubscribes to the new agent’s topic.

subscribe_pubsub

Subscribe to a pub/sub topic, optionally invoking a callback per message.
topic
str
required
The topic to subscribe to; must be non-empty.
callback
Any
Optional callable invoked with each message dict matching the topic.
returns
Any
The internal handler registered for the callback, or None if no callback was supplied or the topic was empty.

update_instructions

Replace the agent’s system prompt mid-session.
instructions
str
required
The new system prompt.

update_interrupt_config

Tune barge-in / interruption behavior mid-session.
mode
Optional[str]
Detection mode: "VAD_ONLY", "STT_ONLY" or "HYBRID".
interrupt_min_duration
Optional[float]
Minimum speech duration (seconds) to count as a barge-in.
interrupt_min_words
Optional[int]
Minimum words before an interruption is honored.
cooldown_ms
Optional[int]
Minimum gap between interruptions, in ms.
false_interrupt_pause_duration_ms
Optional[int]
Pause length treated as a false interrupt, in ms.
resume_on_false_interrupt
Optional[bool]
Resume the agent’s turn after a false interrupt.

update_provider

Hot-swap a pipeline component to a different provider mid-session.
component
str
required
Which stage to change, e.g. "stt", "llm", "tts".
provider
str
required
The provider to switch to.
params
dict | None
Optional provider-specific settings (model, voice, …).

update_tools

Replace the agent’s tool set mid-session.
tools
list
required
The new list of @function_tool callables.

wait_for_synthesis_done

Wait until the current TTS synthesis finishes. Returns immediately if no synthesis is in progress.
timeout
Optional[float]
Maximum time to wait in seconds; None waits indefinitely.
returns
bool
True if synthesis completed (or none was in progress), False if the wait timed out.

warm_transfer

Not implemented.

Room

Per-session room configuration passed to invoke.

Fields

room_id
Optional[str]
Existing room to join. A new room is created when None.
auth_token
Optional[str]
Auth token for the room. Falls back to the ZRT auth env vars.
agent_name
Optional[str]
Display name to publish the agent under. Defaults to the agent id.
playground
bool
default:"True"
Return a playground_url for opening the session. Default True.
vision
bool
default:"False"
Enable camera-frame capture for the agent. Default False.
recording
bool
default:"False"
Record the session. Default False.
background_audio
bool
default:"False"
Enable background-audio playback. Default False.
audio_listener_enabled
bool
default:"False"
Deliver raw input audio to listeners. Default False.
auto_end_session
bool
default:"True"
End the session when the caller leaves. Default True.
session_timeout_seconds
Optional[int]
Hard cap on session duration, in seconds.
no_participant_timeout_seconds
Optional[int]
End the session if nobody joins in time.
agent_participant_id
Optional[str]
Fixed participant id for the agent.
signaling_base_url
Optional[str]
Override the signaling base URL.
send_logs_to_dashboard
bool
default:"True"
Stream session logs to the dashboard. Default True.
dashboard_log_level
str
default:"INFO"
Log level for dashboard logs. Default "INFO".

Sip

Telephony (SIP) details for a phone-call session passed to invoke.

Fields

call_to
Optional[str]
Destination number/address for an outbound call.
call_from
Optional[str]
Caller id presented on the call.
call_type
Optional[str]
Call direction/type (e.g. inbound/outbound).
call_id
Optional[str]
Identifier correlating the call.
webhook_url
Optional[str]
URL to receive call-event callbacks.
extra
Dict[str, str]
default:"…"
Additional provider-specific key/values forwarded with the call.