Build Your First Voice AI Agent

Prerequisites

Before you begin, ensure you have:

A ZRT Auth token or API key + secret from app.zeroruntime.ai (see Authentication).
Python 3.11 or higher.

Understanding the Architecture

Before diving into implementation, choose the pipeline architecture that fits your use case.

Cascade
Realtime

Cascade processes audio through distinct stages for maximum control:

The cascade processes audio through sequential stages:User Voice Input → STT (Speech-to-Text) → LLM (Large Language Model) → TTS (Text-to-Speech) → Agent Voice OutputThis approach provides better control over each processing stage and supports more complex AI reasoning. Power each stage through the Zero Runtime Inference gateway (one token, no per-provider keys) or bring your own provider keys.

Build the Agent

Pick your language. Each track builds the same agent on the Cascade pipeline (Deepgram STT → Google Gemini LLM → Cartesia TTS).

Python

Setup the project

Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install zrt. Every provider plugin ships with it.

uv venv --python 3.11
source .venv/bin/activate     # macOS/Linux
.\.venv\Scripts\activate      # Windows

python3.11 -m venv venv
source venv/bin/activate      # macOS/Linux
.\venv\Scripts\activate       # Windows

New to uv? See the uv install guide.

Cascade
Realtime

uv pip install zrt

pip install zrt

Prefer to bring your own provider keys? Every provider ships with zrt. Just import from zrt.plugins.*. See the plugin pages for STT, LLM, and TTS.

uv pip install zrt

pip install zrt

Configure environment variables

Store API keys and tokens in a .env file in your project root.

Cascade
Realtime

.env

# The Inference gateway authenticates with your Zero Runtime token -
# no separate STT, LLM, or TTS provider keys needed.

# Option A: pre-generated token
ZRT_AUTH_TOKEN = "ZRT Auth token"

# Option B: SDK auto-generates token (omit ZRT_AUTH_TOKEN)
# ZRT_API_KEY = "Your Zero Runtime API Key"
# ZRT_SECRET_KEY = "Your Zero Runtime Secret Key"

.env

# Option A: pre-generated token
ZRT_AUTH_TOKEN="ZRT Auth token"

# Option B: SDK auto-generates token (omit ZRT_AUTH_TOKEN)
# ZRT_API_KEY="Your Zero Runtime API Key"
# ZRT_SECRET_KEY="Your Zero Runtime Secret Key"

OPENAI_API_KEY="Your OpenAI API Key"

# For Google Live API
# GOOGLE_API_KEY="Google Live API Key"

# For AWS Nova API
# AWS_ACCESS_KEY_ID="AWS Key Id"
# AWS_SECRET_ACCESS_KEY="AWS Secret Key"
# AWS_DEFAULT_REGION="AWS Region"

Get keys from OpenAI, Gemini, or AWS. For Zero Runtime, use an auth token from the Zero Runtime Dashboard or an API key + secret. Follow the guide.

Create the pipeline and agent

Build the pipeline first, then create the agent that uses it. The Pipeline auto-detects its mode from the components you pass it: provide STT, LLM, and TTS for Cascade, or a single realtime model for Realtime. Define it at module scope so the agent class can reference it directly. The agent inherits from the base Agent class, the same for both pipelines. Only the pipeline setup and imports differ. instructions defines the agent’s personality, on_enter() runs when it joins the room, and on_exit() runs when it leaves.

Cascade
Realtime

main.py

import zrt
from zrt import Agent, Pipeline, Room
from zrt.plugins import SileroVAD, TurnDetector, CartesiaSTT, GoogleLLM, CartesiaTTS
from zrt.inference import AICousticsDenoise

AGENT_ID = "assistant"

# Create the pipeline first - STT, LLM, and TTS resolve through the Zero Runtime
# Inference gateway. Build it at module scope so the agent class can reference it.
pipeline = Pipeline(
    stt=CartesiaSTT(model="ink-2"),
    llm=GoogleLLM(model="gemini-2.5-flash"),
    tts=CartesiaTTS(model="sonic-2"),
    vad=SileroVAD(threshold=0.4),
    turn_detector=TurnDetector(model="echo-large"),
    denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"),
)


class MyVoiceAgent(Agent):
    def __init__(self):
        super().__init__(
            agent_id=AGENT_ID,
            instructions="You are a helpful voice assistant that can answer questions and help with tasks.",
            pipeline=pipeline,
        )

    async def on_enter(self):
        await self.session.say("Hello! How can I help?")

    async def on_exit(self):
        await self.session.say("Goodbye!")

main.py

import zrt
from zrt import Agent, Pipeline, Room
from zrt.plugins import OpenAIRealtime, OpenAIRealtimeConfig

AGENT_ID = "assistant"

# Create the pipeline first - a single realtime model, built at module scope
# so the agent class can reference it.
model = OpenAIRealtime(
    config=OpenAIRealtimeConfig(
        model="gpt-4o-realtime-preview",
        voice="alloy",  # alloy, ash, ballad, breeze, cedar, cinnamon, coral, echo, ember, juniper, marin, sage, shimmer, verse
        modalities=["text", "audio"],
        # turn_detection defaults to server VAD (threshold 0.5); pass an
        # OpenAIRealtimeConfig turn_detection to customize it.
    ),
)
pipeline = Pipeline(llm=model)


class MyVoiceAgent(Agent):
    def __init__(self):
        super().__init__(
            agent_id=AGENT_ID,
            instructions="You are a helpful voice assistant that can answer questions and help with tasks.",
            pipeline=pipeline,
        )

    async def on_enter(self):
        await self.session.say("Hello! How can I help?")

    async def on_exit(self):
        await self.session.say("Goodbye!")

Start the session

With the pipeline and agent defined, hand the agent class to zrt.serve(...). This wiring is the same for both pipelines.

main.py

def invoke_agent():
    # room_id="YOUR_MEETING_ID"  # Set to join a pre-created room; omit to auto-create
    zrt.invoke(AGENT_ID, room=Room(playground=True))


if __name__ == "__main__":
    # Pass the class itself (not an instance): serve() builds a fresh MyVoiceAgent +
    # pipeline per call, which is required for correct per-call state under concurrent calls.
    zrt.serve(MyVoiceAgent, on_ready=invoke_agent)

serve() registers the agent under its agent_id and blocks, starting a fresh session for each caller. Passing the MyVoiceAgent class itself (not a pre-built instance) lets serve() construct a fresh agent + pipeline per call, which is required for correct per-call state under concurrent calls. on_ready=invoke_agent fires zrt.invoke(...) once the agent is registered, which resolves a room and returns a playground URL to open the session.

Run the project

With your .env configured and dependencies installed, run the project with Python.

Playground
Console mode

Once you run this command, a playground URL appears in your terminal. Use this URL to interact with your AI agent.

uv run python main.py

python main.py

uv run python main.py console

python main.py console

Want to see the magic instantly? Console mode lets you interact with your agent directly through the terminal, no meeting room needed, just speak and listen through your local system. Perfect for quick testing and development.

Learn more about Console Mode.

Verify

The agent connects to the room and greets you.
Audio flows both ways: the agent hears you and you hear it.
In Cascade mode, the LLM’s responses are spoken back through TTS; in Realtime mode, the model streams speech directly.

Common errors:

Missing auth token: set ZRT_AUTH_TOKEN in your .env.
Plugin import error: make sure zrt is installed in the active virtual environment (every provider ships with it).
No audio: check microphone permissions, or confirm the client joined the same room.

Get started quickly with the Quick Start Example for the Zero Runtime AI Agent SDK: everything you need to build your first AI agent fast.

Troubleshooting

Common issues when setting up the project:

No interpreter found for Python 3.11

Make sure Python 3.11 is installed, then create the virtual environment with it.

python3.11 -m venv venv
source venv/bin/activate      # macOS/Linux

RuntimeError: There is no current event loop

Newer Python versions (3.13+) removed the implicit event loop. Create your virtual environment with Python 3.11.

python3.11 -m venv venv
source venv/bin/activate      # macOS/Linux

ModuleNotFoundError when running main.py

You’re running a Python outside the project’s virtual environment. Activate it first, then install the dependencies.

source venv/bin/activate      # macOS/Linux
pip install zrt
python main.py

Get started

Quickstarts

SDKs

Build Your First Voice AI Agent

Prerequisites

Understanding the Architecture

Build the Agent

Verify

Troubleshooting

What’s Next

Add tools to an agent

Pipelines

References

​Prerequisites

​Understanding the Architecture

​Build the Agent

​Verify

​Troubleshooting

​What’s Next

Add tools to an agent

Pipelines

​References

Prerequisites

Understanding the Architecture

Build the Agent

Verify

Troubleshooting

What’s Next

References