> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Build Your First Voice AI Agent

> Ship a voice agent in minutes using a realtime speech-to-speech model or a cascading STT-LLM-TTS pipeline.

## Prerequisites

Before you begin, ensure you have:

* A Zero Runtime Auth token **or** API key + secret from [app.zeroruntime.ai](https://app.zeroruntime.ai/api-keys) (see [Authentication](/authentication)).
* Python 3.11 or higher, or Node.js 20.11 or higher.

## Understanding the Architecture

Before diving into implementation, choose the pipeline architecture that fits your use case.

<Tabs>
  <Tab title="Cascade">
    **Cascade** processes audio through distinct stages for maximum control:

    <Frame>
      <img src="https://mintcdn.com/zeroruntime/dSfKUNLGJQu7XhPO/images/cascade.svg?fit=max&auto=format&n=dSfKUNLGJQu7XhPO&q=85&s=8876d95a726fb1deab5c13c325e98965" className="block w-full my-4" width="1160" height="200" data-path="images/cascade.svg" />
    </Frame>

    The cascade processes audio through sequential stages:

    **User Voice Input** → **STT (Speech-to-Text)** → **LLM (Large Language Model)** → **TTS (Text-to-Speech)** → **Agent Voice Output**

    This approach provides better control over each processing stage and supports more complex AI reasoning. Power each stage through the **Zero Runtime Inference gateway** (one token, no per-provider keys) or bring your own provider keys.
  </Tab>

  <Tab title="Realtime">
    **Realtime** provides direct speech-to-speech processing with minimal latency:

    <Frame>
      <img src="https://mintcdn.com/zeroruntime/dSfKUNLGJQu7XhPO/images/realtime.svg?fit=max&auto=format&n=dSfKUNLGJQu7XhPO&q=85&s=545348ab856240015f71226b8ceea2bf" className="block w-full my-4" width="880" height="280" data-path="images/realtime.svg" />
    </Frame>

    The realtime pipeline processes audio directly through a unified model:

    **User Voice Input** → **Speech-to-Speech model** → **Agent Voice Output**

    This approach offers the fastest response times and is ideal for real-time conversations.
  </Tab>
</Tabs>

## Build the Agent

Pick your language. Each track builds the same agent on the **Cascade** pipeline (Deepgram STT → Google Gemini LLM → Cartesia TTS).

<Tabs>
  <Tab title="Python">
    <Steps>
      <Step title="Setup the project" icon="download">
        Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install `zeroruntime`. Every provider plugin ships with it.

        <CodeGroup>
          ```bash uv theme={null}
          uv venv --python 3.11
          source .venv/bin/activate     # macOS/Linux
          .\.venv\Scripts\activate      # Windows
          ```

          ```bash pip theme={null}
          python3.11 -m venv venv
          source venv/bin/activate      # macOS/Linux
          .\venv\Scripts\activate       # Windows
          ```
        </CodeGroup>

        <Tip>
          New to uv? See the [uv install guide](https://docs.astral.sh/uv/getting-started/installation/).
        </Tip>

        <Tabs>
          <Tab title="Cascade">
            <CodeGroup>
              ```bash uv theme={null}
              uv pip install zeroruntime
              ```

              ```bash pip theme={null}
              pip install zeroruntime
              ```
            </CodeGroup>

            <Tip>
              Prefer to bring your own provider keys? Every provider ships with the SDK. Just import from `zeroruntime.plugins`. See the plugin pages for [STT](/plugins/stt/openai), [LLM](/plugins/llm/openai), and [TTS](/plugins/tts/elevenlabs).
            </Tip>
          </Tab>

          <Tab title="Realtime">
            <CodeGroup>
              ```bash uv theme={null}
              uv pip install zeroruntime
              ```

              ```bash pip theme={null}
              pip install zeroruntime
              ```
            </CodeGroup>
          </Tab>
        </Tabs>
      </Step>

      <Step title="Configure environment variables" icon="key">
        Store API keys and tokens in a `.env` file in your project root.

        <Tabs>
          <Tab title="Cascade">
            ```shell title=".env" theme={null}
            # The Inference gateway authenticates with your Zero Runtime token -
            # no separate STT, LLM, or TTS provider keys needed.

            # pre-generated token
            ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token"

            # Runtime Target
            ZERORUNTIME_TARGET=us2.zeroruntime.ai:443
            ```
          </Tab>

          <Tab title="Realtime">
            ```shell title=".env" theme={null}
            # pre-generated token
            ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token"

            # Runtime Target
            ZERORUNTIME_TARGET=us2.zeroruntime.ai:443

            OPENAI_API_KEY="Your OpenAI API Key"

            # For Google Live API
            # GOOGLE_API_KEY="Google Live API Key"

            # For AWS Nova API
            # AWS_ACCESS_KEY_ID="AWS Key Id"
            # AWS_SECRET_ACCESS_KEY="AWS Secret Key"
            # AWS_DEFAULT_REGION="AWS Region"
            ```

            <Note>
              Get keys from [OpenAI](https://platform.openai.com/api-keys), [Gemini](https://aistudio.google.com/app/apikey), or [AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html). For Zero Runtime, use an auth token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/api-keys) or an API key + secret. Follow the [guide](/authentication).
            </Note>
          </Tab>
        </Tabs>
      </Step>

      <Step title="Create the pipeline and agent" icon="robot">
        Build the pipeline first, then create the agent that uses it. The `Pipeline` auto-detects its mode from the components you pass it: provide STT, LLM, and TTS for Cascade, or a single realtime model for Realtime. Define it at module scope so the agent class can reference it directly. The agent inherits from the base `Agent` class, the same for both pipelines. Only the pipeline setup and imports differ. `instructions` defines the agent's personality, `on_enter()` runs when it joins the room, and `on_exit()` runs when it leaves.

        <Tabs>
          <Tab title="Cascade">
            ```python title="main.py" theme={null}
            import zeroruntime
            from zeroruntime import Agent, Pipeline, Room
            from zeroruntime.plugins import SileroVAD, CartesiaSTT, GoogleLLM, CartesiaTTS
            from zeroruntime.inference import AICousticsDenoise, TurnDetector

            AGENT_ID = "assistant"

            # Create the pipeline first - STT, LLM, and TTS resolve through the Zero Runtime
            # Inference gateway. Build it at module scope so the agent class can reference it.
            pipeline = Pipeline(
                stt=CartesiaSTT(model="ink-2"),
                llm=GoogleLLM(model="gemini-2.5-flash"),
                tts=CartesiaTTS(model="sonic-2"),
                vad=SileroVAD(threshold=0.4),
                turn_detector=TurnDetector(model="echo-large"),
                denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"),
            )


            class MyVoiceAgent(Agent):
                def __init__(self):
                    super().__init__(
                        agent_id=AGENT_ID,
                        instructions="You are a helpful voice assistant that can answer questions and help with tasks.",
                        pipeline=pipeline,
                    )

                async def on_enter(self):
                    await self.session.say("Hello! How can I help?")

                async def on_exit(self):
                    await self.session.say("Goodbye!")
            ```
          </Tab>

          <Tab title="Realtime">
            ```python title="main.py" theme={null}
            import zeroruntime
            from zeroruntime import Agent, Pipeline, Room
            from zeroruntime.plugins import OpenAIRealtime

            AGENT_ID = "assistant"

            # Create the pipeline first - a single realtime model, built at module scope
            # so the agent class can reference it.
            model = OpenAIRealtime(
                model="gpt-4o-realtime-preview",
                config={
                    "voice": "alloy",  # alloy, ash, ballad, breeze, cedar, cinnamon, coral, echo, ember, juniper, marin, sage, shimmer, verse
                    "modalities": ["text", "audio"],
                    # turn_detection defaults to server VAD (threshold 0.5); add a
                    # "turn_detection" key to customize it.
                },
            )
            pipeline = Pipeline(llm=model)


            class MyVoiceAgent(Agent):
                def __init__(self):
                    super().__init__(
                        agent_id=AGENT_ID,
                        instructions="You are a helpful voice assistant that can answer questions and help with tasks.",
                        pipeline=pipeline,
                    )

                async def on_enter(self):
                    await self.session.say("Hello! How can I help?")

                async def on_exit(self):
                    await self.session.say("Goodbye!")
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Start the session" icon="play">
        With the pipeline and agent defined, hand the agent *class* to `zeroruntime.serve(...)`. This wiring is the same for both pipelines.

        ```python title="Python" theme={null}
        def invoke_agent():
            # room_id="YOUR_MEETING_ID"  # Set to join a pre-created room; omit to auto-create
            zeroruntime.invoke(AGENT_ID, room=Room(playground=True))


        if __name__ == "__main__":
            # Pass the class itself (not an instance): serve() builds a fresh MyVoiceAgent +
            # pipeline per call, which is required for correct per-call state under concurrent calls.
            zeroruntime.serve(MyVoiceAgent, on_ready=invoke_agent)
        ```

        `serve()` registers the agent under its `agent_id` and blocks, starting a fresh session for each caller. Passing the `MyVoiceAgent` class itself (not a pre-built instance) lets `serve()` construct a fresh agent + pipeline per call, which is required for correct per-call state under concurrent calls. `on_ready=invoke_agent` fires `zeroruntime.invoke(...)` once the agent is registered, which resolves a room and returns a playground URL to open the session.
      </Step>

      <Step title="Run the project" icon="terminal">
        With your `.env` configured and dependencies installed, run the project with Python.

        <Tabs>
          <Tab title="Playground">
            Once you run this command, a playground URL appears in your terminal. Use this URL to interact with your AI agent.

            <CodeGroup>
              ```bash uv theme={null}
              uv run python main.py
              ```

              ```bash pip theme={null}
              python main.py
              ```
            </CodeGroup>
          </Tab>

          <Tab title="Console mode">
            <CodeGroup>
              ```bash uv theme={null}
              uv run python main.py console
              ```

              ```bash pip theme={null}
              python main.py console
              ```
            </CodeGroup>

            Want to see the magic instantly? Console mode lets you interact with your agent directly through the terminal, no meeting room needed, just speak and listen through your local system. Perfect for quick testing and development.

            <Frame>
              <img src="https://cdn.videosdk.live/website-resources/docs-resources/ai_agents_console_mode_image.png" alt="Console Mode" />
            </Frame>

            Learn more about [Console Mode](/build/configure-a-pipeline/console-mode).
          </Tab>
        </Tabs>
      </Step>
    </Steps>
  </Tab>

  <Tab title="Node JS">
    <Steps>
      <Step title="Setup the project" icon="download">
        Use Node.js 20.11 or higher, then install the Zero Runtime Agents SDK. Add `tsx` and `typescript` as dev dependencies so you can run the TypeScript entrypoint directly. Every provider plugin ships with the SDK.

        ```bash theme={null}
        npm install @zeroruntime/js-sdk dotenv
        npm install -D tsx typescript
        ```

        <Tip>
          New to `tsx`? It runs `.ts` files with no build step. See the [tsx docs](https://tsx.is/).
        </Tip>

        <Note>
          The SDK is ESM-only. Add `"type": "module"` to your `package.json`.
        </Note>

        <Tabs>
          <Tab title="Cascade">
            <Tip>
              Prefer to bring your own provider keys? Every provider ships with the SDK. Import from `@zeroruntime/js-sdk/plugins` instead of `@zeroruntime/js-sdk/inference`. See the plugin pages for [STT](/plugins/stt/openai), [LLM](/plugins/llm/openai), and [TTS](/plugins/tts/elevenlabs).
            </Tip>
          </Tab>

          <Tab title="Realtime">
            A realtime model needs only its own vendor key on top of your Zero Runtime token.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Configure environment variables" icon="key">
        Store API keys and tokens in a `.env` file in your project root. The SDK loads it via `import 'dotenv/config';` at the top of your entrypoint.

        <Tabs>
          <Tab title="Cascade">
            ```shell title=".env" theme={null}
            # The Inference gateway authenticates with your Zero Runtime token -
            # no separate STT, LLM, or TTS provider keys needed.

            # pre-generated token
            ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token"

            # Runtime Target
            ZERORUNTIME_TARGET=us2.zeroruntime.ai:443
            ```
          </Tab>

          <Tab title="Realtime">
            ```shell title=".env" theme={null}
            # pre-generated token
            ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token"

            # Runtime Target
            ZERORUNTIME_TARGET=us2.zeroruntime.ai:443

            OPENAI_API_KEY="Your OpenAI API Key"

            # For Google Live API
            # GOOGLE_API_KEY="Google Live API Key"

            # For AWS Nova API
            # AWS_ACCESS_KEY_ID="AWS Key Id"
            # AWS_SECRET_ACCESS_KEY="AWS Secret Key"
            # AWS_DEFAULT_REGION="AWS Region"
            ```

            <Note>
              Get keys from [OpenAI](https://platform.openai.com/api-keys), [Gemini](https://aistudio.google.com/app/apikey), or [AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html). For Zero Runtime, use an auth token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/api-keys) or an API key + secret. Follow the [guide](/authentication).
            </Note>
          </Tab>
        </Tabs>
      </Step>

      <Step title="Create the pipeline and agent" icon="robot">
        Build the pipeline first, then create the agent that uses it. `Pipeline` auto-detects its mode from the components you pass it: provide STT, LLM, and TTS for Cascade, or a single realtime model for Realtime. Define it at module scope so the agent class can reference it directly. The agent extends the base `Agent` class, the same for both pipelines. Only the pipeline setup and imports differ. `instructions` defines the agent's personality, `on_enter()` runs when it joins the room, and `on_exit()` runs when it leaves.

        <Note>
          `Pipeline` and every provider are factory calls, not constructors — no `new` — and each takes a single options object. Option names match the Python SDK, so they are `snake_case` in both.
        </Note>

        <Tabs>
          <Tab title="Cascade">
            ```typescript title="main.ts" theme={null}
            import 'dotenv/config';

            import { Agent, Pipeline } from '@zeroruntime/js-sdk';
            import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference';
            import { CartesiaSTT, CartesiaTTS, GoogleLLM, SileroVAD } from '@zeroruntime/js-sdk/plugins';

            const AGENT_ID = 'assistant';

            // Create the pipeline first - STT, LLM, and TTS resolve through the Zero Runtime
            // Inference gateway. Build it at module scope so the agent class can reference it.
            const pipeline = Pipeline({
              stt: CartesiaSTT({ model: 'ink-2' }),
              llm: GoogleLLM({ model: 'gemini-2.5-flash' }),
              tts: CartesiaTTS({ model: 'sonic-2' }),
              vad: SileroVAD({ threshold: 0.4 }),
              turn_detector: TurnDetector({ model: 'echo-large' }),
              denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }),
            });

            class MyVoiceAgent extends Agent {
              constructor() {
                super({
                  agent_id: AGENT_ID,
                  instructions:
                    'You are a helpful voice assistant that can answer questions and help with tasks.',
                  pipeline,
                });
              }

              async on_enter(): Promise<void> {
                await this.session!.say('Hello! How can I help?');
              }

              async on_exit(): Promise<void> {
                await this.session!.say('Goodbye!');
              }
            }
            ```
          </Tab>

          <Tab title="Realtime">
            ```typescript title="main.ts" theme={null}
            import 'dotenv/config';

            import { Agent, Pipeline } from '@zeroruntime/js-sdk';
            import { OpenAIRealtime } from '@zeroruntime/js-sdk/plugins';

            const AGENT_ID = 'assistant';

            // Create the pipeline first - a single realtime model, built at module scope
            // so the agent class can reference it.
            const model = OpenAIRealtime({
              model: 'gpt-4o-realtime-preview',
              config: {
                voice: 'alloy', // alloy, ash, ballad, breeze, cedar, cinnamon, coral, echo, ember, juniper, marin, sage, shimmer, verse
                modalities: ['text', 'audio'],
                // turn_detection defaults to server VAD (threshold 0.5); add a
                // turn_detection key to customize it.
              },
            });
            const pipeline = Pipeline({ llm: model });

            class MyVoiceAgent extends Agent {
              constructor() {
                super({
                  agent_id: AGENT_ID,
                  instructions:
                    'You are a helpful voice assistant that can answer questions and help with tasks.',
                  pipeline,
                });
              }

              async on_enter(): Promise<void> {
                await this.session!.say('Hello! How can I help?');
              }

              async on_exit(): Promise<void> {
                await this.session!.say('Goodbye!');
              }
            }
            ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Start the session" icon="play">
        With the pipeline and agent defined, hand the agent *class* to `zeroruntime.serve(...)`. This wiring is the same for both pipelines.

        ```typescript title="Node JS" theme={null}
        import * as zeroruntime from '@zeroruntime/js-sdk';
        import { Room } from '@zeroruntime/js-sdk';

        async function invoke_agent(): Promise<void> {
          // room_id: 'YOUR_MEETING_ID'  // Set to join a pre-created room; omit to auto-create
          await zeroruntime.invoke(AGENT_ID, { room: Room({ playground: true }) });
        }

        // Pass the class itself (not an instance): serve() builds a fresh MyVoiceAgent +
        // pipeline per call, which is required for correct per-call state under concurrent calls.
        await zeroruntime.serve(MyVoiceAgent, { on_ready: invoke_agent });
        ```

        `serve()` registers the agent under its `agent_id` and starts a fresh session for each caller. Passing the `MyVoiceAgent` class itself (not a pre-built instance) lets `serve()` construct a fresh agent + pipeline per call, which is required for correct per-call state under concurrent calls. `on_ready: invoke_agent` fires `zeroruntime.invoke(...)` once the agent is registered, which resolves a room and returns a playground URL to open the session.
      </Step>

      <Step title="Run the project" icon="terminal">
        With your `.env` configured and dependencies installed, run the project with `tsx`.

        <Tabs>
          <Tab title="Playground">
            Once you run this command, a playground URL appears in your terminal. Use this URL to interact with your AI agent.

            ```bash theme={null}
            npx tsx main.ts
            ```
          </Tab>

          <Tab title="Console mode">
            ```bash theme={null}
            npx tsx main.ts console
            ```

            Want to see the magic instantly? Console mode lets you interact with your agent directly through the terminal, no meeting room needed, just speak and listen through your local system. Perfect for quick testing and development.

            <Frame>
              <img src="https://cdn.videosdk.live/website-resources/docs-resources/ai_agents_console_mode_image.png" alt="Console Mode" />
            </Frame>

            Console mode is the same feature described in the Python track above.
          </Tab>
        </Tabs>
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Verify

* The agent connects to the room and greets you.
* Audio flows both ways: the agent hears you and you hear it.
* In Cascade mode, the LLM's responses are spoken back through TTS; in Realtime mode, the model streams speech directly.

<Warning>
  **Common errors:**

  * **Missing auth token:** set `ZERORUNTIME_AUTH_TOKEN` in your `.env`.
  * **Plugin import error:** make sure `zeroruntime` is installed in the active virtual environment, or `@zeroruntime/js-sdk` in the project. Every provider ships with the SDK.
  * **No audio:** check microphone permissions, or confirm the client joined the same room.
</Warning>

<Tip>
  Get started quickly with the [Quick Start Example](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples) for the Zero Runtime AI Agent SDK: everything you need to build your first AI agent fast.
</Tip>

## Troubleshooting

Common issues when setting up the project:

<AccordionGroup>
  <Accordion title="No interpreter found for Python 3.11" icon="code">
    Make sure Python 3.11 is installed, then create the virtual environment with it.

    ```bash theme={null}
    python3.11 -m venv venv
    source venv/bin/activate      # macOS/Linux
    ```
  </Accordion>

  <Accordion title="RuntimeError: There is no current event loop" icon="triangle-exclamation">
    Newer Python versions (3.13+) removed the implicit event loop. Create your virtual environment with Python 3.11.

    ```bash theme={null}
    python3.11 -m venv venv
    source venv/bin/activate      # macOS/Linux
    ```
  </Accordion>

  <Accordion title="ERR_REQUIRE_ESM or Cannot use import statement outside a module" icon="js">
    The SDK is ESM-only. Add `"type": "module"` to your `package.json`, or use a `.mts` entrypoint.

    ```json theme={null}
    { "type": "module" }
    ```
  </Accordion>

  <Accordion title="ModuleNotFoundError when running main.py" icon="box">
    You're running a Python outside the project's virtual environment. Activate it first, then install the dependencies.

    ```bash theme={null}
    source venv/bin/activate      # macOS/Linux
    pip install zeroruntime
    python main.py
    ```
  </Accordion>
</AccordionGroup>

## What's Next

<CardGroup cols={2}>
  <Card title="Add tools to an agent" icon="wrench" href="/quickstarts/add-tools-to-an-agent">
    Let your agent call functions and act.
  </Card>

  <Card title="Pipelines" icon="diagram-project" href="/concepts/pipeline">
    Understand the STT, LLM, and TTS flow.
  </Card>
</CardGroup>

## References

**Python**

* [AI Agent (Complete Example)](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples/blob/main/getting_started/cascade_basic.py)
* [Zero Runtime Inference Agent (Example)](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples/blob/main/inference/zeroruntime_cascade.py)

**Node JS**

* [AI Agent (Complete Example)](https://github.com/ZeroRuntimeAI/zeroruntime-js-examples/blob/main/getting_started/cascade_basic.ts)
* [Zero Runtime Inference Agent (Example)](https://github.com/ZeroRuntimeAI/zeroruntime-js-examples/blob/main/inference/zeroruntime_cascade.ts)