AI Voice Agent with Unity

This guide demonstrates how to integrate a real-time Zero Runtime AI agent with a Unity application. The agent, powered by Google’s Gemini Live API, acts as a high-energy game show host, guiding the user through a number-guessing game via voice.

Prerequisites

Before you begin, ensure you have the following:

A Zero Runtime account (create one from the Zero Runtime Dashboard).
Unity 2022.3 LTS or later.
Python 3.11+ for the AI agent.
A Google API Key for the Gemini Live API.

You need a Zero Runtime account to generate a token and a Google API key for the Gemini Live API. Generate a token from the Zero Runtime Dashboard and a Google API key from Google AI Studio.

1. Unity Frontend

Step 1: Install the RTC SDK package

Open Unity’s Package Manager by selecting from the top bar: Window -> Package Manager.
Click the + button in the top left corner and select Add package from git URL.
Paste the following URL and click Add:

https://github.com/videosdk-live/videosdk-rtc-unity-sdk.git

Install the package from the Unity Package Manager using the git URL above.

Add the com.unity.nuget.newtonsoft-json package by following the instructions provided here.

Step 2: Platform setup

Android
iOS

To integrate the RTC SDK into your Android project, follow these steps:

Add the following repository configuration to your settingsTemplate.gradle file:

settingsTemplate.gradle

dependencyResolutionManagement {
    repositoriesMode.set(RepositoriesMode.PREFER_SETTINGS)
    repositories {
        **ARTIFACTORYREPOSITORY**
        google()
        mavenCentral()
        jcenter()
        maven {
            url = uri("https://maven.aliyun.com/repository/jcenter")
        }
        flatDir {
            dirs "${project(':unityLibrary').projectDir}/libs"
        }
    }
}

Install the Android SDK in mainTemplate.gradle:

mainTemplate.gradle

dependencies {
    implementation 'live.videosdk:rtc-android-sdk:0.3.1'
}

If your project has set android.useAndroidX=true, then set android.enableJetifier=true in the gradleTemplate.properties file to migrate your project to AndroidX and avoid duplicate class conflicts.

gradleTemplate.gradle

android.enableJetifier = true;
android.useAndroidX = true;
android.suppressUnsupportedCompileSdk = 34;

Step 3: Create a meeting room

Create a static roomId using the Zero Runtime API. Both the Unity app and the AI agent will use this ID to connect to the same meeting.

curl -X POST https://api.videosdk.live/v2/rooms \
  -H "Authorization: YOUR_JWT_TOKEN_HERE" \
  -H "Content-Type: application/json"

Replace YOUR_JWT_TOKEN_HERE with your Zero Runtime auth token. Copy the roomId from the response for the next steps.

Step 4: Configure the Unity project

Update GameManager.cs with your Zero Runtime auth token and the roomId you just created.

Unity/Assets/Scripts/GameManager.cs

// ... existing code ...
public class GameManager : MonoBehaviour
{
    // ... existing code ...
    private readonly string _token = "YOUR_ZRT_AUTH_TOKEN";
    private readonly string _staticMeetingId = "YOUR_MEETING_ID";
    // ... existing code ...
}

Step 5: Set up platform-specific permissions

Android
iOS

Add the following permissions to your AndroidManifest.xml file:

AndroidManifest.xml

<uses-permission android:name="android.permission.CAMERA"/>
<uses-permission android:name="android.permission.RECORD_AUDIO"/>
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS"/>

Ensure your Info.plist includes descriptions for camera and microphone usage:

Info.plist

<key>NSCameraUsageDescription</key>
<string>Camera access is required for video calls.</string>
<key>NSMicrophoneUsageDescription</key>
<string>Microphone access is required for audio calls.</string>

2. Python AI Agent

Step 1: Setup the project

Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install zrt and python-dotenv. Every provider plugin ships with zrt.

uv venv --python 3.11
source .venv/bin/activate     # macOS/Linux
.\.venv\Scripts\activate      # Windows

python3.11 -m venv venv
source venv/bin/activate      # macOS/Linux
.\venv\Scripts\activate       # Windows

New to uv? See the uv install guide.

uv pip install zrt python-dotenv

pip install zrt python-dotenv

Step 2: Configure environment variables

Create a .env file in the Unity-quickstart directory to store your keys and token securely for the Python agent:

.env

# Google API Key for the Gemini Live API
GOOGLE_API_KEY=your_google_api_key_here

# Option A: pre-generated ZRT auth token
ZRT_AUTH_TOKEN=your_zrt_auth_token_here

# Option B: SDK auto-generates the token (omit ZRT_AUTH_TOKEN)
# ZRT_API_KEY=your_zrt_api_key
# ZRT_SECRET_KEY=your_zrt_secret_key

Step 3: Create the AI agent backend

The Python agent joins the same meeting room to interact with the user. Create agent-unity.py with your roomId.

agent-unity.py

import os
import zrt
from zrt import Agent, Pipeline, Room
from zrt.plugins import GeminiRealtime, GeminiLiveConfig
from zrt.inference import AICousticsDenoise
from dotenv import load_dotenv

load_dotenv()

AGENT_ID = "unity-game-show-agent"

pipeline = Pipeline(
    llm=GeminiRealtime(
        api_key=os.getenv("GOOGLE_API_KEY"),  # optional when GOOGLE_API_KEY is in .env
        config=GeminiLiveConfig(model="gemini-3.1-flash-live-preview", voice="Leda", response_modalities=["AUDIO"]),
    ),
    denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"),
)


class MyVoiceAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            name="MyVoiceAgent",
            agent_id=AGENT_ID,
            instructions="You are a high-energy game-show host guiding the caller to guess a secret number from 1 to 100 to win 1,000,000$.",
            pipeline=pipeline,
        )

    async def on_enter(self) -> None:
        await self.session.say(
            "Welcome to the Zero Runtime AI Agent game show! I'm your host, and we're about to play for 1,000,000$. Are you ready to play?"
        )

    async def on_exit(self) -> None:
        await self.session.say("Goodbye!")


def invoke_agent() -> None:
    # Join the same room the Unity frontend created (its meeting ID).
    zrt.invoke(AGENT_ID, room=Room(room_id="YOUR_MEETING_ID"))


if __name__ == "__main__":
    zrt.serve(MyVoiceAgent, on_ready=invoke_agent)

3. Run the Application

Step 1: Run the AI agent

Open a terminal, navigate to the Unity-quickstart directory, activate your virtual environment, and run the agent. It reads GOOGLE_API_KEY from the .env file.

uv run python agent-unity.py

python agent-unity.py

Step 2: Run the Unity application

Open the Unity/ project in Unity Hub.
Once the project is loaded, press the Play button in the Unity Editor to start the application.
Click the Join Meeting button in the app to connect to the session.

Once you join the meeting, the AI agent will greet you and start the game.

Troubleshooting

'Waiting for participant...' but no connection

Ensure both the Unity application and the agent are running.
Check that the roomId matches in GameManager.cs and agent-unity.py.
Verify your Zero Runtime token is valid.

Audio not working

Check platform permissions for microphone access (see Step 5 in Part 1).
Ensure your Google API key has Gemini Live API access enabled.

Agent not responding

Verify your GOOGLE_API_KEY is set in the .env file.
Check that the Gemini Live API is enabled in your Google Cloud Console.

Next Steps

Quickstart Example

Complete working example with source code.

Build Your First Voice Agent

Learn the Cascade and Realtime pipelines in depth.

Get started

Quickstarts

SDKs

AI Voice Agent with Unity

Prerequisites

1. Unity Frontend

2. Python AI Agent

3. Run the Application

Troubleshooting

Next Steps

Quickstart Example

Build Your First Voice Agent

​Prerequisites

​1. Unity Frontend

​2. Python AI Agent

​3. Run the Application

​Troubleshooting

​Next Steps

Quickstart Example

Build Your First Voice Agent

Prerequisites

1. Unity Frontend

2. Python AI Agent

3. Run the Application

Troubleshooting

Next Steps