Attach RAG to your Agent

RAG (Retrieval-Augmented Generation) helps your AI agent find relevant information from documents to give better answers. It searches a knowledge base and uses that context to respond more accurately.

Architecture

The RAG pipeline flow:

Voice Input

STT converts speech to text.

Retrieval

The knowledge base fetches relevant documents based on the transcript.

Augmentation

Retrieved context is injected into the LLM prompt.

Generation

The LLM generates a grounded response using the context.

Voice Output

TTS converts the response to speech.

Managed RAG

Managed RAG lets you upload knowledge bases from the Zero runtime dashboard and connect them to your agent. During a conversation, the agent retrieves relevant information from these knowledge bases to provide more accurate, context-aware responses.

Managed RAG is supported in Python SDK 0.1.5+. JavaScript SDK support is coming soon.

Step 1: Upload a Knowledge Base

Upload your documents and create a knowledge base from the Zero runtime dashboard.

After the knowledge base is created, the dashboard provides a unique Knowledge ID.

Step 2: Attach the Knowledge Base

Pass the Knowledge ID to KnowledgeBase and set it as the pipeline’s knowledge_base. On every user turn the agent searches the attached knowledge bases and places the best-matching sections above the user’s message in the LLM prompt. Retrieval runs inside the agent process, so no extra code or hooks are needed.

from zeroruntime import KnowledgeBase, Pipeline
from zeroruntime.inference import CartesiaTTS, DeepgramSTT, GoogleLLM, TurnDetector
from zeroruntime.plugins import SileroVAD

pipeline = Pipeline(
    stt=DeepgramSTT(model="nova-2"),
    llm=GoogleLLM(model="gemini-3.5-flash"),
    tts=CartesiaTTS(),
    vad=SileroVAD(),
    turn_detector=TurnDetector(),
    knowledge_base=KnowledgeBase(
        knowledge_ids=["kb_id_1", "kb_id_2"],
        top_k=5,
    ),
)

You can attach multiple knowledge bases to the same agent. All of them are searched on every turn.

The knowledge_base pipeline parameter works with cascade pipelines (separate STT, LLM, and TTS) only. For a realtime pipeline, use the knowledge base as a function tool.

Configuration

Option	Default	Description
`knowledge_ids`	required	Dashboard IDs of the knowledge bases to search. Must list at least one non-empty ID.
`top_k`	`5`	Number of sections retrieved per turn. Must be at least `1`.
`context_prefix`	built-in line	Replaces the line placed above the retrieved sections in the prompt, e.g. to tell the LLM how to treat the context.
`api_base_url`	Zero Runtime	Host of the knowledge API. Leave unset unless you were given a different endpoint.

Customize the context prefix

Use context_prefix to control how the LLM is told about the retrieved sections.

knowledge_base = KnowledgeBase(
    knowledge_ids=["kb_id_1"],
    context_prefix="Answer only from the product documentation below. If it is not covered, say you don't know.",
)

report_unanswered is deprecated and ignored. Unanswered questions are no longer filed to the dashboard, so remove it from your configuration.

Knowledge Base as a Function Tool

knowledge_base= searches on every user turn. To let the model decide when to search instead, call KnowledgeBase.retrieve() from a function tool. This works with any pipeline, cascade or realtime, and skips the lookup on turns that don’t need it (greetings, small talk).

from zeroruntime import Agent, KnowledgeBase, function_tool

kb = KnowledgeBase(knowledge_ids=["kb_id_1"], top_k=5)


@function_tool
async def search_docs(query: str) -> str:
    """Search the product documentation and support policies."""
    return await kb.retrieve(query)


class SupportAgent(Agent):
    def __init__(self, pipeline):
        super().__init__(
            instructions=(
                "You are a support assistant. Call search_docs before answering "
                "product questions, and answer only from its results."
            ),
            pipeline=pipeline,
            tools=[search_docs],
        )

retrieve() returns the best sections as text, under the knowledge base’s context_prefix. Override settings for a single call with keyword arguments:

Argument	Description
`knowledge_ids`	Knowledge bases to search instead of the configured ones.
`top_k`	Sections to return instead of the configured `top_k`.
`context_prefix`	Line above the sections instead of the configured one. `""` drops it.

Use either the tool or Pipeline(knowledge_base=...), not both, or each turn searches twice.

Say in the agent’s instructions when to call the tool. Models skip tools they aren’t told to use and answer from their own knowledge.

Custom RAG

For full control, Build your own RAG pipeline using any vector database (ChromaDB, Pinecone, etc.) with the user_turn_start hook.This hook fires when the user’s transcript is ready, before the LLM is called, giving you the perfect place to retrieve documents and inject context. 1. Set up the vector store. Create a collection to hold your document embeddings. ChromaDB ships a Python client. The snippet below keeps embeddings in a minimal in-memory index.

self.chroma_client = chromadb.Client()
self.collection = self.chroma_client.create_collection(name="rag_docs")

// A vector store of your choosing; the SDK does not ship one.
this.collection = await vector_store.create_collection('rag_docs');

2. Implement retrieve(). Generate a query embedding and search the vector store for the top matching documents.

async def retrieve(self, query: str, k: int = 2) -> list[str]:
    response = await self.openai_client.embeddings.create(
        input=query, model="text-embedding-ada-002"
    )
    query_embedding = response.data[0].embedding
    results = self.collection.query(
        query_embeddings=[query_embedding], n_results=k
    )
    return results["documents"][0] if results["documents"] else []

async retrieve(query: string, k = 2): Promise<string[]> {
  const response = await this.openai_client.embeddings.create({
    input: query,
    model: 'text-embedding-ada-002',
  });
  const query_embedding = response.data[0].embedding;
  const results = await this.collection.query({
    query_embeddings: [query_embedding],
    n_results: k,
  });
  return results.documents?.[0] ?? [];
}

3. Inject context with the user_turn_start hook. Retrieve documents from the transcript and fold them into the system prompt with session.change_component(instructions=...) before the LLM is invoked. Rebuild from a base string each turn so the retrieved context doesn’t accumulate.

BASE_INSTRUCTIONS = "You are a helpful assistant. Ground your answers in the retrieved context."

@pipeline.on("user_turn_start")
async def on_user_turn_start(transcript: str):
    context_docs = await agent.retrieve(transcript)
    if context_docs:
        context_str = "\n\n".join(
            f"Document {i+1}: {doc}" for i, doc in enumerate(context_docs)
        )
        await agent.session.change_component(
            instructions=(
                f"{BASE_INSTRUCTIONS}\n\nRetrieved Context:\n{context_str}\n\n"
                "Use this context to answer the user's question."
            )
        )

const BASE_INSTRUCTIONS =
  'You are a helpful assistant. Ground your answers in the retrieved context.';

pipeline.on('user_turn_start', async (transcript: string) => {
  const context_docs = await agent.retrieve(transcript);
  if (context_docs.length) {
    const context_str = context_docs
      .map((doc, i) => `Document ${i + 1}: ${doc}`)
      .join('\n\n');
    await agent.session.change_component({
      instructions:
        `${BASE_INSTRUCTIONS}\n\nRetrieved Context:\n${context_str}\n\n` +
        "Use this context to answer the user's question.",
    });
  }
});

The LLM then sees the injected context and uses it to generate a grounded answer.

Best Practices

Start by retrieving k=2-3 documents and adjust based on performance.
Keep document chunk sizes between 300-800 words.
Use persistent storage for your vector database in production.
Cache embeddings for frequently asked questions to reduce latency.
Handle retrieval failures gracefully so the agent can still respond.

Overview

Build

Plugins

Deploy

Attach RAG to your Agent

Architecture

Managed RAG

Step 1: Upload a Knowledge Base

Step 2: Attach the Knowledge Base

Configuration

Customize the context prefix

Knowledge Base as a Function Tool

Custom RAG

Best Practices

What’s Next

Function Tools

MCP

​Architecture

​Managed RAG

​Step 1: Upload a Knowledge Base

​Step 2: Attach the Knowledge Base

​Configuration

​Customize the context prefix

​Knowledge Base as a Function Tool

​Custom RAG

​Best Practices

​What’s Next

Function Tools

MCP

Architecture

Managed RAG

Step 1: Upload a Knowledge Base

Step 2: Attach the Knowledge Base

Configuration

Customize the context prefix

Knowledge Base as a Function Tool

Custom RAG

Best Practices

What’s Next