Attach RAG to your Agent

RAG (Retrieval-Augmented Generation) helps your AI agent find relevant information from documents to give better answers. It searches a knowledge base and uses that context to respond more accurately.

Architecture

The RAG pipeline flow:

Voice Input

STT converts speech to text.

Retrieval

The knowledge base fetches relevant documents based on the transcript.

Augmentation

Retrieved context is injected into the LLM prompt.

Generation

The LLM generates a grounded response using the context.

Voice Output

TTS converts the response to speech.

Custom RAG

For full control, Build your own RAG pipeline using any vector database (ChromaDB, Pinecone, etc.) with the user_turn_start hook.This hook fires when the user’s transcript is ready, before the LLM is called, giving you the perfect place to retrieve documents and inject context. 1. Set up the vector store. Create a collection to hold your document embeddings. ChromaDB ships Python and JavaScript clients; in Go, use any vector store client. The snippet below keeps embeddings in a minimal in-memory index.

self.chroma_client = chromadb.Client()
self.collection = self.chroma_client.create_collection(name="rag_docs")

2. Implement retrieve(). Generate a query embedding and search the vector store for the top matching documents.

async def retrieve(self, query: str, k: int = 2) -> list[str]:
    response = await self.openai_client.embeddings.create(
        input=query, model="text-embedding-ada-002"
    )
    query_embedding = response.data[0].embedding
    results = self.collection.query(
        query_embeddings=[query_embedding], n_results=k
    )
    return results["documents"][0] if results["documents"] else []

3. Inject context with the user_turn_start hook. Retrieve documents from the transcript and fold them into the system prompt with session.update_instructions(...) (updateInstructions in JavaScript, UpdateInstructions in Go) before the LLM is invoked. Rebuild from a base string each turn so the retrieved context doesn’t accumulate.

BASE_INSTRUCTIONS = "You are a helpful assistant. Ground your answers in the retrieved context."

@pipeline.on("user_turn_start")
async def on_user_turn_start(transcript: str):
    context_docs = await agent.retrieve(transcript)
    if context_docs:
        context_str = "\n\n".join(
            f"Document {i+1}: {doc}" for i, doc in enumerate(context_docs)
        )
        await agent.session.update_instructions(
            f"{BASE_INSTRUCTIONS}\n\nRetrieved Context:\n{context_str}\n\n"
            "Use this context to answer the user's question."
        )

The LLM then sees the injected context and uses it to generate a grounded answer.

Best Practices

Start by retrieving k=2-3 documents and adjust based on performance.
Keep document chunk sizes between 300-800 words.
Use persistent storage for your vector database in production.
Cache embeddings for frequently asked questions to reduce latency.
Handle retrieval failures gracefully so the agent can still respond.

Get started

Modalities

Configure Pipeline

Tools and Capabilities

Turn Detection and Interruptions

Telephony Agents

Context Management

Observability and Monitoring

Attach RAG to your Agent

Architecture

Custom RAG

Best Practices

What’s Next

Function Tools

MCP

​Architecture

​Custom RAG

​Best Practices

​What’s Next

Function Tools

MCP

Architecture

Custom RAG

Best Practices

What’s Next