TTS Caching

Many phrases in a voice agent never change: the greeting, the “let me check that for you” hold message, the goodbye. Synthesizing them through a TTS provider on every call costs roughly 300 to 800 ms per phrase and burns provider credits. Synthesize each one once, keep the PCM bytes, and replay them on every subsequent session.say().

How it works

session.say() accepts an audio_data argument: pre-synthesized PCM that bypasses TTS entirely. When it is set, the runtime plays your bytes instead of calling the provider, so playback starts immediately. text is still required alongside audio_data. It is what lands in the transcript and the chat context; the audio is only what the caller hears.

The bytes must be PCM in the room’s audio format. What produces them is up to you: your TTS vendor’s SDK, a file you decoded ahead of time, or a fetch from your own storage. The runtime plays what it is given.

Cache the phrases

Wrap a synthesizer in a cache that stores each phrase on first use. Later calls for the same phrase return the stored bytes without touching the provider, concurrent calls for one phrase share a single synthesis, and entries fall out LRU past max_entries.

Python

import asyncio
import hashlib
from collections import OrderedDict
from typing import Awaitable, Callable


class TTSAudioCache:
    """Reuse-by-key cache for TTS-synthesized audio."""

    def __init__(
        self,
        synthesize: Callable[[str], Awaitable[bytes]],
        max_entries: int = 128,
    ) -> None:
        self._synthesize = synthesize
        self._max_entries = max_entries
        self._store: OrderedDict[str, bytes] = OrderedDict()
        self._locks: dict[str, asyncio.Lock] = {}

    async def fetch(self, text: str) -> bytes:
        key = hashlib.sha256(text.encode()).hexdigest()
        cached = self._store.get(key)
        if cached is not None:
            self._store.move_to_end(key)
            return cached

        # One synthesis per phrase, even if several turns ask at once.
        async with self._locks.setdefault(key, asyncio.Lock()):
            cached = self._store.get(key)
            if cached is not None:
                self._store.move_to_end(key)
                return cached
            audio = await self._synthesize(text)
            self._store[key] = audio
            self._store.move_to_end(key)
            while len(self._store) > self._max_entries:
                self._store.popitem(last=False)
            return audio

    async def preload(self, texts: list[str]) -> None:
        """Synthesize and cache a batch up front, e.g. at startup."""
        for text in texts:
            await self.fetch(text)
        self._locks.clear()

Warm the fixed phrases before the first caller arrives, so nobody pays the synthesis:

Python

cache = TTSAudioCache(synthesize)

if __name__ == "__main__":
    asyncio.run(cache.preload([GREETING, HOLD, GOODBYE]))
    zeroruntime.serve(SupportAgent, on_ready=on_ready)

Supplying the synthesizer

The pipeline’s TTS plugin is a configuration object, not a synthesizer: synthesis happens on the runtime, so the plugin cannot produce audio in your process. The cache needs its own coroutine that returns raw PCM at the agent track’s sample rate, 24 kHz mono 16-bit signed. Name the model and voice once and pass them to both the plugin and the synthesizer, or cached lines will sound like a different speaker than the agent’s dynamic speech.

Python

from cartesia import AsyncCartesia

TTS_MODEL, TTS_VOICE, TTS_LANGUAGE = "sonic-2", "<voice-id>", "en"

client = AsyncCartesia(api_key=os.environ["CARTESIA_API_KEY"])


async def synthesize(text: str) -> bytes:
    chunks = []
    async for chunk in client.tts.bytes(
        model_id=TTS_MODEL,
        transcript=text,
        voice={"mode": "id", "id": TTS_VOICE},
        language=TTS_LANGUAGE,
        output_format={
            "container": "raw",
            "encoding": "pcm_s16le",
            "sample_rate": 24_000,
        },
    ):
        chunks.append(chunk)
    return b"".join(chunks)


pipeline = Pipeline(
    tts=CartesiaTTS(model=TTS_MODEL, voice=TTS_VOICE, language=TTS_LANGUAGE),
    # ...stt, llm, vad, turn_detector
)

Any vendor works. The cache only needs a coroutine returning raw PCM at the track’s sample rate.

Replay in `on_enter` and `on_exit`

The agent’s fixed opening and closing lines are the clearest win: both are known before the call starts, and both sit on the critical path where latency is most audible.

Python

class SupportAgent(Agent):
    async def on_enter(self) -> None:
        await self.session.say(GREETING, audio_data=await cache.fetch(GREETING))

    async def on_exit(self) -> None:
        await self.session.say(GOODBYE, audio_data=await cache.fetch(GOODBYE))

Overlap a hold phrase with a slow operation

Cached audio suits filler speech during database lookups, API calls, or RAG retrieval. Start the phrase as a task and let it play while the work runs, rather than waiting for it to finish first. Pass add_to_chat_context=False so the filler line does not enter the LLM’s context.

Python

@function_tool
async def check_order_status(self, order_id: str) -> dict:
    """Look up an order.

    Args:
        order_id: The order number the caller gives you.
    """
    hold_audio = await cache.fetch(HOLD)
    hold = asyncio.create_task(
        self.session.say(HOLD, audio_data=hold_audio, add_to_chat_context=False)
    )
    order = await db.get_order(order_id)
    await hold
    return order

Pre-recorded audio

The same slot plays produced or branded audio, a recorded human voice or an IVR jingle. Decode the file to 24 kHz mono 16-bit PCM first, then pass the bytes; no TTS provider is involved either way.

Python

with wave.open("greeting.wav", "rb") as wav:
    audio = wav.readframes(wav.getnframes())

await self.session.say(GREETING, audio_data=audio)

Parameters

session.say(text, *, interrupt=False, interruptible=None, add_to_chat_context=None, audio_data=None)

Parameter	Type	Default	Description
`text`	`str`	—	The line to speak. Required even when `audio_data` is set: it is what enters the transcript and chat context.
`audio_data`	`bytes`	`None`	Pre-synthesized PCM in the room’s audio format. Bypasses TTS entirely. Unset means the runtime synthesizes `text` normally.
`add_to_chat_context`	`bool`	`True`	Whether the line joins the chat context. Set `False` for filler and hold phrases.
`interrupt`	`bool`	`False`	Cut off whatever is playing before speaking this.
`interruptible`	`bool`	`True`	Whether the caller can barge in over this line.

Get started

Modalities

Configure Pipeline

Tools and Capabilities

Turn Detection and Interruptions

Telephony Agents

Context Management

Observability and Monitoring

How it works

Cache the phrases

Supplying the synthesizer

Replay in `on_enter` and `on_exit`

Overlap a hold phrase with a slow operation

Pre-recorded audio

Parameters

What’s Next

Background Audio

Audio Customization

References

Examples

Cached TTS

Examples

Cached TTS

​How it works

​Cache the phrases

​Supplying the synthesizer

​Replay in on_enter and on_exit

​Overlap a hold phrase with a slow operation

​Pre-recorded audio

​Parameters

​What’s Next

Background Audio

Audio Customization

​References

​Examples

Cached TTS

​Examples

Cached TTS

How it works

Cache the phrases

Supplying the synthesizer

Replay in `on_enter` and `on_exit`

Overlap a hold phrase with a slow operation

Pre-recorded audio

Parameters

What’s Next

References

Examples

Examples