Modes

The Pipeline is self-configuring: based on the components you hand it, it automatically runs in Cascade, Hybrid, or Realtime mode. Switch between the tabs below to compare each mode’s architecture, when to reach for it, and the code that selects it.

Supported Modes

There are three modes supported by the Pipeline.

1. Cascade (STT-LLM-TTS)

2. Realtime (LLM)

3. Hybrid (Cascade + Realtime)

Cascade
Realtime
Hybrid

Cascade mode runs the pipeline as separate stages: VAD → STT → Turn Detector → LLM → TTS. The Pipeline auto-detects it when you pass all these components, giving you full control over each stage.ArchitectureAudio flows: VAD detects speech, STT transcribes, the turn detector signals end of turn, the LLM generates a reply, and TTS synthesizes audio.

When to use it

You need granular control over each stage (STT, LLM, TTS, VAD, turn detection).
You want to mix and match providers for individual stages.
You rely on custom hooks for turn-taking, RAG, or content filtering.
You can trade higher latency for maximum flexibility.

Example

import zrt
from zrt import Agent, Pipeline
from zrt.plugins import DeepgramSTT, OpenAILLM, ElevenLabsTTS, SileroVAD, TurnDetector
from zrt.inference import AICousticsDenoise

AGENT_ID = "assistant"

pipeline = Pipeline(
    stt=DeepgramSTT(),
    llm=OpenAILLM(),
    tts=ElevenLabsTTS(),
    vad=SileroVAD(),
    turn_detector=TurnDetector(model="echo-large"),
    denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"),
)

class Assistant(Agent):
    def __init__(self):
        super().__init__(
            agent_id=AGENT_ID,
            instructions="You are a helpful voice assistant.",
            pipeline=pipeline,
        )

if __name__ == "__main__":
    # Pass the class itself (not an instance): serve() builds a fresh Assistant +
    # pipeline per call, which is required for correct per-call state under concurrent calls.
    zrt.serve(Assistant, on_ready=lambda: zrt.invoke(AGENT_ID, room=zrt.Room(playground=True)))

Realtime mode runs a single end-to-end speech-to-speech model (for example, OpenAI Realtime, Google Gemini Live, or AWS Nova Sonic). The Pipeline auto-detects it when you pass a realtime model as llm, so no separate STT or TTS stages are required.ArchitectureUser audio streams to the realtime model, which transcribes, reasons, synthesizes, and streams audio back.

When to use it

Latency is your top priority.
You want the simplest setup with the fewest moving parts.
You need fast, natural conversational flow.
You don’t need to swap individual STT or TTS providers.

Example

from zrt import Pipeline
from zrt.plugins import OpenAIRealtime, OpenAIRealtimeConfig

model = OpenAIRealtime(
    config=OpenAIRealtimeConfig(model="gpt-4o-realtime-preview", voice="alloy", modalities=["audio", "text"])
)

# A realtime model goes in the LLM slot - the Pipeline auto-detects Realtime mode.
pipeline = Pipeline(llm=model)

Hybrid mode pairs a realtime model with an external STT or TTS. The Pipeline auto-detects the sub-mode from the extra component: hybrid_stt (external STT) or hybrid_tts (external TTS).Architecture

Hybrid STT: your STT produces a transcript, then both audio and transcript go to the realtime model. Use this when you need searchable text (for KB lookups).
Hybrid TTS: the realtime model handles input and reasoning, and its text output is routed to your TTS for the final voice.

When to use it

sub‑mode	Recommended use cases
Hybrid STT (`hybrid_stt`)	When you need a searchable transcript for knowledge‑base lookups or RAG, or when the realtime model lacks the language support or accuracy you require.
Hybrid TTS (`hybrid_tts`)	When you need a specific custom voice or advanced TTS features (for example, voice tuning or SSML) that the realtime model doesn’t provide.

ExampleProvide an external STT and the Pipeline auto-detects hybrid_stt:

from zrt import Pipeline
from zrt.plugins import GeminiRealtime, GeminiLiveConfig, SarvamAISTT, SileroVAD
from zrt.inference import AICousticsDenoise

model = GeminiRealtime(
    config=GeminiLiveConfig(
        model="gemini-3.1-flash-live-preview",
        voice="Puck",
        response_modalities=["AUDIO"]
    )
)

pipeline = Pipeline(
    stt=SarvamAISTT(),
    llm=model,
    vad=SileroVAD(),
    denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz")
)

Provide an external TTS and the Pipeline auto-detects hybrid_tts:

from zrt import Pipeline
from zrt.plugins import OpenAIRealtime, OpenAIRealtimeConfig, ElevenLabsTTS
from zrt.inference import AICousticsDenoise

model = OpenAIRealtime(
    config=OpenAIRealtimeConfig(model="gpt-4o-realtime-preview", voice="alloy")
)

pipeline = Pipeline(
    llm=model,
    tts=ElevenLabsTTS(),
    denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz")
)

Other Pipeline Combinations

The Pipeline is not limited to the named modes. It accepts any subset of components and configures itself accordingly.

Combination	Components	Use case
LLM + TTS	`llm` and `tts`	Text in, voice out. The agent receives text input and replies with speech.
STT + LLM	`stt` and `llm`	Voice in, text out. The user speaks and the agent replies with text.
Partial	Any other subset	Custom setups where you only need part of the conversational loop.

# Text in, voice out
pipeline = Pipeline(llm=OpenAILLM(), tts=ElevenLabsTTS())

# Voice in, text out
pipeline = Pipeline(stt=DeepgramSTT(), llm=OpenAILLM())

What’s Next

Runtime Hooks

Tap into the pipeline at runtime to inspect or transform each stage.

Observability Hooks

Observe metrics and traces at each pipeline stage.

Fallback Adapter

Add automatic provider failover for any mode.

References

Python

Examples

Cascade Mode

Run STT, LLM and TTS as a cascade.

Realtime Mode

Run a speech-to-speech realtime model.

SDK Reference

Pipeline

Pipeline in the Python API reference.

Get started

Modalities

Configure Pipeline

Tools and Capabilities

Turn Detection and Interruptions

Telephony Agents

Context Management

Observability and Monitoring

Supported Modes

1. Cascade (STT-LLM-TTS)

2. Realtime (LLM)

3. Hybrid (Cascade + Realtime)

Other Pipeline Combinations

What’s Next

Runtime Hooks

Observability Hooks

Fallback Adapter

References

Examples

Cascade Mode

Realtime Mode

SDK Reference

Pipeline

​Supported Modes

1. Cascade (STT-LLM-TTS)

2. Realtime (LLM)

3. Hybrid (Cascade + Realtime)

​Other Pipeline Combinations

​What’s Next

Runtime Hooks

Observability Hooks

Fallback Adapter

​References

​Examples

Cascade Mode

Realtime Mode

​SDK Reference

Pipeline

Supported Modes

Other Pipeline Combinations

What’s Next

References

Examples

SDK Reference