Pipeline wires these components together and
exposes hooks for observing and shaping each turn.
Pipeline
The providers a call runs on: what hears, what thinks, what speaks. Two shapes are valid. A cascade namesstt, llm and tts
separately; a realtime pipeline names a speech-to-speech model in llm
alone, because a realtime model is the language model. realtime= is
accepted as the older spelling of the same slot; naming both is rejected.
A fallback slot — stt, llm, tts — also takes a list, in which
case the head serves and the tail stands by. Wrap that list in FallbackSTT,
FallbackLLM or FallbackTTS to say how it demotes as well as what it
falls back to; each slot carries its own settings.
Fields
Any
Speech-to-text, or a
FallbackSTT chain. Omit it on a realtime pipeline.Any
The language model — a text LLM for a cascade, a
FallbackLLM chain of them, or a speech-to-speech model for a realtime pipeline.Any
Text-to-speech, or a
FallbackTTS chain. Omit it on a realtime pipeline.Any
Voice activity detection — what notices the caller is talking.
Any
End-of-turn detection, for deciding when they finished.
Any
Noise cancellation applied to the inbound audio.
Any
The older spelling of a speech-to-speech
llm.Any
A video avatar to render the agent’s speech through.
Any
EOUConfig — how patiently the agent waits before answering.Any
InterruptConfig — what counts as the caller barging in.list[PronunciationRule]
default:"…"
PronunciationRule substitutions applied to generated text on its way to TTS.Optional[ContextWindow]
ContextWindow — how a long conversation is kept inside the model’s limit.Any
DTMFHandler — deliver keypad tones to the agent. None drops them.Any
VoiceMailDetector — answering-machine detection, for outbound calls that may be picked up by one.dict[str, tuple[ProviderSpec, ...]]
default:"…"
dict[str, _Fallback]
default:"…"
dict[str, list]
default:"…"
on
stt
and llm, and the turn hooks user_turn_start, user_turn_end,
agent_turn_start and agent_turn_end.
An async generator registered on llm is filed as llm_stream: it
is handed the model’s output as a stream and yields what should be
spoken, so it can buffer, rewrite or drop the turn. A plain coroutine on
the same event sees the finished text instead.
Per-component latency is a separate family, registered through its own
namespace rather than by spelling the prefix here::
@pipeline.metrics.on(“stt”)
def on_stt(data: dict) -> None:
…
str
required
Which hook to attach to.
Any
The handler. Omitted, this returns a decorator.
Any
The decorator, or the handler it registered.
hooks
str
required
list
chain
str
required
tuple[ProviderSpec, ...]
primary
None if it has none.
str
required
ProviderSpec | None
fallbacks
str
required
tuple[ProviderSpec, ...]
fallback_settings
None where nothing was tuned.
fallbacks is who a slot falls back to; this is how.
str
required
Optional[_Fallback]
providers
Iterable[ProviderSpec]
NO_CHANGE
change_component(denoise=None) takes denoising off a running call, while
leaving denoise out keeps whatever the call is already denoising with.
PronunciationRule
Rewrite generated text on its way to TTS.Fields
str
required
The text to look for.
str
required
What to say instead.
bool
default:"False"
Match case exactly. Off by default, because the model capitalises the same word differently at the start of a sentence.
ContextWindow
Keep a long conversation inside the model’s context, automatically.Fields
Optional[int]
Compress once the context exceeds this.
None for no token ceiling.Optional[int]
Or bound by item count.
None for none.int
default:"3"
Recent user turns kept verbatim. The SDK’s default is 3, and it is worth keeping several — summarising the sentence the caller just said is how an agent starts answering the wrong question.
int
default:"10"
The SDK’s default is 10.
Any
The model that compresses. A second LLM, described here and built in the agent process like every other provider.
EOUConfig
When the caller is judged to have finished speaking.Fields
str
default:"DEFAULT"
DEFAULT waits a fixed time after speech stops. ADAPTIVE varies it with how certain the turn detector is — longer when the sentence sounds unfinished.List[float]
default:"…"
[min, max] seconds to wait before treating silence as the end of a turn. [0.0, 0.0] answers as soon as the detector says the turn is complete, which is snappy but cuts off anyone who pauses to think.float
default:"0.75"
How sure the detector must be. Lower answers sooner and interrupts more.
Optional[bool]
Treat “mhm”, “haan”, “right” as listening noises rather than turns.
None takes the runtime’s default; the SDK’s own default is on for detectors that support it.InterruptConfig
What counts as the caller interrupting, rather than just making a noise.Fields
str
default:"HYBRID"
VAD_ONLY reacts to any speech-like audio — fastest, and the most easily fooled by a cough or a door. STT_ONLY waits for words. HYBRID uses both.float
default:"0.0"
Seconds of speech before it counts. Raising this is the usual fix for an agent that stops every time someone breathes.
int
default:"0"
Words required before it counts.
2 ignores a stray “yeah”.float
default:"0.0"
STT confidence floor for those words.
float
default:"0.0"
How long to stay paused after an interruption that turned out to be nothing.
bool
default:"False"
Pick the sentence back up when the interruption proves false, rather than dropping it.
float
default:"0.0"
Seconds to fade the agent’s audio out over. An instant cut sounds like a dropped call.
FallbackSTT
Speech-to-text providers to try in order, and how they demote.Fields
Sequence[ProviderSpec]
default:"()"
Optional[float]
Optional[int]
Optional[float]
Optional[int]
FallbackLLM
Language models to try in order, and how they demote.Fields
Sequence[ProviderSpec]
default:"()"
Optional[float]
Optional[int]
Optional[float]
Optional[int]
FallbackTTS
Text-to-speech providers to try in order, and how they demote.Fields
Sequence[ProviderSpec]
default:"()"
Optional[float]
Optional[int]
Optional[float]
Optional[int]
DTMFHandler
Deliver the caller’s keypad tones to the agent instead of dropping them.Fields
Any
What runs per keypress, taking
(key) or (key, payload). Left None, the agent’s on_dtmf method is called instead — which is the usual shape, because a keypad menu almost always wants the agent’s own state to accumulate a multi-digit entry into. The runtime delivers one key per call either way; a PIN is accumulated on your side, not handed over whole.VoiceMailDetector
Detect an answering machine on an outbound call. The detector runs in the agent process the ZeroRuntime starts: it buffers the opening speech forduration seconds and asks llm whether it is a
person or a greeting.
Fields
Any
The classifier. Required: without one there is nothing to classify with, and the detector is simply not installed.
Any
What runs on detection. Left
None, the agent’s on_voicemail method is called instead. Either is awaited, so anything said in it finishes before the call is ended.float
default:"2.0"
Seconds of speech to buffer before deciding. The default is 2.0 — long enough for “Hi, you’ve reached…”.
Optional[str]
Replace the built-in classification prompt.
bool
default:"True"
Set False to configure it without turning it on.
PipelineMode
Which shape a pipeline resolved to. Derived from the slots that were filled rather than set by hand; read it offPipeline.mode.