Pipeline

The voice stack an agent runs on: speech-to-text, LLM, and text-to-speech, plus voice-activity and turn detection. Pipeline wires these components together and exposes hooks for observing and shaping each turn.

The voice stack an agent runs on: how it hears, thinks, and speaks. Compose a cascade by passing components (stt, llm, tts plus vad and turn_detector), or a speech-to-speech setup via realtime_config. Also carries behavior configuration like interruption, end-of-utterance, the context window, DTMF, and wake-up. Register pipeline-event hooks with the @pipeline.on(...) decorator.

Constructor

Pipeline(stt: 'Any' = None, llm: 'Any' = None, tts: 'Any' = None, vad: 'Any' = None, turn_detector: 'Any' = None, avatar: 'Any' = None, denoise: 'Any' = None, eou_config: 'EOUConfig | None' = None, interrupt_config: 'InterruptConfig | None' = None, conversational_graph: 'Any' = None, context_window: 'ContextWindow | Any' = None, voice_mail_detector: 'Any' = None, dtmf_handler: 'Any' = None, wake_up: 'int | None' = None, wake_up_max_attempts: 'int | None' = None, realtime_config: 'RealtimeConfig | None' = None, llm_stream_hook_enabled: 'bool' = False, llm_stream_hook_timeout_ms: 'int' = 100, playback_grace_ms: 'int' = 0, stt_filter_patterns: 'list[str] | None' = None, stt_word_substitutions: 'dict[str, str] | None' = None, inactivity_timeout_seconds: 'int | None' = None) -> 'None'

stt

Any

Speech-to-text component (cascade mode).

llm

Any

Language model. A realtime/speech-to-speech model enables realtime mode.

tts

Any

Text-to-speech component (cascade mode).

vad

Any

Voice-activity detector (e.g. Silero).

turn_detector

Any

End-of-turn detector deciding when the caller has finished.

avatar

Any

Optional visual avatar component.

denoise

Any

Optional input noise-cancellation component.

eou_config

EOUConfig | None

End-of-utterance tuning (EOUConfig).

interrupt_config

InterruptConfig | None

Barge-in / interruption tuning (InterruptConfig).

conversational_graph

Any

Optional structured conversation flow.

context_window

ContextWindow | Any

Conversation-history budget (ContextWindow).

voice_mail_detector

Any

Optional voicemail detector for outbound calls.

dtmf_handler

Any

Handler for touch-tone (DTMF) keypad input.

wake_up

int | None

Seconds of caller silence before a wake-up prompt fires.

wake_up_max_attempts

int | None

Maximum wake-up prompts before giving up.

realtime_config

RealtimeConfig | None

Speech-to-speech configuration (RealtimeConfig).

llm_stream_hook_enabled

bool

default:"False"

Enable the streaming @pipeline.on("llm") hook.

llm_stream_hook_timeout_ms

int

default:"100"

Per-chunk timeout for the LLM stream hook. Default 100.

playback_grace_ms

int

default:"0"

Grace period before treating playback as finished. Default 0.

stt_filter_patterns

list[str] | None

Regexes whose matching transcript fragments are dropped.

stt_word_substitutions

dict[str, str] | None

Literal word replacements applied to transcripts.

inactivity_timeout_seconds

int | None

End the session after this much silence.

change_component

def change_component(self, *, stt: 'Any' = NO_CHANGE, llm: 'Any' = NO_CHANGE, tts: 'Any' = NO_CHANGE, vad: 'Any' = NO_CHANGE, turn_detector: 'Any' = NO_CHANGE, denoise: 'Any' = NO_CHANGE) -> 'None'

Hot-swap pipeline components mid-session, without tearing the call down.

stt

Any

default:"NO_CHANGE"

llm

Any

default:"NO_CHANGE"

tts

Any

default:"NO_CHANGE"

vad

Any

default:"NO_CHANGE"

turn_detector

Any

default:"NO_CHANGE"

denoise

Any

default:"NO_CHANGE"

change_pipeline

def change_pipeline(self, *, stt: 'Any' = None, llm: 'Any' = None, tts: 'Any' = None, vad: 'Any' = None, turn_detector: 'Any' = None, interrupt_config: 'Any' = None, **unsupported: 'Any') -> 'None'

Reconfigure several pipeline pieces mid-session in one call.

stt

Any

llm

Any

tts

Any

vad

Any

turn_detector

Any

interrupt_config

Any

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

Emit an event, invoking all registered handlers with the given arguments. Handlers are called in registration order. Coroutine handlers are scheduled on the running event loop. If the emitter is closed, the call is a no-op.

event

required

The event to emit.

get_latest_frames

def get_latest_frames(self, num_frames: 'int' = 1) -> 'list'

Return the most recent buffered vision frames.

num_frames

int

default:"1"

Number of trailing frames to return. Default 1.

returns

list

Up to num_frames most recent frames in chronological order, or an empty list when num_frames is non-positive.

get_session_metrics_snapshot

def get_session_metrics_snapshot(self) -> 'dict'

Return a snapshot of session metrics.

returns

dict

An empty dictionary. Metrics collection is reported through the PipelineMetricsHooks hooks rather than this snapshot.

interrupt

def interrupt(self) -> 'None'

Interrupt the agent’s current speech. No-op when no agent or active session is attached.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'str', callback: 'Callable | None' = None) -> 'Callable'

Register a pipeline-event hook. Usable either as a decorator (@pipeline.on("event")) or by passing the callback directly.

event

str

required

Name of the pipeline event to subscribe to.

callback

Callable | None

Callback to register. If omitted, a decorator is returned.

returns

Callable

The registered callback when callback is provided, otherwise a decorator that registers the callable it wraps.

process_text

def process_text(self, text: 'str') -> 'None'

Feed text into the session as if it were user input. No-op when no agent or active session is attached.

text

str

required

The text to process through the session.

push_audio_frame

def push_audio_frame(self, frame: 'dict') -> 'None'

Notify audio_delta hooks with an incoming audio frame.

frame

dict

required

The audio frame to dispatch to registered hooks.

push_vision_frame

def push_vision_frame(self, frame: 'dict') -> 'None'

Buffer a vision frame and notify vision_frame hooks.

frame

dict

required

The vision frame to append to the buffer and dispatch.

reply_with_context

def reply_with_context(self, instructions: 'str', **kwargs) -> 'None'

Generate an agent reply guided by the given instructions. No-op when no agent or active session is attached.

instructions

str

required

Instructions steering the generated reply.

send_message

def send_message(self, text: 'str', **kwargs) -> 'None'

Have the agent speak the given text. No-op when no agent or active session is attached.

text

str

required

The text for the agent to say.

set_agent

def set_agent(self, agent: 'Any') -> 'None'

Attach the agent that owns this pipeline.

agent

Any

required

The agent instance to associate with the pipeline.

PipelineHooks

Registry of callbacks bound to pipeline events. Tracks single-slot stream hooks (stt, tts, llm_stream) and multi-subscriber event hooks. Callbacks are registered through the on decorator and retrieved by the pipeline at the appropriate points in the conversation lifecycle.

Constructor

PipelineHooks() -> 'None'

clear_all_hooks

def clear_all_hooks(self) -> 'None'

Remove every registered hook, resetting the registry to empty.

get_hooks

def get_hooks(self, event: 'str') -> 'list[Callable]'

Return the callbacks registered for a multi-subscriber event.

event

str

required

Name of the event to look up.

returns

list[Callable]

The list of registered callbacks, or an empty list if none.

has_llm_observation_hooks

def has_llm_observation_hooks(self) -> 'bool'

Return True if any non-streaming llm observation hooks exist.

returns

bool

has_llm_stream_hook

def has_llm_stream_hook(self) -> 'bool'

Return True if an LLM stream hook is registered.

returns

bool

has_stt_stream_hook

def has_stt_stream_hook(self) -> 'bool'

Return True if a speech-to-text stream hook is registered.

returns

bool

has_tts_stream_hook

def has_tts_stream_hook(self) -> 'bool'

Return True if a text-to-speech stream hook is registered.

returns

bool

on

def on(self, event: 'str') -> 'Callable'

Create a decorator that registers a callback for an event.

event

str

required

Name of the pipeline event to subscribe to.

returns

Callable

A decorator that registers the wrapped callable and returns it unchanged.

registered_names

def registered_names(self) -> 'list[str]'

List the names of all events that currently have a hook registered.

returns

list[str]

Event names with at least one registered callback, including the single-slot stt, tts, and llm_stream hooks when set.

PipelineHookError

Raised when a requested @pipeline.on(...) hook is not supported.

Constructor

PipelineHookError(hook_name: 'str', reason: 'str', alternative: 'Optional[str]' = None) -> 'None'

hook_name

str

required

Name of the unsupported hook.

reason

str

required

Explanation of why the hook is unsupported.

alternative

Optional[str]

Optional suggested alternative, appended to the message.

PipelineComponent

Individual components that can participate in the processing pipeline.

class PipelineComponent(Enum):
    STT = 'stt'
    LLM = 'llm'
    TTS = 'tts'
    VAD = 'vad'
    TURN_DETECTOR = 'turn_detector'
    AVATAR = 'avatar'
    DENOISE = 'denoise'
    REALTIME_MODEL = 'realtime_model'

PipelineConfig

Immutable description of a resolved pipeline configuration.

Fields

pipeline_mode

PipelineMode

required

The high-level operating mode of the pipeline.

realtime_mode

RealtimeMode | None

required

The realtime model mode, or None if not in a realtime mode.

is_realtime

bool

required

Whether the pipeline operates in a realtime mode.

active_components

frozenset[PipelineComponent]

required

The set of components active in this pipeline.

has_component

def has_component(self, component: 'PipelineComponent') -> 'bool'

Check whether a component is active in this pipeline.

component

PipelineComponent

required

The pipeline component to look for.

returns

bool

True if the component is part of the active components.

PipelineMode

Operating modes of the processing pipeline, defining which components are active.

class PipelineMode(Enum):
    REALTIME = 'realtime'
    FULL_CASCADING = 'full_cascading'
    LLM_TTS_ONLY = 'llm_tts_only'
    STT_LLM_ONLY = 'stt_llm_only'
    LLM_ONLY = 'llm_only'
    STT_ONLY = 'stt_only'
    TTS_ONLY = 'tts_only'
    STT_TTS_ONLY = 'stt_tts_only'
    HYBRID = 'hybrid'
    PARTIAL_CASCADING = 'partial_cascading'

RealtimeMode

Modes for a realtime model, ranging from full speech-to-speech to hybrid configurations.

class RealtimeMode(Enum):
    FULL_S2S = 'full_s2s'
    HYBRID_STT = 'hybrid_stt'
    HYBRID_TTS = 'hybrid_tts'
    LLM_ONLY = 'llm_only'

STT

Abstract base class for speech-to-text providers. Implementations convert audio input into transcription events and may emit an 'error' event when a failure occurs.

Constructor

STT() -> 'None'

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

on_stt_transcript

def on_stt_transcript(self, callback: 'Callable[[STTResponse], Awaitable[None]]') -> 'None'

callback

Callable[[STTResponse], Awaitable[None]]

required

An async callable that receives each STTResponse.

process_audio

def process_audio(self, audio_frames: 'bytes', language: 'Optional[str]' = None, **kwargs: 'Any') -> 'None'

Process a chunk of audio for transcription. Implementations should consume the audio frames and emit transcription results through the registered transcript callback.

audio_frames

bytes

required

Raw audio bytes to transcribe.

language

Optional[str]

Optional language code to guide recognition.

stream_transcribe

def stream_transcribe(self, audio_stream: 'AsyncIterator[bytes]', **kwargs: 'Any') -> 'AsyncIterator[STTResponse]'

Transcribe a continuous audio stream.

audio_stream

AsyncIterator[bytes]

required

An async iterator yielding raw audio byte chunks.

returns

AsyncIterator[STTResponse]

STTResponse: Transcription events produced as audio is consumed.

STTResponse

Event emitted by a speech-to-text provider.

Fields

event_type

SpeechEventType

required

The type of speech event this response represents.

data

SpeechData

required

The transcription payload associated with the event.

metadata

Optional[dict[str, Any]]

Optional provider-specific metadata.

SpeechData

Transcription payload produced by a speech-to-text provider.

Fields

text

str

required

The recognized text.

confidence

float

default:"0.0"

Recognition confidence score in the range 0.0 to 1.0.

language

Optional[str]

Detected or configured language code, if available.

start_time

float

default:"0.0"

Start time of the recognized segment, in seconds.

end_time

float

default:"0.0"

End time of the recognized segment, in seconds.

duration

float

default:"0.0"

Duration of the recognized segment, in seconds.

LLM

Abstract base class for language model providers. Implementations generate streamed chat responses and may emit an 'error' event when a failure occurs.

Constructor

LLM() -> 'None'

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

cancel_current_generation

def cancel_current_generation(self) -> 'None'

Cancel the in-progress generation for the active session, if any.

chat

def chat(self, messages: 'Any', tools: 'Any' = None, conversational_graph: 'Any' = None, **kwargs: 'Any') -> 'AsyncIterator[LLMResponse]'

Generate a streamed chat completion.

messages

Any

required

The conversation history to send to the model.

tools

Any

Optional tool definitions the model may call.

conversational_graph

Any

Optional conversational graph guiding the exchange.

returns

AsyncIterator[LLMResponse]

LLMResponse: Response chunks produced as the model generates output.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

LLMResponse

A single response chunk produced by a language model provider.

Fields

content

str

required

The text content of the response.

role

ChatRole

required

The chat role associated with the content.

metadata

Optional[dict[str, Any]]

Optional provider-specific metadata.

TTS

Abstract base class for text-to-speech providers. Implementations synthesize text into audio output and may emit an 'error' event when a failure occurs.

Constructor

TTS(sample_rate: 'int' = 24000, num_channels: 'int' = 1) -> 'None'

sample_rate

int

default:"24000"

Sample rate of the synthesized audio, in Hz.

num_channels

int

default:"1"

Number of audio channels in the synthesized output.

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

interrupt

def interrupt(self) -> 'None'

Interrupt and cancel the current synthesis for the active session, if any.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

on_first_audio_byte

def on_first_audio_byte(self, callback: "'Callable[[int, int], Awaitable[None] | None]'") -> 'None'

callback

'Callable[[int, int], Awaitable[None] | None]'

required

A callable, optionally async, invoked with synthesis timing information when the first audio byte is emitted.

synthesize

def synthesize(self, text: 'Any', **kwargs: 'Any') -> 'None'

Synthesize the given text into speech. Implementations should convert the text to audio. The default behavior forwards the text to the active session for playback when one is present.

text

Any

required

The text to synthesize; coerced to a string if needed.

VAD

Abstract base class for voice activity detection providers. Implementations analyze audio to detect speech and silence boundaries and may emit 'error' and 'info' events.

Constructor

VAD(sample_rate: 'int' = 16000, threshold: 'float' = 0.5, min_speech_duration: 'float' = 0.5, min_silence_duration: 'float' = 0.5) -> 'None'

sample_rate

int

default:"16000"

Sample rate of the input audio, in Hz.

threshold

float

default:"0.5"

Speech probability threshold above which audio is treated as speech.

min_speech_duration

float

default:"0.5"

Minimum duration, in seconds, required to confirm a speech segment.

min_silence_duration

float

default:"0.5"

Minimum duration, in seconds, required to confirm a silence segment.

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

flush

def flush(self) -> 'None'

Flush any buffered audio and finalize pending detection state.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

on_vad_event

def on_vad_event(self, callback: 'Callable[[VADResponse], Awaitable[None]]') -> 'None'

callback

Callable[[VADResponse], Awaitable[None]]

required

An async callable that receives each VADResponse.

process_audio

def process_audio(self, audio_frames: 'bytes', **kwargs: 'Any') -> 'None'

Analyze a chunk of audio for voice activity. Implementations should detect speech and silence transitions and emit results through the registered voice activity callback.

audio_frames

bytes

required

Raw audio bytes to analyze.

VADResponse

Event emitted by a voice activity detection provider.

Fields

event_type

VADEventType

required

The type of voice activity event this response represents.

data

VADData

required

The voice activity payload associated with the event.

metadata

Optional[dict[str, Any]]

Optional provider-specific metadata.

VADData

Voice activity detection payload describing the current speech state.

Fields

is_speech

bool

required

Whether speech is currently detected.

confidence

float

default:"0.0"

Detection confidence score in the range 0.0 to 1.0.

timestamp

float

default:"0.0"

Time of the detection, in seconds.

speech_duration

float

default:"0.0"

Accumulated duration of detected speech, in seconds.

silence_duration

float

default:"0.0"

Accumulated duration of detected silence, in seconds.

EOU

Abstract base class for end-of-utterance detection providers. Implementations estimate the likelihood that a speaker has finished their turn and may emit an 'error' event when a failure occurs.

Constructor

EOU(threshold: 'float' = 0.7) -> 'None'

threshold

float

default:"0.7"

Probability threshold above which an utterance is considered complete.

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

cleanup

def cleanup(self) -> 'None'

Perform any additional teardown required by the provider.

detect_end_of_utterance

def detect_end_of_utterance(self, chat_context: 'Any', threshold: 'Optional[float]' = None) -> 'bool'

Determine whether the current utterance has ended.

chat_context

Any

required

The conversation context used to evaluate the utterance.

threshold

Optional[float]

Optional probability threshold overriding the configured one.

returns

bool

True if the end-of-utterance probability meets or exceeds the threshold, otherwise False.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

get_eou_probability

def get_eou_probability(self, chat_context: 'Any') -> 'float'

Return the estimated probability that the current utterance has ended. Reads the most recent detection result from the active session, falling back to 0.5 when no result is available or the value cannot be parsed.

chat_context

Any

required

The conversation context used to evaluate the utterance.

returns

float

The end-of-utterance probability in the range 0.0 to 1.0.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

set_threshold

def set_threshold(self, threshold: 'float') -> 'None'

Update the end-of-utterance probability threshold.

threshold

float

required

The new probability threshold.

RealtimeBaseModel

Base class for realtime models supporting asynchronous lifecycle management. Provides an asynchronous context manager interface and a no-op cleanup contract that subclasses may override to release resources.

Constructor

RealtimeBaseModel()

aclose

def aclose(self) -> None

Release resources held by the model. The base implementation performs no work. Subclasses should override this method to close connections or free resources.

returns

None.

cleanup

def cleanup(self) -> None

Clean up the model by delegating to aclose.

returns

None.

RealtimeLLMAdapter

def RealtimeLLMAdapter(realtime_model, agent=None)

Mark a realtime model so the pipeline treats it as a realtime LLM. Tags the given model with the internal flag the pipeline checks to route it through the realtime (speech-to-speech) path, then returns the same object.

realtime_model

required

The realtime model instance to adapt.

agent

Optional agent the model is associated with. Currently unused.

returns

The same realtime_model instance, tagged as a realtime model.

FallbackSTT

Speech-to-text provider that falls back across an ordered list of providers. The first provider is used as the primary. When a provider fails, the fallback machinery temporarily disables it and routes to the next available provider, optionally disabling it permanently after repeated failures.

Constructor

FallbackSTT(providers: 'List[STT]', *, temporary_disable_sec: 'int' = 60, permanent_disable_after_attempts: 'int' = 3, recovery_check_interval_sec: 'int' = 300, **kwargs: 'Any') -> 'None'

providers

List[STT]

required

Ordered list of STT providers; the first is the primary.

temporary_disable_sec

int

default:"60"

Duration in seconds a failing provider is temporarily disabled before being retried.

permanent_disable_after_attempts

int

default:"3"

Number of failed attempts after which a provider is permanently disabled.

recovery_check_interval_sec

int

default:"300"

Interval in seconds between checks for recovery of a disabled provider.

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

on_stt_transcript

def on_stt_transcript(self, callback: 'Callable[[STTResponse], Awaitable[None]]') -> 'None'

callback

Callable[[STTResponse], Awaitable[None]]

required

An async callable that receives each STTResponse.

process_audio

def process_audio(self, audio_frames: 'bytes', language: 'Optional[str]' = None, **kwargs: 'Any') -> 'None'

Process a chunk of audio for transcription. Implementations should consume the audio frames and emit transcription results through the registered transcript callback.

audio_frames

bytes

required

Raw audio bytes to transcribe.

language

Optional[str]

Optional language code to guide recognition.

stream_transcribe

def stream_transcribe(self, audio_stream: 'AsyncIterator[bytes]', **kwargs: 'Any') -> 'AsyncIterator[STTResponse]'

Transcribe a continuous audio stream.

audio_stream

AsyncIterator[bytes]

required

An async iterator yielding raw audio byte chunks.

returns

AsyncIterator[STTResponse]

STTResponse: Transcription events produced as audio is consumed.

FallbackLLM

Language model provider that falls back across an ordered list of providers. The first provider is used as the primary. When a provider fails, the fallback machinery temporarily disables it and routes to the next available provider, optionally disabling it permanently after repeated failures.

Constructor

FallbackLLM(providers: 'List[LLM]', *, temporary_disable_sec: 'int' = 60, permanent_disable_after_attempts: 'int' = 3, recovery_check_interval_sec: 'int' = 300, **kwargs: 'Any') -> 'None'

providers

List[LLM]

required

Ordered list of LLM providers; the first is the primary.

temporary_disable_sec

int

default:"60"

Duration in seconds a failing provider is temporarily disabled before being retried.

permanent_disable_after_attempts

int

default:"3"

Number of failed attempts after which a provider is permanently disabled.

recovery_check_interval_sec

int

default:"300"

Interval in seconds between checks for recovery of a disabled provider.

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

cancel_current_generation

def cancel_current_generation(self) -> 'None'

Cancel the in-progress generation for the active session, if any.

chat

def chat(self, messages: 'Any', tools: 'Any' = None, conversational_graph: 'Any' = None, **kwargs: 'Any') -> 'AsyncIterator[LLMResponse]'

Generate a streamed chat completion.

messages

Any

required

The conversation history to send to the model.

tools

Any

Optional tool definitions the model may call.

conversational_graph

Any

Optional conversational graph guiding the exchange.

returns

AsyncIterator[LLMResponse]

LLMResponse: Response chunks produced as the model generates output.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

FallbackTTS

Text-to-speech provider that falls back across an ordered list of providers. The first provider is used as the primary. When a provider fails, the fallback machinery temporarily disables it and routes to the next available provider, optionally disabling it permanently after repeated failures.

Constructor

FallbackTTS(providers: 'List[TTS]', *, temporary_disable_sec: 'int' = 60, permanent_disable_after_attempts: 'int' = 3, recovery_check_interval_sec: 'int' = 300, **kwargs: 'Any') -> 'None'

providers

List[TTS]

required

Ordered list of TTS providers; the first is the primary.

temporary_disable_sec

int

default:"60"

Duration in seconds a failing provider is temporarily disabled before being retried.

permanent_disable_after_attempts

int

default:"3"

Number of failed attempts after which a provider is permanently disabled.

recovery_check_interval_sec

int

default:"300"

Interval in seconds between checks for recovery of a disabled provider.

aclose

def aclose(self) -> 'None'

Release any resources held by the provider.

emit

def emit(self, event: 'T', *args: 'Any') -> 'None'

event

required

The event to emit.

interrupt

def interrupt(self) -> 'None'

Interrupt and cancel the current synthesis for the active session, if any.

off

def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.

event

required

The event the handler was registered for.

callback

Callable[..., Any]

required

The handler to remove.

on

def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.

event

required

The event to listen for.

callback

Callable[..., Any] | None

The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.

returns

Callable[..., Any]

The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

on_first_audio_byte

def on_first_audio_byte(self, callback: "'Callable[[int, int], Awaitable[None] | None]'") -> 'None'

callback

'Callable[[int, int], Awaitable[None] | None]'

required

A callable, optionally async, invoked with synthesis timing information when the first audio byte is emitted.

synthesize

def synthesize(self, text: 'Any', **kwargs: 'Any') -> 'None'

Synthesize the given text into speech. Implementations should convert the text to audio. The default behavior forwards the text to the active session for playback when one is present.

text

Any

required

The text to synthesize; coerced to a string if needed.

UtteranceHandle

Tracks the lifecycle of a single spoken utterance. Provides identity and interruption state for an utterance, lets callers register callbacks for when the final audio byte is emitted, and can be awaited to block until the utterance completes or is interrupted.

Constructor

UtteranceHandle(utterance_id: 'str' = None, interruptible: 'bool' = True)

utterance_id

str

Identifier for the utterance. A random UUID string is generated when not provided.

interruptible

bool

default:"True"

Whether the utterance may be interrupted without forcing. Defaults to True.

done

def done(self) -> 'bool'

Report whether the utterance has finished or been interrupted.

returns

bool

True if the utterance has completed or been interrupted, otherwise False.

interrupt

def interrupt(self, *, force: 'bool' = False) -> 'None'

Interrupt the utterance, marking it done and firing callbacks. When the utterance is interruptible, or when force is set, the utterance is marked as interrupted and completed and any registered last-audio-byte callbacks are invoked. Otherwise the call is a no-op.

force

bool

default:"False"

Interrupt even when the utterance is not interruptible. Defaults to False.

on_last_audio_byte

def on_last_audio_byte(self, callback: 'Callable[[], Any]') -> 'Callable[[], Any]'

Register a callback for when the final audio byte is emitted. If the utterance is already done, the callback is invoked immediately; otherwise it is queued and invoked when the utterance completes or is interrupted. The callback may be a regular function or a coroutine function.

callback

Callable[[], Any]

required

Zero-argument callable invoked when the last audio byte is reached.

returns

Callable[[], Any]

The same callback, allowing use as a decorator.

​Pipeline

​Constructor

​change_component

​change_pipeline

​emit

​get_latest_frames

​get_session_metrics_snapshot

​interrupt

​off

​on

​process_text

​push_audio_frame

​push_vision_frame

​reply_with_context

​send_message

​set_agent

​PipelineHooks

​Constructor

​clear_all_hooks

​get_hooks

​has_llm_observation_hooks

​has_llm_stream_hook

​has_stt_stream_hook

​has_tts_stream_hook

​on

​registered_names

​PipelineHookError

​Constructor

​PipelineComponent

​PipelineConfig

​Fields

​has_component

​PipelineMode

​RealtimeMode

​STT

​Constructor

​aclose

​emit

​off

​on

​on_stt_transcript

​process_audio

​stream_transcribe

​STTResponse

​Fields

​SpeechData

​Fields

​LLM

​Constructor

​aclose

​cancel_current_generation

​chat

​emit

​off

​on

​LLMResponse

​Fields

​TTS

​Constructor

​aclose

​emit

​interrupt

​off

​on

​on_first_audio_byte

​synthesize

​VAD

​Constructor

​aclose

​emit

​flush

​off

​on

​on_vad_event

​process_audio

​VADResponse

​Fields

​VADData

​Fields

​EOU

Pipeline

Constructor

change_component

change_pipeline

emit

get_latest_frames

get_session_metrics_snapshot

interrupt

off

on

process_text

push_audio_frame

push_vision_frame

reply_with_context

send_message

set_agent

PipelineHooks

Constructor

clear_all_hooks

get_hooks

has_llm_observation_hooks

has_llm_stream_hook

has_stt_stream_hook

has_tts_stream_hook

on

registered_names

PipelineHookError

Constructor

PipelineComponent

PipelineConfig

Fields

has_component

PipelineMode

RealtimeMode

STT

Constructor

aclose

emit

off

on

on_stt_transcript

process_audio

stream_transcribe

STTResponse

Fields

SpeechData

Fields

LLM

Constructor

aclose

cancel_current_generation

chat

emit

off

on

LLMResponse

Fields

TTS

Constructor

aclose

emit

interrupt

off

on

on_first_audio_byte

synthesize

VAD

Constructor

aclose

emit

flush

off

on

on_vad_event

process_audio

VADResponse

Fields

VADData

Fields

EOU