Ultravox

Ultravox is a speech-to-speech model. It handles transcription, reasoning, and voice synthesis end-to-end in a single model, so it goes in the pipeline’s llm slot with no separate STT or TTS.

Setup

Set your Ultravox API key in the worker environment. Generate a key from the Ultravox dashboard:

export ULTRAVOX_API_KEY=<key>

Usage

Pass the realtime model to the pipeline’s llm slot, no stt or tts needed.

from zrt import Pipeline
from zrt.plugins import UltravoxRealtime, UltravoxLiveConfig

model = UltravoxRealtime(
    config=UltravoxLiveConfig(
        model="fixie-ai/ultravox",
        voice="54ebeae1-88df-4d66-af13-6c41283b4332",
        language_hint="en",
    ),
)

pipeline = Pipeline(llm=model)

Parameters

Constructor parameters for the Python SDK (UltravoxRealtime). Model behavior is configured through the config object (UltravoxLiveConfig).

Constructor

Parameter	Type	Default	Description
`api_key`	`str`	`None`	Ultravox API key. Falls back to the `ULTRAVOX_API_KEY` environment variable when unset.
`config`	`UltravoxLiveConfig`	`None`	Model behavior configuration (see below). When omitted, a default `UltravoxLiveConfig` is used.

`UltravoxLiveConfig`

Field	Type	Default	Description
`model`	`str`	`"fixie-ai/ultravox"`	Ultravox model to use.
`voice`	`str`	`None`	Voice ID for the synthesized speech.
`language_hint`	`str`	`"en"`	Hint for the conversation’s language.
`temperature`	`float`	`None`	Controls the randomness of responses (0.0 to 1.0).
`max_duration`	`str`	`None`	Maximum duration of the call (e.g. `"600s"`).
`time_exceeded_message`	`str`	`None`	Message spoken when the maximum duration is exceeded.
`input_sample_rate`	`int`	`48000`	Sample rate (Hz) of the input audio.
`output_sample_rate`	`int`	`24000`	Sample rate (Hz) of the synthesized output audio.
`client_buffer_size_ms`	`int`	`30000`	Client-side audio buffer size in milliseconds.
`vad_turn_endpoint_delay_ms`	`int`	`800`	Milliseconds of silence before voice activity detection ends a turn.
`vad_minimum_turn_duration_ms`	`int`	`600`	Minimum duration in milliseconds for a valid speech turn.
`vad_minimum_interruption_duration_ms`	`int`	`None`	Minimum duration in milliseconds of speech required to interrupt the agent.
`vad_frame_activation_threshold`	`float`	`0.4`	Frame activation threshold for voice activity detection.
`first_speaker`	`str`	`"FIRST_SPEAKER_USER"`	Determines who speaks first.
`enable_greeting_prompt`	`bool`	`False`	Whether to enable an initial greeting prompt.
`base_url`	`str`	`None`	Override the Ultravox streaming endpoint.

To pair the realtime model with an external STT or TTS, see Hybrid mode.

Plugins

Inference Gateway

Speech-to-text

LLM

Text-to-speech

Realtime models

Avatars

Turn detection

Voice activity detection

Noise cancellation

Setup

Usage

Parameters

Constructor

`UltravoxLiveConfig`

​Setup

​Usage

​Parameters

​Constructor

​UltravoxLiveConfig

Setup

Usage

Parameters

Constructor

`UltravoxLiveConfig`