Ultravox

Ultravox is a speech-to-speech model. It handles transcription, reasoning, and voice synthesis end-to-end in a single model, so it goes in the pipeline’s llm slot with no separate STT or TTS.

Setup

Set your Ultravox API key in the worker environment. Generate a key from the Ultravox dashboard:

export ULTRAVOX_API_KEY=<key>

Usage

Pass the realtime model to the pipeline’s llm slot, no stt or tts needed.

from zeroruntime import Pipeline
from zeroruntime.plugins import UltravoxRealtime

model = UltravoxRealtime(
    model="fixie-ai/ultravox",
    config={
        "voice": "54ebeae1-88df-4d66-af13-6c41283b4332",
        "language_hint": "en",
    },
)

pipeline = Pipeline(llm=model)

import { Pipeline } from '@zeroruntime/js-sdk';
import { UltravoxRealtime } from '@zeroruntime/js-sdk/plugins';

const model = UltravoxRealtime({
  model: 'fixie-ai/ultravox',
  config: {
    voice: '54ebeae1-88df-4d66-af13-6c41283b4332',
    language_hint: 'en',
  },
});

const pipeline = Pipeline({ llm: model });

Parameters

Constructor parameters for UltravoxRealtime. The Python and Node JS SDKs share these field names. Model behavior is configured through the config mapping.

Constructor

Parameter	Type	Default	Description
`model`	`str`	`"fixie-ai/ultravox"`	Ultravox model to use.
`api_key`	`str`	`None`	Ultravox API key. Falls back to the `ULTRAVOX_API_KEY` environment variable when unset.
`config`	`dict`	`None`	Model behavior configuration (see below). When omitted, provider defaults are used.

`config` keys

Field	Type	Default	Description
`voice`	`str`	`None`	Voice ID for the synthesized speech.
`language_hint`	`str`	`"en"`	Hint for the conversation’s language.
`temperature`	`float`	`None`	Controls the randomness of responses (0.0 to 1.0).
`max_duration`	`str`	`None`	Maximum duration of the call (e.g. `"600s"`).
`time_exceeded_message`	`str`	`None`	Message spoken when the maximum duration is exceeded.
`input_sample_rate`	`int`	`48000`	Sample rate (Hz) of the input audio.
`output_sample_rate`	`int`	`24000`	Sample rate (Hz) of the synthesized output audio.
`client_buffer_size_ms`	`int`	`30000`	Client-side audio buffer size in milliseconds.
`vad_turn_endpoint_delay_ms`	`int`	`800`	Milliseconds of silence before voice activity detection ends a turn.
`vad_minimum_turn_duration_ms`	`int`	`600`	Minimum duration in milliseconds for a valid speech turn.
`vad_minimum_interruption_duration_ms`	`int`	`None`	Minimum duration in milliseconds of speech required to interrupt the agent.
`vad_frame_activation_threshold`	`float`	`0.4`	Frame activation threshold for voice activity detection.
`first_speaker`	`str`	`"FIRST_SPEAKER_USER"`	Determines who speaks first.
`enable_greeting_prompt`	`bool`	`False`	Whether to enable an initial greeting prompt.
`base_url`	`str`	`None`	Override the Ultravox streaming endpoint.

To pair the realtime model with an external STT or TTS, see Hybrid mode.

Plugins

Inference Gateway

Speech-to-text

LLM

Text-to-speech

Realtime models

Avatars

Turn detection

Voice activity detection

Noise cancellation

Setup

Usage

Parameters

Constructor

`config` keys

​Setup

​Usage

​Parameters

​Constructor

​config keys

Setup

Usage

Parameters

Constructor

`config` keys