OpenAI Realtime

OpenAI Realtime is a speech-to-speech model. It handles transcription, reasoning, and voice synthesis end-to-end in a single model, so it goes in the pipeline’s llm slot with no separate STT or TTS. The Pipeline auto-detects Realtime mode when you pass it.

Setup

Set your OpenAI API key in the worker environment. Generate a key from the OpenAI dashboard:

export OPENAI_API_KEY=<key>

Usage

Pass the realtime model to the pipeline’s llm slot, no stt or tts needed.

from zrt import Pipeline
from zrt.plugins import OpenAIRealtime, OpenAIRealtimeConfig

model = OpenAIRealtime(
    config=OpenAIRealtimeConfig(
        model="gpt-4o-realtime-preview",
        voice="alloy",
        modalities=["text", "audio"],
    ),
)

pipeline = Pipeline(llm=model)

Configuration

OpenAIRealtime constructor and OpenAIRealtimeConfig fields for the Python SDK. The JavaScript SDK passes the same settings via the flat config object; the Go SDK via RealtimeOptions.

Constructor

Parameter	Type	Default	Description
`api_key`	`str`	`None`	OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable when unset.
`config`	`OpenAIRealtimeConfig`	`None`	Model behavior configuration (see below).

`OpenAIRealtimeConfig`

Field	Type	Default	Description
`model`	`str`	`"gpt-4o-realtime-preview"`	Realtime model ID.
`voice`	`str`	`"alloy"`	Output voice (e.g. `alloy`, `ash`, `marin`, `cedar`, `verse`).
`temperature`	`float`	`0.8`	Sampling randomness.
`turn_detection`	`TurnDetectionConfig`	server VAD	Turn detection (`server_vad`, threshold `0.5`, `prefix_padding_ms` `300`, `silence_duration_ms` `200`). Pass `None` to disable.
`input_audio_transcription`	`InputAudioTranscriptionConfig`	`gpt-4o-mini-transcribe`	Transcription model for the user’s audio.
`tool_choice`	`str`	`"auto"`	How the model decides when to call tools.
`modalities`	`list[str]`	`["text", "audio"]`	Enabled response modalities. Drop `"audio"` for text-only.

Import paths

SDK	Import	Constructor
Python	`from zrt.plugins import OpenAIRealtime, OpenAIRealtimeConfig`	`OpenAIRealtime(config=...)`
JavaScript	`from '@zrt/js-sdk/plugins/openai_realtime'`	`OpenAIRealtime({ config: { ... } })`
Go	`.../zrt-golang-sdk/plugins/openai_realtime`	`openai_realtime.NewRealtime(openai_realtime.RealtimeOptions{...})`

To pair the realtime model with an external STT or TTS, see Hybrid mode.

Plugins

Inference Gateway

Speech-to-text

LLM

Text-to-speech

Realtime models

Avatars

Turn detection

Voice activity detection

Noise cancellation

Setup

Usage

Configuration

Constructor

`OpenAIRealtimeConfig`

Import paths

​Setup

​Usage

​Configuration

​Constructor

​OpenAIRealtimeConfig

​Import paths

Setup

Usage

Configuration

Constructor

`OpenAIRealtimeConfig`

Import paths