OpenAI

OpenAI is a speech-to-text plugin. It transcribes the caller’s audio into text for the LLM using OpenAI’s transcription models. By default it streams over OpenAI’s realtime transcription WebSocket; it can also run in non-streaming HTTP mode with built-in voice activity detection.

Setup

Set your OpenAI API key in the worker environment. Generate a key from the OpenAI dashboard:

export OPENAI_API_KEY=<key>

Usage

Import the plugin and pass it to the pipeline’s stt slot.

from zrt.plugins import OpenAISTT

stt = OpenAISTT(
    model="gpt-4o-transcribe",
    language="en",
)

# Pipeline(stt=stt, ...)

Azure OpenAI

To run OpenAI transcription through Azure OpenAI, use the separate AzureOpenAISTT plugin. It reads its configuration from Azure environment variables when arguments are omitted:

export AZURE_OPENAI_API_KEY=<key>
export AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com
export AZURE_OPENAI_STT_DEPLOYMENT=<deployment-name>

Python

from zrt.plugins import AzureOpenAISTT

stt = AzureOpenAISTT(
    azure_endpoint="https://<resource>.openai.azure.com",
    deployment="<deployment-name>",
    api_version="2025-03-01-preview",
)

# Pipeline(stt=stt, ...)

api_key falls back to AZURE_API_KEY, then AZURE_OPENAI_API_KEY. deployment falls back to AZURE_OPENAI_STT_DEPLOYMENT. AzureOpenAISTT runs non-streaming by default (stream=False).

Parameters

Constructor parameters for the Python SDK (OpenAISTT).

Parameter	Type	Default	Description
`api_key`	`str`	`None`	OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable when unset.
`model`	`str`	`"gpt-4o-transcribe"`	Transcription model. `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, `gpt-4o-transcribe-diarize`, or `whisper-1`.
`language`	`str`	`"en"`	Expected language of the caller’s speech (ISO-639-1).
`stream`	`bool`	`True`	Stream partial transcripts as the caller speaks.
`input_sample_rate`	`int`	`48000`	Input audio sample rate in Hz.
`output_sample_rate`	`int`	`24000`	Resampled rate sent to the model in Hz.
`prompt`	`str`	`None`	Biasing prompt to guide transcription vocabulary/style.
`turn_detection`	`str`	`"server_vad"`	VAD mode: `"server_vad"` or `"semantic_vad"`.
`vad_threshold`	`float`	`None`	Speech-detection sensitivity, 0.0-1.0.
`vad_prefix_padding_ms`	`int`	`None`	Audio kept before detected speech, in ms.
`vad_silence_duration_ms`	`int`	`None`	Silence before end-of-turn, in ms.
`noise_reduction`	`str`	`"near_field"`	Input denoising profile: `"near_field"`, `"far_field"`, or `None` to disable.
`response_format`	`str`	`"json"`	Transcript format (`"json"`, `"text"`, …).
`base_url`	`str`	`None`	Override the OpenAI API base URL.

Azure OpenAI (`AzureOpenAISTT`)

Parameter	Type	Default	Description
`api_key`	`str`	`None`	Azure OpenAI API key. Falls back to `AZURE_API_KEY`, then `AZURE_OPENAI_API_KEY`.
`azure_endpoint`	`str`	`None`	Azure OpenAI endpoint. Falls back to `AZURE_OPENAI_ENDPOINT`.
`deployment`	`str`	`None`	Deployment name. Falls back to `AZURE_OPENAI_STT_DEPLOYMENT`.
`api_version`	`str`	`"2025-03-01-preview"`	Azure OpenAI REST API version.
`model`	`str`	`""`	Underlying model name passed in the request body (e.g. `"whisper-1"`). Usually left empty when `deployment` is set.
`language`	`str`	`"en"`	ISO-639-1 language code of the spoken audio.
`stream`	`bool`	`False`	When `True`, receive transcription results as a streaming response.
`input_sample_rate`	`int`	`48000`	Input audio sample rate in Hz.
`output_sample_rate`	`int`	`24000`	Resampled rate sent to the Azure endpoint in Hz.
`prompt`	`str`	`None`	Optional text to guide transcription style or continue prior context.
`temperature`	`float`	`None`	Sampling temperature in `[0.0, 1.0]`.
`response_format`	`str`	`"json"`	Response format: `"json"`, `"verbose_json"`, `"text"`, `"srt"`, or `"vtt"`.
`turn_detection`	`str`	`"server_vad"`	Turn-detection strategy for streaming: `"server_vad"` or `"none"`.

Import paths

SDK	Import	Constructor
Python	`from zrt.plugins import OpenAISTT, AzureOpenAISTT`	`OpenAISTT(...)` / `AzureOpenAISTT(...)`
JavaScript	-	Coming soon
Go	-	Coming soon

Transcribed text passes to the LLM once turn detection decides the caller has finished speaking.

Plugins

Inference Gateway

Speech-to-text

LLM

Text-to-speech

Realtime models

Avatars

Turn detection

Voice activity detection

Noise cancellation

Setup

Usage

Azure OpenAI

Parameters

Azure OpenAI (`AzureOpenAISTT`)

Import paths

​Setup

​Usage

​Azure OpenAI

​Parameters

​Azure OpenAI (AzureOpenAISTT)

​Import paths

Setup

Usage

Azure OpenAI

Parameters

Azure OpenAI (`AzureOpenAISTT`)

Import paths