Zero Runtime Inference Gateway

The Inference Gateway is a unified entry point for the model stack. Instead of holding a separate API key for each provider, you authenticate once with your Zero Runtime auth token and route speech-to-text, LLM, text-to-speech, turn detection, and denoise through the gateway. It manages the upstream provider connections, resampling, and streaming server-side.

Setup

Gateway providers connect using your Zero Runtime auth token, no provider API keys are required. Set it in the worker environment:

export ZRT_AUTH_TOKEN=<token>

Usage

Import gateway providers from zrt.inference and drop them into the pipeline exactly like normal plugins. Every gateway class is a drop-in replacement for its zrt.plugins equivalent. Turn detection is the unified TurnDetector (imported from zrt.plugins) using an Echo model, which also routes through the gateway:

from zrt import Pipeline
from zrt.inference import SarvamAISTT, GoogleLLM, CartesiaTTS, AICousticsDenoise
from zrt.plugins import SileroVAD, TurnDetector

pipeline = Pipeline(
    stt=SarvamAISTT(),        # via gateway
    llm=GoogleLLM(),          # via gateway
    tts=CartesiaTTS(),        # via gateway
    vad=SileroVAD(),          # local plugin
    turn_detector=TurnDetector(model="echo-large"),  # Echo, via gateway
    denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"),
)

You can mix gateway and direct-provider components freely in the same pipeline. For example, run STT and TTS through the gateway while the LLM uses its own key.

Supported providers

Stage	Gateway providers
Speech-to-text	Deepgram, Google, Sarvam AI, AssemblyAI, Cartesia
LLM	Google, Sarvam AI
Text-to-speech	Cartesia, Google, Deepgram, Sarvam AI
Turn detection	Echo (echo-small, echo-large)
Denoise	ai-coustics, Sanas

Import paths

Every gateway class is imported from the single zrt.inference module:

Stage	Import
Speech-to-text	`from zrt.inference import DeepgramSTT, GoogleSTT, SarvamAISTT, AssemblyAISTT, CartesiaSTT`
LLM	`from zrt.inference import GoogleLLM, SarvamAILLM`
Text-to-speech	`from zrt.inference import CartesiaTTS, GoogleTTS, DeepgramTTS, SarvamAITTS`
Turn detection	`from zrt.plugins import TurnDetector` (Echo models route through the gateway)
Denoise	`from zrt.inference import Denoise, AICousticsDenoise, SanasDenoise`

Denoisers also expose factory methods, Denoise.sanas() and Denoise.aicoustics(), which return the same configured component as the named aliases above.

Why use it

One credential. Authenticate with a single Zero Runtime token instead of provisioning and rotating a key for every provider.
Less client work. Connection management, resampling, and streaming happen server-side.
Easy switching. Swap a gateway provider without onboarding a new API key.

What’s next

Configure a Pipeline

How gateway providers slot into a pipeline, with cascading and realtime examples.

Plugins

The full catalog of STT, LLM, TTS, and denoise providers.

Plugins

Inference Gateway

Speech-to-text

LLM

Text-to-speech

Realtime models

Avatars

Turn detection

Voice activity detection

Noise cancellation

Zero Runtime Inference Gateway

Setup

Usage

Supported providers

Import paths

Why use it

What’s next

Configure a Pipeline

Plugins

​Setup

​Usage

​Supported providers

​Import paths

​Why use it

​What’s next

Configure a Pipeline

Plugins

Setup

Usage

Supported providers

Import paths

Why use it

What’s next