# Anam Source: https://docs.zeroruntime.ai/api-reference/node-js/avatar/anam TypeScript API reference for the anam avatars provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AnamAvatar Initialize the Anam Avatar plugin. ```ts theme={null} import { AnamAvatar } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Simli Source: https://docs.zeroruntime.ai/api-reference/node-js/avatar/simli TypeScript API reference for the simli avatars provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SimliAvatar Initialize the Simli Avatar plugin. ```ts theme={null} import { SimliAvatar } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Agent & Session Source: https://docs.zeroruntime.ai/api-reference/node-js/core/agent-and-session TypeScript API reference for Agent & Session. ## Agent What the agent is, what it can do, and how it behaves on a call. Subclass it to give the agent behaviour: fields holding `function_tool` callables are registered automatically, and the `on_*` hooks below are called by the session as the call progresses. An instance is inert until `serve` registers it and a session is started against it. The pipeline is the other half. This class carries the prompt, the tools and the call-shaped options; `Pipeline` carries the providers that hear and speak. ### Options ```ts theme={null} Agent(options: AgentOptions) ``` Required. The name `serve` registers under and the runtime routes sessions to. Other agents this one hands off to, kept on `alternates`. Nothing registers them for you: `serve` registers one `agent_id` per call, and `Session.add_handoff` names the target by that id. Summarise the conversation at teardown and POST it somewhere. Spoken on the way out, when the agent ends the call itself. Spoken as soon as the agent joins, before the caller says anything. Carry the chat history over when another agent hands off to this one. Off means it starts the conversation fresh. The system prompt. Persona, task, and the rules the model is expected to hold to. Hard ceiling on the call. The runtime ends the session when it is reached. `MCPServerStdio` and `MCPServerHTTP` whose tools join the agent's own. Connected on the first call to `initialize_mcp`, not here. Display name. Defaults to `agent_id`. The `Pipeline` this agent runs on. How long the runtime waits for one of this agent's tools to return before handing the model a timeout. `null` keeps the runtime's own default of 30 seconds. Raise it only for an agent whose tools genuinely run long. The case it exists for is a tool that awaits `Session.warm_transfer` -- that runs for `supervisor_join_timeout + briefing_timeout`, minutes rather than seconds, and the default would kill the round trip while the transfer carried on underneath. The cost is symmetric: a tool that hangs holds the turn for exactly this long. Tools defined elsewhere. Tools declared as fields on the agent are found on their own, so this is for the ones that are not. Seconds of caller silence before the agent nudges. 0 disables it; negative is rejected. What the nudge says. ### agent\_id ```ts theme={null} agent_id(): void ``` ### instructions ```ts theme={null} instructions(): void ``` ### session ```ts theme={null} session(): void ``` ### tools ```ts theme={null} tools(): void ``` ### add\_server ```ts theme={null} add_server(mcp_server: MCPServer): Promise ``` Connect one MCP server and append what it publishes to `tools`. Ignores anything that is not an `MCPServerStdio` or `MCPServerHTTP`. The server to connect. ### cleanup ```ts theme={null} cleanup(): Promise ``` Close the MCP connections and drop the tools they contributed. Called for you at teardown. Safe to call twice, and safe with no MCP servers configured. ### emit ```ts theme={null} emit(event: 'agent_started', args: unknown[]): void ``` Call every handler subscribed to `event`. Returns as soon as the handlers have been started, not when they finish, and never throws: an exception inside a handler is logged against the event name. Ignored once the emitter has been closed. The event name. Passed through to each handler. ### hangup ```ts theme={null} hangup(reason: string, farewell: string): Promise ``` End the call from the agent's side. A no-op with no live session. Recorded against the session. Spoken before hanging up. Empty uses the agent's own `farewell`. ### initialize\_mcp ```ts theme={null} initialize_mcp(): Promise ``` Connect every configured MCP server and adopt its tools. Runs once; later calls return immediately. `serve` calls it while building the agent, so a server that is down fails the build rather than the first tool call. ### off ```ts theme={null} off(event: 'agent_started', callback: EventHandler): void ``` Unsubscribe one handler. Matched by identity, so it has to be the same object that was registered -- a fresh arrow function will not match. Removing something that was never subscribed does nothing. ### on ```ts theme={null} on(event: 'agent_started', callback: EventHandler): EventHandler ``` Subscribe to an event. Handlers fire in the order they were registered, and registering the same one twice calls it twice. The event name. The handler. ### on\_enter ```ts theme={null} on_enter(): Promise ``` Called once the agent has joined and the session is live. Override it to open the conversation, prime the context, or set up standing configuration such as `Session.set_thinking_audio`. ### on\_exit ```ts theme={null} on_exit(): Promise ``` Called as the call is ending, before teardown finishes. ### on\_participant\_joined ```ts theme={null} on_participant_joined(participant: Participant): Promise ``` Called when somebody joins the room the agent is in. ### on\_participant\_left ```ts theme={null} on_participant_left(participant: Participant): Promise ``` Called when somebody leaves. The agent is not one of them. ### register\_tools ```ts theme={null} register_tools(): void ``` Collect the `function_tool` callables declared on this agent. Called for you the first time `tools` is read. Call it again after adding one at runtime; it never registers the same tool twice. ### toString ```ts theme={null} toString(): string ``` Returns a string representation of an object. ### update\_tools ```ts theme={null} update_tools(tools: FunctionTool[]): void ``` Replace the tool list outright. This drops the tools that were found on the agent and anything MCP added, so pass the full set you want. *** ## AgentContext What the runtime knows about a job, handed to the agent factory. `serve` passes one to your factory if it takes an argument, so the agent can be built around the call it is about to take -- a caller's number out of `metadata`, a different prompt per room. ### Options ```ts theme={null} AgentContext(options: AgentContextOptions) ``` Which registration the job was routed to. The runtime's id for this dispatch. Whatever the dispatcher attached -- the caller's number, a tenant id, anything the job was started with. The room the agent is joining. The `Room` the runtime resolved for this job. The auth token for this job. ### toString ```ts theme={null} toString(): string ``` *** ## CallSummary Summarise the conversation when it ends, and POST the result. ### Options ```ts theme={null} CallSummary(options: CallSummaryOptions) ``` Set false to carry the configuration without summarising. Where the summary is POSTed. Without one it is generated and logged but sent nowhere. Sent with the POST -- an auth header, usually. Replaces the built-in summarising prompt. A second model for the summary alone. `null` reuses the session's. Worth pointing at something cheap: it runs once, over the whole transcript, while the process is already shutting down. Seconds the summariser gets. 0 takes the runtime's default. Summarising is an LLM round trip over the whole conversation, so it is a different order of work from the rest of teardown. The ceiling is the runtime's SIGTERM grace, since this runs while the agent is shutting down -- asking for more than that gets the process killed mid-request rather than granting the time. *** ## serve ```ts theme={null} serve(agent: AgentFactory, options: ServeOptions): Promise ``` Register `agent` and serve every call dispatched to it. Every option left here describes *this worker* -- how many calls it takes, what it logs, where it answers probes. Nothing here describes the session: that is Room's job, and the twelve options which used to say it a second time are gone, because two spellings of `recording` that could disagree is a question with no right answer. The returned promise settles when the worker is interrupted or shut down. The agent is registered with the runtime's registry, and every call dispatched to that id builds a fresh agent from the factory -- one instance per call, never one shared across them. A callable returning an `Agent`. Your subclass is usually the callable. An instance is rejected: concurrent calls would share one conversation. *** ## invoke ```ts theme={null} invoke(agent_id: string, options: InvokeOptions): Promise> ``` Start a session against an agent this process is already serving. Resolves once the runtime has accepted the session, which is the point at which the room is joinable. An ordinary async function: `await` it from `on_ready`, or from anywhere else once `serve()` is running. The agent has to be registered in *this* process. There is no dispatch RPC to route a start to another one: the pipeline, credentials and tool schemas all live where `serve` was called. The id passed to `serve`. *** ## Session One live call, and everything you can do to it while it runs. You are handed one rather than constructing one: `invoke` returns it, and inside a `function_tool` or an `on_*` hook it is the session the agent is bound to. Every method here is a message to the agent process, so they are all awaited and none of them block the call. The object stays usable after the call ends -- `ended` goes true and the send methods become no-ops -- so a hook can read `ended_reason` without guarding first. ### Constructor ```ts theme={null} Session(stream: any, accepted: ProtoMessage, outbound: Outbound, options: object) ``` ### agent ```ts theme={null} agent(): void ``` ### ended ```ts theme={null} ended(): void ``` ### playground\_url ```ts theme={null} playground_url(): void ``` ### \[asyncDispose] ```ts theme={null} [asyncDispose](): Promise ``` ### aclose ```ts theme={null} aclose(): Promise ``` Release everything this call holds. Idempotent, and safe after any ending. Every way a call can finish routes here. Closing the transport is the part that cannot be skipped: the outbound queue feeds a writer this SDK holds, and a live writer keeps the stream -- and the process -- alive. ### add\_handoff ```ts theme={null} add_handoff(to_agent: string, __namedParameters: object): Promise ``` Hand the conversation to another agent, by id. The target has to be registered -- `serve` registers one `agent_id` per call -- and whether it inherits the history is that agent's `inherit_context`. The `agent_id` taking over. ### add\_message ```ts theme={null} add_message(role: string, content: string, __namedParameters: object): Promise ``` Write a message into the conversation without speaking it. Useful for handing the model something it should know -- a CRM lookup, a verification result -- without a turn being spent saying it out loud. `system`, `developer`, `user` or `assistant`. A `ChatRole` works too. The message body. ### change\_component ```ts theme={null} change_component(options: ChangeComponentOptions): Promise ``` Swap the named components and leave the rest of the call alone. ### change\_pipeline ```ts theme={null} change_pipeline(pipeline: Pipeline, __namedParameters: object): Promise ``` Replace the whole pipeline mid-call. Everything not named is rebuilt from the pipeline you pass, so state a complete one. To change a provider or two and leave the rest alone, use `change_component`. The `Pipeline` to switch to. ### destroy ```ts theme={null} destroy(reason: string): Promise ``` Tear the session down at the runtime, without a goodbye. The blunt one: no farewell, no teardown hooks worth waiting for. `end` is the polite version. ### detach ```ts theme={null} detach(): Promise ``` Stop watching this session without ending the call. The agent keeps running runtime-side; this process just stops listening. For a call that should actually stop, use `end` or `destroy`. ### emit ```ts theme={null} emit(event: string, args: unknown[]): void ``` Call every handler subscribed to `event`. Returns as soon as the handlers have been started, not when they finish, and never throws: an exception inside a handler is logged against the event name. Ignored once the emitter has been closed. The event name. Passed through to each handler. ### end ```ts theme={null} end(reason: string, farewell: string): Promise ``` Ask the runtime to end the call. A no-op once it has ended. Recorded against the session. Spoken before hanging up. ### events ```ts theme={null} events(): AsyncGenerator ``` Iterate everything the runtime reports, until the call ends. Keeps yielding briefly past the `ended` event so the last transcripts and metrics are not cut off mid-teardown. ### get\_context\_history ```ts theme={null} get_context_history(__namedParameters: object): Promise ``` Fetch the conversation as the agent process holds it. ### get\_metrics ```ts theme={null} get_metrics(__namedParameters: object): Promise ``` Fetch the per-turn latency metrics collected so far. ### get\_participants ```ts theme={null} get_participants(__namedParameters: object): Promise ``` Who is in the room right now, as `Participant` records. ### interrupt ```ts theme={null} interrupt(__namedParameters: object): Promise ``` Stop the agent talking. ### log\_playground ```ts theme={null} log_playground(): void ``` Announce the join URL on stdout, at most once per session. Both the direct-start path and the registry worker path call this, and `invoke` adds a third; the guard is what keeps one session to one URL. It prints rather than logs because the URL is the one line the operator is waiting to copy, and a log record buries it behind level and module columns that are noise for a value you paste into a browser. ### off ```ts theme={null} off(event: string, callback: EventHandler): void ``` Unsubscribe one handler. Matched by identity, so it has to be the same object that was registered -- a fresh arrow function will not match. Removing something that was never subscribed does nothing. ### on ```ts theme={null} on(event: string, callback: EventHandler): EventHandler ``` Subscribe to an event. Handlers fire in the order they were registered, and registering the same one twice calls it twice. The event name. The handler. ### play\_background\_audio ```ts theme={null} play_background_audio(file: string | object | null, __namedParameters: object): Promise ``` Start a sound bed under the call. Needs the room's mixing track, `Room({ background_audio: true })`. A path, or a `BackgroundAudio` carrying one along with its volume and looping. A disabled `BackgroundAudio` plays nothing. ### process\_text ```ts theme={null} process_text(text: string): Promise ``` Feed text to the agent as though the caller had said it. The full turn runs: the model answers and the answer is spoken. ### publish\_to\_pubsub ```ts theme={null} publish_to_pubsub(pubsub_config: PubSubPublishConfig): Promise ``` Publish one pubsub frame into the room. The topic, body and options to publish with. ### reply ```ts theme={null} reply(instructions: string, options: ReplyOptions): Promise ``` Ask the model for a line, generated against `instructions`. Unlike `say`, this is a generation: the model sees the conversation so far plus these instructions, and what it produces is spoken. What to tell the model for this turn alone. ### say ```ts theme={null} say(text: string, options: SayOptions): Promise ``` Speak an exact line. No model involved. What to say. ### set\_thinking\_audio ```ts theme={null} set_thinking_audio(file: string | null, __namedParameters: object): Promise ``` Play a sound while the agent is generating its reply. Standing configuration rather than a one-shot: once set, the session starts it at the top of every generation and stops it when the answer begins, for as long as the session lasts. Set it from `on_enter`. `file` omitted takes the runtime's own thinking sound. Any file libav can decode works -- wav, mp3, ogg, flac, m4a. Needs the room's mixing track, `Room({ background_audio: true })`, like all audio that is not speech. ### stop ```ts theme={null} stop(reason: string): Promise ``` End the call and release it. What `await using` calls. ### stop\_background\_audio ```ts theme={null} stop_background_audio(): Promise ``` Stop the sound bed. Leaves the thinking sound alone. ### subscribe\_to\_pubsub ```ts theme={null} subscribe_to_pubsub(pubsub_config: PubSubSubscribeConfig): Promise ``` Listen on one pubsub topic for the rest of the call. The topic and its handler. ### toString ```ts theme={null} toString(): string ``` Returns a string representation of an object. ### transfer\_call ```ts theme={null} transfer_call(transfer_to: string, __namedParameters: object): Promise> ``` Cold-transfer the call and leave. The caller is handed to `transfer_to` with no introduction and the agent drops out. Where to send them -- a SIP URI or a phone number. ### wait ```ts theme={null} wait(__namedParameters: object): Promise ``` Consume `events` until the call ends, logging as it goes. The usual last line of a script that started a session and has nothing else to do. *** ## current\_session ```ts theme={null} current_session(): Session ``` The session whose callback is currently running. Valid anywhere the SDK invokes your code on a session's behalf: a `function_tool`, a lifecycle hook, a pipeline hook. Useful for code that runs on the call without holding a reference to it -- a module-level `pipeline.on` handler, say. Anything defined on the agent should reach the call through `this.session` instead, and a pubsub handler subscribed with `Session.subscribe_to_pubsub` already has the session in hand. *** ## Participant ### Constructor ```ts theme={null} Participant(options: object) ``` *** ## PubSubSubscribeConfig One topic to listen on, and what to call for each frame. Passed to `Session.subscribe_to_pubsub` and `Room.subscribe_to_pubsub`. ### Options ```ts theme={null} PubSubSubscribeConfig(options: PubSubSubscribeConfigOptions) ``` Called once per frame with the raw frame as an object -- the whole thing as the transport delivered it, since pubsub promises no schema beyond `message`. Sync or async; a promise is awaited. A handler that declares a second parameter is additionally told whether the frame is backlog: ```ts theme={null} function cb(frame, backlog) { if (backlog) return; // already in the topic before we joined } ``` Subscribing replays whatever the topic already held, so a one-argument handler sees that history as ordinary traffic -- which is the transport's own shape, and worth knowing about before replying to it. The topic to subscribe to. *** ## PubSubPublishConfig One frame to publish into the room. Passed to `Session.publish_to_pubsub` and `Room.publish_to_pubsub`. ### Options ```ts theme={null} PubSubPublishConfig(options: PubSubPublishConfigOptions) ``` The text body. Transport options. Only `sendOnly` -- a list of participant ids to deliver to -- is carried; anything else is dropped, because the wire has nowhere to put it. Omitted, everyone subscribed to the topic gets the frame. Structured data alongside it, JSON-encoded on the way out. The topic to publish on. *** ## UtteranceHandle One thing the agent is saying, and how it went. Returned by `Session.say` and `Session.reply`. Call `wait()` to block until the line finishes -- it resolves whether the line played out or was cut off, so check `interrupted` rather than assuming it was heard. ```ts theme={null} const handle = await session.say('One moment while I look that up.'); await handle.wait(); if (handle.interrupted) { // ... } ``` The wait is a method rather than the handle itself being awaitable: a thenable would be unwrapped by `await session.say(...)`, so that line would not return until the audio had finished playing. As it is, `say()` hands the handle back the moment the request is on its way. Waiting is optional. Ignore the handle and the line still plays. ### Constructor ```ts theme={null} UtteranceHandle(utterance_id: string) ``` ### interrupted ```ts theme={null} interrupted(): void ``` ### state ```ts theme={null} state(): void ``` ### done ```ts theme={null} done(): boolean ``` Whether the utterance has settled, either way. Never blocks. ### toString ```ts theme={null} toString(): string ``` ### wait ```ts theme={null} wait(): Promise ``` Resolve once the line has settled, either way. Resolves to this handle. *** ## Room Which room the agent joins. `room_id: null` asks for a new one. ### Options ```ts theme={null} Room(options: RoomOptions) ``` The 0.1.2 spelling of `name`. Folded into it when set. Publish the agent under a fixed participant id. `null` lets the room mint one, which is what a single-agent session wants. Codec the room negotiates -- `"opus"` (the default when `null`), `"pcmu"`, `"pcma"`, `"g722"`. This is what tells a meeting from a phone call: opus is 48 kHz, pcmu/pcma are 8 kHz and g722 is 16 kHz, and that rate reaches the VAD and STT. Leave it `null` for meetings; set `"pcmu"` or `"pcma"` for SIP so the pipeline sizes itself to the narrowband audio instead of resampling it up to 48 kHz and back down again. VideoSDK token for the room. Falls back to the environment. It travels in `StartSession.credentials`, never in `params_json`. End the session when the last participant leaves. `null` leaves the runtime's default (TRUE) in place. Open the room's mixing audio track. Nothing plays by itself -- this is the track, not the sound. Start audio on it with `session.play_background_audio({ file, volume, looping })` and stop it with `session.stop_background_audio()`. Required before either call: without the track the SDK refuses, and the failure is a silent one from the caller's side. Seconds of silence before the session is considered idle. Whether the agent joins the meeting. `null` leaves the runtime's default (TRUE) in place. Display name the agent publishes under. Seconds with nobody attached before the session is reclaimed. Carried in `SessionLimits.inactivity_timeout_seconds`, same as above. Where this session's telemetry goes. `null` takes the runtime's defaults. Ask the runtime for a playground URL for this session. Record the session. Off unless asked for -- recording a call is a decision about the person on the other end, so the default has to be the safe one and the same runtime serves recorded and unrecorded sessions side by side. Audio is always captured when this is on; the two below add tracks to it. Also record the screen share. Needs `recording` and `vision` -- there is no screen-share track on a session that is not receiving video. Also record camera video. Needs `recording`. Existing room to join. A new room is created when `null`. Hard cap on session length, in seconds. Carried in `SessionLimits.max_session_duration_seconds` rather than in `RoomSpec`, so it is read by the transport and not by `to_proto`. `null` inherits the runtime's own ceiling. Override the signaling base URL. Deliver camera frames to the agent process. Whether the agent waits for someone to join before speaking. `null` leaves the runtime's default (TRUE) in place. ### explicitly\_set ```ts theme={null} explicitly_set(): void ``` *** ## Sip The telephony leg of a session -- who is being called, and from what. ### Options ```ts theme={null} Sip(options: SipOptions) ``` Caller id presented on the call. Identifier correlating the call. Destination number/address for an outbound call. Call direction/type. Additional key/values merged into the metadata map. URL to receive call-event callbacks. ### to\_metadata ```ts theme={null} to_metadata(): Record ``` This leg as the metadata dict the room is created with. *** ## UNSET ```ts theme={null} const UNSET: any ``` Annotated `any` so `recording: any = UNSET` type-checks while the honest default stays visible in the signature a reader sees. *** ## ZeroRuntimeChannel A connection to one runtime, and the sessions started over it. `serve` and `invoke` own one for you. Hold one directly to start sessions from async code, to reach a runtime other than the configured one, or to ask a runtime what capacity it has. Connect explicitly with `await channel.connect()`; every method connects on first use anyway. TLS is decided by the target unless `secure` says otherwise: local hosts are plaintext, port 443 is TLS. `ZERORUNTIME_INSECURE=1` forces plaintext. ### Options ```ts theme={null} ZeroRuntimeChannel(options: ZeroRuntimeChannelOptions) ``` ### auth\_token ```ts theme={null} auth_token(): void ``` ### connected ```ts theme={null} connected(): void ``` ### stub ```ts theme={null} stub(): void ``` ### aclose ```ts theme={null} aclose(): Promise ``` Close the connection. Sessions started over it stop being watched. ### attach ```ts theme={null} attach(session_id: string, __namedParameters: object): Promise ``` Attach to a session that is already running. The session to attach to. ### connect ```ts theme={null} connect(__namedParameters: object): Promise ``` Open the connection. Returns immediately if it is already open. ### destroy ```ts theme={null} destroy(session_id: string, reason: string): Promise ``` Tear a session down at the runtime, by id. The session to destroy. Recorded against it. ### session\_info ```ts theme={null} session_info(session_id: string): Promise ``` Look up one session's state, pid, room and uptime. Works for any session on this runtime, including ones this process did not start. ### start ```ts theme={null} start(options: StartOptions): Promise ``` Start a session and return it, ready to run. The async counterpart to `invoke`, and what to reach for inside async code, where `invoke` cannot be called. ### status ```ts theme={null} status(): Promise ``` Ask the runtime what capacity it has. ### use\_tls ```ts theme={null} use_tls(): boolean ``` Whether this channel will speak TLS. `secure` wins if it was set, then `ZERORUNTIME_INSECURE`, then the target itself: a local host is plaintext and port 443 is TLS. # Audio Source: https://docs.zeroruntime.ai/api-reference/node-js/core/audio TypeScript API reference for Audio. ## BackgroundAudio A sound bed the runtime plays into the room alongside the agent. Needs the room's mixing track, `Room({ background_audio: true })`, like all audio that is not speech. ### Options ```ts theme={null} BackgroundAudio(options: BackgroundAudioOptions) ``` Set false to keep the configuration and play nothing. The file to play. Anything libav decodes -- wav, mp3, ogg, flac, m4a. Empty is rejected. Restart when the file ends instead of falling silent. `mixing` or `playback`. Only `mixing` has a counterpart on the runtime today; `playback` is accepted, warned about, and treated as `mixing`. Gain. 1.0 is the file as recorded; must not be negative. *** ## run\_stt ```ts theme={null} run_stt(audio_stream?: AsyncIterable | null): AsyncGenerator ``` Yield the transcript an `stt` hook was called with. The shim that lets a hook written against a local STT run unchanged. STT itself runs in the agent process and only the transcript crosses, so the audio stream is empty here -- draining it is allowed and yields nothing. Filtering or rewriting the text works as written; transforming the audio does not. The hook's audio stream, if it takes one. Drained and discarded. # Errors & Enums Source: https://docs.zeroruntime.ai/api-reference/node-js/core/errors-and-config TypeScript API reference for Errors & Enums. ## ZeroRuntimeError Base for everything this SDK raises on its own behalf. Catch it to handle any ZeroRuntime failure without naming each one. Errors the language raises -- `TypeError` for a bad argument, `RangeError` for one out of bounds -- are left as themselves rather than wrapped. ### Constructor ```ts theme={null} ZeroRuntimeError(message: string) ``` *** ## SessionRejected The ZeroRuntime refused to create the session. ### Constructor ```ts theme={null} SessionRejected(message: string, options: object) ``` *** ## ProviderUnavailable A provider in the pipeline could not be reached or would not serve. Usually a missing or rejected vendor key, or a model the account is not entitled to. A fallback on the same slot is what keeps this from ending the call. ### Constructor ```ts theme={null} ProviderUnavailable(message: string) ``` *** ## ZeroRuntimeUnreachable The runtime could not be reached, or did not answer in time. A transport failure rather than a refusal: nothing was decided about the session, so retrying is reasonable. ### Constructor ```ts theme={null} ZeroRuntimeUnreachable(message: string) ``` *** ## ToolTimeout A `function_tool` did not return before its deadline. The turn continues without the tool's answer. A tool that calls out to a slow API wants its own timeout, so it can say something useful instead of being cut off. ### Constructor ```ts theme={null} ToolTimeout(message: string) ``` *** ## AgentState Where the agent is in a turn -- listening, thinking, speaking, or on its way in or out of the call. ```ts theme={null} enum AgentState { CLOSING = 'closing', IDLE = 'idle', LISTENING = 'listening', SPEAKING = 'speaking', STARTING = 'starting', THINKING = 'thinking', } ``` *** ## UserState What the caller is doing, as the runtime reports it. ```ts theme={null} enum UserState { IDLE = 'idle', LISTENING = 'listening', SPEAKING = 'speaking', } ``` *** ## ChatRole Who a message in the conversation came from. A plain string works anywhere one of these is accepted -- the members *are* strings -- so `'user'` and `ChatRole.USER` are interchangeable. ```ts theme={null} enum ChatRole { ASSISTANT = 'assistant', DEVELOPER = 'developer', SYSTEM = 'system', TOOL = 'tool', USER = 'user', } ``` # Events & Observability Source: https://docs.zeroruntime.ai/api-reference/node-js/core/events TypeScript API reference for Events & Observability. ## Observability Where this session's traces, metrics and logs go. Each stream is configured on its own, and a stream left `null` follows the runtime's own settings rather than being switched off. ### Options ```ts theme={null} Observability(options: ObservabilityOptions) ``` Level the agent process logs at. One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. The agent process's log records. Latency and usage counters. Per-turn spans -- what the model was asked, what each provider took. *** ## Exporter Where one stream of telemetry is shipped. ### Options ```ts theme={null} Exporter(options: ExporterOptions) ``` Set false to turn this stream off while leaving the rest of the configuration in place. Sent with each export -- the collector's auth header, usually. An OTLP endpoint. `null` sends it wherever the runtime is configured to send its own. # Pipeline Source: https://docs.zeroruntime.ai/api-reference/node-js/core/pipeline TypeScript API reference for Pipeline. ## Pipeline The providers a call runs on: what hears, what thinks, what speaks. Two shapes are valid. A cascade names `stt`, `llm` and `tts` separately; a realtime pipeline names a speech-to-speech model in `llm` alone, because a realtime model is the language model. `realtime` is accepted as the older spelling of the same slot; naming both is rejected. A fallback slot -- `stt`, `llm`, `tts` -- also takes an array, in which case the head serves and the tail stands by. Wrap that array in `FallbackSTT`, `FallbackLLM` or `FallbackTTS` to say how it demotes as well as what it falls back to; each slot carries its own settings. ### Options ```ts theme={null} Pipeline(options: PipelineOptions) ``` A video avatar to render the agent's speech through. How a long conversation is kept inside the model's limit. Noise cancellation applied to the inbound audio. Deliver keypad tones to the agent. `null` drops them. How patiently the agent waits before answering. What counts as the caller barging in. The language model -- a text LLM for a cascade, a `FallbackLLM` chain of them, or a speech-to-speech model for a realtime pipeline. `PronunciationRule` substitutions applied to generated text on its way to TTS. The older spelling of a speech-to-speech `llm`. Speech-to-text, or a `FallbackSTT` chain. Omit it on a realtime pipeline. Text-to-speech, or a `FallbackTTS` chain. Omit it on a realtime pipeline. End-of-turn detection, for deciding when they finished. Voice activity detection -- what notices the caller is talking. Answering-machine detection, for outbound calls that may be picked up by one. ### is\_realtime ```ts theme={null} is_realtime(): void ``` ### mode ```ts theme={null} mode(): void ``` ### chain ```ts theme={null} chain(slot: string): void ``` Everything configured for one slot: the primary, then its fallbacks. Empty for a slot nothing was named for, and for a slot that is not one. ### fallback\_settings ```ts theme={null} fallback_settings(slot: string): _Fallback | null ``` How one slot demotes, or `null` where nothing was tuned. `fallbacks` is who a slot falls back to; this is how. ### fallbacks ```ts theme={null} fallbacks(slot: string): void ``` What a slot falls back to, in the order they are tried. ### hooks ```ts theme={null} hooks(event: string): function[] ``` The handlers registered for one event, in registration order. ### on ```ts theme={null} on(event: PipelineHookEvent, callback: H): H ``` Register a hook that runs in this process, mid-turn. The events are the transcript hooks `stt` and `llm`, and the turn hooks `user_turn_start`, `user_turn_end`, `agent_turn_start` and `agent_turn_end`. An async generator function registered on `llm` is filed as `llm_stream`: it is handed the model's output as a stream and yields what should be spoken, so it can buffer, rewrite or drop the turn. A plain async function on the same event sees the finished text instead. Per-component latency is a separate family, registered through its own namespace rather than by spelling the prefix here: ```ts theme={null} pipeline.metrics.on('stt', (data) => { ... }); ``` Which hook to attach to. The handler. ### primary ```ts theme={null} primary(slot: string): ProviderSpec | null ``` The provider a slot uses first, or `null` if it has none. ### providers ```ts theme={null} providers(): Iterable ``` Every provider in the pipeline, fallbacks included. ### toString ```ts theme={null} toString(): string ``` *** ## NO\_CHANGE ```ts theme={null} const NO_CHANGE: any ``` Annotated `any` on purpose: it lets `tts: any = NO_CHANGE` type-check while the honest annotation stays visible in the signature a reader sees. *** ## PronunciationRule Rewrite generated text on its way to TTS. Positional rather than an options object, because two strings in a fixed order read better than four words of keys: `PronunciationRule('nginx', 'engine x')`. ### Constructor ```ts theme={null} PronunciationRule(find: string, replace: string, case_sensitive: boolean) ``` *** ## ContextWindow Keep a long conversation inside the model's context, automatically. ### Options ```ts theme={null} ContextWindow(options: ContextWindowOptions) ``` Recent user turns kept verbatim. The SDK's default is 3, and it is worth keeping several -- summarising the sentence the caller just said is how an agent starts answering the wrong question. Or bound by item count. `null` for none. Compress once the context exceeds this. `null` for no token ceiling. The SDK's default is 10. The model that compresses. A second LLM, described here and built in the agent process like every other provider. *** ## EOUConfig When the caller is judged to have finished speaking. ### Options ```ts theme={null} EOUConfig(options: EOUConfigOptions) ``` Treat "mhm", "haan", "right" as listening noises rather than turns. `null` takes the runtime's default; the SDK's own default is on for detectors that support it. How sure the detector must be. Lower answers sooner and interrupts more. `[min, max]` seconds to wait before treating silence as the end of a turn. `[0.0, 0.0]` answers as soon as the detector says the turn is complete, which is snappy but cuts off anyone who pauses to think. `DEFAULT` waits a fixed time after speech stops. `ADAPTIVE` varies it with how certain the turn detector is -- longer when the sentence sounds unfinished. *** ## InterruptConfig What counts as the caller interrupting, rather than just making a noise. ### Options ```ts theme={null} InterruptConfig(options: InterruptConfigOptions) ``` How long to stay paused after an interruption that turned out to be nothing. Seconds to fade the agent's audio out over. An instant cut sounds like a dropped call. STT confidence floor for those words. Seconds of speech before it counts. Raising this is the usual fix for an agent that stops every time someone breathes. Words required before it counts. `2` ignores a stray "yeah". `VAD_ONLY` reacts to any speech-like audio -- fastest, and the most easily fooled by a cough or a door. `STT_ONLY` waits for words. `HYBRID` uses both. Pick the sentence back up when the interruption proves false, rather than dropping it. *** ## FallbackSTT Speech-to-text providers to try in order, and how they demote. ### Constructor ```ts theme={null} FallbackSTT(providers: , options: FallbackOptions) ``` ### SLOT ```ts theme={null} SLOT(): void ``` *** ## FallbackLLM Language models to try in order, and how they demote. ### Constructor ```ts theme={null} FallbackLLM(providers: , options: FallbackOptions) ``` ### SLOT ```ts theme={null} SLOT(): void ``` *** ## FallbackTTS Text-to-speech providers to try in order, and how they demote. ### Constructor ```ts theme={null} FallbackTTS(providers: , options: FallbackOptions) ``` ### SLOT ```ts theme={null} SLOT(): void ``` *** ## DTMFHandler Deliver the caller's keypad tones to the agent instead of dropping them. ### Options ```ts theme={null} DTMFHandler(options: DTMFHandlerOptions) ``` What runs per keypress, taking `(key)` or `(key, payload)`. Left `null`, the agent's `on_dtmf` method is called instead -- which is the usual shape, because a keypad menu almost always wants the agent's own state to accumulate a multi-digit entry into. The runtime delivers one key per call either way; a PIN is accumulated on your side, not handed over whole. *** ## VoiceMailDetector Detect an answering machine on an outbound call. The detector runs in the agent process the ZeroRuntime starts: it buffers the opening speech for `duration` seconds and asks `llm` whether it is a person or a greeting. ### Options ```ts theme={null} VoiceMailDetector(options: VoiceMailDetectorOptions) ``` What runs on detection. Left `null`, the agent's `on_voicemail` method is called instead. Either is awaited, so anything said in it finishes before the call is ended. Replace the built-in classification prompt. Seconds of speech to buffer before deciding. The default is 2.0 -- long enough for "Hi, you've reached...". Set false to configure it without turning it on. The classifier. Required: without one there is nothing to classify with, and the detector is simply not installed. *** ## PipelineMode Which shape a pipeline resolved to. Derived from the slots that were filled rather than set by hand; read it off `Pipeline.mode`. ```ts theme={null} enum PipelineMode { FULL_CASCADING = 'full_cascading', HYBRID = 'hybrid', LLM_ONLY = 'llm_only', LLM_TTS_ONLY = 'llm_tts_only', PARTIAL_CASCADING = 'partial_cascading', REALTIME = 'realtime', STT_LLM_ONLY = 'stt_llm_only', STT_ONLY = 'stt_only', STT_TTS_ONLY = 'stt_tts_only', TTS_ONLY = 'tts_only', } ``` # Tools & MCP Source: https://docs.zeroruntime.ai/api-reference/node-js/core/tools-and-mcp TypeScript API reference for Tools & MCP. ## function\_tool ```ts theme={null} function_tool(options: FunctionToolOptions): FunctionTool, R> ``` Expose a function to the model as a callable tool. ```ts theme={null} const lookup_order = function_tool({ name: 'lookup_order', description: 'Find an order by its id.', parameters: { order_id: { type: 'string', description: "The customer's order number." }, }, execute: async ({ order_id }) => find(order_id), }); ``` The name, description and parameter schema are stated rather than derived: JavaScript keeps neither comments nor type annotations at runtime, so there is nothing to read a tool schema off. What you write here is exactly what the model is shown. Tools declared as fields on an `Agent` subclass are registered automatically; tools defined elsewhere go in `Agent({ tools: [...] })`. *** ## MCPServerStdio A local MCP server, run as a subprocess and spoken to over stdio. ### Options ```ts theme={null} MCPServerStdio(options: MCPServerStdioOptions) ``` Environment for the child. `null` inherits this process's, which is usually what you want: the server needs the same API keys you already have. The program to run, e.g. `process.execPath`. Its arguments -- typically the server script. Seconds to wait for the connection and for each tool call. The vendor's default is 5, which is tight for a server that starts an interpreter; its own examples pass 30. Directory to run it in. ### toString ```ts theme={null} toString(): string ``` *** ## MCPServerHTTP A remote MCP server, reached over HTTP. ### Options ```ts theme={null} MCPServerHTTP(options: MCPServerHTTPOptions) ``` Seconds to wait for the connection. The server's URL. Sent with every request -- an API key usually lives here. Read in this process, so the credential stays with you. Seconds to wait for the session, and for each tool call. Seconds a quiet stream may stay open. Long by default: an MCP server that has nothing to say is not broken. ### transport\_mode ```ts theme={null} transport_mode(): void ``` ### toString ```ts theme={null} toString(): string ``` # Aicoustics Source: https://docs.zeroruntime.ai/api-reference/node-js/denoise/aicoustics TypeScript API reference for the aicoustics noise cancellation provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AICousticsDenoise Create a Denoise instance configured for AI-Coustics. ```ts theme={null} import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key ``` ### Options # Rnnoise Source: https://docs.zeroruntime.ai/api-reference/node-js/denoise/rnnoise TypeScript API reference for the rnnoise noise cancellation provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## RNNoise Initialize the RNNoise denoise plugin. ```ts theme={null} import { RNNoise } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Sanas Source: https://docs.zeroruntime.ai/api-reference/node-js/denoise/sanas TypeScript API reference for the sanas noise cancellation provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SanasDenoise Create a Denoise instance configured for Sanas. ```ts theme={null} import { SanasDenoise } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key ``` ### Options # Anthropic Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/anthropic TypeScript API reference for the anthropic llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AnthropicLLM Initialize the Anthropic LLM. ```ts theme={null} import { AnthropicLLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Aws Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/aws TypeScript API reference for the aws llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AWSBedrockLLM AWS Bedrock LLM (Converse API) plugin for VideoSDK Agents. ```ts theme={null} import { AWSBedrockLLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Cerebras Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/cerebras TypeScript API reference for the cerebras llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CerebrasLLM Cerebras LLM implementation using the Cerebras Cloud SDK. ```ts theme={null} import { CerebrasLLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Cometapi Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/cometapi TypeScript API reference for the cometapi llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CometAPILLM Initialize the CometAPI LLM plugin. ```ts theme={null} import { CometAPILLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Google Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/google TypeScript API reference for the google llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GoogleLLM Create an LLM instance configured for Google Gemini. ```ts theme={null} import { GoogleLLM } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { GoogleLLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Openai Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/openai TypeScript API reference for the openai llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAILLM Initialize the OpenAI LLM plugin. ```ts theme={null} import { OpenAILLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options *** ## ZRTLLM Create an LLM instance configured for VideoSDK LLM. ```ts theme={null} import { ZRTLLM } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { ZRTLLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Sarvamai Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/sarvamai TypeScript API reference for the sarvamai llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SarvamAILLM Create an LLM instance configured for Sarvam AI. ```ts theme={null} import { SarvamAILLM } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { SarvamAILLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Xai Source: https://docs.zeroruntime.ai/api-reference/node-js/llm/xai TypeScript API reference for the xai llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## XAILLM LLM Plugin for xAI (Grok) API. ```ts theme={null} import { XAILLM } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Node JS SDK Source: https://docs.zeroruntime.ai/api-reference/node-js/overview API reference for the Zero Runtime Node JS SDK. Reference for every public class, factory, function, and type in the Zero Runtime Node JS SDK. Start with **Core** for agents, sessions, pipelines, tools, and context, then browse the provider sections (Speech-to-text, LLM, Text-to-speech, and more) for the plugins you compose into a pipeline. Install, conventions, and end-to-end examples. # Aws Source: https://docs.zeroruntime.ai/api-reference/node-js/realtime/aws TypeScript API reference for the aws realtime models provider. ## NovaSonicRealtime Nova Sonic's realtime model implementation ```ts theme={null} import { NovaSonicRealtime } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Azure Source: https://docs.zeroruntime.ai/api-reference/node-js/realtime/azure TypeScript API reference for the azure realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AzureVoiceLive Azure Voice Live realtime model implementation ```ts theme={null} import { AzureVoiceLive } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Gemini Source: https://docs.zeroruntime.ai/api-reference/node-js/realtime/gemini TypeScript API reference for the gemini realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GeminiRealtime Create a Realtime instance configured for Google Gemini. ```ts theme={null} import { GeminiRealtime } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { GeminiRealtime } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Openai Source: https://docs.zeroruntime.ai/api-reference/node-js/realtime/openai TypeScript API reference for the openai realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAIRealtime OpenAI's realtime model implementation. ```ts theme={null} import { OpenAIRealtime } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Ultravox Source: https://docs.zeroruntime.ai/api-reference/node-js/realtime/ultravox TypeScript API reference for the ultravox realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## UltravoxRealtime Ultravox's realtime model for audio-only communication ```ts theme={null} import { UltravoxRealtime } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Xai Source: https://docs.zeroruntime.ai/api-reference/node-js/realtime/xai TypeScript API reference for the xai realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## XAIRealtime xAI's Grok realtime model implementation ```ts theme={null} import { XAIRealtime } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Assemblyai Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/assemblyai TypeScript API reference for the assemblyai speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AssemblyAISTT Create an STT instance configured for AssemblyAI Universal Streaming. ```ts theme={null} import { AssemblyAISTT } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { AssemblyAISTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Azure Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/azure TypeScript API reference for the azure speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AzureSTT Initialize the Azure STT plugin. ```ts theme={null} import { AzureSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Cartesia Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/cartesia TypeScript API reference for the cartesia speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CartesiaSTT Create an STT instance configured for Cartesia (Ink) realtime STT. ```ts theme={null} import { CartesiaSTT } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { CartesiaSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Cometapi Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/cometapi TypeScript API reference for the cometapi speech-to-text provider. ## CometAPISTT Initialize the CometAPI STT plugin. ```ts theme={null} import { CometAPISTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Deepgram Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/deepgram TypeScript API reference for the deepgram speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## DeepgramSTT Create an STT instance configured for Deepgram. ```ts theme={null} import { DeepgramSTT } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { DeepgramSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options *** ## DeepgramSTTV2 Initialize the Deepgram STT plugin (Flux / v2 API). ```ts theme={null} import { DeepgramSTTV2 } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Elevenlabs Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/elevenlabs TypeScript API reference for the elevenlabs speech-to-text provider. ## ElevenLabsSTT ElevenLabs Realtime Speech-to-Text (STT) client. ```ts theme={null} import { ElevenLabsSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Gladia Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/gladia TypeScript API reference for the gladia speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GladiaSTT Initialize the Gladia STT plugin with WebSocket support. ```ts theme={null} import { GladiaSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Google Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/google TypeScript API reference for the google speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GoogleSTT Create an STT instance configured for Google Cloud Speech-to-Text. ```ts theme={null} import { GoogleSTT } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { GoogleSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Navana Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/navana TypeScript API reference for the navana speech-to-text provider. ## NavanaSTT VideoSDK Agent Framework STT plugin for Navana's Bodhi API. ```ts theme={null} import { NavanaSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Nvidia Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/nvidia TypeScript API reference for the nvidia speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## NvidiaSTT ```ts theme={null} import { NvidiaSTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Openai Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/openai TypeScript API reference for the openai speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAISTT Initialize the OpenAI STT plugin. ```ts theme={null} import { OpenAISTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Sarvamai Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/sarvamai TypeScript API reference for the sarvamai speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SarvamAISTT Create an STT instance configured for Sarvam AI. ```ts theme={null} import { SarvamAISTT } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { SarvamAISTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Xai Source: https://docs.zeroruntime.ai/api-reference/node-js/stt/xai TypeScript API reference for the xai speech-to-text provider. ## XAISTT Initialize the xAI STT plugin. ```ts theme={null} import { XAISTT } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Aws Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/aws TypeScript API reference for the aws text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AWSPollyTTS AWS Polly TTS implementation (plug-and-play for VideoSDK Agents). ```ts theme={null} import { AWSPollyTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Azure Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/azure TypeScript API reference for the azure text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AzureTTS Initialize the Azure TTS plugin. ```ts theme={null} import { AzureTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Cambai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/cambai TypeScript API reference for the cambai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CambAITTS CambAI Text-to-Speech plugin for VideoSDK agents. ```ts theme={null} import { CambAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Cartesia Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/cartesia TypeScript API reference for the cartesia text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CartesiaTTS Create a TTS instance configured for Cartesia. ```ts theme={null} import { CartesiaTTS } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Cometapi Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/cometapi TypeScript API reference for the cometapi text-to-speech provider. ## CometAPITTS Initialize the CometAPI TTS plugin. ```ts theme={null} import { CometAPITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Deepgram Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/deepgram TypeScript API reference for the deepgram text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## DeepgramTTS Create a TTS instance configured for Deepgram Aura. ```ts theme={null} import { DeepgramTTS } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { DeepgramTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Elevenlabs Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/elevenlabs TypeScript API reference for the elevenlabs text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## ElevenLabsTTS Initialize the ElevenLabs TTS plugin. ```ts theme={null} import { ElevenLabsTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Google Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/google TypeScript API reference for the google text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GoogleTTS Create a TTS instance configured for Google Text-to-Speech. ```ts theme={null} import { GoogleTTS } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { GoogleTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Groq Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/groq TypeScript API reference for the groq text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GroqTTS Initialize the Groq TTS plugin. ```ts theme={null} import { GroqTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Humeai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/humeai TypeScript API reference for the humeai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## HumeAITTS Initialize the HumeAI TTS plugin. ```ts theme={null} import { HumeAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Inworldai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/inworldai TypeScript API reference for the inworldai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## InworldAITTS Inworld AI Text-to-Speech plugin. ```ts theme={null} import { InworldAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Lmnt Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/lmnt TypeScript API reference for the lmnt text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## LMNTTTS Initialize the LMNT TTS plugin (WebSocket streaming). ```ts theme={null} import { LMNTTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Murfai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/murfai TypeScript API reference for the murfai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## MurfAITTS Initialize the Murf.ai TTS plugin. ```ts theme={null} import { MurfAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Neuphonic Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/neuphonic TypeScript API reference for the neuphonic text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## NeuphonicTTS Initialize the Neuphonic TTS plugin. ```ts theme={null} import { NeuphonicTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Nvidia Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/nvidia TypeScript API reference for the nvidia text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## NvidiaTTS ```ts theme={null} import { NvidiaTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Openai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/openai TypeScript API reference for the openai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAITTS Initialize the OpenAI TTS plugin. ```ts theme={null} import { OpenAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Papla Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/papla TypeScript API reference for the papla text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## PaplaTTS Initialize the Papla TTS plugin. ```ts theme={null} import { PaplaTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Resemble Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/resemble TypeScript API reference for the resemble text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## ResembleTTS Initialize the Resemble TTS plugin. ```ts theme={null} import { ResembleTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Rime Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/rime TypeScript API reference for the rime text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## RimeTTS Initialize the Rime TTS plugin. ```ts theme={null} import { RimeTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Sarvamai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/sarvamai TypeScript API reference for the sarvamai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SarvamAITTS Create a TTS instance configured for Sarvam AI. ```ts theme={null} import { SarvamAITTS } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key import { SarvamAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Smallestai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/smallestai TypeScript API reference for the smallestai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SmallestAITTS Initialize the SmallestAI TTS plugin. ```ts theme={null} import { SmallestAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Speechify Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/speechify TypeScript API reference for the speechify text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SpeechifyTTS Initialize the Speechify TTS plugin. ```ts theme={null} import { SpeechifyTTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Xai Source: https://docs.zeroruntime.ai/api-reference/node-js/tts/xai TypeScript API reference for the xai text-to-speech provider. ## XAITTS Initialize the xAI TTS plugin. ```ts theme={null} import { XAITTS } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Turn Detector Source: https://docs.zeroruntime.ai/api-reference/node-js/turn-detection/namo TypeScript API reference for the unified turn detector. Setup, environment variables, and Python/JavaScript/Go usage examples. ## TurnDetector End-of-turn detection through the ZeroRuntime inference gateway. ```ts theme={null} import { TurnDetector } from '@zeroruntime/js-sdk/inference'; // same fields, through the gateway, no vendor key ``` ### Options # Silero Source: https://docs.zeroruntime.ai/api-reference/node-js/vad/silero TypeScript API reference for the silero voice activity detection provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SileroVAD Silero Voice Activity Detection with advanced streaming features. ```ts theme={null} import { SileroVAD } from '@zeroruntime/js-sdk/plugins'; // direct to the vendor, with your key ``` ### Options # Anam Source: https://docs.zeroruntime.ai/api-reference/python/avatar/anam Python API reference for the anam avatars provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AnamAvatar Initialize the Anam Avatar plugin. Slot: avatar. Route: plugins -- direct to the vendor, with ANAM\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import AnamAvatar # direct to the vendor, with your key ``` ### Fields # Simli Source: https://docs.zeroruntime.ai/api-reference/python/avatar/simli Python API reference for the simli avatars provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SimliAvatar Initialize the Simli Avatar plugin. Slot: avatar. Route: plugins -- direct to the vendor, with SIMLI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import SimliAvatar # direct to the vendor, with your key ``` ### Fields # Agent & Session Source: https://docs.zeroruntime.ai/api-reference/python/core/agent-and-session Python API reference for Agent & Session. The core classes that define an agent and drive a live call: the `Agent` you subclass to give it instructions and tools, and the `AgentSession` that runs it, speaking, listening, and managing the call lifecycle. `serve` and `invoke` register an agent and start sessions against it. ## Agent What the agent is, what it can do, and how it behaves on a call. Subclass it to give the agent behaviour: methods decorated with `@function_tool` are registered by the constructor, and the `on_*` hooks below are called by the session as the call progresses. An instance is inert until `zeroruntime.serve` registers it and a session is started against it. The pipeline is the other half. This class carries the prompt, the tools and the call-shaped options; `Pipeline` carries the providers that hear and speak. ### Constructor ```python theme={null} Agent(instructions: 'str', name: 'Optional[str]' = None, pipeline: 'Any' = None, tools: 'Optional[List[FunctionTool]]' = None, agent_id: 'Optional[str]' = None, mcp_servers: 'Optional[list]' = None, inherit_context: 'bool' = False, greeting: 'Optional[str]' = None, farewell: 'Optional[str]' = None, wake_up: 'int' = 0, wake_up_message: 'str' = '', call_summary: 'Optional[CallSummary]' = None, agents: "Optional[List['Agent']]" = None, max_session_duration_seconds: 'Optional[int]' = None, tool_timeout_seconds: 'Optional[int]' = None) -> 'None' ``` The system prompt. Persona, task, and the rules the model is expected to hold to. Display name. Defaults to `agent_id`. The `Pipeline` this agent runs on. `@function_tool` callables defined elsewhere. Tools defined as methods on the agent are found on their own, so this is for the ones that are not. Required. The name `zeroruntime.serve` registers under and the runtime routes sessions to. `MCPServerStdio` and `MCPServerHTTP` whose tools join the agent's own. Connected on the first call to `initialize_mcp`, not here. Carry the chat history over when another agent hands off to this one. Off means it starts the conversation fresh. Spoken as soon as the agent joins, before the caller says anything. Spoken on the way out, when the agent ends the call itself. Seconds of caller silence before the agent nudges. 0 disables it; negative is rejected. What the nudge says. `CallSummary` -- summarise the conversation at teardown and POST it somewhere. Other agents this one hands off to, kept on `alternates`. Nothing registers them for you: `zeroruntime.serve` registers one `agent_id` per call, and `Session.add_handoff` names the target by that id. Hard ceiling on the call. The runtime ends the session when it is reached. How long the runtime waits for one of this agent's tools to return before handing the model a timeout. `None` keeps the runtime's own default of 30 seconds. Raise it only for an agent whose tools genuinely run long. The case it exists for is a tool that awaits `Session.warm_transfer` -- that runs for `supervisor_join_timeout + briefing_timeout`, minutes rather than seconds, and the default would kill the round trip while the transfer carried on underneath. The cost is symmetric: a tool that hangs holds the turn for exactly this long. ### register\_tools ```python theme={null} def register_tools(self) -> 'None' ``` Collect `@function_tool` methods off `self`. Called by the constructor. Call it again after adding one at runtime. ### update\_tools ```python theme={null} def update_tools(self, tools: 'List[FunctionTool]') -> 'None' ``` Replace the tool list outright. This drops the tools that were found on the agent and anything MCP added, so pass the full set you want. ### on\_enter ```python theme={null} async def on_enter(self) -> 'None' ``` Called once the agent has joined and the session is live. Override it to open the conversation, prime the context, or set up standing configuration such as `Session.set_thinking_audio`. ### on\_exit ```python theme={null} async def on_exit(self) -> 'None' ``` Called as the call is ending, before teardown finishes. ### on\_participant\_joined ```python theme={null} async def on_participant_joined(self, participant: "'Participant'") -> 'None' ``` Called when somebody joins the room the agent is in. ### on\_participant\_left ```python theme={null} async def on_participant_left(self, participant: "'Participant'") -> 'None' ``` Called when somebody leaves. The agent is not one of them. ### hangup ```python theme={null} async def hangup(self, reason: 'str' = 'agent ended the call', farewell: 'str' = '') -> 'None' ``` End the call from the agent's side. A no-op with no live session. Recorded against the session. Spoken before hanging up. Empty uses the agent's own `farewell`. ### cleanup ```python theme={null} async def cleanup(self) -> 'None' ``` Close the MCP connections and drop the tools they contributed. Called for you at teardown. Safe to call twice, and safe with no MCP servers configured. ### initialize\_mcp ```python theme={null} async def initialize_mcp(self) -> 'None' ``` Connect every configured MCP server and adopt its tools. Runs once; later calls return immediately. `zeroruntime.serve` calls it while building the agent, so a server that is down fails the build rather than the first tool call. ### add\_server ```python theme={null} async def add_server(self, mcp_server: 'Any') -> 'None' ``` Connect one MCP server and append what it publishes to `tools`. Ignores anything that is not an `MCPServerStdio` or `MCPServerHTTP`. The server to connect. ### on ```python theme={null} def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]' ``` Subscribe to an event. Decorator or direct call. Handlers fire in the order they were registered, and registering the same one twice calls it twice. The event name. The handler. Omitted, this returns a decorator. The decorator, or the handler it registered. ### off ```python theme={null} def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None' ``` Unsubscribe one handler. Matched by identity, so it has to be the same object that was registered -- a fresh lambda or a re-decorated function will not match. Removing something that was never subscribed does nothing. ### emit ```python theme={null} def emit(self, event: 'T', *args: 'Any') -> 'None' ``` Call every handler subscribed to `event`. Returns as soon as the handlers have been started, not when they finish, and never raises: an exception inside a handler is logged against the event name. Ignored once the emitter has been closed. The event name. *** ## AgentContext What the runtime knows about a job, handed to the agent factory. `zeroruntime.serve` passes one to your factory if it takes an argument, so the agent can be built around the call it is about to take -- a caller's number out of `metadata`, a different prompt per room. ### Fields The runtime's id for this dispatch. The room the agent is joining. Which registration the job was routed to. The auth token for this job. Kept out of `repr` so it does not end up in a log line. Whatever the dispatcher attached -- the caller's number, a tenant id, anything the job was started with. The `Room` the runtime resolved for this job. *** ## CallSummary Summarise the conversation when it ends, and POST the result. ### Fields Set False to carry the configuration without summarising. Where the summary is POSTed. Without one it is generated and logged but sent nowhere. Sent with the POST -- an auth header, usually. Replaces the built-in summarising prompt. A second model for the summary alone. `None` reuses the session's. Worth pointing at something cheap: it runs once, over the whole transcript, while the process is already shutting down. Seconds the summariser gets. 0 takes the runtime's default. Summarising is an LLM round trip over the whole conversation, so it is a different order of work from the rest of teardown. The ceiling is the runtime's SIGTERM grace, since this runs while the agent is shutting down -- asking for more than that gets the process killed mid-request rather than granting the time. *** ## serve ```python theme={null} def serve(agent: 'Any', *, room: 'Any' = None, on_ready: 'Optional[Callable[[], None]]' = None, capacity: 'int' = 10, load_threshold: 'float' = 0.75, initialize_timeout: 'float' = 10.0, log_level: 'str' = 'INFO', host: 'str' = '0.0.0.0', debug: 'bool' = True, debug_port: 'int' = 8081, audio_listener_enabled: 'bool' = False, avatar: 'Any' = None, **moved_to_room: 'Any') -> 'None' ``` Register `agent` and serve every call dispatched to it. Every keyword left here describes *this worker* -- how many calls it takes, what it logs, where it answers probes. Nothing here describes the session: that is Room's job, and the twelve keywords which used to say it a second time are gone, because two spellings of `recording` that could disagree is a question with no right answer. Blocks until interrupted. The agent is registered with the runtime's registry, and every call dispatched to that id builds a fresh agent from the factory -- one instance per call, never one shared across them. A callable returning an `Agent`. Your subclass is usually the callable. An instance is rejected: concurrent calls would share one conversation. The Room every dispatched call is given. All 21 fields are reachable, including `observability`, `join_meeting`, `wait_for_participant` and `idle_timeout_seconds`, which no keyword ever covered. A dispatched call never passes through `zeroruntime.invoke`, so this is the only place its session is described; `invoke(room=...)` inherits from it and overrides only the fields it names. Called on its own thread once the agent is registered. Where `zeroruntime.invoke` belongs, for a process that starts its own calls rather than waiting to be dispatched to. Concurrent sessions this worker accepts. The `ZERORUNTIME_MAX_CONCURRENT_SESSIONS` environment variable wins when it is set to a number. Fraction of capacity above which the worker reports itself loaded and the registry prefers another. Inert. Warned about and ignored. Applied to the `zeroruntime` logger, and installs colored logging if nothing else has configured the root logger. Interface the status server binds. Set False to skip the status server entirely. Port for the status server. Inert. Warned about and ignored. Inert -- the avatar is a pipeline slot. Put it on `Pipeline(avatar=...)`. *** ## invoke ```python theme={null} def invoke(agent_id: 'str', *, room: 'Optional[Any]' = None, sip: 'Optional[Any]' = None, labels: 'Optional[Dict[str, str]]' = None, metadata: 'Optional[Dict[str, Any]]' = None, recording_config: 'Optional[Dict[str, Any]]' = None, session_id: 'Optional[str]' = None, runtime_address: 'Optional[str]' = None, timeout: 'float' = 30.0) -> 'Dict[str, str]' ``` Start a session against an agent this process is already serving. Blocking, and deliberately so -- it returns once the runtime has accepted the session, which is the point at which the room is joinable. That also means it cannot be called from inside a running event loop: call it from `serve(on_ready=...)`, which runs on its own thread, or await `ZeroRuntimeChannel.start` from async code. The agent has to be registered in *this* process. There is no dispatch RPC to route a start to another one: the pipeline, credentials and tool schemas all live where `zeroruntime.serve` was called. The id passed to `zeroruntime.serve`. A `Room` layered over the one `serve()` was given -- name only what differs. Without `room_id` a room is created. A `Sip` leg to dial out on, for an outbound call. Not applied. Warned about and ignored. Attached to the job, and readable from `AgentContext`. Only `enabled` travels; anything else is warned about and dropped. Not applied. Warned about and ignored. Start against a different runtime than the one `serve()` is connected to. The channel is then owned by this call and closed when the session ends. Seconds to wait for the runtime to accept. `session_id`, `room_id` and `worker_id`, plus `playground_url` when the room asked for one. *** ## Session One live call, and everything you can do to it while it runs. You are handed one rather than constructing one: `zeroruntime.invoke` returns it, and inside a `@function_tool` or an `on_*` hook it is the session the agent is bound to. Every method here is a message to the agent process, so they are all awaited and none of them block the call. The object stays usable after the call ends -- `ended` goes True and the send methods become no-ops -- so a hook can read `ended_reason` without guarding first. ### Constructor ```python theme={null} Session(stream: 'Any', accepted: "'pb.SessionAccepted'", outbound: "'asyncio.Queue'", *, stub: 'Any' = None, auth_token: 'Optional[str]' = None, tools: 'Any' = None) -> 'None' ``` ### log\_playground ```python theme={null} def log_playground(self) -> 'None' ``` Announce the join URL on stdout, at most once per session. Both the direct-start path and the registry worker path call this, and invoke() adds a third; the guard is what keeps one session to one URL. It prints rather than logs because the URL is the one line the operator is waiting to copy, and a log record buries it behind level and module columns that are noise for a value you paste into a browser. ### events ```python theme={null} async def events(self) -> 'AsyncIterator[Event]' ``` Iterate everything the runtime reports, until the call ends. Keeps yielding briefly past the `ended` event so the last transcripts and metrics are not cut off mid-teardown. One `Event` per server message. ### wait ```python theme={null} async def wait(self, *, log: 'bool' = True) -> 'None' ``` Consume `events` until the call ends, logging as it goes. The usual last line of a script that started a session and has nothing else to do. Set False to drain silently. State frames are never logged either way -- they arrive every few seconds and say nothing a reader wants. ### aclose ```python theme={null} async def aclose(self) -> 'None' ``` Release everything this call holds. Idempotent, and safe after any ending. Every way a call can finish routes here. Closing the transport is the part that cannot be skipped: the outbound queue feeds a generator gRPC holds inside a task, and a live task is a collection root. ### say ```python theme={null} async def say(self, text: 'str', *, interrupt: 'bool' = False, interruptible: 'Optional[bool]' = None, add_to_chat_context: 'Optional[bool]' = None, audio_data: 'Optional[bytes]' = None) -> "'UtteranceHandle'" ``` Speak an exact line. No model involved. What to say. Cut off whatever is playing instead of queueing behind it. Whether the caller can barge in over this line. `None` leaves the pipeline's own setting alone. Whether the line joins the conversation history. `None` defers to the runtime. Pre-rendered audio to play instead of sending `text` through TTS. `text` is still what the transcript records. A handle for this utterance -- await it to know when it finished, or read it to find out it was interrupted. ### reply ```python theme={null} async def reply(self, instructions: 'str', *, frames: 'int' = 0, interruptible: 'Optional[bool]' = None, wait_for_playback: 'Optional[bool]' = None) -> "'UtteranceHandle'" ``` Ask the model for a line, generated against `instructions`. Unlike `say`, this is a generation: the model sees the conversation so far plus these instructions, and what it produces is spoken. What to tell the model for this turn alone. How many of the newest camera frames to show the model. The count travels, not the pixels -- they are captured in the agent process. Needs `Room(vision=True)`; without it there is no video track to capture from. Whether the caller can barge in over the answer. Resolve the handle only once the audio has finished playing, rather than when generation completes. A handle for the utterance this produces. ### get\_context\_history ```python theme={null} async def get_context_history(self, *, limit: 'int' = 0, timeout: 'float' = 10.0) -> 'list' ``` Fetch the conversation as the agent process holds it. Newest N messages. 0 is all of them. Seconds to wait for the answer. The messages, oldest first. ### get\_metrics ```python theme={null} async def get_metrics(self, *, timeout: 'float' = 10.0) -> 'list' ``` Fetch the per-turn latency metrics collected so far. ### change\_pipeline ```python theme={null} async def change_pipeline(self, pipeline: 'Any', *, instructions: 'str' = '', timeout: 'float' = 30.0) -> 'str' ``` Replace the whole pipeline mid-call. Everything not named is rebuilt from the pipeline you pass, so state a complete one. To change a provider or two and leave the rest alone, use `change_component`. The `Pipeline` to switch to. New system prompt to apply with the swap. Empty keeps the current one. Seconds to wait for the runtime to confirm. The mode the session is running in after the swap. ### change\_component ```python theme={null} async def change_component(self, *, stt: 'Any' = NO_CHANGE, llm: 'Any' = NO_CHANGE, tts: 'Any' = NO_CHANGE, vad: 'Any' = NO_CHANGE, turn_detector: 'Any' = NO_CHANGE, denoise: 'Any' = NO_CHANGE, instructions: 'str' = '', timeout: 'float' = 30.0) -> 'str' ``` Swap the named components and leave the rest of the call alone. ### play\_background\_audio ```python theme={null} async def play_background_audio(self, file: 'Any' = None, *, volume: 'float' = 1.0, looping: 'bool' = False, override_thinking: 'Optional[bool]' = None) -> 'None' ``` Start a sound bed under the call. Needs the room's mixing track, `Room(background_audio=True)`. A path, or a `BackgroundAudio` carrying one along with its volume and looping. A disabled `BackgroundAudio` plays nothing. Gain, when `file` is a path. Restart at the end, when `file` is a path. Let this bed play over the thinking sound rather than yielding to it. ### set\_thinking\_audio ```python theme={null} async def set_thinking_audio(self, file: 'Optional[str]' = None, *, volume: 'float' = 0.3) -> 'None' ``` Play a sound while the agent is generating its reply. Standing configuration rather than a one-shot: once set, the session starts it at the top of every generation and stops it when the answer begins, for as long as the session lasts. Set it from `on_enter`. `file=None` takes the runtime's own thinking sound. Any file libav can decode works -- wav, mp3, ogg, flac, m4a. Needs the room's mixing track, `Room(background_audio=True)`, like all audio that is not speech. Without it the SDK declines and says so in the runtime's log only, so this reports it back as a diagnostic instead. ### stop\_background\_audio ```python theme={null} async def stop_background_audio(self) -> 'None' ``` Stop the sound bed. Leaves the thinking sound alone. ### subscribe\_to\_pubsub ```python theme={null} async def subscribe_to_pubsub(self, pubsub_config: 'PubSubSubscribeConfig') -> 'None' ``` Listen on one pubsub topic for the rest of the call. The topic and its handler. ### publish\_to\_pubsub ```python theme={null} async def publish_to_pubsub(self, pubsub_config: 'PubSubPublishConfig') -> 'None' ``` Publish one pubsub frame into the room. The topic, body and options to publish with. ### add\_message ```python theme={null} async def add_message(self, role: 'str', content: 'str', *, agent_id: 'str' = '', replace: 'bool' = False) -> 'None' ``` Write a message into the conversation without speaking it. Useful for handing the model something it should know -- a CRM lookup, a verification result -- without a turn being spent saying it out loud. `system`, `developer`, `user` or `assistant`. A `ChatRole` works too. The message body. Whose context to write into, when the call has more than one agent. Empty means the one running. Replace the previous message of this role instead of appending -- how a rolling system prompt is kept from growing. ### add\_handoff ```python theme={null} async def add_handoff(self, to_agent: 'str', *, from_agent: 'str' = '', reason: 'str' = '') -> 'None' ``` Hand the conversation to another agent, by id. The target has to be registered -- `zeroruntime.serve` registers one `agent_id` per call -- and whether it inherits the history is that agent's `inherit_context`. The `agent_id` taking over. Who is handing off. Empty means the agent running. Recorded with the handoff, and shown in traces. ### process\_text ```python theme={null} async def process_text(self, text: 'str') -> 'None' ``` Feed text to the agent as though the caller had said it. The full turn runs: the model answers and the answer is spoken. ### get\_participants ```python theme={null} async def get_participants(self, *, timeout: 'float' = 10.0) -> 'list' ``` Who is in the room right now, as `Participant` records. ### transfer\_call ```python theme={null} async def transfer_call(self, transfer_to: 'str', *, timeout: 'float' = 30.0) -> 'dict' ``` Cold-transfer the call and leave. The caller is handed to `transfer_to` with no introduction and the agent drops out. Where to send them -- a SIP URI or a phone number. Seconds to wait for the runtime to report the outcome. What the runtime reported about the transfer. ### interrupt ```python theme={null} async def interrupt(self, *, force: 'bool' = False) -> 'None' ``` Stop the agent talking. Cut off even an utterance that was marked non-interruptible. ### end ```python theme={null} async def end(self, reason: 'str' = 'client ended the session', farewell: 'str' = '') -> 'None' ``` Ask the runtime to end the call. A no-op once it has ended. Recorded against the session. Spoken before hanging up. ### stop ```python theme={null} async def stop(self, reason: 'str' = 'client requested stop') -> 'None' ``` End the call and release it. What `async with` calls. ### detach ```python theme={null} async def detach(self) -> 'None' ``` Stop watching this session without ending the call. The agent keeps running runtime-side; this process just stops listening. For a call that should actually stop, use `end` or `destroy`. ### destroy ```python theme={null} async def destroy(self, reason: 'str' = 'client requested destroy') -> 'None' ``` Tear the session down at the runtime, without a goodbye. The blunt one: no farewell, no teardown hooks worth waiting for. `end` is the polite version. ### on ```python theme={null} def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]' ``` Subscribe to an event. Decorator or direct call. Handlers fire in the order they were registered, and registering the same one twice calls it twice. The event name. The handler. Omitted, this returns a decorator. The decorator, or the handler it registered. ### off ```python theme={null} def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None' ``` Unsubscribe one handler. Matched by identity, so it has to be the same object that was registered -- a fresh lambda or a re-decorated function will not match. Removing something that was never subscribed does nothing. ### emit ```python theme={null} def emit(self, event: 'T', *args: 'Any') -> 'None' ``` Call every handler subscribed to `event`. Returns as soon as the handlers have been started, not when they finish, and never raises: an exception inside a handler is logged against the event name. Ignored once the emitter has been closed. The event name. *** ## current\_session ```python theme={null} def current_session() -> "'Session'" ``` The session whose callback is currently running. Valid anywhere the SDK invokes your code on a session's behalf: a `@function_tool`, a lifecycle hook, a pipeline hook. Useful for code that runs on the call without holding a reference to it -- a module-level `@pipeline.on` handler, say. Anything defined on the agent should reach the call through `self.session` instead, and a pubsub handler subscribed with `Session.subscribe_to_pubsub` already has the session in hand. *** ## Participant Somebody in the room with the agent. ### Fields The room's id for this peer. Display name, which may be empty. The peer's send/receive mode, as the transport labels it. Whatever metadata the peer joined with. *** ## PubSubSubscribeConfig One topic to listen on, and what to call for each frame. Passed to `Session.subscribe_to_pubsub`. ### Fields The topic to subscribe to. Called once per frame with the raw frame as a dict -- the whole thing as the transport delivered it, since pubsub promises no schema beyond `message`. Sync or async; a coroutine is awaited. A handler that declares a second parameter is additionally told whether the frame is backlog:: def cb(frame, backlog): if backlog: return # already in the topic before we joined Subscribing replays whatever the topic already held, so a one-argument handler sees that history as ordinary traffic -- which is the transport's own shape, and worth knowing about before replying to it. *** ## PubSubPublishConfig One frame to publish into the room. Passed to `Session.publish_to_pubsub`. ### Fields The topic to publish on. The text body. Structured data alongside it, JSON-encoded on the way out. Transport options. Only `sendOnly` -- a list of participant ids to deliver to -- is carried; anything else is dropped, because the wire has nowhere to put it. Omitted, everyone subscribed to the topic gets the frame. *** ## UtteranceHandle One thing the agent is saying, and how it went. Returned by `Session.say` and `Session.reply`. Await it to block until the line finishes -- awaiting resolves whether it played out or was cut off, so check `interrupted` rather than assuming it was heard. ```python theme={null} handle = await session.say("One moment while I look that up.") await handle if handle.interrupted: ... ``` Awaiting is optional. Ignore the handle and the line still plays. ### Constructor ```python theme={null} UtteranceHandle(utterance_id: 'str') -> 'None' ``` ### done ```python theme={null} def done(self) -> 'bool' ``` Whether the utterance has settled, either way. Never blocks. *** ## Room Which room the agent joins. `room_id=None` asks for a new one. ### Fields Existing room to join. A new room is created when `None`. Display name the agent publishes under. The 0.1.2 spelling of `name`. Folded into it when set. VideoSDK token for the room. Falls back to the environment. It travels in `StartSession.credentials`, never in `params_json`. Ask the runtime for a playground URL for this session. Deliver camera frames to the agent process. Codec the room negotiates -- `"opus"` (the default when `None`), `"pcmu"`, `"pcma"`, `"g722"`. This is what tells a meeting from a phone call: opus is 48 kHz, pcmu/pcma are 8 kHz and g722 is 16 kHz, and that rate reaches the VAD and STT. Leave it `None` for meetings; set `"pcmu"` or `"pcma"` for SIP so the pipeline sizes itself to the narrowband audio instead of resampling it up to 48 kHz and back down again. Record the session. Off unless asked for -- recording a call is a decision about the person on the other end, so the default has to be the safe one and the same runtime serves recorded and unrecorded sessions side by side. Audio is always captured when this is on; the two below add tracks to it. Also record camera video. Needs `recording`. Also record the screen share. Needs `recording` and `vision` -- there is no screen-share track on a session that is not receiving video. Publish the agent under a fixed participant id. `None` lets the room mint one, which is what a single-agent session wants. End the session when the last participant leaves. `None` leaves the runtime's default (TRUE) in place. Open the room's mixing audio track. Nothing plays by itself -- this is the track, not the sound. Start audio on it with `session.play_background_audio(file=..., volume=..., looping=...)` and stop it with `session.stop_background_audio()`. Required before either call: without the track the SDK refuses, and the failure is a silent one from the caller's side. Whether the agent joins the meeting. `None` leaves the runtime's default (TRUE) in place -- see `to_proto`. Whether the agent waits for someone to join before speaking. `None` leaves the runtime's default (TRUE) in place. Override the signaling base URL. Where this session's telemetry goes. `None` takes the runtime's defaults. Hard cap on session length, in seconds. Carried in `SessionLimits.max_session_duration_seconds` rather than in `RoomSpec`, so it is read by the transport and not by `to_proto`. `None` inherits the runtime's own ceiling. Seconds with nobody attached before the session is reclaimed. Carried in `SessionLimits.inactivity_timeout_seconds`, same as above. *** ## Sip The telephony leg of a session -- who is being called, and from what. ### Fields Destination number/address for an outbound call. Caller id presented on the call. Call direction/type. Identifier correlating the call. URL to receive call-event callbacks. Additional key/values merged into the metadata map. ### to\_metadata ```python theme={null} def to_metadata(self) -> 'Dict[str, str]' ``` This leg as the metadata dict the room is created with. *** ## UNSET ```python theme={null} UNSET = UNSET ``` "You did not pass this." Distinct from every real value, including False. `serve(recording=False)` and leaving `recording` out have to mean different things once a Room template exists -- the first overrides the template, the second defers to it -- and no ordinary default can express that, because the value a caller most often means to override *with* is the default itself. *** ## ZeroRuntimeChannel A connection to one runtime, and the sessions started over it. `zeroruntime.serve` and `zeroruntime.invoke` own one for you. Hold one directly to start sessions from async code, to reach a runtime other than the configured one, or to ask a runtime what capacity it has. Connect explicitly with `await channel.connect()`, or use it as an async context manager -- every method connects on first use anyway. TLS is decided by the target unless `secure` says otherwise: local hosts are plaintext, port 443 is TLS. `ZERORUNTIME_INSECURE=1` forces plaintext. ### Constructor ```python theme={null} ZeroRuntimeChannel(target: 'Optional[str]' = None, *, auth_token: 'Optional[str]' = None, secure: 'Optional[bool]' = None, options: 'Optional[list]' = None) -> 'None' ``` ### use\_tls ```python theme={null} def use_tls(self) -> 'bool' ``` Whether this channel will speak TLS. `secure` wins if it was set, then `ZERORUNTIME_INSECURE`, then the target itself: a local host is plaintext and port 443 is TLS. ### connect ```python theme={null} async def connect(self, *, timeout: 'float' = 5.0) -> "'ZeroRuntimeChannel'" ``` Open the connection. Returns immediately if it is already open. Seconds to wait for the channel to become ready. 0 returns without waiting, and the first RPC finds out instead. This channel, so it can be chained. ### aclose ```python theme={null} async def aclose(self) -> 'None' ``` Close the connection. Sessions started over it stop being watched. ### status ```python theme={null} async def status(self) -> 'ZeroRuntimeStatus' ``` Ask the runtime what capacity it has. Its `ZeroRuntimeStatus`. ### session\_info ```python theme={null} async def session_info(self, session_id: 'str') -> 'Any' ``` Look up one session's state, pid, room and uptime. Works for any session on this runtime, including ones this process did not start. ### destroy ```python theme={null} async def destroy(self, session_id: 'str', reason: 'str' = 'client requested destroy') -> 'Any' ``` Tear a session down at the runtime, by id. The session to destroy. Recorded against it. The runtime's response. ### attach ```python theme={null} async def attach(self, session_id: 'str', *, tools: 'Any' = None, heartbeat_interval_ms: 'int' = 0) -> 'Session' ``` Attach to a session that is already running. The session to attach to. Tools to answer this session's tool calls with. How often to ping. The attached `Session`. ### start ```python theme={null} async def start(self, *, agent: 'Any' = None, pipeline: 'Any' = None, room: 'Any' = None, sip: 'Any' = None, tools: 'Any' = None, metadata: 'Optional[dict]' = None, agent_id: 'Optional[str]' = None, extra_credentials: 'Optional[dict]' = None, inactivity_timeout_s: 'Optional[float]' = None, max_duration_s: 'Optional[float]' = None) -> 'Session' ``` Start a session and return it, ready to run. The async counterpart to `zeroruntime.invoke`, and what to reach for inside async code, where `invoke` cannot be called. The `Agent` to run. Its pipeline and tools are used unless overridden here. The `Pipeline` to run, when there is no agent to take one from. The `Room` to join. A `Sip` leg, for an outbound call. Folded into the metadata. Tools to expose. Defaults to the agent's. Attached to the session. Recorded in the metadata, so traces name the agent. Vendor keys beyond the ones the pipeline's providers name for themselves. Seconds without a participant before the runtime reclaims the session. Defaults to the room's setting. Hard ceiling on the call. Defaults to the room's, then the agent's. The live `Session`. # Audio Source: https://docs.zeroruntime.ai/api-reference/python/core/audio Python API reference for Audio. ## BackgroundAudio A sound bed the runtime plays into the room alongside the agent. Needs the room's mixing track, `Room(background_audio=True)`, like all audio that is not speech. ### Fields The file to play. Anything libav decodes -- wav, mp3, ogg, flac, m4a. Empty is rejected. Set False to keep the configuration and play nothing. `mixing` or `playback`. Only `mixing` has a counterpart on the runtime today; `playback` is accepted, warned about, and treated as `mixing`. Gain. 1.0 is the file as recorded; must not be negative. Restart when the file ends instead of falling silent. *** ## run\_stt ```python theme={null} async def run_stt(audio_stream: 'Any' = None) -> 'AsyncIterator[SpeechEvent]' ``` Yield the transcript an `stt` hook was called with. The shim that lets a hook written against a local STT run unchanged. STT itself runs in the agent process and only the transcript crosses, so the audio stream is empty here -- draining it is allowed and yields nothing. Filtering or rewriting the text works as written; transforming the audio does not. The hook's audio stream, if it takes one. Drained and discarded. The `SpeechEvent` for this call. Nothing, outside an `stt` hook. # Errors & Enums Source: https://docs.zeroruntime.ai/api-reference/python/core/errors-and-config Python API reference for Errors & Enums. Shared configuration and state types: session and user state enums, speech and VAD event types, and the config objects for end-of-utterance, interruption, and realtime behavior. ## ZeroRuntimeError Base for everything this SDK raises on its own behalf. Catch it to handle any ZeroRuntime failure without naming each one. Errors the standard library raises -- `ValueError` for a bad argument, `TimeoutError` for a request that went unanswered -- are left as themselves rather than wrapped. *** ## SessionRejected The ZeroRuntime refused to create the session. ### Constructor ```python theme={null} SessionRejected(message: 'str' = '', *, code: 'str' = '', current_sessions: 'int' = 0, max_sessions: 'int' = 0) -> 'None' ``` The ZeroRuntime's explanation. Machine-readable reason, `at_capacity` | `invalid_config` | `internal`. `at_capacity` is worth retrying elsewhere; `invalid_config` never is. Sessions the ZeroRuntime is already running. The ZeroRuntime's ceiling, when it reported one. *** ## ProviderUnavailable A provider in the pipeline could not be reached or would not serve. Usually a missing or rejected vendor key, or a model the account is not entitled to. A fallback on the same slot is what keeps this from ending the call. *** ## ZeroRuntimeUnreachable The runtime could not be reached, or did not answer in time. A transport failure rather than a refusal: nothing was decided about the session, so retrying is reasonable. *** ## ToolTimeout A `@function_tool` did not return before its deadline. The turn continues without the tool's answer. A tool that calls out to a slow API wants its own timeout, so it can say something useful instead of being cut off. *** ## AgentState Where the agent is in a turn -- listening, thinking, speaking, or on its way in or out of the call. ```python theme={null} class AgentState(Enum): STARTING = 'starting' IDLE = 'idle' SPEAKING = 'speaking' LISTENING = 'listening' THINKING = 'thinking' CLOSING = 'closing' ``` *** ## UserState What the caller is doing, as the runtime reports it. ```python theme={null} class UserState(Enum): IDLE = 'idle' SPEAKING = 'speaking' LISTENING = 'listening' ``` *** ## ChatRole Who a message in the conversation came from. A plain string works anywhere one of these is accepted -- it subclasses `str` -- so `"user"` and `ChatRole.USER` are interchangeable. ```python theme={null} class ChatRole(Enum): SYSTEM = 'system' USER = 'user' ASSISTANT = 'assistant' DEVELOPER = 'developer' TOOL = 'tool' ``` # Events & Observability Source: https://docs.zeroruntime.ai/api-reference/python/core/events Python API reference for Events & Observability. Event emission and observability: subscribe to session and pipeline events, and configure logging, metrics, and tracing for a running agent. ## Observability Where this session's traces, metrics and logs go. Each stream is configured on its own, and a stream left `None` follows the runtime's own settings rather than being switched off. ### Fields Per-turn spans -- what the model was asked, what each provider took. Latency and usage counters. The agent process's log records. Level the agent process logs at. One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. *** ## Exporter Where one stream of telemetry is shipped. ### Fields Set False to turn this stream off while leaving the rest of the configuration in place. An OTLP endpoint. `None` sends it wherever the runtime is configured to send its own. Sent with each export -- the collector's auth header, usually. # Pipeline Source: https://docs.zeroruntime.ai/api-reference/python/core/pipeline Python API reference for Pipeline. The voice stack an agent runs on: speech-to-text, LLM, and text-to-speech, plus voice-activity and turn detection. `Pipeline` wires these components together and exposes hooks for observing and shaping each turn. ## Pipeline The providers a call runs on: what hears, what thinks, what speaks. Two shapes are valid. A cascade names `stt`, `llm` and `tts` separately; a realtime pipeline names a speech-to-speech model in `llm` alone, because a realtime model is the language model. `realtime=` is accepted as the older spelling of the same slot; naming both is rejected. A fallback slot -- `stt`, `llm`, `tts` -- also takes a list, in which case the head serves and the tail stands by. Wrap that list in `FallbackSTT`, `FallbackLLM` or `FallbackTTS` to say how it demotes as well as what it falls back to; each slot carries its own settings. ### Fields Speech-to-text, or a `FallbackSTT` chain. Omit it on a realtime pipeline. The language model -- a text LLM for a cascade, a `FallbackLLM` chain of them, or a speech-to-speech model for a realtime pipeline. Text-to-speech, or a `FallbackTTS` chain. Omit it on a realtime pipeline. Voice activity detection -- what notices the caller is talking. End-of-turn detection, for deciding when they finished. Noise cancellation applied to the inbound audio. The older spelling of a speech-to-speech `llm`. A video avatar to render the agent's speech through. `EOUConfig` -- how patiently the agent waits before answering. `InterruptConfig` -- what counts as the caller barging in. `PronunciationRule` substitutions applied to generated text on its way to TTS. `ContextWindow` -- how a long conversation is kept inside the model's limit. `DTMFHandler` -- deliver keypad tones to the agent. `None` drops them. `VoiceMailDetector` -- answering-machine detection, for outbound calls that may be picked up by one. ### on ```python theme={null} def on(self, event: 'str', callback: 'Any' = None) -> 'Any' ``` Register a hook that runs in this process, mid-turn. Decorator or direct call. The events are the transcript hooks `stt` and `llm`, and the turn hooks `user_turn_start`, `user_turn_end`, `agent_turn_start` and `agent_turn_end`. An async generator registered on `llm` is filed as `llm_stream`: it is handed the model's output as a stream and yields what should be spoken, so it can buffer, rewrite or drop the turn. A plain coroutine on the same event sees the finished text instead. Per-component latency is a separate family, registered through its own namespace rather than by spelling the prefix here:: @pipeline.metrics.on("stt") def on\_stt(data: dict) -> None: ... Which hook to attach to. The handler. Omitted, this returns a decorator. The decorator, or the handler it registered. ### hooks ```python theme={null} def hooks(self, event: 'str') -> 'list' ``` The handlers registered for one event, in registration order. ### chain ```python theme={null} def chain(self, slot: 'str') -> 'tuple[ProviderSpec, ...]' ``` Everything configured for one slot: the primary, then its fallbacks. Empty for a slot nothing was named for, and for a slot that is not one. ### primary ```python theme={null} def primary(self, slot: 'str') -> 'ProviderSpec | None' ``` The provider a slot uses first, or `None` if it has none. ### fallbacks ```python theme={null} def fallbacks(self, slot: 'str') -> 'tuple[ProviderSpec, ...]' ``` What a slot falls back to, in the order they are tried. ### fallback\_settings ```python theme={null} def fallback_settings(self, slot: 'str') -> 'Optional[_Fallback]' ``` How one slot demotes, or `None` where nothing was tuned. `fallbacks` is who a slot falls back to; this is how. ### providers ```python theme={null} def providers(self) -> 'Iterable[ProviderSpec]' ``` Every provider in the pipeline, fallbacks included. *** ## NO\_CHANGE ```python theme={null} NO_CHANGE = NO_CHANGE ``` "Leave this slot as it is." Distinct from None, which is a real answer: `change_component(denoise=None)` takes denoising off a running call, while leaving `denoise` out keeps whatever the call is already denoising with. *** ## PronunciationRule Rewrite generated text on its way to TTS. ### Fields The text to look for. What to say instead. Match case exactly. Off by default, because the model capitalises the same word differently at the start of a sentence. *** ## ContextWindow Keep a long conversation inside the model's context, automatically. ### Fields Compress once the context exceeds this. `None` for no token ceiling. Or bound by item count. `None` for none. Recent user turns kept verbatim. The SDK's default is 3, and it is worth keeping several -- summarising the sentence the caller just said is how an agent starts answering the wrong question. The SDK's default is 10. The model that compresses. A second LLM, described here and built in the agent process like every other provider. *** ## EOUConfig When the caller is judged to have finished speaking. ### Fields `DEFAULT` waits a fixed time after speech stops. `ADAPTIVE` varies it with how certain the turn detector is -- longer when the sentence sounds unfinished. `[min, max]` seconds to wait before treating silence as the end of a turn. `[0.0, 0.0]` answers as soon as the detector says the turn is complete, which is snappy but cuts off anyone who pauses to think. How sure the detector must be. Lower answers sooner and interrupts more. Treat "mhm", "haan", "right" as listening noises rather than turns. `None` takes the runtime's default; the SDK's own default is on for detectors that support it. *** ## InterruptConfig What counts as the caller interrupting, rather than just making a noise. ### Fields `VAD_ONLY` reacts to any speech-like audio -- fastest, and the most easily fooled by a cough or a door. `STT_ONLY` waits for words. `HYBRID` uses both. Seconds of speech before it counts. Raising this is the usual fix for an agent that stops every time someone breathes. Words required before it counts. `2` ignores a stray "yeah". STT confidence floor for those words. How long to stay paused after an interruption that turned out to be nothing. Pick the sentence back up when the interruption proves false, rather than dropping it. Seconds to fade the agent's audio out over. An instant cut sounds like a dropped call. *** ## FallbackSTT Speech-to-text providers to try in order, and how they demote. ### Fields *** ## FallbackLLM Language models to try in order, and how they demote. ### Fields *** ## FallbackTTS Text-to-speech providers to try in order, and how they demote. ### Fields *** ## DTMFHandler Deliver the caller's keypad tones to the agent instead of dropping them. ### Fields What runs per keypress, taking `(key)` or `(key, payload)`. Left `None`, the agent's `on_dtmf` method is called instead -- which is the usual shape, because a keypad menu almost always wants the agent's own state to accumulate a multi-digit entry into. The runtime delivers one key per call either way; a PIN is accumulated on your side, not handed over whole. *** ## VoiceMailDetector Detect an answering machine on an outbound call. The detector runs in the agent process the ZeroRuntime starts: it buffers the opening speech for `duration` seconds and asks `llm` whether it is a person or a greeting. ### Fields The classifier. Required: without one there is nothing to classify with, and the detector is simply not installed. What runs on detection. Left `None`, the agent's `on_voicemail` method is called instead. Either is awaited, so anything said in it finishes before the call is ended. Seconds of speech to buffer before deciding. The default is 2.0 -- long enough for "Hi, you've reached...". Replace the built-in classification prompt. Set False to configure it without turning it on. *** ## PipelineMode Which shape a pipeline resolved to. Derived from the slots that were filled rather than set by hand; read it off `Pipeline.mode`. ```python theme={null} class PipelineMode(Enum): REALTIME = 'realtime' FULL_CASCADING = 'full_cascading' LLM_TTS_ONLY = 'llm_tts_only' STT_LLM_ONLY = 'stt_llm_only' LLM_ONLY = 'llm_only' STT_ONLY = 'stt_only' TTS_ONLY = 'tts_only' STT_TTS_ONLY = 'stt_tts_only' HYBRID = 'hybrid' PARTIAL_CASCADING = 'partial_cascading' ``` # Tools & MCP Source: https://docs.zeroruntime.ai/api-reference/python/core/tools-and-mcp Python API reference for Tools & MCP. Give an agent capabilities beyond conversation: define function tools it can call, connect MCP servers to expose external tools, and handle DTMF and voicemail on telephony calls. ## function\_tool ```python theme={null} def function_tool(func: 'Optional[Callable]' = None, *, name: 'Optional[str]' = None) ``` Expose a function to the model as a callable tool. Works bare or with arguments. The name, description and parameter schema are read off the function itself -- its `__name__`, the prose above its `Args:` block, and its type hints -- so a well-written docstring is what the model sees. ```python theme={null} @function_tool async def lookup_order(order_id: str) -> dict: '''Find an order by its id. Args: order_id: The customer's order number. ''' ``` Methods on an `Agent` subclass are registered automatically; tools defined elsewhere go in `Agent(tools=[...])`. The function, when used bare. Override the exposed name. Everything else the model is told comes off the function itself, so this is the only knob. The function, marked as a tool. *** ## MCPServerStdio A local MCP server, run as a subprocess and spoken to over stdio. ### Fields The program to run, e.g. `sys.executable`. Its arguments -- typically the server script. Environment for the child. `None` inherits this process's, which is usually what you want: the server needs the same API keys you already have. Directory to run it in. Seconds to wait for the connection and for each tool call. The vendor's default is 5, which is tight for a server that starts an interpreter; its own examples pass 30. Always `stdio`. Set for you -- it is what tells the SDK which transport to connect with. *** ## MCPServerHTTP A remote MCP server, reached over HTTP. ### Fields The server's URL. Sent with every request -- an API key usually lives here. Read in this process, so the credential stays with you. Seconds to wait for the connection. Seconds a quiet stream may stay open. Long by default: an MCP server that has nothing to say is not broken. Seconds to wait for the session, and for each tool call. Always `http`. Set for you. # Aicoustics Source: https://docs.zeroruntime.ai/api-reference/python/denoise/aicoustics Python API reference for the aicoustics noise cancellation provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AICousticsDenoise Create a Denoise instance configured for AI-Coustics. Slot: denoise. Route: inference -- through the ZeroRuntime inference gateway, which authenticates with your ZeroRuntime token -- this class takes no api\_key. ```python theme={null} from zeroruntime.inference import AICousticsDenoise # same fields, through the gateway, no vendor key ``` ### Fields # Rnnoise Source: https://docs.zeroruntime.ai/api-reference/python/denoise/rnnoise Python API reference for the rnnoise noise cancellation provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## RNNoise Initialize the RNNoise denoise plugin. Slot: denoise. Route: plugins -- direct to the vendor; it needs no key. ```python theme={null} from zeroruntime.plugins import RNNoise # direct to the vendor, with your key ``` ### Fields # Sanas Source: https://docs.zeroruntime.ai/api-reference/python/denoise/sanas Python API reference for the sanas noise cancellation provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SanasDenoise Create a Denoise instance configured for Sanas. Slot: denoise. Route: inference -- through the ZeroRuntime inference gateway, which authenticates with your ZeroRuntime token -- this class takes no api\_key. ```python theme={null} from zeroruntime.inference import SanasDenoise # same fields, through the gateway, no vendor key ``` ### Fields # Anthropic Source: https://docs.zeroruntime.ai/api-reference/python/llm/anthropic Python API reference for the anthropic llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AnthropicLLM Initialize the Anthropic LLM. Slot: llm. Route: plugins -- direct to the vendor, with ANTHROPIC\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import AnthropicLLM # direct to the vendor, with your key ``` ### Fields # Aws Source: https://docs.zeroruntime.ai/api-reference/python/llm/aws Python API reference for the aws llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AWSBedrockLLM AWS Bedrock LLM (Converse API) plugin for VideoSDK Agents. Slot: llm. Route: plugins -- direct to the vendor, with AWS\_ACCESS\_KEY\_ID or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import AWSBedrockLLM # direct to the vendor, with your key ``` ### Fields # Cerebras Source: https://docs.zeroruntime.ai/api-reference/python/llm/cerebras Python API reference for the cerebras llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CerebrasLLM Cerebras LLM implementation using the Cerebras Cloud SDK. Slot: llm. Route: plugins -- direct to the vendor, with CEREBRAS\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import CerebrasLLM # direct to the vendor, with your key ``` ### Fields # Cometapi Source: https://docs.zeroruntime.ai/api-reference/python/llm/cometapi Python API reference for the cometapi llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CometAPILLM Initialize the CometAPI LLM plugin. Slot: llm. Route: plugins -- direct to the vendor, with COMETAPI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import CometAPILLM # direct to the vendor, with your key ``` ### Fields # Google Source: https://docs.zeroruntime.ai/api-reference/python/llm/google Python API reference for the google llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GoogleLLM Initialize the Google LLM plugin. Slot: llm. Route: plugins -- direct to the vendor, with GOOGLE\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import GoogleLLM # direct to the vendor, with your key from zeroruntime.inference import GoogleLLM # same fields, through the gateway, no vendor key ``` ### Fields # Openai Source: https://docs.zeroruntime.ai/api-reference/python/llm/openai Python API reference for the openai llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAILLM Initialize the OpenAI LLM plugin. Slot: llm. Route: plugins -- direct to the vendor, with OPENAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import OpenAILLM # direct to the vendor, with your key ``` ### Fields *** ## ZRTLLM Initialize the VideoSDK LLM plugin against its OpenAI-compatible endpoint. Slot: llm. Route: plugins -- direct to the vendor, with OPENAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import ZRTLLM # direct to the vendor, with your key from zeroruntime.inference import ZRTLLM # same fields, through the gateway, no vendor key ``` ### Fields # Sarvamai Source: https://docs.zeroruntime.ai/api-reference/python/llm/sarvamai Python API reference for the sarvamai llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SarvamAILLM Initialize the SarvamAI LLM plugin. Slot: llm. Route: plugins -- direct to the vendor, with SARVAMAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import SarvamAILLM # direct to the vendor, with your key from zeroruntime.inference import SarvamAILLM # same fields, through the gateway, no vendor key ``` ### Fields # Xai Source: https://docs.zeroruntime.ai/api-reference/python/llm/xai Python API reference for the xai llm provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## XAILLM LLM Plugin for xAI (Grok) API. Slot: llm. Route: plugins -- direct to the vendor, with XAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import XAILLM # direct to the vendor, with your key ``` ### Fields # Python SDK Source: https://docs.zeroruntime.ai/api-reference/python/overview API reference for the Zero Runtime Python SDK. Reference for every public class, function, and type in the Zero Runtime Python SDK. Start with **Core** for agents, sessions, pipelines, tools, and context, then browse the provider sections (Speech-to-text, LLM, Text-to-speech, and more) for the plugins you compose into a pipeline. Install, conventions, and end-to-end examples. # Aws Source: https://docs.zeroruntime.ai/api-reference/python/realtime/aws Python API reference for the aws realtime models provider. ## NovaSonicRealtime Nova Sonic's realtime model implementation Slot: realtime. Route: plugins -- direct to the vendor, with AWS\_ACCESS\_KEY\_ID or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import NovaSonicRealtime # direct to the vendor, with your key ``` ### Fields # Azure Source: https://docs.zeroruntime.ai/api-reference/python/realtime/azure Python API reference for the azure realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AzureVoiceLive Azure Voice Live realtime model implementation Slot: realtime. Route: plugins -- direct to the vendor, with AZURE\_VOICE\_LIVE\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import AzureVoiceLive # direct to the vendor, with your key ``` ### Fields # Gemini Source: https://docs.zeroruntime.ai/api-reference/python/realtime/gemini Python API reference for the gemini realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GeminiRealtime Gemini's realtime model for audio-only communication Slot: realtime. Route: plugins -- direct to the vendor, with GOOGLE\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import GeminiRealtime # direct to the vendor, with your key from zeroruntime.inference import GeminiRealtime # same fields, through the gateway, no vendor key ``` ### Fields # Openai Source: https://docs.zeroruntime.ai/api-reference/python/realtime/openai Python API reference for the openai realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAIRealtime OpenAI's realtime model implementation. Slot: realtime. Route: plugins -- direct to the vendor, with OPENAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import OpenAIRealtime # direct to the vendor, with your key ``` ### Fields # Ultravox Source: https://docs.zeroruntime.ai/api-reference/python/realtime/ultravox Python API reference for the ultravox realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## UltravoxRealtime Ultravox's realtime model for audio-only communication Slot: realtime. Route: plugins -- direct to the vendor, with ULTRAVOX\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import UltravoxRealtime # direct to the vendor, with your key ``` ### Fields # Xai Source: https://docs.zeroruntime.ai/api-reference/python/realtime/xai Python API reference for the xai realtime models provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## XAIRealtime xAI's Grok realtime model implementation Slot: realtime. Route: plugins -- direct to the vendor, with XAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import XAIRealtime # direct to the vendor, with your key ``` ### Fields # Assemblyai Source: https://docs.zeroruntime.ai/api-reference/python/stt/assemblyai Python API reference for the assemblyai speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AssemblyAISTT Initialize the AssemblyAI STT plugin. Slot: stt. Route: plugins -- direct to the vendor, with ASSEMBLYAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import AssemblyAISTT # direct to the vendor, with your key from zeroruntime.inference import AssemblyAISTT # same fields, through the gateway, no vendor key ``` ### Fields # Azure Source: https://docs.zeroruntime.ai/api-reference/python/stt/azure Python API reference for the azure speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AzureSTT Initialize the Azure STT plugin. Slot: stt. Route: plugins -- direct to the vendor, with AZURE\_SPEECH\_KEY or an explicit api\_key. The target accepts \*\*kwargs, so the fields below are what it documents rather than everything it tolerates; the rest go through `extra`. ```python theme={null} from zeroruntime.plugins import AzureSTT # direct to the vendor, with your key ``` ### Fields # Cartesia Source: https://docs.zeroruntime.ai/api-reference/python/stt/cartesia Python API reference for the cartesia speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CartesiaSTT Initialize the Cartesia STT plugin Slot: stt. Route: plugins -- direct to the vendor, with CARTESIA\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import CartesiaSTT # direct to the vendor, with your key from zeroruntime.inference import CartesiaSTT # same fields, through the gateway, no vendor key ``` ### Fields # Cometapi Source: https://docs.zeroruntime.ai/api-reference/python/stt/cometapi Python API reference for the cometapi speech-to-text provider. ## CometAPISTT Initialize the CometAPI STT plugin. Slot: stt. Route: plugins -- direct to the vendor, with COMETAPI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import CometAPISTT # direct to the vendor, with your key ``` ### Fields # Deepgram Source: https://docs.zeroruntime.ai/api-reference/python/stt/deepgram Python API reference for the deepgram speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## DeepgramSTT Initialize the Deepgram STT plugin Slot: stt. Route: plugins -- direct to the vendor, with DEEPGRAM\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import DeepgramSTT # direct to the vendor, with your key from zeroruntime.inference import DeepgramSTT # same fields, through the gateway, no vendor key ``` ### Fields *** ## DeepgramSTTV2 Initialize the Deepgram STT plugin (Flux / v2 API). Slot: stt. Route: plugins -- direct to the vendor, with DEEPGRAM\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import DeepgramSTTV2 # direct to the vendor, with your key ``` ### Fields # Elevenlabs Source: https://docs.zeroruntime.ai/api-reference/python/stt/elevenlabs Python API reference for the elevenlabs speech-to-text provider. ## ElevenLabsSTT ElevenLabs Realtime Speech-to-Text (STT) client. Slot: stt. Route: plugins -- direct to the vendor, with ELEVENLABS\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import ElevenLabsSTT # direct to the vendor, with your key ``` ### Fields # Gladia Source: https://docs.zeroruntime.ai/api-reference/python/stt/gladia Python API reference for the gladia speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GladiaSTT Initialize the Gladia STT plugin with WebSocket support. Slot: stt. Route: plugins -- direct to the vendor, with GLADIA\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import GladiaSTT # direct to the vendor, with your key ``` ### Fields # Google Source: https://docs.zeroruntime.ai/api-reference/python/stt/google Python API reference for the google speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GoogleSTT Initialize the Google STT plugin. Slot: stt. Route: plugins -- direct to the vendor, with GOOGLE\_APPLICATION\_CREDENTIALS or an explicit api\_key. The target accepts \*\*kwargs, so the fields below are what it documents rather than everything it tolerates; the rest go through `extra`. ```python theme={null} from zeroruntime.plugins import GoogleSTT # direct to the vendor, with your key from zeroruntime.inference import GoogleSTT # same fields, through the gateway, no vendor key ``` ### Fields # Navana Source: https://docs.zeroruntime.ai/api-reference/python/stt/navana Python API reference for the navana speech-to-text provider. ## NavanaSTT VideoSDK Agent Framework STT plugin for Navana's Bodhi API. Slot: stt. Route: plugins -- direct to the vendor, with NAVANA\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import NavanaSTT # direct to the vendor, with your key ``` ### Fields # Nvidia Source: https://docs.zeroruntime.ai/api-reference/python/stt/nvidia Python API reference for the nvidia speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## NvidiaSTT NvidiaSTT, stt slot. Slot: stt. Route: plugins -- direct to the vendor, with NVIDIA\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import NvidiaSTT # direct to the vendor, with your key ``` ### Fields # Openai Source: https://docs.zeroruntime.ai/api-reference/python/stt/openai Python API reference for the openai speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAISTT Initialize the OpenAI STT plugin. Slot: stt. Route: plugins -- direct to the vendor, with OPENAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import OpenAISTT # direct to the vendor, with your key ``` ### Fields # Sarvamai Source: https://docs.zeroruntime.ai/api-reference/python/stt/sarvamai Python API reference for the sarvamai speech-to-text provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SarvamAISTT Initialize the SarvamAI STT plugin with WebSocket support. Slot: stt. Route: plugins -- direct to the vendor, with SARVAMAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import SarvamAISTT # direct to the vendor, with your key from zeroruntime.inference import SarvamAISTT # same fields, through the gateway, no vendor key ``` ### Fields # Xai Source: https://docs.zeroruntime.ai/api-reference/python/stt/xai Python API reference for the xai speech-to-text provider. ## XAISTT Initialize the xAI STT plugin. Slot: stt. Route: plugins -- direct to the vendor, with XAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import XAISTT # direct to the vendor, with your key ``` ### Fields # Aws Source: https://docs.zeroruntime.ai/api-reference/python/tts/aws Python API reference for the aws text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AWSPollyTTS AWS Polly TTS implementation (plug-and-play for VideoSDK Agents). Slot: tts. Route: plugins -- direct to the vendor, with AWS\_ACCESS\_KEY\_ID or an explicit api\_key. The target accepts \*\*kwargs, so the fields below are what it documents rather than everything it tolerates; the rest go through `extra`. ```python theme={null} from zeroruntime.plugins import AWSPollyTTS # direct to the vendor, with your key ``` ### Fields # Azure Source: https://docs.zeroruntime.ai/api-reference/python/tts/azure Python API reference for the azure text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## AzureTTS Initialize the Azure TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with AZURE\_SPEECH\_KEY or an explicit api\_key. The target accepts \*\*kwargs, so the fields below are what it documents rather than everything it tolerates; the rest go through `extra`. ```python theme={null} from zeroruntime.plugins import AzureTTS # direct to the vendor, with your key ``` ### Fields # Cambai Source: https://docs.zeroruntime.ai/api-reference/python/tts/cambai Python API reference for the cambai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CambAITTS CambAI Text-to-Speech plugin for VideoSDK agents. Slot: tts. Route: plugins -- direct to the vendor, with CAMBAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import CambAITTS # direct to the vendor, with your key ``` ### Fields # Cartesia Source: https://docs.zeroruntime.ai/api-reference/python/tts/cartesia Python API reference for the cartesia text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## CartesiaTTS Initialize the Cartesia TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with CARTESIA\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import CartesiaTTS # direct to the vendor, with your key from zeroruntime.inference import CartesiaTTS # same fields, through the gateway, no vendor key ``` ### Fields # Cometapi Source: https://docs.zeroruntime.ai/api-reference/python/tts/cometapi Python API reference for the cometapi text-to-speech provider. ## CometAPITTS Initialize the CometAPI TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with COMETAPI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import CometAPITTS # direct to the vendor, with your key ``` ### Fields # Deepgram Source: https://docs.zeroruntime.ai/api-reference/python/tts/deepgram Python API reference for the deepgram text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## DeepgramTTS Initialize the Deepgram TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with DEEPGRAM\_API\_KEY or an explicit api\_key. The target accepts \*\*kwargs, so the fields below are what it documents rather than everything it tolerates; the rest go through `extra`. ```python theme={null} from zeroruntime.plugins import DeepgramTTS # direct to the vendor, with your key from zeroruntime.inference import DeepgramTTS # same fields, through the gateway, no vendor key ``` ### Fields # Elevenlabs Source: https://docs.zeroruntime.ai/api-reference/python/tts/elevenlabs Python API reference for the elevenlabs text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## ElevenLabsTTS Initialize the ElevenLabs TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with ELEVENLABS\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import ElevenLabsTTS # direct to the vendor, with your key ``` ### Fields # Google Source: https://docs.zeroruntime.ai/api-reference/python/tts/google Python API reference for the google text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GoogleTTS Initialize the Google TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with GOOGLE\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import GoogleTTS # direct to the vendor, with your key from zeroruntime.inference import GoogleTTS # same fields, through the gateway, no vendor key ``` ### Fields # Groq Source: https://docs.zeroruntime.ai/api-reference/python/tts/groq Python API reference for the groq text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## GroqTTS Initialize the Groq TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with GROQ\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import GroqTTS # direct to the vendor, with your key ``` ### Fields # Humeai Source: https://docs.zeroruntime.ai/api-reference/python/tts/humeai Python API reference for the humeai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## HumeAITTS Initialize the HumeAI TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with HUMEAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import HumeAITTS # direct to the vendor, with your key ``` ### Fields # Inworldai Source: https://docs.zeroruntime.ai/api-reference/python/tts/inworldai Python API reference for the inworldai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## InworldAITTS Inworld AI Text-to-Speech plugin. Slot: tts. Route: plugins -- direct to the vendor, with INWORLD\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import InworldAITTS # direct to the vendor, with your key ``` ### Fields # Lmnt Source: https://docs.zeroruntime.ai/api-reference/python/tts/lmnt Python API reference for the lmnt text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## LMNTTTS Initialize the LMNT TTS plugin (WebSocket streaming). Slot: tts. Route: plugins -- direct to the vendor, with LMNT\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import LMNTTTS # direct to the vendor, with your key ``` ### Fields # Murfai Source: https://docs.zeroruntime.ai/api-reference/python/tts/murfai Python API reference for the murfai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## MurfAITTS Initialize the Murf.ai TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with MURFAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import MurfAITTS # direct to the vendor, with your key ``` ### Fields # Neuphonic Source: https://docs.zeroruntime.ai/api-reference/python/tts/neuphonic Python API reference for the neuphonic text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## NeuphonicTTS Initialize the Neuphonic TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with NEUPHONIC\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import NeuphonicTTS # direct to the vendor, with your key ``` ### Fields # Nvidia Source: https://docs.zeroruntime.ai/api-reference/python/tts/nvidia Python API reference for the nvidia text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## NvidiaTTS NvidiaTTS, tts slot. Slot: tts. Route: plugins -- direct to the vendor, with NVIDIA\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import NvidiaTTS # direct to the vendor, with your key ``` ### Fields # Openai Source: https://docs.zeroruntime.ai/api-reference/python/tts/openai Python API reference for the openai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## OpenAITTS Initialize the OpenAI TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with OPENAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import OpenAITTS # direct to the vendor, with your key ``` ### Fields # Papla Source: https://docs.zeroruntime.ai/api-reference/python/tts/papla Python API reference for the papla text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## PaplaTTS Initialize the Papla TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with PAPLA\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import PaplaTTS # direct to the vendor, with your key ``` ### Fields # Resemble Source: https://docs.zeroruntime.ai/api-reference/python/tts/resemble Python API reference for the resemble text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## ResembleTTS Initialize the Resemble TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with RESEMBLE\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import ResembleTTS # direct to the vendor, with your key ``` ### Fields # Rime Source: https://docs.zeroruntime.ai/api-reference/python/tts/rime Python API reference for the rime text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## RimeTTS Initialize the Rime TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with RIME\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import RimeTTS # direct to the vendor, with your key ``` ### Fields # Sarvamai Source: https://docs.zeroruntime.ai/api-reference/python/tts/sarvamai Python API reference for the sarvamai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SarvamAITTS A unified Sarvam.ai Text-to-Speech (TTS) plugin that supports both real- time Slot: tts. Route: plugins -- direct to the vendor, with SARVAMAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import SarvamAITTS # direct to the vendor, with your key from zeroruntime.inference import SarvamAITTS # same fields, through the gateway, no vendor key ``` ### Fields # Smallestai Source: https://docs.zeroruntime.ai/api-reference/python/tts/smallestai Python API reference for the smallestai text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SmallestAITTS Initialize the SmallestAI TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with SMALLEST\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import SmallestAITTS # direct to the vendor, with your key ``` ### Fields # Speechify Source: https://docs.zeroruntime.ai/api-reference/python/tts/speechify Python API reference for the speechify text-to-speech provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SpeechifyTTS Initialize the Speechify TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with SPEECHIFY\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import SpeechifyTTS # direct to the vendor, with your key ``` ### Fields # Xai Source: https://docs.zeroruntime.ai/api-reference/python/tts/xai Python API reference for the xai text-to-speech provider. ## XAITTS Initialize the xAI TTS plugin. Slot: tts. Route: plugins -- direct to the vendor, with XAI\_API\_KEY or an explicit api\_key. ```python theme={null} from zeroruntime.plugins import XAITTS # direct to the vendor, with your key ``` ### Fields # Turn Detector Source: https://docs.zeroruntime.ai/api-reference/python/turn-detection/namo Python API reference for the unified turn detector. Setup, environment variables, and Python/JavaScript/Go usage examples. ## TurnDetector End-of-turn detection through the ZeroRuntime inference gateway. Slot: turn\_detector. Route: inference -- through the ZeroRuntime inference gateway, which authenticates with your ZeroRuntime token -- this class takes no api\_key. `model` selects the backend: echo-large echo-small ```python theme={null} from zeroruntime.inference import TurnDetector # same fields, through the gateway, no vendor key ``` ### Fields # Silero Source: https://docs.zeroruntime.ai/api-reference/python/vad/silero Python API reference for the silero voice activity detection provider. Setup, environment variables, and Python/JavaScript/Go usage examples. ## SileroVAD Silero Voice Activity Detection with advanced streaming features. Slot: vad. Route: plugins -- direct to the vendor; it needs no key. ```python theme={null} from zeroruntime.plugins import SileroVAD # direct to the vendor, with your key ``` ### Fields # Authentication Source: https://docs.zeroruntime.ai/authentication Connect your worker to Zero Runtime and authenticate it: runtime address, auth token, API key and secret, and provider keys. Your worker connects to the Zero Runtime runtime over a secure connection. Configure the connection and credentials with environment variables. ## Connect to the runtime Point the worker at your runtime address: ```bash theme={null} export ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 ``` ## Authenticate the worker Get your auth token, or your API key and secret, from the [Zero Runtime dashboard](https://app.zeroruntime.ai/). Then authenticate in one of two ways: use a JWT auth token, or provide an API key and secret and let the SDK mint a JWT for you. ```bash theme={null} # JWT auth token export ZERORUNTIME_AUTH_TOKEN= ``` ## Provider keys Set the key for each provider your pipeline uses: ```bash theme={null} export DEEPGRAM_API_KEY= # speech-to-text export GOOGLE_API_KEY= # LLM export CARTESIA_API_KEY= # text-to-speech ``` ## Reference | Variable | Purpose | | ------------------------ | ----------------------------- | | `ZERORUNTIME_TARGET` | Runtime address | | `ZERORUNTIME_AUTH_TOKEN` | JWT auth token | | `_API_KEY` | Key for each provider you use | # Configure a Pipeline Source: https://docs.zeroruntime.ai/build/configure-a-pipeline Wire STT, LLM, and TTS into the Pipeline your agent runs on. The `Pipeline` is what your agent hears and speaks through. Pass the components you need (STT, LLM, TTS, plus optional VAD and turn detection) and the pipeline auto-detects its mode from what you give it. Pipeline flow from user audio through input processing to cascading, realtime, or hybrid mode, then audio output ```python title="Python" Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import DeepgramSTT, GoogleLLM, CartesiaTTS, SileroVAD from zeroruntime.inference import AICousticsDenoise, TurnDetector pipeline = Pipeline( stt=DeepgramSTT(), llm=GoogleLLM(), tts=CartesiaTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { DeepgramSTT, GoogleLLM, CartesiaTTS, SileroVAD } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: GoogleLLM(), tts: CartesiaTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` Pass `pipeline` to your [Agent](/build/creating-an-agent), then [run it](/build/run-the-runtime). ## Configure it further Cascade, Realtime, and Hybrid, chosen automatically from the components you pass. Tap into the pipeline at runtime to inspect or transform each stage. Fail over between providers to keep sessions resilient. ## What's Next Register and serve the agent. Give the agent function tools and external services. ## References #### Examples Minimal STT-LLM-TTS cascade agent you can run. #### Examples Minimal STT-LLM-TTS cascade agent you can run. # Fallback Adapter Source: https://docs.zeroruntime.ai/build/configure-a-pipeline/fallback-adapter Automatic failover between multiple STT, LLM, or TTS providers The Fallback Adapter provides automatic failover between multiple STT, LLM, or TTS providers. It switches providers on two conditions: first, on **errors** when a provider fails or becomes unavailable, and second, on **latency** when a provider stays slower than its configured budget. In both cases the system automatically switches to the next configured provider without interrupting the session. ## Features Switches to lower-priority providers if the primary provider fails. Optionally switches providers when a component stays above its latency budget for several consecutive turns. Implements a cooldown period before retrying a failed provider, preventing immediate repeated failures. Automatically switches back to a higher-priority provider once it becomes healthy again. Permanently disables a provider after a configured number of failed recovery attempts. ## Error-based Fallback Here is how you can implement error-based fallback providers for STT, LLM, and TTS in your agent's `Pipeline`. When a provider fails or becomes unavailable, the system switches to the next configured provider. ```python Python theme={null} from zeroruntime import FallbackSTT, FallbackLLM, FallbackTTS, Pipeline from zeroruntime.plugins import SarvamAISTT, DeepgramSTT, OpenAILLM, GoogleLLM, CartesiaTTS, DeepgramTTS # The head serves, the tail stands by. pipeline = Pipeline( stt=FallbackSTT( [SarvamAISTT(model="saarika:v2"), DeepgramSTT(model="nova-2")], temporary_disable_sec=30.0, permanent_disable_after_attempts=3, ), llm=FallbackLLM( [OpenAILLM(model="gpt-4o-mini"), GoogleLLM(model="gemini-2.5-flash")], temporary_disable_sec=30.0, permanent_disable_after_attempts=3, ), tts=FallbackTTS( [CartesiaTTS(model="sonic-2"), DeepgramTTS(model="aura-2-thalia-en")], temporary_disable_sec=30.0, permanent_disable_after_attempts=3, ), ) ``` ```typescript Node JS theme={null} import { FallbackLLM, FallbackSTT, FallbackTTS, Pipeline } from '@zeroruntime/js-sdk'; import { CartesiaTTS, DeepgramSTT, DeepgramTTS, GoogleLLM, OpenAILLM, SarvamAISTT, } from '@zeroruntime/js-sdk/plugins'; // The head serves, the tail stands by. const pipeline = Pipeline({ stt: FallbackSTT([SarvamAISTT({ model: 'saarika:v2' }), DeepgramSTT({ model: 'nova-2' })], { temporary_disable_sec: 30.0, permanent_disable_after_attempts: 3, }), llm: FallbackLLM([OpenAILLM({ model: 'gpt-4o-mini' }), GoogleLLM({ model: 'gemini-2.5-flash' })], { temporary_disable_sec: 30.0, permanent_disable_after_attempts: 3, }), tts: FallbackTTS([CartesiaTTS({ model: 'sonic-2' }), DeepgramTTS({ model: 'aura-2-thalia-en' })], { temporary_disable_sec: 30.0, permanent_disable_after_attempts: 3, }), }); ``` Each wrapper goes straight into its normal slot on `Pipeline`. The rest of your agent setup (`Agent`, `Room`, `on_enter`/`on_exit`, etc.) stays unchanged. A bare list — `stt=[primary, standby]` — is the same thing with every option left at its default. ### Configuration Options Set these on the wrapper for the slot they apply to. Each slot carries its own policy. The duration (in seconds) to wait before retrying a failed provider. The maximum number of recovery attempts allowed before a provider is permanently disabled. ## Latency-based Fallback Beyond hard failures, the Fallback Adapter can switch providers when a healthy provider becomes too slow. This is useful for keeping conversations responsive when a provider degrades without erroring out. Latency-based fallback is **off by default**. Set `latency_threshold_ms` on a component to enable it. * Each component measures a relevant latency metric: STT uses `stt_latency`, LLM uses `llm_ttft` (time to first token), and TTS uses `ttfb` (time to first byte). The budget on a wrapper is checked against its own slot's metric. * A provider is only switched after it stays above the threshold for `consecutive_latency_hits` turns in a row, avoiding switches caused by a single slow turn. * Recovery and cooldown for a latency-disabled provider use the same `temporary_disable_sec` and `permanent_disable_after_attempts` settings as the error path. To enable latency-based fallback, add `latency_threshold_ms` (and optionally `consecutive_latency_hits`) to the wrapper. Budget each slot on its own: a few hundred milliseconds is generous for STT and TTS and punishing for an LLM's first token. ```python Python theme={null} pipeline = Pipeline( stt=FallbackSTT( [SarvamAISTT(model="saarika:v2"), DeepgramSTT(model="nova-2")], temporary_disable_sec=30.0, permanent_disable_after_attempts=3, latency_threshold_ms=350, # enable latency-based fallback consecutive_latency_hits=3, ), llm=FallbackLLM( [OpenAILLM(model="gpt-4o-mini"), GoogleLLM(model="gemini-2.5-flash")], temporary_disable_sec=30.0, permanent_disable_after_attempts=3, latency_threshold_ms=800, consecutive_latency_hits=3, ), tts=FallbackTTS( [CartesiaTTS(model="sonic-2"), DeepgramTTS(model="aura-2-thalia-en")], temporary_disable_sec=30.0, permanent_disable_after_attempts=3, latency_threshold_ms=250, consecutive_latency_hits=3, ), ) ``` ```typescript Node JS theme={null} const pipeline = Pipeline({ stt: FallbackSTT([SarvamAISTT({ model: 'saarika:v2' }), DeepgramSTT({ model: 'nova-2' })], { temporary_disable_sec: 30.0, permanent_disable_after_attempts: 3, latency_threshold_ms: 350, // enable latency-based fallback consecutive_latency_hits: 3, }), llm: FallbackLLM([OpenAILLM({ model: 'gpt-4o-mini' }), GoogleLLM({ model: 'gemini-2.5-flash' })], { temporary_disable_sec: 30.0, permanent_disable_after_attempts: 3, latency_threshold_ms: 800, consecutive_latency_hits: 3, }), tts: FallbackTTS([CartesiaTTS({ model: 'sonic-2' }), DeepgramTTS({ model: 'aura-2-thalia-en' })], { temporary_disable_sec: 30.0, permanent_disable_after_attempts: 3, latency_threshold_ms: 250, consecutive_latency_hits: 3, }), }); ``` ### Configuration Options You can configure the latency-based fallback behavior using the following parameters: This slot's latency budget in milliseconds, checked against its own metric (STT `stt_latency`, LLM `llm_ttft`, TTS `ttfb`). Off by default. Pass a value to enable latency-based fallback. The number of consecutive turns that must exceed `latency_threshold_ms` before switching providers. ## References #### Examples Checkout the full implementation on GitHub #### Examples Checkout the full implementation on GitHub # Modes Source: https://docs.zeroruntime.ai/build/configure-a-pipeline/modes Cascade, Hybrid, and Realtime, the three pipeline modes the Pipeline auto-detects from the components you pass in. The `Pipeline` is self-configuring: based on the components you hand it, it automatically runs in **Cascade**, **Hybrid**, or **Realtime** mode. Switch between the tabs below to compare each mode's architecture, when to reach for it, and the code that selects it. #### Supported Modes There are three modes supported by the Pipeline. ##### 1. Cascade (STT-LLM-TTS) ##### 2. Realtime (LLM) ##### 3. Hybrid (Cascade + Realtime) Cascade mode runs the pipeline as separate stages: VAD → STT → Turn Detector → LLM → TTS. The `Pipeline` auto-detects it when you pass all these components, giving you full control over each stage. **Architecture** Audio flows: VAD detects speech, STT transcribes, the turn detector signals end of turn, the LLM generates a reply, and TTS synthesizes audio. Cascade pipeline **When to use it** * You need granular control over each stage (STT, LLM, TTS, VAD, turn detection). * You want to mix and match providers for individual stages. * You rely on custom hooks for turn-taking, RAG, or content filtering. * You can trade higher latency for maximum flexibility. **Example** ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Agent, Pipeline from zeroruntime.plugins import DeepgramSTT, OpenAILLM, ElevenLabsTTS, SileroVAD from zeroruntime.inference import AICousticsDenoise, TurnDetector AGENT_ID = "assistant" pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=ElevenLabsTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class Assistant(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions="You are a helpful voice assistant.", pipeline=pipeline, ) if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh Assistant + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(Assistant, on_ready=lambda: zeroruntime.invoke(AGENT_ID, room=zeroruntime.Room(playground=True))) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, Pipeline } from '@zeroruntime/js-sdk'; import { DeepgramSTT, OpenAILLM, ElevenLabsTTS, SileroVAD } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const AGENT_ID = 'assistant'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: ElevenLabsTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); class Assistant extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful voice assistant.', pipeline, }); } } // Pass the class itself (not an instance): serve() builds a fresh Assistant + // pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(Assistant, { on_ready: () => zeroruntime.invoke(AGENT_ID, { room: zeroruntime.Room({ playground: true }) }) }); ``` Realtime mode runs a single end-to-end speech-to-speech model (for example, OpenAI Realtime, Google Gemini Live, or AWS Nova Sonic). The Pipeline auto-detects it when you pass a realtime model as `llm`, so no separate STT or TTS stages are required. **Architecture** User audio streams to the realtime model, which transcribes, reasons, synthesizes, and streams audio back. Realtime pipeline **When to use it** * Latency is your top priority. * You want the simplest setup with the fewest moving parts. * You need fast, natural conversational flow. * You don't need to swap individual STT or TTS providers. **Example** ```python title="Python" Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import OpenAIRealtime model = OpenAIRealtime( model="gpt-4o-realtime-preview", config={"voice": "alloy", "modalities": ["audio", "text"]} ) # A realtime model goes in the LLM slot - the Pipeline auto-detects Realtime mode. pipeline = Pipeline(llm=model) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { OpenAIRealtime } from '@zeroruntime/js-sdk/plugins'; const model = OpenAIRealtime({ model: 'gpt-4o-realtime-preview', config: {voice: 'alloy', modalities: ['audio', 'text']}, }); // A realtime model goes in the LLM slot - the Pipeline auto-detects Realtime mode. const pipeline = Pipeline({ llm: model }); ``` Hybrid mode pairs a realtime model with an external STT or TTS. The Pipeline auto-detects the sub-mode from the extra component: `hybrid_stt` (external STT) or `hybrid_tts` (external TTS). **Architecture** * **Hybrid STT**: your STT produces a transcript, then both audio and transcript go to the realtime model. Use this when you need searchable text (for KB lookups). * **Hybrid TTS**: the realtime model handles input and reasoning, and its text output is routed to your TTS for the final voice. Hybrid pipeline **When to use it** | sub‑mode | Recommended use cases | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | Hybrid STT (`hybrid_stt`) | When you need a searchable transcript for knowledge‑base lookups or RAG, or when the realtime model lacks the language support or accuracy you require. | | Hybrid TTS (`hybrid_tts`) | When you need a specific custom voice or advanced TTS features (for example, voice tuning or SSML) that the realtime model doesn't provide. | **Example** Provide an external STT and the Pipeline auto-detects `hybrid_stt`: ```python title="Python" Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import GeminiRealtime, SarvamAISTT, SileroVAD from zeroruntime.inference import AICousticsDenoise model = GeminiRealtime( model="gemini-3.1-flash-live-preview", config={ "voice": "Puck", "response_modalities": ["AUDIO"], } ) pipeline = Pipeline( stt=SarvamAISTT(), llm=model, vad=SileroVAD(), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz") ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { GeminiRealtime, SarvamAISTT, SileroVAD } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const model = GeminiRealtime({ model: 'gemini-3.1-flash-live-preview', config: { voice: 'Puck', response_modalities: ['AUDIO'], }, }); const pipeline = Pipeline({ stt: SarvamAISTT(), llm: model, vad: SileroVAD(), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` Provide an external TTS and the Pipeline auto-detects `hybrid_tts`: ```python title="Python" Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import OpenAIRealtime, ElevenLabsTTS from zeroruntime.inference import AICousticsDenoise model = OpenAIRealtime( model="gpt-4o-realtime-preview", config={"voice": "alloy"} ) pipeline = Pipeline( llm=model, tts=ElevenLabsTTS(), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz") ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { OpenAIRealtime, ElevenLabsTTS } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const model = OpenAIRealtime({ model: 'gpt-4o-realtime-preview', config: {voice: 'alloy'}, }); const pipeline = Pipeline({ llm: model, tts: ElevenLabsTTS(), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` ## Other Pipeline Combinations The `Pipeline` is not limited to the named modes. It accepts any subset of components and configures itself accordingly. | Combination | Components | Use case | | :---------- | :--------------- | :------------------------------------------------------------------------- | | LLM + TTS | `llm` and `tts` | Text in, voice out. The agent receives text input and replies with speech. | | STT + LLM | `stt` and `llm` | Voice in, text out. The user speaks and the agent replies with text. | | Partial | Any other subset | Custom setups where you only need part of the conversational loop. | ```python title="Python" Python theme={null} # Text in, voice out pipeline = Pipeline(llm=OpenAILLM(), tts=ElevenLabsTTS()) # Voice in, text out pipeline = Pipeline(stt=DeepgramSTT(), llm=OpenAILLM()) ``` ```typescript title="Node JS" Node JS theme={null} // Text in, voice out const pipeline = Pipeline({ llm: OpenAILLM(), tts: ElevenLabsTTS() }); // Voice in, text out const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM() }); ``` ## What's Next Tap into the pipeline at runtime to inspect or transform each stage. Observe metrics and traces at each pipeline stage. Add automatic provider failover for any mode. ## References #### Examples Run STT, LLM and TTS as a cascade. Run a speech-to-speech realtime model. #### Examples Run STT, LLM and TTS as a cascade. Run a speech-to-speech realtime model. # Observability Hooks Source: https://docs.zeroruntime.ai/build/configure-a-pipeline/observability-hooks Capture component-level latency and token usage without changing the data flow. Observability hooks capture component-level latency and token usage without changing the data flow. They are side-effect-only, so you can register many safely, each fires at the end of a component's turn with a payload of metrics (a `dict`). ## Component Metrics Register a metrics hook with the `@pipeline.metrics.on("")` decorator. Each component delivers different payload keys. | Component | Mode | Key Payload Fields | | :--------- | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `stt` | Cascade | `stt_latency` | | `llm` | Cascade | `llm_ttft`, `llm_duration`, `prompt_tokens`, `completion_tokens`, `total_tokens` | | `tts` | Cascade | `ttfb`, `tts_latency` | | `eou` | Cascade | `eou_latency`, `eou_wait_ms` | | `realtime` | Realtime / Hybrid | `realtime_ttfb`, `realtime_input_tokens`, `realtime_output_tokens`, `realtime_total_tokens`, `realtime_input_text_tokens`, `realtime_output_text_tokens`, `realtime_input_audio_tokens`, `realtime_output_audio_tokens` | ## Use cases Use `stt`, `llm`, `tts`, and `eou` metrics with Cascade pipelines. Use `realtime` metrics when the pipeline runs in Realtime or Hybrid mode with a realtime model as the LLM. ```python title="Python" Python theme={null} @pipeline.metrics.on("stt") def on_stt_metrics(metrics: dict): """Fired when an STT turn completes.""" print(f"[METRICS] STT Latency: {metrics.get('stt_latency')}ms") ``` ```typescript title="Node JS" Node JS theme={null} pipeline.metrics.on('stt', async (metrics: Record) => { // Fired when an STT turn completes. console.log(`[METRICS] STT Latency: ${metrics.stt_latency}ms`); }); ``` ```python title="Python" Python theme={null} @pipeline.metrics.on("llm") def on_llm_metrics(metrics: dict): """Fired when LLM generation completes.""" print( f"[METRICS] LLM TTFT: {metrics.get('llm_ttft')}ms | " f"Total Duration: {metrics.get('llm_duration')}ms" ) print( "[METRICS] LLM Tokens (P/C/T): " f"{metrics.get('prompt_tokens')}/" f"{metrics.get('completion_tokens')}/" f"{metrics.get('total_tokens')}" ) ``` ```typescript title="Node JS" Node JS theme={null} pipeline.metrics.on('llm', async (metrics: Record) => { // Fired when LLM generation completes. console.log( `[METRICS] LLM TTFT: ${metrics.llm_ttft}ms | ` + `Total Duration: ${metrics.llm_duration}ms` ); console.log( '[METRICS] LLM Tokens (P/C/T): ' + `${metrics.prompt_tokens}/` + `${metrics.completion_tokens}/` + `${metrics.total_tokens}` ); }); ``` ```python title="Python" Python theme={null} @pipeline.metrics.on("tts") def on_tts_metrics(metrics: dict): """Fired when TTS finishes speaking.""" print( f"[METRICS] TTS TTFB: {metrics.get('ttfb')}ms | " f"Total Latency: {metrics.get('tts_latency')}ms" ) ``` ```typescript title="Node JS" Node JS theme={null} pipeline.metrics.on('tts', async (metrics: Record) => { // Fired when TTS finishes speaking. console.log( `[METRICS] TTS TTFB: ${metrics.ttfb}ms | ` + `Total Latency: ${metrics.tts_latency}ms` ); }); ``` ```python title="Python" Python theme={null} @pipeline.metrics.on("eou") def on_eou_metrics(metrics: dict): """Fired when the Turn Detector matches end-of-utterance.""" print( f"[METRICS] EOU Latency: {metrics.get('eou_latency')}ms | " f"EOU Wait: {metrics.get('eou_wait_ms')}ms" ) ``` ```typescript title="Node JS" Node JS theme={null} pipeline.metrics.on('eou', async (metrics: Record) => { // Fired when the Turn Detector matches end-of-utterance. console.log( `[METRICS] EOU Latency: ${metrics.eou_latency}ms | ` + `EOU Wait: ${metrics.eou_wait_ms}ms` ); }); ``` ```python title="Python" Python theme={null} @pipeline.metrics.on("realtime") def on_realtime_metrics(metrics: dict): """Fired for realtime (speech-to-speech) models.""" print( "[METRICS] Realtime " f"TTFB: {metrics.get('realtime_ttfb')}ms | " f"Tokens (in/out/total): " f"{metrics.get('realtime_input_tokens')}/" f"{metrics.get('realtime_output_tokens')}/" f"{metrics.get('realtime_total_tokens')} | " f"TextTokens (in/out): " f"{metrics.get('realtime_input_text_tokens')}/" f"{metrics.get('realtime_output_text_tokens')} | " f"AudioTokens (in/out): " f"{metrics.get('realtime_input_audio_tokens')}/" f"{metrics.get('realtime_output_audio_tokens')}" ) ``` ```typescript title="Node JS" Node JS theme={null} pipeline.metrics.on('realtime', async (metrics: Record) => { // Fired for realtime (speech-to-speech) models. console.log( '[METRICS] Realtime ' + `TTFB: ${metrics.realtime_ttfb}ms | ` + `Tokens (in/out/total): ` + `${metrics.realtime_input_tokens}/` + `${metrics.realtime_output_tokens}/` + `${metrics.realtime_total_tokens} | ` + `TextTokens (in/out): ` + `${metrics.realtime_input_text_tokens}/` + `${metrics.realtime_output_text_tokens} | ` + `AudioTokens (in/out): ` + `${metrics.realtime_input_audio_tokens}/` + `${metrics.realtime_output_audio_tokens}` ); }); ``` ## What's Next View the captured metrics, traces, and logs on the Dashboard. Monitor agent and user state changes. ## References #### Examples Emit metrics and traces from pipeline hooks. #### Examples Emit metrics and traces from pipeline hooks. # Overview Source: https://docs.zeroruntime.ai/build/configure-a-pipeline/overview A single, self-configuring component that auto-detects Cascade, Realtime, or Hybrid mode. The `Pipeline` is a single, self-configuring component. You hand it the components you need, and based on what you pass in it automatically selects **Cascade**, **Realtime**, or **Hybrid** mode and connects them for you. ## Pipeline Code Snippet A basic Cascade pipeline built from STT, LLM, TTS, VAD, and a Turn Detector. ```python title="Python" Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import DeepgramSTT, OpenAILLM, ElevenLabsTTS, SileroVAD from zeroruntime.inference import AICousticsDenoise, TurnDetector pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=ElevenLabsTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { DeepgramSTT, OpenAILLM, ElevenLabsTTS, SileroVAD } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: ElevenLabsTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` ## What's Next #### Plugins Speech-to-text providers. Text-to-speech providers. Language model providers. Speech-to-speech realtime models. #### Feature Observe metrics and traces at each pipeline stage. Cascade, Realtime, and Hybrid, chosen automatically from your components. ## References #### Examples Minimal cascade pipeline. Minimal Realtime pipeline. #### Examples Minimal cascade pipeline. Minimal Realtime pipeline. # Runtime Hooks Source: https://docs.zeroruntime.ai/build/configure-a-pipeline/runtime-hooks Intercept and process pipeline data at the STT, LLM, TTS, vision, and lifecycle stages. Runtime hooks let you intercept and process data at each stage of the Pipeline without subclassing it. Register them with the `@pipeline.on()` decorator. ## Audio Processing Hooks The `stt` and `tts` hooks replace the built-in component processing. Each receives an async iterator and yields processed results. Only one of each can be registered. The `stt` and `tts` hooks fully replace the built-in component processing: you receive the raw stream and yield the processed result yourself. #### STT hook: clean up transcripts A common use case is preprocessing the audio and normalizing the transcript before it reaches the LLM. This hook drops tiny audio chunks, then strips filler words from the transcribed text. ```python title="Python" Python theme={null} @pipeline.on("stt") async def stt_hook(audio_stream): # Drop very small audio chunks before transcribing async def filtered(): async for audio in audio_stream: if len(audio) >= 300: yield audio # Transcribe, then remove filler words from the text async for event in run_stt(filtered()): if event.data and event.data.text: text = re.sub(r"\b(uh|um|like)\b", "", event.data.text) event.data.text = " ".join(text.split()) yield event ``` ```typescript title="Node JS" Node JS theme={null} import { run_stt } from '@zeroruntime/js-sdk'; const FILLERS = /\b(?:uh|um|like)\b/g; pipeline.on('stt', async function* (audio_stream) { // The audio phase is a passthrough here: STT runs in the agent process and // only its transcript crosses, so the hook reshapes text rather than audio. // Yield nothing for an utterance and the turn drops it. for await (const event of run_stt(audio_stream)) { const text = event.data.text.replace(FILLERS, '').split(/\s+/).filter(Boolean).join(' '); if (!text) continue; event.data.text = text; yield event; } }); ``` #### TTS hook: fix pronunciation A common use case is reshaping text so the voice reads it correctly. This hook spells out abbreviations so the TTS pronounces them as expected. ```python title="Python" Python theme={null} @pipeline.on("tts") async def tts_hook(text_stream): # Rewrite text before it is synthesized async def preprocess(): async for text in text_stream: yield text.replace("AM", "A M").replace("PM", "P M") async for audio in run_tts(preprocess()): yield audio ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline, PronunciationRule } from '@zeroruntime/js-sdk'; // There is no `tts` hook in the Node JS SDK. The same reshaping is declared on // the pipeline and applied in the agent process between the LLM and TTS. const pipeline = Pipeline({ // ...stt, llm, tts pronunciations: [ PronunciationRule('AM', 'A M'), PronunciationRule('PM', 'P M'), ], }); ``` ## Lifecycle Hooks Lifecycle hooks are side-effect-only. Use them for logging, analytics, and triggering external actions. You can register multiple per event. * `user_turn_start(transcript: str)`: user's final transcript is available. * `user_turn_end()`: agent finished responding for the turn. * `agent_turn_start()`: agent starts speaking. * `agent_turn_end()`: agent finishes speaking. Fetch context for the turn. The final transcript is available here, so it is a good place to run a knowledge-base lookup before the LLM responds. ```python title="Python" Python theme={null} @pipeline.on("user_turn_start") async def on_user_start(transcript: str): docs = await knowledge_base.search(transcript) agent.metadata["retrieved"] = docs ``` ```typescript title="Node JS" Node JS theme={null} pipeline.on('user_turn_start', async (transcript: string) => { const docs = await knowledge_base.search(transcript); agent.metadata['retrieved'] = docs; }); ``` Save the completed turn. Fires once the agent has finished responding, so the turn is complete. Use it to persist the exchange. ```python title="Python" Python theme={null} @pipeline.on("user_turn_end") async def on_user_end(): await db.save_turn(session_id, turn_count) ``` ```typescript title="Node JS" Node JS theme={null} pipeline.on('user_turn_end', async () => { await db.save_turn(session_id, turn_count); }); ``` Show a speaking indicator. Fires when the agent starts speaking. Push a status update so the client UI can show the agent is talking. ```python title="Python" Python theme={null} from zeroruntime import PubSubPublishConfig class StatusAgent(Agent): # An agent method named after the hook is called when the pipeline # registers no handler for it, and reaches the call through self.session. async def on_agent_turn_start(self) -> None: await self.session.publish_to_pubsub( PubSubPublishConfig(topic="AGENT_STATUS", message="speaking") ) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, PubSubPublishConfig } from '@zeroruntime/js-sdk'; class StatusAgent extends Agent { // An agent method named after the hook is called when the pipeline // registers no handler for it, and reaches the call through this.session. async on_agent_turn_start(): Promise { await this.session!.publish_to_pubsub( PubSubPublishConfig({ topic: 'AGENT_STATUS', message: 'speaking' }), ); } } ``` Clear the indicator. Fires when the agent finishes speaking. Use it to reset UI state or measure response latency. ```python title="Python" Python theme={null} from zeroruntime import PubSubPublishConfig class StatusAgent(Agent): # An agent method named after the hook is called when the pipeline # registers no handler for it, and reaches the call through self.session. async def on_agent_turn_end(self) -> None: await self.session.publish_to_pubsub( PubSubPublishConfig(topic="AGENT_STATUS", message="idle") ) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, PubSubPublishConfig } from '@zeroruntime/js-sdk'; class StatusAgent extends Agent { // An agent method named after the hook is called when the pipeline // registers no handler for it, and reaches the call through this.session. async on_agent_turn_end(): Promise { await this.session!.publish_to_pubsub( PubSubPublishConfig({ topic: 'AGENT_STATUS', message: 'idle' }), ); } } ``` ## What's Next Review how the pipeline is configured before hooking into it. Run the pipeline with an agent. ## References #### Examples Hook into a cascade pipeline at runtime. #### Examples Hook into a cascade pipeline at runtime. # Wakeup Call Source: https://docs.zeroruntime.ai/build/configure-a-pipeline/wakeup-call Automatically nudge users back into the conversation after a period of inactivity A Wakeup Call automatically triggers an action when the user has been inactive for a specified period of time. Instead of leaving a silent gap when the caller goes quiet, the agent can gently check in, re-prompt, or offer help, keeping the conversation alive and maintaining engagement. ## How it works Set `wake_up` on your `Agent` to the number of seconds of caller silence to allow before nudging. When the caller stays quiet for that long, the runtime calls your agent's `on_wake_up` method, where you decide what happens next. ```python Python theme={null} pipeline = Pipeline( stt=CartesiaSTT(model="ink-2"), llm=OpenAILLM(model="gpt-5.4-nano-2026-03-17", streaming=True), tts=SarvamAITTS(streaming=True), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), ) class PatientAgent(Agent): def __init__(self) -> None: super().__init__( name="PatientAgent", agent_id=AGENT_ID, instructions=( "You are a patient assistant. Answer questions and help the caller. If they go " "quiet, you'll gently check in on them." ), pipeline=pipeline, wake_up=10, # nudge after 10s of caller silence ) self._nudges = 0 async def on_enter(self) -> None: await self.session.say("Hi! Take your time — I'm here whenever you're ready.") async def on_exit(self) -> None: await self.session.say("Goodbye!") async def on_wake_up(self) -> None: # Called by the runtime when the caller has been silent for `wake_up` seconds. self._nudges += 1 await self.session.say("Are you still there? I'm happy to keep helping.") ``` ```typescript Node JS theme={null} const pipeline = Pipeline({ stt: CartesiaSTT({ model: 'ink-2' }), llm: OpenAILLM({ model: 'gpt-5.4-nano-2026-03-17', streaming: true }), tts: SarvamAITTS({ streaming: true }), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), }); class PatientAgent extends Agent { _nudges: number; constructor() { super({ name: 'PatientAgent', agent_id: AGENT_ID, instructions: 'You are a patient assistant. Answer questions and help the caller. If they go ' + "quiet, you'll gently check in on them.", pipeline, wake_up: 10, // nudge after 10s of caller silence }); this._nudges = 0; } async on_enter() { await this.session!.say("Hi! Take your time — I'm here whenever you're ready."); } async on_exit() { await this.session!.say('Goodbye!'); } async on_wake_up() { // Called by the runtime when the caller has been silent for `wake_up` seconds. this._nudges += 1; await this.session!.say("Are you still there? I'm happy to keep helping."); } } ``` Track state across nudges (like the `_nudges` counter above) to escalate your response — for example, offer more help on the first nudge and end the call after several unanswered check-ins. ### Configuration Options Set on the `Agent`. Seconds of caller silence to allow before triggering `on_wake_up`. `0` disables wake-up calls; a negative value is rejected. ### Callback Override this method on your `Agent` to define the wake-up action. The runtime calls it each time the caller stays silent for `wake_up` seconds. ## References #### Examples Checkout the full implementation on GitHub #### Examples Checkout the full implementation on GitHub # Agent Handoffs Source: https://docs.zeroruntime.ai/build/context-management/agent-handoffs Split a workflow across specialized agents and switch control between them. Agent handoffs let you split complex workflows across specialized agents. The primary agent detects intent and calls a `function_tool` to hand control to the right agent, which completes the request. ## Handoff vs Transfer "Handoff" and "transfer" are easy to confuse. They differ in who receives the conversation and whether context is passed along. | Term | Conversation moves to | Context passed | Covered in | | :--------------------------------- | :--------------------------------------------------------- | :------------------------------ | :---------------------------------------------- | | **Agent Handoff** | Another **AI agent** in the same session | Optional, via `inherit_context` | This page | | **Agent Transfer** (Call Transfer) | A **human or another phone number**, connected immediately | No | [Call Transfer](/build/telephony/call-transfer) | In short: a **handoff between agents** keeps the conversation with the bot and only switches which agent is in control. A **transfer** hands the live call to a person. A transfer is **cold** when the person is connected with no briefing, and **warm** when they receive context before the call connects. ## Context Sharing When switching agents, the `inherit_context` flag controls whether the new agent is aware of the previous conversation. | Value | Behavior | When to Use | | --------------------------------- | --------------------------------------------- | --------------------------------------------------------------- | | `inherit_context=True` | The new agent receives the full chat context. | Maintaining continuity so the user does not repeat information. | | `inherit_context=False` (default) | The new agent starts with a fresh state. | Switching to a completely unrelated task. | ## Example A travel agent hands off to a booking specialist from a function tool, passing the conversation context along. The handoff is driven by the tool's **return value**: return the next `Agent` instance (constructed with `inherit_context=True`). ```python title="Python" Python theme={null} from zeroruntime import Agent, function_tool class TravelAgent(Agent): def __init__(self): super().__init__( agent_id="travel", instructions="You are a travel assistant. Help with travel questions and guide users to booking.", ) @function_tool() async def transfer_to_booking(self) -> Agent: """Transfer the user to a booking specialist.""" # Returning an Agent instance performs the handoff; # inherit_context=True passes the existing chat context. return BookingAgent(inherit_context=True) class BookingAgent(Agent): def __init__(self, inherit_context: bool = False): super().__init__( agent_id="booking", instructions="You are a booking specialist. Help users book or modify reservations.", inherit_context=inherit_context, ) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, function_tool } from '@zeroruntime/js-sdk'; class BookingAgent extends Agent { constructor(inherit_context = false) { super({ agent_id: 'booking', instructions: 'You are a booking specialist. Help users book or modify reservations.', inherit_context, }); } } class TravelAgent extends Agent { constructor() { super({ agent_id: 'travel', instructions: 'You are a travel assistant. Help with travel questions and guide users to booking.', }); } transfer_to_booking = function_tool({ name: 'transfer_to_booking', description: 'Transfer the user to a booking specialist.', parameters: {}, // Returning an Agent performs the handoff; inherit_context passes the // existing chat context to the agent taking over. execute: async () => new BookingAgent(true), }); } ``` `inherit_context` controls whether the receiving agent starts with the full chat context (continuity) or a clean slate (unrelated task). ## What's Next Manage conversation history automatically. How ChatContext records conversation memory. ## References #### Examples Switch control between multiple agents. #### Examples Switch control between multiple agents. # Context Control Source: https://docs.zeroruntime.ai/build/context-management/context-control How ChatContext records conversation memory, survives pipeline switches, and powers multi-agent fork. Every `Agent` keeps a `ChatContext`: a structured log of messages (with their tool calls) plus agent handoffs. It’s the single source of truth across cascade and realtime pipelines and between agents, so you can switch pipelines or hand off mid-call without losing history. For automatic summaries and token-budget trimming on long calls, see [Context Window](/build/context-management/context-window). ## What `ChatContext` Records Unlike a plain message list, `ChatContext` keeps an ordered log of `ChatMessage` items (accessible via `.items`) plus a separate list of agent handoffs, so the full history survives a pipeline switch or an agent handoff. | Record | Where It Lives | | :----------------- | :---------------------------------------------------------------------------------------------------------- | | **`ChatMessage`** | User and assistant turns (final transcripts) and system instructions, returned by `.items` and `messages()` | | **`FunctionCall`** | Every tool the agent invoked, with its arguments, embedded in each message's `tool_calls` | | **`AgentHandoff`** | Transfer markers when one agent hands the conversation to another, read via `handoffs()` / `last_handoff()` | ```python title="Python" Python theme={null} # Inspect the agent's context at any time items = await self.session.get_context_history() recent_user_message = items[-1] ``` ```typescript Node JS theme={null} // Inspect the agent's context at any time const items = await this.session!.get_context_history(); const recent_user_message = items[-1]; ``` Python doesn't expose a `ChatContext`-returning accessor yet: `session.fetch_context_history()` returns the same conversation as a plain list of message dicts (`role`, `content`, `message_id`, `tool_calls`, ...) instead of `ChatMessage` objects with `.items` / `.messages()`. `self.chat_context` is reserved for future use and is `None` today. Don't read from it. ## Mid-Call Pipeline Switching You can switch an `AgentSession`'s pipeline mid-call (for example, from a cascade stack, `STT` → `LLM` → `TTS`, to a realtime speech-to-speech model) using `pipeline.change_pipeline(...)`. The agent's `chat_context` is preserved, and the realtime model seeds itself from that context when it connects. ### Idempotency The switch tool stays available on the agent after the switch happens. Without a guard, a realtime model, seeded with a conversation that's all about switching, can loop on the same tool. Track a flag such as `self._switched` and make the tool a safe no-op on repeat calls. ### Supported Realtime Providers `change_pipeline(...)` works with every realtime provider that records back into `ChatContext`: | Provider | Model class | | :------------------- | :----------------- | | Google Gemini Live | `GeminiRealtime` | | OpenAI Realtime (GA) | `OpenAIRealtime` | | xAI | `XAIRealtime` | | Ultravox | `UltravoxRealtime` | Each provider seeds its prior conversation into the realtime session's instructions on connect, so the realtime half starts already aware of what was said and which tools were called. For the full mid-call switch, see [Configure a Pipeline](/build/configure-a-pipeline/overview). ## Multi-Agent Context Patterns When two or more agents share a conversation, `ChatContext` provides primitives for transferring control, isolating sub-agent work, and merging results back. ### Hand Off to a Peer Agent `add_handoff(...)` records a transfer marker on the shared context so the receiving agent's first turn is informed by what the previous agent did and why. ```python title="Python" Python theme={null} # In the intake agent, before transferring to billing self.chat_context.add_handoff( to_agent="billing", from_agent="intake", reason="Caller wants to dispute a charge on order 456.", ) ``` ```typescript Node JS theme={null} // In the intake agent, before transferring to billing this.chat_context.add_handoff('billing', { from_agent: 'intake', reason: 'Caller wants to dispute a charge on order 456.', }); ``` The billing agent reads the handoff marker on takeover and can greet the caller with full context: *"Hi, I see you'd like to dispute the charge on order 456. Let me pull that up."* ## Realtime Tool-Call Recording Realtime tool calls are automatically logged to ChatContext as both a `FunctionCall` and its `FunctionCallOutput` (deduped by `call_id`). This lets a cascade LLM read prior results after a realtime→cascade switch, preventing duplicate tool invocations like re-calling `lookup_order(456)`. No configuration is needed. This happens automatically for every realtime provider listed above. ## What's Next Manage conversation history automatically. Share context across specialized agents. ## References #### Examples Read and pass conversation context. #### Examples Read and pass conversation context. # Context Window Source: https://docs.zeroruntime.ai/build/context-management/context-window Automatically manage conversation history with token and item budgets. Automatically manage history for the `Pipeline`. Configure a `ContextWindow` with token and item budgets. Before each LLM call it summarizes older turns and truncates excess items so your agent preserves key memory without exceeding limits. Context Window compress and truncate cycle Context Window replaces manual context management. All token budgeting, history compression, and truncation is handled automatically through a single configuration object. ## How Context Window Works Context Window is configured on a `Pipeline` instance via the `context_window` parameter. It runs a two-step management cycle before every LLM call: | Step | Action | Purpose | | :-------------- | :--------------------------------------- | :------------------------------------------------------ | | **1. Compress** | Summarize old conversation turns via LLM | Preserve long-term memory without keeping every message | | **2. Truncate** | Remove oldest non-protected items | Enforce hard token and item count limits | Three item types are **always protected** and never removed: | Protected Item | Reason | | :-------------------- | :---------------------------------------------------- | | **System message** | Agent instructions must persist | | **Summary message** | Compressed history is the agent's long-term memory | | **Last user message** | LLMs require the conversation to end with a user turn | ```python title="Python" Python theme={null} from zeroruntime import Pipeline, ContextWindow from zeroruntime.plugins import DeepgramSTT, OpenAILLM, CartesiaTTS, SileroVAD from zeroruntime.inference import AICousticsDenoise, TurnDetector pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=CartesiaTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), # Configure context window management context_window=ContextWindow( max_tokens=4000, max_context_items=20, keep_recent_turns=3, max_tool_calls_per_turn=10, ), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline, ContextWindow } from '@zeroruntime/js-sdk'; import { DeepgramSTT, OpenAILLM, CartesiaTTS, SileroVAD } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: CartesiaTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), // Configure context window management context_window: ContextWindow({ max_tokens: 4000, max_context_items: 20, keep_recent_turns: 3, max_tool_calls_per_turn: 10, }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` ## Configuration Parameters Adjust these parameters to match your application's token limits, cost targets, and expected tool-call patterns. | Parameter | Type | Default | Description | | :------------------------ | :------------ | :------ | :-------------------------------------------------------------------------------------------------------- | | `max_tokens` | `int \| None` | `None` | • Token budget for the full history.
• On overflow, old turns compress, then truncate. | | `max_context_items` | `int \| None` | `None` | • Max items (messages + tool calls + results).
• Either limit triggers compression/truncation. | | `keep_recent_turns` | `int` | `3` | • Recent exchanges kept verbatim.
• Older ones get summarized. | | `max_tool_calls_per_turn` | `int` | `10` | • Max tool calls per user turn.
• Prevents infinite tool-call loops. | | `summary_llm` | `LLM \| None` | `None` | • Optional LLM for summaries.
• Falls back to the main LLM.
• Use a cheaper model to cut costs. | ## Processing Cycle Before each LLM call, an internal process is automatically executed. This process performs two steps in sequence. ### Step 1: Compress When the context exceeds the token or item budget **and** there are enough old turns to compress (more user turns than `keep_recent_turns`), compression kicks in: 1. **Split** - Separate items into old turns and recent turns (keeping the last N user exchanges). 2. **Render** - Convert old items into human-readable text for the summarization prompt. 3. **Summarize** - Call the LLM (or `summary_llm`) to generate a concise summary. 4. **Replace** - Remove all old items and insert the summary as a system message tagged with the internal Summary source (rendered as `[Conversation Summary]`). The summary preserves: * Key facts, names, and numbers * Decisions made and their reasoning * Tool/function call results and outcomes * Commitments or promises the assistant made * User objectives, preferences, and unresolved tasks ### Step 2: Truncate After compression (or if compression wasn't needed), truncation enforces hard limits: 1. Remove the oldest non-protected items one at a time. 2. Function call/output pairs are removed together to avoid orphaned tool calls. 3. Continue until both `max_tokens` and `max_context_items` are satisfied. 4. If only protected items remain, stop even if still over budget. ## How Tool Chaining Works Context Window works with tool chaining. Here's the lifecycle of a multi-tool turn: User says "Plan for Dubai" → LLM returns `get_weather(Dubai)`. Tool executes → result added to context → LLM called again. LLM returns `get_clothing_advice(22°C)` → execute → call LLM again. LLM returns `get_activity_suggestion(22°C, "jacket")` → execute → call LLM. LLM returns text "Dubai is 22°C, wear a jacket, go hiking!" → spoken by TTS. That's 3 tool calls + 1 text response = 4 rounds, well within `max_tool_calls_per_turn=10`. Some LLMs (Anthropic Claude, OpenAI GPT-4o) can return multiple tool calls in a single response. These are collected and executed in parallel using `asyncio.gather`, then all results are added to context before the next LLM call. Google Gemini sends one tool call at a time (always sequential). ## Example A full example combining Context Window with tool chaining for a production-ready travel assistant: ```python title="main.py" Python theme={null} import aiohttp import zeroruntime from zeroruntime import Agent, Pipeline, function_tool, ContextWindow from zeroruntime.plugins import DeepgramSTT, CartesiaTTS, OpenAILLM, SileroVAD from zeroruntime.inference import AICousticsDenoise, TurnDetector AGENT_ID = "travel-assistant" @function_tool async def get_weather(city: str) -> dict: """Get the current weather temperature for a given city.""" city_coords = { "dubai": (25.2048, 55.2708), "mumbai": (19.0760, 72.8777), "new york": (40.7128, -74.0060), } coords = city_coords.get(city.lower(), (25.2048, 55.2708)) lat, lon = coords url = f"https://api.open-meteo.com/v1/forecast?latitude={lat}&longitude={lon}¤t=temperature_2m" async with aiohttp.ClientSession() as session: async with session.get(url) as response: if response.status == 200: data = await response.json() temp = data["current"]["temperature_2m"] return {"city": city, "temperature": temp, "unit": "Celsius"} else: return {"city": city, "temperature": 25, "unit": "Celsius", "note": "fallback"} @function_tool async def get_clothing_advice(temperature: float) -> dict: """Get clothing recommendation based on temperature.""" if temperature > 35: advice = "Very light breathable clothes, hat, and sunscreen." elif temperature > 25: advice = "Light clothes like t-shirt and shorts." elif temperature > 15: advice = "Light jacket or sweater with comfortable pants." elif temperature > 5: advice = "Warm coat, scarf, and layered clothing." else: advice = "Heavy winter coat, gloves, hat, and thermal layers." return {"temperature": temperature, "clothing_advice": advice} pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=CartesiaTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), # ── Context Window Configuration ─────────────────────────── # max_tokens: token budget for the conversation (~8 city plans + chat). # max_context_items: max messages + tool calls before compression/truncation. # keep_recent_turns: recent exchanges kept verbatim; older ones summarized. # max_tool_calls_per_turn: safety limit to prevent infinite tool-call loops. context_window=ContextWindow( max_tokens=4000, max_context_items=20, keep_recent_turns=3, max_tool_calls_per_turn=10, ), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class TravelAgent(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions=( "You are a helpful travel assistant. When a user asks what to do in a city:\n" "1. FIRST call get_weather to get the temperature\n" "2. THEN call get_clothing_advice with that temperature\n" "3. Combine results into a natural spoken response (2-3 sentences max)." ), pipeline=pipeline, tools=[get_weather, get_clothing_advice], ) async def on_enter(self) -> None: await self.session.say("Hi! I'm your travel assistant. Ask me about any city!") async def on_exit(self) -> None: pass if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh TravelAgent + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(TravelAgent, on_ready=lambda: zeroruntime.invoke(AGENT_ID, room=zeroruntime.Room(playground=True))) ``` ```typescript title="main.ts" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, ContextWindow, Pipeline, Room, function_tool } from '@zeroruntime/js-sdk'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; import { CartesiaTTS, DeepgramSTT, OpenAILLM, SileroVAD } from '@zeroruntime/js-sdk/plugins'; const AGENT_ID = 'travel-assistant'; const CITY_COORDS: Record = { dubai: [25.2048, 55.2708], mumbai: [19.076, 72.8777], 'new york': [40.7128, -74.006], }; const get_weather = function_tool({ name: 'get_weather', description: 'Get the current weather temperature for a given city.', parameters: { city: { type: 'string', description: 'The city to look up.' }, }, execute: async ({ city }) => { const [lat, lon] = CITY_COORDS[city.toLowerCase()] ?? CITY_COORDS.dubai; const url = `https://api.open-meteo.com/v1/forecast?latitude=${lat}&longitude=${lon}` + '¤t=temperature_2m'; const response = await fetch(url); if (!response.ok) { return { city, temperature: 25, unit: 'Celsius', note: 'fallback' }; } const data = (await response.json()) as { current: { temperature_2m: number } }; return { city, temperature: data.current.temperature_2m, unit: 'Celsius' }; }, }); const get_clothing_advice = function_tool({ name: 'get_clothing_advice', description: 'Get clothing recommendation based on temperature.', parameters: { temperature: { type: 'number', description: 'The temperature in Celsius.' }, }, execute: async ({ temperature }) => { let advice: string; if (temperature > 35) advice = 'Very light breathable clothes, hat, and sunscreen.'; else if (temperature > 25) advice = 'Light clothes like t-shirt and shorts.'; else if (temperature > 15) advice = 'Light jacket or sweater with comfortable pants.'; else if (temperature > 5) advice = 'Warm coat, scarf, and layered clothing.'; else advice = 'Heavy winter coat, gloves, hat, and thermal layers.'; return { temperature, clothing_advice: advice }; }, }); const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: CartesiaTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), // Context Window Configuration // max_tokens: token budget for the conversation (~8 city plans + chat). // max_context_items: max messages + tool calls before compression/truncation. // keep_recent_turns: recent exchanges kept verbatim; older ones summarized. // max_tool_calls_per_turn: safety limit to prevent infinite tool-call loops. context_window: ContextWindow({ max_tokens: 4000, max_context_items: 20, keep_recent_turns: 3, max_tool_calls_per_turn: 10, }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); class TravelAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful travel assistant. When a user asks what to do in a city:\n' + '1. FIRST call get_weather to get the temperature\n' + '2. THEN call get_clothing_advice with that temperature\n' + '3. Combine results into a natural spoken response (2-3 sentences max).', pipeline, tools: [get_weather, get_clothing_advice], }); } async on_enter(): Promise { await this.session!.say("Hi! I'm your travel assistant. Ask me about any city!"); } } // Pass the class itself (not an instance): serve() builds a fresh TravelAgent + // pipeline per call, which is required for correct per-call state under concurrent calls. await zeroruntime.serve(TravelAgent, { on_ready: () => zeroruntime.invoke(AGENT_ID, { room: Room({ playground: true }) }), }); ``` ## What's Next How ChatContext records conversation memory. Share context across specialized agents. ## References #### Examples Compress and truncate long conversations. #### Examples Compress and truncate long conversations. # Create an Agent Source: https://docs.zeroruntime.ai/build/creating-an-agent Subclass Agent, set its instructions, and wire it into a runnable session. The `Agent` class is the foundation for every voice agent you build. You define a custom agent by subclassing `Agent`, give it instructions, and override its lifecycle hooks to act when it joins or leaves a session. ## Architecture The `Agent` defines behavior and carries its `Pipeline` (the STT, LLM, and TTS components). You register the agent with `zeroruntime.serve()`, and `zeroruntime.invoke()` starts a session for it; the runtime orchestrates both into a running workflow inside a session. Agent orchestrator wired to instructions, function tools, MCP, and its pipeline ## Initialization Every custom agent calls `super().__init__()` inside its own `__init__`, passing its `instructions`, a unique `agent_id` (required: it's the handle callers reach the agent by), and the `pipeline` it runs on. This runs the base `Agent` so it can register lifecycle hooks and tools, prepare internal state, and store the system prompt. ### Instructions Provide a clear system prompt via the `instructions` argument. Describe the agent's role, tone, and constraints; it is sent with every conversation turn. ```python Python theme={null} from zeroruntime import Agent class MyAgent(Agent): def __init__(self, pipeline): super().__init__( agent_id="assistant", instructions="You are a helpful assistant. Keep replies short and friendly.", pipeline=pipeline, ) ``` ```typescript Node JS theme={null} import { Agent } from '@zeroruntime/js-sdk'; class MyAgent extends Agent { constructor(pipeline) { super({ agent_id: 'assistant', instructions: 'You are a helpful assistant. Keep replies short and friendly.', pipeline, }); } } ``` ## Lifecycle Agents provide two async lifecycle hooks you override to act on join and leave: `on_enter` (runs after the agent joins) and `on_exit` (runs before it leaves). Together with `instructions`, both hooks are expected on every agent you define. ### on\_enter() `on_enter` is called once when the agent successfully joins the session. A common use case is greeting participants with `session.say()`. Limitation: it runs only after the room connection succeeds, so it cannot be used for pre-connection setup. ```python Python theme={null} async def on_enter(self): print("Agent has entered the session.") await self.session.say("Hello everyone! I'm here to help.") ``` ```typescript Node JS theme={null} async on_enter() { console.log('Agent has entered the session.'); await this.session!.say("Hello everyone! I'm here to help."); } ``` ### on\_exit() `on_exit` is called when the agent is about to leave the session. A common use case is saying goodbye or running cleanup tasks. Limitation: the session is shutting down, so long-running work may not complete before resources are released. ```python Python theme={null} async def on_exit(self): print("Agent is exiting the session.") await self.session.say("It was a pleasure assisting you. Goodbye!") ``` ```typescript Node JS theme={null} async on_exit() { console.log('Agent is exiting the session.'); await this.session!.say('It was a pleasure assisting you. Goodbye!'); } ``` ## Example A full agent you can run directly. It builds a cascade pipeline, registers the agent with `zeroruntime.serve()`, and starts a playground session with `zeroruntime.invoke()`. ```python title="main.py" Python theme={null} import zeroruntime from zeroruntime import Agent, Pipeline from zeroruntime.plugins import DeepgramSTT, GoogleLLM, CartesiaTTS from zeroruntime.inference import AICousticsDenoise, TurnDetector AGENT_ID = "assistant" pipeline = Pipeline( stt=DeepgramSTT(), llm=GoogleLLM(), tts=CartesiaTTS(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class MyAgent(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions="You are a helpful voice assistant.", pipeline=pipeline, ) async def on_enter(self): await self.session.say("Hello! How can I help you today?") async def on_exit(self): await self.session.say("Goodbye!") if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh MyAgent + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(MyAgent, on_ready=lambda: zeroruntime.invoke(AGENT_ID, room=zeroruntime.Room(playground=True))) ``` ```typescript title="main.ts" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, Pipeline } from '@zeroruntime/js-sdk'; import { DeepgramSTT, GoogleLLM, CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const AGENT_ID = 'assistant'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: GoogleLLM(), tts: CartesiaTTS(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); class MyAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful voice assistant.', pipeline, }); } async on_enter() { await this.session!.say('Hello! How can I help you today?'); } async on_exit() { await this.session!.say('Goodbye!'); } } // Pass the class itself (not an instance): serve() builds a fresh MyAgent + // pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(MyAgent, { on_ready: () => zeroruntime.invoke(AGENT_ID, { room: zeroruntime.Room({ playground: true }) }) }); ``` ## What's Next Wire up STT, LLM, and TTS. Register and serve the agent. ## References #### Examples Minimal STT-LLM-TTS cascade agent you can run. Minimal speech-to-speech realtime agent. #### Examples Minimal STT-LLM-TTS cascade agent you can run. Minimal speech-to-speech realtime agent. # Installation Source: https://docs.zeroruntime.ai/build/installation Learn to install the Zero Runtime Python SDK, connect to the runtime, and build voice AI agents with Python 3.11+. The Zero Runtime Python SDK lets you build voice AI agents on Python 3.11+ with a clean, type-hinted API. ## Prerequisites Before installing, make sure you have: * **Python 3.11 or later** * A **Zero Runtime auth token** — get these from the [Zero Runtime dashboard](https://app.zeroruntime.ai/) * **API keys** for any AI/voice providers you plan to use (e.g., STT, TTS, LLM providers) ## Installation Install the SDK via pip: ```bash theme={null} pip install zeroruntime ``` ## Connect to the Runtime Point the worker at your runtime address, and set your auth token, using the values from the dashboard: ```bash theme={null} export ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 export ZERORUNTIME_AUTH_TOKEN= ``` # Avatar Integration Source: https://docs.zeroruntime.ai/build/modalities/avatars/integration Add a managed photorealistic avatar from Anam or Simli to your agent. The quickest way to add a face is a managed avatar provider. Construct the avatar and pass it to the pipeline's `avatar` slot; the SDK streams the agent's speech to the provider, which renders a lip-synced video track into the room. ## Anam Set your Anam credentials, then construct `AnamAvatar`: ```bash theme={null} export ANAM_API_KEY= ``` ```python title="Python" Python theme={null} from zeroruntime.plugins import AnamAvatar from zeroruntime.inference import AICousticsDenoise avatar = AnamAvatar( avatar_id="", # falls back to ANAM_AVATAR_ID / a default ) pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=ElevenLabsTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), avatar=avatar, denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript title="Node JS" Node JS theme={null} import { AnamAvatar } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const avatar = AnamAvatar({ avatar_id: '', // falls back to ANAM_AVATAR_ID / a default }); const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: ElevenLabsTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), avatar, denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` | Parameter | Type | Default | Description | | :------------- | :---- | :-------- | :---------------------------------------------------- | | `api_key` | `str` | `None` | Anam API key. Falls back to `ANAM_API_KEY`. | | `avatar_id` | `str` | a default | The avatar to render. Falls back to `ANAM_AVATAR_ID`. | | `persona_name` | `str` | `None` | Optional persona name for the avatar. | | `voice_id` | `str` | `None` | Optional voice override for the avatar. | ## Simli Set your Simli credentials, then construct `SimliAvatar` with the face to render: ```bash theme={null} export SIMLI_API_KEY= ``` ```python title="Python" Python theme={null} from zeroruntime.plugins import SimliAvatar from zeroruntime.inference import AICousticsDenoise avatar = SimliAvatar( face_id="", # falls back to SIMLI_FACE_ID / a default ) pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=ElevenLabsTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), avatar=avatar, denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript title="Node JS" Node JS theme={null} import { SimliAvatar } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const avatar = SimliAvatar({ config: { faceId: '' }, // Simli's own config object carries the face }); const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: ElevenLabsTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), avatar, denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` | Parameter | Type | Default | Description | | :----------- | :----- | :-------- | :------------------------------------------------------ | | `api_key` | `str` | `None` | Simli API key. Falls back to `SIMLI_API_KEY`. | | `face_id` | `str` | a default | The face to render. Falls back to `SIMLI_FACE_ID`. | | `model` | `str` | `None` | Optional Simli model override. | | `is_trinity` | `bool` | `False` | Set when the face ID is a Trinity avatar, to keep sync. | # Avatars Source: https://docs.zeroruntime.ai/build/modalities/avatars/overview Give your agent a real-time visual face: a talking avatar that lip-syncs to the agent's speech. An avatar gives your voice agent a **face**: a real-time video presence that lip-syncs to the agent's speech and publishes a video track into the room. Add one by passing an avatar to the pipeline's `avatar` slot; it works with both [Cascade and Realtime](/build/configure-a-pipeline/modes) pipelines. ```python title="Python" Python theme={null} # anam_avatar = AnamAvatar(...) - see Avatar Integration below from zeroruntime.inference import AICousticsDenoise pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=ElevenLabsTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), avatar=anam_avatar, # ← the agent now has a visual face denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript title="Node JS" Node JS theme={null} // anam_avatar = AnamAvatar(...) - see Avatar Integration below import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: ElevenLabsTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), avatar: anam_avatar, // ← the agent now has a visual face denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` Use a managed avatar provider: Anam or Simli. ## Options | Approach | When to use | | :------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------- | | **Managed provider** ([Anam](/build/modalities/avatars/integration), [Simli](/build/modalities/avatars/integration)) | Drop-in photorealistic avatars with one API key. | Avatar providers (Anam and Simli) are available in the Python SDK. # Modalities Source: https://docs.zeroruntime.ai/build/modalities/overview The input and output channels your agent can work with: speech and audio, text, vision, and avatars. A **modality** is a channel your agent uses to perceive or respond. Zero Runtime agents are voice-first, but the same pipeline can take text in, see video, and render a visual avatar, often at the same time. Pick the modalities your experience needs; the pipeline wires them in. Voice in, voice out, plus background audio and TTS caching. Text input and output for chatbots, omnichannel, and debugging. Let the agent see: send camera or screen frames to the model. Give the agent a real-time visual face. ## How modalities map to the pipeline Modalities are selected by the components you pass to the [`Pipeline`](/build/configure-a-pipeline/overview) and by per-session flags on `zeroruntime.Room(...)`: | Modality | Turned on by | | :--------------- | :---------------------------------------------------------------------------- | | Speech & audio | `stt` + `tts` (or a realtime model), the default voice loop | | Text | Sending text with `pipeline.process_text(...)`; LLM-only or LLM+TTS pipelines | | Vision | `zeroruntime.Room(vision=True)` + `session.reply(..., frames=N)` | | Avatars | `Pipeline(avatar=...)` | | Background audio | `zeroruntime.Room(background_audio=True)` | Modalities compose: a pipeline can run voice, vision, and an avatar together. ## References #### Examples Combine speech, text and vision in one agent. Voice agent built from a cascade pipeline. Speech-to-speech realtime agent. #### Examples Combine speech, text and vision in one agent. Voice agent built from a cascade pipeline. Speech-to-speech realtime agent. # Audio Customization Source: https://docs.zeroruntime.ai/build/modalities/speech-and-audio/audio-customization Choose the agent's voice and tune how it sounds (speed, pitch, emotion) through the TTS plugin. How the agent sounds is controlled by the [text-to-speech](/plugins/tts/cartesia) plugin in the pipeline. Each TTS provider exposes a voice plus tuning knobs; set them on the plugin's constructor. ## Choose a voice Pass the provider's voice identifier. Voices and the exact option names vary by provider. See each [TTS plugin page](/plugins/overview) for the full list. ```python Python theme={null} from zeroruntime.plugins import CartesiaTTS tts = CartesiaTTS( voice="", generation_config={"speed": 1.0}, ) # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; const tts = CartesiaTTS({ voice: '', generation_config: {speed: 1.0}, }); // Pipeline({ tts, ... }) ``` ## Common tuning knobs The exact set depends on the provider, but most TTS plugins support some of: | Knob | Effect | | :------------------ | :---------------------------------------------- | | `voice` / `speaker` | The voice identity. | | `speed` / `pace` | How fast the agent talks. | | `pitch` | Higher or lower voice. | | `emotion` | Emotional tone, where the provider supports it. | | `language` | Spoken language / accent. | | `sample_rate` | Output audio sample rate. | For voice agents, lower latency usually beats richer prosody. Prefer a fast model and streaming synthesis, and keep replies short. ## Speaking on demand Make the agent speak a specific line at any time with `session.say()`: ```python title="Python" Python theme={null} await self.session.say("Let me look that up for you.") ``` ```typescript title="Node JS" Node JS theme={null} await this.session!.say('Let me look that up for you.'); ``` To fix how specific words are pronounced (acronyms, brand names), transform the text before it reaches TTS. See [Text Transformation](/build/modalities/text/transformation). ## References #### Examples Customize pronunciation of names and terms. Switch speech languages within an agent. #### Examples Customize pronunciation of names and terms. Switch speech languages within an agent. # Background Audio Source: https://docs.zeroruntime.ai/build/modalities/speech-and-audio/background-audio Play thinking sounds while the agent generates and ambient music or hold audio during a call. Background audio fills the silences. The agent can play a subtle **thinking sound** while the LLM is generating, and **ambient audio** (hold music, office noise) on demand, so the call never feels dead. Any libav-decodable file works: WAV, MP3, Ogg/Vorbis, Ogg/Opus, FLAC, M4A/AAC. Pass `background_audio=True` to `zeroruntime.serve(room= zeroruntime.Room(...))`. An explicit file URL is required. An unset or empty file disables the audio. Pass `background_audio: true` to `zeroruntime.serve(Agent, { room: Room({ ... }) })`. An explicit file URL is required. An unset or empty file disables the audio. ## Thinking audio `set_thinking_audio()` plays a short sound while the agent is thinking (LLM generation). Call it as the agent enters, in the constructor. Provide a `file` to play; an unset file disables the audio. ```python title="Python" Python theme={null} class VoiceAgent(Agent): def __init__(self, pipeline): super().__init__( agent_id="assistant", instructions="You are a helpful assistant.", pipeline=pipeline, ) self.set_thinking_audio( file="https://cdn.zeroruntime.ai/zrt/bg-audio/bg-noise-1.wav", volume=0.3, ) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent } from '@zeroruntime/js-sdk'; class VoiceAgent extends Agent { constructor(pipeline) { super({ agent_id: 'assistant', instructions: 'You are a helpful assistant.', pipeline, }); } // The thinking sound is set on the live session, so it is armed once the // agent has joined rather than in the constructor. async on_enter(): Promise { await this.session!.set_thinking_audio( 'https://cdn.zeroruntime.ai/zrt/bg-audio/bg-noise-1.wav', { volume: 0.3 }, ); } } ``` ## Ambient / background music Start and stop ambient audio on demand, for example, from a function tool the LLM can call: ```python title="Python" Python theme={null} @function_tool async def control_background_music(self, action: str): """Play or stop background music. action: 'play' or 'stop'.""" if action == "play": await self.play_background_audio( file="https://cdn.zeroruntime.ai/zrt/bg-audio/bg-noise-1.wav", volume=0.8, looping=True, override_thinking=False, ) return "Music started." await self.stop_background_audio() return "Music stopped." ``` ```typescript title="Node JS" Node JS theme={null} import { current_session, function_tool } from '@zeroruntime/js-sdk'; const control_background_music = function_tool({ name: 'control_background_music', description: "Play or stop background music. action: 'play' or 'stop'.", parameters: { action: { type: 'string', description: "Either 'play' or 'stop'." }, }, // Background audio is a session call, so this reads the live session rather // than the agent. Hold the tool on an agent field and `this` is bound for you. execute: async ({ action }) => { const session = current_session(); if (action === 'play') { await session.play_background_audio( 'https://cdn.zeroruntime.ai/zrt/bg-audio/bg-noise-1.wav', { volume: 0.8, looping: true, override_thinking: false }, ); return 'Music started.'; } await session.stop_background_audio(); return 'Music stopped.'; }, }); ``` ## Parameters `set_thinking_audio(file=None, volume=0.3)` | Parameter | Type | Default | Description | | :-------- | :------ | :------ | :----------------------------------------------------------------------------------------- | | `file` | `str` | `None` | Audio file to play while the agent generates a reply. Required. An unset file disables it. | | `volume` | `float` | `0.3` | Playback volume. | `play_background_audio(file=None, volume=1.0, looping=False, override_thinking=True)` | Parameter | Type | Default | Description | | :------------------ | :------ | :------ | :---------------------------------------------------------------------------------------------------------------------- | | `file` | `str` | `None` | Audio file to play in the background. Required. An unset file disables it. | | `volume` | `float` | `1.0` | Playback volume. | | `looping` | `bool` | `False` | Loop the file until stopped. | | `override_thinking` | `bool` | `True` | `True`: thinking audio layers over the music. `False`: music is exclusive and suppresses thinking audio while it plays. | Call `stop_background_audio()` to stop ambient playback. ## What's Next Tune the agent's voice. Browse vision, text, and avatar modalities. ## References #### Examples Play ambient or hold audio during a call. #### Examples Play ambient or hold audio during a call. # Speech & Audio Source: https://docs.zeroruntime.ai/build/modalities/speech-and-audio/overview The voice-first modality (speech in, speech out) plus background audio and TTS caching. Speech is the default modality. The caller speaks, [speech-to-text](/plugins/stt/deepgram) transcribes, the [LLM](/plugins/llm/google) replies, and [text-to-speech](/plugins/tts/cartesia) voices the answer, or a single [realtime](/build/configure-a-pipeline/modes) model does it end to end. Beyond the core loop, this modality covers how the agent *sounds* and the audio it plays around the conversation. Choose the voice and tune speed, pitch, and emotion. Play thinking sounds and ambient music during a call. Browse the speech-to-text and text-to-speech providers. ## The voice loop ```python title="Python" Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import DeepgramSTT, OpenAILLM, CartesiaTTS, SileroVAD from zeroruntime.inference import AICousticsDenoise, TurnDetector pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=CartesiaTTS(), vad=SileroVAD(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { DeepgramSTT, OpenAILLM, CartesiaTTS, SileroVAD } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: CartesiaTTS(), vad: SileroVAD(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` To swap voice providers, change the `stt`/`tts` plugins. To collapse the loop into one low-latency model, pass a realtime model as `llm`. See [Pipeline Modes](/build/configure-a-pipeline/modes). ## References #### Examples Play ambient or hold audio during a call. Tune how the agent pronounces words. #### Examples Play ambient or hold audio during a call. Tune how the agent pronounces words. # TTS Caching Source: https://docs.zeroruntime.ai/build/modalities/speech-and-audio/tts-caching Synthesize fixed phrases once and replay the audio, skipping the TTS round trip on every call. Many phrases in a voice agent never change: the greeting, the "let me check that for you" hold message, the goodbye. Synthesizing them through a TTS provider on every call costs roughly 300 to 800 ms per phrase and burns provider credits. Synthesize each one **once**, keep the PCM bytes, and replay them on every subsequent `session.say()`. ## How it works `session.say()` accepts an `audio_data` argument: pre-synthesized PCM that bypasses TTS entirely. When it is set, the runtime plays your bytes instead of calling the provider, so playback starts immediately. `text` is still required alongside `audio_data`. It is what lands in the transcript and the chat context; the audio is only what the caller hears. The bytes must be PCM in the room's audio format. What produces them is up to you: your TTS vendor's SDK, a file you decoded ahead of time, or a fetch from your own storage. The runtime plays what it is given. ## Cache the phrases Wrap a synthesizer in a cache that stores each phrase on first use. Later calls for the same phrase return the stored bytes without touching the provider, concurrent calls for one phrase share a single synthesis, and entries fall out LRU past `max_entries`. ```python title="Python" Python theme={null} import asyncio import hashlib from collections import OrderedDict from typing import Awaitable, Callable class TTSAudioCache: """Reuse-by-key cache for TTS-synthesized audio.""" def __init__( self, synthesize: Callable[[str], Awaitable[bytes]], max_entries: int = 128, ) -> None: self._synthesize = synthesize self._max_entries = max_entries self._store: OrderedDict[str, bytes] = OrderedDict() self._locks: dict[str, asyncio.Lock] = {} async def fetch(self, text: str) -> bytes: key = hashlib.sha256(text.encode()).hexdigest() cached = self._store.get(key) if cached is not None: self._store.move_to_end(key) return cached # One synthesis per phrase, even if several turns ask at once. async with self._locks.setdefault(key, asyncio.Lock()): cached = self._store.get(key) if cached is not None: self._store.move_to_end(key) return cached audio = await self._synthesize(text) self._store[key] = audio self._store.move_to_end(key) while len(self._store) > self._max_entries: self._store.popitem(last=False) return audio async def preload(self, texts: list[str]) -> None: """Synthesize and cache a batch up front, e.g. at startup.""" for text in texts: await self.fetch(text) self._locks.clear() ``` Warm the fixed phrases before the first caller arrives, so nobody pays the synthesis: ```python title="Python" Python theme={null} cache = TTSAudioCache(synthesize) if __name__ == "__main__": asyncio.run(cache.preload([GREETING, HOLD, GOODBYE])) zeroruntime.serve(SupportAgent, on_ready=on_ready) ``` ## Supplying the synthesizer The pipeline's TTS plugin is a configuration object, not a synthesizer: synthesis happens on the runtime, so the plugin cannot produce audio in your process. The cache needs its own coroutine that returns raw PCM at the agent track's sample rate, 24 kHz mono 16-bit signed. Name the model and voice once and pass them to both the plugin and the synthesizer, or cached lines will sound like a different speaker than the agent's dynamic speech. ```python title="Python" Python theme={null} from cartesia import AsyncCartesia TTS_MODEL, TTS_VOICE, TTS_LANGUAGE = "sonic-2", "", "en" client = AsyncCartesia(api_key=os.environ["CARTESIA_API_KEY"]) async def synthesize(text: str) -> bytes: chunks = [] async for chunk in client.tts.bytes( model_id=TTS_MODEL, transcript=text, voice={"mode": "id", "id": TTS_VOICE}, language=TTS_LANGUAGE, output_format={ "container": "raw", "encoding": "pcm_s16le", "sample_rate": 24_000, }, ): chunks.append(chunk) return b"".join(chunks) pipeline = Pipeline( tts=CartesiaTTS(model=TTS_MODEL, voice=TTS_VOICE, language=TTS_LANGUAGE), # ...stt, llm, vad, turn_detector ) ``` Any vendor works. The cache only needs a coroutine returning raw PCM at the track's sample rate. ## Replay in `on_enter` and `on_exit` The agent's fixed opening and closing lines are the clearest win: both are known before the call starts, and both sit on the critical path where latency is most audible. ```python title="Python" Python theme={null} class SupportAgent(Agent): async def on_enter(self) -> None: await self.session.say(GREETING, audio_data=await cache.fetch(GREETING)) async def on_exit(self) -> None: await self.session.say(GOODBYE, audio_data=await cache.fetch(GOODBYE)) ``` ## Overlap a hold phrase with a slow operation Cached audio suits filler speech during database lookups, API calls, or RAG retrieval. Start the phrase as a task and let it play while the work runs, rather than waiting for it to finish first. Pass `add_to_chat_context=False` so the filler line does not enter the LLM's context. ```python title="Python" Python theme={null} @function_tool async def check_order_status(self, order_id: str) -> dict: """Look up an order. Args: order_id: The order number the caller gives you. """ hold_audio = await cache.fetch(HOLD) hold = asyncio.create_task( self.session.say(HOLD, audio_data=hold_audio, add_to_chat_context=False) ) order = await db.get_order(order_id) await hold return order ``` ## Pre-recorded audio The same slot plays produced or branded audio, a recorded human voice or an IVR jingle. Decode the file to 24 kHz mono 16-bit PCM first, then pass the bytes; no TTS provider is involved either way. ```python title="Python" Python theme={null} with wave.open("greeting.wav", "rb") as wav: audio = wav.readframes(wav.getnframes()) await self.session.say(GREETING, audio_data=audio) ``` ## Parameters `session.say(text, *, interrupt=False, interruptible=None, add_to_chat_context=None, audio_data=None)` | Parameter | Type | Default | Description | | :-------------------- | :------ | :------ | :-------------------------------------------------------------------------------------------------------------------------- | | `text` | `str` | — | The line to speak. Required even when `audio_data` is set: it is what enters the transcript and chat context. | | `audio_data` | `bytes` | `None` | Pre-synthesized PCM in the room's audio format. Bypasses TTS entirely. Unset means the runtime synthesizes `text` normally. | | `add_to_chat_context` | `bool` | `True` | Whether the line joins the chat context. Set `False` for filler and hold phrases. | | `interrupt` | `bool` | `False` | Cut off whatever is playing before speaking this. | | `interruptible` | `bool` | `True` | Whether the caller can barge in over this line. | ## What's Next Play thinking sounds and ambient audio. Tune the agent's voice. ## References #### Examples Replay fixed phrases without a TTS round trip. #### Examples Replay fixed phrases without a TTS round trip. # Chat Source: https://docs.zeroruntime.ai/build/modalities/text/chat Drive the agent with text instead of speech, and exchange real-time messages with clients over room pub/sub. The pipeline isn't limited to voice. You can feed it **text** and have it reply with text or speech, and you can exchange **real-time messages** with client apps over the room's pub/sub channel. Together these power chat widgets, omnichannel agents, and command channels (a "capture frame" button, a menu choice) alongside the voice conversation. ## Send text in Push text into the live session with `session.process_text()`: the LLM treats it as if the user said it. Pair it with a text-capable [pipeline](/build/modalities/text/overview) (`llm` only, or `llm` + `tts`). ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import GoogleLLM, CartesiaTTS # Text in, voice out pipeline = Pipeline(llm=GoogleLLM(), tts=CartesiaTTS()) # ... when a text message arrives, feed it to the live session: await self.session.process_text("What are your opening hours?") ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { GoogleLLM, CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; // Text in, voice out const pipeline = Pipeline({ llm: GoogleLLM(), tts: CartesiaTTS() }); // ... when a text message arrives, feed it to the live session: await this.session!.process_text('What are your opening hours?'); ``` For a pure text chatbot (text in, text out), use an `llm`-only pipeline and read replies from the `llm` event: ```python Python theme={null} pipeline = Pipeline(llm=GoogleLLM()) def on_reply(data): print(f"Agent: {data['text']}") pipeline.on("llm", on_reply) # `process_text` lives on the live session, not on the pipeline: await session.process_text("Hello!") ``` ```typescript Node JS theme={null} const pipeline = Pipeline({ llm: GoogleLLM() }); function on_reply(data) { console.log(`Agent: ${data['text']}`); } pipeline.on('llm', on_reply); // `process_text` lives on the live session, not on the pipeline: await session.process_text('Hello!'); ``` ## Room pub/sub messaging Pub/sub lets the agent and your clients exchange messages on named **topics** in the room: push data to clients, receive commands, or run a side text channel next to the voice call. ### Publish a message Publish through the session. This works well inside a `function_tool`, so the LLM itself can send messages: ```python Python theme={null} from zeroruntime import PubSubPublishConfig # Typically called from a function_tool, e.g. self.session inside the agent: await self.session.publish_to_pubsub( PubSubPublishConfig(topic="CHAT", message="Hello from the agent") ) ``` ```typescript Node JS theme={null} import { PubSubPublishConfig } from '@zeroruntime/js-sdk'; // Typically called from a function_tool, e.g. this.session inside the agent: await this.session!.publish_to_pubsub( PubSubPublishConfig({ topic: 'CHAT', message: 'Hello from the agent' }), ); ``` ### Subscribe to a topic Subscribe from `on_enter`, once the session exists, and hand it the method that should receive each frame: ```python Python theme={null} from zeroruntime import Agent, PubSubSubscribeConfig class ChatAgent(Agent): async def on_enter(self) -> None: await self.session.subscribe_to_pubsub( PubSubSubscribeConfig(topic="CHAT", cb=self.on_chat) ) async def on_chat(self, frame: dict, backlog: bool) -> None: # Subscribing replays the topic's history, so `backlog` is what # separates it from anything sent since -- usually worth skipping. if backlog: return print("Received:", frame.get("message")) ``` ```typescript Node JS theme={null} import { Agent, PubSubSubscribeConfig } from '@zeroruntime/js-sdk'; class ChatAgent extends Agent { async on_enter(): Promise { await this.session!.subscribe_to_pubsub( PubSubSubscribeConfig({ topic: 'CHAT', cb: this.on_chat.bind(this) }), ); } async on_chat(frame: Record, backlog: boolean): Promise { // Subscribing replays the topic's history, so `backlog` is what // separates it from anything sent since -- usually worth skipping. if (backlog) { return; } console.log('Received:', frame.message); } } ``` A common pattern wires a client message to agent behavior: for example, the client publishes `"capture_frames"` and the agent responds by capturing [vision](/build/modalities/vision/image-input) frames, or forwards inbound chat text to `process_text()`. ### Common topics | Topic | Used by | | :----------- | :--------------------------------------------------------------------- | | `CHAT` | App-defined messaging between client and agent. | | `DTMF_EVENT` | Keypad presses, consumed by the [DTMF handler](/build/telephony/dtmf). | ## What's Next Rewrite the text stream before it's spoken Trigger frame captures from a client message. ## References #### Examples Exchange text messages over pub/sub. #### Examples Exchange text messages over pub/sub. # Text Source: https://docs.zeroruntime.ai/build/modalities/text/overview Text input and output: chatbots, text-driven agents, and transforming text in the pipeline. The pipeline isn't limited to voice. You can feed it **text** and have it reply with text or speech, which is useful for chat widgets, omnichannel agents that share one brain across voice and chat, and for testing an agent from the terminal. Send text in and receive text or voice out. Rewrite the text stream before TTS: pronunciation, filtering, normalization. ## Text-capable pipelines Drop the STT and/or TTS stages to run text in or out: | Pipeline | Behavior | | :-------------------- | :---------------------------------- | | `llm` only | Text in, text out (a pure chatbot). | | `llm` + `tts` | Text in, voice out. | | `stt` + `llm` | Voice in, text out. | | `stt` + `llm` + `tts` | The full voice loop. | See [Pipeline Modes](/build/configure-a-pipeline/modes) for how the pipeline auto-detects the configuration from the components you pass. # Transformation Source: https://docs.zeroruntime.ai/build/modalities/text/transformation Rewrite the text stream before it's spoken (fix pronunciation, filter content, or normalize formatting) with a pipeline hook. Sometimes the LLM's text needs a touch-up before it's spoken: expand an acronym so TTS says it correctly, strip markdown, or filter sensitive content. The `tts` [pipeline hook](/build/configure-a-pipeline/runtime-hooks) gives you the reply's text stream so you can transform it before synthesis. ## The `tts` hook Register an async hook with `@pipeline.on("tts")`. It receives the streaming reply text; yield the transformed text, then hand it to `run_tts()` to produce the audio. ```python title="Python" Python theme={null} import re from zeroruntime import run_tts pronunciation_map = {"nginx": "engine x", "API": "A P I", "SQL": "sequel"} @pipeline.on("tts") async def tts_node(text_stream): async def text_phase(): async for response in text_stream: processed = response for word, say_as in pronunciation_map.items(): processed = re.sub(rf"\b{word}\b", say_as, processed, flags=re.IGNORECASE) yield processed async for audio_chunk in run_tts(text_phase()): yield audio_chunk ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline, PronunciationRule } from '@zeroruntime/js-sdk'; // The Node JS SDK has no `tts` hook: substitutions are declared on the pipeline // and applied in the agent process between the LLM and TTS, so nothing crosses // the wire per chunk and the turn pays nothing for them. const pipeline = Pipeline({ // ...stt, llm, tts pronunciations: [ PronunciationRule('nginx', 'engine x'), PronunciationRule('API', 'A P I'), PronunciationRule('SQL', 'sequel'), ], }); ``` ## What you can do in the hook * **Fix pronunciation** of acronyms, brand names, and technical terms (shown above). * **Filter or redact** content before it's spoken. * **Normalize formatting**: strip markdown, expand numbers/dates, clean up symbols. Because the hook sits between the LLM and TTS, the change affects only what's *spoken*: the text recorded in [chat context](/build/context-management/access-context) is unchanged. This is one of several [pipeline hooks](/build/configure-a-pipeline/runtime-hooks). The same mechanism lets you intercept other stages of the pipeline. ## References #### Examples Rewrite text before synthesis. #### Examples Rewrite text before synthesis. # Image Input Source: https://docs.zeroruntime.ai/build/modalities/vision/image-input Capture video frames on demand and send them to the LLM with a prompt. For a snapshot-style "look at this" interaction, ask for frames on the reply itself: `session.reply()` takes a `frames` count and the runtime shows the model that many of the newest camera frames alongside your prompt. The count travels, not the pixels. This works in [Cascade](/build/configure-a-pipeline/modes) pipelines and is the simplest way to add vision. ## Capture and send ```python title="Python" Python theme={null} # Show the model the two most recent frames and ask it to describe them await session.reply( "Describe what you see in this frame in one sentence.", frames=2, ) ``` ```typescript title="Node JS" Node JS theme={null} // Show the model the two most recent frames and ask it to describe them await session.reply('Describe what you see in this frame in one sentence.', { frames: 2, }); ``` `frames` is a count, not a list of images -- at most `Session.MAX_FRAMES` (5). A negative count or one above the maximum raises `ValueError`. Capturing needs `Room(vision=True)`; without it there is no video track to capture from. ## Trigger a capture A common pattern is to capture when the client sends a [pub/sub](/build/modalities/text/chat) message, for example, a "capture" button in your app: ```python title="Python" Python theme={null} from zeroruntime import Agent, PubSubSubscribeConfig, Room class VisionAgent(Agent): async def on_enter(self) -> None: await self.session.subscribe_to_pubsub( PubSubSubscribeConfig(topic="CHAT", cb=self.on_chat) ) async def on_chat(self, frame: dict, backlog: bool) -> None: if backlog or frame.get("message") != "capture_frames": return await self.session.reply( "Analyze this frame and describe what you see.", frames=2 ) # Turn the video track on; the topic is subscribed once the session exists. zeroruntime.invoke(AGENT_ID, room=Room(vision=True)) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, PubSubSubscribeConfig, Room } from '@zeroruntime/js-sdk'; class VisionAgent extends Agent { async on_enter(): Promise { await this.session!.subscribe_to_pubsub( PubSubSubscribeConfig({ topic: 'CHAT', cb: this.on_chat.bind(this) }), ); } async on_chat(frame: Record, backlog: boolean): Promise { if (backlog || frame.message !== 'capture_frames') { return; } await this.session!.reply('Analyze this frame and describe what you see.', { frames: 2 }); } } // Turn the video track on; the topic is subscribed once the session exists. zeroruntime.invoke(AGENT_ID, { room: Room({ vision: true }) }); ``` You can also build `ImageContent` directly from PIL images, NumPy arrays, or `av.VideoFrame`s for fully custom flows. ## Encoding Before frames reach the model they're encoded with `EncodeOptions`: JPEG by default, resized (default `1024×1024`) and compressed (`quality=75`). Raise these when you need higher fidelity (for example, to read fine text on camera) at the cost of more tokens and latency. ## References #### Examples Send images to a cascade agent. #### Examples Send images to a cascade agent. # Vision Source: https://docs.zeroruntime.ai/build/modalities/vision/overview Give your agent eyes: send camera or screen frames to the model so it can describe, inspect, or reason about what it sees. Vision lets your agent **see**. With vision enabled, it can pull frames from the participant's video and send them to the model alongside a prompt, to describe a scene, read a document on camera, or inspect a product. It works in both [Cascade and Realtime](/build/configure-a-pipeline/modes) modes. Turn on the video subscription per session with the vision option on `serve()` (applies to every caller) or on the room you invoke. ```python title="Python" Python theme={null} zeroruntime.serve(VisionAgent, vision=True) # or, per room: zeroruntime.invoke(AGENT_ID, room=zeroruntime.Room(playground=True, vision=True)) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; // In Node JS, vision is a Room field. Set it on the Room every dispatched call // gets, or on the Room you invoke with. await zeroruntime.serve(VisionAgent, { room: Room({ vision: true }) }); // or, per room: await zeroruntime.invoke(AGENT_ID, { room: Room({ playground: true, vision: true }) }); ``` Capture frames on demand and send them to the LLM. Continuous vision with a realtime model, from camera or screen share. ## How it works Frames are captured as `av.VideoFrame` objects and encoded to JPEG (resized and compressed) before they reach the model. Whether you snapshot a frame on demand or stream video to a realtime model, the agent reasons over the image together with the conversation. ## What's Next Compare Cascade and Realtime pipeline modes. Browse speech, text, and avatar modalities. ## References #### Examples Capture camera frames and describe them with a vision-capable LLM. #### Examples Capture camera frames and describe them with a vision-capable LLM. # Video Input Source: https://docs.zeroruntime.ai/build/modalities/vision/video-input Stream live video to a realtime model for continuous vision, from the participant's camera or a shared screen. For continuous "watch what I'm doing" interactions, pair vision with a [realtime](/build/configure-a-pipeline/modes) model. The model receives video frames as the conversation flows, so the agent can react to what it sees in real time (no explicit capture step needed). Video input rides on whatever video track the participant publishes. That can be their **camera** or a **screen share**; the agent sees either the same way. ## Realtime vision Use a realtime model in the pipeline and turn on `vision=True` for the session, on the `zeroruntime.Room` you invoke (pass the same `Room` to `zeroruntime.serve()` to apply it to every caller). Frames from the participant's video are forwarded to the model alongside audio: ```python title="main.py" Python theme={null} import zeroruntime from zeroruntime import Agent, Pipeline from zeroruntime.plugins import OpenAIRealtime from zeroruntime.inference import AICousticsDenoise AGENT_ID = "vision-agent" pipeline = Pipeline( llm=OpenAIRealtime( model="gpt-4o-realtime-preview", config={"voice": "alloy"} ), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class VisionAgent(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions="You can see the participant's video. Describe what you observe.", pipeline=pipeline, ) if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh VisionAgent + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve( VisionAgent, on_ready=lambda: zeroruntime.invoke( AGENT_ID, room=zeroruntime.Room(playground=True, vision=True) ), ) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, Pipeline } from '@zeroruntime/js-sdk'; import { OpenAIRealtime } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const AGENT_ID = 'vision-agent'; const pipeline = Pipeline({ llm: OpenAIRealtime({ model: 'gpt-4o-realtime-preview', config: {voice: 'alloy'}, }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); class VisionAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: "You can see the participant's video. Describe what you observe.", pipeline, }); } } // Pass the class itself (not an instance): serve() builds a fresh VisionAgent + // pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(VisionAgent, { on_ready: () => zeroruntime.invoke(AGENT_ID, { room: zeroruntime.Room({ playground: true, vision: true }) }), }); ``` For on-demand snapshots in a cascade pipeline instead, see [Image Input](/build/modalities/vision/image-input). ## References #### Examples Stream video frames to a realtime agent. Hook into frames before they reach the model. #### Examples Stream video frames to a realtime agent. Hook into frames before they reach the model. # Observability Options Source: https://docs.zeroruntime.ai/build/observability-and-monitoring/analytics/observability-options Configure recording. Traces and metrics are captured for every session and shown on the Zero Runtime Dashboard. Zero Runtime captures recording, traces, and metrics for every session. Recording is configured directly on `serve()` and on the per-session `Room` passed to `invoke()`. Traces and metrics are collected automatically and shown on the Zero Runtime Dashboard with no extra setup. | Capability | Where you configure it | | ---------------- | ------------------------------------------------------------------------------------------------------- | | Recording | `serve(recording=...)` or the per-session `Room(recording=True)` / `invoke(..., recording_config=...)`. | | Traces & metrics | Collected automatically for every session and shown on the Dashboard. | ## Quick Start Record sessions (audio + video): ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Room AGENT_ID = "assistant" if __name__ == "__main__": zeroruntime.serve( Assistant, room=Room(recording=True, recording_video=True), on_ready=lambda: zeroruntime.invoke( AGENT_ID, room=Room( playground=True, recording=True, ), ), ) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; const AGENT_ID = 'assistant'; zeroruntime.serve(Assistant, { room: Room({ recording: true, recording_video: true }), on_ready: () => zeroruntime.invoke(AGENT_ID, { room: Room({ playground: true, recording: true, }), }), }); ``` ## Recording Audio is recorded by default; video and screen share are opt-in. Set the extra stream fields on the `Room` you pass to `serve()` or `invoke()` to set defaults for every session. See [Recording](/build/observability-and-monitoring/recording) for the full lifecycle. | Parameter | Type | Default | Description | | ------------------------ | ------ | ------- | ------------------------------------------------------------------------- | | `recording_video` | `bool` | `False` | Record the agent's camera video track (composite audio + video). | | `recording_screen_share` | `bool` | `False` | Record the screen-share track. Requires `vision=True` on the same `Room`. | ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Room zeroruntime.serve( Assistant, room=Room( vision=True, recording=True, recording_video=True, recording_screen_share=True, ), on_ready=lambda: zeroruntime.invoke("assistant", room=Room(playground=True)), ) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; zeroruntime.serve(Assistant, { room: Room({ vision: true, recording: true, recording_video: true, recording_screen_share: true, }), on_ready: () => zeroruntime.invoke('assistant', { room: Room({ playground: true }) }), }); ``` ## Traces & metrics Traces and metrics are collected for every session and shown on the Zero Runtime Dashboard automatically (no configuration needed). For programmatic, in-process access to metrics, use Pipeline Observability hooks (`@pipeline.metrics.on(...)`). ## What's Next Review sessions on the Dashboard. Drill into per-turn traces and spans. ## References #### Examples Configure what is observed in a session. #### Examples Configure what is observed in a session. # Overview Source: https://docs.zeroruntime.ai/build/observability-and-monitoring/analytics/overview Inspect agent telemetry (traces, metrics, and logs) on the Dashboard. Zero Runtime's observability tools give you deep insight into your agent's performance and behavior. Every session emits telemetry automatically. The Dashboard collects and visualizes it by default, with no extra setup. ## What Gets Captured Every agent session emits three signals: | Signal | Captures | | ------- | ------------------------------------------------ | | Traces | Spans for STT, LLM, TTS, EOU, and tool calls. | | Metrics | Latency percentiles, durations, and token usage. | | Logs | Runtime log records, filtered by level. | Recording captures the session audio and video alongside this telemetry. See [Recording](/build/observability-and-monitoring/recording). ## What's Next Capture session audio and video alongside this telemetry. Configure recording. Traces, metrics, and logs are captured for every session and shown on the Dashboard. Review per-session metrics, transcripts, and recording playback on the Dashboard. Break a session into traces and spans to find bottlenecks and debug errors. ## References #### Examples Collect metrics and traces from a session. #### Examples Collect metrics and traces from a session. # Session Analytics Source: https://docs.zeroruntime.ai/build/observability-and-monitoring/analytics/session-analytics Review per-session metrics, transcripts, and recording playback on the Dashboard. The **Sessions** view on the Zero Runtime Dashboard lists every conversation, each one a unique user-and-agent pairing. Use it to spot slow sessions, frequent interruptions, and failures at a glance. ## Sessions View | Column | Meaning | | --------------------- | --------------------------------------------------------------------------------------------------- | | `Session ID` | Unique identifier for the conversation. | | `Room ID` | The room the session ran in. | | `TTFW` | Time to First Word: how long the agent takes to say its first word after the user stops speaking. | | `P50` / `P90` / `P95` | Response latency percentiles. P90, for example, is the latency that 90% of responses come in under. | | `Interruption` | Number of times the user interrupted the agent. | | `Duration` | Total length of the session. | | `Recording` | Whether the session was recorded, with playback when available. | | `Created At` | When the session started. | | `Actions` | Open **View Analytics** for the full session breakdown. | Sessions list showing Session ID, Room ID, TTFW, latency percentiles, interruptions, duration, and recording ## Session View Click **View Analytics** to open the **Session view**. It shows the full transcript with timestamps and speaker labels (Caller and Agent), and plays back the recording with an autoscrolling transcript, so you can review the experience and spot areas to improve. Session view showing the conversation transcript with timestamps and speaker labels alongside recording playback ## What's Next Drill into per-turn traces and spans. Configure recording and review telemetry on the Dashboard. ## References #### Examples Gather per-session analytics. #### Examples Gather per-session analytics. # Trace Insights Source: https://docs.zeroruntime.ai/build/observability-and-monitoring/analytics/traces Break a session into traces and spans to find bottlenecks and debug errors. The **Trace View** on the Dashboard breaks a session into a hierarchy of traces and spans for granular analysis. Use it to find where time goes in a turn and to debug errors across STT, LLM, TTS, and tool calls. The Trace View offers an even deeper level of insight, breaking down the entire session into a hierarchical structure of traces and spans. ## Session Configuration At the top level, the **Session Configuration** details every parameter the agent was initialized with: the STT, LLM, and TTS models, plus any function tools and MCP tools. Use it to reproduce and debug agent behavior. Trace view hierarchy with the Session Configuration node selected, showing STT, LLM, and TTS models and function and MCP tools in the Properties panel ## User & Agent Turns The core of the Trace View is the breakdown of the conversation into **User & Agent Turns**. Each turn is a single exchange between the user and the agent, with a detailed timeline of these spans: | Span | Measures | | ------------------- | ------------------------------------------------- | | STT | Speech-to-text transcription duration. | | EOU | End-of-utterance detection timing. | | LLM | Response generation latency. | | TTS | Text-to-speech synthesis duration. | | Time to First Byte | Initial delay before the agent starts responding. | | User Input Speech | How long the user spoke. | | Agent Output Speech | How long the agent's response played. | A user and agent turn expanded to show STT, EOU, LLM, and TTS processing spans along the timeline ## Turn Properties For each turn, inspect the properties of the components involved: the transcript of the user's input, the response from the LLM, and any errors that occurred. A turn selected in the trace view with its transcript and LLM response shown in the Properties panel ## Tool Calls When the LLM invokes a tool, the Trace View shows details about the call, including the tool's name and the parameters it was called with. Use it to validate integrations and debug tool behavior. A tool call span selected in the trace view, showing the tool name and the parameters passed to it ## What's Next View per-turn traces and metrics on the Dashboard. See session-level metrics and transcripts. ## References #### Examples Emit traces from a session. #### Examples Emit traces from a session. # Recording Source: https://docs.zeroruntime.ai/build/observability-and-monitoring/recording Record an agent session automatically by enabling recording on serve or invoke. Recording captures session activity for analysis, compliance, and quality assurance. Zero Runtime Agents support automatic recording with a single configuration flag. Recordings can be accessed in the Zero Runtime Dashboard with transcripts and timestamps, or downloaded for offline review. ## How It Works Enable recording when you serve or invoke the agent, and the framework automatically: * Starts recording when the agent joins the session. * Starts recording for each participant as they join. * Stops and merges the recordings when the session ends. Audio is always recorded when recording is on. Recording defaults to off, and no pipeline changes are needed once you enable it. ## What You Can Record By default, only audio is captured. Set the additional stream fields on `Room` to opt in: | Option | Type | Default | Records | | ------------------------ | ---- | ------- | ------------------------------------------------------------------ | | `recording_video` | bool | `False` | The agent's camera video track. | | `recording_screen_share` | bool | `False` | The screen-share track. Requires `vision=True` on the same `Room`. | ## Configure Recording The quickest way to record every session a process serves is a `Room(recording=True)` on `zeroruntime.serve()`: ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Room # Record every session this process serves. zeroruntime.serve(Assistant, room=Room(recording=True), on_ready=lambda: zeroruntime.invoke(AGENT_ID)) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; // Record every session this process serves. zeroruntime.serve(Assistant, { room: Room({ recording: true }), on_ready: () => zeroruntime.invoke(AGENT_ID) }); ``` You can also turn recording on for a single session by setting `recording=True` on the `Room` you pass to `zeroruntime.invoke()`: ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Room zeroruntime.invoke(AGENT_ID, room=Room(recording=True)) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; zeroruntime.invoke(AGENT_ID, { room: Room({ recording: true }) }); ``` To also record camera video, add `recording_video=True` to the same `Room`: ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Room zeroruntime.serve( Assistant, room=Room(recording=True, recording_video=True), on_ready=lambda: zeroruntime.invoke(AGENT_ID), ) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; zeroruntime.serve(Assistant, { room: Room({ recording: true, recording_video: true }), on_ready: () => zeroruntime.invoke(AGENT_ID), }); ``` To record the screen-share track, set `recording_screen_share=True` along with `vision=True` on the `Room`: ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Room zeroruntime.serve( Assistant, room=Room( vision=True, # required for screen-share recording recording=True, recording_screen_share=True, ), on_ready=lambda: zeroruntime.invoke(AGENT_ID, room=Room(vision=True)), ) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; zeroruntime.serve(Assistant, { room: Room({ vision: true, // required for screen-share recording recording: true, recording_screen_share: true, }), on_ready: () => zeroruntime.invoke(AGENT_ID, { room: Room({ vision: true }) }), }); ``` `recording_screen_share=True` requires `vision=True` because vision is what subscribes to the video and share streams. Without `vision=True`, there is no screen-share track to capture, so the screen-share recording has no effect. ## Recording Lifecycle Events Recording lifecycle hooks let you observe when recording starts, stops, or fails without polling any API. They are side-effect-only: they react to events without changing the data flow. Hooks fire only when recording is enabled. Turn recording on with `recording=True` on `zeroruntime.serve()`, or by setting `recording=True` on the `Room` you pass to `zeroruntime.invoke()`. | Hook | Fires when | | ------------------- | ------------------------------------------------------------------------------------------------------------- | | `recording_started` | Recording starts successfully. | | `recording_stopped` | Recording stops successfully, typically at session end. | | `recording_failed` | Recording fails to start or stop. Use it to surface issues to your monitoring system before the session ends. | Register the hooks on your pipeline to react to each event: ```python title="Python" Python theme={null} @pipeline.on("recording_started") def on_recording_started(data): """Fired when recording starts successfully.""" print(f"[RECORDING HOOK] Started: {data}") @pipeline.on("recording_stopped") def on_recording_stopped(data): """Fired when recording stops successfully.""" print(f"[RECORDING HOOK] Stopped: {data}") @pipeline.on("recording_failed") def on_recording_failed(data): """Fired when recording fails to start or stop.""" print(f"[RECORDING HOOK] Failed: {data}") ``` ```typescript title="Node JS" Node JS theme={null} pipeline.on('recording_started', async (data) => { // Fired when recording starts successfully. console.log(`[RECORDING HOOK] Started: ${data}`); }); pipeline.on('recording_stopped', async (data) => { // Fired when recording stops successfully. console.log(`[RECORDING HOOK] Stopped: ${data}`); }); pipeline.on('recording_failed', async (data) => { // Fired when recording fails to start or stop. console.log(`[RECORDING HOOK] Failed: ${data}`); }); ``` ## Best Practices * Inform participants that the session is being recorded. * Ensure requests are properly authenticated with a valid token. * Monitor recording status and errors using the recording lifecycle hooks. * Implement a data retention policy and meet the regulations that apply to your users. ## What's Next Pair recordings with traces, metrics, and logs for each session. Run an agent session and capture it with recording. # Run Your Agent Source: https://docs.zeroruntime.ai/build/run-the-runtime Register and run your agent with serve, start sessions with invoke or the Dispatch API, and connect it to the phone network. `zeroruntime.serve()` runs your agent as a worker: it registers the agent under its `agent_id` with the Zero Runtime and serves a session to each caller. Fire `zeroruntime.invoke()` from `on_ready` to launch a session and print a link you can open. ```python title="Python" Python theme={null} import zeroruntime from zeroruntime import Agent, Pipeline from zeroruntime.plugins import DeepgramSTT, GoogleLLM, CartesiaTTS from zeroruntime.inference import AICousticsDenoise, TurnDetector AGENT_ID = "assistant" pipeline = Pipeline( stt=DeepgramSTT(), llm=GoogleLLM(), tts=CartesiaTTS(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class Assistant(Agent): def __init__(self) -> None: super().__init__( agent_id=AGENT_ID, instructions="You are a helpful assistant.", pipeline=pipeline, ) if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh Assistant + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(Assistant, on_ready=lambda: zeroruntime.invoke(AGENT_ID, room=zeroruntime.Room(playground=True))) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, Pipeline } from '@zeroruntime/js-sdk'; import { DeepgramSTT, GoogleLLM, CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const AGENT_ID = 'assistant'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: GoogleLLM(), tts: CartesiaTTS(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); class Assistant extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful assistant.', pipeline, }); } } // Pass the class itself (not an instance): serve() builds a fresh Assistant + // pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(Assistant, { on_ready: () => zeroruntime.invoke(AGENT_ID, { room: zeroruntime.Room({ playground: true }) }) }); ``` ```bash Python theme={null} python main.py ``` `serve()` blocks and keeps the worker running, accepting a fresh session for every caller until you stop it. Scale out by running more `serve()` workers, each registering the same `agent_id`. Pass the `Agent` subclass itself (`Assistant`, not `Assistant()`): `serve(Assistant)`. `serve()` starts a fresh session per call, which is required for correct per-call state and pipeline hooks under concurrent sessions. ## Start a session `serve()` keeps your agent registered and idle. You then **start a session** to connect a caller to it. There are two ways, depending on where you're calling from: | Method | Best for | Called from | | :------------------------------ | :---------------------------------------------- | :---------------------- | | **SDK: `zeroruntime.invoke()`** | Local development, scripts, CLIs, `on_ready` | Your agent's codebase | | **Dispatch API: HTTP `POST`** | Production, remote triggers, Agent Cloud agents | Any language or service | Both do the same thing: hand a session to a running worker registered under your `agent_id`. Use whichever fits where you start the session. ### Invoke with the SDK During development, call `zeroruntime.invoke()` from a script, a CLI, a web handler, or your agent's `on_ready` hook. It returns the session details (`session_id`, `room_id`, `worker_id`), plus a `playground_url` when the room has `playground=True` (the default). Open it to talk to your agent right away. ```python title="Python" Python theme={null} import zeroruntime # Invoke a playground session and print a link to talk to the agent. result = zeroruntime.invoke("assistant", room=zeroruntime.Room(playground=True)) print(result["playground_url"]) ``` ```typescript title="Node JS" Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; // Invoke a playground session and print a link to talk to the agent. const result = zeroruntime.invoke('assistant', { room: zeroruntime.Room({ playground: true }) }); console.log(result['playground_url']); ``` #### Place a phone call Pass `Sip(...)` to start the session on an outbound (or inbound) call instead of a playground room: ```python Python theme={null} zeroruntime.invoke( "assistant", sip=zrt.Sip(call_to="+1XXXXXXXXXX", call_from="+1XXXXXXXXXX"), ) ``` ```typescript Node JS theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Sip } from '@zeroruntime/js-sdk'; zeroruntime.invoke('assistant', { sip: Sip({ call_to: '+1XXXXXXXXXX', call_from: '+1XXXXXXXXXX' }), }); ``` ### Dispatch with the API In production, start a session over HTTP with the Dispatch API. It needs no SDK, works from any language or service, and supports both Agent Cloud and self-hosted agents, for both playground and SIP-connected sessions. `meetingId` and `room_id` (the SDK's `Room(room_id=...)`) are the same identifier. The Dispatch API just calls it `meetingId`. Pass a room ID you already have, or omit it to auto-create one. #### Endpoint ```bash theme={null} POST https://api.videosdk.live/v2/agent/dispatch ``` #### Request body parameters | Parameter | Type | Required | Description | | :----------- | :----- | :------- | :-------------------------------------------------------------------------------------------------------------------------------------------- | | `meetingId` | string | Yes | The room ID to dispatch the agent into (same value as the SDK's `room_id`). | | `agentId` | string | Yes | The ID of the agent to dispatch. | | `metadata` | object | No | Optional metadata to pass to the agent, such as variables. | | `versionTag` | string | No | The specific commit version of an Agent Cloud agent to dispatch. If omitted, the latest deployed version is used. Not for self-hosted agents. | #### Example request ```bash theme={null} curl -X POST "https://api.videosdk.live/v2/agent/dispatch" \ -H "Authorization: YOUR_ZRT_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "meetingId": "xxxx-xxxx-xxxx", "agentId": "ag_xxxxxx", "metadata": { "variables":[ { "name":"fname", "value":"john" } ] }, "versionTag":"cm_xxxxxx" }' ``` #### Responses **On success**, the request returns a confirmation that the dispatch has been initiated: ```json theme={null} { "message": "Agent dispatch requested successfully.", "data": { "success": true, "status": "assigned", "roomId": "xxxx-xxxx-xxxx", "agentId": "ag_xxxxxx" } } ``` **On error**, you receive one of the following: No servers and agents are configured to handle the request. ```json theme={null} { "message": "No workers available" } ``` Specific to self-hosted agents: the `agentId` is valid, but no server has registered for it. ```json theme={null} { "message": "No workers have registered with agentId 'ag_xxxxxx'" } ``` Specific to Agent Cloud agents: the agent exists but has no deployed version available for dispatch. ```json theme={null} { "message": "No agent is deployed with agentId 'ag_xxxxxx'" } ``` ## Inside the session However you started it, once a session is live you control it from the agent's hooks and tools. Reach it as `self.session`: | Method | What it does | | :-------------------- | :------------------------------------ | | `say(text)` | Speak a fixed line (straight to TTS). | | `reply(instructions)` | Generate a reply through the LLM. | | `close()` | End the session and run `on_exit()`. | ```python Python theme={null} async def on_enter(self): await self.session.say("Hi! How can I help?") ``` ```typescript Node JS theme={null} async on_enter() { await this.session!.say('Hi! How can I help?'); } ``` ## Connect your agent to the phone network With your agent running locally, connect it to the outside world by setting up gateways and routing rules in your Zero Runtime Dashboard or via the Zero Runtime API. * Go to the Zero Runtime Dashboard. * Click on **Add Number**. * Click on **Configure SIP**. * Give a name and add your phone number. * Copy the **Inbound URL** from the Zero Runtime Dashboard. * Go to Twilio and create a new SIP Trunk. * Go to the **Origination** section, paste the Inbound URL there, and save it. * Go to the **Termination** section in Twilio, create a URI, and paste it into the outbound section in the Zero Runtime Dashboard. * Create a username and password in Twilio and add it to the Zero Runtime outbound section. * Click on **Configure rule** then **Create new routing rule**. * Add **Routing Rule Name**. * Select **API Key**. * Add **Call Direction** (Inbound or Outbound). * Add **Phone Number**. * Add **Room Type**. * Add **Agent ID**. * Click **Save**. Register your phone numbers and provision inbound and outbound SIP gateways in a single request. Replace `$YOUR_TOKEN` and update the phone numbers, SIP region, and outbound gateway settings for your provider. ```bash theme={null} curl -H 'Authorization: $YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "name": "My SIP Setup", "phoneNumbers": [ "+14155551234", "+14155555678" ], "isShared": true, "mediaEncryption": "disable", "inbound": { "sipRegion": "us002" }, "outbound": { "sipRegion": "us002", "address": "sip.telnyx.com:5061", "transport": "tls", "auth": { "username": "sip-user", "password": "sip-pass" } } }' \ -X POST https://api.videosdk.live/v2/sip/phone-numbers ``` API reference: [Create Phone Numbers + SIP Gateways](https://docs.videosdk.live/api-reference/realtime-communication/sip/phone-numbers/create-phone-number). Create a routing rule to route inbound calls to your agent. Set `agentId` to your agent's ID (from above) and replace `$YOUR_TOKEN`, `$YOUR_API_KEY`, and the phone numbers with your own values. ```bash theme={null} curl -H 'Authorization: $YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "name": "Inbound support rule", "type": "inbound", "phoneNumbers": [ "+14155551234" ], "apiKey": "$YOUR_API_KEY", "agentId": "$YOUR_AGENT_ID", "agentMetadata": { "team": "support" }, "room": { "type": "dynamic", "prefix": "blank" }, "includeHeaders": "SIP_X_HEADERS", "headersToAttributes": { "X-Caller-Id": "callerId" }, "headers": { "X-Customer-Id": "cust_42" }, "allowedNumbers": [], "allowedIpAddresses": [], "tags": [ "support", "v2" ], "recording": false, "dtmf": false, "noiseCancellation": false, "hidePhoneNumber": true }' \ -X POST https://api.videosdk.live/v2/sip/routing-rule ``` API reference: [Create Routing Rule](https://docs.videosdk.live/api-reference/realtime-communication/sip/routing-rules/create-routing-rule). You have now configured Zero Runtime to route inbound calls from your configured phone number(s) to your running agent. ## Make and receive calls Your setup is complete. Test it out. Make sure your AI agent is running locally before configuring the telephony settings. The agent must be active to receive incoming calls. ### Making an inbound call * Using any phone, dial the SIP number you configured. * Your local agent will automatically answer. * You'll hear the greeting: "Hello! I'm your real-time assistant. How can I help you today?" * Start talking. The agent will listen and respond in real time. ### Making an outbound call Trigger an outbound call from your agent with a simple API request. Use `curl` or any API client to make a `POST` request to the Zero Runtime API. Replace `$YOUR_TOKEN` and the `routingRuleId` with your own. ```bash theme={null} curl -X POST https://api.videosdk.live/v2/sip/call \ -H "Authorization: $YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "sipCallFrom" : "+14155550100", "sipCallTo" : "+14155550199", "routingRuleId" : "rr_2554md" }' ``` This commands your agent to dial the specified number and start a conversation. For optimal performance, run your agent in the same geographic region as your SIP provider (for example, US East for Twilio, US West for Telnyx, Europe for Plivo). This reduces latency and improves call quality. ## What's Next Define the agent that serve runs. Give the agent function tools and external services. DTMF, call transfer, and voicemail detection. Take your agent to production. ## References #### Examples Smallest complete agent showing serve + invoke. #### Examples Smallest complete agent showing serve + invoke. # Call Transfer Source: https://docs.zeroruntime.ai/build/telephony/call-transfer Move an ongoing SIP call to another phone number without ending the session. Call Transfer lets your AI Agent move an ongoing SIP call to another phone number without ending the current session. Instead of making the caller hang up and redial, the agent routes the call automatically. This page covers transferring from your **agent code**. For the underlying SIP REFER mechanism and provider requirements, see [Call Transfer](/telephony/managing-calls/call-transfer) in the Telephony platform docs. ## How Call Transfer Works The agent evaluates the user’s intent to determine when a call transfer is required and then triggers the function tool. When the function tool is triggered, it tells the system to move the call to another phone number. The ongoing SIP call is forwarded to the new number instantly, without disconnecting or redialing. ## Setup Expose a function tool on your agent that calls `session.transfer_call` with your auth token and the destination number. The agent invokes the tool when it detects the user wants to be transferred. ```python title="Python" Python theme={null} import os from zeroruntime import Agent, Pipeline, function_tool class CallTransferAgent(Agent): def __init__(self, pipeline: Pipeline): super().__init__( agent_id="call-transfer", instructions="You are the Call Transfer Agent. Use the call_transfer tool to transfer the ongoing call to a new number.", pipeline=pipeline, ) async def on_enter(self) -> None: await self.session.say("Hello, how can I help you today?") async def on_exit(self) -> None: await self.session.say("Goodbye, thank you for calling!") @function_tool async def call_transfer(self) -> None: """Transfer the call to the provided number""" token = os.getenv("ZERORUNTIME_AUTH_TOKEN") transfer_to = os.getenv("CALL_TRANSFER_TO") return await self.session.transfer_call(transfer_to) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, Pipeline, function_tool } from '@zeroruntime/js-sdk'; class CallTransferAgent extends Agent { constructor(pipeline: Pipeline) { super({ agent_id: 'call-transfer', instructions: 'You are the Call Transfer Agent. Use the call_transfer tool to transfer ' + 'the ongoing call to a new number.', pipeline, }); } async on_enter(): Promise { await this.session!.say('Hello, how can I help you today?'); } async on_exit(): Promise { await this.session!.say('Goodbye, thank you for calling!'); } // A tool held on a field is registered for you; the arrow keeps `this` bound // to the agent, so the tool reaches the live session the way `self` does. call_transfer = function_tool({ name: 'call_transfer', description: 'Transfer the call to the provided number', parameters: {}, execute: async () => { const transfer_to = process.env.CALL_TRANSFER_TO ?? ''; return await this.session!.transfer_call(transfer_to); }, }); } ``` See [Handoff vs Transfer](/build/agent-handoffs#handoff-vs-transfer) for the difference between Call Transfer and Agent Handoff. ## What's Next Detect voicemail systems on outbound calls. Capture caller key presses. # DTMF Events Source: https://docs.zeroruntime.ai/build/telephony/dtmf Listen for caller key presses during a call to capture input and drive IVR flows. DTMF (Dual-Tone Multi-Frequency) events occur when a caller presses keys (0–9, \*, #) during a call. Agents can listen for these events to capture input and respond immediately. This page covers handling DTMF from your **agent code**. For the SIP gateway setup and the raw `DTMF_EVENT` payload, see [DTMF Events](/telephony/managing-calls/dtmf-events) in the Telephony platform docs. ## Features * Detect key presses during a call session. * Deliver events in real time to the agent. * Handle events with a user-defined callback. * Trigger actions or IVR flows based on the input. ## Activation DTMF detection is enabled on the Inbound SIP gateway, in one of two ways. When creating an Inbound SIP gateway in the Zero Runtime dashboard, enable the `DTMF` option. DTMF events Set `enableDtmf` to `true` when creating or updating a SIP gateway. ```bash theme={null} curl -H 'Authorization: $YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "name": "Twilio Inbound Gateway", "enableDtmf": "true", "numbers": ["+0123456789"] }' \ -XPOST https://api.videosdk.live/v2/sip/inbound-gateways ``` Once the gateway has DTMF enabled, implement the handler as shown below. ## Setup Put a `DTMFHandler` on the `Pipeline` so keypad tones are delivered rather than dropped, then define an `on_dtmf` method on the agent. Once the session is started with `zeroruntime.invoke()`, the runtime calls it for each keypress. ```python title="Python" Python theme={null} from zeroruntime import Agent, DTMFHandler, Pipeline from zeroruntime.plugins import DeepgramSTT, GoogleLLM, CartesiaTTS from zeroruntime.inference import AICousticsDenoise, TurnDetector pipeline = Pipeline( stt=DeepgramSTT(), llm=GoogleLLM(), tts=CartesiaTTS(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), dtmf_handler=DTMFHandler(), # deliver keypad tones instead of dropping them ) class KeypadAgent(Agent): def __init__(self): super().__init__( agent_id="keypad", instructions="You are a phone menu. Ask the caller to press 1 or 2.", pipeline=pipeline, ) self.pressed = "" async def on_dtmf(self, key: str, payload: dict) -> None: """Called once per keypress. A one-argument `on_dtmf(self, key)` works too.""" if key == "1": await self.session.say("Routing you to Sales. How can I help?") elif key == "2": await self.session.say("Routing you to Support. What issue are you facing?") # Multi-digit sequences (a PIN, say) are accumulated by you -- the runtime # delivers one key at a time. self.pressed = (self.pressed + key)[-4:] if self.pressed == "1234": await self.session.say("PIN accepted.") ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, DTMFHandler, Pipeline } from '@zeroruntime/js-sdk'; import { DeepgramSTT, GoogleLLM, CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: GoogleLLM(), tts: CartesiaTTS(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), dtmf_handler: DTMFHandler(), // deliver keypad tones instead of dropping them }); class KeypadAgent extends Agent { pressed: string; constructor() { super({ agent_id: 'keypad', instructions: 'You are a phone menu. Ask the caller to press 1 or 2.', pipeline, }); this.pressed = ''; } async on_dtmf(key: string, payload: Record) { // Called once per keypress. A one-argument `on_dtmf(self, key)` works too. if (key === '1') { await this.session!.say('Routing you to Sales. How can I help?'); } else if (key === '2') { await this.session!.say('Routing you to Support. What issue are you facing?'); } // Multi-digit sequences (a PIN, say) are accumulated by you -- the runtime // delivers one key at a time. this.pressed = (this.pressed + key).slice(-4); if (this.pressed === '1234') { await this.session!.say('PIN accepted.'); } } } ``` A `DTMFHandler` on the pipeline subscribes the agent to the room's DTMF events—no manual subscription is required. `on_dtmf` is looked up on your agent by name and may take either `(key)` or `(key, payload)`; `DTMFHandler(callback)` routes to a plain function instead. The runtime delivers one key per call, so multi-digit input such as a PIN is accumulated in your own agent state. ## What's Next Move a live call to another number. Handle voicemail on outbound calls. # Overview Source: https://docs.zeroruntime.ai/build/telephony/overview Connect an agent to the phone network and handle real call flows like IVR, transfers, and voicemail. Telephony connects your AI agent to the phone network over SIP so it can take inbound and place outbound calls. This section covers keypad input, routing, transfers, and voicemail so your agent can handle real phone interactions. ## Inbound and Outbound Telephony works in both directions, and the features you reach for depend on the call's origin: * **Inbound:** a caller dials a number routed to your agent through an Inbound SIP gateway. DTMF and IVR routing are most common here, letting callers navigate menus or reach the right department. * **Outbound:** your agent places the call. Voicemail detection matters most here, since unanswered calls are often forwarded to a voicemail system. Transfers apply to either direction once a call is connected. Route inbound calls to the right room or agent. Capture caller key presses and drive IVR flows on inbound calls. Detect voicemail systems on outbound calls. ## Transfers vs Handoff Moving a caller can mean two different things, and it's worth keeping them straight: * **Call Transfer** forwards the SIP call to a new number directly. * **Agent Handoff** keeps the call in place and switches which agent is responsible for it. Move a live call to another number. Switch which agent is responsible for a call without moving it. # Voicemail Detection Source: https://docs.zeroruntime.ai/build/telephony/voicemail-detection Detect voicemail systems on outbound calls. Voice Mail Detection automatically detects when outbound calls are routed to voicemail, so the agent does not speak to a recording or wait for a person who is not there. Voice Mail Detection lets you: * Detect voicemail systems automatically. * Control how your agent responds. * End calls cleanly after voicemail handling. ## Setup Import `VoiceMailDetector` and put it on the `Pipeline`, alongside the providers. Give it an `llm` to classify with, and the runtime runs the detector when the session is started with `zeroruntime.invoke()`. To set up outbound calling and routing rules, check out [Handling Calls](/telephony/managing-calls/handling-calls). ```python title="Python" Python theme={null} from zeroruntime import Agent, Pipeline, VoiceMailDetector from zeroruntime.plugins import OpenAILLM, DeepgramSTT, CartesiaTTS from zeroruntime.inference import AICousticsDenoise, TurnDetector pipeline = Pipeline( stt=DeepgramSTT(), llm=OpenAILLM(), tts=CartesiaTTS(), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), voice_mail_detector=VoiceMailDetector( llm=OpenAILLM(), duration=5, ), ) class OutboundAgent(Agent): def __init__(self): super().__init__( agent_id="outbound", instructions="You are calling to confirm an appointment.", pipeline=pipeline, ) async def on_voicemail(self) -> None: """Awaited, so anything said here finishes before the call ends.""" print("Voice Mail detected, Shutting down the agent") await self.hangup(reason="reached voicemail") ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, Pipeline, VoiceMailDetector } from '@zeroruntime/js-sdk'; import { OpenAILLM, DeepgramSTT, CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ stt: DeepgramSTT(), llm: OpenAILLM(), tts: CartesiaTTS(), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), voice_mail_detector: VoiceMailDetector({ llm: OpenAILLM(), duration: 5, }), }); class OutboundAgent extends Agent { constructor() { super({ agent_id: 'outbound', instructions: 'You are calling to confirm an appointment.', pipeline, }); } async on_voicemail() { // Awaited, so anything said here finishes before the call ends. console.log('Voice Mail detected, Shutting down the agent'); await this.hangup('reached voicemail'); } } ``` ## How it works The detector buffers the opening speech on an outbound call for `duration` seconds, then asks the `llm` to classify the transcript as a human or a voicemail greeting (a one-word yes/no). Detection is handled by the runtime, which calls the agent's `on_voicemail` back when it fires. Pass `callback=` on the detector instead to route it to a plain function. ## Parameters | Parameter | Type | Default | Description | | --------------- | ----------------- | ---------- | ------------------------------------------------------------------------------------ | | `llm` | `LLM` | *required* | LLM instance used to classify the opening transcript as human vs. voicemail. | | `callback` | `Callable` | `None` | Runs on detection. `None` calls the agent's `on_voicemail` method. | | `duration` | `float` (seconds) | `2.0` | How long to buffer the opening speech before classifying. | | `custom_prompt` | `str` | `None` | Override the built-in classifier system prompt for custom voicemail-detection logic. | | `enabled` | `bool` | `True` | Set `False` to carry the configuration without turning detection on. | ## What's Next Capture caller key presses. Move a live call to another number. # Function Tools Source: https://docs.zeroruntime.ai/build/tools-and-capabilities/function-tools Let your agent run custom Python functions to act and call external services. Function tools are Python functions decorated with `@function_tool` that let an agent perform actions and call external services during a conversation. The LLM decides when to invoke them. Tools may be synchronous or asynchronous, suitable for I/O like HTTP requests. ## External Tools External tools are standalone functions passed into the agent's constructor via the `tools` parameter. Use this to share common tools across multiple agents. ```python title="Python" Python theme={null} from zeroruntime import Agent, function_tool # External tool defined outside the class @function_tool def get_weather(location: str) -> str: """Get weather information for a specific location.""" return f"Weather in {location}: Sunny, 72°F" class WeatherAgent(Agent): def __init__(self, pipeline): super().__init__( agent_id="weather-assistant", instructions="You are a weather assistant.", pipeline=pipeline, tools=[get_weather], # Register the external tool ) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, function_tool } from '@zeroruntime/js-sdk'; // External tool defined outside the class const get_weather = function_tool({ name: 'get_weather', description: 'Get weather information for a specific location.', parameters: { location: { type: 'string', description: 'The location to look up.' }, }, execute: async ({ location }) => `Weather in ${location}: Sunny, 72F`, }); class WeatherAgent extends Agent { constructor(pipeline) { super({ agent_id: 'weather-assistant', instructions: 'You are a weather assistant.', pipeline, tools: [get_weather], // Register the external tool }); } } ``` ## Internal Tools Internal tools are methods defined inside your agent class and decorated with `@function_tool`. Use this for logic specific to the agent that needs its internal state via `self`. ```python title="Python" Python theme={null} from zeroruntime import Agent, function_tool class FinanceAgent(Agent): def __init__(self, pipeline): super().__init__( agent_id="finance-assistant", instructions="You are a helpful financial assistant.", pipeline=pipeline, ) self.portfolio = {"AAPL": 10, "GOOG": 5} @function_tool def get_portfolio_value(self) -> dict: """Get the current value of the user's stock portfolio.""" # Access agent state via self return {"total_value": 5000, "holdings": self.portfolio} ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, function_tool } from '@zeroruntime/js-sdk'; class FinanceAgent extends Agent { portfolio: Record; constructor(pipeline) { super({ agent_id: 'finance-assistant', instructions: 'You are a helpful financial assistant.', pipeline, }); this.portfolio = { AAPL: 10, GOOG: 5 }; } // A tool held on a field is registered automatically -- no `tools` entry // needed. The arrow keeps `this` bound to the agent, so the tool reads its // state the way the Python one reads `self`. get_portfolio_value = function_tool({ name: 'get_portfolio_value', description: "Get the current value of the user's stock portfolio.", parameters: {}, execute: async () => ({ total_value: 5000, holdings: this.portfolio }), }); } ``` ## Tool Chaining and Parallel Calls Agents can chain tools in a turn: call a tool, pass its result back to the LLM, and repeat until a final text reply. Some LLMs (Anthropic Claude, OpenAI GPT‑4o) may request multiple tool calls in one response; those run concurrently using `asyncio.gather`. Google Gemini issues one tool call at a time. To cap how many tool calls a single turn may make, set `max_tool_calls_per_turn` on the Context Window config of the `Pipeline`. It defaults to `10` and acts as a safety limit against infinite tool-call loops. ```python title="Python" Python theme={null} from zeroruntime import Pipeline, ContextWindow pipeline = Pipeline( # ...stt, llm, tts, vad, turn_detector context_window=ContextWindow( max_tool_calls_per_turn=10, # Allow up to 10 tool calls per turn ), ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline, ContextWindow } from '@zeroruntime/js-sdk'; const pipeline = Pipeline({ // ...stt, llm, tts, vad, turn_detector context_window: ContextWindow({ max_tool_calls_per_turn: 10, // Allow up to 10 tool calls per turn }), }); ``` ## Updating Tools at Runtime You can change an agent's tools while a session is running with `Agent.update_tools(...)`. This adds, removes, or replaces tools without restarting the session. * `Agent.update_tools(tools)` updates the agent's tool list. It is synchronous. * Every entry must be a valid tool. Invalid entries raise an error when the tools are registered. ```python Python theme={null} # add a tool agent.update_tools(agent.tools + [get_horoscope]) # remove a tool agent.update_tools([t for t in agent.tools if t is not get_horoscope]) # replace all tools agent.update_tools([get_horoscope]) ``` ```typescript Node JS theme={null} // add a tool agent.update_tools([...agent.tools, get_horoscope]); // remove a tool agent.update_tools(agent.tools.filter((t) => t !== get_horoscope)); // replace all tools agent.update_tools([get_horoscope]); ``` To update tools mid-session, schedule the change as a background task from `on_enter`, where the live session is available: ```python title="Python" Python theme={null} import asyncio from zeroruntime import Agent class VoiceAgent(Agent): def __init__(self, pipeline): super().__init__( agent_id="assistant", instructions="You are a helpful voice assistant.", pipeline=pipeline, ) async def on_enter(self): await self.session.say("Hi! How can I help?") async def update_tools_later(): await asyncio.sleep(20) # Apply the change to the live session after a delay. self.update_tools(self.tools + [get_horoscope]) asyncio.create_task(update_tools_later()) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent } from '@zeroruntime/js-sdk'; class VoiceAgent extends Agent { constructor(pipeline) { super({ agent_id: 'assistant', instructions: 'You are a helpful voice assistant.', pipeline, }); } async on_enter() { await this.session!.say('Hi! How can I help?'); const update_tools_later = async () => { await new Promise((resolve) => setTimeout(resolve, 20 * 1000)); // Apply the change to the live session after a delay. this.update_tools([...this.tools, get_horoscope]); } void update_tools_later(); } } ``` Sarvam AI LLM: When using Sarvam AI as the LLM option, function tool calls and MCP tools will not work. Consider using alternative LLM providers if you need function tool support. ## What's Next Auto-discover tools from an MCP server instead of writing them by hand. Ground answers in a knowledge base with custom retrieval. Run the agent with its tools in a live session. ## References #### Examples Chain multiple function tools in one turn. #### Examples Chain multiple function tools in one turn. # Integrate MCP Source: https://docs.zeroruntime.ai/build/tools-and-capabilities/mcp Attach Model Context Protocol servers to extend your agent with external tools. MCP is an open standard for securely connecting AI assistants to external data sources and tools. In Zero Runtime AI Agents, attach MCP servers to auto-discover services, databases, and APIs. The agent picks and runs the right tools and turns results into natural responses. ## How It Works MCP tools are automatically discovered and made available to your agent, which intelligently chooses which tools to use based on user requests. When a user asks for information that requires external data, the agent will: Detect that the user's request requires external data. Choose the appropriate tools from the available MCP servers. Run the tools with the relevant parameters. Process the results and provide a natural language response. This integration allows your voice agent to access real-time data and external services while maintaining a natural conversational flow. ## Transport Methods Zero Runtime supports two transport methods for MCP servers. | Transport | Communication | Best For | | :-------------------------------- | :----------------------------------------------------- | :--------------------------------------------------- | | **STDIO** | Direct process communication with local Python scripts | Custom tools and functions, server-side integrations | | **HTTP** (Streamable HTTP or SSE) | Network-based communication with external MCP services | Third-party integrations, remote MCP servers | ## Example Pass MCP servers to the agent via the `mcp_servers` parameter. Use `MCPServerStdio` for local scripts and `MCPServerHTTP` for remote services. ```python title="Python" Python theme={null} import sys from zeroruntime import Agent, MCPServerStdio, MCPServerHTTP class MyVoiceAgent(Agent): def __init__(self, pipeline): super().__init__( agent_id="assistant", instructions="You are a helpful assistant with access to real-time data.", pipeline=pipeline, mcp_servers=[ # STDIO - local script MCPServerStdio( executable_path=sys.executable, process_arguments=["mcp_stdio_example.py"], ), # HTTP - remote service MCPServerHTTP( endpoint_url="https://your-mcp-service.com/api/mcp", request_headers={"Authorization": "Bearer "}, ), ], ) ``` ```typescript title="Node JS" Node JS theme={null} import { Agent, MCPServerStdio, MCPServerHTTP } from '@zeroruntime/js-sdk'; class MyVoiceAgent extends Agent { constructor(pipeline) { super({ agent_id: 'assistant', instructions: 'You are a helpful assistant with access to real-time data.', pipeline, mcp_servers: [ // STDIO - local script MCPServerStdio({ executable_path: 'python3', process_arguments: ['mcp_stdio_example.py'], }), // HTTP - remote service MCPServerHTTP({ endpoint_url: 'https://your-mcp-service.com/api/mcp', request_headers: { Authorization: 'Bearer ' }, }), ], }); } } ``` The agent auto-discovers every tool exposed by each server and chooses among them at runtime, so you don't register tools individually. ## Server options ### `MCPServerStdio` (local process) | Parameter | Type | Default | Description | | :------------------ | :----------------------- | :--------- | :---------------------------------------------------------------------------------------------------------------------------------------------- | | `executable_path` | `str` | *required* | Program to run (e.g. `sys.executable` for Python). | | `process_arguments` | `list[str]` | `[]` | Its arguments -- typically the server script. | | `environment_vars` | `dict[str, str] \| None` | `None` | Environment for the child. `None` inherits this process's, which is usually what you want: the server needs the same API keys you already have. | | `working_directory` | `str \| Path \| None` | `None` | Directory to run it in. | | `session_timeout` | `float` | `5.0` | Seconds to wait for the connection and for each tool call. 5 is tight for a server that starts an interpreter; pass `30` for those. | ### `MCPServerHTTP` (remote service) | Parameter | Type | Default | Description | | :-------------------- | :----------------------- | :--------- | :---------------------------------------------------------------------------------------------------------------- | | `endpoint_url` | `str` | *required* | The server's URL. | | `request_headers` | `dict[str, str] \| None` | `None` | Sent with every request -- an API key usually lives here. Read in this process, so the credential stays with you. | | `connection_timeout` | `float` | `10.0` | Seconds to wait for the connection. | | `stream_read_timeout` | `float` | `300.0` | Seconds a quiet stream may stay open. Long by default: an MCP server that has nothing to say is not broken. | | `session_timeout` | `float` | `5.0` | Seconds to wait for the session, and for each tool call. | ## What's Next Add custom Python functions as tools. Ground responses in a knowledge base. ## References #### Examples Consume tools from an MCP server. #### Examples Consume tools from an MCP server. # Overview Source: https://docs.zeroruntime.ai/build/tools-and-capabilities/overview Extend an agent with function tools, MCP servers, and a knowledge base. An AI agent can do more than chat. You can extend it with custom actions, external services, and a knowledge base so it can call APIs, query databases, or fetch facts and include the results in its replies. There are three ways to extend an agent, and they work well together: * **Function Tools** run your own code. * **MCP** connects standardized external tool servers. * **RAG** grounds answers in your own documents. ## Ways to Extend an Agent Functions your agent calls to run custom logic, call APIs, or access state. Attach Model Context Protocol servers to extend your agent with external tools. Ground agent answers in a knowledge base with custom retrieval.. ## What's Next Add custom actions to your agent. Register the tools on an agent you define. Run the tool-equipped agent in a live session. ## References #### Examples Chain multiple function tools in one turn. Connect tools through an MCP server. #### Examples Chain multiple function tools in one turn. Connect tools through an MCP server. # Attach RAG to your Agent Source: https://docs.zeroruntime.ai/build/tools-and-capabilities/rag Ground agent answers in a knowledge base with custom retrieval. RAG (Retrieval-Augmented Generation) helps your AI agent find relevant information from documents to give better answers. It searches a knowledge base and uses that context to respond more accurately. ## Architecture The RAG pipeline flow: STT converts speech to text. The knowledge base fetches relevant documents based on the transcript. Retrieved context is injected into the LLM prompt. The LLM generates a grounded response using the context. TTS converts the response to speech. ## Custom RAG For full control, Build your own RAG pipeline using any vector database (ChromaDB, Pinecone, etc.) with the `user_turn_start` hook.This hook fires when the user's transcript is ready, before the LLM is called, giving you the perfect place to retrieve documents and inject context. **1. Set up the vector store.** Create a collection to hold your document embeddings. ChromaDB ships a Python client. The snippet below keeps embeddings in a minimal in-memory index. ```python title="Python" Python theme={null} self.chroma_client = chromadb.Client() self.collection = self.chroma_client.create_collection(name="rag_docs") ``` ```typescript title="Node JS" Node JS theme={null} // A vector store of your choosing; the SDK does not ship one. this.collection = await vector_store.create_collection('rag_docs'); ``` **2. Implement `retrieve()`.** Generate a query embedding and search the vector store for the top matching documents. ```python title="Python" Python theme={null} async def retrieve(self, query: str, k: int = 2) -> list[str]: response = await self.openai_client.embeddings.create( input=query, model="text-embedding-ada-002" ) query_embedding = response.data[0].embedding results = self.collection.query( query_embeddings=[query_embedding], n_results=k ) return results["documents"][0] if results["documents"] else [] ``` ```typescript title="Node JS" Node JS theme={null} async retrieve(query: string, k = 2): Promise { const response = await this.openai_client.embeddings.create({ input: query, model: 'text-embedding-ada-002', }); const query_embedding = response.data[0].embedding; const results = await this.collection.query({ query_embeddings: [query_embedding], n_results: k, }); return results.documents?.[0] ?? []; } ``` **3. Inject context with the `user_turn_start` hook.** Retrieve documents from the transcript and fold them into the system prompt with `session.change_component(instructions=...)` before the LLM is invoked. Rebuild from a base string each turn so the retrieved context doesn't accumulate. ```python title="Python" Python theme={null} BASE_INSTRUCTIONS = "You are a helpful assistant. Ground your answers in the retrieved context." @pipeline.on("user_turn_start") async def on_user_turn_start(transcript: str): context_docs = await agent.retrieve(transcript) if context_docs: context_str = "\n\n".join( f"Document {i+1}: {doc}" for i, doc in enumerate(context_docs) ) await agent.session.change_component( instructions=( f"{BASE_INSTRUCTIONS}\n\nRetrieved Context:\n{context_str}\n\n" "Use this context to answer the user's question." ) ) ``` ```typescript title="Node JS" Node JS theme={null} const BASE_INSTRUCTIONS = 'You are a helpful assistant. Ground your answers in the retrieved context.'; pipeline.on('user_turn_start', async (transcript: string) => { const context_docs = await agent.retrieve(transcript); if (context_docs.length) { const context_str = context_docs .map((doc, i) => `Document ${i + 1}: ${doc}`) .join('\n\n'); await agent.session.change_component({ instructions: `${BASE_INSTRUCTIONS}\n\nRetrieved Context:\n${context_str}\n\n` + "Use this context to answer the user's question.", }); } }); ``` The LLM then sees the injected context and uses it to generate a grounded answer. ## Best Practices * Start by retrieving `k=2-3` documents and adjust based on performance. * Keep document chunk sizes between 300-800 words. * Use persistent storage for your vector database in production. * Cache embeddings for frequently asked questions to reduce latency. * Handle retrieval failures gracefully so the agent can still respond. ## What's Next Add custom actions to your agent. Connect external MCP servers as tools. # De-noise Source: https://docs.zeroruntime.ai/build/turn-detection-and-interruptions/denoise Filter background noise before it reaches the pipeline with RNNoise or cloud providers. De-noise improves audio quality by filtering out background noise before it reaches your pipeline. This creates clearer conversations, especially in noisy environments. Zero Runtime supports two approaches to denoising: * **RNNoise**: a deep learning plugin that runs inside Zero Runtime. * **Inference providers**: speech enhancement via the Zero Runtime Inference gateway (Sanas and AIcoustics). ## RNNoise The Zero Runtime Agents framework provides real-time audio denoising through the `RNNoise` plugin, which runs inside Zero Runtime. RNNoise uses deep learning to tell speech apart from noise. It: * Removes background sounds like keyboard typing, air conditioning, and ambient noise. * Improves speech clarity and speech-to-text accuracy. * Processes audio in real time with minimal latency. * Works with the `Pipeline` in both cascading and realtime modes. Initialize RNNoise and pass it to your `Pipeline` as the `denoise` component: ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import RNNoise, DeepgramSTT, OpenAILLM, ElevenLabsTTS, SileroVAD pipeline = Pipeline( stt=DeepgramSTT(api_key="your-deepgram-key"), llm=OpenAILLM(api_key="your-openai-key", model="gpt-5.4-nano"), tts=ElevenLabsTTS(api_key="your-elevenlabs-key", voice="your-voice-id"), vad=SileroVAD(), denoise=RNNoise() # Enable noise removal ) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { RNNoise, DeepgramSTT, OpenAILLM, ElevenLabsTTS, SileroVAD } from '@zeroruntime/js-sdk/plugins'; const pipeline = Pipeline({ stt: DeepgramSTT({ api_key: 'your-deepgram-key' }), llm: OpenAILLM({ api_key: 'your-openai-key', model: 'gpt-5.4-nano' }), tts: ElevenLabsTTS({ api_key: 'your-elevenlabs-key', voice: 'your-voice-id' }), vad: SileroVAD(), denoise: RNNoise(), // Enable noise removal }); ``` ## Inference Providers The Zero Runtime Inference gateway exposes denoise providers as the `SanasDenoise` and `AICousticsDenoise` classes. Import the one you want and pass it to your `Pipeline` as the `denoise` component. `SanasDenoise` integrates Sanas for real-time speech enhancement and noise suppression. No required parameters. ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.inference import SanasDenoise pipeline = Pipeline( # ... other config denoise=SanasDenoise() ) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { SanasDenoise } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ // ... other config denoise: SanasDenoise(), }); ``` `AICousticsDenoise` provides live audio cleanup with on-the-fly speech enhancement and background noise reduction. No required parameters. ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.inference import AICousticsDenoise pipeline = Pipeline( # ... other config denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const pipeline = Pipeline({ // ... other config denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` ## Choosing a Denoise Provider | Provider | Type | Language Support | Extra Config | | :------------- | :------------- | :---------------- | :----------- | | **RNNoise** | Runtime plugin | Language-agnostic | None | | **Sanas** | Inference | Language-agnostic | None | | **AIcoustics** | Inference | Language-agnostic | None | Use **RNNoise** when you want a low-latency solution that runs inside Zero Runtime with no external dependencies. Use **Sanas** or **AIcoustics** when you prefer to route denoising through the Zero Runtime Inference gateway alongside your other components. ## What's Next Send clean audio to turn detection. Feed clean audio into VAD next. # Overview Source: https://docs.zeroruntime.ai/build/turn-detection-and-interruptions/overview Read timing and audio quality so the agent replies, yields, and hears clearly. In a voice conversation, the agent must get timing and audio quality right. It needs to know when the user has finished speaking, when to yield if someone interrupts, and it needs clean audio to make accurate decisions. This section covers the features that handle all three. Turn detection, VAD, and de-noise in the pipeline ## What this section covers Decide when the user has actually finished a turn so the agent replies at the right moment. Detect whether speech is present and let the user barge in while the agent is talking. Remove background noise before audio reaches the pipeline so every stage works from clean input. ## Turn Detection and VAD VAD and turn detection answer two different questions, and work best paired, with VAD as the first-pass speech detector and the turn detector deciding when the turn is over: * **VAD**: *is anyone speaking?* Fast and lightweight, but tracks silence only, not meaning. * **Turn Detection**: *is the user actually done?* A semantic model shifts from raw audio analysis to Natural Language Understanding (NLU), reading words and context to tell a real endpoint from a thinking pause. | Aspect | Traditional VAD (Silence-Based) | Semantic Turn Detection | | :--------------- | :---------------------------------------------- | :------------------------------------------------------- | | Detection method | Listens for silence. | Understands words and context. | | Approach | Relies on a fixed timer (e.g. 800ms). | Uses a transformer model to predict intent. | | Performance | Often interrupts or lags. | Knows when to wait and when to respond instantly. | | Handling pauses | Struggles with natural pauses and filler words. | Distinguishes between a brief pause and a true endpoint. | ## Interruptions Real conversations aren't strictly turn by turn. When a user cuts in, interruption handling lets the agent stop and listen, using VAD or STT to confirm they genuinely want the turn. ## Noise Cancellation Background noise such as typing, fans, or street sound lowers transcription accuracy and confuses turn detection. De-noise strips it from the incoming audio in real time, so STT, VAD, and the turn detector all work from a clean signal. ## What's Next Set up semantic turn detection first. Add VAD and handle user interruptions. Remove background noise before the pipeline. ## References #### Examples Voice agent with VAD and turn detection configured. #### Examples Voice agent with VAD and turn detection configured. # Turn Detection Source: https://docs.zeroruntime.ai/build/turn-detection-and-interruptions/turn-detection Understand when a user is truly finished speaking with semantic turn-detection models. Timing matters. VAD detects silence; turn detection reads meaning so your agent knows when to respond and when to keep listening. ## Echo Turn Detector Zero Runtime's turn detector is **Echo**, a server-hosted model passed to your `Pipeline` as the `turn_detector`. It is exposed through the **Zero Runtime Inference Gateway** via the unified `TurnDetector` class (`model="echo-small"` or `model="echo-large"`). No model is downloaded or loaded on your machine; authentication requires `ZERORUNTIME_AUTH_TOKEN`. Choose `echo-small` for speed or `echo-large` for accuracy. VAD detects **that** speech is happening; the turn detector decides **when** the turn is over. ### How It Works As the user speaks, VAD detects the speech and STT produces a transcript. After each user utterance, the latest transcript is sent to the Inference Gateway, where the selected Echo model (`echo-small` or `echo-large`) classifies the turn into one of four states: | State | Meaning | | :------------ | :------------------------------------------------------------------ | | `Complete` | The user has finished their turn. | | `Incomplete` | The user is still mid-sentence or not finished yet. | | `Backchannel` | A short acknowledgement (e.g. "uh-huh", "okay okay"). | | `Wait` | The user wants the agent to hold (e.g. "wait a minute", "hold on"). | ### Models Both models share the same four-state classification; they differ only in the latency/accuracy trade-off: | Provider | Model Name | Model ID | | :----------- | :--------- | :----------- | | Zero Runtime | Echo Small | `echo-small` | | Zero Runtime | Echo Large | `echo-large` | * **`model="echo-small"`**: the lowest-latency model, optimized for the fastest possible turn detection. Best when responsiveness matters most. * **`model="echo-large"`**: a higher-accuracy model that trades a little latency for better classification. Best when accuracy matters more than raw speed. ### Supported Languages Both `echo-small` and `echo-large` support 12 languages: English, Hindi, Gujarati, Marathi, Tamil, Telugu, Urdu, Bengali, French, German, Italian, and Spanish. ### Usage Set your auth token, then construct `TurnDetector` with the `echo-small` or `echo-large` model: ```python Python theme={null} # Set ZERORUNTIME_AUTH_TOKEN in your environment. from zeroruntime.inference import TurnDetector # Fastest, lowest latency turn_detector = TurnDetector(model="echo-small") # Higher accuracy turn_detector = TurnDetector(model="echo-large") ``` ```typescript Node JS theme={null} // Set ZERORUNTIME_AUTH_TOKEN in your environment. import { TurnDetector } from '@zeroruntime/js-sdk/inference'; // Fastest, lowest latency const turn_detector = TurnDetector({ model: 'echo-small' }); // Higher accuracy const turn_detector = TurnDetector({ model: 'echo-large' }); ``` ### Performance Benchmarked on the [TURNS2K](https://huggingface.co/datasets/latishab/turns-2k) dataset against a leading third-party turn-detection model, referred to here as **Baseline**. Each sample is labeled **Complete** (the user has finished speaking) or **Incomplete** (the user is still speaking). | Metric | Echo-Small | Echo-Large | Baseline | | :------------------ | :--------- | :--------- | :------- | | Accuracy | 93.60% | 96.20% | 61.13% | | Recall (Complete) | 97.31% | 96.50% | 32.83% | | Specificity | 88.91% | 95.81% | 96.83% | | F1 Score (Complete) | 0.9443 | 0.9659 | 0.4851 | Results are measured on the benchmark dataset described above, on samples labeled Complete or Incomplete. Performance may vary depending on language, deployment configuration, user behavior, and application requirements. ### Choosing a Model | Model | Latency | Accuracy | Best For | | :------------- | :------ | :------- | :-------------------------------- | | **Echo Small** | Lowest | High | Lowest-latency, responsive agents | | **Echo Large** | Low | Highest | Highest accuracy | ## End-of-Utterance Handling End-of-Utterance (EOU) handling decides when the pipeline treats the user as finished speaking. In `ADAPTIVE` mode, the wait timeout adjusts based on confidence scores, so the agent waits longer when the user is hesitant and responds faster when intent is clear. Configure it with `EOUConfig` in your pipeline options: ```python Python theme={null} from zeroruntime import Pipeline, EOUConfig pipeline = Pipeline( # ... other config eou_config=EOUConfig( mode="ADAPTIVE", min_max_speech_wait_timeout=[0.5, 0.8] ) ) ``` ```typescript Node JS theme={null} import { Pipeline, EOUConfig } from '@zeroruntime/js-sdk'; const pipeline = Pipeline({ // ... other config eou_config: EOUConfig({ mode: 'ADAPTIVE', min_max_speech_wait_timeout: [0.5, 0.8], }), }); ``` | Parameter | Type | Default | Description | | :---------------------------- | :-------------------------- | :----------- | :-------------------------------------------------- | | `mode` | `"DEFAULT"` \| `"ADAPTIVE"` | `"DEFAULT"` | `ADAPTIVE` uses LLM confidence to adjust wait time. | | `min_max_speech_wait_timeout` | `[float, float]` | `[0.5, 0.8]` | Min and max wait time in seconds after speech ends. | ### How the modes behave * `DEFAULT`: Fixed wait within your min–max range. For clear utterances you respond near the minimum; for hesitations you wait near the maximum. * `ADAPTIVE`: The wait scales with confidence. Low confidence (hesitation) increases wait; high confidence shortens it, always clamped to your min–max. ## What's Next Pair turn detection with Silero VAD. Feed the detector cleaner input audio. ## References #### Examples Voice agent with a turn detector configured. #### Examples Voice agent with a turn detector configured. # VAD and Interruptions Source: https://docs.zeroruntime.ai/build/turn-detection-and-interruptions/vad-and-interruptions Detect speech with Silero VAD, let users barge in while the agent is talking, and gracefully recover from false interruptions. Voice Activity Detection (VAD) identifies when speech is present in an audio stream. It acts as a first-pass filter and enables interruption handling in your pipeline by signaling when a user starts talking. Zero Runtime uses Silero VAD for speech-activity detection; for the most reliable results, pair VAD with a turn detector. ## Configure Silero VAD Configure VAD and pass it to your `Pipeline` as the `vad` component: ```python Python theme={null} from zeroruntime.plugins import SileroVAD # Configure VAD to detect speech activity vad = SileroVAD( threshold=0.5, # Sensitivity to speech (0.3 to 0.8) min_speech_duration=0.1, # Ignore very brief sounds min_silence_duration=0.75 # Wait time before considering speech ended ) ``` ```typescript Node JS theme={null} import { SileroVAD } from '@zeroruntime/js-sdk/plugins'; // Configure VAD to detect speech activity const vad = SileroVAD({ threshold: 0.5, // Sensitivity to speech (0.3 to 0.8) min_speech_duration: 0.1, // Ignore very brief sounds min_silence_duration: 0.75, // Wait time before considering speech ended }); ``` ## Interruption Detection Interruption Detection decides when user speech should stop the agent mid-response. It can use VAD, STT, or both. This avoids false triggers from short noises, filler words, or background audio. **Using VAD** detects raw speech activity. It is faster but can be triggered by background noise. ```python Python theme={null} from zeroruntime import Pipeline, InterruptConfig pipeline = Pipeline( # ... other config interrupt_config=InterruptConfig( mode="VAD_ONLY", interrupt_min_duration=0.2 # 200ms of continuous speech ) ) ``` ```typescript Node JS theme={null} import { Pipeline, InterruptConfig } from '@zeroruntime/js-sdk'; const pipeline = Pipeline({ // ... other config interrupt_config: InterruptConfig({ mode: 'VAD_ONLY', interrupt_min_duration: 0.2, // 200ms of continuous speech }), }); ``` **Using STT** relies only on recognized words from the transcript. It is slower but ensures the speech is intelligible. ```python Python theme={null} from zeroruntime import Pipeline, InterruptConfig pipeline = Pipeline( # ... other config interrupt_config=InterruptConfig( mode="STT_ONLY", interrupt_min_words=2 # At least 2 words recognized ) ) ``` ```typescript Node JS theme={null} import { Pipeline, InterruptConfig } from '@zeroruntime/js-sdk'; const pipeline = Pipeline({ // ... other config interrupt_config: InterruptConfig({ mode: 'STT_ONLY', interrupt_min_words: 2, // At least 2 words recognized }), }); ``` Use `HYBRID` mode to require both audio detection and recognized words. ## False Interruption Recovery A false interruption is a brief accidental noise, like a cough, an "mm-hmm", or background noise, that should not stop the agent. Instead of cutting off immediately, `InterruptConfig` lets the agent pause and confirm whether an interruption is genuine before deciding to stop or resume speaking. ### How it works The user makes a sound while the agent is speaking. Instead of cutting off, the agent pauses TTS and starts a timer (`false_interrupt_pause_duration`). If a transcript with at least `interrupt_min_words` words arrives before the timer expires (or VAD confirms sustained speech), it's a real interruption and the agent stops. Otherwise the timer expires and the agent resumes speaking from where it paused. ### Enable it Set `resume_on_false_interrupt=True` on the pipeline's `InterruptConfig`. Tune `false_interrupt_pause_duration` for how long to wait before deciding it was a false alarm. ```python title="Python" Python theme={null} from zeroruntime import Pipeline, InterruptConfig pipeline = Pipeline( # ... other config interrupt_config=InterruptConfig( false_interrupt_pause_duration=2.0, # Wait 2 seconds to confirm resume_on_false_interrupt=True # Auto-resume if interruption is brief ) ) ``` ```typescript title="Node JS" Node JS theme={null} import { Pipeline, InterruptConfig } from '@zeroruntime/js-sdk'; const pipeline = Pipeline({ // ... other config interrupt_config: InterruptConfig({ false_interrupt_pause_duration: 2.0, // Wait 2 seconds to confirm resume_on_false_interrupt: true, // Auto-resume if interruption === brief }), }); ``` ## Configuration | Parameter | Type | Default | Description | | :------------------------------- | :----------------------------------------- | :--------- | :----------------------------------------------------------------------------------------- | | `mode` | `"VAD_ONLY"` \| `"STT_ONLY"` \| `"HYBRID"` | `"HYBRID"` | Detection method for interruptions. | | `interrupt_min_duration` | `float` | `0.5` | Minimum speech duration in seconds to trigger an interrupt. | | `interrupt_min_words` | `int` | `2` | Minimum words needed to confirm an interrupt. | | `false_interrupt_pause_duration` | `float` | `2.0` | Pause duration in seconds on a false interrupt, before deciding it was false and resuming. | | `resume_on_false_interrupt` | `bool` | `True` | Whether to resume agent speech after a false interrupt. | ## What's Next Pair VAD with semantic turn detection. Give VAD a cleaner audio signal. ## References #### Examples Voice agent with VAD configured. #### Examples Voice agent with VAD configured. # CLI Reference Source: https://docs.zeroruntime.ai/deployments/cli-reference Every Zero Runtime CLI command, grouped by what you use it for, with its flags and defaults. A complete reference for the `zeroruntime` CLI. For a guided walkthrough, see [Deploy an Agent](/deployments/deploy-an-agent) and [Managing Deployments](/deployments/managing-deployments). Install the CLI with pip (Python 3.11+): ```bash theme={null} pip install zeroruntime ``` Add `--help` to any command to see its flags, e.g. `zeroruntime up --help`. Most commands read missing values (agent ID, image, version) from `zeroruntime.yaml` in the current folder, so you rarely pass them by hand. Run commands from your project root. ## How values are resolved Every value a command needs is found in this order: 1. The flag you pass on the command line. 2. The matching field in `zeroruntime.yaml` (e.g. `agent.id`, `build.image`). 3. A built-in default, otherwise the command tells you what's missing. ## The typical flow These are the commands you'll run most, in the order you'll use them: | Command | What it does | | --------------------------- | ------------------------------------------------------------------ | | `zeroruntime auth login` | Sign in (opens the browser). | | `zeroruntime quickstart` | Download an example agent and run it locally. | | `zeroruntime run ` | Run an agent project locally. | | `zeroruntime init` | Register the agent and generate `zeroruntime.yaml` + `Dockerfile`. | | `zeroruntime up` | Deploy your agent to Zero Runtime Cloud. | | `zeroruntime invoke` | Start your deployed agent in a room so you can talk to it. | | `zeroruntime logs` | Stream your agent's console logs. | | `zeroruntime down` | Take the agent offline. | ## Authentication | Command | Description | | ------------------------- | --------------------------------------------------------- | | `zeroruntime auth login` | Sign in to your Zero Runtime account (opens the browser). | | `zeroruntime auth logout` | Sign out on this machine. | | `zeroruntime --version` | Print the CLI version. | ## Quickstart `zeroruntime quickstart` downloads a ready-made example, drops it in a new folder, and runs it locally so you can talk to it immediately. If you're not signed in, it logs you in first. ```bash theme={null} zeroruntime quickstart # pick an example interactively zeroruntime quickstart --template # use a specific example, skip the picker zeroruntime quickstart --no-open # run locally but don't open the browser ``` | Flag | Description | Default | | ------------------- | ------------------------------------------ | ------- | | `--template` / `-t` | Example id to use (skips the picker). | prompt | | `--no-open` | Don't open the playground in your browser. | off | ## Run locally `zeroruntime run ` runs an agent on your machine. Point it at a project folder or a single `.py` file. It creates a virtual environment, installs `requirements.txt` once, and reads provider keys from the project's `.env`. ```bash theme={null} zeroruntime run my-agent --open # playground mode, open in the browser zeroruntime run my-agent --console # console mode, talk in the terminal zeroruntime run main.py --console # run a single file ``` | Flag | Description | Default | | --------------------- | ------------------------------------------------------------- | ------------ | | `--playground` / `-p` | Run in playground mode. | on (default) | | `--console` / `-c` | Run in console mode (talk in the terminal). Requires sign-in. | off | | `--open` / `-o` | Open the playground in your browser once the agent starts. | off | | `--reset-env` | Re-pick the Python environment for this file. | off | ## Initialize `zeroruntime init` registers an agent + deployment, writes the IDs to `zeroruntime.yaml`, and generates a `Dockerfile` if your project doesn't have one. ```bash theme={null} zeroruntime init --name my-agent ``` | Flag | Description | Default | | ------------------- | ---------------------------------------- | ---------------- | | `--name` / `-n` | Agent name. | server-generated | | `--template` / `-t` | Template id to associate with the agent. | none | ## Deploy `zeroruntime up` deploys your agent to Zero Runtime Cloud in a single command, using sensible defaults. ```bash theme={null} zeroruntime up # deploy the agent zeroruntime up --env .env # also upload provider keys from .env as secrets ``` | Flag | Description | Default | | ---------------- | ---------------------------------------------------------------- | ----------------------- | | `--env` / `-e` | Upload keys from this `.env` file as a secret set for the agent. | none | | `--image` / `-i` | Image name and tag (e.g. `my-agent:0.0.1`). | `build.image` from yaml | | `--file` / `-f` | Path to the Dockerfile. | `./Dockerfile` | On success, the new `version.id` is saved to `zeroruntime.yaml`. ## Take it down `zeroruntime down` deactivates all active versions so they stop using resources. ```bash theme={null} zeroruntime down # confirms first zeroruntime down --yes # skip the confirmation zeroruntime down --force # deactivate even versions with active sessions ``` | Flag | Description | Default | | -------------- | ---------------------------------------------- | ------- | | `--yes` / `-y` | Skip the confirmation prompt. | off | | `--force` | Deactivate even versions with active sessions. | off | ## Versions A **version** is one immutable, deployed configuration. Every `zeroruntime up` creates a new one. ```bash theme={null} zeroruntime version list # all versions (the active one is marked) zeroruntime version status # rollout state of the latest version zeroruntime version describe -v # full details for one version zeroruntime version activate -v # make a version live zeroruntime version deactivate -v # take a version offline ``` | Command | Description | | ---------------------------------------- | ------------------------------------------------------------- | | `zeroruntime version list` | List all versions (paginated). | | `zeroruntime version status` | Rollout status of a version (latest if no `-v`). | | `zeroruntime version describe` | Full details for a version. | | `zeroruntime version activate -v ` | Activate a version. | | `zeroruntime version deactivate -v ` | Deactivate a version (`--force` to override active sessions). | ### `version list` flags | Flag | Description | Default | | ------------ | ------------------------------------ | ------- | | `--page` | Page number (`1`+). | `1` | | `--per-page` | Items per page (`1` to `100`). | `10` | | `--sort` | `-1` newest first, `1` oldest first. | `-1` | ## Secrets Secrets are environment variables (provider keys, tokens) injected into your running agent. The easiest way to set them is `zeroruntime up --env .env`; you can also manage sets directly: ```bash theme={null} zeroruntime secrets list # all secret sets zeroruntime secrets create my-secrets -f .env # create a set from a file zeroruntime secrets describe my-secrets # show keys (values hidden) zeroruntime secrets add my-secrets # add keys interactively zeroruntime secrets remove my-secrets # remove keys interactively zeroruntime secrets delete my-secrets # delete the whole set ``` | Command | Description | | ------------------------------------------- | ----------------------------------------------------------------- | | `zeroruntime secrets list` | List all secret sets. | | `zeroruntime secrets create -f .env` | Create a set from a file (omit `-f` to enter keys interactively). | | `zeroruntime secrets describe ` | Show the keys in a set (values hidden). | | `zeroruntime secrets add [name]` | Add keys interactively (picks a set if `name` is omitted). | | `zeroruntime secrets remove ` | Remove keys interactively. | | `zeroruntime secrets delete ` | Delete the whole set. | ## Invoke `zeroruntime invoke` starts your deployed agent in a room so you can talk to it. ```bash theme={null} zeroruntime invoke # start in a fresh room, print a playground link zeroruntime invoke --console # start, then talk to it in your terminal zeroruntime invoke --room-id # start in a specific room ``` | Flag | Description | Default | | ------------------- | ----------------------------------------------------- | -------------------- | | `--agent-id` / `-a` | Agent to start. | `agent.id` from yaml | | `--room-id` / `-r` | Room to join. | new room | | `--console` / `-c` | Talk to the agent in your terminal. Requires sign-in. | off | ## Sessions A session is one live conversation. `zeroruntime invoke` starts one; `session` lists and stops them. ```bash theme={null} zeroruntime session list # sessions for this agent zeroruntime session stop --room-id # stop a session by room zeroruntime session stop --session-id # stop a session by session id ``` | Command | Description | | --------------------------------------- | -------------------------------------------------------------- | | `zeroruntime session list` | List sessions for the agent. | | `zeroruntime session stop -r ` | Stop a session by room (`-s ` to stop by session). | ### `session list` flags | Flag | Description | Default | | ------------ | ------------------------------------ | ------- | | `--room-id` | Only sessions in this room. | none | | `--page` | Page number (`1`+). | `1` | | `--per-page` | Items per page (`1` to `100`). | `10` | | `--sort` | `-1` newest first, `1` oldest first. | `-1` | ## Logs `zeroruntime logs` streams console output from your running agent. ```bash theme={null} zeroruntime logs # most recent 50 lines, newest first zeroruntime logs --limit 200 --sort 1 # 200 lines, oldest first zeroruntime logs --since 1h # last hour zeroruntime logs --since 30m --until 5m # a specific window ``` | Flag | Description | Default | | --------------------- | -------------------------------------------------------- | -------------------- | | `--limit` / `-n` | Number of log lines (`1` to `1000`). | `50` | | `--sort` | `-1` newest first, `1` oldest first. | `-1` | | `--since` | Start of the window (e.g. `30m`, `2h`, `1d`, or a date). | none | | `--until` | End of the window. | now | | `--version-id` / `-v` | Logs for a specific version. | latest | | `--agent-id` | Agent to read logs for. | `agent.id` from yaml | ## The `zeroruntime.yaml` file The CLI stores deployment state in `zeroruntime.yaml` so commands can run without repeating flags. You rarely edit it by hand; each command fills in the values it produces. ```yaml zeroruntime.yaml theme={null} version: "1.0" agent: id: ag_xxxxxxxx # set by `zeroruntime init` name: my-agent build: image: my-agent:0.0.1 # set by `zeroruntime up` deploy: id: dep_xxxxxxxx # set by `zeroruntime init` version: vr_xxxxxxxx # set by `zeroruntime up` secrets: env: my-secrets # set by `zeroruntime up --env` / `zeroruntime secrets create` ``` # CLI Setup Source: https://docs.zeroruntime.ai/deployments/cli-setup Install the Zero Runtime CLI with pip and authenticate your account so you can build and deploy agents. The Zero Runtime CLI ships in the `zeroruntime` Python package: it is both the developer CLI and the console-mode engine, so there is just one thing to install. Install it once, authenticate, and you are ready to deploy. ## Install the CLI Install the `zeroruntime` package with pip; it puts the `zeroruntime` command on your PATH: ```bash theme={null} pip install zeroruntime ``` Requires **Python 3.11 or newer**. If `pip` maps to Python 2 on your system, use `pip3 install zeroruntime`. To keep it isolated, install it inside a virtual environment. ### Verify the install ```bash theme={null} zeroruntime --version ``` You should see the installed version printed back, for example `Zero Runtime CLI, version 0.0.7`. ## Authenticate Log in to connect the CLI to your Zero Runtime account. This opens your browser to confirm the login; the CLI waits for approval and then stores your token locally. ```bash theme={null} zeroruntime auth login ``` ```text Expected output theme={null} ◆ Authentication ▸ Initiating browser authentication... ✓ Opened authentication URL in browser https://app.zeroruntime.ai/cli/confirm-auth?requestId=... ⠋ Waiting for authentication... ✓ Successfully authenticated! ``` If the link expires or the browser doesn't open, run `zeroruntime auth login` again. You can copy the printed URL into a browser manually. ## Try a live agent instantly Once you're signed in, the fastest way to see Zero Runtime working is `zeroruntime quickstart`. It downloads a ready-made example into a new folder and runs it locally, opening a **playground** in your browser so you can talk to it in about 30 seconds. ```bash theme={null} zeroruntime quickstart ``` Pass an example to skip the picker, or `--no-open` to run without opening the browser: ```bash theme={null} zeroruntime quickstart --template zeroruntime quickstart --no-open ``` `quickstart` is the quickest way to try an agent and get a working project to build on. To deploy it to the cloud, follow [Deploy an Agent](/deployments/deploy-an-agent). ## Sign out When you want to disconnect the CLI from your account on this machine: ```bash theme={null} zeroruntime auth logout ``` ## Next Create a project, test it locally, then deploy, step by step. Browse every command and flag. # Deploy an Agent Source: https://docs.zeroruntime.ai/deployments/deploy-an-agent Go from an empty folder to a live agent you can talk to: create a project, test it locally, deploy it to Zero Runtime Cloud with one command, then invoke it. This guide takes you from nothing to a live agent in a handful of commands. Run them in order; each one builds on the last. First, install the Zero Runtime CLI with pip (Python 3.11+), and make sure **Docker is running** (needed for `zeroruntime up`): ```bash theme={null} pip install zeroruntime ``` See [CLI Setup](/deployments/cli-setup) for more detail. Run every `zeroruntime` command from **inside your project folder**. The CLI reads and writes a `zeroruntime.yaml` file there to remember your agent, image, and version, so most commands need no flags at all. Authenticate the CLI with your Zero Runtime account. This opens your browser to confirm the login, then stores your token on this machine: ```bash theme={null} zeroruntime auth login ``` You only need to do this once per machine. To sign out later, run `zeroruntime auth logout`. The fastest way to get a working agent is `zeroruntime quickstart`. It downloads a ready-made example into a new folder and runs it locally so you can talk to it right away. ```bash theme={null} zeroruntime quickstart ``` You'll be asked to pick an example, or skip the picker by naming one (replace `` with the example you want): ```bash theme={null} zeroruntime quickstart --template ``` This creates a `.//` folder with everything you need: | File | Purpose | | ------------------ | ------------------------------------------- | | `main.py` | Your agent's entrypoint and pipeline. | | `requirements.txt` | Python dependencies. | | `.env` | Provider API keys and secrets (kept local). | | `config.json` | Local run metadata. | Already have an agent project? You can skip this step. Just make sure it has an entrypoint named `main.py`, `agent.py`, or `app.py`. `zeroruntime init` (below) generates the `Dockerfile` and `zeroruntime.yaml` for you. Move into the project folder (the one quickstart just created) and open `.env`. Fill in the API keys for the STT, LLM, and TTS providers your agent uses: ```bash theme={null} cd ``` ```bash .env theme={null} ZERORUNTIME_AUTH_TOKEN=... # added for you by quickstart DEEPGRAM_API_KEY=... # your provider keys GOOGLE_API_KEY=... CARTESIA_API_KEY=... ``` `zeroruntime run` and your deployed agent both read keys from this `.env` file. Never commit it or bake keys into the Docker image. Run the agent on your own machine to confirm it works before deploying. This creates a virtual environment, installs the requirements, and starts the agent in the **playground** so you can talk to it in your browser: ```bash theme={null} zeroruntime run . --open ``` ```text Expected output theme={null} Running project ''... Playground ready: https://playground.zeroruntime.ai/... Opening the playground in your browser... ``` Prefer your terminal over the browser? Run in **console mode** instead: ```bash theme={null} zeroruntime run . --console ``` You can also run a single file directly: `zeroruntime run main.py --console`. This runs entirely on your machine; no cloud resources are used yet. Register the agent with Zero Runtime Cloud. This creates the IDs your later commands need, writes them to `zeroruntime.yaml`, and generates a `Dockerfile` if your project doesn't have one: ```bash theme={null} zeroruntime init --name my-agent ``` After it runs, `zeroruntime.yaml` holds your `agent.id` and `deploy.id`: ```yaml zeroruntime.yaml theme={null} version: "1.0" agent: id: ag_xxxxxxxx name: my-agent deploy: id: dep_xxxxxxxx ``` You won't edit `zeroruntime.yaml` by hand. Each command fills in the values it produces (image name, version ID, secrets), so the commands that follow run with no flags. `zeroruntime up` deploys your agent to Zero Runtime Cloud in a single command. Pass `--env .env` to also upload your provider keys so the deployed agent can use them: ```bash theme={null} zeroruntime up --env .env ``` ```text Expected output theme={null} ✓ Agent is up! Version ID: vr_xxxxxxxx ``` That's it. Your agent is now running on Zero Runtime Cloud. The new `version.id` is saved to `zeroruntime.yaml`. `zeroruntime up` deploys with sensible defaults, so there's nothing else to configure. This step needs Docker running. If you see `'docker' command not found`, make sure Docker is installed and running, then try again. Check that the version deployed and is healthy: ```bash theme={null} zeroruntime version list # every version, with the active one marked zeroruntime version status # rollout state of the latest version ``` **Invoke** your agent to start it in a room. The CLI prints a playground link you can open to talk to it: ```bash theme={null} zeroruntime invoke ``` ```text Expected output theme={null} ✓ Agent invoked successfully Room ID: Interact with your agent here: https://playground.zeroruntime.ai?token=...&meetingId= ``` Want to talk to it right in your terminal? Add `--console`: ```bash theme={null} zeroruntime invoke --console ``` List and stop live sessions when you're done: ```bash theme={null} zeroruntime session list zeroruntime session stop --room-id ``` Stream your agent's console output to see what it's doing in production: ```bash theme={null} zeroruntime logs # most recent 50 lines zeroruntime logs --limit 100 --sort 1 # 100 lines, oldest first ``` See [Managing Deployments](/deployments/managing-deployments#logs) for time filters. ## Ship an update Changed your code? Redeploy with the same command; it creates a new version: ```bash theme={null} zeroruntime up --env .env ``` ## Take it down When you're finished, stop the running version(s) so they stop using resources: ```bash theme={null} zeroruntime down ``` This lists the active versions and asks you to confirm. Add `--yes` to skip the prompt, or `--force` if a version still has active sessions. ## What's next Versions, secrets, sessions, and logs for a live agent. Every command and flag in one place. # Managing Deployments Source: https://docs.zeroruntime.ai/deployments/managing-deployments Operate a live agent day to day: manage versions, update secrets, inspect sessions, and read logs. Once your agent is [deployed](/deployments/deploy-an-agent), these commands let you operate it. Run them from inside your project folder so the CLI can read IDs from `zeroruntime.yaml`; most flags are optional because of that. If you don't have the CLI yet, install it with pip (Python 3.11+): ```bash theme={null} pip install zeroruntime ``` ## Versions Every `zeroruntime up` creates a new immutable **version**. You keep the one you want live, and can roll back by reactivating an older one. ```bash theme={null} zeroruntime version list # all versions, with the active one marked zeroruntime version status # rollout state of the latest version zeroruntime version describe -v # full details for one version ``` Activate or deactivate a specific version: ```bash theme={null} zeroruntime version activate -v zeroruntime version deactivate -v zeroruntime version deactivate -v --force # even if it has active sessions ``` `zeroruntime down` is the quick way to deactivate **all** active versions at once. Use `zeroruntime version deactivate` when you want to target a single version. ## Secrets Secrets are environment variables (provider keys, tokens) injected into your running agent. The simplest way to set them is at deploy time: ```bash theme={null} zeroruntime up --env .env ``` This uploads the keys from your `.env` file and applies them to the deployment. You can also manage secret sets directly: ```bash theme={null} zeroruntime secrets list # all secret sets zeroruntime secrets create my-secrets -f .env zeroruntime secrets describe my-secrets # keys in the set (values are hidden) zeroruntime secrets add my-secrets # add keys interactively zeroruntime secrets remove my-secrets # remove keys interactively zeroruntime secrets delete my-secrets # delete the whole set ``` Changing secrets affects new sessions. Redeploy with `zeroruntime up` to roll the change out to running workers. ## Sessions A session is one live conversation handled by your agent. Start the agent with `zeroruntime invoke`, then list and stop sessions with `zeroruntime session`: ```bash theme={null} zeroruntime invoke # start the agent in a room zeroruntime invoke --console # start it, then talk in your terminal zeroruntime session list # list sessions for this agent zeroruntime session list --room-id # only sessions in a specific room zeroruntime session stop --room-id # stop a session by room ``` `zeroruntime invoke` prints a playground link and the room ID it joined. Stop a session by room with `--room-id`, or by session ID with `--session-id`. ## Logs Stream console output from your running agent. Filter by count, direction, and time window: ```bash theme={null} zeroruntime logs # most recent 50 lines, newest first zeroruntime logs --limit 200 --sort 1 # 200 lines, oldest first zeroruntime logs --since 1h # last hour zeroruntime logs --since 30m --until 5m # a specific window ``` | Flag | Description | Default | | --------------------- | --------------------------------------------- | ---------------------------------- | | `--limit` / `-n` | Number of log lines (`1` to `1000`). | `50` | | `--sort` | `-1` newest first, `1` oldest first. | `-1` | | `--since` | Start of the window (e.g. `30m`, `2h`, `1d`). | none | | `--until` | End of the window. | now | | `--version-id` / `-v` | Logs for a specific version. | latest | | `--agent-id` | Agent to read logs for. | `agent.id` from `zeroruntime.yaml` | ## Tearing down Deactivate all active versions for the deployment: ```bash theme={null} zeroruntime down # confirms first zeroruntime down --yes # skip confirmation zeroruntime down --force # deactivate even with active sessions ``` ## What's next Every command and flag in one place. Revisit the end-to-end walkthrough. # Deployments Source: https://docs.zeroruntime.ai/deployments/overview How a Zero Runtime agent runs in production - the runtime handles real-time media, and you deploy your agent to Zero Runtime Cloud with a single zeroruntime up command. Your agent runs as a **worker** - your code, wrapped by `zeroruntime.serve(...)`. The Zero Runtime **runtime** handles the real-time media (transport, GPUs, STT/LLM/TTS orchestration, and turn-taking); your worker just registers with the runtime and receives sessions. You never operate low-latency media infrastructure. Everything here is driven by the `zeroruntime` CLI, installed with the `zeroruntime` package (Python 3.11+): ```bash theme={null} pip install zeroruntime ``` See [CLI Setup](/deployments/cli-setup) for authentication and next steps. Deployment architecture: a caller's live session reaches the Zero Runtime runtime, which manages transport, GPUs, and real-time media. The runtime streams the session to your agent worker, deployed on Zero Runtime Cloud via the CLI. The control plane invokes your agent to start the session. ## How a session flows 1. A **caller** starts a live session (from a playground, a web/mobile client, or an inbound phone call). 2. The session reaches the **Zero Runtime**, which Zero Runtime manages: transport, GPUs, and real-time media. 3. The **control plane** invokes your agent to start the session. 4. The runtime streams that session to one of your **registered workers**, which runs your agent code and responds back through the runtime. ## Deploying your agent You deploy your agent to **Zero Runtime Cloud** with `zeroruntime up`. Zero Runtime hosts, scales, and runs your agent for you: it pulls your image and schedules the pods on **our** compute, so you never operate a server. You manage everything (versions, secrets, sessions, and logs) through the `zeroruntime` CLI. ```bash theme={null} zeroruntime up ``` ## Get started Install the `zeroruntime` package with pip and authenticate. Create a project, test it locally, then deploy end to end. Versions, secrets, sessions, and logs. Every command and flag in one place. # Installation Source: https://docs.zeroruntime.ai/installation Learn to install the Zero Runtime Python SDK, connect to the runtime, and build voice AI agents with Python 3.11+. The Zero Runtime Python SDK lets you build voice AI agents on Python 3.11+ with a clean, type-hinted API. ## Prerequisites Before installing, make sure you have: * **Python 3.11 or later** * A **Zero Runtime auth token** — get these from the [Zero Runtime dashboard](https://app.zeroruntime.ai/) * **API keys** for any AI/voice providers you plan to use (e.g., STT, TTS, LLM providers) ## Installation Install the SDK via pip: ```bash theme={null} pip install zeroruntime ``` ## Connect to the Runtime Point the worker at your runtime address, and set your auth token, using the values from the dashboard: ```bash theme={null} export ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 export ZERORUNTIME_AUTH_TOKEN= ``` ## Key Conventions Once installed, keep these conventions in mind as you build: * **Agents** subclass `Agent` and implement the `async def on_enter` and `on_exit` hooks. * **Tools** use the `@function_tool` decorator — the schema is automatically inferred from type hints and the docstring. * **Provider plugins** are imported from `zrt.plugins`. ## Next Steps * Follow the [Quickstart](https://docs.zeroruntime.ai/quickstarts/build-your-first-voice-agent) guide to build your first complete agent. * Browse runnable examples in the [zeroruntime-python-examples](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples/tree/main) and [zeroruntime-js-examples](https://github.com/ZeroRuntimeAI/zeroruntime-js-examples/tree/main) repositories. # Introduction Source: https://docs.zeroruntime.ai/introduction Build voice AI agents in Python. Zero Runtime runs the real-time voice pipeline so you never manage latency, turn-taking, or scale. Zero Runtime architecture: users connect through integrations to the fully managed real-time voice pipeline (denoise, VAD, STT, turn detection, LLM/realtime model, TTS), driven by your agent code in the Python SDK. **Zero Runtime** lets you build real-time voice AI agents in Python. You write the agent code; we run the entire voice infrastructure including speech-to-text, LLM processing, text-to-speech, turn detection, noise cancellation, scaling, and GPUs. Your code runs in your process, while all real-time voice handling runs on ours. > **Think serverless for voice AI: you build the agent, we run the runtime.** ## Build Build voice agents with the Python SDK. Build voice agents with the Node JS SDK. *** ## Integration Connect your agents with communication channels and client applications. #### Communication Handle inbound and outbound voice calls. Engage users through WhatsApp messaging. #### Client SDKs Build voice-enabled React experiences. Integrate voice features into mobile apps. Add voice experiences to iOS apps. *** ## Observe Monitor, analyze, and troubleshoot agent interactions and performance. Track agent execution step by step. Measure usage, performance, and outcomes. Review detailed agent activity records. Access and review conversation recordings. ## Deploy Take your agent from local development to production on Zero Runtime. Understand how Zero Runtime deploys and scales your agents. Ship your agent to production step by step. ## Use Cases Explore production-ready examples and reference implementations to accelerate development. Phone-tuned booking agent whose tools are an n8n workflow reached over MCP. Support agent that escalates to a human over MCP instead of guessing. Detects the caller's language each turn and rebuilds the pipeline to answer in it. Detects the user's language automatically and provides support in their preferred language. ## Need Help? Join the community and connect with other developers. Runnable Python agents to read, copy, and adapt. The same agents in TypeScript, one file per idea. Book a demo, contact support, or speak with our sales team. # Setup with MCP and Skills Source: https://docs.zeroruntime.ai/mcp-and-skills Connect the Zero Runtime MCP server and Agent Skill to your AI coding agent so it builds with the current docs instead of guessing. Two pieces make an AI coding agent good at Zero Runtime: * **The MCP server** — a live search over these docs, hosted at `https://mcp.zeroruntime.ai`. The agent looks up real API details mid-task instead of recalling them. * **The Agent Skill** — the procedure for building on Zero Runtime: scoping the agent, wiring auth, and picking a pipeline. Install both. The skill knows *what* to build; the MCP server tells it the *exact* method name. This page is about your **coding** agent. To give your **voice** agent MCP tools at runtime, see [Integrate MCP](/build/tools-and-capabilities/mcp). ## Install the MCP server Pick your client. Most clients need a restart before a new server loads. Run this from your project, or add `--scope user` to make it available everywhere: ```bash theme={null} claude mcp add --transport http zeroruntime https://mcp.zeroruntime.ai ``` Confirm it registered with `claude mcp list`. Add the server to `~/.cursor/mcp.json` for every project, or `.cursor/mcp.json` for this one: ```json theme={null} { "mcpServers": { "zeroruntime": { "url": "https://mcp.zeroruntime.ai" } } } ``` Create `.vscode/mcp.json` in your workspace. Copilot picks it up once the file is saved. ```json theme={null} { "servers": { "zeroruntime": { "type": "http", "url": "https://mcp.zeroruntime.ai" } } } ``` Add the server to `~/.codex/config.toml`: ```toml theme={null} [mcp_servers.zeroruntime] url = "https://mcp.zeroruntime.ai" ``` Add the server to `~/.codeium/windsurf/mcp_config.json`: ```json theme={null} { "mcpServers": { "zeroruntime": { "serverUrl": "https://mcp.zeroruntime.ai" } } } ``` Add the server to `~/.gemini/config/mcp_config.json` for every project, or `.agents/mcp_config.json` for this one: ```json theme={null} { "mcpServers": { "zeroruntime": { "serverUrl": "https://mcp.zeroruntime.ai" } } } ``` A remote server must use `serverUrl`. The legacy `url` and `httpUrl` fields are not supported. Type `/mcp` in the prompt panel to manage servers from inside the CLI. Open the MCP Servers panel, choose to edit the configuration, and add the server to `cline_mcp_settings.json`: ```json theme={null} { "mcpServers": { "zeroruntime": { "type": "streamableHttp", "url": "https://mcp.zeroruntime.ai" } } } ``` Add the server under `context_servers` in your Zed `settings.json`: ```json theme={null} { "context_servers": { "zeroruntime": { "source": "custom", "url": "https://mcp.zeroruntime.ai" } } } ``` ### Verify the connection Ask your agent: > Search the Zero Runtime docs for how to generate an auth token, and tell me which page it came from. A connected agent replies with a real `docs.zeroruntime.ai` URL it just fetched. If it answers from memory, cites no page, or says it has no such tool, the server has not loaded — restart the client and recheck the file you edited. ## Install the Agent Skill One skill, `zeroruntime`, covering voice agents, telephony, and the auth each needs. ```bash theme={null} npx skills add ZeroRuntimeAI/skills ``` `npx skills` is agent-agnostic. It asks which agents to install into — pick the ones you use, then restart them. To place it by hand instead, copy the `zeroruntime` folder into your agent's skills directory, for example `~/.claude/skills/zeroruntime`. ## Skill or MCP server They do different jobs and work best together. | | Agent Skill | MCP Server | | ------------- | ----------------------------------------------- | -------------------------------------- | | What it is | The procedure the agent follows | The lookup the agent calls | | What it holds | How to scope the build, wire auth, pick plugins | The current docs, searchable | | When it runs | At the start of a Zero Runtime task | Whenever the agent needs an API detail | ## Troubleshooting Restart the client. Several of them only initialise MCP servers at startup. Search for the full name as it appears in the docs — for example a plugin's provider name — rather than an abbreviation. Ask the agent to send feedback through the server, including the page and the query you used. That reaches the team maintaining the index. # Anam Source: https://docs.zeroruntime.ai/plugins/avatar/anam Use the Anam avatar plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Anam is an **avatar** plugin. It renders a real-time, lip-synced AI avatar synchronized to the agent's speech output. ## Setup Set your Anam API key in the worker environment. Generate a key from the [Anam site](https://www.anam.ai/): ```bash theme={null} export ANAM_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `avatar` slot. ```python Python theme={null} from zeroruntime.plugins import AnamAvatar avatar = AnamAvatar( avatar_id="", ) # Pipeline(avatar=avatar, ...) ``` ```typescript Node JS theme={null} import { AnamAvatar } from '@zeroruntime/js-sdk/plugins'; const avatar = AnamAvatar({ avatar_id: '', }); // Pipeline({ avatar, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | -------------- | ----- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Anam API key. Falls back to the `ANAM_API_KEY` environment variable when unset. | | `avatar_id` | `str` | `""` | Identifier of the avatar to use. Falls back to the `ANAM_AVATAR_ID` environment variable, then to a built-in default avatar when empty. | | `persona_name` | `str` | `None` | Optional persona name to apply to the avatar. | | `voice_id` | `str` | `None` | Optional voice identifier for the avatar. | The avatar renders a lip-synced video track from the agent's speech; see the [avatar integration guide](/build/modalities/avatars/integration) for how it wires into the pipeline, or browse the full [plugins overview](/plugins/overview). # Simli Source: https://docs.zeroruntime.ai/plugins/avatar/simli Use the Simli avatar plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Simli is an **avatar** plugin. It renders a real-time, lip-synced AI avatar video track from the agent's speech. ## Setup Set your Simli API key in the worker environment. Generate a key from the [Simli dashboard](https://app.simli.com/apikey): ```bash theme={null} export SIMLI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `avatar` slot. ```python Python theme={null} from zeroruntime.plugins import SimliAvatar avatar = SimliAvatar( face_id="", # falls back to SIMLI_FACE_ID / a default ) # Pipeline(avatar=avatar, ...) ``` ```typescript Node JS theme={null} import { SimliAvatar } from '@zeroruntime/js-sdk/plugins'; const avatar = SimliAvatar({ config: { faceId: '' }, // Simli's own config object }); // Pipeline({ avatar, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------------- | ------------- | ------------------------ | ---------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Simli API key. Falls back to the `SIMLI_API_KEY` environment variable when unset. | | `config` | `SimliConfig` | *required* | Simli's own configuration object, from the `simli` package. Carries the face id and the rest of the vendor's settings. | | `transport_mode` | `str` | `'P2P'` | Transport used to reach Simli. | | `simli_url` | `str` | `'https://api.simli.ai'` | Simli API base URL. | | `is_trinity_avatar` | `bool` | `False` | Set to `True` when the face belongs to a Trinity avatar, to keep lip-sync correct. | The avatar renders the [TTS](/plugins/overview) output as a lip-synced video track in the room. # Aicoustics Source: https://docs.zeroruntime.ai/plugins/denoise/aicoustics Use the ai-coustics acoustic denoiser in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. ai-coustics is an acoustic **noise cancellation** denoiser. Like [Sanas](/plugins/denoise/sanas), it runs server-side over the Zero Runtime **inference gateway** (so it needs a gateway auth token), unlike the local [RNNoise](/plugins/denoise/rnnoise) plugin. It occupies the pipeline's `denoise` slot, cleaning the caller's audio before the rest of the pipeline runs. ## Setup The denoiser connects to the inference gateway using your Zero Runtime auth token: ```bash theme={null} export ZERORUNTIME_AUTH_TOKEN= ``` The token can also be passed explicitly to the factory. ## Usage Build the denoiser with `AICousticsDenoise` and pass it to the pipeline's `denoise` slot. ```python Python theme={null} from zeroruntime.inference import AICousticsDenoise denoise = AICousticsDenoise(model_id="sparrow-xxs-48khz") # Pipeline(denoise=denoise, ...) ``` ```typescript Node JS theme={null} import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'; const denoise = AICousticsDenoise({ model_id: 'sparrow-xxs-48khz' }); // Pipeline({ denoise, ... }) ``` ## Models ai-coustics offers two model families: | Family | Use case | Sample rate | Examples | | ------- | --------------------------------- | ----------- | ---------------------------------------------------------------------------------- | | Sparrow | Human-to-human audio | 48 kHz | `sparrow-xxs-48khz` (fastest), `sparrow-s-48khz`, `sparrow-l-48khz` (best quality) | | Quail | Human-to-machine (voice AI / STT) | 16 kHz | `quail-vf-l-16khz`, `quail-l-16khz`, `quail-s-16khz` | Match `sample_rate` to the model family (48000 for Sparrow, 16000 for Quail). ## Parameters *Parameters for `AICousticsDenoise`.* | Parameter | Type | Default | Description | | ------------- | ----- | ---------------- | ----------------------------------------------------- | | `model_id` | `str` | `"rook-l-48khz"` | ai-coustics model. | | `sample_rate` | `int` | `48000` | Audio sample rate (Hz). Use `16000` for Quail models. | | `base_url` | `str` | `None` | Override the inference gateway URL. | ## Import paths | SDK | Import | Constructor | | ------- | ------------------------------------------------------------------- | ---------------------------- | | Python | `from zeroruntime.inference import AICousticsDenoise` | `AICousticsDenoise(...)` | | Node JS | `import { AICousticsDenoise } from '@zeroruntime/js-sdk/inference'` | `AICousticsDenoise({ ... })` | Noise cancellation runs first, so [voice activity detection](/plugins/vad/silero) and [speech-to-text](/plugins/stt/deepgram) see cleaner audio. # RNNoise Source: https://docs.zeroruntime.ai/plugins/denoise/rnnoise Use the RNNoise noise cancellation plugin in a Zero Runtime pipeline to clean the caller's audio. Setup and usage in Python and JavaScript. RNNoise is a **noise cancellation** plugin. It cleans the caller's incoming audio before the rest of the pipeline runs, which improves voice activity detection and transcription on noisy connections. It occupies the pipeline's `denoise` slot. ## Usage Import the plugin and pass it to the pipeline's `denoise` slot. ```python Python theme={null} from zeroruntime.plugins import RNNoise denoise = RNNoise() # Pipeline(denoise=denoise, ...) ``` ```typescript Node JS theme={null} import { RNNoise } from '@zeroruntime/js-sdk/plugins'; const denoise = RNNoise(); // Pipeline({ denoise, ... }) ``` ## Parameters `RNNoise()` takes no constructor parameters. It runs the bundled RNNoise model at a fixed 48 kHz on 20 ms (480-sample) frames and resamples other input rates automatically, so there is nothing to configure. ## Import paths | SDK | Import | Constructor | | ------- | ------------------------------------------------------- | ----------- | | Python | `from zeroruntime.plugins import RNNoise` | `RNNoise()` | | Node JS | `import { RNNoise } from '@zeroruntime/js-sdk/plugins'` | `RNNoise()` | Noise cancellation runs first, so [voice activity detection](/plugins/vad/silero) and [speech-to-text](/plugins/stt/deepgram) see cleaner audio. # Sanas Source: https://docs.zeroruntime.ai/plugins/denoise/sanas Use the Sanas noise cancellation denoiser in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Sanas is a professional **noise cancellation** denoiser. Unlike the local [RNNoise](/plugins/denoise/rnnoise) plugin, Sanas runs server-side over the Zero Runtime **inference gateway**, so it needs a gateway auth token. It occupies the pipeline's `denoise` slot, cleaning the caller's audio before the rest of the pipeline runs. ## Setup The denoiser connects to the inference gateway using your Zero Runtime auth token: ```bash theme={null} export ZERORUNTIME_AUTH_TOKEN= ``` The token can also be passed explicitly to the factory. ## Usage Build the denoiser with `SanasDenoise` and pass it to the pipeline's `denoise` slot. ```python Python theme={null} from zeroruntime.inference import SanasDenoise denoise = SanasDenoise(model_id="VI_G_NC3.0") # Pipeline(denoise=denoise, ...) ``` ```typescript Node JS theme={null} import { SanasDenoise } from '@zeroruntime/js-sdk/inference'; const denoise = SanasDenoise({ model_id: 'VI_G_NC3.0' }); // Pipeline({ denoise, ... }) ``` ## Parameters *Parameters for `SanasDenoise`.* | Parameter | Type | Default | Description | | --------------- | ----- | -------------- | ----------------------------------------------------------------------------- | | `model_id` | `str` | `"VI_G_NC3.0"` | Sanas noise-cancellation model. | | `sample_rate` | `int` | `16000` | Audio sample rate (Hz) the model expects. | | `chunk_ms` | `int` | `20` | Audio chunk size in milliseconds. | | `gateway_token` | `str` | `None` | Auth token for the inference gateway. Falls back to `ZERORUNTIME_AUTH_TOKEN`. | | `base_url` | `str` | `None` | Override the inference gateway URL. | ## Import paths | SDK | Import | Constructor | | ------- | -------------------------------------------------------------- | ----------------------- | | Python | `from zeroruntime.inference import SanasDenoise` | `SanasDenoise(...)` | | Node JS | `import { SanasDenoise } from '@zeroruntime/js-sdk/inference'` | `SanasDenoise({ ... })` | Noise cancellation runs first, so [voice activity detection](/plugins/vad/silero) and [speech-to-text](/plugins/stt/deepgram) see cleaner audio. # Zero Runtime Inference Gateway Source: https://docs.zeroruntime.ai/plugins/inference/zero-runtime Route STT, LLM, TTS, turn detection, and denoise through the Zero Runtime Inference Gateway with a single auth token, no per-provider API keys. The **Inference Gateway** is a unified entry point for the model stack. Instead of holding a separate API key for each provider, you authenticate once with your Zero Runtime auth token and route speech-to-text, LLM, text-to-speech, turn detection, and denoise through the gateway. It manages the upstream provider connections, resampling, and streaming server-side. ## Setup Gateway providers connect using your Zero Runtime auth token, no provider API keys are required. Set it in the worker environment: ```bash theme={null} export ZERORUNTIME_AUTH_TOKEN= ``` ## Usage Import gateway providers from `zrt.inference` and drop them into the pipeline exactly like normal plugins. Every gateway class is a drop-in replacement for its `zrt.plugins` equivalent. Turn detection is the unified `TurnDetector` (imported from `zrt.plugins`) using an Echo model, which also routes through the gateway: ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.inference import AICousticsDenoise, CartesiaTTS, GoogleLLM, SarvamAISTT, TurnDetector from zeroruntime.plugins import SileroVAD pipeline = Pipeline( stt=SarvamAISTT(), # via gateway llm=GoogleLLM(), # via gateway tts=CartesiaTTS(), # via gateway vad=SileroVAD(), # local plugin turn_detector=TurnDetector(model="echo-large"), # Echo, via gateway denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { AICousticsDenoise, CartesiaTTS, GoogleLLM, SarvamAISTT, TurnDetector } from '@zeroruntime/js-sdk/inference'; import { SileroVAD } from '@zeroruntime/js-sdk/plugins'; const pipeline = Pipeline({ stt: SarvamAISTT(), // via gateway llm: GoogleLLM(), // via gateway tts: CartesiaTTS(), // via gateway vad: SileroVAD(), // local plugin turn_detector: TurnDetector({ model: 'echo-large' }), // Echo, via gateway denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); ``` You can mix gateway and direct-provider components freely in the same pipeline. For example, run STT and TTS through the gateway while the LLM uses its own key. ## Supported providers | Stage | Gateway providers | | :----------------- | :------------------------------------------------ | | **Speech-to-text** | Deepgram, Google, Sarvam AI, AssemblyAI, Cartesia | | **LLM** | Google, Sarvam AI | | **Text-to-speech** | Cartesia, Google, Deepgram, Sarvam AI | | **Turn detection** | Echo (echo-small, echo-large) | | **Denoise** | ai-coustics, Sanas | ## Import paths Every gateway class is imported from the single `zrt.inference` module: | Stage | Import | | :------------- | :-------------------------------------------------------------------------------------------------- | | Speech-to-text | `from zeroruntime.inference import DeepgramSTT, GoogleSTT, SarvamAISTT, AssemblyAISTT, CartesiaSTT` | | LLM | `from zeroruntime.inference import GoogleLLM, SarvamAILLM` | | Text-to-speech | `from zeroruntime.inference import CartesiaTTS, GoogleTTS, DeepgramTTS, SarvamAITTS` | | Turn detection | `from zeroruntime.plugins import TurnDetector` (Echo models route through the gateway) | | Denoise | `from zeroruntime.inference import Denoise, AICousticsDenoise, SanasDenoise` | Denoisers also expose factory methods, `Denoise.sanas()` and `Denoise.aicoustics()`, which return the same configured component as the named aliases above. ## Why use it * **One credential.** Authenticate with a single Zero Runtime token instead of provisioning and rotating a key for every provider. * **Less client work.** Connection management, resampling, and streaming happen server-side. * **Easy switching.** Swap a gateway provider without onboarding a new API key. ## What's next How gateway providers slot into a pipeline, with cascading and realtime examples. The full catalog of STT, LLM, TTS, and denoise providers. # Anthropic Source: https://docs.zeroruntime.ai/plugins/llm/anthropic Use the Anthropic Claude LLM plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Anthropic Claude is an **LLM** plugin. It takes the transcribed conversation and generates the reply, with support for tool calling, prompt caching, and extended thinking. ## Setup Set your Anthropic API key in the worker environment. Generate a key from the [Anthropic console](https://console.anthropic.com/dashboard): ```bash theme={null} export ANTHROPIC_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import AnthropicLLM llm = AnthropicLLM( model="claude-sonnet-4-20250514", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { AnthropicLLM } from '@zeroruntime/js-sdk/plugins'; const llm = AnthropicLLM({ model: 'claude-sonnet-4-20250514', temperature: 0.7, }); // Pipeline({ llm, ... }) ``` ## Configuration Options *Constructor parameters for `AnthropicLLM`. The Python and Node JS SDKs share these field names.* ### Core | Parameter | Type | Default | Description | | ------------------- | ------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `api_key` | `str` | `None` | Anthropic API key. Falls back to the `ANTHROPIC_API_KEY` environment variable when unset. | | `model` | `str` | `"claude-sonnet-4-20250514"` | Claude model used to generate replies. | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `max_output_tokens` | `int` | `1024` | Maximum number of tokens the model may generate per response. The legacy `max_tokens` kwarg is accepted as a deprecated alias. | ### Sampling | Parameter | Type | Default | Description | | ---------------- | -------------------- | ------- | ------------------------------------------------------------------------------------------------------ | | `top_k` | `int` | `None` | Top-k sampling cutoff. | | `top_p` | `float` | `None` | Nucleus-sampling probability cutoff. | | `stop_sequences` | `str` or `list[str]` | `None` | One or more strings at which the model stops generating. Accepts a single string or a list of strings. | ### Advanced | Parameter | Type | Default | Description | | ----------------- | ----- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `thinking_budget` | `int` | `None` | Token budget for Claude's extended thinking (chain-of-thought reasoning). A positive value (e.g. `1024`) enables thinking; `None` disables it. The legacy `thinking={"budget_tokens": N}` dict form is also accepted. | For voice, short replies feel best. Keep `max_output_tokens` modest and the system instruction concise to keep latency down. ## Import paths | SDK | Import | Constructor | | ------- | ------------------------------------------------------------ | ----------------------- | | Python | `from zeroruntime.plugins import AnthropicLLM` | `AnthropicLLM(...)` | | Node JS | `import { AnthropicLLM } from '@zeroruntime/js-sdk/plugins'` | `AnthropicLLM({ ... })` | The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which synthesizes the agent's voice. # AWS Bedrock Source: https://docs.zeroruntime.ai/plugins/llm/aws Use the AWS Bedrock LLM plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. AWS Bedrock is an **LLM** plugin. It takes the transcribed conversation and generates the reply using any Bedrock-hosted model (Amazon Nova, Anthropic Claude, Meta Llama, Mistral, and more) through the unified Converse API. ## Setup Set your AWS region in the worker environment. Credentials resolve as explicit constructor argument, then environment variable, then the default AWS credential chain (IAM role or shared profile). Get your access keys from the [AWS console](https://console.aws.amazon.com/): ```bash theme={null} export AWS_DEFAULT_REGION= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import AWSBedrockLLM llm = AWSBedrockLLM( model="amazon.nova-lite-v1:0", region="us-east-1", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { AWSBedrockLLM } from '@zeroruntime/js-sdk/plugins'; const llm = AWSBedrockLLM({ model: 'amazon.nova-lite-v1:0', region: 'us-east-1', temperature: 0.7, }); // Pipeline({ llm, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | --------------------------- | -------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `str` | `"amazon.nova-lite-v1:0"` | Bedrock model id or inference profile ARN. Falls back to the `BEDROCK_INFERENCE_PROFILE_ARN` environment variable when unset. | | `region` | `str \| None` | `None` | AWS region for Bedrock Runtime. Falls back to `AWS_DEFAULT_REGION`, then `AWS_REGION`, then `"us-east-1"`. | | `aws_access_key_id` | `str \| None` | `None` | AWS access key ID. Falls back to the `AWS_ACCESS_KEY_ID` environment variable. | | `aws_secret_access_key` | `str \| None` | `None` | AWS secret access key. Falls back to the `AWS_SECRET_ACCESS_KEY` environment variable. | | `aws_session_token` | `str \| None` | `None` | Session token for temporary credentials. Falls back to the `AWS_SESSION_TOKEN` environment variable. | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `max_output_tokens` | `int` | `1024` | Maximum tokens generated per response. The legacy `max_tokens` keyword argument is also accepted. | | `top_p` | `float \| None` | `None` | Nucleus-sampling probability mass. | | `top_k` | `int \| None` | `None` | Restricts sampling to the top-k most probable tokens. Sent via additional model request fields; support varies by model. | | `stop_sequences` | `list[str] \| str \| None` | `None` | Sequence or list of sequences that stop generation. | | `tool_choice` | `str` | `"auto"` | How the model decides when to call tools: `"auto"`, `"required"`, `"none"`, or a specific tool name. | | `cache_system` | `bool \| None` | `None` | Add a prompt-cache checkpoint after the system prompt to reduce input token usage. | | `cache_tools` | `bool \| None` | `None` | Add a prompt-cache checkpoint after the tool definitions. | | `strip_thinking` | `bool \| None` | `None` | Remove `...` spans from the streamed text. Amazon Nova models emit chain-of-thought in these tags, which would otherwise be read aloud by TTS. | | `text_tool_calls` | `bool \| None` | `None` | Parse function calls a model prints as plain text instead of native Converse tool use. Auto-enabled for models that lack native tool use (e.g. Gemma) when left unset. | | `additional_request_fields` | `dict \| None` | `None` | Extra fields merged into `additionalModelRequestFields` for model-specific parameters. | See the [plugins overview](/plugins/overview) for how the LLM slot fits into the rest of the pipeline. # Azure OpenAI Source: https://docs.zeroruntime.ai/plugins/llm/azure Use the Azure OpenAI LLM plugin in a Zero Runtime pipeline. Setup and usage in Python. Azure OpenAI is an **LLM** plugin. It generates the agent's text responses from the caller's transcribed speech, with support for tool calling and streaming. ## Setup Set your Azure OpenAI API key in the worker environment. Generate a key from the [Azure AI Foundry portal](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/create-resource?pivots=web-portal): ```bash theme={null} export AZURE_OPENAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import AzureOpenAILLM llm = AzureOpenAILLM( azure_endpoint="https://.openai.azure.com/", deployment="gpt-4o", ) # Pipeline(llm=llm, ...) ``` `AzureOpenAILLM` ships in the Python SDK only. On Node JS, use [the OpenAI LLM plugin](/plugins/llm/openai) instead. ## Parameters *Constructor parameters for `AzureOpenAILLM`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | --------------------- | ------------------------------------ | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Azure OpenAI API key. Falls back to the `AZURE_OPENAI_API_KEY` environment variable when unset. | | `azure_endpoint` | `str \| None` | `None` | Full Azure OpenAI resource endpoint URL, e.g. `"https://.openai.azure.com/"`. Falls back to the `AZURE_OPENAI_ENDPOINT` environment variable when unset. | | `deployment` | `AzureOpenAILLMModel \| str \| None` | `None` | Name of the Azure OpenAI deployment (not the underlying model name) - the custom name chosen when deploying a model in the Azure portal or AI Foundry, e.g. `"gpt-4o-deployment"`. Must be provided. | | `api_version` | `str` | `"2024-10-21"` | Azure OpenAI REST API version date string. | | `temperature` | `float` | `0.7` | Sampling temperature in `[0.0, 2.0]`. Lower values produce more deterministic output. | | `max_output_tokens` | `int` | `1024` | Maximum tokens the model may generate per response. | | `top_p` | `float \| None` | `None` | Nucleus sampling probability mass in `(0.0, 1.0]`. Mutually exclusive with `temperature`. | | `frequency_penalty` | `float \| None` | `None` | Float in `[-2.0, 2.0]`. Penalizes token repetition based on cumulative frequency so far. | | `presence_penalty` | `float \| None` | `None` | Float in `[-2.0, 2.0]`. Penalizes tokens that have already appeared, encouraging topic diversity. | | `seed` | `int \| None` | `None` | Integer seed for deterministic sampling. | | `stop` | `str \| None` | `None` | String (or up to four strings) at which the model stops generating further tokens. | | `user` | `str \| None` | `None` | Opaque end-user identifier forwarded to Azure for abuse monitoring. | | `tool_choice` | `str \| None` | `None` | Controls how the model selects tools: `"none"`, `"auto"`, `"required"`, or a named tool dict. Defaults to `"auto"` when tools are present. | | `parallel_tool_calls` | `bool \| None` | `None` | When `True`, the model may emit multiple tool calls in a single response turn. | | `response_format` | `dict \| None` | `None` | Enforce a structured output schema - `{"type": "json_object"}` for JSON mode, or a JSON Schema dict for strict structured outputs. | | `reasoning_effort` | `str \| None` | `None` | Controls reasoning token budget for o-series (reasoning) models: `"low"`, `"medium"`, or `"high"`. Ignored for non-reasoning models. | The LLM's response is synthesized to speech by the [TTS](/plugins/overview) plugin configured in the pipeline. # Cerebras Source: https://docs.zeroruntime.ai/plugins/llm/cerebras Use the Cerebras LLM plugin in a Zero Runtime pipeline for fast inference. Setup, options, and usage in Python and JavaScript. Cerebras is an **LLM** plugin built for fast inference. It takes the transcribed conversation and generates the reply, running open models such as Llama and Qwen on Cerebras hardware. ## Setup Set your Cerebras API key in the worker environment. Generate a key from the [Cerebras cloud](https://cloud.cerebras.ai/): ```bash theme={null} export CEREBRAS_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import CerebrasLLM llm = CerebrasLLM( model="llama3.3-70b", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { CerebrasLLM } from '@zeroruntime/js-sdk/plugins'; const llm = CerebrasLLM({ model: 'llama3.3-70b', temperature: 0.7, }); // Pipeline({ llm, ... }) ``` ## Configuration Options *Constructor parameters for `CerebrasLLM`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ----------------------- | ------- | ---------------- | --------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Cerebras API key. Falls back to the `CEREBRAS_API_KEY` environment variable when unset. | | `model` | `str` | `"llama3.3-70b"` | Cerebras model used to generate replies (e.g. `gpt-oss-120b`, `zai-glm-4.7`, `llama3.3-70b`). | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `tool_choice` | `str` | `"auto"` | How the model decides when to call tools (`"auto"`, `"none"`, or `"required"`). | | `max_completion_tokens` | `int` | `None` | Caps the length of each reply. | | `top_p` | `float` | `None` | Nucleus-sampling probability cutoff. | | `seed` | `int` | `None` | Seed for reproducible sampling. | | `stop` | `str` | `None` | Stop sequence that halts generation. | | `user` | `str` | `None` | End-user identifier passed to the API. | ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | ---------------------- | | Python | `from zeroruntime.plugins import CerebrasLLM` | `CerebrasLLM(...)` | | Node JS | `import { CerebrasLLM } from '@zeroruntime/js-sdk/plugins'` | `CerebrasLLM({ ... })` | The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which synthesizes the agent's voice. # CometAPI Source: https://docs.zeroruntime.ai/plugins/llm/cometapi Use the CometAPI LLM plugin in a Zero Runtime pipeline to access many models through one endpoint. Setup, options, and usage in Python and JavaScript. CometAPI is an **LLM** plugin. It exposes many models behind a single OpenAI-compatible endpoint, so you can switch models without changing providers. It takes the transcribed conversation and generates the reply. ## Setup Set your CometAPI key in the worker environment. Generate a key from the [CometAPI console](https://www.cometapi.com/): ```bash theme={null} export COMETAPI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import CometAPILLM llm = CometAPILLM( model="gpt-4o-mini", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { CometAPILLM } from '@zeroruntime/js-sdk/plugins'; const llm = CometAPILLM({ model: 'gpt-4o-mini', temperature: 0.7, }); // Pipeline({ llm, ... }) ``` ## Configuration Options *Constructor parameters for `CometAPILLM`. The Python and Node JS SDKs share these field names. CometAPI is OpenAI-compatible and built on the [OpenAI](/plugins/llm/openai) plugin.* | Parameter | Type | Default | Description | | ----------------------- | ------- | --------------- | ----------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | CometAPI key. Falls back to the `COMETAPI_API_KEY` environment variable when unset. | | `model` | `str` | `"gpt-4o-mini"` | Model used to generate replies. | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `max_completion_tokens` | `int` | `None` | Caps the length of each reply. | ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | ---------------------- | | Python | `from zeroruntime.plugins import CometAPILLM` | `CometAPILLM(...)` | | Node JS | `import { CometAPILLM } from '@zeroruntime/js-sdk/plugins'` | `CometAPILLM({ ... })` | The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which synthesizes the agent's voice. # Google Gemini Source: https://docs.zeroruntime.ai/plugins/llm/google Use the Google Gemini LLM plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Google Gemini is an **LLM** plugin. It takes the transcribed conversation and generates the reply. ## Setup Set your Google API key in the worker environment. Generate a key from the [Google AI Studio](https://aistudio.google.com/apikey): ```bash theme={null} export GOOGLE_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import GoogleLLM llm = GoogleLLM( model="gemini-2.5-flash", thinking_budget=0, include_thoughts=False, max_output_tokens=8192, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { GoogleLLM } from '@zeroruntime/js-sdk/plugins'; const llm = GoogleLLM({ model: 'gemini-2.5-flash', thinking_budget: 0, include_thoughts: false, max_output_tokens: 8192, }); // Pipeline({ llm, ... }) ``` ## Vertex AI By default the plugin calls the public Gemini API with `GOOGLE_API_KEY`. To run Gemini through **Vertex AI** instead, enable the Vertex backend and supply your Google Cloud project and location. Authenticate with a service account. Set `GOOGLE_APPLICATION_CREDENTIALS`: ```bash theme={null} export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json ``` ```python Python theme={null} from zeroruntime.plugins import GoogleLLM llm = GoogleLLM( model="gemini-2.5-flash", vertexai=True, project_id="your-gcp-project-id", location="us-central1", ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { GoogleLLM } from '@zeroruntime/js-sdk/plugins'; const llm = GoogleLLM({ model: 'gemini-2.5-flash', vertexai: true, vertexai_config: { project_id: 'your-gcp-project-id', location: 'us-central1', }, }); // Pipeline({ llm, ... }) ``` `location` defaults to `us-central1` in all SDKs. In Python, `project_id` is required when `vertexai=True` and must be passed explicitly. Node JS carries the same two settings inside `vertexai_config`. ## Configuration Options *Constructor parameters for `GoogleLLM`. The Python and Node JS SDKs share these field names.* ### Core | Parameter | Type | Default | Description | | ------------------- | ------- | ------------------------- | ----------------------------------------------------------------------------------- | | `model` | `str` | `"gemini-2.5-flash-lite"` | Gemini model used to generate replies. | | `api_key` | `str` | `None` | Google API key. Falls back to the `GOOGLE_API_KEY` environment variable when unset. | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `max_output_tokens` | `int` | `8192` | Caps the length of each reply. | | `top_p` | `float` | `None` | Nucleus-sampling probability cutoff. | | `top_k` | `int` | `None` | Top-k sampling cutoff. | | `presence_penalty` | `float` | `None` | Penalizes tokens already present, encouraging new topics. | | `frequency_penalty` | `float` | `None` | Penalizes frequent tokens, reducing repetition. | ### Generation knobs | Parameter | Type | Default | Description | | --------- | ----- | ------- | ------------------------------- | | `seed` | `int` | `None` | Seed for reproducible sampling. | ### Vertex AI | Parameter | Type | Default | Description | | ---------------------- | ------------- | --------------- | ----------------------------------------------------------------------------------------------------------------- | | `vertexai` | `bool` | `False` | Route requests through Vertex AI instead of the Gemini API. | | `project_id` | `str` | `None` | Google Cloud project ID. Required when `vertexai` is enabled. | | `location` | `str` | `"us-central1"` | Vertex AI regional endpoint used when `vertexai` is enabled. | | `service_account_json` | `str \| dict` | `None` | Service-account credentials as a JSON string or dict, used for Vertex AI authentication. | | `service_account_path` | `str` | `None` | Path to a service-account JSON key file. Falls back to the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. | ### Safety | Parameter | Type | Default | Description | | ----------------- | ------ | ------- | -------------------------- | | `safety_settings` | `list` | `None` | Content-safety thresholds. | ### Extended thinking | Parameter | Type | Default | Description | | ------------------ | ------ | ------- | ------------------------------------------------------------------------ | | `thinking_budget` | `int` | `0` | Tokens the model may spend reasoning. `0` disables it for lower latency. | | `include_thoughts` | `bool` | `False` | Include the model's reasoning in the output. | For voice, short replies feel best. A zero or small thinking budget and a concise system instruction keep latency down. ## Import paths | SDK | Import | Constructor | | ------- | --------------------------------------------------------- | -------------------- | | Python | `from zeroruntime.plugins import GoogleLLM` | `GoogleLLM(...)` | | Node JS | `import { GoogleLLM } from '@zeroruntime/js-sdk/plugins'` | `GoogleLLM({ ... })` | # Groq Source: https://docs.zeroruntime.ai/plugins/llm/groq Use the Groq LLM plugin in a Zero Runtime pipeline for low-latency inference. Setup, options, and usage in Python. Groq is an **LLM** plugin built for low-latency inference. It takes the transcribed conversation and generates the reply, running open models such as Llama on Groq's LPU hardware. ## Setup Set your Groq API key in the worker environment. Generate a key from the [Groq console](https://console.groq.com/keys): ```bash theme={null} export GROQ_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import GroqLLM llm = GroqLLM( model="llama-3.3-70b-versatile", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` `GroqLLM` ships in the Python SDK only. On Node JS, use [another LLM plugin](/plugins/llm/openai) instead. ## Configuration Options *Constructor parameters for `GroqLLM`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | --------------------- | ------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `api_key` | `str` | `None` | Groq API key. Falls back to the `GROQ_API_KEY` environment variable when unset. | | `model` | `str` | `"llama-3.3-70b-versatile"` | Groq model used to generate replies (e.g. `llama-3.1-8b-instant`, `openai/gpt-oss-20b`, `openai/gpt-oss-120b`, `groq/compound`, `groq/compound-mini`). | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `max_output_tokens` | `int` | `1024` | Maximum tokens to generate per response. | | `top_p` | `float` | `None` | Nucleus-sampling probability cutoff. | | `frequency_penalty` | `float` | `None` | Penalises tokens by how often they appear, reducing repetition. | | `presence_penalty` | `float` | `None` | Penalises tokens that have already appeared, encouraging topic diversity. | | `seed` | `int` | `None` | Seed for reproducible sampling. | | `stop` | `str` | `None` | Stop sequence that halts generation. | | `user` | `str` | `None` | End-user identifier passed to the API. | | `tool_choice` | `str` | `None` | How the model decides when to call tools (`"auto"`, `"none"`, `"required"`, or a specific tool name). | | `parallel_tool_calls` | `bool` | `None` | When `True`, the model may emit multiple tool calls in a single turn. | | `response_format` | `dict` | `None` | Structured output format descriptor (e.g. `{"type": "json_object"}`). | | `reasoning_effort` | `str` | `None` | Reasoning token budget hint for supported reasoning models (`"low"`, `"medium"`, `"high"`). | | `reasoning_format` | `str` | `None` | Controls how the reasoning chain is surfaced (`"parsed"`, `"raw"`, `"hidden"`). | | `service_tier` | `str` | `None` | Groq service tier (`"on_demand"`, `"flex"`, `"performance"`, `"auto"`). | ## Import paths | SDK | Import | Constructor | | ------ | ----------------------------------------- | -------------- | | Python | `from zeroruntime.plugins import GroqLLM` | `GroqLLM(...)` | The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which synthesizes the agent's voice. # OpenAI Source: https://docs.zeroruntime.ai/plugins/llm/openai Use the OpenAI LLM plugin in a Zero Runtime pipeline, including Azure OpenAI. Setup, options, and usage in Python and JavaScript. OpenAI is an **LLM** plugin. It takes the transcribed conversation and generates the reply using OpenAI's chat models, with tool calling and an optional low-latency WebSocket streaming mode. ## Setup Set your OpenAI API key in the worker environment. Generate a key from the [OpenAI dashboard](https://platform.openai.com/api-keys): ```bash theme={null} export OPENAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import OpenAILLM llm = OpenAILLM( model="gpt-5.4-nano", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { OpenAILLM } from '@zeroruntime/js-sdk/plugins'; const llm = OpenAILLM({ model: 'gpt-5.4-nano', temperature: 0.7, }); // Pipeline({ llm, ... }) ``` The default model is `gpt-5.4-nano`. ## Low-latency streaming Set `streaming` to use OpenAI's WebSocket Responses API instead of HTTP chat completions. The connection is reused across turns and continues with `previous_response_id` for lower per-turn latency. ```python Python theme={null} from zeroruntime.plugins import OpenAILLM llm = OpenAILLM(model="gpt-5.4-nano", streaming=True) ``` ```typescript Node JS theme={null} import { OpenAILLM } from '@zeroruntime/js-sdk/plugins'; const llm = OpenAILLM({ model: 'gpt-5.4-nano', streaming: true }); ``` ## Azure OpenAI To run OpenAI chat models through **Azure OpenAI**, use the separate `AzureOpenAILLM` plugin. It reads its configuration from Azure environment variables when arguments are omitted: ```bash theme={null} export AZURE_OPENAI_API_KEY= export AZURE_OPENAI_ENDPOINT=https://.openai.azure.com ``` ```python Python theme={null} from zeroruntime.plugins import AzureOpenAILLM llm = AzureOpenAILLM( azure_endpoint="https://.openai.azure.com", deployment="", api_version="2024-10-21", ) # Pipeline(llm=llm, ...) ``` `AzureOpenAILLM` ships in the Python SDK only. The Node JS SDK does not ship an Azure OpenAI provider. `deployment` is the custom name you chose when deploying a model in Azure (not the underlying model name). Authenticate with `api_key`, or fall back to the `AZURE_OPENAI_API_KEY` environment variable. ## Configuration Options *Constructor parameters for `OpenAILLM`. The Python and Node JS SDKs share these field names.* ### Core | Parameter | Type | Default | Description | | ------------------- | ------- | ---------------- | ----------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable when unset. | | `model` | `str` | `"gpt-5.4-nano"` | Chat model used to generate replies. | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `tool_choice` | `str` | `None` | How the model decides when to call tools (`"none"`, `"auto"`, `"required"`, or a specific tool spec). | | `max_output_tokens` | `int` | `1024` | Caps the length of each reply. The legacy `max_completion_tokens` kwarg is accepted as an alias. | | `response_format` | `dict` | `None` | Output format spec, e.g. `{"type": "json_object"}` or a `json_schema` for Structured Outputs. | | `stop` | `str` | `None` | Up to 4 stop sequences at which generation halts. | | `user` | `str` | `None` | Stable end-user identifier for abuse monitoring. | ### Sampling | Parameter | Type | Default | Description | | --------------------- | ------- | ------- | --------------------------------------------------------- | | `top_p` | `float` | `None` | Nucleus-sampling probability cutoff. | | `frequency_penalty` | `float` | `None` | Penalizes frequent tokens, reducing repetition. | | `presence_penalty` | `float` | `None` | Penalizes tokens already present, encouraging new topics. | | `seed` | `int` | `None` | Seed for deterministic sampling. | | `parallel_tool_calls` | `bool` | `None` | Allow the model to call multiple tools in one turn. | ### Reasoning models | Parameter | Type | Default | Description | | ------------------ | ----- | ------- | -------------------------------------------------------------------------------------------------------------------------------- | | `reasoning_effort` | `str` | `None` | Reasoning depth (`"minimal"`, `"low"`, `"medium"`, `"high"`) for reasoning / GPT-5 models. `"none"` requires `gpt-5.1` or later. | | `verbosity` | `str` | `None` | Output verbosity (`"low"`, `"medium"`, `"high"`) for reasoning / GPT-5 models. | ### Streaming & client | Parameter | Type | Default | Description | | ----------- | ------ | ------- | ----------------------------------------------------------- | | `streaming` | `bool` | `False` | Use the WebSocket Responses API for lower per-turn latency. | | `store` | `bool` | `True` | Persist responses server-side for later retrieval. | | `wss_url` | `str` | `None` | Override the WebSocket Responses URL. | ## Import paths | SDK | Import | Constructor | | ------- | --------------------------------------------------------- | -------------------- | | Python | `from zeroruntime.plugins import OpenAILLM` | `OpenAILLM(...)` | | Node JS | `import { OpenAILLM } from '@zeroruntime/js-sdk/plugins'` | `OpenAILLM({ ... })` | Azure OpenAI is provided by the separate `AzureOpenAILLM` plugin (`from zeroruntime.plugins import AzureOpenAILLM`). The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which synthesizes the agent's voice. # Sarvam AI Source: https://docs.zeroruntime.ai/plugins/llm/sarvamai Use the Sarvam AI LLM plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Sarvam AI is an **LLM** plugin tuned for Indian languages. It takes the transcribed conversation and generates the reply, with optional Wikipedia grounding. Sarvam AI is also available as a [speech-to-text](/plugins/stt/sarvamai) plugin. ## Setup Set your Sarvam AI API key in the worker environment. Generate a key from the [Sarvam AI dashboard](https://dashboard.sarvam.ai/key-management): ```bash theme={null} export SARVAMAI_API_KEY= ``` The runtime picks up `SARVAMAI_API_KEY` from the worker environment. If you pass the key directly to the constructor's `api_key` parameter and leave it unset, the SDK falls back to the `SARVAM_API_KEY` environment variable instead. ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import SarvamAILLM llm = SarvamAILLM( model="sarvam-105b", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { SarvamAILLM } from '@zeroruntime/js-sdk/plugins'; const llm = SarvamAILLM({ model: 'sarvam-105b', temperature: 0.7, }); // Pipeline({ llm, ... }) ``` ## Configuration Options *Constructor parameters for `SarvamAILLM`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ----------------------- | ------- | -------------- | -------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Sarvam AI API key. Falls back to the `SARVAM_API_KEY` environment variable when unset. | | `model` | `str` | `"sarvam-30b"` | Sarvam model used to generate replies. | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `tool_choice` | `str` | `"auto"` | How the model decides when to call tools. | | `max_completion_tokens` | `int` | `None` | Caps the length of each reply. | | `reasoning_effort` | `str` | `None` | Reasoning depth: `"low"`, `"medium"`, or `"high"`. | | `wiki_grounding` | `bool` | `False` | Enable Wikipedia search to ground responses. | | `top_p` | `float` | `None` | Nucleus-sampling cutoff, as an alternative to `temperature`. | | `frequency_penalty` | `float` | `None` | Penalizes frequent tokens (range -2.0 to 2.0). | | `presence_penalty` | `float` | `None` | Penalizes tokens already present (range -2.0 to 2.0). | | `stop` | `str` | `None` | Up to 4 sequences that halt generation. | ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | ---------------------- | | Python | `from zeroruntime.plugins import SarvamAILLM` | `SarvamAILLM(...)` | | Node JS | `import { SarvamAILLM } from '@zeroruntime/js-sdk/plugins'` | `SarvamAILLM({ ... })` | The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which synthesizes the agent's voice. # xAI Grok Source: https://docs.zeroruntime.ai/plugins/llm/xai Use the xAI Grok LLM plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. xAI Grok is an **LLM** plugin. It takes the transcribed conversation and generates the reply using xAI's Grok models, with client-side function calling. ## Setup Set your xAI API key in the worker environment. Generate a key from the [xAI console](https://console.x.ai): ```bash theme={null} export XAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `llm` slot. ```python Python theme={null} from zeroruntime.plugins import XAILLM llm = XAILLM( model="grok-4-1-fast-non-reasoning", temperature=0.7, ) # Pipeline(llm=llm, ...) ``` ```typescript Node JS theme={null} import { XAILLM } from '@zeroruntime/js-sdk/plugins'; const llm = XAILLM({ model: 'grok-4-1-fast-non-reasoning', temperature: 0.7, }); // Pipeline({ llm, ... }) ``` ## Configuration Options *Constructor parameters for `XAILLM`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ----------------------- | ------- | ------------------------------- | ----------------------------------------------------------------------------- | | `api_key` | `str` | `None` | xAI API key. Falls back to the `XAI_API_KEY` environment variable when unset. | | `model` | `str` | `"grok-4-1-fast-non-reasoning"` | Grok model used to generate replies (e.g. `grok-4`, `grok-4-1-fast`). | | `base_url` | `str` | `"https://api.x.ai/v1"` | xAI API base URL. | | `temperature` | `float` | `0.7` | Sampling randomness; lower is more deterministic. | | `tool_choice` | `str` | `"auto"` | How the model decides when to call tools. | | `max_completion_tokens` | `int` | `None` | Caps the length of each reply. | ## Import paths | SDK | Import | Constructor | | ------- | ------------------------------------------------------ | ----------------- | | Python | `from zeroruntime.plugins import XAILLM` | `XAILLM(...)` | | Node JS | `import { XAILLM } from '@zeroruntime/js-sdk/plugins'` | `XAILLM({ ... })` | The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which synthesizes the agent's voice. # Plugins Source: https://docs.zeroruntime.ai/plugins/overview The Zero Runtime plugin catalog. Choose a plugin for each pipeline stage: speech-to-text, the LLM, text-to-speech, and more, with usage for Python and JavaScript. A plugin handles one stage of the [pipeline](/concepts/pipeline). You import it from `zeroruntime.plugins` (Python) or `@zeroruntime/js-sdk/plugins` (Node JS) and pass it to the pipeline. Each plugin page below shows setup and usage in Python and JavaScript. Plugins are grouped by stage, because that's how you choose them: pick the best speech-to-text, LLM, and text-to-speech for your agent. ## Plugins by provider | Provider | STT | LLM | TTS | Realtime | | ------------ | :-------------------------------------: | :-------------------------------------: | :----------------------------: | :------------------------------: | | OpenAI | [Yes](/plugins/stt/openai) | [Yes](/plugins/llm/openai) | [Yes](/plugins/tts/openai) | [Yes](/plugins/realtime/openai) | | Google | [Yes](/plugins/stt/google) | [Gemini](/plugins/llm/google) | [Yes](/plugins/tts/google) | [Live](/plugins/realtime/gemini) | | Deepgram | [Yes](/plugins/stt/deepgram) | - | [Yes](/plugins/tts/deepgram) | - | | Cartesia | [Yes](/plugins/stt/cartesia) | - | [Yes](/plugins/tts/cartesia) | - | | AssemblyAI | [Yes](/plugins/stt/assemblyai) | - | - | - | | Sarvam AI | [Yes](/plugins/stt/sarvamai) | [Yes](/plugins/llm/sarvamai) | Supported | - | | Anthropic | - | [Yes](/plugins/llm/anthropic) | - | - | | Groq | - | [Yes](/plugins/llm/groq) | Supported | - | | Cerebras | - | [Yes](/plugins/llm/cerebras) | - | - | | xAI Grok | - | [Yes](/plugins/llm/xai) | - | - | | CometAPI | - | [Yes](/plugins/llm/cometapi) | - | - | | ElevenLabs | - | - | [Yes](/plugins/tts/elevenlabs) | - | | Smallest AI | - | - | [Yes](/plugins/tts/smallestai) | - | | Azure OpenAI | [Yes](/plugins/stt/openai#azure-openai) | [Yes](/plugins/llm/openai#azure-openai) | Supported | - | A "Supported" cell works today but is not documented yet; "-" means that provider doesn't serve that stage. Availability can vary by SDK, so check each plugin page. Turn detection (Echo), voice activity detection (Silero), and noise cancellation ([RNNoise](/plugins/denoise/rnnoise), [Sanas](/plugins/denoise/sanas), [ai-coustics](/plugins/denoise/aicoustics)) are listed below. ## Speech-to-text | Plugin | Status | | -------------------------------------------------- | ---------- | | [Deepgram](/plugins/stt/deepgram) | Documented | | [AssemblyAI](/plugins/stt/assemblyai) | Documented | | [Google Cloud STT](/plugins/stt/google) | Documented | | [Sarvam AI](/plugins/stt/sarvamai) | Documented | | [OpenAI](/plugins/stt/openai) (incl. Azure OpenAI) | Documented | | [Cartesia](/plugins/stt/cartesia) | Documented | | Azure, Gladia, NVIDIA | Supported | ## LLM | Plugin | Status | | -------------------------------------------------- | ---------- | | [Google Gemini](/plugins/llm/google) | Documented | | [OpenAI](/plugins/llm/openai) (incl. Azure OpenAI) | Documented | | [Anthropic Claude](/plugins/llm/anthropic) | Documented | | [Groq](/plugins/llm/groq) | Documented | | [Cerebras](/plugins/llm/cerebras) | Documented | | [xAI Grok](/plugins/llm/xai) | Documented | | [Sarvam AI](/plugins/llm/sarvamai) | Documented | | [CometAPI](/plugins/llm/cometapi) | Documented | ## Text-to-speech | Plugin | Status | | ---------------------------------------------------------- | ---------- | | [Cartesia](/plugins/tts/cartesia) | Documented | | [ElevenLabs](/plugins/tts/elevenlabs) | Documented | | [Google Cloud TTS](/plugins/tts/google) | Documented | | [Deepgram](/plugins/tts/deepgram) | Documented | | [Smallest AI](/plugins/tts/smallestai) | Documented | | [OpenAI](/plugins/tts/openai) | Documented | | AWS Polly, Azure, Rime, LMNT, Neuphonic, Hume AI, and more | Supported | ## Realtime models Speech-to-speech models that run in the pipeline's `llm` slot (no separate STT/TTS). See [Realtime mode](/build/configure-a-pipeline/modes). | Plugin | Status | | ------------------------------------------- | ---------- | | [OpenAI Realtime](/plugins/realtime/openai) | Documented | | [Gemini Live](/plugins/realtime/gemini) | Documented | ## Turn detection | Plugin | Status | | --------------------------------------------- | ---------- | | [Turn Detector](/plugins/turn-detection/namo) | Documented | ## Voice activity detection | Plugin | Status | | ----------------------------- | ---------- | | [Silero](/plugins/vad/silero) | Documented | ## Noise cancellation | Plugin | Status | | ------------------------------------------ | -------------------- | | [RNNoise](/plugins/denoise/rnnoise) | Documented (local) | | [Sanas](/plugins/denoise/sanas) | Documented (gateway) | | [ai-coustics](/plugins/denoise/aicoustics) | Documented (gateway) | Plugins marked **Supported** work today and are added the same way; their pages will follow. Availability can vary by SDK, so check your SDK's plugin list. # Azure Voice Live Source: https://docs.zeroruntime.ai/plugins/realtime/azure Use the Azure Voice Live speech-to-speech model in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Azure Voice Live is a **speech-to-speech** model. It unifies speech recognition, generative AI, and text-to-speech into a single Microsoft Azure endpoint, so it goes in the pipeline's `llm` slot with no separate STT or TTS. The `Pipeline` auto-detects [Realtime mode](/build/configure-a-pipeline/modes) when you pass it. ## Setup Set your Azure Voice Live API key in the worker environment. Generate a key from the [Azure AI Foundry portal](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/create-resource?pivots=web-portal): ```bash theme={null} export AZURE_VOICE_LIVE_API_KEY= ``` Azure Voice Live also needs a service endpoint. Pass it explicitly as `endpoint` (or set an `endpoint` key in `config`), or export it as `AZURE_VOICE_LIVE_ENDPOINT` and it will be picked up automatically. ```bash theme={null} export AZURE_VOICE_LIVE_ENDPOINT= ``` ## Usage Pass the realtime model to the pipeline's `llm` slot, no `stt` or `tts` needed. ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import AzureVoiceLive llm = AzureVoiceLive( model="gpt-4o-realtime-preview", config={ "voice": "en-US-AvaNeural", "modalities": ["text", "audio"], }, ) pipeline = Pipeline(llm=llm) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { AzureVoiceLive } from '@zeroruntime/js-sdk/plugins'; const llm = AzureVoiceLive({ model: 'gpt-4o-realtime-preview', config: { voice: 'en-US-AvaNeural', modalities: ['text', 'audio'], }, }); const pipeline = Pipeline({ llm }); ``` ## Parameters *Constructor parameters for `AzureVoiceLive`. The Python and Node JS SDKs share these field names. Model behavior is configured through the `config` mapping.* | Parameter | Type | Default | Description | | ---------- | ------ | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | `str` | `"gpt-4o-realtime-preview"` | Voice Live model ID. | | `api_key` | `str` | `None` | Azure Voice Live API key. Falls back to the `AZURE_VOICE_LIVE_API_KEY` environment variable when unset. | | `config` | `dict` | `None` | Model behavior configuration (see below). When omitted, provider defaults are used. | | `endpoint` | `str` | `None` | Azure Voice Live service endpoint. Falls back to `config.endpoint`, then the `AZURE_VOICE_LIVE_ENDPOINT` environment variable, then an empty string. | ### `config` keys | Field | Type | Default | Description | | ---------------------------- | ------------------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `voice` | `str` | `"en-US-AvaNeural"` | Output voice: an Azure neural voice (e.g. `"en-US-AvaNeural"`) or an OpenAI voice (e.g. `"alloy"`). | | `endpoint` | `str` | `None` | Azure Voice Live service endpoint. | | `modalities` | `list[str]` | `["text", "audio"]` | Enabled response modalities. Drop `"audio"` for text-only. | | `temperature` | `float` | `None` | Sampling randomness. | | `max_response_output_tokens` | `int \| str` | `None` | Caps the length of each response. | | `turn_detection` | `TurnDetectionConfig` | server VAD | Turn detection (`server_vad`, threshold `0.5`, `prefix_padding_ms` `300`, `silence_duration_ms` `500`, `create_response` `True`, `interrupt_response` `True`). | | `input_audio_transcription` | `InputAudioTranscriptionConfig` | `gpt-4o-mini-transcribe` | Transcription model for the caller's audio. | | `tool_choice` | `str` | `"auto"` | How the model decides when to call tools. | To pair the realtime model with an external STT or TTS instead, see [the plugins overview](/plugins/overview). # Gemini Live Source: https://docs.zeroruntime.ai/plugins/realtime/gemini Use the Google Gemini Live speech-to-speech model in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Gemini Live is a **speech-to-speech** model. It handles transcription, reasoning, and voice synthesis end-to-end, so it goes in the pipeline's `llm` slot with no separate STT or TTS. The `Pipeline` auto-detects [Realtime mode](/build/configure-a-pipeline/modes) when you pass it. ## Setup Set your Google API key in the worker environment. Generate a key from the [Google AI Studio](https://aistudio.google.com/apikey): ```bash theme={null} export GOOGLE_API_KEY= ``` To run Gemini Live through **Vertex AI**, set `vertexai=True` and authenticate with a service account (`GOOGLE_APPLICATION_CREDENTIALS`), as with the [Gemini LLM](/plugins/llm/google#vertex-ai) plugin. ## Usage Pass the realtime model to the pipeline's `llm` slot, no `stt` or `tts` needed. ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import GeminiRealtime model = GeminiRealtime( model="gemini-3.1-flash-live-preview", config={ "voice": "Puck", "response_modalities": ["AUDIO"], }, ) pipeline = Pipeline(llm=model) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { GeminiRealtime } from '@zeroruntime/js-sdk/plugins'; const model = GeminiRealtime({ model: 'gemini-3.1-flash-live-preview', config: { voice: 'Puck', response_modalities: ['AUDIO'], }, }); const pipeline = Pipeline({ llm: model }); ``` The default model is `gemini-3.1-flash-live-preview` in every SDK. Pass `model` explicitly for consistency. ## Configuration *`GeminiRealtime` constructor and `config` keys in both SDKs. The Python and Node JS SDKs share these field names.* ### Constructor | Parameter | Type | Default | Description | | --------- | ------ | --------------------------------- | ----------------------------------------------------------------------------------- | | `model` | `str` | `"gemini-3.1-flash-live-preview"` | Gemini Live model ID. | | `api_key` | `str` | `None` | Gemini API key. Falls back to the `GOOGLE_API_KEY` environment variable when unset. | | `config` | `dict` | `None` | Model behavior configuration (see below). | ### `config` keys | Field | Type | Default | Description | | ------------------------------------ | --------------- | --------------- | -------------------------------------------------------------------------------------------------------- | | `voice` | `str` | `"Puck"` | Output voice: `Puck`, `Charon`, `Kore`, `Fenrir`, or `Aoede`. | | `language_code` | `str` | `None` | Language for speech synthesis. | | `response_modalities` | `list` | `["AUDIO"]` | Enabled response types (`"TEXT"`, `"AUDIO"`). | | `top_p` | `float` | `None` | Nucleus-sampling cutoff. | | `top_k` | `int` | `None` | Top-k sampling cutoff. | | `max_output_tokens` | `int` | `None` | Caps the length of each response. | | `thinking_budget` | `int` | `None` | Thinking budget; `0` disables it for low-latency voice (native-audio models only). | | `include_thoughts` | `bool` | `False` | Include the model's thought summaries in the response. | | `vad_start_sensitivity` | `str` | `None` | Start-of-speech sensitivity for automatic activity detection. | | `vad_end_sensitivity` | `str` | `None` | End-of-speech sensitivity for automatic activity detection. | | `vad_prefix_padding_ms` | `int` | `None` | Audio padding retained before detected speech, in milliseconds. | | `vad_silence_duration_ms` | `int` | `None` | Silence duration that ends a turn, in milliseconds. | | `context_compression_trigger_tokens` | `int` | `None` | Token count that triggers context-window compression to extend sessions past the connection cap. | | `session_resumption_handle` | `str` | `None` | Handle used to resume a previous session on reconnect. | | `vertexai` | `bool` | `False` | Route through Vertex AI instead of the Gemini API. | | `vertex_project_id` | `str` | `None` | Google Cloud project ID for Vertex AI (required when `vertexai=True`). | | `vertex_location` | `str` | `"us-central1"` | Vertex AI region. | | `vertex_service_account_json` | `str` or `dict` | `None` | Inline service-account credentials for Vertex AI. | | `vertex_service_account_path` | `str` | `None` | Path to a service-account JSON for Vertex AI. Falls back to `GOOGLE_APPLICATION_CREDENTIALS` when unset. | ## Import paths | SDK | Import | Constructor | | ------- | -------------------------------------------------------------- | ----------------------------------------- | | Python | `from zeroruntime.plugins import GeminiRealtime` | `GeminiRealtime(model=..., config={...})` | | Node JS | `import { GeminiRealtime } from '@zeroruntime/js-sdk/plugins'` | `GeminiRealtime({ ... })` | To pair the realtime model with an external STT or TTS, see [Hybrid mode](/build/configure-a-pipeline/modes#hybrid). # OpenAI Realtime Source: https://docs.zeroruntime.ai/plugins/realtime/openai Use the OpenAI Realtime speech-to-speech model in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. OpenAI Realtime is a **speech-to-speech** model. It handles transcription, reasoning, and voice synthesis end-to-end in a single model, so it goes in the pipeline's `llm` slot with no separate STT or TTS. The `Pipeline` auto-detects [Realtime mode](/build/configure-a-pipeline/modes) when you pass it. ## Setup Set your OpenAI API key in the worker environment. Generate a key from the [OpenAI dashboard](https://platform.openai.com/api-keys): ```bash theme={null} export OPENAI_API_KEY= ``` ## Usage Pass the realtime model to the pipeline's `llm` slot, no `stt` or `tts` needed. ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import OpenAIRealtime model = OpenAIRealtime( model="gpt-4o-realtime-preview", config={ "voice": "alloy", "modalities": ["text", "audio"], }, ) pipeline = Pipeline(llm=model) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { OpenAIRealtime } from '@zeroruntime/js-sdk/plugins'; const model = OpenAIRealtime({ model: 'gpt-4o-realtime-preview', config: { voice: 'alloy', modalities: ['text', 'audio'], }, }); const pipeline = Pipeline({ llm: model }); ``` ## Configuration *`OpenAIRealtime` constructor and `config` keys in both SDKs. The Python and Node JS SDKs share these field names.* ### Constructor | Parameter | Type | Default | Description | | --------- | ------ | --------------------------- | ----------------------------------------------------------------------------------- | | `model` | `str` | `"gpt-4o-realtime-preview"` | Realtime model ID. | | `api_key` | `str` | `None` | OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable when unset. | | `config` | `dict` | `None` | Model behavior configuration (see below). | ### `config` keys | Field | Type | Default | Description | | --------------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | `voice` | `str` | `"alloy"` | Output voice (e.g. `alloy`, `ash`, `marin`, `cedar`, `verse`). | | `temperature` | `float` | `0.8` | Sampling randomness. | | `turn_detection` | `TurnDetectionConfig` | server VAD | Turn detection (`server_vad`, threshold `0.5`, `prefix_padding_ms` `300`, `silence_duration_ms` `200`). Pass `None` to disable. | | `input_audio_transcription` | `InputAudioTranscriptionConfig` | `gpt-4o-mini-transcribe` | Transcription model for the user's audio. | | `tool_choice` | `str` | `"auto"` | How the model decides when to call tools. | | `modalities` | `list[str]` | `["text", "audio"]` | Enabled response modalities. Drop `"audio"` for text-only. | ## Import paths | SDK | Import | Constructor | | ------- | -------------------------------------------------------------- | ----------------------------------------- | | Python | `from zeroruntime.plugins import OpenAIRealtime` | `OpenAIRealtime(model=..., config={...})` | | Node JS | `import { OpenAIRealtime } from '@zeroruntime/js-sdk/plugins'` | `OpenAIRealtime({ ... })` | To pair the realtime model with an external STT or TTS, see [Hybrid mode](/build/configure-a-pipeline/modes#hybrid). # Ultravox Source: https://docs.zeroruntime.ai/plugins/realtime/ultravox Use the Ultravox speech-to-speech model in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Ultravox is a **speech-to-speech** model. It handles transcription, reasoning, and voice synthesis end-to-end in a single model, so it goes in the pipeline's `llm` slot with no separate STT or TTS. ## Setup Set your Ultravox API key in the worker environment. Generate a key from the [Ultravox dashboard](https://app.ultravox.ai/): ```bash theme={null} export ULTRAVOX_API_KEY= ``` ## Usage Pass the realtime model to the pipeline's `llm` slot, no `stt` or `tts` needed. ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import UltravoxRealtime model = UltravoxRealtime( model="fixie-ai/ultravox", config={ "voice": "54ebeae1-88df-4d66-af13-6c41283b4332", "language_hint": "en", }, ) pipeline = Pipeline(llm=model) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { UltravoxRealtime } from '@zeroruntime/js-sdk/plugins'; const model = UltravoxRealtime({ model: 'fixie-ai/ultravox', config: { voice: '54ebeae1-88df-4d66-af13-6c41283b4332', language_hint: 'en', }, }); const pipeline = Pipeline({ llm: model }); ``` ## Parameters *Constructor parameters for `UltravoxRealtime`. The Python and Node JS SDKs share these field names. Model behavior is configured through the `config` mapping.* ### Constructor | Parameter | Type | Default | Description | | --------- | ------ | --------------------- | --------------------------------------------------------------------------------------- | | `model` | `str` | `"fixie-ai/ultravox"` | Ultravox model to use. | | `api_key` | `str` | `None` | Ultravox API key. Falls back to the `ULTRAVOX_API_KEY` environment variable when unset. | | `config` | `dict` | `None` | Model behavior configuration (see below). When omitted, provider defaults are used. | ### `config` keys | Field | Type | Default | Description | | -------------------------------------- | ------- | ---------------------- | --------------------------------------------------------------------------- | | `voice` | `str` | `None` | Voice ID for the synthesized speech. | | `language_hint` | `str` | `"en"` | Hint for the conversation's language. | | `temperature` | `float` | `None` | Controls the randomness of responses (0.0 to 1.0). | | `max_duration` | `str` | `None` | Maximum duration of the call (e.g. `"600s"`). | | `time_exceeded_message` | `str` | `None` | Message spoken when the maximum duration is exceeded. | | `input_sample_rate` | `int` | `48000` | Sample rate (Hz) of the input audio. | | `output_sample_rate` | `int` | `24000` | Sample rate (Hz) of the synthesized output audio. | | `client_buffer_size_ms` | `int` | `30000` | Client-side audio buffer size in milliseconds. | | `vad_turn_endpoint_delay_ms` | `int` | `800` | Milliseconds of silence before voice activity detection ends a turn. | | `vad_minimum_turn_duration_ms` | `int` | `600` | Minimum duration in milliseconds for a valid speech turn. | | `vad_minimum_interruption_duration_ms` | `int` | `None` | Minimum duration in milliseconds of speech required to interrupt the agent. | | `vad_frame_activation_threshold` | `float` | `0.4` | Frame activation threshold for voice activity detection. | | `first_speaker` | `str` | `"FIRST_SPEAKER_USER"` | Determines who speaks first. | | `enable_greeting_prompt` | `bool` | `False` | Whether to enable an initial greeting prompt. | | `base_url` | `str` | `None` | Override the Ultravox streaming endpoint. | To pair the realtime model with an external STT or TTS, see [Hybrid mode](/build/configure-a-pipeline/modes#hybrid). # xAI Grok Source: https://docs.zeroruntime.ai/plugins/realtime/xai Use the xAI Grok speech-to-speech model in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. xAI Grok Realtime is a **speech-to-speech** model. It handles transcription, reasoning, and voice synthesis end-to-end in a single model, so it goes in the pipeline's `llm` slot with no separate STT or TTS. The `Pipeline` auto-detects [Realtime mode](/build/configure-a-pipeline/modes) when you pass it. ## Setup Set your xAI API key in the worker environment. Generate a key from the [xAI console](https://console.x.ai): ```bash theme={null} export XAI_API_KEY= ``` ## Usage Pass the realtime model to the pipeline's `llm` slot, no `stt` or `tts` needed. ```python Python theme={null} from zeroruntime import Pipeline from zeroruntime.plugins import XAIRealtime llm = XAIRealtime( model="grok-realtime", config={ "voice": "Ara", "modalities": ["text", "audio"], }, ) pipeline = Pipeline(llm=llm) ``` ```typescript Node JS theme={null} import { Pipeline } from '@zeroruntime/js-sdk'; import { XAIRealtime } from '@zeroruntime/js-sdk/plugins'; const llm = XAIRealtime({ model: 'grok-realtime', config: { voice: 'Ara', modalities: ['text', 'audio'], }, }); const pipeline = Pipeline({ llm }); ``` ## Parameters *Constructor parameters for `XAIRealtime`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | --------- | ------ | ----------------- | ----------------------------------------------------------------------------------- | | `model` | `str` | `"grok-realtime"` | Grok realtime model ID. | | `api_key` | `str` | `None` | xAI API key. Falls back to the `XAI_API_KEY` environment variable when unset. | | `config` | `dict` | `None` | Model behavior configuration (see below). When omitted, provider defaults are used. | ### `config` keys | Field | Type | Default | Description | | ---------------------------- | ------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | | `voice` | `str` | `"Ara"` | Output voice. One of `"Ara"`, `"Rex"`, `"Sal"`, `"Eve"`, `"Leo"`. | | `modalities` | `list[str]` | `["text", "audio"]` | Enabled response types. Drop `"audio"` for text-only. | | `temperature` | `float` | `0.8` | Sampling randomness. | | `max_response_output_tokens` | `int \| str` | `"inf"` | Caps the length of each response. `"inf"` disables the cap. | | `turn_detection` | `TurnDetectionConfig` | server VAD | Turn detection (`server_vad`, threshold `0.5`, `prefix_padding_ms` `300`, `silence_duration_ms` `200`). Pass `None` to disable. | | `input_audio_transcription` | `InputAudioTranscriptionConfig` | `gpt-4o-mini-transcribe` | Transcription model for the user's audio. | | `tool_choice` | `str` | `"auto"` | How the model decides when to call tools. | | `base_url` | `str` | `None` | Override the xAI realtime API base URL. | To pair the realtime model with an external STT or TTS, see [Hybrid mode](/build/configure-a-pipeline/modes#hybrid). For general plugin concepts, see the [plugins overview](/plugins/overview). # AssemblyAI Source: https://docs.zeroruntime.ai/plugins/stt/assemblyai Use the AssemblyAI speech-to-text plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. AssemblyAI is a **speech-to-text** plugin. It transcribes the caller's audio into text for the LLM, using AssemblyAI's universal streaming model with built-in end-of-turn detection. ## Setup Set your AssemblyAI API key in the worker environment. Generate a key from the [AssemblyAI dashboard](https://www.assemblyai.com/dashboard/docs/your-api-key): ```bash theme={null} export ASSEMBLYAI_API_KEY= ``` The Python SDK requires `scipy` for audio resampling. Install it with `pip install scipy` if it isn't already present. ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import AssemblyAISTT stt = AssemblyAISTT() # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { AssemblyAISTT } from '@zeroruntime/js-sdk/plugins'; const stt = AssemblyAISTT(); // Pipeline({ stt, ... }) ``` ## Parameters *Constructor parameters for `AssemblyAISTT`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ---------------------------------------- | ----------- | ------------------------------- | ------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | AssemblyAI API key. Falls back to the `ASSEMBLYAI_API_KEY` environment variable when unset. | | `input_sample_rate` | `int` | `48000` | Sample rate (Hz) of the incoming audio. | | `output_sample_rate` | `int` | `16000` | Sample rate (Hz) the audio is resampled to before it is sent to AssemblyAI. | | `format_turns` | `bool` | `True` | Apply capitalization and punctuation to completed turns. | | `keyterms_prompt` | `list[str]` | `None` | Words or phrases to bias recognition toward. | | `end_of_turn_confidence_threshold` | `float` | `0.4` | Confidence above which a turn is considered finished. | | `min_end_of_turn_silence_when_confident` | `int` | `560` | Milliseconds of silence required to end a turn when end-of-turn confidence is high. | | `max_turn_silence` | `int` | `2400` | Maximum silence (ms) allowed within a turn before it ends. | | `speech_model` | `str` | `"universal-streaming-english"` | Recognition model: `"universal-streaming-english"` or `"universal-streaming-multilingual"`. | | `language_detection` | `bool` | `False` | Automatically detect the spoken language. | ## Import paths | SDK | Import | Constructor | | ------- | ------------------------------------------------------------- | ----------------- | | Python | `from zeroruntime.plugins import AssemblyAISTT` | `AssemblyAISTT()` | | Node JS | `import { AssemblyAISTT } from '@zeroruntime/js-sdk/plugins'` | `AssemblyAISTT()` | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # Azure Source: https://docs.zeroruntime.ai/plugins/stt/azure Use the Azure speech-to-text plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Azure is a **speech-to-text** plugin. It transcribes the caller's audio into text using Microsoft Azure's Speech service for the LLM. ## Setup Set your Azure Speech key in the worker environment. Create a Speech resource and key in the [Azure portal](https://portal.azure.com/): ```bash theme={null} export AZURE_SPEECH_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import AzureSTT stt = AzureSTT() # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { AzureSTT } from '@zeroruntime/js-sdk/plugins'; const stt = AzureSTT(); // Pipeline({ stt, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | -------------------- | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `speech_key` | `str` | `None` | Azure Speech resource subscription key. Falls back to the `AZURE_SPEECH_KEY` environment variable when unset. | | `speech_region` | `str` | `None` | Azure region identifier for the Speech resource (e.g. `"eastus"`, `"westeurope"`, `"southeastasia"`). Falls back to the `AZURE_REGION` environment variable, then `"eastus"`. Must match the region where your Azure Speech resource was created. | | `language` | `str` | `"en-US"` | BCP-47 locale tag for the spoken language. Over 100 locales are supported. | | `sample_rate` | `int` | `16000` | PCM input sample rate in Hz sent to Azure. | | `enable_phrase_list` | `bool` | `False` | When `True`, the phrases supplied in `phrase_list` are registered as recognition hints to improve accuracy for domain-specific terms. | | `phrase_list` | `list[str]` | `None` | List of words or phrases to use as recognition hints when `enable_phrase_list` is `True`. | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # Cartesia Source: https://docs.zeroruntime.ai/plugins/stt/cartesia Use the Cartesia speech-to-text plugin in a Zero Runtime pipeline. Setup, options, and usage. Cartesia is a **speech-to-text** plugin. It transcribes the caller's audio into text for the LLM over Cartesia's streaming WebSocket API, using the `ink-whisper` model. Cartesia is also available as a [text-to-speech](/plugins/tts/cartesia) plugin. ## Setup Set your Cartesia API key in the worker environment. Generate a key from the [Cartesia dashboard](https://play.cartesia.ai/keys): ```bash theme={null} export CARTESIA_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import CartesiaSTT stt = CartesiaSTT( model="ink-whisper", language="en", ) # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { CartesiaSTT } from '@zeroruntime/js-sdk/plugins'; const stt = CartesiaSTT({ model: 'ink-whisper', language: 'en', }); // Pipeline({ stt, ... }) ``` ## Parameters *Constructor parameters for `CartesiaSTT`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | -------------------- | ----- | --------------- | -------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Cartesia API key. Falls back to the `CARTESIA_API_KEY` environment variable when unset. | | `model` | `str` | `"ink-whisper"` | Cartesia speech-to-text model. | | `language` | `str` | `"en"` | Expected language of the caller's speech. | | `input_sample_rate` | `int` | `48000` | Sample rate (Hz) of the incoming PCM audio; must match the capture rate. | | `output_sample_rate` | `int` | `16000` | Sample rate (Hz) at which Cartesia processes audio internally. | | `base_url` | `str` | `None` | Override the Cartesia WebSocket base URL. Defaults to `None` (uses `wss://api.cartesia.ai`). | ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | ---------------------- | | Python | `from zeroruntime.plugins import CartesiaSTT` | `CartesiaSTT(...)` | | Node JS | `import { CartesiaSTT } from '@zeroruntime/js-sdk/plugins'` | `CartesiaSTT({ ... })` | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # Deepgram Source: https://docs.zeroruntime.ai/plugins/stt/deepgram Use the Deepgram speech-to-text plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Deepgram is a **speech-to-text** plugin. It transcribes the caller's audio into text for the LLM. ## Setup Set your Deepgram API key in the worker environment. Generate a key from the [Deepgram console](https://console.deepgram.com/): ```bash theme={null} export DEEPGRAM_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import DeepgramSTT stt = DeepgramSTT() # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { DeepgramSTT } from '@zeroruntime/js-sdk/plugins'; const stt = DeepgramSTT(); // Pipeline({ stt, ... }) ``` ## Parameters *Constructor parameters for `DeepgramSTT`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | -------------------- | ------------------ | ------------------------------------ | --------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Deepgram API key. Falls back to the `DEEPGRAM_API_KEY` environment variable when unset. | | `model` | `str` | `"nova-2"` | Deepgram speech-to-text model. | | `language` | `str` | `"en-US"` | Expected language (BCP-47 code) of the caller's speech. | | `interim_results` | `bool` | `True` | Emit partial transcripts while the caller is still speaking. | | `punctuate` | `bool` | `True` | Add punctuation to transcripts. | | `smart_format` | `bool` | `True` | Apply readable formatting to dates, numbers, and similar entities. | | `sample_rate` | `int` | `48000` | Sample rate (Hz) of the input audio. | | `endpointing` | `int` | `50` | Milliseconds of silence before an utterance is finalized. | | `filler_words` | `bool` | `True` | Keep filler words such as "uh" and "um" in transcripts. | | `keywords` | `list[str]` | `None` | Keywords to boost recognition for (legacy keyword boosting). | | `keyterm` | `list[str]` | `None` | Key terms to boost (key-term prompting). | | `profanity_filter` | `bool` | `False` | Mask profanity in transcripts. | | `numerals` | `bool` | `False` | Convert spoken numbers into digits. | | `tag` | `str \| list[str]` | `None` | Tags attached to the request for usage tracking. | | `enable_diarization` | `bool` | `False` | Label which speaker said each word. | | `base_url` | `str` | `"wss://api.deepgram.com/v1/listen"` | Deepgram streaming endpoint. | ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | --------------- | | Python | `from zeroruntime.plugins import DeepgramSTT` | `DeepgramSTT()` | | Node JS | `import { DeepgramSTT } from '@zeroruntime/js-sdk/plugins'` | `DeepgramSTT()` | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # Gladia Source: https://docs.zeroruntime.ai/plugins/stt/gladia Use the Gladia speech-to-text plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Gladia is a **speech-to-text** plugin. It transcribes the caller's audio into text for the LLM, with support for 90+ languages and automatic code-switching between them. ## Setup Set your Gladia API key in the worker environment. Generate a key from the [Gladia dashboard](https://app.gladia.io/signup): ```bash theme={null} export GLADIA_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import GladiaSTT stt = GladiaSTT() # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { GladiaSTT } from '@zeroruntime/js-sdk/plugins'; const stt = GladiaSTT(); // Pipeline({ stt, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ----------------------------- | ------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Gladia API key. Falls back to the `GLADIA_API_KEY` environment variable when unset. | | `model` | `str` | `"solaria-1"` | Gladia recognition model. `"solaria-1"` is the only production model and the default. | | `languages` | `List[str] \| None` | `None` | ISO 639-1 language codes to hint at (e.g. `["en", "fr"]`). Only the first element is used. When `None`, the default is `"english"`. | | `code_switching` | `bool` | `True` | When `True`, language is re-detected on every utterance. When `False`, language is detected once on the first utterance and held for the session. | | `input_sample_rate` | `int` | `48000` | Sample rate (Hz) of the incoming audio before any resampling. Valid values: `8000`, `16000`, `32000`, `44100`, `48000`. | | `output_sample_rate` | `int` | `16000` | Sample rate (Hz) forwarded to the Gladia WebSocket session. | | `encoding` | `str` | `"wav/pcm"` | PCM encoding format sent over the WebSocket. Valid values: `"wav/pcm"`, `"wav/alaw"`, `"wav/ulaw"`. | | `bit_depth` | `int` | `16` | Bit depth of the PCM samples. Valid values: `8`, `16`, `24`, `32`. | | `channels` | `int` | `1` | Number of audio channels (1-8). Default is mono. | | `receive_partial_transcripts` | `bool` | `False` | When `True`, partial (non-final) transcripts are emitted over the WebSocket before the utterance is complete. | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # Google Cloud STT Source: https://docs.zeroruntime.ai/plugins/stt/google Use the Google Cloud Speech-to-Text plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Google Cloud Speech-to-Text is a **speech-to-text** plugin. It transcribes the caller's audio into text for the LLM using Google Cloud's Speech-to-Text V2 streaming API. ## Setup Google Cloud STT authenticates with a Google Cloud service account rather than a simple API key. Point Application Default Credentials at your service-account JSON. Create the service account and key in the [Google Cloud console](https://console.cloud.google.com/apis/credentials): ```bash theme={null} export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json ``` In the Python SDK you can also pass the service-account JSON path (or raw JSON string) as the `api_key` argument. The value is used directly as the service-account credentials; it is not written to the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import GoogleSTT stt = GoogleSTT( languages="en-US", model="latest_long", ) # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { GoogleSTT } from '@zeroruntime/js-sdk/plugins'; const stt = GoogleSTT({ languages: 'en-US', model: 'latest_long', }); // Pipeline({ stt, ... }) ``` ## Parameters *Constructor parameters for `GoogleSTT`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ------------------------------ | ------------------ | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `credentials_json` | `str` | `None` | Service-account credentials as a raw JSON string or a path to a JSON key file. Takes highest precedence. | | `service_account_path` | `str` | `None` | Path to a service-account JSON key file. Used when `credentials_json` is not set. | | `api_key` | `str` | `None` | Fallback credential source; the value is treated as a JSON file path (or raw JSON) first. Used only when the above are absent. Service-account auth is recommended for STT. | | `project_id` | `str` | `None` | Google Cloud project ID. Extracted from the service-account JSON when not set. | | `languages` | `str \| list[str]` | `"en-US"` | Language code(s) to recognize. The first entry is the primary language. | | `model` | `str` | `"latest_long"` | Google Cloud Speech recognition model. | | `sample_rate` | `int` | `48000` | Target sample rate (Hz). Matches the native input rate so no resampling occurs; set a lower value (e.g. `16000`) to downsample. | | `interim_results` | `bool` | `True` | Emit partial transcripts while the caller is still speaking. | | `punctuate` | `bool` | `True` | Add automatic punctuation to transcripts. | | `min_confidence_threshold` | `float` | `0.0` | Drop results below this confidence. `0.0` disables filtering. | | `location` | `str` | `"us"` | Cloud Speech-to-Text regional endpoint: `"us"` or `"eu"`. | | `profanity_filter` | `bool` | `False` | Mask profane words in transcripts. | | `enable_voice_activity_events` | `bool` | `False` | Emit server-side speech-start/-end events independently of transcripts. | | `speech_start_timeout` | `float` | `None` | Seconds to wait for speech to begin before a voice-activity timeout fires. `None` uses the API default. | | `speech_end_timeout` | `float` | `None` | Seconds of silence after speech before an end-of-speech event fires. `None` uses the API default. | | `audio_channel_count` | `int` | `1` | Number of audio channels in the input (mono by default). | | `min_speaker_count` | `int` | `None` | Minimum expected speakers for diarization. `None` disables diarization. | | `max_speaker_count` | `int` | `None` | Maximum expected speakers for diarization. `None` disables diarization. | ## Import paths | SDK | Import | Constructor | | ------- | --------------------------------------------------------- | -------------------- | | Python | `from zeroruntime.plugins import GoogleSTT` | `GoogleSTT(...)` | | Node JS | `import { GoogleSTT } from '@zeroruntime/js-sdk/plugins'` | `GoogleSTT({ ... })` | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # NVIDIA Source: https://docs.zeroruntime.ai/plugins/stt/nvidia Use the NVIDIA Riva speech-to-text plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. NVIDIA Riva is a **speech-to-text** plugin that provides GPU-accelerated, low-latency speech recognition for the caller's audio. ## Setup Set your NVIDIA API key in the worker environment. Generate a key from the [NVIDIA build portal](https://build.nvidia.com/): ```bash theme={null} export NVIDIA_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import NvidiaSTT stt = NvidiaSTT() # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { NvidiaSTT } from '@zeroruntime/js-sdk/plugins'; const stt = NvidiaSTT(); // Pipeline({ stt, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ----------------------- | ------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | NVIDIA API key. Falls back to the `NVIDIA_API_KEY` environment variable when unset. | | `model` | `str` | `"parakeet-1.1b-en-US-asr-streaming-silero-vad-sortformer"` | Riva/NIM ASR model used for transcription (Parakeet 1.1B with Silero VAD + Sortformer diarization). | | `server` | `str` | `"grpc.nvcf.nvidia.com:443"` | Riva gRPC endpoint. | | `function_id` | `str` | `""` | NVCF function ID for the hosted model. | | `language_code` | `str` | `"en-US"` | BCP-47 language tag of the caller's speech. | | `sample_rate` | `int` | `16000` | Sample rate (Hz) of the input audio. | | `use_ssl` | `bool` | `True` | Use a TLS/SSL gRPC channel. | | `profanity_filter` | `bool` | `False` | Mask profanity in transcripts. | | `automatic_punctuation` | `bool` | `True` | Add punctuation to transcripts. | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # OpenAI Source: https://docs.zeroruntime.ai/plugins/stt/openai Use the OpenAI speech-to-text plugin in a Zero Runtime pipeline, including Azure OpenAI. Setup, options, and usage. OpenAI is a **speech-to-text** plugin. It transcribes the caller's audio into text for the LLM using OpenAI's transcription models. By default it streams over OpenAI's realtime transcription WebSocket; it can also run in non-streaming HTTP mode with built-in voice activity detection. ## Setup Set your OpenAI API key in the worker environment. Generate a key from the [OpenAI dashboard](https://platform.openai.com/api-keys): ```bash theme={null} export OPENAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import OpenAISTT stt = OpenAISTT( model="gpt-4o-transcribe", language="en", ) # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { OpenAISTT } from '@zeroruntime/js-sdk/plugins'; const stt = OpenAISTT({ model: 'gpt-4o-transcribe', language: 'en', }); // Pipeline({ stt, ... }) ``` ## Azure OpenAI To run OpenAI transcription through **Azure OpenAI**, use the separate `AzureOpenAISTT` plugin. It reads its configuration from Azure environment variables when arguments are omitted: ```bash theme={null} export AZURE_OPENAI_API_KEY= export AZURE_OPENAI_ENDPOINT=https://.openai.azure.com export AZURE_OPENAI_STT_DEPLOYMENT= ``` ```python Python theme={null} from zeroruntime.plugins import AzureOpenAISTT stt = AzureOpenAISTT( azure_endpoint="https://.openai.azure.com", deployment="", api_version="2025-03-01-preview", ) # Pipeline(stt=stt, ...) ``` `AzureOpenAISTT` ships in the Python SDK only. The Node JS SDK does not ship an Azure OpenAI provider. `api_key` falls back to `AZURE_API_KEY`, then `AZURE_OPENAI_API_KEY`. `deployment` falls back to `AZURE_OPENAI_STT_DEPLOYMENT`. `AzureOpenAISTT` runs non-streaming by default (`stream=False`). ## Parameters *Constructor parameters for `OpenAISTT`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ------------------------- | ------- | --------------------- | ---------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable when unset. | | `model` | `str` | `"gpt-4o-transcribe"` | Transcription model. `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, `gpt-4o-transcribe-diarize`, or `whisper-1`. | | `language` | `str` | `"en"` | Expected language of the caller's speech (ISO-639-1). | | `stream` | `bool` | `True` | Stream partial transcripts as the caller speaks. | | `input_sample_rate` | `int` | `48000` | Input audio sample rate in Hz. | | `output_sample_rate` | `int` | `24000` | Resampled rate sent to the model in Hz. | | `prompt` | `str` | `None` | Biasing prompt to guide transcription vocabulary/style. | | `turn_detection` | `str` | `"server_vad"` | VAD mode: `"server_vad"` or `"semantic_vad"`. | | `vad_threshold` | `float` | `None` | Speech-detection sensitivity, 0.0-1.0. | | `vad_prefix_padding_ms` | `int` | `None` | Audio kept before detected speech, in ms. | | `vad_silence_duration_ms` | `int` | `None` | Silence before end-of-turn, in ms. | | `noise_reduction` | `str` | `"near_field"` | Input denoising profile: `"near_field"`, `"far_field"`, or `None` to disable. | | `response_format` | `str` | `"json"` | Transcript format (`"json"`, `"text"`, ...). | | `base_url` | `str` | `None` | Override the OpenAI API base URL. | ### Azure OpenAI (`AzureOpenAISTT`) | Parameter | Type | Default | Description | | -------------------- | ------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Azure OpenAI API key. Falls back to `AZURE_API_KEY`, then `AZURE_OPENAI_API_KEY`. | | `azure_endpoint` | `str` | `None` | Azure OpenAI endpoint. Falls back to `AZURE_OPENAI_ENDPOINT`. | | `deployment` | `str` | `None` | Deployment name. Falls back to `AZURE_OPENAI_STT_DEPLOYMENT`. | | `api_version` | `str` | `"2025-03-01-preview"` | Azure OpenAI REST API version. | | `model` | `str` | `""` | Underlying model name passed in the request body (e.g. `"whisper-1"`). Usually left empty when `deployment` is set. | | `language` | `str` | `"en"` | ISO-639-1 language code of the spoken audio. | | `stream` | `bool` | `False` | When `True`, receive transcription results as a streaming response. | | `input_sample_rate` | `int` | `48000` | Input audio sample rate in Hz. | | `output_sample_rate` | `int` | `24000` | Resampled rate sent to the Azure endpoint in Hz. | | `prompt` | `str` | `None` | Optional text to guide transcription style or continue prior context. | | `temperature` | `float` | `None` | Sampling temperature in `[0.0, 1.0]`. | | `response_format` | `str` | `"json"` | Response format: `"json"`, `"verbose_json"`, `"text"`, `"srt"`, or `"vtt"`. | | `turn_detection` | `str` | `"server_vad"` | Turn-detection strategy for streaming: `"server_vad"` or `"none"`. | ## Import paths | SDK | Import | Constructor | | ------ | ----------------------------------------------------------- | ---------------------------------------- | | Python | `from zeroruntime.plugins import OpenAISTT, AzureOpenAISTT` | `OpenAISTT(...)` / `AzureOpenAISTT(...)` | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # Sarvam AI Source: https://docs.zeroruntime.ai/plugins/stt/sarvamai Use the Sarvam AI speech-to-text plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Sarvam AI is a **speech-to-text** plugin tuned for Indian languages. It transcribes the caller's audio into text over Sarvam's streaming WebSocket API, with optional speech-to-text translation. ## Setup Set your Sarvam AI API key in the worker environment. Generate a key from the [Sarvam AI dashboard](https://dashboard.sarvam.ai/key-management): ```bash theme={null} export SARVAMAI_API_KEY= ``` The Python SDK requires `scipy` for audio resampling (`pip install scipy`). ## Usage Import the plugin and pass it to the pipeline's `stt` slot. ```python Python theme={null} from zeroruntime.plugins import SarvamAISTT stt = SarvamAISTT( model="saaras:v3", language="en-IN", ) # Pipeline(stt=stt, ...) ``` ```typescript Node JS theme={null} import { SarvamAISTT } from '@zeroruntime/js-sdk/plugins'; const stt = SarvamAISTT({ model: 'saaras:v3', language: 'en-IN', }); // Pipeline({ stt, ... }) ``` ## Parameters *Constructor parameters for `SarvamAISTT`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ---------------------- | ------ | ------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Sarvam AI API key. Falls back to the `SARVAM_API_KEY` environment variable when unset. | | `model` | `str` | `"saaras:v3"` | Sarvam speech-to-text model. | | `language` | `str` | `"en-IN"` | Expected language (BCP-47 code) of the caller's speech. | | `input_sample_rate` | `int` | `48000` | Sample rate (Hz) of the incoming audio. | | `output_sample_rate` | `int` | `16000` | Sample rate (Hz) the audio is resampled to before it is sent. | | `mode` | `str` | `None` | Recognition mode (`"transcribe"`, `"translate"`, `"verbatim"`, `"translit"`, `"codemix"`). Applies only to the `saaras:v3` model. | | `high_vad_sensitivity` | `bool` | `None` | Use higher voice-activity detection sensitivity. | | `flush_signals` | `bool` | `None` | Emit flush signals to force transcript boundaries. | | `translation` | `bool` | `False` | Translate the audio to the target language instead of transcribing verbatim. | | `prompt` | `str` | `None` | Biasing prompt to guide recognition. Applies only when `translation` is enabled. | ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | ---------------------- | | Python | `from zeroruntime.plugins import SarvamAISTT` | `SarvamAISTT(...)` | | Node JS | `import { SarvamAISTT } from '@zeroruntime/js-sdk/plugins'` | `SarvamAISTT({ ... })` | Transcribed text passes to the [LLM](/plugins/llm/google) once [turn detection](/plugins/turn-detection/namo) decides the caller has finished speaking. # AWS Polly Source: https://docs.zeroruntime.ai/plugins/tts/aws Use the AWS Polly text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. AWS Polly is a **text-to-speech** plugin. It synthesizes the agent's text responses into natural-sounding speech using Amazon Polly's neural, standard, generative, and long-form voice engines. ## Setup Set your AWS access key in the worker environment. Get your access keys from the [AWS console](https://console.aws.amazon.com/): ```bash theme={null} export AWS_ACCESS_KEY_ID= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import AWSPollyTTS tts = AWSPollyTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { AWSPollyTTS } from '@zeroruntime/js-sdk/plugins'; const tts = AWSPollyTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ----------------------- | ------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `aws_access_key_id` | `str \| None` | `None` | AWS access key ID. Falls back to the `AWS_ACCESS_KEY_ID` environment variable when unset. | | `aws_secret_access_key` | `str \| None` | `None` | AWS secret access key paired with `aws_access_key_id`. Falls back to the `AWS_SECRET_ACCESS_KEY` environment variable when unset. | | `aws_session_token` | `str \| None` | `None` | Temporary session token for short-lived IAM role credentials (e.g. from `sts:AssumeRole`). Falls back to the `AWS_SESSION_TOKEN` environment variable when unset. Omit for long-term credentials. | | `region` | `str` | `"us-east-1"` | AWS region hosting the Polly endpoint, e.g. `"us-east-1"`, `"eu-west-1"`, `"ap-southeast-1"`. | | `voice` | `str` | `"Joanna"` | Polly voice ID to use for synthesis (US English, female, neural). Other popular options include `"Matthew"` (US English, male), `"Amy"` (British English, female), `"Brian"` (British English, male), and `"Celine"` (French, female). See the [AWS Polly voice list](https://docs.aws.amazon.com/polly/latest/dg/voicelist.html) for the full catalogue. | | `engine` | `str` | `"neural"` | Polly synthesis engine. One of `"neural"` (high-quality neural TTS; supports a subset of voices), `"standard"` (concatenative synthesis; broadest voice support), `"generative"` (expressive speech; select voices only), or `"long-form"` (optimized for longer text passages; select voices only). | | `speed` | `float` | `1.0` | Playback rate multiplier (`1.0` is normal speed). | | `pitch` | `float` | `0.0` | Pitch adjustment in semitones (`0.0` is normal). | Synthesized audio is streamed back to the caller after the [LLM](/plugins/overview) produces a response. # Azure Source: https://docs.zeroruntime.ai/plugins/tts/azure Use the Azure text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Azure is a **text-to-speech** plugin. It synthesizes the LLM's text response into natural-sounding speech using Azure Neural TTS voices. ## Setup Set your Azure Speech resource key in the worker environment. Create a Speech resource and key in the [Azure portal](https://portal.azure.com/): ```bash theme={null} export AZURE_SPEECH_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import AzureTTS tts = AzureTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { AzureTTS } from '@zeroruntime/js-sdk/plugins'; const tts = AzureTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | --------------- | --------------- | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `speech_key` | `Optional[str]` | `None` | Azure Speech resource subscription key. Falls back to the `AZURE_SPEECH_KEY` environment variable when unset. | | `speech_region` | `Optional[str]` | `None` | Azure region identifier for the Speech resource (e.g. `"eastus"`, `"westeurope"`, `"southeastasia"`). Falls back to the `AZURE_REGION` environment variable, then `"eastus"` if that is also unset. Must match the region where your Azure Speech resource was created, since keys are region-scoped. | | `voice` | `str` | `"en-US-JennyNeural"` | Azure Neural TTS voice name in `"-Neural"` format. Common options include `"en-US-AriaNeural"`, `"en-US-GuyNeural"`, `"en-GB-SoniaNeural"`, `"fr-FR-DeniseNeural"`, `"de-DE-KatjaNeural"`, `"ja-JP-NanamiNeural"`. | | `sample_rate` | `int` | `24000` | PCM output sample rate in Hz. Azure Neural TTS natively supports `24000` (standard) and `48000` (high-fidelity). | The synthesized audio is streamed back to the caller as the [LLM](/plugins/llm/google) produces its response. # Camb AI Source: https://docs.zeroruntime.ai/plugins/tts/cambai Use the Camb AI text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Camb AI is a **text-to-speech** plugin. It synthesizes the LLM's reply into natural-sounding speech via the CambAI MARS TTS API. ## Setup Set your Camb AI API key in the worker environment. Generate a key from the [Camb AI studio](https://studio.camb.ai/): ```bash theme={null} export CAMBAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import CambAITTS tts = CambAITTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { CambAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = CambAITTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | | `api_key` | `str \| None` | `None` | CambAI API key. Falls back to the `CAMBAI_API_KEY` environment variable when unset. | | `voice` | `int` | `147320` | Integer voice identifier from the CambAI voice catalog. Numeric IDs can be retrieved from the CambAI dashboard or the voices API endpoint. | | `model` | `str` | `"mars-pro"` | CambAI TTS model name. `"mars-pro"` is the primary MARS 8 high-quality model. | | `sample_rate` | `int` | `24000` | PCM output sample rate in Hz. | Synthesized audio is streamed back to the caller after the [LLM](/plugins/overview) produces a response. # Cartesia Source: https://docs.zeroruntime.ai/plugins/tts/cartesia Use the Cartesia text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Cartesia is a **text-to-speech** plugin. It turns the LLM's reply into audio the caller hears. ## Setup Set your Cartesia API key in the worker environment. Generate a key from the [Cartesia dashboard](https://play.cartesia.ai/keys): ```bash theme={null} export CARTESIA_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import CartesiaTTS tts = CartesiaTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { CartesiaTTS } from '@zeroruntime/js-sdk/plugins'; const tts = CartesiaTTS(); // Pipeline({ tts, ... }) ``` ## Parameters *Constructor parameters for `CartesiaTTS`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ------------------------ | -------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Cartesia API key. Falls back to the `CARTESIA_API_KEY` environment variable when unset. | | `model` | `str` | `"sonic-2"` | Cartesia voice model. | | `voice_id` | `str \| list[float]` | `"f8f5f1b2-f02d-4d8e-a40d-fd850a487b3d"` | Voice to speak with: a voice ID or an embedding vector. | | `language` | `str` | `"en"` | Language of the spoken output. | | `generation_config` | `GenerationConfig` | `None` | Fine-grained synthesis settings such as speed, volume, and emotion. | | `pronunciation_dict_id` | `str` | `None` | ID of a custom pronunciation dictionary to apply. | | `max_buffer_delay_ms` | `int` | `None` | Maximum time to buffer text before synthesizing, trading latency for smoother audio. | | `enable_word_timestamps` | `bool` | `False` | Return per-word timing alongside the audio. | `generation_config` can be passed as a dictionary with Cartesia's supported voice-shaping fields: * `speed`: `0.6` to `1.5` (`1.0` = default speed) * `volume`: `0.5` to `2.0` (`1.0` = default volume) * `emotion`: one of `"neutral"`, `"calm"`, `"angry"`, `"content"`, or `"sad"` ```python theme={null} from zeroruntime.plugins import CartesiaTTS tts = CartesiaTTS( generation_config={ "speed": 0.9, "volume": 1.2, "emotion": "calm", } ) ``` ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | --------------- | | Python | `from zeroruntime.plugins import CartesiaTTS` | `CartesiaTTS()` | | Node JS | `import { CartesiaTTS } from '@zeroruntime/js-sdk/plugins'` | `CartesiaTTS()` | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. # Deepgram Source: https://docs.zeroruntime.ai/plugins/tts/deepgram Use the Deepgram text-to-speech plugin in a Zero Runtime pipeline. Setup, options, and usage. Deepgram is a **text-to-speech** plugin. It synthesizes the LLM's reply into the agent's voice over a streaming WebSocket using Deepgram's Aura voices. It occupies the pipeline's `tts` slot. Deepgram is also available as a [speech-to-text](/plugins/stt/deepgram) plugin. ## Setup Set your Deepgram API key in the worker environment. Generate a key from the [Deepgram console](https://console.deepgram.com/): ```bash theme={null} export DEEPGRAM_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import DeepgramTTS tts = DeepgramTTS( model="aura-2-andromeda-en", ) # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { DeepgramTTS } from '@zeroruntime/js-sdk/plugins'; const tts = DeepgramTTS({ model: 'aura-2-andromeda-en', }); // Pipeline({ tts, ... }) ``` ## Parameters *Constructor parameters for `DeepgramTTS`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ------------- | ----- | ----------------------------------- | --------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Deepgram API key. Falls back to the `DEEPGRAM_API_KEY` environment variable when unset. | | `model` | `str` | `"aura-2-andromeda-en"` | Deepgram Aura voice/model ID. | | `encoding` | `str` | `"linear16"` | Output audio encoding. | | `sample_rate` | `int` | `24000` | Output sample rate (Hz). | | `base_url` | `str` | `"wss://api.deepgram.com/v1/speak"` | Deepgram streaming endpoint. | ## Import paths | SDK | Import | Constructor | | ------- | ----------------------------------------------------------- | ---------------------- | | Python | `from zeroruntime.plugins import DeepgramTTS` | `DeepgramTTS(...)` | | Node JS | `import { DeepgramTTS } from '@zeroruntime/js-sdk/plugins'` | `DeepgramTTS({ ... })` | The synthesized audio is streamed back to the caller as the final stage of the [pipeline](/concepts/pipeline). # ElevenLabs Source: https://docs.zeroruntime.ai/plugins/tts/elevenlabs Use the ElevenLabs text-to-speech plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. ElevenLabs is a **text-to-speech** plugin. It synthesizes the LLM's reply into the agent's voice, streaming low-latency audio over a WebSocket. It occupies the pipeline's `tts` slot. ## Setup Set your ElevenLabs API key in the worker environment. Generate a key from the [ElevenLabs dashboard](https://elevenlabs.io/app/settings/api-keys): ```bash theme={null} export ELEVENLABS_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import ElevenLabsTTS tts = ElevenLabsTTS( model="eleven_turbo_v2", voice="21m00Tcm4TlvDq8ikWAM", ) # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { ElevenLabsTTS } from '@zeroruntime/js-sdk/plugins'; const tts = ElevenLabsTTS({ model: 'eleven_turbo_v2', voice: '21m00Tcm4TlvDq8ikWAM', }); // Pipeline({ tts, ... }) ``` Both SDKs default to model `eleven_turbo_v2` and voice `21m00Tcm4TlvDq8ikWAM` (the "Rachel" voice). Pass `model` and `voice` explicitly for consistent results across SDKs. ## Voice settings Fine-tune the voice with the `voice_settings` mapping, which mirrors ElevenLabs' native `voice_settings` shape: `stability`, `similarity_boost`, `style` and `use_speaker_boost`. ```python Python theme={null} from zeroruntime.plugins import ElevenLabsTTS tts = ElevenLabsTTS( voice_settings={ "stability": 0.5, "similarity_boost": 0.75, "style": 0.0, "use_speaker_boost": True, }, ) ``` ```typescript Node JS theme={null} import { ElevenLabsTTS } from '@zeroruntime/js-sdk/plugins'; const tts = ElevenLabsTTS({ voice_settings: { stability: 0.5, similarity_boost: 0.75, style: 0.0, use_speaker_boost: true, }, }); ``` ## Parameters *Constructor parameters for `ElevenLabsTTS`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | -------------------------- | ------- | ------------------------ | ----------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | ElevenLabs API key. Falls back to the `ELEVENLABS_API_KEY` environment variable when unset. | | `voice` | `str` | `"21m00Tcm4TlvDq8ikWAM"` | Voice ID (the "Rachel" voice). | | `model` | `str` | `"eleven_turbo_v2"` | ElevenLabs synthesis model (`eleven_v3`, `eleven_multilingual_v2`, `eleven_flash_v2_5`, `eleven_flash_v2`). | | `sample_rate` | `int` | `24000` | Output sample rate in Hz. | | `stability` | `float` | `0.5` | Voice consistency, 0.0-1.0 (higher is steadier). | | `similarity_boost` | `float` | `0.75` | Adherence to the original voice, 0.0-1.0. | | `style` | `float` | `0.0` | Style exaggeration, 0.0-1.0 (0 disables, adds latency). | | `use_speaker_boost` | `bool` | `True` | Boost similarity to the speaker. | | `apply_text_normalization` | `str` | `None` | Text normalization mode: `"auto"`, `"on"`, or `"off"`. | | `enable_word_timestamps` | `bool` | `False` | Return per-word timing metadata. | | `speed` | `float` | `None` | Speaking-rate multiplier (\~0.7-1.2). Provider default when unset. | | `language` | `str` | `None` | ISO-639-1 language hint for supported models. | | `enable_ssml_parsing` | `bool` | `None` | Interpret SSML tags in the input text. | | `stream` | `bool` | `True` | Stream audio as it is synthesized. | ## Import paths | SDK | Import | Constructor | | ------- | ------------------------------------------------------------- | ------------------------ | | Python | `from zeroruntime.plugins import ElevenLabsTTS` | `ElevenLabsTTS(...)` | | Node JS | `import { ElevenLabsTTS } from '@zeroruntime/js-sdk/plugins'` | `ElevenLabsTTS({ ... })` | The synthesized audio is streamed back to the caller as the final stage of the [pipeline](/concepts/pipeline). # Google Cloud TTS Source: https://docs.zeroruntime.ai/plugins/tts/google Use the Google Cloud Text-to-Speech plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Google Cloud Text-to-Speech is a **text-to-speech** plugin. It synthesizes the LLM's reply into the agent's voice. It occupies the pipeline's `tts` slot. ## Setup Cloud TTS authenticates with either a service account or an API key. Create one in the [Google Cloud console](https://console.cloud.google.com/apis/credentials) and export it in the worker environment: ```bash API key theme={null} export GOOGLE_API_KEY= ``` ```bash Service account theme={null} export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json ``` Streaming synthesis needs a service account. Cloud TTS refuses an API key on `StreamingSynthesize`, and `streaming` defaults to `True` — so an API key alone works only with `streaming=False`. The plugin also prefers `GOOGLE_API_KEY` whenever it is set: leave it unset when you mean to authenticate with the service account. ### Credential resolution The plugin chooses one credential when it is constructed, and takes the first that is present: 1. `GOOGLE_API_KEY` when it is unset it will use `GOOGLE_APPLICATION_CREDENTIALS`. 2. `GOOGLE_APPLICATION_CREDENTIALS`, when it points at a readable file — used as service-account credentials. The fallback is on **absence**, not on failure. Step 2 is reached only when no API key is set at all — so if `GOOGLE_API_KEY` is present and Google rejects it, the plugin does not then try the service-account file, even when both are configured. Unset `GOOGLE_API_KEY` to make the service account the credential in use. ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import GoogleTTS tts = GoogleTTS( voice_config={"name": "en-US-Chirp3-HD-Charon", "languageCode": "en-US"}, speed=1.0, ) # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { GoogleTTS } from '@zeroruntime/js-sdk/plugins'; const tts = GoogleTTS({ voice_config: {name: 'en-US-Chirp3-HD-Charon', languageCode: 'en-US'}, speed: 1.0, }); // Pipeline({ tts, ... }) ``` Everything about the voice lives in `voice_config` — there is no separate `voice` or `language_code` argument. Omit it entirely and the plugin uses `en-US-Chirp3-HD-Charon` / `en-US` / `MALE`. | Key | Type | Default | Description | | -------------- | ----- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | `str` | `"en-US-Chirp3-HD-Charon"` | Cloud TTS voice name, or a bare Gemini voice (e.g. `"Kore"`) when `model` is set. | | `languageCode` | `str` | `"en-US"` | BCP-47 language code for the voice. | | `ssmlGender` | `str` | `"MALE"` | `"MALE"`, `"FEMALE"`, or `"NEUTRAL"`. Sent only for non-Studio Cloud TTS voices — it is omitted for Studio voices and whenever `model` is set. | ## Streaming `streaming=True` (the default) synthesizes over gRPC `StreamingSynthesize`, which starts returning audio while the LLM is still producing text. Without a Gemini-TTS `model`, that path only accepts **Chirp 3 HD** voices; any other voice raises `ValueError` from the constructor. For a Neural2, Studio, WaveNet, or Standard voice, turn streaming off and the plugin falls back to per-segment `SynthesizeSpeech` requests: ```python Python theme={null} from zeroruntime.plugins import GoogleTTS tts = GoogleTTS( voice_config={"name": "en-US-Neural2-F", "languageCode": "en-US"}, streaming=False, ) ``` ```typescript Node JS theme={null} import { GoogleTTS } from '@zeroruntime/js-sdk/plugins'; const tts = GoogleTTS({ voice_config: {name: 'en-US-Neural2-F', languageCode: 'en-US'}, streaming: false, }); ``` `pitch` is only sent on the non-streaming path — Cloud TTS's streaming audio config carries `speaking_rate` but has no pitch field, so `pitch` is silently inert when `streaming=True`. `streaming=True` and `vertexai=True` cannot be combined; the constructor raises `ValueError`. ## Gemini-TTS Set `model` to a Gemini-TTS engine to synthesize with Gemini instead of standard Cloud TTS. The voice name becomes a bare Gemini voice, and `prompt` takes a natural-language style instruction: ```python Python theme={null} from zeroruntime.plugins import GoogleTTS tts = GoogleTTS( model="gemini-3.1-flash-tts-preview", voice_config={"name": "Kore", "languageCode": "en-US"}, prompt="Speak in a warm, professional tone", ) ``` ```typescript Node JS theme={null} import { GoogleTTS } from '@zeroruntime/js-sdk/plugins'; const tts = GoogleTTS({ model: 'gemini-3.1-flash-tts-preview', voice_config: {name: 'Kore', languageCode: 'en-US'}, prompt: 'Speak in a warm, professional tone', }); ``` Known engines are `"gemini-3.1-flash-tts-preview"`, `"gemini-2.5-flash-tts"`, `"gemini-2.5-flash-lite-preview-tts"`, and `"gemini-2.5-pro-tts"`. A Gemini model lifts the Chirp 3 HD restriction on streaming, and `prompt` is only valid alongside one — passing `prompt` without `model` raises `ValueError`. ## Vertex AI `vertexai=True` routes synthesis through the regional Vertex AI endpoint (`{location}-texttospeech.googleapis.com`) using Application Default Credentials rather than an API key. It requires `streaming=False`. ```python Python theme={null} from zeroruntime.plugins import GoogleTTS tts = GoogleTTS( vertexai=True, vertexai_config={"project_id": "my-project", "location": "us-central1"}, streaming=False, ) ``` ```typescript Node JS theme={null} import { GoogleTTS } from '@zeroruntime/js-sdk/plugins'; const tts = GoogleTTS({ vertexai: true, vertexai_config: {project_id: 'my-project', location: 'us-central1'}, streaming: false, }); ``` | Key | Type | Default | Description | | ------------ | ------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `project_id` | `str \| None` | `None` | GCP project. Falls back to `GOOGLE_CLOUD_PROJECT`, then `GCLOUD_PROJECT`, then the `project_id` inside the `GOOGLE_APPLICATION_CREDENTIALS` service-account file. `ValueError` when none of them resolve. | | `location` | `str` | `"us-central1"` | Vertex AI region, which also determines the endpoint host. `GOOGLE_CLOUD_LOCATION` is consulted only if this is explicitly emptied — the default is already a value. | ## Custom pronunciations `custom_pronunciations` overrides how specific phrases are read. The short form is a mapping of phrase to IPA: ```python theme={null} tts = GoogleTTS(custom_pronunciations={"tomato": "təˈmeɪtoʊ"}) ``` The long form is a list, and lets each entry pick its phonetic encoding — `"ipa"` (default) or `"x-sampa"`. An unrecognized encoding logs a warning and falls back to IPA: ```python theme={null} tts = GoogleTTS( custom_pronunciations=[ {"phrase": "tomato", "pronunciation": "təˈmeɪtoʊ", "encoding": "ipa"}, {"phrase": "Nginx", "pronunciation": "ˈɛndʒɪnˈɛks"}, ] ) ``` Cloud TTS only applies custom pronunciations to `en-US`. With any other `languageCode` the plugin logs a warning and the overrides are ignored. They apply on both the streaming and non-streaming paths. ## Parameters *Constructor parameters for `GoogleTTS`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ----------------------- | ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `voice_config` | `dict \| None` | `None` | Voice selection — `name`, `languageCode`, `ssmlGender`. Unset means `en-US-Chirp3-HD-Charon` / `en-US` / `MALE`. | | `model` | `str \| None` | `None` | Gemini-TTS engine, e.g. `"gemini-3.1-flash-tts-preview"`. Unset uses standard Cloud TTS, where the voice name determines the family. | | `prompt` | `str \| None` | `None` | Natural-language style instruction. Gemini-TTS only — `ValueError` without `model`. | | `streaming` | `bool` | `True` | Use gRPC `StreamingSynthesize`. Restricted to Chirp 3 HD voices unless `model` is set, and cannot be combined with `vertexai`. | | `speed` | `float` | `1.0` | Speaking-rate multiplier, passed through as `speaking_rate`. Cloud TTS accepts `0.25`–`4.0`. | | `pitch` | `float` | `0.0` | Pitch shift in semitones, `-20.0`–`20.0`. Applied only when `streaming=False`. | | `response_format` | `Literal["pcm"]` | `"pcm"` | Output encoding. `"pcm"` is the only accepted value. | | `custom_pronunciations` | `list[dict] \| dict \| None` | `None` | IPA or X-SAMPA pronunciation overrides. `en-US` only. | | `vertexai` | `bool` | `False` | Route through the regional Vertex AI endpoint with ADC instead of the global API. Requires `streaming=False`. | | `vertexai_config` | `dict \| None` | `None` | `project_id` and `location` for Vertex AI. | Output audio is fixed at 24 kHz, mono, 16-bit PCM — there is no `sample_rate` argument. ## Import paths | SDK | Import | Constructor | | ------- | --------------------------------------------------------- | -------------------- | | Python | `from zeroruntime.plugins import GoogleTTS` | `GoogleTTS(...)` | | Node JS | `import { GoogleTTS } from '@zeroruntime/js-sdk/plugins'` | `GoogleTTS({ ... })` | The same plugin is also reachable without a Google credential of your own, billed against your Zero Runtime token, as `from zeroruntime.inference import GoogleTTS` — see [Zero Runtime inference](/plugins/inference/zero-runtime). The synthesized audio is streamed back to the caller as the final stage of the [pipeline](/concepts/pipeline). # Groq Source: https://docs.zeroruntime.ai/plugins/tts/groq Use the Groq text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Groq is a **text-to-speech** plugin. It synthesizes the LLM's reply into natural-sounding speech via Groq's low-latency inference API. ## Setup Set your Groq API key in the worker environment. Generate a key from the [Groq console](https://console.groq.com/keys): ```bash theme={null} export GROQ_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import GroqTTS tts = GroqTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { GroqTTS } from '@zeroruntime/js-sdk/plugins'; const tts = GroqTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ----------------- | ------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Groq API key. Falls back to the `GROQ_API_KEY` environment variable when unset. | | `model` | `str` | `"playai-tts"` | TTS model identifier. Options include `"playai-tts"` (PlayAI Dialog, English), `"playai-tts-arabic"` (PlayAI Dialog, Arabic), `"canopylabs/orpheus-v1-english"` (Orpheus English with vocal direction support), and `"canopylabs/orpheus-arabic-saudi"` (Orpheus Arabic, Saudi dialect). | | `voice` | `str` | `"Fritz-PlayAI"` | Voice identifier used for synthesis. For `"playai-tts"`, example voices include `"Fritz-PlayAI"`, `"Aaliyah-PlayAI"`, and `"Adelaide-PlayAI"`. For the Orpheus English model, available voices include `"troy"`, `"hannah"`, `"austin"`, `"diana"`, `"autumn"`, and `"daniel"`. | | `speed` | `float` | `1.0` | Speaking speed multiplier. | | `response_format` | `str` | `"wav"` | Audio container format for the returned audio. | | `sample_rate` | `int` | `24000` | Output audio sample rate in Hz. | Synthesized audio is streamed back to the caller after the [LLM](/plugins/overview) produces a response. # Hume AI Source: https://docs.zeroruntime.ai/plugins/tts/humeai Use the Hume AI text-to-speech plugin in a Zero Runtime pipeline. Hume AI is a **text-to-speech** plugin. It turns the LLM's reply into expressive, emotion-aware audio the caller hears. ## Setup Set your Hume AI API key in the worker environment. Generate a key from the [Hume AI dashboard](https://platform.hume.ai/settings/keys): ```bash theme={null} export HUMEAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import HumeAITTS tts = HumeAITTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { HumeAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = HumeAITTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Hume AI API key. Falls back to the `HUMEAI_API_KEY` environment variable when unset. | | `voice` | `str` | `"Serene Assistant"` | Voice name or UUID. Can be a name from Hume's built-in voice library (e.g. `"Serene Assistant"`, `"Male English Actor"`), the name of a custom saved voice, or a UUID string. | | `speed` | `float` | `1.0` | Speaking speed multiplier on a non-linear scale from `0.5` (much slower) to `2.0` (much faster), where `1.0` is normal pace. | | `sample_rate` | `int` | `24000` | Advertised output sample rate in Hz. Hume AI's API always returns audio at 48 kHz regardless of this value. | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. See the [plugins overview](/plugins/overview) for how TTS fits into the rest of the pipeline. # Inworld AI Source: https://docs.zeroruntime.ai/plugins/tts/inworldai Use the Inworld AI text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Inworld AI is a **text-to-speech** plugin. It synthesizes the LLM's reply into natural-sounding speech via Inworld AI's realtime, low-latency voice API. ## Setup Set your Inworld AI API key in the worker environment. Generate a key from the [Inworld AI studio](https://studio.inworld.ai/login): ```bash theme={null} export INWORLDAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import InworldAITTS tts = InworldAITTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { InworldAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = InworldAITTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Inworld AI API key. Falls back to the `INWORLDAI_API_KEY` environment variable when unset. | | `voice_id` | `str` | `"Hades"` | Voice identifier for synthesis. Use any voice ID from the Inworld AI voice library (e.g. `"Hades"`, `"Sarah"`), or a custom voice UUID obtained from the voice cloning endpoint. Voice IDs are consistent across the TTS API and the Inworld Playground. | | `model_id` | `str` | `"inworld-tts-1"` | TTS model to use. Options include `"inworld-tts-1.5-max"` (optimized for quality) and `"inworld-tts-1.5-mini"` (\~120 ms median latency, optimized for speed). | | `sample_rate` | `int` | `24000` | Output audio sample rate in Hz. Supports values in the range 8000-48000 Hz. | Synthesized audio is streamed back to the caller after the [LLM](/plugins/overview) produces a response. # LMNT Source: https://docs.zeroruntime.ai/plugins/tts/lmnt Use the LMNT text-to-speech plugin in a Zero Runtime pipeline. LMNT is a **text-to-speech** plugin. It turns the LLM's reply into audio the caller hears. ## Setup Set your LMNT API key in the worker environment. Generate a key from the [LMNT dashboard](https://app.lmnt.com/account): ```bash theme={null} export LMNT_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import LMNTTTS tts = LMNTTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { LMNTTTS } from '@zeroruntime/js-sdk/plugins'; const tts = LMNTTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | LMNT API key. Falls back to the `LMNT_API_KEY` environment variable when unset. | | `voice` | `str` | `"ava"` | Voice ID to use for synthesis. LMNT provides a library of pre-built voices (e.g. `"ava"`) as well as cloned voices identified by their UUID. | | `model` | `str` | `"blizzard"` | TTS model identifier. `"blizzard"` is the production flagship (Blizzard 2.0); `"aurora"` is an alias that routes to Blizzard. | | `sample_rate` | `int` | `24000` | Desired output sample rate in Hz. | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. See the [plugins overview](/plugins/overview) for how TTS fits into the rest of the pipeline. # Murf AI Source: https://docs.zeroruntime.ai/plugins/tts/murfai Use the Murf AI text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Murf AI is a **text-to-speech** plugin. It turns the LLM's reply into audio the caller hears, using Murf AI's real-time `Falcon` model or studio-grade `Gen2` model across 150+ voices and 35+ languages. ## Setup Set your Murf AI API key in the worker environment. Generate a key from the [Murf AI site](https://murf.ai/): ```bash theme={null} export MURFAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import MurfAITTS tts = MurfAITTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { MurfAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = MurfAITTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------------- | ----------------- | -------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Murf API key. Falls back to the `MURFAI_API_KEY` environment variable when unset. | | `voice` | `str` | `"en-US-natalie"` | Murf `voiceId` (e.g. `"en-US-natalie"`) or the voice actor name (e.g. `"natalie"`). | | `model` | `str` | `"Falcon"` | Synthesis model - `"Falcon"` (low-latency, real-time conversational) or `"Gen2"` (most realistic). | | `sample_rate` | `int` | `24000` | Output audio sample rate, in Hz. | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. See the [plugin overview](/plugins/overview) for how TTS fits into the rest of the pipeline. # Neuphonic Source: https://docs.zeroruntime.ai/plugins/tts/neuphonic Use the Neuphonic text-to-speech plugin in a Zero Runtime pipeline. Neuphonic is a **text-to-speech** plugin. It turns the LLM's reply into audio the caller hears. ## Setup Set your Neuphonic API key in the worker environment. Generate a key from the [Neuphonic dashboard](https://app.neuphonic.com/apikey): ```bash theme={null} export NEUPHONIC_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import NeuphonicTTS tts = NeuphonicTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { NeuphonicTTS } from '@zeroruntime/js-sdk/plugins'; const tts = NeuphonicTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | --------------- | ------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `api_key` | `str \| None` | `None` | Neuphonic API key. Falls back to the `NEUPHONIC_API_KEY` environment variable when unset. | | `voice_id` | `str \| None` | `None` | Voice identifier (a UUID string from your Neuphonic voice library). The chosen voice also determines which underlying model is used. Defaults to letting Neuphonic select a default voice. | | `lang_code` | `str` | `"en"` | Language code for synthesis. Other supported codes include `"es"`, `"pt"`, `"fr"`, `"de"`, `"zh"`, `"ja"`, `"ko"`, and `"ur"`. | | `sampling_rate` | `int` | `22050` | Output audio sampling rate in Hz. Valid values: `8000`, `16000`, `22050`, `24000`. | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. See the [plugins overview](/plugins/overview) for how the `tts` slot fits into the rest of the pipeline. # NVIDIA Source: https://docs.zeroruntime.ai/plugins/tts/nvidia Use the NVIDIA Riva text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. NVIDIA is a **text-to-speech** plugin. It converts the agent's text responses into natural-sounding audio using NVIDIA's GPU-accelerated Riva voices. ## Setup Set your NVIDIA API key in the worker environment. Generate a key from the [NVIDIA build portal](https://build.nvidia.com/): ```bash theme={null} export NVIDIA_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import NvidiaTTS tts = NvidiaTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { NvidiaTTS } from '@zeroruntime/js-sdk/plugins'; const tts = NvidiaTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | --------------- | ------------- | ----------------------- | ------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | NVIDIA API key. Falls back to the `NVIDIA_API_KEY` environment variable when unset. | | `voice` | `str` | `"English-US-Female-1"` | Riva voice name, e.g. `"English-US-Female-1"` or a Magpie voice like `"Magpie-Multilingual.EN-US.Mia"`. | | `language_code` | `str` | `"en-US"` | BCP-47 language tag for synthesis. | | `sample_rate` | `int` | `22050` | Output audio sample rate in Hz. | | `server` | `str` | `""` | Riva gRPC endpoint. | The synthesized audio is streamed back to the caller as soon as it's generated, after the [LLM](/plugins/llm/google) produces its response. # OpenAI Source: https://docs.zeroruntime.ai/plugins/tts/openai Use the OpenAI text-to-speech plugin in a Zero Runtime pipeline. Setup, options, and usage. OpenAI is a **text-to-speech** plugin. It synthesizes the LLM's reply into the agent's voice using OpenAI's TTS models. It occupies the pipeline's `tts` slot. ## Setup Set your OpenAI API key in the worker environment. Generate a key from the [OpenAI dashboard](https://platform.openai.com/api-keys): ```bash theme={null} export OPENAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import OpenAITTS tts = OpenAITTS( model="gpt-4o-mini-tts", voice="ash", ) # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { OpenAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = OpenAITTS({ model: 'gpt-4o-mini-tts', voice: 'ash', }); // Pipeline({ tts, ... }) ``` ## Speed control Use `speed` to adjust the speaking rate. The value is a multiplier in the range `0.25` - `4.0`; leaving it unset uses the provider default of `1.0`: ```python Python theme={null} from zeroruntime.plugins import OpenAITTS tts = OpenAITTS( model="gpt-4o-mini-tts", voice="marin", speed=1.1, ) ``` ```typescript Node JS theme={null} import { OpenAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = OpenAITTS({ model: 'gpt-4o-mini-tts', voice: 'marin', speed: 1.1, }); ``` ## Parameters *Constructor parameters for `OpenAITTS`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ------------- | ------- | ------------------- | ------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | OpenAI API key. Falls back to the `OPENAI_API_KEY` environment variable when unset. | | `voice` | `str` | `"ash"` | Built-in voice name (e.g. `"marin"`, `"cedar"`, `"ash"`, `"coral"`). | | `model` | `str` | `"gpt-4o-mini-tts"` | TTS model: `"gpt-4o-mini-tts"` (newest, most capable), `"tts-1"` (lower latency), or `"tts-1-hd"` (higher quality). | | `sample_rate` | `int` | `24000` | Output audio sample rate in Hz. | | `speed` | `float` | `None` | Speaking speed multiplier (0.25 - 4.0). `None` uses the provider default of 1.0. | | `stream` | `bool` | `True` | When `True`, audio is returned as a streaming response; `False` requests the full audio before playback. | ## Import paths | SDK | Import | Constructor | | ------- | --------------------------------------------------------- | -------------------- | | Python | `from zeroruntime.plugins import OpenAITTS` | `OpenAITTS(...)` | | Node JS | `import { OpenAITTS } from '@zeroruntime/js-sdk/plugins'` | `OpenAITTS({ ... })` | The synthesized audio is streamed back to the caller as the final stage of the [pipeline](/concepts/pipeline). # Papla Source: https://docs.zeroruntime.ai/plugins/tts/papla Use the Papla text-to-speech plugin in a Zero Runtime pipeline. Papla is a **text-to-speech** plugin. It turns the LLM's reply into audio the caller hears. ## Setup Set your Papla API key in the worker environment. Generate a key from the [Papla Media site](https://papla.media/): ```bash theme={null} export PAPLA_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import PaplaTTS tts = PaplaTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { PaplaTTS } from '@zeroruntime/js-sdk/plugins'; const tts = PaplaTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Papla API key. Falls back to the `PAPLA_API_KEY` environment variable when unset. | | `model_id` | `str` | `"papla_p1"` | TTS model identifier, also used as the voice ID for the synthesis request. Defaults to the P1 ultra-realistic English model. | | `sample_rate` | `int` | `24000` | Output audio sample rate in Hz. | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. See the [plugins overview](/plugins/overview) for how TTS fits into the rest of the pipeline. # Resemble AI Source: https://docs.zeroruntime.ai/plugins/tts/resemble Use the Resemble AI text-to-speech plugin in a Zero Runtime pipeline. Resemble AI is a **text-to-speech** plugin. It turns the LLM's reply into audio the caller hears, using Resemble AI's custom neural voice clones. ## Setup Set your Resemble AI API key in the worker environment. Generate a key from the [Resemble AI dashboard](https://app.resemble.ai/account/api): ```bash theme={null} export RESEMBLE_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import ResembleTTS tts = ResembleTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { ResembleTTS } from '@zeroruntime/js-sdk/plugins'; const tts = ResembleTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `api_key` | `str \| None` | `None` | Resemble AI API key. Falls back to the `RESEMBLE_API_KEY` environment variable when unset. | | `voice_uuid` | `str` | `"55592656"` | UUID of the Resemble AI voice (custom or pre-built) to use for synthesis. Retrieve voice UUIDs from the Resemble AI dashboard or the `/voices` API endpoint. | | `sample_rate` | `int` | `22050` | Output audio sample rate in Hz. | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. See the [plugins overview](/plugins/overview) for how TTS fits into the rest of the pipeline. # Rime Source: https://docs.zeroruntime.ai/plugins/tts/rime Use the Rime text-to-speech plugin in a Zero Runtime pipeline. Setup and usage in Python and JavaScript. Rime is a **text-to-speech** plugin. It converts the agent's text responses into low-latency, natural-sounding speech. ## Setup Set your Rime API key in the worker environment. Generate a key from the [Rime site](https://rime.ai/): ```bash theme={null} export RIME_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import RimeTTS tts = RimeTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { RimeTTS } from '@zeroruntime/js-sdk/plugins'; const tts = RimeTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | --------------- | ------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Rime API key. Falls back to the `RIME_API_KEY` environment variable when unset. | | `speaker` | `str` | `"river"` | Voice/speaker identifier to use for synthesis. Available speakers depend on the chosen `model_id`; example cross-model voices include `"astra"`, `"luna"`, `"celeste"`, `"masonry"`, `"albion"`, and `"cove"`. | | `model_id` | `str` | `"mist"` | Rime model identifier. Known options: `"mist"` / `"mistv2"` (fast, streaming-optimised conversational model), `"coda"` (Rime's most realistic conversational voices), `"v1"` (legacy model). | | `sampling_rate` | `int` | `24000` | Output audio sample rate in Hz. | Synthesized audio is streamed back to the caller after the [LLM](/plugins/overview) produces a response. # Sarvam AI Source: https://docs.zeroruntime.ai/plugins/tts/sarvamai Use the Sarvam AI text-to-speech plugin in a Zero Runtime pipeline. Setup, options, and usage. Sarvam AI is a **text-to-speech** plugin that synthesizes the LLM's reply into the agent's voice across 11 Indic languages using the bulbul model family. It occupies the pipeline's `tts` slot. ## Setup Set your Sarvam AI API key in the worker environment. Generate a key from the [Sarvam AI dashboard](https://dashboard.sarvam.ai/key-management): ```bash theme={null} export SARVAM_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import SarvamAITTS tts = SarvamAITTS( model="bulbul:v3", speaker="shubh", language="en-IN", ) # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { SarvamAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = SarvamAITTS({ model: 'bulbul:v3', speaker: 'shubh', language: 'en-IN', }); // Pipeline({ tts, ... }) ``` ## Parameters *Constructor parameters for `SarvamAITTS`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | -------------------- | ------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str \| None` | `None` | Sarvam AI API key. Falls back to the `SARVAM_API_KEY` environment variable when unset. | | `model` | `SarvamAITTSModel \| str` | `"bulbul:v3"` | TTS model: `"bulbul:v3"` (latest, 43+ voices, temperature control) or `"bulbul:v2"` (legacy, pitch/loudness controls). | | `language` | `str` | `"en-IN"` | BCP-47 language code. Supported: `"bn-IN"`, `"en-IN"`, `"gu-IN"`, `"hi-IN"`, `"kn-IN"`, `"ml-IN"`, `"mr-IN"`, `"od-IN"`, `"pa-IN"`, `"ta-IN"`, `"te-IN"`. | | `speaker` | `str` | `"shubh"` | Speaker/voice name, e.g. `"aditya"`, `"ritu"`, `"priya"`, `"neha"`, `"rahul"`. | | `streaming` | `bool` | `True` | Stream audio back in chunks. Set to `False` for a single-response download. | | `sample_rate` | `int` | `24000` | Output sample rate in Hz. Supported: `8000`, `16000`, `22050`, `24000`; `bulbul:v3` also supports `32000`, `44100`, `48000` via the REST API. | | `output_audio_codec` | `str` | `"linear16"` | Audio encoding: `"linear16"`, `"mp3"`, `"mulaw"`, `"alaw"`, `"opus"`, `"flac"`, `"aac"`, `"wav"`. | | `pitch` | `float \| None` | `0.0` | Pitch adjustment (`bulbul:v2` only). Range `-0.75` to `0.75`. No effect on `bulbul:v3`. | | `pace` | `float \| None` | `1.0` | Speaking rate multiplier. `bulbul:v3` range: `0.5`–`2.0`; `bulbul:v2` range: `0.3`–`3.0`. | | `loudness` | `float \| None` | `1.0` | Volume multiplier (`bulbul:v2` only). Range `0.3`–`3.0`. No effect on `bulbul:v3`. | | `temperature` | `float \| None` | `0.6` | Expressiveness/variability control (`bulbul:v3` only). Range `0.01`–`2.0`. | | `bitrate` | `str` | `"128k"` | MP3 bitrate when `output_audio_codec="mp3"`. | | `min_buffer_size` | `int` | `50` | Minimum characters to accumulate before sending a synthesis request in streaming mode. | | `max_chunk_length` | `int` | `150` | Maximum characters per synthesis chunk. | | `preprocessing` | `bool` | `False` | Enable server-side text pre-processing (number normalization, acronym expansion) before synthesis. | The synthesized audio is streamed back to the caller as the final stage of the [pipeline](/concepts/pipeline). # Smallest AI Source: https://docs.zeroruntime.ai/plugins/tts/smallestai Use the Smallest AI text-to-speech plugin in a Zero Runtime pipeline. Setup, options, and usage in Python and JavaScript. Smallest AI is a **text-to-speech** plugin built for very low latency. It synthesizes the LLM's reply into the agent's voice using the Lightning model family. It occupies the pipeline's `tts` slot. ## Setup Set your Smallest AI API key in the worker environment. Generate a key from the [Smallest AI console](https://console.smallest.ai/apikeys): ```bash theme={null} export SMALLESTAI_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import SmallestAITTS tts = SmallestAITTS( model="lightning_v3.1", voice_id="magnus", ) # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { SmallestAITTS } from '@zeroruntime/js-sdk/plugins'; const tts = SmallestAITTS({ model: 'lightning-v3.1', voice_id: 'magnus', }); // Pipeline({ tts, ... }) ``` The default model is `lightning_v3.1`. ## Parameters *Constructor parameters for `SmallestAITTS`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ------------- | ------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------- | | `api_key` | `str` | `None` | Smallest AI API key. Falls back to the `SMALLESTAI_API_KEY` environment variable when unset. | | `voice` | `str` | `"magnus"` | Voice name (e.g. `"magnus"`, `"lauren"`, `"meher"`, `"devansh"`). Takes precedence over `voice_id`. | | `voice_id` | `str` | `None` | Alias for `voice`; ignored when `voice` is also set. | | `model` | `str` | `"lightning_v3.1"` | Model: `"lightning_v3.1"`, `"lightning_v3.1_pro"`, or `"lightning"`. | | `sample_rate` | `int` | `24000` | Output sample rate: one of `8000`, `16000`, `24000`, `44100`. | | `language` | `str` | `"en"` | ISO 639-1 language code: `"en"`, `"hi"`, `"mr"`, `"kn"`, `"ta"`, `"bn"`, `"gu"`, `"te"`, `"ml"`, `"pa"`, `"or"`, or `"es"`. | | `speed` | `float` | `None` | Speaking rate multiplier relative to the voice default; `None` uses the model default. | | `stream` | `bool` | `True` | When `True`, audio is returned as a stream of chunks for lower latency; `False` returns a single synchronous response. | ## Import paths | SDK | Import | Constructor | | ------- | ------------------------------------------------------------- | ------------------------ | | Python | `from zeroruntime.plugins import SmallestAITTS` | `SmallestAITTS(...)` | | Node JS | `import { SmallestAITTS } from '@zeroruntime/js-sdk/plugins'` | `SmallestAITTS({ ... })` | The synthesized audio is streamed back to the caller as the final stage of the [pipeline](/concepts/pipeline). # Speechify Source: https://docs.zeroruntime.ai/plugins/tts/speechify Use the Speechify text-to-speech plugin in a Zero Runtime pipeline. Speechify is a **text-to-speech** plugin. It turns the LLM's reply into audio the caller hears using Speechify's SIMBA model family. ## Setup Set your Speechify API key in the worker environment. Generate a key from the [Speechify console](https://console.sws.speechify.com/): ```bash theme={null} export SPEECHIFY_API_KEY= ``` ## Usage Import the plugin and pass it to the pipeline's `tts` slot. ```python Python theme={null} from zeroruntime.plugins import SpeechifyTTS tts = SpeechifyTTS() # Pipeline(tts=tts, ...) ``` ```typescript Node JS theme={null} import { SpeechifyTTS } from '@zeroruntime/js-sdk/plugins'; const tts = SpeechifyTTS(); // Pipeline({ tts, ... }) ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `api_key` | `str \| None` | `None` | Speechify API key. Falls back to the `SPEECHIFY_API_KEY` environment variable when unset. | | `voice_id` | `str` | `"kristy"` | Speechify voice identifier. Browse the full catalogue at the Speechify voice library. | | `model` | `str` | `"simba-english"` | SIMBA model variant. `"simba-english"` is the flagship English model with the highest quality, lowest streaming latency, and full SSML support. `"simba-multilingual"` adds cross-language support. `"simba-base"` and `"simba-turbo"` are deprecated. | | `sample_rate` | `int` | `24000` | Output audio sample rate in Hz. | Spoken replies support interruptions. When the caller talks over the agent, the runtime stops the audio and listens again. See the [plugins overview](/plugins/overview) for how TTS fits into the rest of the pipeline. # Turn Detector Source: https://docs.zeroruntime.ai/plugins/turn-detection/namo Use the unified Turn Detector plugin with the server-hosted Echo model to decide when the caller has finished speaking, in Python and JavaScript. The **Turn Detector** decides when the caller has finished their turn, so the agent replies at the right moment rather than interrupting or pausing too long. It uses **Echo**, a server-hosted model exposed through the Zero Runtime Inference Gateway via the unified `TurnDetector` class. Nothing is downloaded or loaded on your machine; authentication requires `ZERORUNTIME_AUTH_TOKEN`. Choose `echo-small` for the lowest latency or `echo-large` for higher accuracy. ## Usage Set your auth token, then construct `TurnDetector` with the `echo-small` or `echo-large` model and pass it to the pipeline's `turn_detector` slot. ```python Python theme={null} # Set ZERORUNTIME_AUTH_TOKEN in your environment. from zeroruntime.inference import TurnDetector turn_detector = TurnDetector(model="echo-small") # Pipeline(turn_detector=turn_detector, ...) ``` ```typescript Node JS theme={null} // Set ZERORUNTIME_AUTH_TOKEN in your environment. import { TurnDetector } from '@zeroruntime/js-sdk/inference'; const turn_detector = TurnDetector({ model: 'echo-small' }); // Pipeline({ turn_detector, ... }) ``` The turn detector pairs with [voice activity detection](/plugins/vad/silero): VAD finds that the caller is speaking, the turn detector decides when they have stopped. ## What's next For the four-state classification, model comparison, supported languages, and end-of-utterance handling, see the full [Turn Detection guide](/build/turn-detection-and-interruptions/turn-detection). # Silero Source: https://docs.zeroruntime.ai/plugins/vad/silero Use the Silero voice activity detection plugin in a Zero Runtime pipeline to detect when the caller is speaking. Setup, options, and usage in Python and JavaScript. Silero is a **voice activity detection (VAD)** plugin. It tells the pipeline when the caller is speaking rather than silent, so the agent reacts to speech instead of background noise. ## Usage Import the plugin and pass it to the pipeline's `vad` slot. ```python Python theme={null} from zeroruntime.plugins import SileroVAD vad = SileroVAD(threshold=0.4) # Pipeline(vad=vad, ...) ``` ```typescript Node JS theme={null} import { SileroVAD } from '@zeroruntime/js-sdk/plugins'; const vad = SileroVAD({ threshold: 0.4 }); // Pipeline({ vad, ... }) ``` ## Parameters *Constructor parameters for `SileroVAD`. The Python and Node JS SDKs share these field names.* | Parameter | Type | Default | Description | | ----------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------- | | `threshold` | `float` | `0.4` | Speech-probability cutoff for classifying a frame as speech (0-1). Lower catches softer speech but allows more false positives. | | `start_threshold` | `float` | `None` | Probability needed to start a speech segment. When set, overrides `threshold`. | | `end_threshold` | `float` | `None` | Probability below which a speech segment ends. When set, overrides `stop_threshold`. | | `stop_threshold` | `float` | `0.25` | Probability below which a speech segment ends (used when `end_threshold` is unset). | | `min_speech_duration` | `float` | `0.3` | Minimum seconds of speech before it counts as a segment. | | `min_silence_duration` | `float` | `0.4` | Silence (seconds) required to mark the end of speech. | | `padding_duration` | `float` | `0.5` | Audio (seconds) kept before and after each segment. | | `max_buffered_speech` | `float` | `60.0` | Maximum seconds of speech buffered in memory. | | `sample_rate` | `int` | `16000` | Sample rate the model runs at (used when `model_sample_rate` is unset). | | `input_sample_rate` | `int` | `48000` | Sample rate (Hz) of the incoming audio. | | `model_sample_rate` | `int` | `None` | Sample rate the model runs at; falls back to `sample_rate` when unset. | | `smoothing_factor` | `float` | `0.35` | EMA smoothing weight applied to speech probabilities. | | `force_cpu` | `bool` | `False` | Run inference on CPU even if a GPU is available. | | `min_volume` | `float` | `0.0` | Minimum normalized volume for a frame to be considered. | | `energy_filter_enabled` | `bool` | `True` | Apply the `min_volume` energy filter (when disabled, `min_volume` is treated as `0.0`). | Silero works with [turn detection](/plugins/turn-detection/namo), which decides when a detected speaker has finished their turn. # Build a Telephony Agent with Zero Runtime Source: https://docs.zeroruntime.ai/quickstarts/build-a-telephony-agent Handle inbound and outbound PSTN calls via SIP with an AI voice agent. ## Architecture PSTN callers dial your number; your SIP provider forwards the call to the Zero Runtime SIP Gateway, and a routing rule routes it to your self-hosted agent by matching its `agent_id`.\ For outbound calls, your agent dials through an outbound gateway to the SIP provider, which places the PSTN call. Telephony call flow diagram showing inbound and outbound PSTN call routing through the Zero Runtime SIP Gateway. ## Prerequisites * **Python 3.11 or higher.** * A Zero Runtime Auth token (`ZERORUNTIME_AUTH_TOKEN`). Generate one in the [Zero Runtime Dashboard](https://app.zeroruntime.ai/). * A **SIP provider** (such as [Twilio](https://www.twilio.com/)) with a phone number, plus access to its origination and termination settings. Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install `zrt` and `python-dotenv`. Every provider plugin ships with `zrt`. ```bash uv theme={null} uv venv --python 3.11 source .venv/bin/activate # macOS/Linux .\.venv\Scripts\activate # Windows ``` ```bash pip theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux .\venv\Scripts\activate # Windows ``` New to uv? See the [uv install guide](https://docs.astral.sh/uv/getting-started/installation/). ```bash uv theme={null} uv pip install zeroruntime python-dotenv ``` ```bash pip theme={null} pip install zeroruntime python-dotenv ``` ```bash Node JS theme={null} npm install @zeroruntime/js-sdk dotenv npm install -D tsx typescript ``` Store API keys and tokens in a `.env` file in your project root. ```shell title=".env" theme={null} # The Inference gateway authenticates with your Zero Runtime token - # no separate STT, LLM, or TTS provider keys needed. # pre-generated token ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token" # Runtime Target ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 ``` ```shell title=".env" theme={null} # pre-generated token ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token" # Runtime Target ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 # Used by GeminiRealtime GOOGLE_API_KEY="Google Live API Key" ``` Get a Gemini key from [Google AI Studio](https://aistudio.google.com/app/apikey). For Zero Runtime, use an auth token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/api-keys) or an API key + secret. Follow the [guide](/authentication). For telephony, the agent must register itself so calls can be routed to it. `zeroruntime.serve()` registers the agent under its `agent_id` automatically. The agent half is the same as any voice agent. Pick your language. For telephony, the agent must register itself with Zero Runtime so calls can be routed to it. Give the agent a unique `agent_id` and start it with `zeroruntime.serve()`, which registers it automatically. Inbound calls arrive through routing, so there is no need to self-invoke a session. ```python title="main.py" theme={null} import logging import zeroruntime from zeroruntime import Agent, Pipeline from zeroruntime.plugins import SileroVAD from zeroruntime.plugins import CartesiaSTT, GoogleLLM, CartesiaTTS from zeroruntime.inference import AICousticsDenoise, TurnDetector from dotenv import load_dotenv logging.basicConfig(level=logging.INFO) load_dotenv() AGENT_ID = "MyTelephonyAgent" # unique ID used for call routing pipeline = Pipeline( stt=CartesiaSTT(model="ink-2"), llm=GoogleLLM(model="gemini-2.5-flash"), tts=CartesiaTTS(model="sonic-2"), vad=SileroVAD(threshold=0.35), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class TelephonyAgent(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions="You are a helpful AI assistant that answers phone calls. Keep your responses concise and friendly.", pipeline=pipeline, ) async def on_enter(self): await self.session.say("Hello! I'm your AI telephony assistant. How can I help you today?") async def on_exit(self): await self.session.say("Goodbye! It was great talking with you!") def on_ready(): logging.info("Agent registered as %s and ready for calls.", AGENT_ID) if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh TelephonyAgent + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(TelephonyAgent, on_ready=on_ready) ``` ```typescript title="main.ts" theme={null} import 'dotenv/config'; import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, Pipeline, get_logger } from '@zeroruntime/js-sdk'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; import { CartesiaSTT, CartesiaTTS, GoogleLLM, SileroVAD } from '@zeroruntime/js-sdk/plugins'; const logger = get_logger('telephony'); const AGENT_ID = 'MyTelephonyAgent'; // unique ID used for call routing const pipeline = Pipeline({ stt: CartesiaSTT({ model: 'ink-2' }), llm: GoogleLLM({ model: 'gemini-2.5-flash' }), tts: CartesiaTTS({ model: 'sonic-2' }), vad: SileroVAD({ threshold: 0.35 }), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); class TelephonyAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful AI assistant that answers phone calls. Keep your responses concise and friendly.', pipeline, }); } async on_enter(): Promise { await this.session!.say("Hello! I'm your AI telephony assistant. How can I help you today?"); } async on_exit(): Promise { await this.session!.say('Goodbye! It was great talking with you!'); } } function on_ready(): void { logger.info(`Agent registered as ${AGENT_ID} and ready for calls.`); } // Pass the class itself (not an instance): serve() builds a fresh TelephonyAgent + // pipeline per call, which is required for correct per-call state under concurrent calls. await zeroruntime.serve(TelephonyAgent, { on_ready }); ``` ```python title="main.py" theme={null} import os, logging import zeroruntime from zeroruntime import Agent, Pipeline from zeroruntime.plugins import GeminiRealtime from dotenv import load_dotenv logging.basicConfig(level=logging.INFO) load_dotenv() AGENT_ID = "MyTelephonyAgent" # unique ID used for call routing model = GeminiRealtime( api_key=os.getenv("GOOGLE_API_KEY"), model="gemini-3.1-flash-live-preview", config={ "voice": "Leda", "response_modalities": ["AUDIO"], }, ) pipeline = Pipeline(llm=model) class TelephonyAgent(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions="You are a helpful AI assistant that answers phone calls. Keep your responses concise and friendly.", pipeline=pipeline, ) async def on_enter(self): await self.session.say("Hello! I'm your real-time assistant. How can I help you today?") async def on_exit(self): await self.session.say("Goodbye! It was great talking with you!") def on_ready(): logging.info("Agent registered as %s and ready for calls.", AGENT_ID) if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh TelephonyAgent + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(TelephonyAgent, on_ready=on_ready) ``` ```typescript title="main.ts" theme={null} import 'dotenv/config'; import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, Pipeline, get_logger } from '@zeroruntime/js-sdk'; import { GeminiRealtime } from '@zeroruntime/js-sdk/plugins'; const logger = get_logger('telephony'); const AGENT_ID = 'MyTelephonyAgent'; // unique ID used for call routing const model = GeminiRealtime({ api_key: process.env.GOOGLE_API_KEY, model: 'gemini-3.1-flash-live-preview', config: { voice: 'Leda', response_modalities: ['AUDIO'], }, }); const pipeline = Pipeline({ llm: model }); class TelephonyAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful AI assistant that answers phone calls. Keep your responses concise and friendly.', pipeline, }); } async on_enter(): Promise { await this.session!.say("Hello! I'm your real-time assistant. How can I help you today?"); } async on_exit(): Promise { await this.session!.say('Goodbye! It was great talking with you!'); } } function on_ready(): void { logger.info(`Agent registered as ${AGENT_ID} and ready for calls.`); } // Pass the class itself (not an instance): serve() builds a fresh TelephonyAgent + // pipeline per call, which is required for correct per-call state under concurrent calls. await zeroruntime.serve(TelephonyAgent, { on_ready }); ``` With your `.env` configured and dependencies installed, run the agent: ```bash uv theme={null} uv run python main.py ``` ```bash pip theme={null} python main.py ``` ```bash Node JS theme={null} npx tsx main.ts ``` Keep the process running. It registers with Zero Runtime using the ID `MyTelephonyAgent`. Agent running locally and registering with Zero Runtime 1. Go to the [Zero Runtime Dashboard](https://app.zeroruntime.ai/). 2. Click **Add Number**. 3. Click **Configure SIP**. 4. Give a name and add your phone number. Set up two gateways and point your SIP provider (Twilio in this example) at them so calls flow both ways. Use the Dashboard or the API for each. **1. Inbound gateway**: the entry point for incoming calls. 1. Copy the **Inbound URL** from the Zero Runtime dashboard. 2. Go to **Twilio** and create a new SIP Trunk. 3. Go to the Origination section, paste the Inbound URL there, and save it. ```bash theme={null} curl --request POST \ --url https://api.videosdk.live/v2/sip/inbound-gateways \ --header 'Authorization: YOUR_ZRT_AUTH_TOKEN' \ --header 'Content-Type: application/json' \ --data '{ "name": "My Inbound Gateway", "numbers": ["+1234567890"] }' ``` Then set the **origination URI** of your Twilio SIP trunk to the inbound gateway hostname (shown in the dashboard), e.g. `sip:9WXXXXXXX.sip.videosdk.live`. **2. Outbound gateway**: lets the agent dial out to the PSTN. 1. In Twilio, open the same SIP Trunk and go to the **Termination** section. Create a **Termination URI** and add username/password credentials. 2. Back in the Zero Runtime Dashboard, open the outbound gateway configuration. 3. Paste the Twilio **Termination URI** as the address, add the same **username/password** credentials, and save. ```bash theme={null} curl --request POST \ --url https://api.videosdk.live/v2/sip/outbound-gateways \ --header 'Authorization: YOUR_ZRT_AUTH_TOKEN' \ --header 'Content-Type: application/json' \ --data '{ "name": "My Outbound Gateway", "numbers": ["+12065551234"], "address": "sip.myprovider.com", "transport": "udp", "auth": { "username": "your-username", "password": "your-password" } }' ``` Set `address` and `auth` to match your Twilio **Termination URI** and credentials so outbound calls reach the PSTN. Connect the inbound gateway to your agent by ID. The agent ID must match the `agent_id` you pass to your agent (registered by `zeroruntime.serve()`). 1. From your inbound gateway, click **Configure rule**, then **Create new routing rule**. 2. Enter a **Routing Rule Name** and select **API Key** authentication. 3. Choose the **Call Direction** (Inbound). 4. Enter the **Phone Number** and select a **Room Type**. 5. Enter the **Agent ID** (e.g. `MyTelephonyAgent`) and click **Save**. ```bash theme={null} curl --request POST \ --url https://api.videosdk.live/v2/sip/routing-rule \ --header 'Authorization: YOUR_ZRT_AUTH_TOKEN' \ --header 'Content-Type: application/json' \ --data '{ "gatewayId": "gateway_in_123456789", "name": "Support Line Rule", "numbers": ["+1234567890"], "agentId": "MyTelephonyAgent" }' ``` Make sure your **main.py** is running locally before configuring the telephony settings. The agent must be active to receive incoming calls. Dial your provider's phone number from any phone. The worker waits for the caller to join, then greets you as soon as the call connects. Trigger a call from the agent. Replace `routingRuleId` with your rule ID. Use **curl** or any **API client** to make a POST request to the Zero Runtime API. Replace \$YOUR\_TOKEN and the routingRuleId with your own. ```bash theme={null} curl --request POST \ --url https://api.videosdk.live/v2/sip/call \ --header 'Authorization: YOUR_ZRT_AUTH_TOKEN' \ --header 'Content-Type: application/json' \ --data '{ "sipCallFrom": "+14155550100", "sipCallTo": "+14155550199", "routingRuleId": "rr_2554md" }' ``` For **optimal performance**, run your agent in the same geographic region as your SIP provider (e.g., US East for Twilio, US West for Telnyx, Europe for Plivo). This reduces latency and improves call quality. ## Verify * The agent registers under the ID `MyTelephonyAgent` and the worker process stays running. * An incoming call reaches the agent and you hear the greeting. * Outbound calls dial the target number through your outbound gateway. **Common errors:** * **`agent_id` mismatch:** the routing rule's `agentId` must exactly match the agent's `agent_id`. * **Agent not running:** the agent process must be alive to accept the inbound call. * **SIP misconfiguration:** the provider's origination and termination URIs must point at your Zero Runtime gateways. ## What's Next Explore SIP connect, call routing, and integrations. Handle call transfer, DTMF events, and webhooks. ## References * [Inbound Gateway API](https://docs.videosdk.live/api-reference/realtime-communication/sip/inbound-gateway/create-inbound-gateway) * [Outbound Gateway API](https://docs.videosdk.live/api-reference/realtime-communication/sip/outbound-gateway/create-outbound-gateway) * [Routing Rule API](https://docs.videosdk.live/api-reference/realtime-communication/sip/routing-rules/create-routing-rule) # Build Your First Voice AI Agent Source: https://docs.zeroruntime.ai/quickstarts/build-your-first-voice-agent Ship a voice agent in minutes using a realtime speech-to-speech model or a cascading STT-LLM-TTS pipeline. ## Prerequisites Before you begin, ensure you have: * A Zero Runtime Auth token **or** API key + secret from [app.zeroruntime.ai](https://app.zeroruntime.ai/api-keys) (see [Authentication](/authentication)). * Python 3.11 or higher, or Node.js 20.11 or higher. ## Understanding the Architecture Before diving into implementation, choose the pipeline architecture that fits your use case. **Cascade** processes audio through distinct stages for maximum control: The cascade processes audio through sequential stages: **User Voice Input** → **STT (Speech-to-Text)** → **LLM (Large Language Model)** → **TTS (Text-to-Speech)** → **Agent Voice Output** This approach provides better control over each processing stage and supports more complex AI reasoning. Power each stage through the **Zero Runtime Inference gateway** (one token, no per-provider keys) or bring your own provider keys. **Realtime** provides direct speech-to-speech processing with minimal latency: The realtime pipeline processes audio directly through a unified model: **User Voice Input** → **Speech-to-Speech model** → **Agent Voice Output** This approach offers the fastest response times and is ideal for real-time conversations. ## Build the Agent Pick your language. Each track builds the same agent on the **Cascade** pipeline (Deepgram STT → Google Gemini LLM → Cartesia TTS). Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install `zeroruntime`. Every provider plugin ships with it. ```bash uv theme={null} uv venv --python 3.11 source .venv/bin/activate # macOS/Linux .\.venv\Scripts\activate # Windows ``` ```bash pip theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux .\venv\Scripts\activate # Windows ``` New to uv? See the [uv install guide](https://docs.astral.sh/uv/getting-started/installation/). ```bash uv theme={null} uv pip install zeroruntime ``` ```bash pip theme={null} pip install zeroruntime ``` Prefer to bring your own provider keys? Every provider ships with the SDK. Just import from `zeroruntime.plugins`. See the plugin pages for [STT](/plugins/stt/openai), [LLM](/plugins/llm/openai), and [TTS](/plugins/tts/elevenlabs). ```bash uv theme={null} uv pip install zeroruntime ``` ```bash pip theme={null} pip install zeroruntime ``` Store API keys and tokens in a `.env` file in your project root. ```shell title=".env" theme={null} # The Inference gateway authenticates with your Zero Runtime token - # no separate STT, LLM, or TTS provider keys needed. # pre-generated token ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token" # Runtime Target ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 ``` ```shell title=".env" theme={null} # pre-generated token ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token" # Runtime Target ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 OPENAI_API_KEY="Your OpenAI API Key" # For Google Live API # GOOGLE_API_KEY="Google Live API Key" # For AWS Nova API # AWS_ACCESS_KEY_ID="AWS Key Id" # AWS_SECRET_ACCESS_KEY="AWS Secret Key" # AWS_DEFAULT_REGION="AWS Region" ``` Get keys from [OpenAI](https://platform.openai.com/api-keys), [Gemini](https://aistudio.google.com/app/apikey), or [AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html). For Zero Runtime, use an auth token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/api-keys) or an API key + secret. Follow the [guide](/authentication). Build the pipeline first, then create the agent that uses it. The `Pipeline` auto-detects its mode from the components you pass it: provide STT, LLM, and TTS for Cascade, or a single realtime model for Realtime. Define it at module scope so the agent class can reference it directly. The agent inherits from the base `Agent` class, the same for both pipelines. Only the pipeline setup and imports differ. `instructions` defines the agent's personality, `on_enter()` runs when it joins the room, and `on_exit()` runs when it leaves. ```python title="main.py" theme={null} import zeroruntime from zeroruntime import Agent, Pipeline, Room from zeroruntime.plugins import SileroVAD, CartesiaSTT, GoogleLLM, CartesiaTTS from zeroruntime.inference import AICousticsDenoise, TurnDetector AGENT_ID = "assistant" # Create the pipeline first - STT, LLM, and TTS resolve through the Zero Runtime # Inference gateway. Build it at module scope so the agent class can reference it. pipeline = Pipeline( stt=CartesiaSTT(model="ink-2"), llm=GoogleLLM(model="gemini-2.5-flash"), tts=CartesiaTTS(model="sonic-2"), vad=SileroVAD(threshold=0.4), turn_detector=TurnDetector(model="echo-large"), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class MyVoiceAgent(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions="You are a helpful voice assistant that can answer questions and help with tasks.", pipeline=pipeline, ) async def on_enter(self): await self.session.say("Hello! How can I help?") async def on_exit(self): await self.session.say("Goodbye!") ``` ```python title="main.py" theme={null} import zeroruntime from zeroruntime import Agent, Pipeline, Room from zeroruntime.plugins import OpenAIRealtime AGENT_ID = "assistant" # Create the pipeline first - a single realtime model, built at module scope # so the agent class can reference it. model = OpenAIRealtime( model="gpt-4o-realtime-preview", config={ "voice": "alloy", # alloy, ash, ballad, breeze, cedar, cinnamon, coral, echo, ember, juniper, marin, sage, shimmer, verse "modalities": ["text", "audio"], # turn_detection defaults to server VAD (threshold 0.5); add a # "turn_detection" key to customize it. }, ) pipeline = Pipeline(llm=model) class MyVoiceAgent(Agent): def __init__(self): super().__init__( agent_id=AGENT_ID, instructions="You are a helpful voice assistant that can answer questions and help with tasks.", pipeline=pipeline, ) async def on_enter(self): await self.session.say("Hello! How can I help?") async def on_exit(self): await self.session.say("Goodbye!") ``` With the pipeline and agent defined, hand the agent *class* to `zeroruntime.serve(...)`. This wiring is the same for both pipelines. ```python title="Python" theme={null} def invoke_agent(): # room_id="YOUR_MEETING_ID" # Set to join a pre-created room; omit to auto-create zeroruntime.invoke(AGENT_ID, room=Room(playground=True)) if __name__ == "__main__": # Pass the class itself (not an instance): serve() builds a fresh MyVoiceAgent + # pipeline per call, which is required for correct per-call state under concurrent calls. zeroruntime.serve(MyVoiceAgent, on_ready=invoke_agent) ``` `serve()` registers the agent under its `agent_id` and blocks, starting a fresh session for each caller. Passing the `MyVoiceAgent` class itself (not a pre-built instance) lets `serve()` construct a fresh agent + pipeline per call, which is required for correct per-call state under concurrent calls. `on_ready=invoke_agent` fires `zeroruntime.invoke(...)` once the agent is registered, which resolves a room and returns a playground URL to open the session. With your `.env` configured and dependencies installed, run the project with Python. Once you run this command, a playground URL appears in your terminal. Use this URL to interact with your AI agent. ```bash uv theme={null} uv run python main.py ``` ```bash pip theme={null} python main.py ``` ```bash uv theme={null} uv run python main.py console ``` ```bash pip theme={null} python main.py console ``` Want to see the magic instantly? Console mode lets you interact with your agent directly through the terminal, no meeting room needed, just speak and listen through your local system. Perfect for quick testing and development. Console Mode Learn more about [Console Mode](/build/configure-a-pipeline/console-mode). Use Node.js 20.11 or higher, then install the Zero Runtime Agents SDK. Add `tsx` and `typescript` as dev dependencies so you can run the TypeScript entrypoint directly. Every provider plugin ships with the SDK. ```bash theme={null} npm install @zeroruntime/js-sdk dotenv npm install -D tsx typescript ``` New to `tsx`? It runs `.ts` files with no build step. See the [tsx docs](https://tsx.is/). The SDK is ESM-only. Add `"type": "module"` to your `package.json`. Prefer to bring your own provider keys? Every provider ships with the SDK. Import from `@zeroruntime/js-sdk/plugins` instead of `@zeroruntime/js-sdk/inference`. See the plugin pages for [STT](/plugins/stt/openai), [LLM](/plugins/llm/openai), and [TTS](/plugins/tts/elevenlabs). A realtime model needs only its own vendor key on top of your Zero Runtime token. Store API keys and tokens in a `.env` file in your project root. The SDK loads it via `import 'dotenv/config';` at the top of your entrypoint. ```shell title=".env" theme={null} # The Inference gateway authenticates with your Zero Runtime token - # no separate STT, LLM, or TTS provider keys needed. # pre-generated token ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token" # Runtime Target ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 ``` ```shell title=".env" theme={null} # pre-generated token ZERORUNTIME_AUTH_TOKEN="Zero Runtime Auth token" # Runtime Target ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 OPENAI_API_KEY="Your OpenAI API Key" # For Google Live API # GOOGLE_API_KEY="Google Live API Key" # For AWS Nova API # AWS_ACCESS_KEY_ID="AWS Key Id" # AWS_SECRET_ACCESS_KEY="AWS Secret Key" # AWS_DEFAULT_REGION="AWS Region" ``` Get keys from [OpenAI](https://platform.openai.com/api-keys), [Gemini](https://aistudio.google.com/app/apikey), or [AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html). For Zero Runtime, use an auth token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/api-keys) or an API key + secret. Follow the [guide](/authentication). Build the pipeline first, then create the agent that uses it. `Pipeline` auto-detects its mode from the components you pass it: provide STT, LLM, and TTS for Cascade, or a single realtime model for Realtime. Define it at module scope so the agent class can reference it directly. The agent extends the base `Agent` class, the same for both pipelines. Only the pipeline setup and imports differ. `instructions` defines the agent's personality, `on_enter()` runs when it joins the room, and `on_exit()` runs when it leaves. `Pipeline` and every provider are factory calls, not constructors — no `new` — and each takes a single options object. Option names match the Python SDK, so they are `snake_case` in both. ```typescript title="main.ts" theme={null} import 'dotenv/config'; import { Agent, Pipeline } from '@zeroruntime/js-sdk'; import { AICousticsDenoise, TurnDetector } from '@zeroruntime/js-sdk/inference'; import { CartesiaSTT, CartesiaTTS, GoogleLLM, SileroVAD } from '@zeroruntime/js-sdk/plugins'; const AGENT_ID = 'assistant'; // Create the pipeline first - STT, LLM, and TTS resolve through the Zero Runtime // Inference gateway. Build it at module scope so the agent class can reference it. const pipeline = Pipeline({ stt: CartesiaSTT({ model: 'ink-2' }), llm: GoogleLLM({ model: 'gemini-2.5-flash' }), tts: CartesiaTTS({ model: 'sonic-2' }), vad: SileroVAD({ threshold: 0.4 }), turn_detector: TurnDetector({ model: 'echo-large' }), denoise: AICousticsDenoise({ model_id: 'quail-vf-2.2-l-16khz' }), }); class MyVoiceAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful voice assistant that can answer questions and help with tasks.', pipeline, }); } async on_enter(): Promise { await this.session!.say('Hello! How can I help?'); } async on_exit(): Promise { await this.session!.say('Goodbye!'); } } ``` ```typescript title="main.ts" theme={null} import 'dotenv/config'; import { Agent, Pipeline } from '@zeroruntime/js-sdk'; import { OpenAIRealtime } from '@zeroruntime/js-sdk/plugins'; const AGENT_ID = 'assistant'; // Create the pipeline first - a single realtime model, built at module scope // so the agent class can reference it. const model = OpenAIRealtime({ model: 'gpt-4o-realtime-preview', config: { voice: 'alloy', // alloy, ash, ballad, breeze, cedar, cinnamon, coral, echo, ember, juniper, marin, sage, shimmer, verse modalities: ['text', 'audio'], // turn_detection defaults to server VAD (threshold 0.5); add a // turn_detection key to customize it. }, }); const pipeline = Pipeline({ llm: model }); class MyVoiceAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful voice assistant that can answer questions and help with tasks.', pipeline, }); } async on_enter(): Promise { await this.session!.say('Hello! How can I help?'); } async on_exit(): Promise { await this.session!.say('Goodbye!'); } } ``` With the pipeline and agent defined, hand the agent *class* to `zeroruntime.serve(...)`. This wiring is the same for both pipelines. ```typescript title="Node JS" theme={null} import * as zeroruntime from '@zeroruntime/js-sdk'; import { Room } from '@zeroruntime/js-sdk'; async function invoke_agent(): Promise { // room_id: 'YOUR_MEETING_ID' // Set to join a pre-created room; omit to auto-create await zeroruntime.invoke(AGENT_ID, { room: Room({ playground: true }) }); } // Pass the class itself (not an instance): serve() builds a fresh MyVoiceAgent + // pipeline per call, which is required for correct per-call state under concurrent calls. await zeroruntime.serve(MyVoiceAgent, { on_ready: invoke_agent }); ``` `serve()` registers the agent under its `agent_id` and starts a fresh session for each caller. Passing the `MyVoiceAgent` class itself (not a pre-built instance) lets `serve()` construct a fresh agent + pipeline per call, which is required for correct per-call state under concurrent calls. `on_ready: invoke_agent` fires `zeroruntime.invoke(...)` once the agent is registered, which resolves a room and returns a playground URL to open the session. With your `.env` configured and dependencies installed, run the project with `tsx`. Once you run this command, a playground URL appears in your terminal. Use this URL to interact with your AI agent. ```bash theme={null} npx tsx main.ts ``` ```bash theme={null} npx tsx main.ts console ``` Want to see the magic instantly? Console mode lets you interact with your agent directly through the terminal, no meeting room needed, just speak and listen through your local system. Perfect for quick testing and development. Console Mode Console mode is the same feature described in the Python track above. ## Verify * The agent connects to the room and greets you. * Audio flows both ways: the agent hears you and you hear it. * In Cascade mode, the LLM's responses are spoken back through TTS; in Realtime mode, the model streams speech directly. **Common errors:** * **Missing auth token:** set `ZERORUNTIME_AUTH_TOKEN` in your `.env`. * **Plugin import error:** make sure `zeroruntime` is installed in the active virtual environment, or `@zeroruntime/js-sdk` in the project. Every provider ships with the SDK. * **No audio:** check microphone permissions, or confirm the client joined the same room. Get started quickly with the [Quick Start Example](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples) for the Zero Runtime AI Agent SDK: everything you need to build your first AI agent fast. ## Troubleshooting Common issues when setting up the project: Make sure Python 3.11 is installed, then create the virtual environment with it. ```bash theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux ``` Newer Python versions (3.13+) removed the implicit event loop. Create your virtual environment with Python 3.11. ```bash theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux ``` The SDK is ESM-only. Add `"type": "module"` to your `package.json`, or use a `.mts` entrypoint. ```json theme={null} { "type": "module" } ``` You're running a Python outside the project's virtual environment. Activate it first, then install the dependencies. ```bash theme={null} source venv/bin/activate # macOS/Linux pip install zeroruntime python main.py ``` ## What's Next Let your agent call functions and act. Understand the STT, LLM, and TTS flow. ## References **Python** * [AI Agent (Complete Example)](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples/blob/main/getting_started/cascade_basic.py) * [Zero Runtime Inference Agent (Example)](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples/blob/main/inference/zeroruntime_cascade.py) **Node JS** * [AI Agent (Complete Example)](https://github.com/ZeroRuntimeAI/zeroruntime-js-examples/blob/main/getting_started/cascade_basic.ts) * [Zero Runtime Inference Agent (Example)](https://github.com/ZeroRuntimeAI/zeroruntime-js-examples/blob/main/inference/zeroruntime_cascade.ts) # AI Voice Agent with Flutter Source: https://docs.zeroruntime.ai/quickstarts/mobile/flutter Connect a Zero Runtime voice AI agent to a native Flutter app so users can talk to it in real time using the Google Gemini Live API. Zero Runtime lets you integrate AI agents with real-time voice interaction into your native Flutter application within minutes. In this quickstart, you'll create an AI agent that joins a Flutter meeting room and interacts with users through voice using the Google Gemini Live API. ## Prerequisites Before proceeding, ensure that your development environment meets the following requirements: * A Zero Runtime account (create one from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/)). * The Flutter toolchain and Python 3.11+ installed on your device. * A Google API key with Gemini Live API access. You need a Zero Runtime account to generate a token and a Google API key for the Gemini Live API. Generate a token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/) and a Google API key from [Google AI Studio](https://aistudio.google.com/api-keys). ## Project Structure Your project structure should look like this: You will be working on the following files: * `join_screen.dart`: Responsible for the user interface to join a meeting. * `meeting_screen.dart`: Displays the meeting interface and handles meeting logic. * `api_call.dart`: Handles API calls for creating meetings. * `agent-flutter.py`: The Python AI agent backend using the Google Gemini Live API. * `.env`: Environment variables for the Python agent's API keys. ## 1. Flutter Frontend Create a new Flutter app using the following command: ```bash theme={null} flutter create zrt_ai_agent_flutter_app ``` Install the RTC SDK using the following Flutter command. Make sure you are in your Flutter app directory before you run this command. ```bash theme={null} flutter pub add videosdk flutter pub add http ``` Update `/android/app/src/main/AndroidManifest.xml` for the permissions used to implement the audio and video features. ```xml title="android/app/src/main/AndroidManifest.xml" theme={null} ``` If necessary, in `build.gradle` you will need to increase the `minSdkVersion` of `defaultConfig` up to `23` (Flutter's generator defaults it to `16`). Add the following entries which allow your app to access the camera and microphone to your `/ios/Runner/Info.plist` file: ```xml title="/ios/Runner/Info.plist" theme={null} NSCameraUsageDescription $(PRODUCT_NAME) Camera Usage! NSMicrophoneUsageDescription $(PRODUCT_NAME) Microphone Usage! ``` Uncomment the following line to define a global platform for your project in `/ios/Podfile`: ```ruby title="/ios/Podfile" theme={null} platform :ios, '12.0' ``` Add the following entries to your `/macos/Runner/Info.plist` file, which allow your app to access the camera and microphone. ```xml title="/macos/Runner/Info.plist" theme={null} NSCameraUsageDescription $(PRODUCT_NAME) Camera Usage! NSMicrophoneUsageDescription $(PRODUCT_NAME) Microphone Usage! ``` Add the following entries to your `/macos/Runner/DebugProfile.entitlements` file, which allow your app to access the camera, microphone, and open outgoing network connections. ```xml title="/macos/Runner/DebugProfile.entitlements" theme={null} com.apple.security.network.client com.apple.security.device.camera com.apple.security.device.microphone ``` Add the following entries to your `/macos/Runner/Release.entitlements` file, which allow your app to access the camera, microphone, and open outgoing network connections. ```xml title="/macos/Runner/Release.entitlements" theme={null} com.apple.security.network.server com.apple.security.network.client com.apple.security.device.camera com.apple.security.device.microphone ``` Create a meeting room using the Zero Runtime API: ```bash theme={null} curl -X POST https://api.videosdk.live/v2/rooms \ -H "Authorization: YOUR_JWT_TOKEN_HERE" \ -H "Content-Type: application/json" ``` Copy the `roomId` from the response and configure it in `lib/join_screen.dart` and `lib/api_call.dart`. ```dart title="lib/api_call.dart" theme={null} import 'dart:convert'; import 'package:http/http.dart' as http; //Auth token we will use to generate a meeting and connect to it const token = 'YOUR_ZRT_AUTH_TOKEN'; // API call to create meeting Future createMeeting() async { final http.Response httpResponse = await http.post( Uri.parse('https://api.videosdk.live/v2/rooms'), headers: {'Authorization': token}, ); //Destructuring the roomId from the response return json.decode(httpResponse.body)['roomId']; } ``` ```dart title="lib/join_screen.dart" theme={null} import 'package:flutter/material.dart'; import 'api_call.dart'; import 'meeting_screen.dart'; class JoinScreen extends StatelessWidget { final _meetingIdController = TextEditingController(); JoinScreen({super.key}); void onCreateButtonPressed(BuildContext context) async { // call api to create meeting and navigate to MeetingScreen with meetingId,token await createMeeting().then((meetingId) { if (!context.mounted) return; Navigator.of(context).push( MaterialPageRoute( builder: (context) => MeetingScreen(meetingId: meetingId, token: token), ), ); }); } void onJoinButtonPressed(BuildContext context) { // check meeting id is not null or invaild // if meeting id is vaild then navigate to MeetingScreen with meetingId,token Navigator.of(context).push( MaterialPageRoute( builder: (context) => MeetingScreen(meetingId: "YOUR_MEETING_ID", token: token), ), ); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar(title: const Text('Zero Runtime QuickStart')), body: Padding( padding: const EdgeInsets.all(12.0), child: Center( child: ElevatedButton( onPressed: () => onJoinButtonPressed(context), child: const Text('Join Meeting'), ), ), ), ); } } ``` Create the main `MeetingScreen` widget with audio-only interaction in `lib/meeting_screen.dart`: ```dart title="lib/meeting_screen.dart" theme={null} import 'package:flutter/material.dart'; import 'package:videosdk/videosdk.dart'; import 'participant_tile.dart'; import 'meeting_controls.dart'; class MeetingScreen extends StatefulWidget { final String meetingId; final String token; const MeetingScreen({ super.key, required this.meetingId, required this.token, }); @override State createState() => _MeetingScreenState(); } class _MeetingScreenState extends State { late Room _room; var micEnabled = true; var camEnabled = true; Map participants = {}; @override void initState() { // create room _room = VideoSDK.createRoom( roomId: widget.meetingId, token: widget.token, displayName: "John Doe", micEnabled: micEnabled, camEnabled: false, defaultCameraIndex: 1, // Index of MediaDevices will be used to set default camera ); setMeetingEventListener(); // Join room _room.join(); super.initState(); } // listening to meeting events void setMeetingEventListener() { _room.on(Events.roomJoined, () { setState(() { participants.putIfAbsent( _room.localParticipant.id, () => _room.localParticipant, ); }); }); _room.on(Events.participantJoined, (Participant participant) { setState( () => participants.putIfAbsent(participant.id, () => participant), ); }); _room.on(Events.participantLeft, (String participantId) { if (participants.containsKey(participantId)) { setState(() => participants.remove(participantId)); } }); _room.on(Events.roomLeft, () { participants.clear(); Navigator.popUntil(context, ModalRoute.withName('/')); }); } // onbackButton pressed leave the room Future _onWillPop() async { _room.leave(); return true; } @override Widget build(BuildContext context) { return WillPopScope( onWillPop: () => _onWillPop(), child: Scaffold( appBar: AppBar(title: const Text('Zero Runtime QuickStart')), body: Padding( padding: const EdgeInsets.all(8.0), child: Column( children: [ Text(widget.meetingId), //render all participant Expanded( child: Padding( padding: const EdgeInsets.all(8.0), child: GridView.builder( gridDelegate: const SliverGridDelegateWithFixedCrossAxisCount( crossAxisCount: 2, crossAxisSpacing: 10, mainAxisSpacing: 10, mainAxisExtent: 300, ), itemBuilder: (context, index) { return ParticipantTile( key: Key(participants.values.elementAt(index).id), participant: participants.values.elementAt(index), ); }, itemCount: participants.length, ), ), ), MeetingControls( onToggleMicButtonPressed: () { micEnabled ? _room.muteMic() : _room.unmuteMic(); micEnabled = !micEnabled; }, onLeaveButtonPressed: () => _room.leave(), ), ], ), ), ), ); } } ``` ```dart title="lib/participant_tile.dart" theme={null} import 'package:flutter/material.dart'; import 'package:videosdk/videosdk.dart'; class ParticipantTile extends StatefulWidget { final Participant participant; const ParticipantTile({super.key, required this.participant}); @override State createState() => _ParticipantTileState(); } class _ParticipantTileState extends State { var pariticpantName; @override void initState() { pariticpantName = widget.participant.displayName; super.initState(); } @override Widget build(BuildContext context) { return Padding( padding: const EdgeInsets.all(8.0), child: Container( color: Colors.grey.shade800, child: Center( child: Text( '$pariticpantName', style: TextStyle(color: Colors.white), ), ), ), ); } } ``` ```dart title="lib/meeting_controls.dart" theme={null} import 'package:flutter/material.dart'; class MeetingControls extends StatelessWidget { final void Function() onToggleMicButtonPressed; final void Function() onLeaveButtonPressed; const MeetingControls({ super.key, required this.onToggleMicButtonPressed, required this.onLeaveButtonPressed, }); @override Widget build(BuildContext context) { return Row( mainAxisAlignment: MainAxisAlignment.spaceEvenly, children: [ ElevatedButton( onPressed: onLeaveButtonPressed, child: const Text('Leave'), ), ElevatedButton( onPressed: onToggleMicButtonPressed, child: const Text('Toggle Mic'), ), ], ); } } ``` ## 2. Python AI Agent Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install `zrt` and `python-dotenv`. Every provider plugin ships with `zrt`. ```bash uv theme={null} uv venv --python 3.11 source .venv/bin/activate # macOS/Linux .\.venv\Scripts\activate # Windows ``` ```bash pip theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux .\venv\Scripts\activate # Windows ``` New to uv? See the [uv install guide](https://docs.astral.sh/uv/getting-started/installation/). ```bash uv theme={null} uv pip install zeroruntime python-dotenv ``` ```bash pip theme={null} pip install zeroruntime python-dotenv ``` Create a `.env` file in your project root to store your keys and token securely for the Python agent: ```shell title=".env" theme={null} # Google API Key for the Gemini Live API GOOGLE_API_KEY=your_google_api_key_here # pre-generated Zero Runtime auth token ZERORUNTIME_AUTH_TOKEN=your_zrt_auth_token_here ``` Create the Python AI agent (`agent-flutter.py`) that joins the same meeting room and interacts with users through voice. ```python title="agent-flutter.py" theme={null} import os import zeroruntime from zeroruntime import Agent, Pipeline, Room from zeroruntime.plugins import GeminiRealtime from zeroruntime.inference import AICousticsDenoise from dotenv import load_dotenv load_dotenv() AGENT_ID = "flutter-voice-agent" pipeline = Pipeline( # Realtime speech-to-speech model goes in the pipeline's llm slot - # no separate STT or TTS needed. llm=GeminiRealtime( # When GOOGLE_API_KEY is set in .env you can omit api_key. api_key=os.getenv("GOOGLE_API_KEY"), model="gemini-3.1-flash-live-preview", config={"voice": "Leda", "response_modalities": ["AUDIO"]}, ), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class MyVoiceAgent(Agent): def __init__(self) -> None: super().__init__( name="MyVoiceAgent", agent_id=AGENT_ID, instructions="You are a high-energy game-show host guiding the caller to guess a secret number from 1 to 100 to win 1,000,000$.", pipeline=pipeline, ) async def on_enter(self) -> None: await self.session.say( "Welcome to the Zero Runtime AI Agent game show! I'm your host, and we're about to play for 1,000,000$. Are you ready to play?" ) async def on_exit(self) -> None: await self.session.say("Goodbye!") def invoke_agent() -> None: # Join the same room the Flutter app created (its meeting ID). zeroruntime.invoke(AGENT_ID, room=Room(room_id="YOUR_MEETING_ID")) if __name__ == "__main__": zeroruntime.serve(MyVoiceAgent, on_ready=invoke_agent) ``` ## 3. Run the Application Once you have completed all the steps above, start your Flutter application: ```bash theme={null} flutter run ``` Open a new terminal, activate your virtual environment, and run the Python agent. It reads `GOOGLE_API_KEY` from the `.env` file. ```bash uv theme={null} uv run python agent-flutter.py ``` ```bash pip theme={null} python agent-flutter.py ``` 1. **Join the meeting from the Flutter app:** * Tap the **Join Meeting** button. * Allow microphone permissions when prompted. 2. **Agent connection:** * Once you join, the Python agent detects your participation. * You should see "Participant joined" in the terminal. * The AI agent greets you and initiates the game. 3. **Start playing:** * The agent guides you through a number guessing game (1–100). * Use your microphone to interact with the AI host. ## Troubleshooting * Ensure both the Flutter app and the agent are running. * Check that the room ID matches in both `lib/join_screen.dart` and `agent-flutter.py`. * Verify your Zero Runtime token is valid. * Check device permissions for microphone access. * Ensure your Google API key has Gemini Live API access enabled. * Verify your `GOOGLE_API_KEY` is set in the `.env` file. * Check that the Gemini Live API is enabled in your Google Cloud Console. * Ensure your Flutter version is compatible. * Try cleaning the build: `flutter clean`. * Delete `pubspec.lock` and run `flutter pub get`. ## Next Steps Complete working example with source code. Learn the Cascade and Realtime pipelines in depth. # AI Voice Agent with iOS Source: https://docs.zeroruntime.ai/quickstarts/mobile/ios Connect a Zero Runtime voice AI agent to a native iOS (SwiftUI) app so users can talk to it in real time using the Google Gemini Live API. Zero Runtime lets you integrate AI agents with real-time voice interaction into your native iOS app within minutes. In this quickstart, you'll create an AI agent that joins a meeting room and interacts with users through voice using the Google Gemini Live API. ## Prerequisites Before proceeding, ensure that your development environment meets the following requirements: * A Zero Runtime account (create one from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/)). * iOS 13.0+, Xcode 13.0+, and Swift 5.0+. * Python 3.11+ installed on your device. * A Google API key with Gemini Live API access. You need a Zero Runtime account to generate a token and a Google API key for the Gemini Live API. Generate a token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/) and a Google API key from [Google AI Studio](https://aistudio.google.com/api-keys). ## Project Structure Your final Xcode project structure should look like this: You will be working on the following files: * `JoinScreenView.swift`: Join screen UI. * `MeetingView.swift`: Meeting interface with audio controls. * `MeetingViewController.swift`: Handles meeting logic and events. * `agent-ios.py`: The Python AI agent backend using the Google Gemini Live API. * `.env`: Environment variables for the Python agent's API keys. ## 1. iOS Frontend Create a new iOS app in Xcode: 1. Create a new Xcode project. 2. Choose the **App** template. 3. Add a **Product Name** and save the project. Install the RTC SDK using Swift Package Manager: 1. In Xcode, go to `File > Add Packages...` 2. Enter the repository URL: `https://github.com/videosdk-live/videosdk-rtc-ios-sdk.git` 3. Choose the latest version and click `Add Package`. Add microphone and camera permissions to `Info.plist`: ```xml title="Info.plist" theme={null} NSCameraUsageDescription Camera permission description NSMicrophoneUsageDescription Microphone permission description ``` Create the Swift models and views for the meeting interface. ```swift title="RoomsStruct.swift" theme={null} struct RoomsStruct: Codable { let createdAt, updatedAt, roomID: String? let links: Links? let id: String? enum CodingKeys: String, CodingKey { case createdAt, updatedAt case roomID = "roomId" case links, id } } struct Links: Codable { let getRoom, getSession: String? enum CodingKeys: String, CodingKey { case getRoom = "get_room" case getSession = "get_session" } } ``` ```swift title="JoinScreenView.swift" theme={null} import SwiftUI struct JoinScreenView: View { // State variables for let meetingId: String = "YOUR_MEETING_ID" @State var name: String var body: some View { NavigationView { VStack { Text("Zero Runtime") .font(.largeTitle) .fontWeight(.bold) Text("AI Agent Quickstart") .font(.largeTitle) .fontWeight(.semibold) .padding(.bottom) TextField("Enter Your Name", text: $name) .foregroundColor(Color.black) .autocorrectionDisabled() .font(.headline) .overlay( Image(systemName: "xmark.circle.fill") .padding() .offset(x: 10) .foregroundColor(Color.gray) .opacity(name.isEmpty ? 0.0 : 1.0) .onTapGesture { UIApplication.shared.endEditing() name = "" } , alignment: .trailing) .padding() .background( RoundedRectangle(cornerRadius: 25) .fill(Color.secondary.opacity(0.5)) .shadow(color: Color.gray.opacity(0.10), radius: 10)) .padding() NavigationLink(destination: MeetingView(meetingId: self.meetingId, userName: name ?? "Guest") .navigationBarBackButtonHidden(true)) { Text("Join Meeting") .foregroundColor(Color.white) .padding() .background( RoundedRectangle(cornerRadius: 25.0) .fill(Color.blue)) } } } } } extension UIApplication { func endEditing() { sendAction(#selector(UIResponder.resignFirstResponder), to: nil, from: nil, for: nil) } } ``` ```swift title="MeetingView.swift" theme={null} import SwiftUI import VideoSDKRTC struct MeetingView: View{ @Environment(\.presentationMode) var presentationMode @ObservedObject var meetingViewController = MeetingViewController() @State var meetingId: String? @State var userName: String? @State var isUnMute: Bool = true var body: some View { VStack { if meetingViewController.participants.count == 0 { Text("Meeting Initializing") } else { VStack { VStack(spacing: 20) { Text("Meeting ID: \(meetingViewController.meetingID)") .padding(.vertical) List { ForEach(meetingViewController.participants.indices, id: \.self) { index in Text("Participant Name: \(meetingViewController.participants[index].displayName)") } } } VStack { HStack(spacing: 15) { // mic button Button { if isUnMute { isUnMute = false meetingViewController.meeting?.muteMic() } else { isUnMute = true meetingViewController.meeting?.unmuteMic() } } label: { Text("Toggle Mic") .foregroundStyle(Color.white) .font(.caption) .padding() .background( RoundedRectangle(cornerRadius: 25) .fill(Color.blue)) } // end meeting button Button { meetingViewController.meeting?.end() presentationMode.wrappedValue.dismiss() } label: { Text("End Call") .foregroundStyle(Color.white) .font(.caption) .padding() .background( RoundedRectangle(cornerRadius: 25) .fill(Color.red)) } } .padding(.bottom) } } } } .onAppear() { /// MARK :- configuring the videoSDK VideoSDK.config(token: meetingViewController.token) print(meetingId) if meetingId?.isEmpty == false { print("i ff meeting isd is emty \(meetingId)") // join an existing meeting with provided meeting Id meetingViewController.joinMeeting(meetingId: meetingId!, userName: userName!) } } } } ``` Create the main meeting view controller to handle meeting events. ```swift title="MeetingViewController.swift" theme={null} import Foundation import VideoSDKRTC class MeetingViewController: ObservableObject { var token = "YOUR_ZRT_AUTH_TOKEN" // Add Your token here var meetingId: String = "" var name: String = "" @Published var meeting: Meeting? = nil @Published var participants: [Participant] = [] @Published var meetingID: String = "" func initializeMeeting(meetingId: String, userName: String) { meeting = VideoSDK.initMeeting( meetingId: meetingId, participantName: userName, micEnabled: true, webcamEnabled: false ) meeting?.addEventListener(self) meeting?.join() } } extension MeetingViewController: MeetingEventListener { func onMeetingJoined() { guard let localParticipant = self.meeting?.localParticipant else { return } // add to list participants.append(localParticipant) localParticipant.addEventListener(self) } func onParticipantJoined(_ participant: Participant) { participants.append(participant) // add listener participant.addEventListener(self) } func onParticipantLeft(_ participant: Participant) { participants = participants.filter({ $0.id != participant.id }) } func onMeetingLeft() { meeting?.localParticipant.removeEventListener(self) meeting?.removeEventListener(self) } func onMeetingStateChanged(meetingState: MeetingState) { switch meetingState { case .DISCONNECTED: participants.removeAll() default: print("") } } } extension MeetingViewController: ParticipantEventListener { } extension MeetingViewController { func joinMeeting(meetingId: String, userName: String) { if !token.isEmpty { self.meetingID = meetingId self.initializeMeeting(meetingId: meetingId, userName: userName) } else { print("Auth token required") } } } ``` Configure the main app entry point. ```swift title="zrt_agents_quickstart_iosApp.swift" theme={null} import SwiftUI @main struct zrt_agents_quickstart_iosApp: App { var body: some Scene { WindowGroup { JoinScreenView(name: "") } } } ``` Create a meeting room using the Zero Runtime API: ```bash theme={null} curl -X POST https://api.videosdk.live/v2/rooms \ -H "Authorization: YOUR_JWT_TOKEN_HERE" \ -H "Content-Type: application/json" ``` Copy the `roomId` from the response and set it as the `meetingId` in `JoinScreenView.swift`, and set your token in `MeetingViewController.swift` (`YOUR_ZRT_AUTH_TOKEN`). ## 2. Python AI Agent Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install `zrt` and `python-dotenv`. Every provider plugin ships with `zrt`. ```bash uv theme={null} uv venv --python 3.11 source .venv/bin/activate # macOS/Linux .\.venv\Scripts\activate # Windows ``` ```bash pip theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux .\venv\Scripts\activate # Windows ``` New to uv? See the [uv install guide](https://docs.astral.sh/uv/getting-started/installation/). ```bash uv theme={null} uv pip install zeroruntime python-dotenv ``` ```bash pip theme={null} pip install zeroruntime python-dotenv ``` Create a `.env` file in your project root to store your keys and token securely for the Python agent: ```shell title=".env" theme={null} # Google API Key for the Gemini Live API GOOGLE_API_KEY=your_google_api_key_here # pre-generated Zero Runtime auth token ZERORUNTIME_AUTH_TOKEN=your_zrt_auth_token_here ``` Create the Python AI agent (`agent-ios.py`) that joins the same meeting room and interacts with users through voice. ```python title="agent-ios.py" theme={null} import os import zeroruntime from zeroruntime import Agent, Pipeline, Room from zeroruntime.plugins import GeminiRealtime from zeroruntime.inference import AICousticsDenoise from dotenv import load_dotenv load_dotenv() AGENT_ID = "ios-quickstart-agent" pipeline = Pipeline( # Realtime speech-to-speech model goes in the pipeline's llm slot - # no separate STT or TTS needed. llm=GeminiRealtime( # When GOOGLE_API_KEY is set in .env you can omit api_key. api_key=os.getenv("GOOGLE_API_KEY"), model="gemini-3.1-flash-live-preview", config={ "voice": "Leda", # Puck, Charon, Kore, Fenrir, Aoede, Leda, Orus, Zephyr "response_modalities": ["AUDIO"], }, ), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class MyVoiceAgent(Agent): def __init__(self) -> None: super().__init__( name="MyVoiceAgent", agent_id=AGENT_ID, instructions="You are a high-energy game-show host guiding the caller to guess a secret number from 1 to 100 to win 1,000,000$.", pipeline=pipeline, ) async def on_enter(self) -> None: await self.session.say( "Welcome to the Zero Runtime AI Agent game show! I'm your host, and we're about to play for 1,000,000$. Are you ready to play?" ) async def on_exit(self) -> None: await self.session.say("Goodbye!") def invoke_agent() -> None: # Join the same room the frontend created (its meeting ID). zeroruntime.invoke(AGENT_ID, room=Room(room_id="YOUR_MEETING_ID")) if __name__ == "__main__": zeroruntime.serve(MyVoiceAgent, on_ready=invoke_agent) ``` ## 3. Run the Application Build and run the app from Xcode on a simulator or physical device. Open a new terminal, activate your virtual environment, and run the Python agent. It reads `GOOGLE_API_KEY` from the `.env` file. ```bash uv theme={null} uv run python agent-ios.py ``` ```bash pip theme={null} python agent-ios.py ``` 1. Run the iOS app on a simulator or device. 2. Join the meeting and allow microphone permissions. 3. When you join, the Python agent detects your participation and starts speaking. 4. Talk to the agent in real time and play the number guessing game. ## Troubleshooting * Ensure the same `room_id` is set in both the iOS app (`JoinScreenView.swift`) and the agent's `Room(room_id=...)`. * Verify microphone permissions in iOS Settings > Privacy & Security > Microphone. * For simulator issues, ensure you're using a physical device for microphone testing. * Check that microphone permissions were granted when prompted. * Ensure your Google API key has Gemini Live API access enabled. * Confirm your Zero Runtime token is valid and the `GOOGLE_API_KEY` is set in the `.env` file. * Check that the Gemini Live API is enabled in your Google Cloud Console. ## Next Steps Complete working example with source code. Learn the Cascade and Realtime pipelines in depth. # AI Voice Agent with React Native Source: https://docs.zeroruntime.ai/quickstarts/mobile/react-native Connect a Zero Runtime voice AI agent to a React Native app so users can talk to it in real time using the Google Gemini Live API. Zero Runtime lets you integrate an AI voice agent into your React Native app (Android/iOS) within minutes. The agent joins the same meeting room and interacts over voice using the Google Gemini Live API. ## Prerequisites Before proceeding, ensure that your development environment meets the following requirements: * A Zero Runtime account (create one from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/)). * Node.js and a working React Native environment (Android Studio and/or Xcode). * Python 3.11+ installed on your device. * A Google API key with Gemini Live API access. You need a Zero Runtime account to generate a token and a Google API key for the Gemini Live API. Generate a token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/) and a Google API key from [Google AI Studio](https://aistudio.google.com/api-keys). ## Project Structure First, create an empty project using `mkdir folder_name` in your preferred location for the React Native frontend. Your final project structure should look like this: You will be working on the following files: * `android/`: Contains the Android-specific project files. * `ios/`: Contains the iOS-specific project files. * `App.js`: The main React Native component, containing the UI and meeting logic. * `constants.js`: Stores the token and meeting ID for the frontend. * `index.js`: The entry point of the React Native application, where the SDK is registered. * `agent-react-native.py`: The Python AI agent backend using the Google Gemini Live API. * `.env`: Environment variables for the Python agent's API keys. ## 1. React Native Frontend Create a React Native app and install the React Native SDK: ```bash theme={null} npx react-native init videosdkAiAgentRN cd videosdkAiAgentRN # Install the SDK npm install "@videosdk.live/react-native-sdk" ``` Add the required permissions in the `AndroidManifest.xml` file. ```xml title="android/app/src/main/AndroidManifest.xml" theme={null} ``` Link the necessary native dependencies: ```java title="android/app/build.gradle" theme={null} dependencies { implementation project(':rnwebrtc') } ``` ```gradle title="android/settings.gradle" theme={null} include ':rnwebrtc' project(':rnwebrtc').projectDir = new File(rootProject.projectDir, '../node_modules/@videosdk.live/react-native-webrtc/android') ``` ```java title="MainApplication.kt" theme={null} import live.videosdk.rnwebrtc.WebRTCModulePackage class MainApplication : Application(), ReactApplication { override val reactNativeHost: ReactNativeHost = object : DefaultReactNativeHost(this) { override fun getPackages(): List { val packages = PackageList(this).packages.toMutableList() packages.add(WebRTCModulePackage()) return packages } // ... } } ``` ```java title="android/gradle.properties" theme={null} /* This one fixes a weird WebRTC runtime problem on some devices. */ android.enableDexingArtifactTransform.desugaring=false ``` Include the following line in your `proguard-rules.pro` file (optional, only if you are using Proguard): ```java title="android/app/proguard-rules.pro" theme={null} -keep class org.webrtc.** { *; } ``` In your `build.gradle` file, update the minimum OS/SDK version to `23`: ```java title="android/build.gradle" theme={null} buildscript { ext { minSdkVersion = 23 } } ``` Ensure that you are using CocoaPods version 1.10 or later. To update CocoaPods, reinstall the gem using `sudo gem install cocoapods`. Change the path of `react-native-webrtc` in your Podfile: ```sh title="ios/Podfile" theme={null} pod 'react-native-webrtc', :path => '../node_modules/@videosdk.live/react-native-webrtc' ``` Update the platform field in the Podfile to iOS 12.0 or above, since `react-native-webrtc` doesn't support earlier versions: `platform :ios, '12.0'`. Install the pods: ```sh theme={null} pod install ``` Declare the camera and microphone permissions in `Info.plist` (located at `project folder/ios/projectname/info.plist`): ```html title="ios/MyApp/Info.plist" theme={null} NSCameraUsageDescription Camera permission description NSMicrophoneUsageDescription Microphone permission description ``` Register the SDK services in your root `index.js` file for the initialization service. ```js title="index.js" theme={null} import { AppRegistry } from "react-native"; import App from "./App"; import { name as appName } from "./app.json"; import { register } from "@videosdk.live/react-native-sdk"; register(); AppRegistry.registerComponent(appName, () => App); ``` Create a meeting room using the Zero Runtime API: ```bash theme={null} curl -X POST https://api.videosdk.live/v2/rooms \ -H "Authorization: YOUR_JWT_TOKEN_HERE" \ -H "Content-Type: application/json" ``` Copy the `roomId` from the response and create a `constants.js` file to store your token and meeting ID. ```js title="constants.js" theme={null} export const token = "YOUR_ZRT_AUTH_TOKEN"; export const meetingId = "YOUR_MEETING_ID"; export const name = "User Name"; ``` ```js title="App.js" theme={null} import React from 'react'; import { SafeAreaView, TouchableOpacity, Text, View, FlatList, } from 'react-native'; import { MeetingProvider, useMeeting, } from '@videosdk.live/react-native-sdk'; import { meetingId, token, name } from './constants'; const Button = ({ onPress, buttonText, backgroundColor }) => { return ( {buttonText} ); }; function ControlsContainer({ join, leave, toggleMic }) { return (
```
Create a meeting room using the Zero Runtime API: ```bash theme={null} curl -X POST https://api.videosdk.live/v2/rooms \ -H "Authorization: YOUR_JWT_TOKEN_HERE" \ -H "Content-Type: application/json" ``` Copy the `roomId` from the response and configure it in `config.js`: ```js title="config.js" theme={null} TOKEN = "YOUR_ZRT_AUTH_TOKEN"; ROOM_ID = "YOUR_MEETING_ID"; // Static room ID shared between frontend and agent ``` In `index.js`, retrieve DOM elements, declare variables, and add the core meeting functionality. ```js title="index.js" theme={null} // getting Elements from Dom const leaveButton = document.getElementById("leaveBtn"); const toggleMicButton = document.getElementById("toggleMicBtn"); const createButton = document.getElementById("createMeetingBtn"); const audioContainer = document.getElementById("audioContainer"); const textDiv = document.getElementById("textDiv"); // declare Variables let meeting = null; let meetingId = ""; let isMicOn = false; // Join Agent Meeting Button Event Listener createButton.addEventListener("click", async () => { document.getElementById("join-screen").style.display = "none"; textDiv.textContent = "Please wait, we are joining the meeting"; meetingId = ROOM_ID; initializeMeeting(); }); // Initialize meeting function initializeMeeting() { window.VideoSDK.config(TOKEN); meeting = window.VideoSDK.initMeeting({ meetingId: meetingId, name: "C.V.Raman", micEnabled: true, webcamEnabled: false, }); meeting.join(); meeting.localParticipant.on("stream-enabled", (stream) => { if (stream.kind === "audio") { setAudioTrack(stream, meeting.localParticipant, true); } }); meeting.on("meeting-joined", () => { textDiv.textContent = null; document.getElementById("grid-screen").style.display = "block"; document.getElementById("meetingIdHeading").textContent = `Meeting Id: ${meetingId}`; }); meeting.on("meeting-left", () => { audioContainer.innerHTML = ""; }); meeting.on("participant-joined", (participant) => { let audioElement = createAudioElement(participant.id); participant.on("stream-enabled", (stream) => { if (stream.kind === "audio") { setAudioTrack(stream, participant, false); audioContainer.appendChild(audioElement); } }); }); meeting.on("participant-left", (participant) => { let aElement = document.getElementById(`a-${participant.id}`); if (aElement) aElement.remove(); }); } // Create audio elements for participants function createAudioElement(pId) { let audioElement = document.createElement("audio"); audioElement.setAttribute("autoPlay", "false"); audioElement.setAttribute("playsInline", "true"); audioElement.setAttribute("controls", "false"); audioElement.setAttribute("id", `a-${pId}`); audioElement.style.display = "none"; return audioElement; } // Set audio track function setAudioTrack(stream, participant, isLocal) { if (stream.kind === "audio") { if (isLocal) { isMicOn = true; } else { const audioElement = document.getElementById(`a-${participant.id}`); if (audioElement) { const mediaStream = new MediaStream(); mediaStream.addTrack(stream.track); audioElement.srcObject = mediaStream; audioElement.play().catch((err) => console.error("audioElem.play() failed", err)); } } } } // Implement controls leaveButton.addEventListener("click", async () => { meeting?.leave(); document.getElementById("grid-screen").style.display = "none"; document.getElementById("join-screen").style.display = "block"; }); toggleMicButton.addEventListener("click", async () => { if (isMicOn) meeting?.muteMic(); else meeting?.unmuteMic(); isMicOn = !isMicOn; }); ```
## Part 2: Python AI Agent Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install `zrt` and `python-dotenv`. Every provider plugin ships with `zrt`. ```bash uv theme={null} uv venv --python 3.11 source .venv/bin/activate # macOS/Linux .\.venv\Scripts\activate # Windows ``` ```bash pip theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux .\venv\Scripts\activate # Windows ``` New to uv? See the [uv install guide](https://docs.astral.sh/uv/getting-started/installation/). ```bash uv theme={null} uv pip install zeroruntime python-dotenv ``` ```bash pip theme={null} pip install zeroruntime python-dotenv ``` Create a `.env` file in your project root to store your keys and token securely for the Python agent: ```shell title=".env" theme={null} # Google API Key for the Gemini Live API GOOGLE_API_KEY=your_google_api_key_here # pre-generated Zero Runtime auth token ZERORUNTIME_AUTH_TOKEN=your_zrt_auth_token_here ``` Create the Python AI agent (`agent-js.py`) that joins the same meeting room and interacts with users through voice. ```python title="agent-js.py" theme={null} import os import zeroruntime from zeroruntime import Agent, Pipeline, Room from zeroruntime.plugins import GeminiRealtime from zeroruntime.inference import AICousticsDenoise from dotenv import load_dotenv load_dotenv() AGENT_ID = "game-show-agent" pipeline = Pipeline( llm=GeminiRealtime( api_key=os.getenv("GOOGLE_API_KEY"), # optional when GOOGLE_API_KEY is in .env model="gemini-3.1-flash-live-preview", config={"voice": "Leda", "response_modalities": ["AUDIO"]}, ), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class MyVoiceAgent(Agent): def __init__(self) -> None: super().__init__( name="MyVoiceAgent", agent_id=AGENT_ID, instructions="You are a high-energy game-show host guiding the caller to guess a secret number from 1 to 100 to win 1,000,000$.", pipeline=pipeline, ) async def on_enter(self) -> None: await self.session.say( "Welcome to the Zero Runtime AI Agent game show! I'm your host, and we're about to play for 1,000,000$. Are you ready to play?" ) async def on_exit(self) -> None: await self.session.say("Goodbye!") def invoke_agent() -> None: # Join the same room the frontend created (its meeting ID). zeroruntime.invoke(AGENT_ID, room=Room(room_id="YOUR_MEETING_ID")) if __name__ == "__main__": zeroruntime.serve(MyVoiceAgent, on_ready=invoke_agent) ``` ## Part 3: Run the Application Once you have completed all the steps above, serve your frontend files: ```bash theme={null} # Using Python's built-in server python3 -m http.server 8000 # Or using the Node.js http-server npx http-server -p 8000 ``` Open `http://localhost:8000` in your web browser. Open a new terminal, activate your virtual environment, and run the Python agent. It reads `GOOGLE_API_KEY` from the `.env` file. ```bash uv theme={null} uv run python agent-js.py ``` ```bash pip theme={null} python agent-js.py ``` 1. **Join the meeting from the frontend:** * Click the **Join Agent Meeting** button in your browser. * Allow microphone permissions when prompted. 2. **Agent connection:** * Once you join, the Python agent detects your participation. * You should see "Participant joined" in the terminal. * The AI agent greets you and initiates the game. 3. **Start playing:** * The agent guides you through a number guessing game (1–100). * Use your microphone to interact with the AI host. ## Troubleshooting * Ensure both the frontend and the agent are running. * Check that the `ROOM_ID` matches in `config.js` and `agent-js.py`. * Verify your Zero Runtime token is valid. * Check browser permissions for microphone access. * Ensure your Google API key has Gemini Live API access enabled. * Verify your `GOOGLE_API_KEY` is set in the `.env` file. * Check that the Gemini Live API is enabled in your Google Cloud Console. ## Next Steps Complete working example with source code. Learn the Cascade and Realtime pipelines in depth. # Integrate a Voice Agent with React Source: https://docs.zeroruntime.ai/quickstarts/web-integration/react Connect a Zero Runtime voice AI agent to a React app so users can talk to it in real time using the Google Gemini Live API. Zero Runtime lets you integrate AI agents with real-time voice interaction into your React application within minutes. In this quickstart, you'll create an AI agent that joins a meeting room and interacts with users through voice using the Google Gemini Live API. ## Prerequisites Before proceeding, ensure that your development environment meets the following requirements: * A Zero Runtime account (create one from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/)). * Node.js and Python 3.11+ installed on your device. * A Google API key with Gemini Live API access. You need a Zero Runtime account to generate a token and a Google API key for the Gemini Live API. Generate a token from the [Zero Runtime Dashboard](https://app.zeroruntime.ai/) and a Google API key from [Google AI Studio](https://aistudio.google.com/api-keys). ## Project Structure Your project structure should look like this: You will be working on the following files: * `App.js`: Creates a basic UI for joining the meeting. * `config.js`: Stores the token and room ID. * `index.js`: The entry point of your React application. * `agent-react.py`: The Python AI agent backend using the Google Gemini Live API. * `.env`: Environment variables for API keys. ## Part 1: React Frontend Create a new React app using the command below. ```bash theme={null} npx create-react-app zrt-ai-agent-react-app ``` Install the Zero Runtime React SDK. Make sure you are in your React app directory before you run this command. ```bash theme={null} npm install "@videosdk.live/react-sdk" ``` Create a meeting room using the Zero Runtime API: ```bash theme={null} curl -X POST https://api.videosdk.live/v2/rooms \ -H "Authorization: YOUR_JWT_TOKEN_HERE" \ -H "Content-Type: application/json" ``` Copy the `roomId` from the response and configure it in `src/config.js`: ```js title="src/config.js" theme={null} export const TOKEN = "YOUR_ZRT_AUTH_TOKEN"; export const ROOM_ID = "YOUR_MEETING_ID"; // Create using the Zero Runtime API (curl -X POST https://api.videosdk.live/v2/rooms) ``` Create the main App component with audio-only interaction in `src/App.js`: ```js title="src/App.js" theme={null} import React, { useEffect, useRef, useState } from "react"; import { MeetingProvider, MeetingConsumer, useMeeting, useParticipant } from "@videosdk.live/react-sdk"; import { TOKEN, ROOM_ID } from "./config"; function ParticipantAudio({ participantId }) { const { micStream, micOn, isLocal, displayName } = useParticipant(participantId); const audioRef = useRef(null); useEffect(() => { if (!audioRef.current) return; if (micOn && micStream) { const mediaStream = new MediaStream(); mediaStream.addTrack(micStream.track); audioRef.current.srcObject = mediaStream; audioRef.current.play().catch(() => {}); } else { audioRef.current.srcObject = null; } }, [micStream, micOn]); return (

Participant: {displayName} | Mic: {micOn ? "ON" : "OFF"}

); } function Controls() { const { leave, toggleMic } = useMeeting(); return (
); } function MeetingView({ meetingId, onMeetingLeave }) { const [joined, setJoined] = useState(null); const { join, participants } = useMeeting({ onMeetingJoined: () => setJoined("JOINED"), onMeetingLeft: onMeetingLeave, }); const joinMeeting = () => { setJoined("JOINING"); join(); }; return (

Meeting Id: {meetingId}

{joined === "JOINED" ? (
{[...participants.keys()].map((pid) => ( ))}
) : joined === "JOINING" ? (

Joining the meeting...

) : ( )}
); } export default function App() { const [meetingId] = useState(ROOM_ID); const onMeetingLeave = () => { // no-op; simple sample }; return ( {() => } ); } ```
## Part 2: Python AI Agent Use Python 3.11 or higher, then install the Zero Runtime Agents SDK. Create and activate a virtual environment, then install `zrt` and `python-dotenv`. Every provider plugin ships with `zrt`. ```bash uv theme={null} uv venv --python 3.11 source .venv/bin/activate # macOS/Linux .\.venv\Scripts\activate # Windows ``` ```bash pip theme={null} python3.11 -m venv venv source venv/bin/activate # macOS/Linux .\venv\Scripts\activate # Windows ``` New to uv? See the [uv install guide](https://docs.astral.sh/uv/getting-started/installation/). ```bash uv theme={null} uv pip install zeroruntime python-dotenv ``` ```bash pip theme={null} pip install zeroruntime python-dotenv ``` Create a `.env` file in your project root to store your keys and token securely for the Python agent: ```shell title=".env" theme={null} # Google API Key for the Gemini Live API GOOGLE_API_KEY=your_google_api_key_here # pre-generated Zero Runtime auth token ZERORUNTIME_AUTH_TOKEN=your_zrt_auth_token_here ``` Create the Python AI agent that joins the same meeting room and interacts with users through voice. ```python title="agent-react.py" theme={null} import os import zeroruntime from zeroruntime import Agent, Pipeline, Room from zeroruntime.plugins import GeminiRealtime from zeroruntime.inference import AICousticsDenoise from dotenv import load_dotenv load_dotenv() AGENT_ID = "react-voice-agent" pipeline = Pipeline( llm=GeminiRealtime( api_key=os.getenv("GOOGLE_API_KEY"), # optional when GOOGLE_API_KEY is in .env model="gemini-3.1-flash-live-preview", config={"voice": "Leda", "response_modalities": ["AUDIO"]}, ), denoise=AICousticsDenoise(model_id="quail-vf-2.2-l-16khz"), ) class MyVoiceAgent(Agent): def __init__(self) -> None: super().__init__( name="MyVoiceAgent", agent_id=AGENT_ID, instructions="You are a high-energy game-show host guiding the caller to guess a secret number from 1 to 100 to win 1,000,000$.", pipeline=pipeline, ) async def on_enter(self) -> None: await self.session.say( "Welcome to the Zero Runtime AI Agent game show! I'm your host, and we're about to play for 1,000,000$. Are you ready to play?" ) async def on_exit(self) -> None: await self.session.say("Goodbye!") def invoke_agent() -> None: # Join the same room the frontend created (its meeting ID). zeroruntime.invoke(AGENT_ID, room=Room(room_id="YOUR_MEETING_ID")) if __name__ == "__main__": zeroruntime.serve(MyVoiceAgent, on_ready=invoke_agent) ``` ## Part 3: Run the Application Once you have completed all the steps above, start your React application: ```bash theme={null} # Install dependencies npm install # Start the development server npm start ``` Open `http://localhost:3000` in your web browser. Open a new terminal, activate your virtual environment, and run the Python agent. It reads `GOOGLE_API_KEY` from the `.env` file. ```bash uv theme={null} uv run python agent-react.py ``` ```bash pip theme={null} python agent-react.py ``` 1. **Join the meeting from the React app:** * Click the **Join** button in your browser. * Allow microphone permissions when prompted. 2. **Agent connection:** * Once you join, the Python backend detects your participation. * You should see "Participant joined" in the terminal. * The AI agent greets you and initiates the game. 3. **Start playing:** * The agent guides you through a number guessing game (1–100). * Use your microphone to interact with the AI host. * The agent provides hints and encouragement throughout the game. ## Troubleshooting * Ensure both the frontend and backend are running. * Check that the room ID matches in both `src/config.js` and `agent-react.py`. * Verify your Zero Runtime token is valid. * Check browser permissions for microphone access. * Ensure your Google API key has Gemini Live API access enabled. * Verify your Google API key is correctly set in the environment. * Check that the Gemini Live API is enabled in your Google Cloud Console. * Ensure your Node.js version is compatible. * Clear the npm cache: `npm cache clean --force`. * Delete `node_modules` and reinstall: `rm -rf node_modules && npm install`. ## Next Steps Complete working example with source code. Learn the Cascade and Realtime pipelines in depth. # Node JS Source: https://docs.zeroruntime.ai/sdks/nodejs The Zero Runtime Node JS SDK: build voice AI agents in JavaScript or TypeScript on Node.js 20.11+ with typed, ESM-first APIs. ## Requirements * Node.js 20.11+ * A Zero Runtime auth token from the [dashboard](https://app.zeroruntime.ai/) * API keys for the providers you use ## Install ```bash theme={null} npm install @zeroruntime/js-sdk ``` The package ships compiled JavaScript plus complete type declarations, so both languages are first class: TypeScript runs directly with `tsx`, JavaScript needs no build step at all. Every provider ships with the SDK; most need no extra package, because their credential is a string you set in the environment. ```bash theme={null} npm install -D tsx typescript ``` ## Connect to the Runtime Point the worker at your runtime address, and set your auth token, using the values from the dashboard: ```bash theme={null} export ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 export ZERORUNTIME_AUTH_TOKEN= ``` The SDK reads a `.env` from the working directory if you add `import 'dotenv/config';` at the top of your entrypoint. ## Conventions * Extend `Agent` and implement `on_enter()` and `on_exit()`. * Use `Pipeline()` and provider functions directly. * Use `function_tool()` to define tools. * Access the current session with `this.session!` inside agents. * Import core APIs from `@zeroruntime/js-sdk` and providers from `@zeroruntime/js-sdk/plugins`. ## Example ```typescript title="main.ts" theme={null} import 'dotenv/config'; import * as zeroruntime from '@zeroruntime/js-sdk'; import { Agent, Pipeline, Room, function_tool } from '@zeroruntime/js-sdk'; import { TurnDetector } from '@zeroruntime/js-sdk/inference'; import { CartesiaTTS, DeepgramSTT, GoogleLLM, SileroVAD } from '@zeroruntime/js-sdk/plugins'; const AGENT_ID = 'assistant'; class VoiceAgent extends Agent { constructor() { super({ agent_id: AGENT_ID, instructions: 'You are a helpful voice assistant.', pipeline: Pipeline({ stt: DeepgramSTT(), llm: GoogleLLM(), tts: CartesiaTTS(), vad: SileroVAD(), turn_detector: TurnDetector(), }), }); } async on_enter(): Promise { await this.session!.say('Hello, how can I help you today?'); } async on_exit(): Promise { await this.session!.say('Goodbye!'); } get_weather = function_tool({ name: 'get_weather', description: 'Fetch the current weather for a location.', parameters: { latitude: { type: 'string', description: 'Latitude of the location. Estimate it; do not ask.' }, longitude: { type: 'string', description: 'Longitude of the location. Estimate it; do not ask.' }, }, execute: async ({ latitude, longitude }) => ({ latitude, longitude, temperature_c: 28 }), }); } async function invoke_agent(): Promise { await zeroruntime.invoke(AGENT_ID, { room: Room({ name: 'Cascade Agent', playground: true }), }); } await zeroruntime.serve(VoiceAgent, { on_ready: invoke_agent }); ``` Run it with `npx tsx main.ts`. The playground URL is printed once, on stdout — open it to talk to the agent. Pass the agent **class** to `serve()`, not an instance. `serve()` builds a fresh agent and pipeline per call, which is what keeps per-call state correct under concurrent calls. ## API reference Every export, type, and option in the Node JS SDK. The [Quickstart](/quickstarts/build-your-first-voice-agent) shows a complete agent. Runnable examples are at [zeroruntime-js-examples](https://github.com/ZeroRuntimeAI/zeroruntime-js-examples). # Python Source: https://docs.zeroruntime.ai/sdks/python The Zero Runtime Python SDK: build voice AI agents on Python 3.11+ with a clean, type-hinted API. ## Requirements * Python 3.11+ * A Zero Runtime auth token from the [dashboard](https://app.zeroruntime.ai/) * API keys for the providers you use ## Install ```bash theme={null} pip install zeroruntime ``` ## Connect to the Runtime Point the worker at your runtime address, and set your auth token, using the values from the dashboard: ```bash theme={null} export ZERORUNTIME_TARGET=us2.zeroruntime.ai:443 export ZERORUNTIME_AUTH_TOKEN= ``` ## Conventions * Agents subclass `Agent` and implement the `async def on_enter` and `on_exit` hooks. * Tools use the `@function_tool` decorator; the schema is inferred from type hints and the docstring. * Provider plugins are imported from `zrt.plugins`. The [Quickstart](/quickstarts/build-your-first-voice-agent) shows a complete agent. Runnable examples are at [zeroruntime-python-examples](https://github.com/ZeroRuntimeAI/zeroruntime-python-examples/tree/main). # Routing SIP Calls to Agents Source: https://docs.zeroruntime.ai/telephony/call-routing/ai-agent-routing Learn how to configure routing rules to direct SIP calls to AI agents in Zero Runtime, including Agent Cloud and self-hosted deployment options. You can route inbound or outbound SIP calls directly to an AI agent using Zero Runtime's routing rules. This is useful for building automated voice assistants, IVRs, or connecting callers to intelligent agents for support, sales, or other workflows. ### Example Request ```bash theme={null} curl -X POST 'https://api.videosdk.live/v2/sip/routing-rule' \ -H 'Authorization: YOUR_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "name": "Inbound support rule", "type": "inbound", "agentId": "Your Agent ID", "phoneNumbers": ["+14155551234"] }' ``` ### Example Response ```json theme={null} { "id": "rr_abc123", "name": "Inbound support rule", "type": "inbound", "numbers": ["+14155551234"], "room": { "type": "dynamic", "prefix": "blank" }, "agentId": "agent_xyz", "agentMetadata": { "team": "support" } } ``` ## Deployment Notes for Agent Cloud When using an Agent Cloud deployment, the agent must read the following environment variables to join the correct Zero Runtime room: * `ZERORUNTIME_ROOM_ID`: The room ID the agent should join for the call session. * `ZERORUNTIME_AUTH_TOKEN`: The authentication token for joining the room. Your agent should read these environment variables at runtime to connect to the correct room and handle the SIP call. You can configure agent routing rules via the Zero Runtime dashboard or using the API for automation and CI/CD workflows. ## API Reference * [Building AI Agent for Telephony](/quickstarts/build-a-telephony-agent) * [Create Routing Rule API](https://docs.videosdk.live/api-reference/realtime-communication/sip/routing-rules/create-routing-rule) * [Fetch all Routing Rule API](https://docs.videosdk.live/api-reference/realtime-communication/sip/routing-rules/fetch-all-routing-rule) * [Fetch a Routing Rule API](https://docs.videosdk.live/api-reference/realtime-communication/sip/routing-rules/fetch-routing-rule) * [Update Routing Rule API](https://docs.videosdk.live/api-reference/realtime-communication/sip/routing-rules/update-routing-rule) * [Delete Routing Rule API](https://docs.videosdk.live/api-reference/realtime-communication/sip/routing-rules/delete-routing-rule) # Setting Up Routing Rules Source: https://docs.zeroruntime.ai/telephony/call-routing/setting-up-routing-rules Learn how to set up routing rules to control how inbound and outbound SIP calls are routed to rooms or agents in Zero Runtime. This guide covers dashboard and API configuration. Routing rules in Zero Runtime allow you to control how inbound and outbound SIP calls are routed to specific rooms or agents based on the configurations you define. With routing rules, you can flexibly direct calls according to your business logic, such as sending support calls to a particular room or routing sales calls to a specific agent. You can configure routing rules using the Zero Runtime dashboard or programmatically via the API. ## Configure routing rules using dashboard The dashboard provides an intuitive interface to create and manage routing rules: 1. Navigate to the SIP section and select "Routing Rules". 2. Click "Add Routing Rule". 3. Fill in the required details: * **Gateway:** Select the SIP gateway (inbound or outbound) for this rule. * **Name:** Enter a descriptive name for the rule. * **Numbers:** Specify the phone numbers this rule applies to. * **Dispatch:** Choose how calls should be routed (to a room or agent). * **Room Options:** For room dispatch, select static or dynamic, and configure prefix, room ID, and PIN as needed. * **Agent Options:** For agent dispatch, select agent type and provide agent ID and metadata. * **Additional Options:** Set tags, metadata, and whether to hide the caller's phone number. 4. Save the rule. The new routing rule will be applied immediately.