Groq

Groq is an LLM plugin built for low-latency inference. It takes the transcribed conversation and generates the reply, running open models such as Llama on Groq’s LPU hardware.

Setup

Set your Groq API key in the worker environment. Generate a key from the Groq console:

export GROQ_API_KEY=<key>

Usage

Import the plugin and pass it to the pipeline’s llm slot.

from zrt.plugins import GroqLLM

llm = GroqLLM(
    model="llama-3.3-70b-versatile",
    temperature=0.7,
)

# Pipeline(llm=llm, ...)

Configuration Options

Constructor parameters for the Python SDK (GroqLLM). The JavaScript and Go SDKs expose equivalent options where supported, in their idiomatic form (camelCase / LLMOptions struct).

Parameter	Type	Default	Description
`api_key`	`str`	`None`	Groq API key. Falls back to the `GROQ_API_KEY` environment variable when unset.
`model`	`str`	`"llama-3.3-70b-versatile"`	Groq model used to generate replies (e.g. `llama-3.1-8b-instant`, `openai/gpt-oss-20b`, `openai/gpt-oss-120b`, `groq/compound`, `groq/compound-mini`).
`temperature`	`float`	`0.7`	Sampling randomness; lower is more deterministic.
`max_output_tokens`	`int`	`1024`	Maximum tokens to generate per response.
`top_p`	`float`	`None`	Nucleus-sampling probability cutoff.
`frequency_penalty`	`float`	`None`	Penalises tokens by how often they appear, reducing repetition.
`presence_penalty`	`float`	`None`	Penalises tokens that have already appeared, encouraging topic diversity.
`seed`	`int`	`None`	Seed for reproducible sampling.
`stop`	`str`	`None`	Stop sequence that halts generation.
`user`	`str`	`None`	End-user identifier passed to the API.
`tool_choice`	`str`	`None`	How the model decides when to call tools (`"auto"`, `"none"`, `"required"`, or a specific tool name).
`parallel_tool_calls`	`bool`	`None`	When `True`, the model may emit multiple tool calls in a single turn.
`response_format`	`dict`	`None`	Structured output format descriptor (e.g. `{"type": "json_object"}`).
`reasoning_effort`	`str`	`None`	Reasoning token budget hint for supported reasoning models (`"low"`, `"medium"`, `"high"`).
`reasoning_format`	`str`	`None`	Controls how the reasoning chain is surfaced (`"parsed"`, `"raw"`, `"hidden"`).
`service_tier`	`str`	`None`	Groq service tier (`"on_demand"`, `"flex"`, `"performance"`, `"auto"`).

Import paths

SDK	Import	Constructor
Python	`from zrt.plugins import GroqLLM`	`GroqLLM(...)`
JavaScript	`from '@zrt/js-sdk/plugins/groq'`	`GroqLLM({ ... })`
Go	`.../zrt-golang-sdk/plugins/groq`	`groq.NewLLM(groq.LLMOptions{...})`

The reply is streamed to the text-to-speech plugin, which synthesizes the agent’s voice.

Plugins

Inference Gateway

Speech-to-text

LLM

Text-to-speech

Realtime models

Avatars

Turn detection

Voice activity detection

Noise cancellation

Setup

Usage

Configuration Options

Import paths

​Setup

​Usage

​Configuration Options

​Import paths

Setup

Usage

Configuration Options

Import paths