> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Groq

> Use the Groq LLM plugin in a Zero Runtime pipeline for low-latency inference. Setup, options, and usage in Python, JavaScript, and Go.

Groq is an **LLM** plugin built for low-latency inference. It takes the transcribed
conversation and generates the reply, running open models such as Llama on Groq's LPU
hardware.

## Setup

Set your Groq API key in the worker environment. Generate a key from the [Groq console](https://console.groq.com/keys):

```bash theme={null}
export GROQ_API_KEY=<key>
```

## Usage

Import the plugin and pass it to the pipeline's `llm` slot.

<CodeGroup>
  ```python Python theme={null}
  from zrt.plugins import GroqLLM

  llm = GroqLLM(
      model="llama-3.3-70b-versatile",
      temperature=0.7,
  )

  # Pipeline(llm=llm, ...)
  ```
</CodeGroup>

## Configuration Options

*Constructor parameters for the Python SDK (`GroqLLM`). The JavaScript and Go SDKs expose equivalent options where supported, in their idiomatic form (camelCase / `LLMOptions` struct).*

| Parameter             | Type    | Default                     | Description                                                                                                                                            |
| --------------------- | ------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `api_key`             | `str`   | `None`                      | Groq API key. Falls back to the `GROQ_API_KEY` environment variable when unset.                                                                        |
| `model`               | `str`   | `"llama-3.3-70b-versatile"` | Groq model used to generate replies (e.g. `llama-3.1-8b-instant`, `openai/gpt-oss-20b`, `openai/gpt-oss-120b`, `groq/compound`, `groq/compound-mini`). |
| `temperature`         | `float` | `0.7`                       | Sampling randomness; lower is more deterministic.                                                                                                      |
| `max_output_tokens`   | `int`   | `1024`                      | Maximum tokens to generate per response.                                                                                                               |
| `top_p`               | `float` | `None`                      | Nucleus-sampling probability cutoff.                                                                                                                   |
| `frequency_penalty`   | `float` | `None`                      | Penalises tokens by how often they appear, reducing repetition.                                                                                        |
| `presence_penalty`    | `float` | `None`                      | Penalises tokens that have already appeared, encouraging topic diversity.                                                                              |
| `seed`                | `int`   | `None`                      | Seed for reproducible sampling.                                                                                                                        |
| `stop`                | `str`   | `None`                      | Stop sequence that halts generation.                                                                                                                   |
| `user`                | `str`   | `None`                      | End-user identifier passed to the API.                                                                                                                 |
| `tool_choice`         | `str`   | `None`                      | How the model decides when to call tools (`"auto"`, `"none"`, `"required"`, or a specific tool name).                                                  |
| `parallel_tool_calls` | `bool`  | `None`                      | When `True`, the model may emit multiple tool calls in a single turn.                                                                                  |
| `response_format`     | `dict`  | `None`                      | Structured output format descriptor (e.g. `{"type": "json_object"}`).                                                                                  |
| `reasoning_effort`    | `str`   | `None`                      | Reasoning token budget hint for supported reasoning models (`"low"`, `"medium"`, `"high"`).                                                            |
| `reasoning_format`    | `str`   | `None`                      | Controls how the reasoning chain is surfaced (`"parsed"`, `"raw"`, `"hidden"`).                                                                        |
| `service_tier`        | `str`   | `None`                      | Groq service tier (`"on_demand"`, `"flex"`, `"performance"`, `"auto"`).                                                                                |

## Import paths

| SDK        | Import                            | Constructor                         |
| ---------- | --------------------------------- | ----------------------------------- |
| Python     | `from zrt.plugins import GroqLLM` | `GroqLLM(...)`                      |
| JavaScript | `from '@zrt/js-sdk/plugins/groq'` | `GroqLLM({ ... })`                  |
| Go         | `.../zrt-golang-sdk/plugins/groq` | `groq.NewLLM(groq.LLMOptions{...})` |

The reply is streamed to the [text-to-speech](/plugins/tts/cartesia) plugin, which
synthesizes the agent's voice.