> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Groq

> Python API reference for the groq llm plugin.

<Card title="Usage guide" icon="book" href="/plugins/llm/groq" horizontal>
  Setup, environment variables, and Python/JavaScript/Go usage examples.
</Card>

## GroqLLM

Groq LLM plugin using Groq's OpenAI-compatible inference API.

Provides ultra-low-latency inference over Groq's LPU hardware, supporting
Llama 3.x models, OpenAI GPT-OSS variants, and Groq Compound agentic
systems. Includes optional reasoning control for supported models.

### Constructor

```python theme={null}
GroqLLM(*, api_key: 'str | None' = None, model: 'GroqLLMModel | str' = 'llama-3.3-70b-versatile', temperature: 'float' = 0.7, max_output_tokens: 'int' = 1024, top_p: 'float | None' = None, frequency_penalty: 'float | None' = None, presence_penalty: 'float | None' = None, seed: 'int | None' = None, stop: 'str | None' = None, user: 'str | None' = None, tool_choice: 'str | None' = None, parallel_tool_calls: 'bool | None' = None, response_format=None, reasoning_effort: 'str | None' = None, reasoning_format: 'str | None' = None, service_tier: 'str | None' = None, **kwargs)
```

<ParamField path="api_key" type="str | None">
  Groq API key. Falls back to the `GROQ_API_KEY` environment variable when `None`.
</ParamField>

<ParamField path="model" type="GroqLLMModel | str" default="llama-3.3-70b-versatile">
  Groq model ID. Known values (`GroqLLMModel`): `"llama-3.3-70b-versatile"` (default, 131k context, \~280 tok/s), `"llama-3.1-8b-instant"` (\~560 tok/s), `"openai/gpt-oss-20b"`, `"openai/gpt-oss-120b"` (\~500 tok/s), `"groq/compound"`, `"groq/compound-mini"`. Any string is accepted for forward compatibility.
</ParamField>

<ParamField path="temperature" type="float" default="0.7">
  Sampling temperature. Higher values produce more varied output. Groq converts `0` to `1e-8` internally. Default: `0.7`.
</ParamField>

<ParamField path="max_output_tokens" type="int" default="1024">
  Maximum tokens to generate in the response. Default: `1024`.
</ParamField>

<ParamField path="top_p" type="float | None">
  Nucleus sampling probability cutoff (0.0–1.0). `None` uses the model default.
</ParamField>

<ParamField path="frequency_penalty" type="float | None">
  Penalises tokens by how often they appear, reducing repetition. `None` uses the model default.
</ParamField>

<ParamField path="presence_penalty" type="float | None">
  Penalises tokens that have already appeared, encouraging topic diversity. `None` uses the model default.
</ParamField>

<ParamField path="seed" type="int | None">
  Random seed for reproducible outputs when supported. `None` uses non-deterministic sampling.
</ParamField>

<ParamField path="stop" type="str | None">
  Stop sequence; generation halts when this string is produced. `None` disables early stopping.
</ParamField>

<ParamField path="user" type="str | None">
  End-user identifier forwarded to the API for monitoring and abuse detection. `None` omits the field.
</ParamField>

<ParamField path="tool_choice" type="str | None">
  Controls function-calling behaviour. Typical values: `"auto"` (model decides), `"none"` (no tools), `"required"` (must call a tool), or a specific tool name. `None` uses the model default.
</ParamField>

<ParamField path="parallel_tool_calls" type="bool | None">
  When `True`, the model may emit multiple tool calls in a single turn. `None` uses the model default.
</ParamField>

<ParamField path="response_format">
  Structured output format descriptor (e.g. `&#123;"type": "json_object"&#125;`). `None` returns plain text.
</ParamField>

<ParamField path="reasoning_effort" type="str | None">
  Reasoning token budget hint for supported reasoning models. For GPT-OSS models: `"low"`, `"medium"`, or `"high"`. `None` omits the parameter.
</ParamField>

<ParamField path="reasoning_format" type="str | None">
  Controls how the model's reasoning chain is surfaced. `"parsed"` places it in a dedicated message field; `"raw"` embeds it inside `&lt;think>` tags in the content; `"hidden"` returns only the final answer. Mutually exclusive with `include_reasoning`. `None` omits the parameter.
</ParamField>

<ParamField path="service_tier" type="str | None">
  Groq service tier. `"on_demand"` (default when omitted), `"flex"` (higher throughput, best-effort), `"performance"` (enterprise, lowest latency), or `"auto"` (platform selects the best available tier). `None` uses the Groq API default (`"on_demand"`).
</ParamField>

### aclose

```python theme={null}
def aclose(self) -> 'None'
```

Release any resources held by the provider.

### cancel\_current\_generation

```python theme={null}
def cancel_current_generation(self) -> 'None'
```

Cancel the in-progress generation for the active session, if any.

### chat

```python theme={null}
def chat(self, messages: 'Any', tools: 'Any' = None, conversational_graph: 'Any' = None, **kwargs: 'Any') -> 'AsyncIterator[LLMResponse]'
```

Generate a streamed chat completion.

<ParamField path="messages" type="Any" required>
  The conversation history to send to the model.
</ParamField>

<ParamField path="tools" type="Any">
  Optional tool definitions the model may call.
</ParamField>

<ParamField path="conversational_graph" type="Any">
  Optional conversational graph guiding the exchange.
</ParamField>

<ResponseField name="returns" type="AsyncIterator[LLMResponse]">
  LLMResponse: Response chunks produced as the model generates output.
</ResponseField>

### emit

```python theme={null}
def emit(self, event: 'T', *args: 'Any') -> 'None'
```

Emit an event, invoking all registered handlers with the given arguments.

Handlers are called in registration order. Coroutine handlers are
scheduled on the running event loop. If the emitter is closed, the call
is a no-op.

<ParamField path="event" type="T" required>
  The event to emit.
</ParamField>

### off

```python theme={null}
def off(self, event: 'T', callback: 'Callable[..., Any]') -> 'None'
```

Remove a previously registered handler for an event.

If the handler is not registered for the event, the call is a no-op.

<ParamField path="event" type="T" required>
  The event the handler was registered for.
</ParamField>

<ParamField path="callback" type="Callable[..., Any]" required>
  The handler to remove.
</ParamField>

### on

```python theme={null}
def on(self, event: 'T', callback: 'Callable[..., Any] | None' = None) -> 'Callable[..., Any]'
```

Register a handler for an event.

Can be used directly by passing a callback, or as a decorator when
`callback` is omitted.

<ParamField path="event" type="T" required>
  The event to listen for.
</ParamField>

<ParamField path="callback" type="Callable[..., Any] | None">
  The handler to invoke when the event is emitted. If `None`, a decorator is returned that registers the decorated function.
</ParamField>

<ResponseField name="returns" type="Callable[..., Any]">
  The registered callback when `callback` is provided, otherwise a decorator that registers and returns the function it wraps.
</ResponseField>

***

## GroqLLMModel

```python theme={null}
GroqLLMModel = typing.Literal['openai/gpt-oss-20b', 'openai/gpt-oss-120b', 'groq/compound', 'groq/compound-mini', 'llama-3.1-8b-instant', 'llama-3.3-70b-versatile']
```