Skip to main content

Usage guide

Setup, environment variables, and Python/JavaScript/Go usage examples.

​
GroqLLM

Groq LLM plugin using Groq’s OpenAI-compatible inference API. Provides ultra-low-latency inference over Groq’s LPU hardware, supporting Llama 3.x models, OpenAI GPT-OSS variants, and Groq Compound agentic systems. Includes optional reasoning control for supported models.

​
Constructor

​
api_key
str | None
Groq API key. Falls back to the GROQ_API_KEY environment variable when None.
​
model
GroqLLMModel | str
default:"llama-3.3-70b-versatile"
Groq model ID. Known values (GroqLLMModel): "llama-3.3-70b-versatile" (default, 131k context, ~280 tok/s), "llama-3.1-8b-instant" (~560 tok/s), "openai/gpt-oss-20b", "openai/gpt-oss-120b" (~500 tok/s), "groq/compound", "groq/compound-mini". Any string is accepted for forward compatibility.
​
temperature
float
default:"0.7"
Sampling temperature. Higher values produce more varied output. Groq converts 0 to 1e-8 internally. Default: 0.7.
​
max_output_tokens
int
default:"1024"
Maximum tokens to generate in the response. Default: 1024.
​
top_p
float | None
Nucleus sampling probability cutoff (0.0–1.0). None uses the model default.
​
frequency_penalty
float | None
Penalises tokens by how often they appear, reducing repetition. None uses the model default.
​
presence_penalty
float | None
Penalises tokens that have already appeared, encouraging topic diversity. None uses the model default.
​
seed
int | None
Random seed for reproducible outputs when supported. None uses non-deterministic sampling.
​
stop
str | None
Stop sequence; generation halts when this string is produced. None disables early stopping.
​
user
str | None
End-user identifier forwarded to the API for monitoring and abuse detection. None omits the field.
​
tool_choice
str | None
Controls function-calling behaviour. Typical values: "auto" (model decides), "none" (no tools), "required" (must call a tool), or a specific tool name. None uses the model default.
​
parallel_tool_calls
bool | None
When True, the model may emit multiple tool calls in a single turn. None uses the model default.
​
response_format
Structured output format descriptor (e.g. {"type": "json_object"}). None returns plain text.
​
reasoning_effort
str | None
Reasoning token budget hint for supported reasoning models. For GPT-OSS models: "low", "medium", or "high". None omits the parameter.
​
reasoning_format
str | None
Controls how the model’s reasoning chain is surfaced. "parsed" places it in a dedicated message field; "raw" embeds it inside <think> tags in the content; "hidden" returns only the final answer. Mutually exclusive with include_reasoning. None omits the parameter.
​
service_tier
str | None
Groq service tier. "on_demand" (default when omitted), "flex" (higher throughput, best-effort), "performance" (enterprise, lowest latency), or "auto" (platform selects the best available tier). None uses the Groq API default ("on_demand").

​
aclose

Release any resources held by the provider.

​
cancel_current_generation

Cancel the in-progress generation for the active session, if any.

​
chat

Generate a streamed chat completion.
​
messages
Any
required
The conversation history to send to the model.
​
tools
Any
Optional tool definitions the model may call.
​
conversational_graph
Any
Optional conversational graph guiding the exchange.
​
returns
AsyncIterator[LLMResponse]
LLMResponse: Response chunks produced as the model generates output.

​
emit

Emit an event, invoking all registered handlers with the given arguments. Handlers are called in registration order. Coroutine handlers are scheduled on the running event loop. If the emitter is closed, the call is a no-op.
​
event
T
required
The event to emit.

​
off

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.
​
event
T
required
The event the handler was registered for.
​
callback
Callable[..., Any]
required
The handler to remove.

​
on

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.
​
event
T
required
The event to listen for.
​
callback
Callable[..., Any] | None
The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.
​
returns
Callable[..., Any]
The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.

​
GroqLLMModel