Skip to main content

Usage guide

Setup, environment variables, and Python/JavaScript/Go usage examples.

​
AWSBedrockLLM

AWS Bedrock LLM plugin (Converse API). Streams text generation and function/tool calling from any Bedrock-hosted model (Amazon Nova, Anthropic Claude, Meta Llama, Mistral, Google Gemma). Credentials resolve as explicit argument -> environment variable -> the default AWS credential chain (IAM role / shared profile).

​
Constructor

​
model
str
default:"amazon.nova-lite-v1:0"
Bedrock model id or inference profile ARN. Defaults to "amazon.nova-lite-v1:0"; falls back to the BEDROCK_INFERENCE_PROFILE_ARN env var.
​
region
Optional[str]
AWS region for Bedrock Runtime. Falls back to AWS_DEFAULT_REGION / AWS_REGION, then "us-east-1".
​
aws_access_key_id
Optional[str]
Falls back to AWS_ACCESS_KEY_ID.
​
aws_secret_access_key
Optional[str]
Falls back to AWS_SECRET_ACCESS_KEY.
​
aws_session_token
Optional[str]
Falls back to AWS_SESSION_TOKEN. When no credentials are resolved, the runtime uses the default AWS credential chain (IAM role, shared profile).
​
temperature
float
default:"0.7"
Sampling temperature. Defaults to 0.7.
​
max_output_tokens
int
default:"1024"
Max tokens generated per response. Defaults to 1024. The legacy max_tokens kwarg is also accepted.
​
top_p
Optional[float]
Nucleus sampling probability mass.
​
top_k
Optional[int]
Top-K tokens considered (sent via additional request fields; model support varies).
​
stop_sequences
list[str] | str | None
Sequences that stop generation; a string or list.
​
tool_choice
str
default:"auto"
"auto", "required", "none", or a tool name.
​
cache_system
Optional[bool]
Add a prompt-cache checkpoint after the system prompt.
​
cache_tools
Optional[bool]
Add a prompt-cache checkpoint after the tool definitions.
​
strip_thinking
Optional[bool]
Remove <thinking>...</thinking> spans from the streamed text. Defaults to enabled in the runtime (Nova emits chain-of-thought that TTS would otherwise read aloud).
​
text_tool_calls
Optional[bool]
Parse function calls a model prints as plain text instead of native Converse tool use. Auto-enabled for models that lack native tool use (e.g. Gemma) when left unset.
​
additional_request_fields
Optional[dict]
Extra additionalModelRequestFields merged into the Converse request for model-specific parameters.

​
aclose

Release any resources held by the provider.

​
cancel_current_generation

Cancel the in-progress generation for the active session, if any.

​
chat

Generate a streamed chat completion.
​
messages
Any
required
The conversation history to send to the model.
​
tools
Any
Optional tool definitions the model may call.
​
conversational_graph
Any
Optional conversational graph guiding the exchange.
​
returns
AsyncIterator[LLMResponse]
LLMResponse: Response chunks produced as the model generates output.

​
emit

Emit an event, invoking all registered handlers with the given arguments. Handlers are called in registration order. Coroutine handlers are scheduled on the running event loop. If the emitter is closed, the call is a no-op.
​
event
T
required
The event to emit.

​
off

Remove a previously registered handler for an event. If the handler is not registered for the event, the call is a no-op.
​
event
T
required
The event the handler was registered for.
​
callback
Callable[..., Any]
required
The handler to remove.

​
on

Register a handler for an event. Can be used directly by passing a callback, or as a decorator when callback is omitted.
​
event
T
required
The event to listen for.
​
callback
Callable[..., Any] | None
The handler to invoke when the event is emitted. If None, a decorator is returned that registers the decorated function.
​
returns
Callable[..., Any]
The registered callback when callback is provided, otherwise a decorator that registers and returns the function it wraps.