Chat & Context - Zero Runtime

The conversation state an agent carries: chat context and messages, function-call records, and image content, plus helpers for windowing history and encoding images for vision.

ChatContext

Mutable container for a conversation’s messages and agent handoffs. Holds the ordered list of ChatMessage items that make up a conversation along with any AgentHandoff records, and provides helpers for adding messages, estimating size, truncating history, and converting to and from transport message formats.

Constructor

ChatContext(initial_messages: 'Optional[List[ChatMessage]]' = None, initial_handoffs: 'Optional[List[AgentHandoff]]' = None)

initial_messages

Optional[List[ChatMessage]]

Messages to seed the context with. Copied into a new list; defaults to empty.

initial_handoffs

Optional[List[AgentHandoff]]

Handoff records to seed the context with. Copied into a new list; defaults to empty.

add_attributed_message

def add_attributed_message(self, role: 'ChatRole', content: 'str', agent_id: 'str', message_id: 'Optional[str]' = None, images: 'Optional[List[ImageContent]]' = None) -> 'ChatMessage'

Append a message attributed to a specific agent.

role

ChatRole

required

The role of the message author.

content

str

required

The message text. Non-string values are coerced to str.

agent_id

str

required

Identifier of the agent that produced the message.

message_id

Optional[str]

Identifier for the message. A new UUID is generated when not provided.

images

Optional[List[ImageContent]]

Optional image attachments for the message.

returns

ChatMessage

The newly created and appended ChatMessage.

add_handoff

def add_handoff(self, to_agent: 'str', from_agent: 'str' = '', reason: 'str' = '') -> 'AgentHandoff'

Record a handoff from one agent to another.

to_agent

str

required

Identifier of the agent receiving control.

from_agent

str

default:""

Identifier of the agent relinquishing control.

reason

str

default:""

Optional explanation for the handoff.

returns

AgentHandoff

The newly created and appended AgentHandoff with a generated id.

add_message

def add_message(self, role: 'ChatRole', content: 'str', message_id: 'Optional[str]' = None, images: 'Optional[List[ImageContent]]' = None, created_at: 'Optional[float]' = None, replace: 'bool' = False) -> 'ChatMessage'

Append a message to the context.

role

ChatRole

required

The role of the message author.

content

str

required

The message text. Non-string values are coerced to str.

message_id

Optional[str]

Identifier for the message. A new UUID is generated when not provided.

images

Optional[List[ImageContent]]

Optional image attachments for the message.

created_at

Optional[float]

Optional creation timestamp (currently unused).

replace

bool

default:"False"

Reserved flag (currently unused).

returns

ChatMessage

The newly created and appended ChatMessage.

copy

def copy(self, **kwargs) -> "'ChatContext'"

Create a copy of this context.

returns

'ChatContext'

A new ChatContext sharing the current messages and handoffs.

empty

def empty() -> "'ChatContext'"

Create an empty chat context.

returns

'ChatContext'

A new ChatContext with no messages or handoffs.

estimated_tokens

def estimated_tokens(self) -> 'int'

Estimate the token count of the conversation. Uses a rough heuristic of two tokens per whitespace-separated word across all message contents.

returns

int

The estimated total token count.

from_context_messages

def from_context_messages(messages: 'Any') -> "'ChatContext'"

Build a context from transport message dictionaries or objects. Accepts an iterable of mappings or attribute-bearing objects and reconstructs ChatMessage entries, including any images and tool calls. Unknown or invalid roles fall back to ChatRole.USER.

messages

Any

required

An iterable of message mappings or objects, or None.

returns

'ChatContext'

A new ChatContext populated with the reconstructed messages.

handoffs

def handoffs(self) -> 'List[AgentHandoff]'

Return a shallow copy of the recorded handoffs.

returns

List[AgentHandoff]

A new list containing the current AgentHandoff records.

last_handoff

def last_handoff(self) -> 'Optional[AgentHandoff]'

Return the most recent handoff, if any.

returns

Optional[AgentHandoff]

The last AgentHandoff recorded, or None when there are none.

messages

def messages(self) -> 'List[ChatMessage]'

Return a shallow copy of the context’s messages.

returns

List[ChatMessage]

A new list containing the current ChatMessage objects.

to_context_messages

def to_context_messages(self) -> 'List[Dict[str, Any]]'

Serialize the context to transport message dictionaries. Each message is emitted with its role and content, and optionally its message_id, images, tool_calls, and tool_call_id when present.

returns

List[Dict[str, Any]]

A list of dictionaries representing the messages in the SDK’s transport format.

to_openai_messages

def to_openai_messages(self, *, reasoning_model: 'bool' = False) -> 'List[Dict[str, Any]]'

Convert the context to OpenAI chat message dictionaries.

reasoning_model

bool

default:"False"

Reserved flag for reasoning-model formatting (currently unused).

returns

List[Dict[str, Any]]

A list of {'role', 'content'} dictionaries, one per message.

truncate

def truncate(self, max_items: 'Optional[int]' = None, max_tokens: 'Optional[int]' = None) -> "'ChatContext'"

Return a new context trimmed to size limits. When max_items is given, only the most recent messages are kept. When max_tokens is given, the oldest messages are dropped (always retaining at least one) until the estimated token total fits, using the same two-tokens-per-word heuristic as estimated_tokens.

max_items

Optional[int]

Maximum number of messages to retain.

max_tokens

Optional[int]

Maximum estimated token budget to retain.

returns

'ChatContext'

A new ChatContext containing the retained messages.

turn_count

def turn_count(self) -> 'int'

Count the number of user turns in the conversation.

returns

int

The number of messages whose role is ChatRole.USER.

ChatMessage

A single message within a conversation.

Fields

role

ChatRole

default:"user"

Role of the message author.

content

str

default:""

The message text.

message_id

str

default:""

Unique identifier for the message.

tool_calls

List[FunctionCall]

default:"…"

Tool invocations requested by this message.

tool_call_id

str

default:""

Identifier of the tool call this message responds to.

images

List[ImageContent]

default:"…"

Image attachments associated with the message.

agent_id

str

default:""

Identifier of the agent that produced the message, if any.

ChatContent

Textual content part of a chat message.

Fields

type

str

default:"text"

Content type discriminator; 'text' for this part.

text

str

default:""

The text payload.

ChatRole

Roles a message can take in a chat conversation.

class ChatRole(Enum):
    SYSTEM = 'system'
    USER = 'user'
    ASSISTANT = 'assistant'
    DEVELOPER = 'developer'

FunctionCall

A tool invocation requested by the model.

Fields

name

str

default:""

Name of the tool to invoke.

arguments

str

default:""

Arguments for the call, encoded as a JSON string.

call_id

str

default:""

Identifier correlating this call with its output.

FunctionCallOutput

The result of executing a tool call.

Fields

name

str

default:""

Name of the tool that was invoked.

output

str

default:""

The tool’s output, encoded as a string.

call_id

str

default:""

Identifier correlating this output with its originating call.

is_error

bool

default:"False"

Whether the output represents an error.

ImageContent

Image content part of a chat message.

Fields

type

str

default:"image"

Content type discriminator; 'image' for this part.

url

str

default:""

The image location, either an http(s) URL or a data: URL embedding the image bytes.

detail

str

default:"auto"

Requested level of detail for the image (e.g. 'auto', 'low', 'high').

from_base64

def from_base64(base64_str: 'str', mime_type: 'str', detail: 'str' = 'auto') -> "'ImageContent'"

Create image content from a base64-encoded string.

base64_str

str

required

The base64-encoded image data.

mime_type

str

required

MIME type of the image (e.g. 'image/jpeg').

detail

str

default:"auto"

Requested level of detail for the image.

returns

'ImageContent'

An ImageContent with a data: URL built from the inputs.

from_bytes

def from_bytes(data: 'bytes', mime_type: 'str', detail: 'str' = 'auto') -> "'ImageContent'"

Create image content from raw image bytes.

data

bytes

required

The raw image bytes.

mime_type

str

required

MIME type of the image (e.g. 'image/jpeg').

detail

str

default:"auto"

Requested level of detail for the image.

returns

'ImageContent'

An ImageContent with a base64 data: URL built from the bytes.

from_data_url

def from_data_url(data_url: 'str', detail: 'str' = 'auto') -> "'ImageContent'"

Create image content from a data: URL.

data_url

str

required

A data: URL embedding the image bytes.

detail

str

default:"auto"

Requested level of detail for the image.

returns

'ImageContent'

An ImageContent referencing the given data URL.

from_file

def from_file(path: 'str', mime_type: 'Optional[str]' = None, detail: 'str' = 'auto') -> "'ImageContent'"

Create image content by reading an image file from disk.

path

str

required

Filesystem path to the image file.

mime_type

Optional[str]

MIME type of the image. When omitted, it is inferred from the file extension, defaulting to 'image/jpeg'.

detail

str

default:"auto"

Requested level of detail for the image.

returns

'ImageContent'

An ImageContent with a base64 data: URL of the file contents.

from_frame

def from_frame(frame: 'Any', detail: 'str' = 'auto') -> "'ImageContent'"

Create image content from a video or image frame. Accepts a mapping carrying raw data and an optional MIME type, or an arbitrary frame object that is coerced to JPEG bytes.

frame

Any

required

The source frame. May be a mapping with a data key (and optional mime_type/mimeType) or any object accepted by coerce_image_to_jpeg_bytes.

detail

str

default:"auto"

Requested level of detail for the image.

returns

'ImageContent'

An ImageContent with a base64 data: URL of the frame image.

from_url

def from_url(url: 'str', detail: 'str' = 'auto') -> "'ImageContent'"

Create image content from an http(s) URL.

url

str

required

The image URL.

detail

str

default:"auto"

Requested level of detail for the image.

returns

'ImageContent'

An ImageContent referencing the given URL.

ContextWindow

Configuration governing how conversation context is bounded and truncated.

Fields

max_tokens

int | None

Maximum number of tokens to retain in the context, or None for no token-based limit.

max_context_items

int | None

Maximum number of context items to retain, or None for no item-based limit.

keep_recent_turns

int

default:"3"

Number of most recent turns to always preserve.

max_tool_calls_per_turn

int

default:"10"

Maximum number of tool calls allowed within a single turn.

summary_llm

Any

Optional language model used to summarize older context when it is trimmed.

EncodeOptions

Options controlling how an image frame is encoded.

Fields

format

Literal['JPEG', 'PNG']

default:"JPEG"

Output image format, either 'JPEG' or 'PNG'.

resize_options

Optional[ResizeOptions]

default:"…"

Dimensions to resize the image to before encoding. When None, the image is encoded at its original size; defaults to the legacy default resolution.

quality

int

default:"75"

JPEG quality (0-100); ignored for PNG output.

ResizeOptions

Target dimensions for resizing an image.

Fields

width

int

required

Target width in pixels.

height

int

required

Target height in pixels.

encode

def encode(frame: 'Any', options: 'Optional[EncodeOptions]' = None) -> 'bytes'

Encode an image frame to JPEG or PNG bytes. Converts the frame to a Pillow image, optionally resizes it, and encodes it using the requested format. For JPEG output the image is converted to RGB when needed and saved with optimization enabled.

frame

Any

required

Source image. Accepted types include raw bytes-like data, a PIL.Image.Image, a numpy array, or an object exposing a to_image method (such as an av.VideoFrame).

options

Optional[EncodeOptions]

Encoding options controlling format, resize dimensions, and JPEG quality. Defaults to EncodeOptions() when not provided.

returns

bytes

The encoded image as bytes.

coerce_image_to_jpeg_bytes

def coerce_image_to_jpeg_bytes(frame: 'Any', *, options: 'Optional[EncodeOptions]' = None) -> 'bytes'

Return JPEG bytes for an image frame, passing through raw bytes. If the frame is already bytes-like it is returned unchanged; otherwise the frame is encoded using the provided options.

frame

Any

required

Source image. Accepted types include raw bytes-like data, a PIL.Image.Image, a numpy array, or an object exposing a to_image method (such as an av.VideoFrame).

options

Optional[EncodeOptions]

Encoding options used when the frame must be encoded. Defaults to DEFAULT_REALTIME_ENCODE_OPTIONS when not provided.

returns

bytes

The image as bytes, either passed through unchanged or freshly encoded.

DEFAULT_REALTIME_ENCODE_OPTIONS

DEFAULT_REALTIME_ENCODE_OPTIONS = EncodeOptions(format='JPEG', resize_options=ResizeOptions(width=1024, height=1024), quality=75)

​ChatContext

​Constructor

​add_attributed_message

​add_handoff

​add_message

​copy

​empty

​estimated_tokens

​from_context_messages

​handoffs

​last_handoff

​messages

​to_context_messages

​to_openai_messages

​truncate

​turn_count

​ChatMessage

​Fields

​ChatContent

​Fields

​ChatRole

​FunctionCall

​Fields

​FunctionCallOutput

​Fields

​ImageContent

​Fields

​from_base64

​from_bytes

​from_data_url

​from_file

​from_frame

​from_url

​ContextWindow

​Fields

​EncodeOptions

​Fields

​ResizeOptions

​Fields

​encode

​coerce_image_to_jpeg_bytes

​DEFAULT_REALTIME_ENCODE_OPTIONS

ChatContext

Constructor

add_attributed_message

add_handoff

add_message

copy

empty

estimated_tokens

from_context_messages

handoffs

last_handoff

messages

to_context_messages

to_openai_messages

truncate

turn_count

ChatMessage

Fields

ChatContent

Fields

ChatRole

FunctionCall

Fields

FunctionCallOutput

Fields

ImageContent

Fields

from_base64

from_bytes

from_data_url

from_file

from_frame

from_url

ContextWindow

Fields

EncodeOptions

Fields

ResizeOptions

Fields

encode

coerce_image_to_jpeg_bytes

DEFAULT_REALTIME_ENCODE_OPTIONS