Image Input

For a snapshot-style “look at this” interaction, capture the most recent frames with capture_frames() and pass them to session.reply() with a prompt. The frames and prompt go to the LLM together. This works in Cascade pipelines and is the simplest way to add vision.

Capture and send

# Capture the two most recent frames and ask the LLM to describe them
frames = agent.capture_frames(num_of_frames=2)
if frames:
    await session.reply(
        "Describe what you see in this frame in one sentence.",
        frames=frames,
    )

capture_frames() never raises: it returns an empty list when vision is not active or no frames have arrived yet, so no exception handling is needed.

Trigger a capture

A common pattern is to capture when the client sends a pub/sub message, for example, a “capture” button in your app:

# Inside the agent, e.g. in on_enter:
async def on_message(message):
    if isinstance(message, dict) and message.get("message") == "capture_frames":
        frames = self.capture_frames(num_of_frames=2)
        if frames:
            await self.session.reply(
                "Analyze this frame and describe what you see.", frames=frames
            )

await self.session.subscribe_pubsub(
    "CHAT", lambda m: asyncio.create_task(on_message(m))
)

You can also build ImageContent directly from PIL images, NumPy arrays, or av.VideoFrames for fully custom flows.

Encoding

Before frames reach the model they’re encoded with EncodeOptions: JPEG by default, resized (default 1024×1024) and compressed (quality=75). Raise these when you need higher fidelity (for example, to read fine text on camera) at the cost of more tokens and latency.

References

Python

Examples

Vision (Cascade)

Send images to a cascade agent.

Overview Video Input

Get started

Modalities

Configure Pipeline

Tools and Capabilities

Turn Detection and Interruptions

Telephony Agents

Context Management

Observability and Monitoring

Capture and send

Trigger a capture

Encoding

References

Examples

Vision (Cascade)

​Capture and send

​Trigger a capture

​Encoding

​References

​Examples

Vision (Cascade)

Capture and send

Trigger a capture

Encoding

References

Examples