Skip to main content
Vision lets your agent see. With vision enabled, it can pull frames from the participant’s video and send them to the model alongside a prompt, to describe a scene, read a document on camera, or inspect a product. It works in both Cascade and Realtime modes.
Turn on the video subscription per session with the vision option on serve() (applies to every caller) or on the room you invoke.

Image Input

Capture frames on demand and send them to the LLM.

Video Input

Continuous vision with a realtime model, from camera or screen share.

​
How it works

Frames are captured as av.VideoFrame objects and encoded to JPEG (resized and compressed) before they reach the model. Whether you snapshot a frame on demand or stream video to a realtime model, the agent reasons over the image together with the conversation.

​
What’s Next

Pipeline Modes

Compare Cascade and Realtime pipeline modes.

Modalities Overview

Browse speech, text, and avatar modalities.

​
References

​
Examples

Vision

Capture camera frames and describe them with a vision-capable LLM.