Skip to main content
An item is one test case. It has three parts:
  • Kind: what part of the agent it tests, such as a full call or one LLM reply. See Item kinds.
  • Input: what to replay, such as the caller’s goal or the conversation so far.
  • Expected result: the checks that must pass, such as the agent called lookup_order.
The Items tab of a dataset lists every item with its kind, where it came from, what it’s checked by, its latest result, and when and by whom it was added. Filter the list by Kind, Tags, Source, and Added by, or search by name. Use Status to see Active or Archived items.
Items tab with the Kind, Tags, Source, and Status filters, the item search, and three conversation items showing their kind, Manual source, number of assertions, and a failed last result

Add items

Items you add yourself are active as soon as you save them, because you’re the reviewer. Imported, generated, and auto-drafted items wait in the review queue until someone approves them.
The best items come from real calls that went wrong. When you spot a bad call in Sessions, add it to a dataset straight away, so you’ll know if the same mistake ever comes back.
Add item menu open on a dataset page with the Add manually, Import JSONL, Generate items, and From a session options

Add manually

Select Add item › Add manually. Pick a Kind, fill in the input and the expected result, and select Add item. Until the item is complete, the footer lists what’s still missing.
Add item panel with the Conversation kind selected, the item name Refund for a damaged order, the Scenario card with the caller's goal, the Angry customer persona, and success and failure criteria, the Tool mocks card, and the Goal Completion and Instruction Following judges ticked under Expected result

From a session

Open a session in Sessions and select Add to dataset. The item is built from the call, and opens in the item form for you to check and edit before you save it.
1

Dataset

Pick a dataset, or choose + New dataset to create one on the spot.
2

How

  • Add now: you’re the reviewer, so the item is active the moment you save it.
  • Send to review: drafts it into the review queue for someone else to approve. Add a note on what went wrong.
3

Kind

Conversation, LLM, STT, Turn detection, or VAD. TTS items can’t be built from a session; add them by hand or import them.
4

Message history

Pick the turn to build the item from, for example Up to turn 6. The turns before it become the item’s history, and the panel on the right shows exactly what the item takes from the call.
What each kind takes from the call:
Add to dataset panel pre-filled from a session, with Add now selected, the Manual · LLM replies dataset, the LLM kind, message history up to turn 3, and the source turns from the call on the right
Judges that ran on the session are suggested as the item’s checks. For an STT item, the transcript from the call is filled in as the expected transcript; correct it if the call misheard the caller. If the dataset already has an item built from this call, a notice names it, and the button changes to Add anyway.

From a trace span

Open a session’s Traces tab to add one exact step of the call. Each turn, and each speech-to-text, LLM, turn detection, and VAD step inside it, has an Add to dataset button. The item’s kind and turn are filled in from the step you picked, so you go straight to reviewing it. Other steps, such as tool calls or knowledge base lookups, can’t become items. Text-to-speech steps can’t either yet; hover the disabled button to see why.
Session Traces tab with the Add to dataset button in the header and on each turn, VAD, STT, and turn detection span, and a tooltip on the TTS span saying TTS items can't be built from a session yet

From a simulation

Open a simulated conversation in Simulation and select Add to dataset. A whole simulated conversation becomes a Conversation item. Judges that failed on the conversation are filled in as the item’s checks. You can do the same from an experiment’s results: select the Add to dataset icon at the end of a conversation item’s row, or Add to dataset in its result. See Review results.

Import JSONL

Select Add item › Import JSONL and upload a .jsonl file of up to 10 MB, with one item per line. Select Download example file for a file to start from. Each line is one item, written as JSON on a single line. Here is one item spread over several lines so it’s easier to read:
In the file, the same item, and an LLM item after it, look like this:

Fields

The input fields for each kind:
Not sure of a judge’s ID? Import the items with assertions only, then add judges to each draft while you review it in the queue.
The file is checked before anything is added. You see how many items are valid and which lines have errors, with the reason for each. Lines with errors are skipped. Select Re-upload to try a fixed file, or Send N items to review to send the valid ones to the review queue.
Import items panel on the Validate step for refund-items.jsonl, with 2 valid items, 2 lines with errors and their reasons, and the Re-upload and Send 2 items to review buttons

Generate items

Select Add item › Generate items to have a model write items from your agent’s prompt. You need to pick an agent or paste a prompt, so the items fit what your agent does. Select Generate N items; it takes up to a minute. Then:
  1. Tick the items you want to keep.
  2. Select Edit on any you want to change.
  3. Select Send N items to review.
Not happy with the results? Select Regenerate to start over with a different description.
Generate items panel on the Preview step with generated LLM cases ticked, one showing its regex assertions and one open for editing, and the Regenerate and Send 3 items to review buttons

Input by kind

The fields you fill in depend on the item’s kind.

Conversation

The same fields as a Simulation scenario:
Items with tool mocks, or with history from a session, run only in Text only mode.

LLM

STT (speech-to-text)

TTS (text-to-speech)

Turn detection

VAD (voice activity detection)

Expected result

The expected result is how an item is scored. Every item needs at least one check, or it can’t be saved: An item passes an iteration when every judge and every assertion on it passes.

Judges that must pass

Tick the judges that must pass. Only Conversation and LLM items use judges.
  • LLM items are scored by LLM judges only.
  • Conversation items can use LLM judges and code checks. Audio judges only score items run in Audio mode.
Judges that don’t fit the kind are hidden; select Show all judges to see them. Judges suggested from the source call are listed below as one-click picks.

Expected output

What the stage should produce. For an STT item, this is the correct transcript of the clip, which Max WER is measured against. For an LLM item, it counts as a check on its own. TTS and turn detection items show the field too, but they still need an assertion.

Assertions

Assertions are precise, automatic checks. Use them for anything you can state exactly, like a tool call or a time limit, and use judges for things that need judgement, like whether the agent was polite. Each kind has its own assertions:

Tags

Tag items to run only some of them in an experiment, for example refund or edge-case. Tags are lowercase, with hyphens for spaces, up to 48 characters each.

View an item

Select an item to open it. You see its kind, source, status, and language, its input, its expected result, its tags, and its Recent results in past experiments. Select Open session or Open conversation to see the call it came from.
Item panel for Reschedule a dentist appointment with its kind, source, status, and language, the Scenario, the Max turns and Tool called assertions, its tags, a failed recent result in Experiment #13, and the Archive item, Copy to, and Edit buttons

Manage items

Open an item’s ⋮ menu:
  • Open item: see the item and its recent results.
  • Edit item: change anything except the kind. Edits apply from the next experiment; past experiments keep their copy.
  • Copy to another dataset: an item belongs to one dataset, so the copy gets a new ID and keeps its source.
  • Archive item: it stops running in new experiments. Past results that used it stay as they are.
  • Restore: make an archived item active again.
To act on several items at once, tick their checkboxes, then select Add tags, Copy to dataset, Archive, or Run N selected. The selection is kept across pages and filters.