> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Items

> Add test cases from live sessions, trace spans, and simulated conversations, write them by hand, import or generate them, and say what each one must pass.

An item is one test case. It has three parts:

* **Kind**: what part of the agent it tests, such as a full call or one LLM reply. See [Item kinds](https://docs.zeroruntime.ai/agent-builder/datasets/overview.md#item-kinds).
* **Input**: what to replay, such as the caller's goal or the conversation so far.
* **Expected result**: the checks that must pass, such as *the agent called `lookup_order`*.

The **Items** tab of a dataset lists every item with its kind, where it came from, what it's checked by, its latest result, and when and by whom it was added.

Filter the list by **Kind**, **Tags**, **Source**, and **Added by**, or search by name. Use **Status** to see **Active** or **Archived** items.

![Items tab with the Kind, Tags, Source, and Status filters, the item search, and three conversation items showing their kind, Manual source, number of assertions, and a failed last result](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/dataset-items.png)

## Add items

| How | Where | Joins the dataset |
| - | - | - |
| **Add manually** | Dataset › **Add item** | Right away |
| **From a session** | A live session, or one of its trace spans | Right away, or as a draft |
| **From a simulation** | A simulated conversation, or an experiment result | Right away |
| **Import JSONL** | Dataset › **Add item** | As drafts, after review |
| **Generate items** | Dataset › **Add item** | As drafts, after review |
| **Auto-drafting** | A failed eval or tool error on a live call | As drafts, after review |

Items you add yourself are active as soon as you save them, because you're the reviewer. Imported, generated, and auto-drafted items wait in the [review queue](https://docs.zeroruntime.ai/agent-builder/datasets/review-queue.md) until someone approves them.

> [!TIP]
>
> The best items come from real calls that went wrong. When you spot a bad call in Sessions, add it to a dataset straight away, so you'll know if the same mistake ever comes back.

![Add item menu open on a dataset page with the Add manually, Import JSONL, Generate items, and From a session options](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/add-item-menu.png)

### Add manually

Select **Add item** › **Add manually**. Pick a **Kind**, fill in the [input](#input-by-kind) and the [expected result](#expected-result), and select **Add item**. Until the item is complete, the footer lists what's still missing.

![Add item panel with the Conversation kind selected, the item name Refund for a damaged order, the Scenario card with the caller's goal, the Angry customer persona, and success and failure criteria, the Tool mocks card, and the Goal Completion and Instruction Following judges ticked under Expected result](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/add-item-conversation.png)

### From a session

Open a session in [Sessions](https://docs.zeroruntime.ai/agent-builder/observability.md#review-sessions) and select **Add to dataset**. The item is built from the call, and opens in the item form for you to check and edit before you save it.

**1. Dataset**

Pick a dataset, or choose **+ New dataset** to create one on the spot.

**2. How**

* **Add now**: you're the reviewer, so the item is active the moment you save it.
* **Send to review**: drafts it into the review queue for someone else to approve. Add a note on what went wrong.

**3. Kind**

**Conversation**, **LLM**, **STT**, **Turn detection**, or **VAD**. TTS items can't be built from a session; add them by hand or import them.

**4. Message history**

Pick the turn to build the item from, for example *Up to turn 6*. The turns before it become the item's history, and the panel on the right shows exactly what the item takes from the call.

What each kind takes from the call:

![Add to dataset panel pre-filled from a session, with Add now selected, the Manual · LLM replies dataset, the LLM kind, message history up to turn 3, and the source turns from the call on the right](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/add-to-dataset-session.png)

| Kind | Built from |
| - | - |
| **Conversation** | A simulated caller re-runs the call from this turn, with the earlier turns as history. |
| **LLM** | The messages up to this turn, replayed against the agent's LLM with its current prompt and tools. |
| **STT** | The caller's audio for this turn, clipped from the call recording. |
| **Turn detection** | The messages up to this turn and the caller's audio, to check the turn ends here. |
| **VAD** | The caller's audio for this turn, with where speech starts and ends. |

Judges that ran on the session are suggested as the item's checks. For an STT item, the transcript from the call is filled in as the expected transcript; correct it if the call misheard the caller.

If the dataset already has an item built from this call, a notice names it, and the button changes to **Add anyway**.

#### From a trace span

Open a session's **Traces** tab to add one exact step of the call. Each turn, and each speech-to-text, LLM, turn detection, and VAD step inside it, has an **Add to dataset** button. The item's kind and turn are filled in from the step you picked, so you go straight to reviewing it.

Other steps, such as tool calls or knowledge base lookups, can't become items. Text-to-speech steps can't either yet; hover the disabled button to see why.

![Session Traces tab with the Add to dataset button in the header and on each turn, VAD, STT, and turn detection span, and a tooltip on the TTS span saying TTS items can't be built from a session yet](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/trace-span-add-to-dataset.png)

### From a simulation

Open a simulated conversation in [Simulation](https://docs.zeroruntime.ai/agent-builder/simulation/review-results.md#a-conversation) and select **Add to dataset**. A whole simulated conversation becomes a **Conversation** item. Judges that failed on the conversation are filled in as the item's checks.

You can do the same from an experiment's results: select the **Add to dataset** icon at the end of a conversation item's row, or **Add to dataset** in its result. See [Review results](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md#every-item-every-judge).

### Import JSONL

Select **Add item** › **Import JSONL** and upload a `.jsonl` file of up to 10 MB, with one item per line. Select **Download example file** for a file to start from.

Each line is one item, written as JSON on a single line. Here is one item spread over several lines so it's easier to read:

```json theme={null}
{
  "kind": "conversation",
  "name": "Refund for damaged item",
  "tags": ["refund"],
  "input": {
    "goal": "Get a refund for a blender that arrived broken.",
    "behaviour": "Polite but firm, has the order number ready.",
    "successCriteria": "The agent looks up the order before refunding.",
    "failureCriteria": "The agent refunds without checking the order."
  },
  "expected": {
    "assertions": [{ "type": "tool_called", "toolName": "lookup_order" }]
  }
}
```

In the file, the same item, and an LLM item after it, look like this:

```json theme={null}
{"kind":"conversation","name":"Refund for damaged item","tags":["refund"],"input":{"goal":"Get a refund for a blender that arrived broken.","behaviour":"Polite but firm, has the order number ready.","successCriteria":"The agent looks up the order before refunding.","failureCriteria":"The agent refunds without checking the order."},"expected":{"assertions":[{"type":"tool_called","toolName":"lookup_order"}]}}
{"kind":"llm","name":"Asks for order number first","input":{"messages":[{"role":"user","content":"I want a refund for my order."}]},"expected":{"assertions":[{"type":"rubric","rubric":"Asks for the order number."}]}}
```

#### Fields

| Field | Required | What it holds |
| - | - | - |
| `kind` | Yes | `conversation`, `llm`, `stt`, `tts`, `turn_detection`, or `vad`. |
| `name` | Yes | The item's name. |
| `tags` | No | A list of tags. |
| `input` | Yes | What to replay. The fields depend on the kind; see below. |
| `expected.assertions` | One of these | A list of assertions, each with a `type` and its fields, such as `{"type": "max_turns", "value": 10}`. See [Assertions](#assertions). |
| `expected.evaluationIds` | One of these | The IDs of judges that must pass. |
| `expected.output` | One of these | The expected output, for LLM and STT items. |

The `input` fields for each kind:

| Kind | `input` fields |
| - | - |
| `conversation` | `goal`, `successCriteria`, `failureCriteria`, and either `personaId` or `behaviour`. Optional: `openingLine`, `language`, `variables`. |
| `llm` | `messages`. Optional: `systemPrompt` as `{"text": "..."}`. |
| `stt` | `audio` as `{"url": "..."}`. Optional: `language`. |
| `tts` | `text`. |
| `turn_detection` | `messages`. Optional: `audio`. |
| `vad` | `audio`. Optional: `speechStartMs`, `speechEndMs`. |

> [!TIP]
>
> Not sure of a judge's ID? Import the items with assertions only, then add judges to each draft while you review it in the queue.

The file is checked before anything is added. You see how many items are valid and which lines have errors, with the reason for each. Lines with errors are skipped. Select **Re-upload** to try a fixed file, or **Send N items to review** to send the valid ones to the [review queue](https://docs.zeroruntime.ai/agent-builder/datasets/review-queue.md).

![Import items panel on the Validate step for refund-items.jsonl, with 2 valid items, 2 lines with errors and their reasons, and the Re-upload and Send 2 items to review buttons](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/import-items.png)

### Generate items

Select **Add item** › **Generate items** to have a model write items from your agent's prompt.

| Field | What to enter |
| - | - |
| **Describe the set** | The cases you need, for example *Callers asking for refunds on damaged or late orders, some refusing to give an order number.* Up to 500 characters. |
| **Kind** | **conversation**, **llm**, **tts**, or **turn\_detection**. STT and VAD items need real audio, so they can't be generated. |
| **How many** | 1 to 20 per batch. |
| **Agent** | The agent whose saved prompt the items are written from. Defaults to the agent last run on this dataset. |
| **Agent prompt** | Optional. Paste a prompt to generate from it instead of the agent's saved one. |

You need to pick an agent or paste a prompt, so the items fit what your agent does. Select **Generate N items**; it takes up to a minute. Then:

1. Tick the items you want to keep.
2. Select **Edit** on any you want to change.
3. Select **Send N items to review**.

Not happy with the results? Select **Regenerate** to start over with a different description.

![Generate items panel on the Preview step with generated LLM cases ticked, one showing its regex assertions and one open for editing, and the Regenerate and Send 3 items to review buttons](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/generate-items.png)

## Input by kind

The fields you fill in depend on the item's kind.

### Conversation

The same fields as a [Simulation scenario](https://docs.zeroruntime.ai/agent-builder/simulation/scenarios.md):

| Field | Required | What to enter |
| - | - | - |
| **Language** | Yes | The language the caller speaks. Defaults to **Agent's language**. |
| **What does the caller want?** | Yes | The caller's goal, for example *dispute an \$840 charge they don't recognise*. Up to 1,000 characters. |
| **Caller's first line** | No | Leave empty and the caller opens in their own words. |
| **Customer behaviour** | Yes | **Use a persona** and pick a [persona](https://docs.zeroruntime.ai/agent-builder/simulation/personas.md), or **Describe the behaviour** in your own words. |
| **Success when** | Yes | What the agent must do, for example *The agent books the refill*. |
| **Failure when** | Yes | What the agent must not do, for example *The agent gives dosage advice*. |
| **Caller metadata (variables)** | No | Key and value pairs the agent receives as [variables](https://docs.zeroruntime.ai/agent-builder/agent/variables.md), for example `loan_amount` = `500000`. |
| **Tool mocks** | No | A tool name and the response to return instead of calling the tool. A tool without a mock calls its live endpoint. |

> [!NOTE]
>
> Items with tool mocks, or with history from a session, run only in **Text only** mode.

### LLM

| Field | Required | What to enter |
| - | - | - |
| **Message history up to this turn** | Yes | A JSON array of messages with role `system`, `user`, `assistant`, or `tool`. |
| **System prompt** | No | Leave empty to use the agent's prompt at run time. |

```json theme={null}
[
  { "role": "user", "content": "I need a refill on RX-44810" },
  { "role": "assistant", "content": "Sure — which pharmacy?" },
  { "role": "user", "content": "The one on 5th street" }
]
```

### STT (speech-to-text)

| Field | Required | What to enter |
| - | - | - |
| **Language** | Yes | The language spoken in the clip. |
| **Audio** | Yes | An audio URL, or a file: wav, mp3, m4a, ogg, webm, or flac, up to 20 MB. |

### TTS (text-to-speech)

| Field | Required | What to enter |
| - | - | - |
| **Text to synthesise** | Yes | What the agent's voice should say. Up to 5,000 characters. |

### Turn detection

| Field | Required | What to enter |
| - | - | - |
| **Messages so far** | Yes | A JSON array of messages with role `user` or `assistant`. The last message must be the caller's. |
| **Audio** | No | The caller's audio for the last turn. |

### VAD (voice activity detection)

| Field | Required | What to enter |
| - | - | - |
| **Audio** | Yes | An audio URL, or a file up to 20 MB. |
| **Speech starts (ms)** | No | Where speech starts in the clip, in milliseconds. Needed for **Max speech-start error**. |
| **Speech ends (ms)** | No | Where speech ends in the clip, in milliseconds. Needed for **Max speech-end error**. |

## Expected result

The expected result is how an item is scored. Every item needs at least one check, or it can't be saved:

| Kind | Needs at least one of |
| - | - |
| **Conversation** | A judge or an assertion |
| **LLM** | A judge, an assertion, or an expected output |
| **STT** | An assertion, or an expected transcript |
| **TTS**, **Turn detection**, **VAD** | An assertion |

An item passes an iteration when every judge and every assertion on it passes.

### Judges that must pass

Tick the [judges](https://docs.zeroruntime.ai/agent-builder/evaluations/judges.md) that must pass. Only Conversation and LLM items use judges.

* **LLM** items are scored by LLM judges only.
* **Conversation** items can use LLM judges and code checks. Audio judges only score items run in **Audio** mode.

Judges that don't fit the kind are hidden; select **Show all judges** to see them. Judges suggested from the source call are listed below as one-click picks.

### Expected output

What the stage should produce. For an **STT** item, this is the correct transcript of the clip, which **Max WER** is measured against. For an **LLM** item, it counts as a check on its own. TTS and turn detection items show the field too, but they still need an assertion.

### Assertions

Assertions are precise, automatic checks. Use them for anything you can state exactly, like a tool call or a time limit, and use judges for things that need judgement, like whether the agent was polite. Each kind has its own assertions:

| Kind | Assertion | Passes when |
| - | - | - |
| **Conversation** | **Tool called** | The agent called the tool. |
| | **Tool called with args** | The agent called the tool with these arguments (JSON). |
| | **Tool not called** | The agent never called the tool. |
| | **Ended with tool** | The last tool the agent called is this one. |
| | **Max turns** | The call took at most this many turns. |
| **LLM** | **Calls tool** | The reply calls the tool, optionally with these arguments. |
| | **Matches regex** / **Does not match regex** | The reply does or doesn't match the regular expression. |
| | **Rubric** | A judge finds the reply meets the rubric you write. |
| **STT** | **Max WER** | The word error rate against the expected transcript is at most this value, from 0 to 1. For example, `0.1` allows one wrong word in ten. |
| | **Contains words** | The transcript contains every word in the list. |
| | **Matches regex** / **Does not match regex** | The transcript does or doesn't match. |
| **TTS** | **Max duration** | The audio is at most this long, in ms. |
| | **Max time to first audio** | The voice started within this many ms. |
| | **Rubric** | A judge finds the audio meets the rubric. |
| **Turn detection** | **End of turn** | The turn ends here, or doesn't, as you set. |
| **VAD** | **Speech detected** | Speech is detected, or isn't, as you set. |
| | **Max speech-start error** / **Max speech-end error** | The detected start or end is within this many ms of the one you set. |

## Tags

Tag items to run only some of them in an experiment, for example `refund` or `edge-case`. Tags are lowercase, with hyphens for spaces, up to 48 characters each.

## View an item

Select an item to open it. You see its kind, source, status, and language, its input, its expected result, its tags, and its **Recent results** in past experiments. Select **Open session** or **Open conversation** to see the call it came from.

![Item panel for Reschedule a dentist appointment with its kind, source, status, and language, the Scenario, the Max turns and Tool called assertions, its tags, a failed recent result in Experiment #13, and the Archive item, Copy to, and Edit buttons](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/item-detail.png)

## Manage items

Open an item's **⋮** menu:

* **Open item**: see the item and its recent results.
* **Edit item**: change anything except the kind. Edits apply from the next experiment; past experiments keep their copy.
* **Copy to another dataset**: an item belongs to one dataset, so the copy gets a new ID and keeps its source.
* **Archive item**: it stops running in new experiments. Past results that used it stay as they are.
* **Restore**: make an archived item active again.

To act on several items at once, tick their checkboxes, then select **Add tags**, **Copy to dataset**, **Archive**, or **Run N selected**. The selection is kept across pages and filters.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.