> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> Keep a set of test cases from real calls, simulations, and your own writing, and run any agent against it to catch regressions.

A dataset is a saved set of test cases that you run your agent against after every change. It answers one question: *did my change break anything that used to work?*

Each test case, called an item, says what to replay and what must be true afterwards. For example: *a caller asks for a refund for a damaged order, and the agent must call `lookup_order` before refunding.* Running your agent against the whole dataset is called an experiment. It shows which items pass, and which ones passed last time but fail now.

Most items come from real problems. When a live call goes wrong, add it to a dataset so you notice if the same mistake ever comes back. You can also write items by hand, import them from a file, or have them generated from your agent's prompt.

Open **Datasets** in the sidebar. The page has three tabs: **Datasets**, **Review queue**, and **Experiments**.

![Datasets page with the New dataset button, the Datasets, Review queue, and Experiments tabs, the Tags and Sort filters, the Show archived switch, and six datasets with their tags, item counts by kind, and last experiment pass rates](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/datasets-list.png)

## Core concepts

| Concept | What it is |
| - | - |
| **Dataset** | A named set of items. A dataset doesn't belong to one agent, so any agent can run against any dataset. |
| **Item** | One test case: its kind, what to replay (the input), and what must pass (the expected result). See [Items](https://docs.zeroruntime.ai/agent-builder/datasets/items.md). |
| **Kind** | How an item runs: a full simulated **Conversation**, or one pipeline stage: **LLM**, **STT**, **TTS**, **Turn detection**, or **VAD**. |
| **Expected result** | The checks that decide whether an item passed: [judges](https://docs.zeroruntime.ai/agent-builder/evaluations/judges.md), assertions, or an expected output. Every item needs at least one. |
| **Assertion** | A precise, automatic check, such as *the agent called `lookup_order`* or *the call took at most 10 turns*. See [Assertions](https://docs.zeroruntime.ai/agent-builder/datasets/items.md#assertions). |
| **Draft** | An item waiting in the review queue. Failed calls, imports, and generated items arrive as drafts, and join a dataset only when someone approves them. See [Review queue](https://docs.zeroruntime.ai/agent-builder/datasets/review-queue.md). |
| **Experiment** | One run of an agent against a dataset. Each experiment has a number, for example `Experiment #4`, a pass rate, and a count of regressions. See [Run an experiment](https://docs.zeroruntime.ai/agent-builder/datasets/run-an-experiment.md). |
| **Iteration** | One run of one item in an experiment. Running each item more than once shows which passes are reliable. |
| **Baseline** | An earlier experiment to compare with. |
| **Regression** | An item that passed in the baseline and fails now: something your change broke. |

## Item kinds

Every item has a kind. A **Conversation** item tests the whole agent in a full simulated call. The other kinds test one stage of the agent's voice pipeline on its own, which is much faster.

| Kind | What it replays | Use it to test |
| - | - | - |
| **Conversation** | A full call. A simulated caller holds the conversation again in every experiment. | The whole agent: did it reach the right outcome and use the right tools? |
| **LLM** | One turn: the conversation up to that point, sent to the agent's LLM. Takes seconds. | What the agent says or which tool it calls at one moment in a call. |
| **STT** (speech-to-text) | An audio clip through speech recognition. | Whether the agent hears the caller correctly, such as names and numbers. |
| **TTS** (text-to-speech) | A piece of text through the agent's voice. | How fast and how well the agent's voice speaks. |
| **Turn detection** | The conversation so far. | Whether the agent knows when the caller has finished speaking. |
| **VAD** (voice activity detection) | An audio clip with known points where speech starts and ends. | Whether the agent detects speech at the right moments. |

> [!TIP]
>
> New to datasets? Start with **Conversation** items for the calls that matter most, and add **LLM** items when you want a fast check on one specific reply.

## When to use datasets

| Feature | Use it to |
| - | - |
| [Simulation](https://docs.zeroruntime.ai/agent-builder/simulation/overview.md) | Explore how the agent handles new situations, with scenarios and personas. |
| [Evaluations](https://docs.zeroruntime.ai/agent-builder/evaluations/overview.md) | Score calls, live or simulated, on accuracy and experience. |
| **Datasets** | Keep the cases you never want to break, and check after every change that they still pass. |

Datasets use the same [judges](https://docs.zeroruntime.ai/agent-builder/evaluations/judges.md) as Evaluations, and the same [personas](https://docs.zeroruntime.ai/agent-builder/simulation/personas.md) and modes as Simulation.

## Before you start

* **Create an agent.** Experiments run an agent against a dataset.
* **Deploy the branch you want to test.** An experiment tests a deployed branch, not your draft. See [Deploy an agent](https://docs.zeroruntime.ai/agent-builder/deploy.md).
* **Have at least one judge**, if you want items scored by judges. See [Judges](https://docs.zeroruntime.ai/agent-builder/evaluations/judges.md).
* **Have enough credits.** Conversation items are billed like simulated calls, and judges are billed as model usage.

## How it works

**1. Create a dataset**

Select **New dataset**, give it a name, and optionally a description and tags.

**2. Add items**

Add a failed call from a session, a trace span, or a simulated conversation, write an item by hand, import a JSONL file, or generate items. See [Items](https://docs.zeroruntime.ai/agent-builder/datasets/items.md).

**3. Review drafts**

Imported items, generated items, and failed live calls wait in the review queue. Approve the ones worth keeping. See [Review queue](https://docs.zeroruntime.ai/agent-builder/datasets/review-queue.md).

**4. Run an experiment**

Pick the agent and branch to test, and start the run. See [Run an experiment](https://docs.zeroruntime.ai/agent-builder/datasets/run-an-experiment.md).

**5. Review the results**

See which items passed, which regressed against the baseline, and why. See [Review results](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md).

## Create a dataset

Select **New dataset** in the page header and fill in:

| Field | Required | What to enter |
| - | - | - |
| **Name** | Yes | A short name, unique on your account, for example *Refund requests*. Up to 256 characters. |
| **Description** | No | What kind of cases the dataset holds. Up to 2,000 characters. |
| **Tags** | No | Labels to group and filter datasets. Lowercase, with hyphens for spaces, up to 48 characters each. Press Enter to add one. |

![New dataset panel with the name Refund requests, a description of refund callers, the refunds and support tags, and the Create dataset button](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/new-dataset.png)

Select **Create dataset**. The dataset opens, ready for items.

## The datasets list

The **Datasets** tab lists every dataset with its tags, how many items it has of each kind, its latest experiment's pass rate, when it was last updated, and who created it. Search by name, filter by **Tags** and **Created by**, and sort by **Last updated**, **Newest**, or **Name A–Z**.

Open a dataset's **⋮** menu to **Open dataset**, **Run experiment**, open its **Settings**, or **Archive dataset**.

## A dataset

Select a dataset to open it. The header shows its tags and the **Add item** and **Run experiment** buttons. Below that you see:

* **Items**: how many items it has, of each kind.
* **Last pass rate**: the result of the latest experiment.
* **Regressions**: items the latest experiment broke, compared with its baseline.
* **Pending review**: drafts waiting in the review queue for this dataset.
* **Updated**: when it last changed, and by whom.

![Dataset page with the Add item and Run experiment buttons, the Items, Last pass rate, Regressions, Pending review, and Updated details, the Items, Experiments, and Settings tabs, and three conversation items with their source, expected checks, and last result](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/dataset-detail.png)

The page has three tabs:

* **Items**: every item in the dataset. See [Items](https://docs.zeroruntime.ai/agent-builder/datasets/items.md).
* **Experiments**: every experiment run against it. See [Review results](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md).
* **Settings**: change the name, description, and tags, or archive the dataset.

## Archive a dataset

Select **Archive dataset** in the dataset's **⋮** menu or on its **Settings** tab.

* Its items become read-only and can't run in new experiments.
* Past experiments and their results stay available.
* It's hidden from the list. Turn on **Show archived** to find it.
* Nothing is deleted.

Select **Restore** on an archived dataset, or **Restore dataset** on its **Settings** tab, to make it active again.

> [!TIP]
>
> Prefer the terminal? `zeroruntime datasets` and `zeroruntime experiments` work with the same datasets, drafts, and experiments, and can fail a CI job when something regresses. See [Datasets in the CLI](https://docs.zeroruntime.ai/runtime-sdk/deployments/datasets.md).

## Limits

| Setting | Limit |
| - | - |
| Items per dataset | 1,000 |
| Iterations per item, per experiment | 1 to 5 |
| Items generated per batch | 20 |
| JSONL import file | 10 MB |
| Audio file per item | 20 MB |

> [card] [Items](https://docs.zeroruntime.ai/agent-builder/datasets/items.md)
>
> Add test cases and say what must pass.

> [card] [Review queue](https://docs.zeroruntime.ai/agent-builder/datasets/review-queue.md)
>
> Approve failed calls, imports, and generated items.

> [card] [Run an experiment](https://docs.zeroruntime.ai/agent-builder/datasets/run-an-experiment.md)
>
> Run any agent against a dataset.

> [card] [Review results](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md)
>
> Find regressions and see why an item failed.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.