> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run an Experiment

> Run any agent against a dataset, all of it or just some items, and compare the result with an earlier experiment.

Select **Run experiment** on a dataset to open the **Run experiment** panel. You can also start one from a dataset's **⋮** menu on the **Datasets** tab, or tick items on the **Items** tab and select **Run N selected**.

**Run experiment** is disabled while the dataset has no items, or while it's archived. Hover the button to see why.

Nothing runs until you select **Start experiment**, and the panel shows exactly what will run before you do.

![Run experiment panel with the name Refund prompt v3, the agent, Judges for this run, the Main branch, Text only mode, 3 iterations, the conversation kind ticked, the baseline set to the last comparable run, and the summary 3 conversation × 3 iterations → 9 sessions](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/run-experiment.png)

## Configure the run

**1. Name**

Optional. Shown as the page title and in the experiments list, for example *Refund prompt v3*. Defaults to *Experiment #N*.

**2. Agent**

The agent to test. Defaults to the agent last run on this dataset. A dataset doesn't belong to one agent, so you can run any agent against it.

**3. Deployment**

The branch to test. It defaults to the agent's default branch, and results show it as *branch @ commit*. An experiment tests what's deployed on the branch, not your draft. See [Deploy an agent](https://docs.zeroruntime.ai/agent-builder/deploy.md).

**4. Mode**

The same modes as [Simulation](https://docs.zeroruntime.ai/agent-builder/simulation/overview.md#modes):

* **Text only**: checks what the agent says. Items with tool mocks run here.
* **Audio**: adds speech, voice, and background noise.

**Audio** is unavailable when some of the items have tool mocks or history from a session, because those run only as text.

**5. Iterations**

How many times each item runs, from 1 to 5. Defaults to 1.

Agents don't answer the same way every time, so one run can pass by luck. With more iterations, an item that passes every time gets **Pass**, and one that passes only some of the time gets **Flaky**, which points to a weak spot. See [Verdicts](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md#verdicts).

**6. Kinds and tags**

Every kind in the dataset starts ticked; untick one to leave it out. Pick **Tags** to run only the items with those tags. When you run selected items, these filters don't appear.

**7. Baseline**

The experiment to compare with. Items that passed there and fail now are regressions. Defaults to **Last comparable run (same agent, judges and mode)**, or pick any completed experiment on this dataset.

> [!TIP]
>
> Use 1 iteration in **Text only** mode for quick checks while you work. Before a release, use 3 iterations, and **Audio** mode if the items support it.

## Judges

There are no judges to pick: each item is scored by its own judges and assertions. Under **Judges for this run**, the panel lists every judge the items use and how many items use each.

If a judge an item uses was deleted, the panel says so. Those items aren't scored by it; edit them to pick another judge.

## What runs

Before you start, the bottom of the panel shows what will run:

* Single-stage items (LLM, STT, TTS, turn detection, and VAD) run together in one quick batch that finishes first, for example *12 llm · 4 stt items → 1 component batch*.
* Conversation items run as simulated calls, one per iteration, for example *3 conversation × 3 iterations → 9 sessions (text only)*. These take longer.

How each kind runs:

| Kind | Runs as |
| - | - |
| **Conversation** | Re-simulated as a full call |
| **LLM** | Replayed as one LLM call |
| **STT** | Replayed through speech-to-text |
| **TTS** | Synthesised and timed |
| **Turn detection** | Replayed through turn detection |
| **VAD** | Replayed through voice activity detection |

Archived items never run. If no active items match the kinds and tags you picked, the panel says what to change.

## Start the experiment

Select **Start experiment**. The experiment opens, and results fill in as items finish. You can leave the page while it runs. See [Review results](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md).

## Run again

Select **Run again** on a finished experiment, or in its **⋮** menu in the experiments list. The panel opens pre-filled with the same agent, branch, mode, iterations, kinds, and tags, with that experiment as the baseline. Change anything, then start. It runs against the dataset's current items, so items added or edited since are included.

> [!TIP]
>
> After you fix your agent, deploy the change, then **Run again** on the last experiment. The new experiment compares with it, so you see straight away what you fixed and what you broke.

## Cancel an experiment

Select **Cancel** on a running experiment, or in its **⋮** menu in the experiments list. Items that already finished keep their results; the rest are marked **Cancelled**.

## Cost

Conversation items are billed like [simulated calls](https://docs.zeroruntime.ai/agent-builder/simulation/run-a-simulation.md#cost), once per iteration. Judges are billed as model usage for each item and iteration they score. Pipeline-stage items are billed for the model or provider call they replay. An experiment's page shows what it cost.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.