Skip to main content
Select Run experiment on a dataset to open the Run experiment panel. You can also start one from a dataset’s ⋮ menu on the Datasets tab, or tick items on the Items tab and select Run N selected. Run experiment is disabled while the dataset has no items, or while it’s archived. Hover the button to see why. Nothing runs until you select Start experiment, and the panel shows exactly what will run before you do.
Run experiment panel with the name Refund prompt v3, the agent, Judges for this run, the Main branch, Text only mode, 3 iterations, the conversation kind ticked, the baseline set to the last comparable run, and the summary 3 conversation × 3 iterations → 9 sessions

Configure the run

1

Name

Optional. Shown as the page title and in the experiments list, for example Refund prompt v3. Defaults to Experiment #N.
2

Agent

The agent to test. Defaults to the agent last run on this dataset. A dataset doesn’t belong to one agent, so you can run any agent against it.
3

Deployment

The branch to test. It defaults to the agent’s default branch, and results show it as branch @ commit. An experiment tests what’s deployed on the branch, not your draft. See Deploy an agent.
4

Mode

The same modes as Simulation:
  • Text only: checks what the agent says. Items with tool mocks run here.
  • Audio: adds speech, voice, and background noise.
Audio is unavailable when some of the items have tool mocks or history from a session, because those run only as text.
5

Iterations

How many times each item runs, from 1 to 5. Defaults to 1.Agents don’t answer the same way every time, so one run can pass by luck. With more iterations, an item that passes every time gets Pass, and one that passes only some of the time gets Flaky, which points to a weak spot. See Verdicts.
6

Kinds and tags

Every kind in the dataset starts ticked; untick one to leave it out. Pick Tags to run only the items with those tags. When you run selected items, these filters don’t appear.
7

Baseline

The experiment to compare with. Items that passed there and fail now are regressions. Defaults to Last comparable run (same agent, judges and mode), or pick any completed experiment on this dataset.
Use 1 iteration in Text only mode for quick checks while you work. Before a release, use 3 iterations, and Audio mode if the items support it.

Judges

There are no judges to pick: each item is scored by its own judges and assertions. Under Judges for this run, the panel lists every judge the items use and how many items use each. If a judge an item uses was deleted, the panel says so. Those items aren’t scored by it; edit them to pick another judge.

What runs

Before you start, the bottom of the panel shows what will run:
  • Single-stage items (LLM, STT, TTS, turn detection, and VAD) run together in one quick batch that finishes first, for example 12 llm · 4 stt items → 1 component batch.
  • Conversation items run as simulated calls, one per iteration, for example 3 conversation × 3 iterations → 9 sessions (text only). These take longer.
How each kind runs: Archived items never run. If no active items match the kinds and tags you picked, the panel says what to change.

Start the experiment

Select Start experiment. The experiment opens, and results fill in as items finish. You can leave the page while it runs. See Review results.

Run again

Select Run again on a finished experiment, or in its ⋮ menu in the experiments list. The panel opens pre-filled with the same agent, branch, mode, iterations, kinds, and tags, with that experiment as the baseline. Change anything, then start. It runs against the dataset’s current items, so items added or edited since are included.
After you fix your agent, deploy the change, then Run again on the last experiment. The new experiment compares with it, so you see straight away what you fixed and what you broke.

Cancel an experiment

Select Cancel on a running experiment, or in its ⋮ menu in the experiments list. Items that already finished keep their results; the rest are marked Cancelled.

Cost

Conversation items are billed like simulated calls, once per iteration. Judges are billed as model usage for each item and iteration they score. Pipeline-stage items are billed for the model or provider call they replay. An experiment’s page shows what it cost.