> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> Manage test datasets from the CLI: import and export items as JSONL, approve drafts, run experiments against your agent, and fail CI when something regresses.

A dataset is a saved set of test cases, called items, that you run your agent against after every change. Running the agent against a dataset is called an experiment, and it tells you which items pass and which ones passed last time but fail now. `zeroruntime datasets` and `zeroruntime experiments` do all of this from your terminal.

Datasets, items, drafts, and experiments are the same ones you see under **Datasets** in the dashboard. For what each one is and how items are checked, see the [Datasets overview](https://docs.zeroruntime.ai/agent-builder/datasets/overview.md).

## Quick start

Run these from your project folder, the one with `zeroruntime.yaml`:

```bash theme={null}
zeroruntime datasets create 'Refund flows' --tag refund     # prints the new dataset id
zeroruntime datasets items import ds_123 items.jsonl --dry-run   # check the file
zeroruntime datasets items import ds_123 items.jsonl --direct    # add the items
zeroruntime experiments run ds_123                          # run them and wait for the result
```

`experiments run` tests the agent in `agent.id` of `zeroruntime.yaml`. Pass `--agent` to test another one, and `--branch` for an [Agent Builder](https://docs.zeroruntime.ai/agent-builder/introduction.md) agent.

## Create and manage datasets

```bash theme={null}
zeroruntime datasets list                                   # your datasets
zeroruntime datasets list --tag refund --search 'refund'
zeroruntime datasets show ds_123                            # one dataset and how many items it has
zeroruntime datasets create 'Refund flows' --description 'Callers asking for refunds' --tag refund,orders
zeroruntime datasets update ds_123 --name 'Refund flows v2'
zeroruntime datasets archive ds_123                         # read-only, hidden from the list
zeroruntime datasets restore ds_123
```

* Dataset names are unique on your account.
* `--agent` links a dataset to an agent, so you can filter by it with `datasets list --agent`. Any agent can still run against any dataset.
* `update` changes only the options you pass. `--tag` replaces all of the dataset's tags.
* Archived datasets are read-only and can't run in experiments. List them with `datasets list --status archived`.

## Items

Items have ids that start with `dsi_`. Every item command takes the dataset id first.

```bash theme={null}
zeroruntime datasets items list ds_123                      # first page of items
zeroruntime datasets items list ds_123 --kind llm --tag refund
zeroruntime datasets items show ds_123 dsi_456              # one item's input and checks
zeroruntime datasets items archive ds_123 dsi_1 dsi_2       # experiments skip archived items
zeroruntime datasets items restore ds_123 dsi_1
```

### Import items

Write items as a `.jsonl` file, one item per line. The format is the same as in the dashboard; see [Import JSONL](https://docs.zeroruntime.ai/agent-builder/datasets/items.md#import-jsonl) for every field.

```json items.jsonl theme={null}
{"kind":"conversation","name":"Refund for damaged item","tags":["refund"],"input":{"goal":"Get a refund for a blender that arrived broken.","behaviour":"Polite but firm, has the order number ready.","successCriteria":"The agent looks up the order before refunding.","failureCriteria":"The agent refunds without checking the order."},"expected":{"assertions":[{"type":"tool_called","toolName":"lookup_order"}]}}
{"kind":"llm","name":"Asks for order number first","input":{"messages":[{"role":"user","content":"I want a refund for my order."}]},"expected":{"assertions":[{"type":"rubric","rubric":"Asks for the order number."}]}}
```

```bash theme={null}
zeroruntime datasets items import ds_123 items.jsonl --dry-run   # only check the file
zeroruntime datasets items import ds_123 items.jsonl             # send the items to the review queue
zeroruntime datasets items import ds_123 items.jsonl --direct    # add them to the dataset straight away
```

The file is checked first, and each line with an error is listed with the reason. Then:

* **By default**, the items go to the [review queue](#review-drafts) as drafts. Approve them there to add them to the dataset.
* **With `--direct`**, they're added to the dataset at once and skip review. Use this for items you've already checked, such as a file in your repo.
* **If any line has an error, nothing is imported.** Fix the file, or pass `--skip-invalid` to import the valid lines anyway.

Audio links in the file (`input.audio.url`) are kept only with `--direct`. For drafts, add the audio afterwards with `datasets review update <draft> --audio clip.wav`.

### Export items

```bash theme={null}
zeroruntime datasets items export ds_123 --out items.jsonl
zeroruntime datasets items export ds_123 --kind stt --tag clean-audio > stt.jsonl
```

The file has only the fields import needs, so you can import it again, into the same dataset or another one.

> [!TIP]
>
> Keep your datasets in your repo: export each one to a `.jsonl` file, review changes to it like code, and import it with `--direct` to update the dataset.

### Change an item

```bash theme={null}
zeroruntime datasets items update ds_123 dsi_456 --name 'Refund window'
zeroruntime datasets items update ds_123 dsi_456 --tag policy,refund    # replaces all tags
zeroruntime datasets items update ds_123 dsi_456 --audio clip.wav       # new audio file
zeroruntime datasets items update ds_123 dsi_456 --file change.json
```

Only the options you pass are changed. To change an item's input or checks, put them in a JSON file and pass `--file`. `input` and `expected` each replace the whole field, so include every part you want to keep:

```json change.json theme={null}
{
  "expected": {
    "assertions": [
      { "type": "tool_called", "toolName": "lookup_order" },
      { "type": "max_turns", "value": 10 }
    ]
  }
}
```

An item's kind can't be changed. Edits apply from the next experiment; past experiments keep their copy.

## Review drafts

Drafts are items waiting for a person to approve them. They come from imports, from [auto-drafting](https://docs.zeroruntime.ai/agent-builder/datasets/review-queue.md#auto-drafting) of failed live calls, and from calls you flag. Draft ids start with `drf_`.

```bash theme={null}
zeroruntime datasets review list                            # pending drafts
zeroruntime datasets review list --dataset ds_123 --reason eval_failed
zeroruntime datasets review show drf_123                    # the draft, its reason, and its re-score
zeroruntime datasets review update drf_123 --dataset ds_456 # pick the dataset it goes to
zeroruntime datasets review update drf_123 --audio clip.wav # add audio before approving
zeroruntime datasets review approve drf_1 drf_2             # each draft becomes an item
zeroruntime datasets review reject drf_3 --reason 'Duplicate'
```

* `approve` adds each draft to its own dataset. Pass `--dataset` to send them all to one dataset, which you need for drafts that don't have one yet.
* A draft can't be approved until it has at least one check. Add one with `review update --file`.
* `reject` asks first; `-y` skips the prompt. Rejected drafts are kept with the reason, and `review list --status rejected` shows them.

`--reason` filters by why the draft was made: `eval_failed`, `tool_error`, `user_flagged`, `imported`, or `generated`.

### Flag a live call

Saw a call go wrong? Flag it, and it becomes a draft in the review queue:

```bash theme={null}
zeroruntime datasets review flag --session 6aeacdd96d4dbef3f7f9b3c7
zeroruntime datasets review flag --session 6aeacdd96d4dbef3f7f9b3c7 --turn 4 --note 'Quoted the wrong price' --dataset ds_123
```

Without `--turn`, the draft covers the whole call. Find session ids with [`zeroruntime session`](https://docs.zeroruntime.ai/runtime-sdk/deployments/cli-reference#session).

## Run an experiment

```bash theme={null}
zeroruntime experiments run ds_123
```

The CLI starts the experiment, shows its progress, and waits until it finishes. Then it prints the pass rate, the regressions against the baseline, and up to five items that failed, with the reason for each.

Press **Ctrl+C** while it runs to choose: stop the experiment (finished items keep their results), detach and let it keep going in the cloud, or keep watching. Pass `--detach` to start it and return at once.

Common options:

```bash theme={null}
zeroruntime experiments run ds_123 --agent ag_123 --branch br_123   # an Agent Builder agent's branch
zeroruntime experiments run ds_123 --tag refund --kind llm          # only some items
zeroruntime experiments run ds_123 --item dsi_1,dsi_2               # only these items
zeroruntime experiments run ds_123 --iterations 3 --mode audio      # each item 3 times, as audio calls
zeroruntime experiments run ds_123 --baseline exp_120 --name 'Refund prompt v3'
```

### Judges

Each item is scored by the judges and assertions set on it, so you don't have to pick judges. Pass `--judge` to run only some of them, such as `--judge goal_completion`; items lose any judge you leave out.

Before it starts, the CLI warns you if some items have nothing left to check. They show as not checked in the results.

### Iterations and passing

With `--iterations` above 1, each item runs several times. An item that passes every time gets **Pass**, and one that passes only some of the time gets **Flaky**. `--pass-threshold` sets the share of iterations an item must pass, from `0` to `1`. See [Verdicts](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md#verdicts).

### Limits and cost

* `--max-cost` sets a spending limit in US dollars.
* `--max-turns` and `--max-duration` (seconds) cap each conversation item.
* `--concurrency` runs up to 10 items at once.
* `--environment` adds background noise to audio runs, for example `--environment noisy`.

Conversation items are billed like [simulated calls](https://docs.zeroruntime.ai/agent-builder/simulation/run-a-simulation.md#cost), and judges as model usage. See [Cost](https://docs.zeroruntime.ai/agent-builder/datasets/run-an-experiment.md#cost).

## Run in CI

```bash theme={null}
zeroruntime experiments run ds_123 --branch br_123 --iterations 3 --fail-under 90
```

`experiments run` waits for the result and exits with a code your CI can check. By default, any failed item fails the job. With `--fail-under`, the job passes as long as the pass rate is at least that number, from `0` to `100`. A regression against the baseline always fails the job.

### Exit codes

| Code | Meaning |
| - | - |
| `0` | Passed. |
| `1` | An item failed, the pass rate was below `--fail-under`, or an item regressed. |
| `2` | Bad usage, or nothing was checked. |
| `3` | The experiment was cancelled, or the API failed. |

## Review results

```bash theme={null}
zeroruntime experiments watch                       # live view of your experiments
zeroruntime experiments list --dataset ds_123       # recent experiments
zeroruntime experiments show exp_123                # summary and every item's verdict
zeroruntime experiments show exp_123 --status failed   # why items failed
zeroruntime experiments show exp_123 --item dsi_456    # every result of one item
zeroruntime experiments show exp_123 --json
```

`experiments watch` opens a live, full-screen view. Open an experiment to see its items, then an item to see each iteration's checks and output as they come in. Use the arrow keys to move, **Enter** to open, **Esc** to go back, **r** to refresh, and **q** to quit.

`experiments show` prints the experiment's summary and its items. `--status` lists every result with that status, with the reason it failed; `--item` lists every result of one item. Add `--json` for the raw response.

You can open every experiment in the dashboard too. See [Review results](https://docs.zeroruntime.ai/agent-builder/datasets/review-results.md).

## Stop an experiment

```bash theme={null}
zeroruntime experiments stop exp_123
```

Items that already finished keep their results.

See the [CLI Reference](https://docs.zeroruntime.ai/runtime-sdk/deployments/cli-reference.md#datasets) for every command and flag.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.