Skip to main content
A dataset is a saved set of test cases, called items, that you run your agent against after every change. Running the agent against a dataset is called an experiment, and it tells you which items pass and which ones passed last time but fail now. zeroruntime datasets and zeroruntime experiments do all of this from your terminal. Datasets, items, drafts, and experiments are the same ones you see under Datasets in the dashboard. For what each one is and how items are checked, see the Datasets overview.

Quick start

Run these from your project folder, the one with zeroruntime.yaml:
experiments run tests the agent in agent.id of zeroruntime.yaml. Pass --agent to test another one, and --branch for an Agent Builder agent.

Create and manage datasets

  • Dataset names are unique on your account.
  • --agent links a dataset to an agent, so you can filter by it with datasets list --agent. Any agent can still run against any dataset.
  • update changes only the options you pass. --tag replaces all of the dataset’s tags.
  • Archived datasets are read-only and can’t run in experiments. List them with datasets list --status archived.

Items

Items have ids that start with dsi_. Every item command takes the dataset id first.

Import items

Write items as a .jsonl file, one item per line. The format is the same as in the dashboard; see Import JSONL for every field.
items.jsonl
The file is checked first, and each line with an error is listed with the reason. Then:
  • By default, the items go to the review queue as drafts. Approve them there to add them to the dataset.
  • With --direct, they’re added to the dataset at once and skip review. Use this for items you’ve already checked, such as a file in your repo.
  • If any line has an error, nothing is imported. Fix the file, or pass --skip-invalid to import the valid lines anyway.
Audio links in the file (input.audio.url) are kept only with --direct. For drafts, add the audio afterwards with datasets review update <draft> --audio clip.wav.

Export items

The file has only the fields import needs, so you can import it again, into the same dataset or another one.
Keep your datasets in your repo: export each one to a .jsonl file, review changes to it like code, and import it with --direct to update the dataset.

Change an item

Only the options you pass are changed. To change an item’s input or checks, put them in a JSON file and pass --file. input and expected each replace the whole field, so include every part you want to keep:
change.json
An item’s kind can’t be changed. Edits apply from the next experiment; past experiments keep their copy.

Review drafts

Drafts are items waiting for a person to approve them. They come from imports, from auto-drafting of failed live calls, and from calls you flag. Draft ids start with drf_.
  • approve adds each draft to its own dataset. Pass --dataset to send them all to one dataset, which you need for drafts that don’t have one yet.
  • A draft can’t be approved until it has at least one check. Add one with review update --file.
  • reject asks first; -y skips the prompt. Rejected drafts are kept with the reason, and review list --status rejected shows them.
--reason filters by why the draft was made: eval_failed, tool_error, user_flagged, imported, or generated.

Flag a live call

Saw a call go wrong? Flag it, and it becomes a draft in the review queue:
Without --turn, the draft covers the whole call. Find session ids with zeroruntime session.

Run an experiment

The CLI starts the experiment, shows its progress, and waits until it finishes. Then it prints the pass rate, the regressions against the baseline, and up to five items that failed, with the reason for each. Press Ctrl+C while it runs to choose: stop the experiment (finished items keep their results), detach and let it keep going in the cloud, or keep watching. Pass --detach to start it and return at once. Common options:

Judges

Each item is scored by the judges and assertions set on it, so you don’t have to pick judges. Pass --judge to run only some of them, such as --judge goal_completion; items lose any judge you leave out. Before it starts, the CLI warns you if some items have nothing left to check. They show as not checked in the results.

Iterations and passing

With --iterations above 1, each item runs several times. An item that passes every time gets Pass, and one that passes only some of the time gets Flaky. --pass-threshold sets the share of iterations an item must pass, from 0 to 1. See Verdicts.

Limits and cost

  • --max-cost sets a spending limit in US dollars.
  • --max-turns and --max-duration (seconds) cap each conversation item.
  • --concurrency runs up to 10 items at once.
  • --environment adds background noise to audio runs, for example --environment noisy.
Conversation items are billed like simulated calls, and judges as model usage. See Cost.

Run in CI

experiments run waits for the result and exits with a code your CI can check. By default, any failed item fails the job. With --fail-under, the job passes as long as the pass rate is at least that number, from 0 to 100. A regression against the baseline always fails the job.

Exit codes

Review results

experiments watch opens a live, full-screen view. Open an experiment to see its items, then an item to see each iteration’s checks and output as they come in. Use the arrow keys to move, Enter to open, Esc to go back, r to refresh, and q to quit. experiments show prints the experiment’s summary and its items. --status lists every result with that status, with the reason it failed; --item lists every result of one item. Add --json for the raw response. You can open every experiment in the dashboard too. See Review results.

Stop an experiment

Items that already finished keep their results. See the CLI Reference for every command and flag.