zeroruntime datasets and zeroruntime experiments do all of this from your terminal.
Datasets, items, drafts, and experiments are the same ones you see under Datasets in the dashboard. For what each one is and how items are checked, see the Datasets overview.
Quick start
Run these from your project folder, the one withzeroruntime.yaml:
experiments run tests the agent in agent.id of zeroruntime.yaml. Pass --agent to test another one, and --branch for an Agent Builder agent.
Create and manage datasets
- Dataset names are unique on your account.
--agentlinks a dataset to an agent, so you can filter by it withdatasets list --agent. Any agent can still run against any dataset.updatechanges only the options you pass.--tagreplaces all of the dataset’s tags.- Archived datasets are read-only and can’t run in experiments. List them with
datasets list --status archived.
Items
Items have ids that start withdsi_. Every item command takes the dataset id first.
Import items
Write items as a.jsonl file, one item per line. The format is the same as in the dashboard; see Import JSONL for every field.
items.jsonl
- By default, the items go to the review queue as drafts. Approve them there to add them to the dataset.
- With
--direct, they’re added to the dataset at once and skip review. Use this for items you’ve already checked, such as a file in your repo. - If any line has an error, nothing is imported. Fix the file, or pass
--skip-invalidto import the valid lines anyway.
input.audio.url) are kept only with --direct. For drafts, add the audio afterwards with datasets review update <draft> --audio clip.wav.
Export items
Change an item
--file. input and expected each replace the whole field, so include every part you want to keep:
change.json
Review drafts
Drafts are items waiting for a person to approve them. They come from imports, from auto-drafting of failed live calls, and from calls you flag. Draft ids start withdrf_.
approveadds each draft to its own dataset. Pass--datasetto send them all to one dataset, which you need for drafts that don’t have one yet.- A draft can’t be approved until it has at least one check. Add one with
review update --file. rejectasks first;-yskips the prompt. Rejected drafts are kept with the reason, andreview list --status rejectedshows them.
--reason filters by why the draft was made: eval_failed, tool_error, user_flagged, imported, or generated.
Flag a live call
Saw a call go wrong? Flag it, and it becomes a draft in the review queue:--turn, the draft covers the whole call. Find session ids with zeroruntime session.
Run an experiment
--detach to start it and return at once.
Common options:
Judges
Each item is scored by the judges and assertions set on it, so you don’t have to pick judges. Pass--judge to run only some of them, such as --judge goal_completion; items lose any judge you leave out.
Before it starts, the CLI warns you if some items have nothing left to check. They show as not checked in the results.
Iterations and passing
With--iterations above 1, each item runs several times. An item that passes every time gets Pass, and one that passes only some of the time gets Flaky. --pass-threshold sets the share of iterations an item must pass, from 0 to 1. See Verdicts.
Limits and cost
--max-costsets a spending limit in US dollars.--max-turnsand--max-duration(seconds) cap each conversation item.--concurrencyruns up to 10 items at once.--environmentadds background noise to audio runs, for example--environment noisy.
Run in CI
experiments run waits for the result and exits with a code your CI can check. By default, any failed item fails the job. With --fail-under, the job passes as long as the pass rate is at least that number, from 0 to 100. A regression against the baseline always fails the job.
Exit codes
Review results
experiments watch opens a live, full-screen view. Open an experiment to see its items, then an item to see each iteration’s checks and output as they come in. Use the arrow keys to move, Enter to open, Esc to go back, r to refresh, and q to quit.
experiments show prints the experiment’s summary and its items. --status lists every result with that status, with the reason it failed; --item lists every result of one item. Add --json for the raw response.
You can open every experiment in the dashboard too. See Review results.