Evaluations

An evaluation grades calls your agent has already had. Judges check each call, and a scorecard turns their verdicts into a score. zeroruntime evals runs evaluations from your terminal, on live calls or on the conversations of a simulation run. Judges, scorecards, and runs are the same ones you see in the dashboard. See Evaluations to work with them there.

Quick start

Run these from your project folder, the one with zeroruntime.yaml:

zeroruntime evals init                       # create evals/ with your judges and scorecards
zeroruntime evals validate                   # check the files for errors
zeroruntime evals run --since 24h --dry-run  # show the calls, judges, and estimated cost
zeroruntime evals run --since 24h            # grade the last day's calls and watch

evals init fills in agent from agent.id in zeroruntime.yaml. If the project isn’t deployed yet, set agent in evals/evals.yaml or pass --agent.

How a call is graded

Each judge answers one question about a call and gives a verdict:

Verdict	Meaning
Pass	The call meets the judge’s pass criteria.
Fail	The call meets the judge’s fail criteria.
Can’t tell	The judge’s topic never came up in the call. The scorecard decides how it counts.
Error	The judge couldn’t run, for example because the recording is missing. It doesn’t count toward the score.

There are three kinds of judge:

Type	What it checks
`llm_judge`	Reads the transcript and the evidence you give it, such as the system prompt or tool calls.
`audio_judge`	Listens to the call recording, for things like tone and pronunciation. Needs the call to be recorded.
`metric`	Compares a measured number, such as response time, to a threshold. Runs no AI and costs nothing. Run `zeroruntime evals metrics` for the list.

Every judge has a category:

Accuracy: did the agent do the right thing, such as reaching the caller’s goal or quoting the right price.
Experience: how the call felt, such as tone, pace, and response time.
Diagnostic: signals to look into, such as speech recognition errors. They’re never scored; a failed diagnostic judge shows up as a flag.

Each graded call gets a status:

Status	When
`passed`	No accuracy or experience judge failed.
`needs_review`	An accuracy or experience judge failed, or a judge errored.
`failed`	A required judge failed: `goal_completion`, `response_accuracy`, or `safety_compliance`.
`skipped`	The call matched a skip rule and wasn’t graded.

A run’s accuracy is the share of graded calls in which every accuracy judge passed, and experience is the same for experience judges.

The `evals/` folder

Judges and scorecards live in your repo as YAML, so you can review and version them like code. The CLI reads evals/ from the folder you run it in.

evals/
  evals.yaml                 the agent, judges, skip rules, scorecard, and gates
  judges/pricing.yaml        one judge per file
  scorecards/sales_qa.yaml   one scorecard per file

Each judge and scorecard file has an id, which is also its file name. Use lowercase letters, numbers, and _, up to 64 characters. judges and scorecards can’t be used as ids.

Settings

evals/evals.yaml sets what evals run grades and the bar a run must clear.

evals/evals.yaml

agent: ag_zqn1az
judges: ["*"]
skip:
  min_duration_sec: 10
  min_turns: 2
  agent_never_spoke: true
  caller_never_spoke: true
  voicemail: false
scorecard: default
gates:
  accuracy: 0.95
  experience: 0.85

Key	Description	Default
`agent`	Agent whose calls are graded.	`agent.id` in `zeroruntime.yaml`
`judges`	Judge ids or globs such as `lead_`. `` selects every judge.	`["*"]`
`skip`	Calls not to grade. See Skip rules.	see below
`scorecard`	Scorecard id that scores each call.	`default`
`gates.accuracy`, `gates.experience`	The lowest accuracy and experience rate a run must reach, from `0` to `1`.	none
`gates.score`	The lowest average score a run must reach.	none

evals init copies judges and scorecard from the agent’s automatic grading settings when it has them. Quote values that start with * in YAML, such as ["*"].

Judge

evals/judges/pricing.yaml

id: pricing
name: Pricing accuracy
type: llm_judge
category: accuracy
source: custom
question: Did the agent quote the right price for the plan the caller asked about?
pass: The agent quotes the price from the price list for the plan the caller named.
fail: The agent quotes a price that isn't on the price list, or for a different plan.
fail_needs_quote: true
cant_tell_when: The caller never asks about price.
evidence:
  - transcript
  - system_prompt
  - knowledge

Key	Required	Description
`name`	Yes	Display name, up to 256 characters.
`type`	Yes	`llm_judge`, `audio_judge`, or `metric`. Can’t change once the judge is pushed.
`category`	Yes	`accuracy`, `experience`, or `diagnostic`.
`question`	No	The question the judge answers about the call.
`pass`	AI judges	When the call passes.
`fail`	AI judges	When the call fails.
`fail_needs_quote`	No	A fail must quote the turn that failed. Defaults to `false`.
`allow_cant_tell`	No	Let the judge answer can’t tell. Defaults to `true`.
`cant_tell_when`	No	When the judge should answer can’t tell.
`default_verdict`	No	`pass` or `fail`: the verdict to give when the evidence doesn’t point either way.
`evidence`	No	What the judge can see besides the transcript: `transcript`, `system_prompt`, `tool_calls`, `knowledge`, `session`, `metrics`, `scenario`.
`context`	No	Extra reference text for the judge, such as a price list.
`levels`, `pass_at`	No	Grade on a scale instead of pass or fail. See below.
`conditions`	Metric judges	The thresholds a metric judge checks. See Metric judge.
`curve`	No	Score a metric judge on a curve. See Metric judge.
`source`	No	Written by the CLI: `standard`, `custom`, or `generated`. Don’t edit it.

session gives the judge the call’s start time and duration, and scenario gives it the simulated caller’s scenario. Use scenario only for judges that grade simulation runs. To grade on a scale, give a judge 2 to 6 levels, each with a score from 0 to 1. The judge picks a level, and the call passes when that level’s score reaches pass_at, or the highest level’s score if you leave pass_at out:

levels:
  - label: Excellent
    score: 1
  - label: Acceptable
    score: 0.6
  - label: Poor
    score: 0
pass_at: 0.6

Metric judge

A metric judge passes when every condition it can measure holds, and answers can’t tell when it can measure none of them. op is <=, >=, <, >, or between (with max):

evals/judges/fast_replies.yaml

id: fast_replies
name: Fast replies
type: metric
category: experience
conditions:
  - metric: e2e_p95
    op: "<="
    value: 1500
  - metric: unanswered_turns
    op: "<="
    value: 0
curve:
  metric: e2e_p95
  points:
    - [800, 1]
    - [3000, 0]

With a curve, the judge’s score follows the curve instead of being all or nothing: here a call scores 1 at 800 ms or less, 0 at 3000 ms or more, and in between in proportion. A judge has up to 20 conditions, and a curve 2 to 20 points. Run zeroruntime evals metrics to list every metric with its unit. Metrics marked needs the recording only work on recorded calls.

Scorecard

evals/scorecards/sales_qa.yaml

id: sales_qa
name: Sales QA
out_of: 100
groups:
  accuracy:
    weight: 0.6
  experience:
    weight: 0.4
judges:
  goal_completion:
    weight: 2
  safety_compliance:
    weight: 1
    knockout: true
    cap: 40
other_judges: 1
not_applicable: reweight
bands:
  - min: 85
    label: Good
  - min: 60
    label: Needs review
  - min: 0
    label: Poor

Key	Description	Default
`out_of`	The highest score, up to `1000`.	`100`
`groups`	How much accuracy and experience count, each from `0` to `1`. At least one must be above `0`.	`0.6` and `0.4`
`judges.<id>.weight`	How much a judge counts within its group, from `0` to `10`. `0` leaves it out.	`other_judges`
`judges.<id>.knockout`, `cap`	When this judge fails, the call scores at most `cap`.	off
`other_judges`	Weight of every judge not listed under `judges`, from `0` to `10`.	`1`
`not_applicable`	How a can’t tell verdict counts: `reweight` (leave the judge out), `pass`, or `fail`.	`reweight`
`bands`	Labels for score ranges. Up to 10, and one must start at `0`.	none
`score`	A formula that replaces the weights, such as `min(judge.pricing, judge.goal_completion) * 100`. Uses numbers, `judge.<id>`, `+ - * / ( )`, `min`, and `max`.	none

A call’s score is worked out like this:

Each judge gives a score from 0 to 1: 1 for a pass and 0 for a fail, or its level or curve score.
Each group’s score is the weighted average of its judges’ scores.
The call’s score is the average of the group scores, weighted by groups, times out_of. A group with no graded judges is left out.
If a knockout judge failed, the score is capped at its cap. With more than one, the lowest cap applies.

Diagnostic judges are never part of the score. Every account has a built-in scorecard, default, with the values above and no judge weights. It can’t be changed or deleted.

Create files

zeroruntime evals judges new pricing       # ask for each field, then write the file
zeroruntime evals judges new fast_replies --type metric
zeroruntime evals judges new pricing -y    # write a template and open it in $EDITOR
zeroruntime evals scorecards new sales_qa  # same, for a scorecard
zeroruntime evals metrics                  # the metrics a metric judge can check

new asks for each field and checks every answer as you type it. With -y it writes a template with empty values instead, which evals validate reports until you fill it in. Then run evals validate and evals push.

Run an evaluation

Pick the calls to grade with one of --session, --since or --from, or --simulation:

zeroruntime evals run --since 24h                  # calls from the last 24 hours
zeroruntime evals run --from 2026-10-01 --to 2026-10-07 --limit 100
zeroruntime evals run --session <sessionId> --session <sessionId>
zeroruntime evals run --simulation run_8b2d4e07f1  # the conversations of a simulation run
zeroruntime evals run --since 7d --judge pricing,'lead_*' --scorecard sales_qa
zeroruntime evals run --since 24h --dry-run        # show calls, judges, and cost; run nothing

Flags override evals.yaml. evals run:

Finds the calls, and shows how many will be graded, how many are skipped and why, the judges, the scorecard, and the estimated cost.
Asks you to confirm (--yes skips it).
Starts the run and prints each call’s status, score, and the reason as it’s graded.
Prints a summary with the gates from evals.yaml, the top failing judges, and exits with a status code.

Press Ctrl-C during a run to stop it, keep watching, or detach and leave it running. Stopping keeps the results of calls already graded. --judge takes judge ids or globs, separated by spaces or commas, and --category keeps only judges of that category. A judge must be on the dashboard to be used, so push new judges first. To grade calls to an agent you ran on your machine with zeroruntime run, pass the agent id it ran under (ZERORUNTIME_AGENT_ID in your .env) with --agent.

Skip rules

Skip rules leave out calls that aren’t worth grading. Skipped calls aren’t charged.

Rule	Skips a call when	Default
`min_duration_sec`	It’s shorter than this many seconds.	`10`
`max_duration_min`	It’s longer than this many minutes.	none
`min_turns`	It has fewer turns than this.	`2`
`agent_never_spoke`	The agent never spoke.	`true`
`caller_never_spoke`	The caller never spoke.	`true`
`voicemail`	It reached voicemail. Found while grading, so the dry run doesn’t count these.	`false`

Set them under skip in evals.yaml, or override one for a run with --skip:

zeroruntime evals run --since 7d --skip min_turns=3 --skip voicemail=on

Cost

AI and audio judges are charged for the tokens the judge model uses on each call. Metric judges, skipped calls, dry runs, and rescoring cost nothing. The estimate is based on what this agent’s recent graded calls cost, so the first run of an agent has no estimate.

Grade calls automatically

Grade an agent’s calls as they happen, without running anything:

zeroruntime evals auto set --live-calls on --judge '*' --sample 0.25 --max-per-day 50
zeroruntime evals auto set --simulation-runs on --scorecard sales_qa
zeroruntime evals auto status  # the agent's current settings
zeroruntime evals auto set --live-calls off --simulation-runs off

Flag	Description
`--live-calls on\|off`	Grade the agent’s calls when they end. Each day’s calls are grouped into one run.
`--simulation-runs on\|off`	Grade every simulation run of this agent when it finishes, in a run of its own.
`--judge`	Judge ids or globs to grade with. Replaces the current judges.
`--scorecard`	Scorecard that scores each call.
`--sample`	Share of live calls to grade, from `0` to `1`. `1` grades every call. Calls are picked at random.
`--max-per-day`	The most live calls to grade in a day.

Flags you don’t pass keep their current value. Skip rules use their defaults. Automatic runs show up with zeroruntime evals runs list --trigger auto.

Automatic grading needs an agent that exists on your account: deployed with zeroruntime up, built in Agent Builder, or self-hosted with a registered worker. To grade a local agent’s calls, use evals run.

Run in CI

zeroruntime evals run --simulation "$SIM_RUN" --yes
zeroruntime evals diff evr_5c1e9a8b2d evr_7a3f0c6e1b --fail-on-drop 2

--yes skips the confirmation prompt.
gates in evals.yaml set the accuracy, experience, and score a run must reach.
evals diff --fail-on-drop fails when accuracy, experience, or any judge’s pass rate drops by more than that many percentage points.

To test a change end to end, run a simulation, then grade its run with evals run --simulation.

Exit codes

Code	Meaning
`0`	The run completed and every gate was met.
`1`	A gate was missed: the accuracy or experience rate, or the average score.
`2`	The run couldn’t start: invalid flags or `evals/` files, unknown judges or scorecard, no calls left after the skip rules, or nothing was graded.
`3`	The run was stopped, or an API error stopped the watch.

evals diff exits 1 when a rate dropped past --fail-on-drop, and 0 otherwise.

Review results

zeroruntime evals runs list            # recent runs
zeroruntime evals runs list --agent ag_zqn1az --trigger auto
zeroruntime evals show evr_5c1e9a8b2d  # a run and its calls
zeroruntime evals show evr_5c1e9a8b2d --status failed --judge pricing
zeroruntime evals show <sessionId>     # a call's latest grade
zeroruntime evals show sims_2k8f1qd9xa --run evr_5c1e9a8b2d --transcript

evals show with a run id prints the run’s summary and a row per call: its status, score, band, and the reason. The A, X, and D columns count the call’s accuracy, experience, and diagnostic verdicts:

Symbol	Verdict
`●`	Pass
`○`	Fail
`◐`	Can’t tell
`×`	Error
`–`	Skipped

With a call id, evals show prints the call’s score and band, any knockout, and every judge’s verdict with its reason and quote, grouped by category. Each scored judge shows the points it gained and lost. Add --transcript for the conversation and --json for the raw response. For a simulation conversation (sims_…), pass the run with --run.

Stop, rescore, and compare

zeroruntime evals stop evr_5c1e9a8b2d                                    # stop a running run
zeroruntime evals rescore evr_5c1e9a8b2d --scorecard sales_qa --dry-run  # print the new scores
zeroruntime evals rescore evr_5c1e9a8b2d --scorecard sales_qa            # save them
zeroruntime evals diff evr_5c1e9a8b2d evr_7a3f0c6e1b                     # compare two runs

Stopping keeps the results of calls already graded. rescore scores a finished run’s verdicts with another scorecard without calling any judge again, so you can try weights and bands for free. diff compares two runs’ accuracy, experience, each judge’s pass rate, and average score. Each run keeps a copy of the judges and scorecard it was graded with, so editing or deleting a judge later doesn’t change past runs.

Sync with the dashboard

evals init writes every judge and scorecard on your account, including the standard judges and the default scorecard. To sync after that:

zeroruntime evals push                                # upload every new or changed file
zeroruntime evals push judges                         # only judges
zeroruntime evals push scorecards sales_qa            # only this scorecard
zeroruntime evals pull                                # pick judges or scorecards to write
zeroruntime evals pull judges 'lead_*'                # only these judges
zeroruntime evals pull '*' --overwrite                # everything, replacing changed files
zeroruntime evals judges list --category accuracy     # dashboard judges
zeroruntime evals scorecards list                     # dashboard scorecards
zeroruntime evals judges delete pricing fast_replies  # delete from the dashboard; files kept

Files are matched to dashboard items by id. push creates the ones that don’t exist yet and updates the ones that changed. pull writes the ones you don’t have and keeps files that differ from the dashboard unless you pass --overwrite. Standard judges and the default scorecard are read-only: push skips them, and delete refuses them. To change one, copy its file to a new id. Deleting a scorecard that an agent grades with automatically moves that agent to default; delete tells you which agents first and asks before deleting. See the CLI Reference for every command and flag.

Overview

Build

Plugins

Deploy

Quick start

How a call is graded

The `evals/` folder

Settings

Judge

Metric judge

Scorecard

Create files

Run an evaluation

Skip rules

Cost

Grade calls automatically

Run in CI

Exit codes

Review results

Stop, rescore, and compare

Sync with the dashboard

​Quick start

​How a call is graded

​The evals/ folder

​Settings

​Judge

​Metric judge

​Scorecard

​Create files

​Run an evaluation

​Skip rules

​Cost

​Grade calls automatically

​Run in CI

​Exit codes

​Review results

​Stop, rescore, and compare

​Sync with the dashboard

Quick start

How a call is graded

The `evals/` folder

Settings

Judge

Metric judge

Scorecard

Create files

Run an evaluation

Skip rules

Cost

Grade calls automatically

Run in CI

Exit codes

Review results

Stop, rescore, and compare

Sync with the dashboard