> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Grade your agent's calls from the CLI: keep judges and scorecards in your repo, grade past calls or simulation runs, grade new calls automatically, and gate CI on the result.

An evaluation grades calls your agent has already had. Judges check each call, and a scorecard turns their verdicts into a score. `zeroruntime evals` runs evaluations from your terminal, on live calls or on the conversations of a [simulation](https://docs.zeroruntime.ai/runtime-sdk/deployments/simulations.md) run.

Judges, scorecards, and runs are the same ones you see in the dashboard. See [Evaluations](https://docs.zeroruntime.ai/agent-builder/evaluations/overview.md) to work with them there.

## Quick start

Run these from your project folder, the one with `zeroruntime.yaml`:

```bash theme={null}
zeroruntime evals init                       # create evals/ with your judges and scorecards
zeroruntime evals validate                   # check the files for errors
zeroruntime evals run --since 24h --dry-run  # show the calls, judges, and estimated cost
zeroruntime evals run --since 24h            # grade the last day's calls and watch
```

`evals init` fills in `agent` from `agent.id` in `zeroruntime.yaml`. If the project isn't deployed yet, set `agent` in `evals/evals.yaml` or pass `--agent`.

## How a call is graded

Each judge answers one question about a call and gives a verdict:

| Verdict | Meaning |
| - | - |
| Pass | The call meets the judge's pass criteria. |
| Fail | The call meets the judge's fail criteria. |
| Can't tell | The judge's topic never came up in the call. The scorecard decides how it counts. |
| Error | The judge couldn't run, for example because the recording is missing. It doesn't count toward the score. |

There are three kinds of judge:

| Type | What it checks |
| - | - |
| `llm_judge` | Reads the transcript and the [evidence](#judge) you give it, such as the system prompt or tool calls. |
| `audio_judge` | Listens to the call recording, for things like tone and pronunciation. Needs the call to be recorded. |
| `metric` | Compares a measured number, such as response time, to a threshold. Runs no AI and costs nothing. Run `zeroruntime evals metrics` for the list. |

Every judge has a category:

* **Accuracy**: did the agent do the right thing, such as reaching the caller's goal or quoting the right price.
* **Experience**: how the call felt, such as tone, pace, and response time.
* **Diagnostic**: signals to look into, such as speech recognition errors. They're never scored; a failed diagnostic judge shows up as a flag.

Each graded call gets a status:

| Status | When |
| - | - |
| `passed` | No accuracy or experience judge failed. |
| `needs_review` | An accuracy or experience judge failed, or a judge errored. |
| `failed` | A required judge failed: `goal_completion`, `response_accuracy`, or `safety_compliance`. |
| `skipped` | The call matched a [skip rule](#skip-rules) and wasn't graded. |

A run's **accuracy** is the share of graded calls in which every accuracy judge passed, and **experience** is the same for experience judges.

## The `evals/` folder

Judges and scorecards live in your repo as YAML, so you can review and version them like code. The CLI reads `evals/` from the folder you run it in.

```
evals/
  evals.yaml                 the agent, judges, skip rules, scorecard, and gates
  judges/pricing.yaml        one judge per file
  scorecards/sales_qa.yaml   one scorecard per file
```

Each judge and scorecard file has an `id`, which is also its file name. Use lowercase letters, numbers, and `_`, up to 64 characters. `judges` and `scorecards` can't be used as ids.

### Settings

`evals/evals.yaml` sets what `evals run` grades and the bar a run must clear.

```yaml evals/evals.yaml theme={null}
agent: ag_zqn1az
judges: ["*"]
skip:
  min_duration_sec: 10
  min_turns: 2
  agent_never_spoke: true
  caller_never_spoke: true
  voicemail: false
scorecard: default
gates:
  accuracy: 0.95
  experience: 0.85
```

| Key | Description | Default |
| - | - | - |
| `agent` | Agent whose calls are graded. | `agent.id` in `zeroruntime.yaml` |
| `judges` | Judge ids or globs such as `lead_*`. `*` selects every judge. | `["*"]` |
| `skip` | Calls not to grade. See [Skip rules](#skip-rules). | see below |
| `scorecard` | Scorecard id that scores each call. | `default` |
| `gates.accuracy`, `gates.experience` | The lowest accuracy and experience rate a run must reach, from `0` to `1`. | none |
| `gates.score` | The lowest average score a run must reach. | none |

`evals init` copies `judges` and `scorecard` from the agent's [automatic grading](#grade-calls-automatically) settings when it has them. Quote values that start with `*` in YAML, such as `["*"]`.

### Judge

```yaml evals/judges/pricing.yaml theme={null}
id: pricing
name: Pricing accuracy
type: llm_judge
category: accuracy
source: custom
question: Did the agent quote the right price for the plan the caller asked about?
pass: The agent quotes the price from the price list for the plan the caller named.
fail: The agent quotes a price that isn't on the price list, or for a different plan.
fail_needs_quote: true
cant_tell_when: The caller never asks about price.
evidence:
  - transcript
  - system_prompt
  - knowledge
```

| Key | Required | Description |
| - | - | - |
| `name` | Yes | Display name, up to 256 characters. |
| `type` | Yes | `llm_judge`, `audio_judge`, or `metric`. Can't change once the judge is pushed. |
| `category` | Yes | `accuracy`, `experience`, or `diagnostic`. |
| `question` | No | The question the judge answers about the call. |
| `pass` | AI judges | When the call passes. |
| `fail` | AI judges | When the call fails. |
| `fail_needs_quote` | No | A fail must quote the turn that failed. Defaults to `false`. |
| `allow_cant_tell` | No | Let the judge answer can't tell. Defaults to `true`. |
| `cant_tell_when` | No | When the judge should answer can't tell. |
| `default_verdict` | No | `pass` or `fail`: the verdict to give when the evidence doesn't point either way. |
| `evidence` | No | What the judge can see besides the transcript: `transcript`, `system_prompt`, `tool_calls`, `knowledge`, `session`, `metrics`, `scenario`. |
| `context` | No | Extra reference text for the judge, such as a price list. |
| `levels`, `pass_at` | No | Grade on a scale instead of pass or fail. See below. |
| `conditions` | Metric judges | The thresholds a metric judge checks. See [Metric judge](#metric-judge). |
| `curve` | No | Score a metric judge on a curve. See [Metric judge](#metric-judge). |
| `source` | No | Written by the CLI: `standard`, `custom`, or `generated`. Don't edit it. |

`session` gives the judge the call's start time and duration, and `scenario` gives it the simulated caller's scenario. Use `scenario` only for judges that grade simulation runs.

To grade on a scale, give a judge 2 to 6 `levels`, each with a score from `0` to `1`. The judge picks a level, and the call passes when that level's score reaches `pass_at`, or the highest level's score if you leave `pass_at` out:

```yaml theme={null}
levels:
  - label: Excellent
    score: 1
  - label: Acceptable
    score: 0.6
  - label: Poor
    score: 0
pass_at: 0.6
```

### Metric judge

A metric judge passes when every condition it can measure holds, and answers can't tell when it can measure none of them. `op` is `<=`, `>=`, `<`, `>`, or `between` (with `max`):

```yaml evals/judges/fast_replies.yaml theme={null}
id: fast_replies
name: Fast replies
type: metric
category: experience
conditions:
  - metric: e2e_p95
    op: "<="
    value: 1500
  - metric: unanswered_turns
    op: "<="
    value: 0
curve:
  metric: e2e_p95
  points:
    - [800, 1]
    - [3000, 0]
```

With a `curve`, the judge's score follows the curve instead of being all or nothing: here a call scores `1` at 800 ms or less, `0` at 3000 ms or more, and in between in proportion. A judge has up to 20 conditions, and a curve 2 to 20 points.

Run `zeroruntime evals metrics` to list every metric with its unit. Metrics marked *needs the recording* only work on recorded calls.

### Scorecard

```yaml evals/scorecards/sales_qa.yaml theme={null}
id: sales_qa
name: Sales QA
out_of: 100
groups:
  accuracy:
    weight: 0.6
  experience:
    weight: 0.4
judges:
  goal_completion:
    weight: 2
  safety_compliance:
    weight: 1
    knockout: true
    cap: 40
other_judges: 1
not_applicable: reweight
bands:
  - min: 85
    label: Good
  - min: 60
    label: Needs review
  - min: 0
    label: Poor
```

| Key | Description | Default |
| - | - | - |
| `out_of` | The highest score, up to `1000`. | `100` |
| `groups` | How much accuracy and experience count, each from `0` to `1`. At least one must be above `0`. | `0.6` and `0.4` |
| `judges.<id>.weight` | How much a judge counts within its group, from `0` to `10`. `0` leaves it out. | `other_judges` |
| `judges.<id>.knockout`, `cap` | When this judge fails, the call scores at most `cap`. | off |
| `other_judges` | Weight of every judge not listed under `judges`, from `0` to `10`. | `1` |
| `not_applicable` | How a can't tell verdict counts: `reweight` (leave the judge out), `pass`, or `fail`. | `reweight` |
| `bands` | Labels for score ranges. Up to 10, and one must start at `0`. | none |
| `score` | A formula that replaces the weights, such as `min(judge.pricing, judge.goal_completion) * 100`. Uses numbers, `judge.<id>`, `+ - * / ( )`, `min`, and `max`. | none |

A call's score is worked out like this:

1. Each judge gives a score from `0` to `1`: `1` for a pass and `0` for a fail, or its level or curve score.
2. Each group's score is the weighted average of its judges' scores.
3. The call's score is the average of the group scores, weighted by `groups`, times `out_of`. A group with no graded judges is left out.
4. If a knockout judge failed, the score is capped at its `cap`. With more than one, the lowest cap applies.

Diagnostic judges are never part of the score. Every account has a built-in scorecard, `default`, with the values above and no judge weights. It can't be changed or deleted.

### Create files

```bash theme={null}
zeroruntime evals judges new pricing       # ask for each field, then write the file
zeroruntime evals judges new fast_replies --type metric
zeroruntime evals judges new pricing -y    # write a template and open it in $EDITOR
zeroruntime evals scorecards new sales_qa  # same, for a scorecard
zeroruntime evals metrics                  # the metrics a metric judge can check
```

`new` asks for each field and checks every answer as you type it. With `-y` it writes a template with empty values instead, which `evals validate` reports until you fill it in. Then run `evals validate` and `evals push`.

## Run an evaluation

Pick the calls to grade with one of `--session`, `--since` or `--from`, or `--simulation`:

```bash theme={null}
zeroruntime evals run --since 24h                  # calls from the last 24 hours
zeroruntime evals run --from 2026-10-01 --to 2026-10-07 --limit 100
zeroruntime evals run --session <sessionId> --session <sessionId>
zeroruntime evals run --simulation run_8b2d4e07f1  # the conversations of a simulation run
zeroruntime evals run --since 7d --judge pricing,'lead_*' --scorecard sales_qa
zeroruntime evals run --since 24h --dry-run        # show calls, judges, and cost; run nothing
```

Flags override `evals.yaml`. `evals run`:

1. Finds the calls, and shows how many will be graded, how many are skipped and why, the judges, the scorecard, and the estimated cost.
2. Asks you to confirm (`--yes` skips it).
3. Starts the run and prints each call's status, score, and the reason as it's graded.
4. Prints a summary with the gates from `evals.yaml`, the top failing judges, and exits with a [status code](#exit-codes).

Press **Ctrl-C** during a run to stop it, keep watching, or detach and leave it running. Stopping keeps the results of calls already graded.

`--judge` takes judge ids or globs, separated by spaces or commas, and `--category` keeps only judges of that category. A judge must be on the dashboard to be used, so push new judges first.

To grade calls to an agent you ran on your machine with `zeroruntime run`, pass the agent id it ran under (`ZERORUNTIME_AGENT_ID` in your `.env`) with `--agent`.

### Skip rules

Skip rules leave out calls that aren't worth grading. Skipped calls aren't charged.

| Rule | Skips a call when | Default |
| - | - | - |
| `min_duration_sec` | It's shorter than this many seconds. | `10` |
| `max_duration_min` | It's longer than this many minutes. | none |
| `min_turns` | It has fewer turns than this. | `2` |
| `agent_never_spoke` | The agent never spoke. | `true` |
| `caller_never_spoke` | The caller never spoke. | `true` |
| `voicemail` | It reached voicemail. Found while grading, so the dry run doesn't count these. | `false` |

Set them under `skip` in `evals.yaml`, or override one for a run with `--skip`:

```bash theme={null}
zeroruntime evals run --since 7d --skip min_turns=3 --skip voicemail=on
```

### Cost

AI and audio judges are charged for the tokens the judge model uses on each call. Metric judges, skipped calls, dry runs, and [rescoring](#stop-rescore-and-compare) cost nothing. The estimate is based on what this agent's recent graded calls cost, so the first run of an agent has no estimate.

## Grade calls automatically

Grade an agent's calls as they happen, without running anything:

```bash theme={null}
zeroruntime evals auto set --live-calls on --judge '*' --sample 0.25 --max-per-day 50
zeroruntime evals auto set --simulation-runs on --scorecard sales_qa
zeroruntime evals auto status  # the agent's current settings
zeroruntime evals auto set --live-calls off --simulation-runs off
```

| Flag | Description |
| - | - |
| `--live-calls on\|off` | Grade the agent's calls when they end. Each day's calls are grouped into one run. |
| `--simulation-runs on\|off` | Grade every simulation run of this agent when it finishes, in a run of its own. |
| `--judge` | Judge ids or globs to grade with. Replaces the current judges. |
| `--scorecard` | Scorecard that scores each call. |
| `--sample` | Share of live calls to grade, from `0` to `1`. `1` grades every call. Calls are picked at random. |
| `--max-per-day` | The most live calls to grade in a day. |

Flags you don't pass keep their current value. Skip rules use their defaults. Automatic runs show up with `zeroruntime evals runs list --trigger auto`.

> [!NOTE]
>
> Automatic grading needs an agent that exists on your account: deployed with `zeroruntime up`, built in [Agent Builder](https://docs.zeroruntime.ai/agent-builder/introduction.md), or self-hosted with a registered worker. To grade a local agent's calls, use `evals run`.

## Run in CI

```bash theme={null}
zeroruntime evals run --simulation "$SIM_RUN" --yes
zeroruntime evals diff evr_5c1e9a8b2d evr_7a3f0c6e1b --fail-on-drop 2
```

* `--yes` skips the confirmation prompt.
* `gates` in `evals.yaml` set the accuracy, experience, and score a run must reach.
* `evals diff --fail-on-drop` fails when accuracy, experience, or any judge's pass rate drops by more than that many percentage points.

To test a change end to end, run a [simulation](https://docs.zeroruntime.ai/runtime-sdk/deployments/simulations.md#run-in-ci), then grade its run with `evals run --simulation`.

### Exit codes

| Code | Meaning |
| - | - |
| `0` | The run completed and every gate was met. |
| `1` | A gate was missed: the accuracy or experience rate, or the average score. |
| `2` | The run couldn't start: invalid flags or `evals/` files, unknown judges or scorecard, no calls left after the skip rules, or nothing was graded. |
| `3` | The run was stopped, or an API error stopped the watch. |

`evals diff` exits `1` when a rate dropped past `--fail-on-drop`, and `0` otherwise.

## Review results

```bash theme={null}
zeroruntime evals runs list            # recent runs
zeroruntime evals runs list --agent ag_zqn1az --trigger auto
zeroruntime evals show evr_5c1e9a8b2d  # a run and its calls
zeroruntime evals show evr_5c1e9a8b2d --status failed --judge pricing
zeroruntime evals show <sessionId>     # a call's latest grade
zeroruntime evals show sims_2k8f1qd9xa --run evr_5c1e9a8b2d --transcript
```

`evals show` with a run id prints the run's summary and a row per call: its status, score, band, and the reason. The `A`, `X`, and `D` columns count the call's accuracy, experience, and diagnostic verdicts:

| Symbol | Verdict |
| - | - |
| `●` | Pass |
| `○` | Fail |
| `◐` | Can't tell |
| `×` | Error |
| `–` | Skipped |

With a call id, `evals show` prints the call's score and band, any knockout, and every judge's verdict with its reason and quote, grouped by category. Each scored judge shows the points it gained and lost. Add `--transcript` for the conversation and `--json` for the raw response. For a simulation conversation (`sims_…`), pass the run with `--run`.

## Stop, rescore, and compare

```bash theme={null}
zeroruntime evals stop evr_5c1e9a8b2d                                    # stop a running run
zeroruntime evals rescore evr_5c1e9a8b2d --scorecard sales_qa --dry-run  # print the new scores
zeroruntime evals rescore evr_5c1e9a8b2d --scorecard sales_qa            # save them
zeroruntime evals diff evr_5c1e9a8b2d evr_7a3f0c6e1b                     # compare two runs
```

Stopping keeps the results of calls already graded. `rescore` scores a finished run's verdicts with another scorecard without calling any judge again, so you can try weights and bands for free. `diff` compares two runs' accuracy, experience, each judge's pass rate, and average score.

Each run keeps a copy of the judges and scorecard it was graded with, so editing or deleting a judge later doesn't change past runs.

## Sync with the dashboard

`evals init` writes every judge and scorecard on your account, including the standard judges and the `default` scorecard. To sync after that:

```bash theme={null}
zeroruntime evals push                                # upload every new or changed file
zeroruntime evals push judges                         # only judges
zeroruntime evals push scorecards sales_qa            # only this scorecard
zeroruntime evals pull                                # pick judges or scorecards to write
zeroruntime evals pull judges 'lead_*'                # only these judges
zeroruntime evals pull '*' --overwrite                # everything, replacing changed files
zeroruntime evals judges list --category accuracy     # dashboard judges
zeroruntime evals scorecards list                     # dashboard scorecards
zeroruntime evals judges delete pricing fast_replies  # delete from the dashboard; files kept
```

Files are matched to dashboard items by `id`. `push` creates the ones that don't exist yet and updates the ones that changed. `pull` writes the ones you don't have and keeps files that differ from the dashboard unless you pass `--overwrite`.

Standard judges and the `default` scorecard are read-only: `push` skips them, and `delete` refuses them. To change one, copy its file to a new id. Deleting a scorecard that an agent grades with automatically moves that agent to `default`; `delete` tells you which agents first and asks before deleting.

See the [CLI Reference](https://docs.zeroruntime.ai/runtime-sdk/deployments/cli-reference.md#evals) for every command and flag.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.