Skip to main content
Evaluations grade your agent’s calls. Judges check each call against one question, such as “Did the agent confirm the booking?”, and a scorecard turns their verdicts into a score. You can grade scenarios played by a simulated caller, a simulation run that already finished, or real calls your agent took, and see which judge failed and why. Open Evals in the sidebar. Evaluations score one agent at a time: pick the agent with the agent picker in the page header. The page has three tabs: Overview, Judges, and Runs.
Evals page with the agent picker, New judge and Run evaluation buttons, the Overview, Judges, and Runs tabs, the How evaluations work cards for Accuracy, Experience, and Diagnostic, and the Set up evaluations checklist with two of three steps done

​
Core concepts

​
Judge categories

Every judge has a category. The category decides whether the judge counts toward the score.

​
Verdicts

Each try also gets an overall status:

​
How a run is scored

A try passes a category when every judge in that category passes. A run’s Accuracy is the share of scenarios whose tries passed every accuracy judge, and Experience is the same for experience judges. When you grade live calls, each call counts as its own scenario. When each scenario is played more than once, use the toggle above the scores to choose how a scenario passes: Separately, the scorecard gives every graded call a score and a band. The run shows the average score, the median, and how many calls landed in each band. See Scorecards.

​
Before you start

  • Create an agent. Evaluations need at least one agent on your account.
  • Have something to grade: scenarios in Simulation, a finished simulation run, or live calls to the agent.
  • Have enough credits. AI and audio judges are billed as model usage for each call they read.

​
Set up evaluations

Until your first run finishes, the Overview tab shows a short checklist:
1

Add judges

Use the default judges, or write your own for what a good call looks like. See Judges.
2

Add scenarios

Scenarios come from Simulation. Write a few callers your agent must handle. See Scenarios. You can skip this step if you only want to grade a past simulation run or live calls.
3

Run your first evaluation

Select Run evaluation, pick what to grade and the judges, and start the run. See Run an evaluation.
After your first run, the Overview tab shows the latest scores, the judges that failed most, and your recent runs. See Review results.
Prefer the terminal? zeroruntime evals uses the same judges, scorecards, and runs. See Evaluations in the CLI.

Judges

Write the questions every call is graded on.

Scorecards

Weigh judges and set score bands.

Run an evaluation

Grade scenarios, a simulation run, or live calls.

Review results

See which judge failed, on which call, and why.