> ## Documentation Index > Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt > Use this file to discover all available pages before exploring further. # Judges > Write the questions every call is graded on, choose how each judge grades, and pick which judges run automatically. A judge asks one question about every call, like *"Was the booking made?"*, and answers pass or fail with a reason. The **Judges** tab lists every judge on your account with its source, category, how it grades, whether it auto runs on this agent, and its pass rate in the latest finished run. Filter the list by **Category**, **How it grades**, **Source**, and **Auto run**, or search by name and question. The number next to the **Judges** tab is how many judges are on [auto run](#auto-run) for this agent. ![Judges tab listing the ten default judges with their source, category, how they grade, auto run switch, and pass rate, with the Category, How it grades, Source, and Auto run filters and the search box](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/judges-tab.png) ## Default and custom judges | Source | What it is | | - | - | | **Default** | Built-in judges every account starts with. You can use and duplicate them, but not change or delete them. | | **Custom** | Judges your team wrote, duplicated, or started from a template. | ### Default judges | Judge | Category | How it grades | What it checks | | - | - | - | - | | **Goal Completion** | Accuracy | LLM judge | Did the agent achieve what the caller wanted? | | **Response Accuracy** | Accuracy | LLM judge | Was what the agent said and did correct, with no made-up facts? | | **Safety & Compliance** | Accuracy | LLM judge | Did the agent protect sensitive data, resist manipulation, and stay within policy? | | **Instruction Following** | Accuracy | LLM judge | Did the agent follow its configured instructions and policies? | | **Conversation Quality** | Experience | LLM judge | How well the conversation flowed. | | **Conversation Outcome** | Experience | LLM judge | How the call ended for the caller. | | **Vocal Tone** | Experience | Audio judge | How the agent's voice sounded. | | **Voice Quality** | Experience | Code check | How accurately and at what pace the agent spoke. | | **Responsiveness** | Diagnostic | Code check | How quickly the agent replied. | | **Speech Recognition** | Diagnostic | Code check | How accurately the caller's speech was transcribed. | Open a default judge to see its full criteria and evidence. ## How a judge grades | Type | What it does | Best for | | - | - | - | | **LLM judge** | Reads the transcript and the evidence you choose. Shows its reasoning for every verdict. | Meaning: faithful, polite, on-policy. | | **Audio judge** | Listens to the call recording. | How the call sounded: pronunciation, pauses, interruptions. | | **Code check** | Compares recorded call data, such as reply latency, to rules you set. Doesn't read the transcript. | Exact, measurable facts. | ## Create a judge Select **New judge** in the page header. The panel opens on **Choose a template**: * **Start from scratch** to write every field yourself. * Or pick a judge from the templates to start from a copy of it. Filter templates by category, or search. Your copy lists right below the judge it came from. ![Choose a template panel with the Start from scratch option, the All, Accuracy, Experience, and Diagnostic tabs, the template search, and default judge templates such as Goal Completion and Response Accuracy](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/judge-templates.png) Then fill in the form: | Field | Required | What to enter | | - | - | - | | **Name** | Yes | A short name, for example *Confirms the booking*. Up to 256 characters. | | **Category** | Yes | **Accuracy**, **Experience**, or **Diagnostic**. See [Judge categories](https://docs.zeroruntime.ai/agent-builder/evaluations/overview.md#judge-categories). | | **How it grades** | Yes | **LLM judge**, **Audio judge**, or **Code check**. Can't be changed after you create the judge. | | **Question** | No | The one thing the judge answers about every call, for example *Did the agent confirm the date and time before booking?* Up to 1000 characters. | ### LLM and audio judges ![New judge form for an LLM judge named Confirms the booking, with the Accuracy category, the question, pass and fail criteria, default verdict, the quoted turn and can't tell options, the evidence checkboxes, and Auto run turned on with the model credits notice](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/new-judge-llm.png) | Field | Required | What to enter | | - | - | - | | **Pass** | Yes | What a passing call looks like. Up to 4000 characters. | | **Fail** | Yes | What a failing call looks like. Up to 4000 characters. | | **Default verdict** | No | **None**, **Pass**, or **Fail**: the verdict to give when the evidence doesn't point either way. Defaults to **None**. | | **Fail needs a quoted turn** | No | The judge must quote the turn that failed. On by default. | | **Allow "can't tell"** | No | Lets the judge answer can't tell when its topic never came up. When off, the judge always picks pass or fail. On by default. | | **When it can't tell** | No | When the judge should answer can't tell, for example *The caller never asked for a booking.* | | **Evidence it can see** | Yes | What the judge reads before answering. Pick at least one. See [Evidence](#evidence). | | **Extra context** | No | Anything the judge should know, such as business hours, a price list, or policy notes. Up to 20,000 characters. | #### Evidence | Evidence | What the judge gets | | - | - | | **Transcript** | The conversation, turn by turn. | | **Instructions & system prompt** | The agent's [system instructions](https://docs.zeroruntime.ai/agent-builder/agent/system-instructions-and-greeting.md). | | **Tool calls & results** | Every tool the agent called and what came back. | | **Knowledge base** | The agent's [knowledge base](https://docs.zeroruntime.ai/agent-builder/knowledge-base.md). | | **Caller goal** | The simulated caller's scenario. Use it only for judges that grade simulated calls. | | **Session details** | The call's start time and duration. | | **Call metrics** | Measured numbers such as latency and interruptions. | New judges start with **Transcript**, **Instructions & system prompt**, and **Tool calls & results**. Audio judges always get the call recording too. ### Code checks A code check has **Rules** instead of pass and fail criteria. Each rule is a field, an operator (`<`, `≤`, `>`, `≥`, or **between**), and a number, for example *Reply latency p95 (ms) ≤ 1500*. The judge passes only when every rule holds. Add up to 20 rules with **Add rule**. The fields a rule can check are below. The dashboard shows each one by name, such as *Reply latency p95 (ms)*; the ID is what you use in the [CLI](https://docs.zeroruntime.ai/runtime-sdk/deployments/evaluations.md#metric-judge). ![New judge form for a code check named Replies without long gaps, with three rules: reply latency p95 at most 1500 ms, longest silence between 0 and 3 seconds, and interruptions at most 2](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/new-judge-code-check.png) | ID | What it measures | | - | - | | `e2e_p50`, `e2e_p95`, `e2e_max` | How long the agent took to reply after the caller stopped speaking: the median, the 95th percentile, and the slowest reply. | | `llm_ttft_p95` | How long the LLM took to start answering (time to first token), 95th percentile. | | `tts_ttfb_p95` | How long the voice took to start speaking (time to first byte), 95th percentile. | | `stt_latency_p95` | How long speech recognition took to return the caller's words, 95th percentile. | | `eou_p95` | How long the agent took to decide the caller had finished speaking (end of utterance), 95th percentile. | | `max_silence` | The longest gap with no one speaking. | | `unanswered_turns` | Caller turns the agent never replied to. | | `interruptions` | How often the caller and the agent spoke over each other. | | `user_turns`, `agent_turns` | How many turns the caller and the agent took. | | `duration` | How long the call lasted. | | `tool_calls` | How many tools the agent called. | | `speaking_rate` | How fast the agent spoke. | | `stt_wer`, `stt_entity_errors` | How accurately the caller's speech was transcribed: the word error rate, and errors in names, numbers, and other key details. | | `tts_wer`, `tts_entity_errors` | How accurately the agent's voice said its words: the word error rate, and errors in key details. | | `caller_sentiment_change` | How the caller's mood changed from the start of the call to the end. | Run `zeroruntime evals metrics` to list every field with its unit. Some fields only work on recorded calls; the list marks them *needs the recording*. ### Auto run and save At the bottom of the form, **Auto run on *agent name*** is on by default. It adds the new judge to the judges that grade this agent's calls automatically. Turn it off to use the judge only in runs you start yourself. Select **Create judge**. The judge opens, and you can pick it in any run. > [!TIP] > > Keep each judge to one question. A judge that checks several things fails without telling you which one went wrong. ## Auto run The **Auto run** column shows whether a judge grades this agent's calls automatically: | Status | Meaning | | - | - | | **Yes** | The judge is on and the agent's auto run is switched on. | | **Paused** | The judge is on, but the agent's auto run is off. It starts grading once you switch auto run on. | | **No** | The judge is off. It doesn't auto run, but you can still pick it in any run you start. | Use the switch in the row to turn a judge on or off. Which calls the agent auto runs on, and with which scorecard, is set in the agent builder. See [Grade calls automatically](https://docs.zeroruntime.ai/agent-builder/evaluations/run-an-evaluation.md#grade-calls-automatically). An agent's auto run needs at least one judge on, so you can't turn off the last one. Switch the agent's auto run off instead. ## View a judge Select a judge to open it. You see: ![Instruction Following judge page with the Duplicate and Edit buttons, the What it checks card with its question, pass, fail, and can't tell criteria and evidence, the Results card with a failed verdict and Open try, and the side card with the auto run switch, category, and source](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/judge-detail.png) * **What it checks**: its question, its pass, fail, and can't tell criteria (or its rules for a code check), its default verdict, and the evidence it can see. * **Results** from the latest finished run that used it: how many tries passed, the change since the previous comparable run, how many failed or couldn't tell, and its most recent verdicts with their quotes and reasons. Select **Open try** to see a call. * Its auto run switch, source, category, how it grades, and when it was last edited. * **Used in**: how many of this agent's runs it graded. ## Manage judges Open a judge's **⋮** menu: * **Open**: see the judge's details and results. * **Edit**: change any field except how it grades. Editing a default judge opens a copy; select **Save as my copy** to save it as a new custom judge. * **Duplicate**: make an editable copy, listed right below the original. Copies don't auto run until you turn them on. * **Turn on auto run** / **Turn off auto run**. * **Delete**: remove a custom judge. Default judges can't be deleted. This can't be undone. To act on several judges at once, tick their checkboxes, then select **Turn on auto run**, **Turn off auto run**, or **Delete**. Delete skips any default judges in the selection. > [!NOTE] > > Edits and deletes apply only to new runs. Every run keeps the version of each judge it was graded with, so past results stay as they are. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.