> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Review Queue

> Turn failed live calls into test cases automatically, check each draft against its original call, and approve the ones worth keeping.

The review queue is where possible new items wait for a person to check them. Each one is called a draft. Approving a draft adds it to a dataset; rejecting it throws it out. Nothing in the queue runs in an experiment until it's approved, so a bad or duplicate case never slips into your tests.

Drafts arrive when:

* A judge fails on a live call, or a tool returns an error, and [auto-drafting](#auto-drafting) is on for the agent.
* Someone adds a session with **Send to review**. See [From a session](https://docs.zeroruntime.ai/agent-builder/datasets/items.md#from-a-session).
* Someone [imports](https://docs.zeroruntime.ai/agent-builder/datasets/items.md#import-jsonl) or [generates](https://docs.zeroruntime.ai/agent-builder/datasets/items.md#generate-items) items.

Open **Datasets** and select the **Review queue** tab. The number next to the tab is how many drafts are waiting.

## Auto-drafting

Auto-drafting turns failures on an agent's live calls into drafts for you, so you don't have to hunt for bad calls yourself.

Select **Auto-drafting settings** on the **Review queue** tab, and pick the agent. You can also set it up in the agent builder's **Evals** tab, or on the [Judges](https://docs.zeroruntime.ai/agent-builder/evaluations/judges.md) tab of **Evals**. Each change saves at once.

> [!NOTE]
>
> Auto-drafting works from graded live calls. Before you can switch it on, the agent needs at least one [judge](https://docs.zeroruntime.ai/agent-builder/evaluations/judges.md) on, and either [auto run on live calls](https://docs.zeroruntime.ai/agent-builder/evaluations/run-an-evaluation.md#grade-calls-automatically) switched on or evaluations run on its live calls.

| Setting | What it does |
| - | - |
| **Draft failed evals for review** | Drafts a live call when one of the judges below fails on it. |
| **Draft when these judges fail** | The judges that trigger a draft. Starts with every judge that's on for the agent. Pick at least one, or nothing is drafted. |
| **Also draft tool errors** | Also drafts calls where a tool returned an error, even if every judge passed. |
| Dataset | The dataset every draft is approved into, or **Pick a dataset in review** to choose one for each draft as you review it. |
| Daily cap | **Up to 5 drafts a day**, or 10, 20, 50, 100, or 500. Keeps the queue small. |

![Auto-drafting panel with an agent picked, the Auto draft to datasets card with the Review queue link and the Draft failed evals for review switch, and the note that duplicates are merged](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/auto-drafting-settings.png)

A failure seen again on the same call doesn't create a second draft. Similar sessions are merged into one draft, and the draft shows how many were merged.

## Re-score

Judges sometimes get it wrong. So before you review an auto-drafted item, its judges and assertions run again on what the agent actually said in the original call. This is the re-score, and it tells you whether the failure is real:

| Re-score | Meaning |
| - | - |
| **Confirmed failure** | The original output still fails, so this is a real regression case. |
| **Passing example** | The original call passes on a second look, so the agent probably did nothing wrong. Approve it only if you want a test that the agent keeps handling this call well; otherwise reject it. |
| **Re-scoring…** | The re-score is still running. The queue updates when it finishes. |
| **Re-score failed** | The re-score couldn't run. Review the call yourself before approving, or select **Re-score** to try again. |

The queue lists confirmed failures first, so the most useful drafts are at the top.

## The queue

Each draft shows its name, why it was drafted (for example *Failed: Goal Completion*, *Tool error: lookup\_order*, or *Flagged*), its kind, its re-score, its source, its target dataset, and when it was created.

Filter by **Source**, **Kind**, **Tags**, **Reason**, and **Target dataset**, or search. When new drafts arrive while you're on the page, select **Load latest** to see them.

![Review queue tab with the Auto-drafting settings button, the Source, Kind, Tags, Reason, and Target dataset filters, and three flagged session drafts with their kind, re-score, source, and target dataset](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/review-queue.png)

## Review a draft

Select a draft, or **Review** in its **⋮** menu, to open it. Use the arrows to move through the queue.

**1. Check the re-score**

See each judge and assertion's verdict on the original output, with its reason.

**2. Pick the target dataset**

Required. Drafts from auto-drafting start on the agent's target dataset, if one is set.

**3. Adjust the item**

Edit the input, tags, judges, and assertions like any [item](https://docs.zeroruntime.ai/agent-builder/datasets/items.md#expected-result). Drafts from tool errors have no judges, so add a judge or an assertion. STT and VAD drafts need a recording before you can approve them.

**4. Save or approve**

Select **Save & re-score** to keep your edits and run the re-score again with them; the draft stays in the queue. Select **Approve** to save your edits and add the item to the dataset.

An approved item is active and runs in the next experiment on its dataset.

![Draft panel for a flagged turn of a live call, with the re-score card and Open session link, the target dataset, tags, the caller's goal, the Nothing to score yet warning, the judges and assertions, and the Reject, Save & re-score, and Approve buttons](https://cdn.zeroruntime.ai/zrt/docs-resource/agent-builder/draft-review.png)

## Reject a draft

Select **Reject** in a draft's **⋮** menu, and pick a reason:

* **Not a real failure (judge too strict)**
* **Duplicate of an existing item**
* **Not worth testing**
* **Other**

Add a note if you want. Rejected drafts are archived with the reason, never deleted, and never run.

> [!TIP]
>
> A draft rejected as *judge too strict* is a sign to tighten that judge's pass and fail criteria. See [Judges](https://docs.zeroruntime.ai/agent-builder/evaluations/judges.md).

## Act on several drafts

* **Approve confirmed (N)** approves every confirmed failure on the page that has a target dataset.
* Tick drafts, then select **Approve N selected** or **Reject**.

Drafts without a target dataset can't be approved in bulk. They're listed after the others are approved; open each one and pick a target.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.