Skip to main content
The review queue is where possible new items wait for a person to check them. Each one is called a draft. Approving a draft adds it to a dataset; rejecting it throws it out. Nothing in the queue runs in an experiment until it’s approved, so a bad or duplicate case never slips into your tests. Drafts arrive when: Open Datasets and select the Review queue tab. The number next to the tab is how many drafts are waiting.

Auto-drafting

Auto-drafting turns failures on an agent’s live calls into drafts for you, so you don’t have to hunt for bad calls yourself. Select Auto-drafting settings on the Review queue tab, and pick the agent. You can also set it up in the agent builder’s Evals tab, or on the Judges tab of Evals. Each change saves at once.
Auto-drafting works from graded live calls. Before you can switch it on, the agent needs at least one judge on, and either auto run on live calls switched on or evaluations run on its live calls.
Auto-drafting panel with an agent picked, the Auto draft to datasets card with the Review queue link and the Draft failed evals for review switch, and the note that duplicates are merged
A failure seen again on the same call doesn’t create a second draft. Similar sessions are merged into one draft, and the draft shows how many were merged.

Re-score

Judges sometimes get it wrong. So before you review an auto-drafted item, its judges and assertions run again on what the agent actually said in the original call. This is the re-score, and it tells you whether the failure is real: The queue lists confirmed failures first, so the most useful drafts are at the top.

The queue

Each draft shows its name, why it was drafted (for example Failed: Goal Completion, Tool error: lookup_order, or Flagged), its kind, its re-score, its source, its target dataset, and when it was created. Filter by Source, Kind, Tags, Reason, and Target dataset, or search. When new drafts arrive while you’re on the page, select Load latest to see them.
Review queue tab with the Auto-drafting settings button, the Source, Kind, Tags, Reason, and Target dataset filters, and three flagged session drafts with their kind, re-score, source, and target dataset

Review a draft

Select a draft, or Review in its ⋮ menu, to open it. Use the arrows to move through the queue.
1

Check the re-score

See each judge and assertion’s verdict on the original output, with its reason.
2

Pick the target dataset

Required. Drafts from auto-drafting start on the agent’s target dataset, if one is set.
3

Adjust the item

Edit the input, tags, judges, and assertions like any item. Drafts from tool errors have no judges, so add a judge or an assertion. STT and VAD drafts need a recording before you can approve them.
4

Save or approve

Select Save & re-score to keep your edits and run the re-score again with them; the draft stays in the queue. Select Approve to save your edits and add the item to the dataset.
An approved item is active and runs in the next experiment on its dataset.
Draft panel for a flagged turn of a live call, with the re-score card and Open session link, the target dataset, tags, the caller's goal, the Nothing to score yet warning, the judges and assertions, and the Reject, Save & re-score, and Approve buttons

Reject a draft

Select Reject in a draft’s ⋮ menu, and pick a reason:
  • Not a real failure (judge too strict)
  • Duplicate of an existing item
  • Not worth testing
  • Other
Add a note if you want. Rejected drafts are archived with the reason, never deleted, and never run.
A draft rejected as judge too strict is a sign to tighten that judge’s pass and fail criteria. See Judges.

Act on several drafts

  • Approve confirmed (N) approves every confirmed failure on the page that has a target dataset.
  • Tick drafts, then select Approve N selected or Reject.
Drafts without a target dataset can’t be approved in bulk. They’re listed after the others are approved; open each one and pick a target.