Simulations

A simulation calls your agent with simulated callers and has a judge score each conversation. zeroruntime sim runs simulations from your terminal, against your agent deployed on Zero Runtime Cloud or a copy running on your machine. Scenarios, personas, and runs are the same ones you see under Simulation in the dashboard. For what each one is, see the Simulation overview.

Quick start

Run these from your project folder, the one with zeroruntime.yaml:

zeroruntime sim init              # create sims/ with the default scenarios and personas
zeroruntime sim validate          # check the files for errors
zeroruntime sim run --dry-run     # show the conversations and estimated cost
zeroruntime sim run               # run it and watch the results

sim init fills in agent from agent.id in zeroruntime.yaml. If the project isn’t deployed yet, set agent or local in the simulation file first.

The `sims/` folder

Simulations live in your repo as YAML, so you can review and version them like code. The CLI reads sims/ from the folder you run it in.

sims/
  simulations/regression.yaml   one simulation per file
  scenarios/refund.yaml         one scenario per file
  personas/angry-customer.yaml  one persona per file

Each file has an id, which is also its file name. Use letters, numbers, -, and _. Every simulation file stands alone: nothing is shared or merged between files.

Scenario

sims/scenarios/close-account.yaml

id: close-account
name: Close a savings account
situation: close their savings account ending in {{metadata.last4}}
first_message: Hi, I'd like to close my savings account.
metadata:
  name: Priya Shah
  last4: "4821"
success_criteria: Agent verifies the caller, confirms which account, and explains when it closes.
failure_criteria: Agent closes an account before verifying the caller.
difficulty: medium
expect_tools:
  called: [verify_identity]
  not_called: [transfer_funds]
  order: [verify_identity, close_account]
  args:
    close_account: { account: savings }

Key	Required	Description
`name`	Yes	Display name. Also links the file to the dashboard, so keep it unique.
`situation`	Yes	What the caller wants. `{{metadata.<key>}}` is replaced with that metadata value.
`first_message`	No	The caller’s first line. Leave it out and the caller opens in their own words.
`metadata`	No	Details the caller can share, such as a name or account number.
`success_criteria`	Yes	What the agent must do to pass.
`failure_criteria`	Yes	What the agent must never do. Anything here fails the conversation.
`difficulty`	No	`easy`, `medium`, or `hard`.
`expect_tools`	No	Tool calls to check after the judge scores the conversation. See Check tool calls.
`source`	No	Written by the CLI: `default`, `manual`, or `generated`. Don’t edit it.

Persona

sims/personas/angry-customer.yaml

id: angry-customer
name: Angry customer
behaviour: Upset before the call starts. Interrupts and asks for a real person.
language: English

Key	Required	Description
`name`	Yes	Display name. Also links the file to the dashboard, so keep it unique.
`behaviour`	Yes	How the caller talks and reacts, for the whole conversation.
`language`	No	The language the caller speaks. Defaults to `English`.

Simulation

sims/simulations/regression.yaml

id: regression
name: Regression
agent: "ag_zqn1az"
mode: audio
environments: [quiet, noisy]
repeat: 1
concurrency: 2
max_duration: 300
scenarios: ["close-*", scn_default_refund]
personas: ["*"]
budget:
  max_cost_usd: 5

Key	Description	Default
`agent`	Deployed agent to test. Set `agent` or `local`, not both.	`agent.id` in `zeroruntime.yaml`
`local`	Project folder or `.py` file to start on this machine and test. Audio only.	none
`branch_id`, `language_id`	Branch and language to test, for an Agent Builder agent.	the agent’s default
`mode`	`audio` or `text`. See Text mode.	`audio`
`environments`	Audio only. Background for each call: `quiet`, `noisy`, or `office`. Each one adds a conversation per scenario and persona pair.	`[quiet]`
`repeat`	Times to run each conversation, `1` to `10`.	`1`
`concurrency`	Conversations to run at once, `1` to `10`.	`1`
`max_duration`	Audio only. Seconds per call, `120` to `600`.	`300`
`max_turns`	Text only. Turns per conversation, `5` to `30`.	none
`scenarios`, `personas`	File ids, globs such as `close-`, or dashboard ids (`scn_…`, `per_…`). `` or leaving the key out selects every file.	every file
`budget.max_cost_usd`	Refuse a run estimated above this, and stop one that costs more.	none
`name`	Run name.	`<id> · <git branch>@<sha>`
`agent_config`	Text only. The agent to chat with. See Text mode.	none

Quote values that start with * in YAML, such as ["*"] and ["close-*"]. A run can have up to 50 scenarios, 50 personas, and 500 conversations. The number of conversations is scenarios × personas × environments × repeat.

Create files

zeroruntime sim simulations new smoke         # write sims/simulations/smoke.yaml and open it in $EDITOR
zeroruntime sim scenarios new close-account   # write a scenario template and open it
zeroruntime sim simulations list              # each simulation with its agent and conversation count

To write a persona, add a file to sims/personas/ by hand, or create it in the dashboard and pull it.

Generate scenarios

Generate scenarios from your agent’s system prompt. They’re saved to the dashboard and written to sims/scenarios/.

zeroruntime sim scenarios generate                               # 8 scenarios from the agent's prompt
zeroruntime sim scenarios generate --count 5 --difficulty hard
zeroruntime sim scenarios generate --prompt-file prompt.txt      # read the prompt from a file

For an agent you deploy with the CLI, pass the prompt with --prompt-file. The prompt is only fetched automatically for Agent Builder agents.

Run a simulation

zeroruntime sim run                                  # the only simulation, or pick one from a list
zeroruntime sim run regression                       # sims/simulations/regression.yaml
zeroruntime sim run --scenario 'close-*' --persona angry-customer
zeroruntime sim run --dry-run                        # show conversations and cost, run nothing

Flags override the simulation file. sim run:

Shows the conversations and estimated cost, and asks you to confirm (--yes skips it).
Creates or updates the selected scenarios and personas on the dashboard. Default scenarios and personas are never changed.
Starts the run and prints each conversation’s verdict as it finishes.
Prints a summary and exits with a status code.

Press Ctrl-C during a run to stop it, keep watching, or detach and leave it running. With --local you can only stop or keep watching. Use --detach to start a run and return its id right away.

Test a local agent

--local starts your agent on this machine and calls it, so you can test changes before you deploy them:

zeroruntime sim run --local .               # this project
zeroruntime sim run --local my-agent.py

The CLI waits for the agent to register, runs the simulation, then stops the agent. The agent’s output goes to .zeroruntime/sim-agent.log. It runs under the agent id in ZERORUNTIME_AGENT_ID from your .env, or zeroruntime_agent.

--local only works with audio mode, and not with --detach. Use an agent id that isn’t deployed: if it is, the call would reach the deployed agent, so the CLI stops the run.

Text mode

Text mode chats with your agent instead of calling it. It’s faster and cheaper, and it tests what the agent says: its prompt, LLM, and tools. For an agent you deploy with the CLI, describe the agent in the simulation file’s agent_config. Text mode runs this copy, not your deployed code, so only HTTP tools are called. Agent Builder agents use their saved configuration instead, so leave agent_config out.

sims/simulations/text-smoke.yaml

id: text-smoke
mode: text
max_turns: 12
scenarios: ["*"]
personas: ["*"]
agent_config:
  name: Support agent
  system_prompt: You are a support agent for Northwind Bank. Always verify the caller first.
  greeting: Hi, thanks for calling Northwind Bank. How can I help?
  llm:
    provider: google
    model: gemini-2.5-flash
    temperature: 0.4
  tools:
    - name: verify_identity
      description: Verify the caller with their date of birth.
      parameters:
        type: object
        properties:
          dob: { type: string }
        required: [dob]
      http:
        url: https://api.example.com/verify
        method: POST
        timeout_s: 10
  end_call: true
  knowledgebase_ids: []

`agent_config` key	Description
`name`, `system_prompt`	Required. The agent’s name and instructions.
`greeting`, `timezone`	Optional.
`llm`	`provider` (`google` or `openai`) and `model` are required. Optional: `temperature`, `max_output_tokens`, `thinking_budget` (Google), `reasoning_effort` (OpenAI).
`tools`	HTTP tools: `name`, `description`, `parameters` (JSON Schema), and `http` with `url`, `method`, `headers`, and `timeout_s`.
`end_call`	Let the agent end the conversation. Defaults to `true`.
`knowledgebase_ids`	Knowledge bases the agent searches.

environments, max_duration, and local don’t apply in text mode, and max_turns and agent_config don’t apply in audio mode. sim validate reports these.

Check tool calls

expect_tools in a scenario checks the agent’s tool calls in every judged conversation:

Key	Passes when
`called`	Each tool listed was called.
`not_called`	None of the tools listed was called.
`order`	The tools were first called in this order.
`args`	At least one call to the tool included these arguments. Other arguments are ignored.

A conversation the judge passed counts as failed in the CLI summary, reports, and exit code if a tool check fails. The dashboard still shows the judge’s verdict.

Run in CI

zeroruntime sim run regression --yes --max-cost 10 --report junit:sim.xml

--yes skips the confirmation prompt.
--max-cost refuses a run estimated above the limit, and stops one that goes over.
--report junit:<path> writes one test case per conversation, with failures and skipped conversations marked. Use --report json:<path> for JSON. Repeat --report for both.

Exit codes

Code	Meaning
`0`	Every conversation passed.
`1`	At least one conversation failed.
`2`	Invalid flags or `sims/` files.
`3`	The run didn’t finish: stopped, over budget, the local agent exited, nothing was judged, or an API error.

Review results

zeroruntime sim watch                               # live view of your runs
zeroruntime sim runs list                           # recent runs
zeroruntime sim show run_8b2d4e07f1                 # a run and its conversations
zeroruntime sim show run_8b2d4e07f1 --result failed
zeroruntime sim show sims_2k8f1qd9xa --run run_8b2d4e07f1 --transcript

sim watch opens a live, full-screen view. Open a run to see its conversations, then a conversation to see its verdict, where it failed, the judge’s suggested fixes, cost, and transcript as it happens. Use the arrow keys to move, Enter to open, Esc to go back, r to refresh, and q to quit. sim show prints a run’s summary and conversations, or one conversation’s verdict, where it failed, and the judge’s suggestions. Add --transcript for the full conversation and --json for the raw response. You can open every run in the dashboard too. See Review results.

Stop, re-run, and delete

zeroruntime sim stop run_8b2d4e07f1                     # stop a running run
zeroruntime sim rerun run_8b2d4e07f1                    # run every conversation again
zeroruntime sim rerun run_8b2d4e07f1 --failed           # only failed, abandoned, or stopped ones
zeroruntime sim delete run_8b2d4e07f1                   # the run's simulation and all its runs
zeroruntime sim delete sims_2k8f1qd9xa --run run_8b2d4e07f1   # one conversation

Stopping keeps the results of finished conversations. A re-run tests the version of the agent that’s live when it starts, so you can fix the agent, run zeroruntime up, and re-run. rerun and delete ask first; --yes skips the prompt.

Sync with the dashboard

sim run uploads the scenarios and personas it uses. To sync without running:

zeroruntime sim push --all                  # upload every scenario and persona
zeroruntime sim push scenarios 'close-*'    # only these scenarios
zeroruntime sim pull --all                  # write dashboard scenarios and personas that aren't in sims/ yet
zeroruntime sim scenarios list              # dashboard scenarios, and which have a local file
zeroruntime sim scenarios delete close-account   # delete from the dashboard; the file is kept

Files are matched to dashboard items by name. Renaming a file’s name creates a new item on the next push. Default scenarios and personas are read-only: push skips them and delete refuses them. pull never overwrites a file. See the CLI Reference for every command and flag.

Overview

Build

Plugins

Deploy

Quick start

The `sims/` folder

Scenario

Persona

Simulation

Create files

Generate scenarios

Run a simulation

Test a local agent

Text mode

Check tool calls

Run in CI

Exit codes

Review results

Stop, re-run, and delete

Sync with the dashboard

​Quick start

​The sims/ folder

​Scenario

​Persona

​Simulation

​Create files

​Generate scenarios

​Run a simulation

​Test a local agent

​Text mode

​Check tool calls

​Run in CI

​Exit codes

​Review results

​Stop, re-run, and delete

​Sync with the dashboard

Quick start

The `sims/` folder

Scenario

Persona

Simulation

Create files

Generate scenarios

Run a simulation

Test a local agent

Text mode

Check tool calls

Run in CI

Exit codes

Review results

Stop, re-run, and delete

Sync with the dashboard