> ## Documentation Index > Fetch the complete documentation index at: https://docs.zeroruntime.ai/llms.txt > Use this file to discover all available pages before exploring further. # Simulations > Test your agent with simulated callers from the CLI: keep scenarios and personas in your repo, run them against a deployed or local agent, and gate CI on the result. A simulation calls your agent with simulated callers and has a judge score each conversation. `zeroruntime sim` runs simulations from your terminal, against your agent deployed on Zero Runtime Cloud or a copy running on your machine. Scenarios, personas, and runs are the same ones you see under **Simulation** in the dashboard. For what each one is, see the [Simulation overview](/agent-builder/simulation/overview). ## Quick start Run these from your project folder, the one with `zeroruntime.yaml`: ```bash theme={null} zeroruntime sim init # create sims/ with the default scenarios and personas zeroruntime sim validate # check the files for errors zeroruntime sim run --dry-run # show the conversations and estimated cost zeroruntime sim run # run it and watch the results ``` `sim init` fills in `agent` from `agent.id` in `zeroruntime.yaml`. If the project isn't deployed yet, set `agent` or `local` in the simulation file first. ## The `sims/` folder Simulations live in your repo as YAML, so you can review and version them like code. The CLI reads `sims/` from the folder you run it in. ``` sims/ simulations/regression.yaml one simulation per file scenarios/refund.yaml one scenario per file personas/angry-customer.yaml one persona per file ``` Each file has an `id`, which is also its file name. Use letters, numbers, `-`, and `_`. Every simulation file stands alone: nothing is shared or merged between files. ### Scenario ```yaml sims/scenarios/close-account.yaml theme={null} id: close-account name: Close a savings account situation: close their savings account ending in {{metadata.last4}} first_message: Hi, I'd like to close my savings account. metadata: name: Priya Shah last4: "4821" success_criteria: Agent verifies the caller, confirms which account, and explains when it closes. failure_criteria: Agent closes an account before verifying the caller. difficulty: medium expect_tools: called: [verify_identity] not_called: [transfer_funds] order: [verify_identity, close_account] args: close_account: { account: savings } ``` | Key | Required | Description | | - | - | - | | `name` | Yes | Display name. Also links the file to the dashboard, so keep it unique. | | `situation` | Yes | What the caller wants. `{{metadata.}}` is replaced with that metadata value. | | `first_message` | No | The caller's first line. Leave it out and the caller opens in their own words. | | `metadata` | No | Details the caller can share, such as a name or account number. | | `success_criteria` | Yes | What the agent must do to pass. | | `failure_criteria` | Yes | What the agent must never do. Anything here fails the conversation. | | `difficulty` | No | `easy`, `medium`, or `hard`. | | `expect_tools` | No | Tool calls to check after the judge scores the conversation. See [Check tool calls](#check-tool-calls). | | `source` | No | Written by the CLI: `default`, `manual`, or `generated`. Don't edit it. | ### Persona ```yaml sims/personas/angry-customer.yaml theme={null} id: angry-customer name: Angry customer behaviour: Upset before the call starts. Interrupts and asks for a real person. language: English ``` | Key | Required | Description | | - | - | - | | `name` | Yes | Display name. Also links the file to the dashboard, so keep it unique. | | `behaviour` | Yes | How the caller talks and reacts, for the whole conversation. | | `language` | No | The language the caller speaks. Defaults to `English`. | ### Simulation ```yaml sims/simulations/regression.yaml theme={null} id: regression name: Regression agent: "ag_zqn1az" mode: audio environments: [quiet, noisy] repeat: 1 concurrency: 2 max_duration: 300 scenarios: ["close-*", scn_default_refund] personas: ["*"] budget: max_cost_usd: 5 ``` | Key | Description | Default | | - | - | - | | `agent` | Deployed agent to test. Set `agent` or `local`, not both. | `agent.id` in `zeroruntime.yaml` | | `local` | Project folder or `.py` file to start on this machine and test. Audio only. | none | | `branch_id`, `language_id` | Branch and language to test, for an [Agent Builder](/agent-builder/introduction) agent. | the agent's default | | `mode` | `audio` or `text`. See [Text mode](#text-mode). | `audio` | | `environments` | Audio only. Background for each call: `quiet`, `noisy`, or `office`. Each one adds a conversation per scenario and persona pair. | `[quiet]` | | `repeat` | Times to run each conversation, `1` to `10`. | `1` | | `concurrency` | Conversations to run at once, `1` to `10`. | `1` | | `max_duration` | Audio only. Seconds per call, `120` to `600`. | `300` | | `max_turns` | Text only. Turns per conversation, `5` to `30`. | none | | `scenarios`, `personas` | File ids, globs such as `close-*`, or dashboard ids (`scn_…`, `per_…`). `*` or leaving the key out selects every file. | every file | | `budget.max_cost_usd` | Refuse a run estimated above this, and stop one that costs more. | none | | `name` | Run name. | ` · @` | | `agent_config` | Text only. The agent to chat with. See [Text mode](#text-mode). | none | Quote values that start with `*` in YAML, such as `["*"]` and `["close-*"]`. A run can have up to 50 scenarios, 50 personas, and 500 conversations. The number of conversations is scenarios × personas × environments × `repeat`. ### Create files ```bash theme={null} zeroruntime sim simulations new smoke # write sims/simulations/smoke.yaml and open it in $EDITOR zeroruntime sim scenarios new close-account # write a scenario template and open it zeroruntime sim simulations list # each simulation with its agent and conversation count ``` To write a persona, add a file to `sims/personas/` by hand, or create it in the dashboard and [pull](#sync-with-the-dashboard) it. ### Generate scenarios Generate scenarios from your agent's system prompt. They're saved to the dashboard and written to `sims/scenarios/`. ```bash theme={null} zeroruntime sim scenarios generate # 8 scenarios from the agent's prompt zeroruntime sim scenarios generate --count 5 --difficulty hard zeroruntime sim scenarios generate --prompt-file prompt.txt # read the prompt from a file ``` For an agent you deploy with the CLI, pass the prompt with `--prompt-file`. The prompt is only fetched automatically for Agent Builder agents. ## Run a simulation ```bash theme={null} zeroruntime sim run # the only simulation, or pick one from a list zeroruntime sim run regression # sims/simulations/regression.yaml zeroruntime sim run --scenario 'close-*' --persona angry-customer zeroruntime sim run --dry-run # show conversations and cost, run nothing ``` Flags override the simulation file. `sim run`: 1. Shows the conversations and estimated cost, and asks you to confirm (`--yes` skips it). 2. Creates or updates the selected scenarios and personas on the dashboard. Default scenarios and personas are never changed. 3. Starts the run and prints each conversation's verdict as it finishes. 4. Prints a summary and exits with a [status code](#exit-codes). Press **Ctrl-C** during a run to stop it, keep watching, or detach and leave it running. With `--local` you can only stop or keep watching. Use `--detach` to start a run and return its id right away. ### Test a local agent `--local` starts your agent on this machine and calls it, so you can test changes before you deploy them: ```bash theme={null} zeroruntime sim run --local . # this project zeroruntime sim run --local my-agent.py ``` The CLI waits for the agent to register, runs the simulation, then stops the agent. The agent's output goes to `.zeroruntime/sim-agent.log`. It runs under the agent id in `ZERORUNTIME_AGENT_ID` from your `.env`, or `zeroruntime_agent`. > [!NOTE] > > `--local` only works with audio mode, and not with `--detach`. Use an agent id that isn't deployed: if it is, the call would reach the deployed agent, so the CLI stops the run. ### Text mode Text mode chats with your agent instead of calling it. It's faster and cheaper, and it tests what the agent says: its prompt, LLM, and tools. For an agent you deploy with the CLI, describe the agent in the simulation file's `agent_config`. Text mode runs this copy, not your deployed code, so only HTTP tools are called. Agent Builder agents use their saved configuration instead, so leave `agent_config` out. ```yaml sims/simulations/text-smoke.yaml theme={null} id: text-smoke mode: text max_turns: 12 scenarios: ["*"] personas: ["*"] agent_config: name: Support agent system_prompt: You are a support agent for Northwind Bank. Always verify the caller first. greeting: Hi, thanks for calling Northwind Bank. How can I help? llm: provider: google model: gemini-2.5-flash temperature: 0.4 tools: - name: verify_identity description: Verify the caller with their date of birth. parameters: type: object properties: dob: { type: string } required: [dob] http: url: https://api.example.com/verify method: POST timeout_s: 10 end_call: true knowledgebase_ids: [] ``` | `agent_config` key | Description | | - | - | | `name`, `system_prompt` | Required. The agent's name and instructions. | | `greeting`, `timezone` | Optional. | | `llm` | `provider` (`google` or `openai`) and `model` are required. Optional: `temperature`, `max_output_tokens`, `thinking_budget` (Google), `reasoning_effort` (OpenAI). | | `tools` | HTTP tools: `name`, `description`, `parameters` (JSON Schema), and `http` with `url`, `method`, `headers`, and `timeout_s`. | | `end_call` | Let the agent end the conversation. Defaults to `true`. | | `knowledgebase_ids` | Knowledge bases the agent searches. | `environments`, `max_duration`, and `local` don't apply in text mode, and `max_turns` and `agent_config` don't apply in audio mode. `sim validate` reports these. ## Check tool calls `expect_tools` in a scenario checks the agent's tool calls in every judged conversation: | Key | Passes when | | - | - | | `called` | Each tool listed was called. | | `not_called` | None of the tools listed was called. | | `order` | The tools were first called in this order. | | `args` | At least one call to the tool included these arguments. Other arguments are ignored. | A conversation the judge passed counts as **failed** in the CLI summary, reports, and exit code if a tool check fails. The dashboard still shows the judge's verdict. ## Run in CI ```bash theme={null} zeroruntime sim run regression --yes --max-cost 10 --report junit:sim.xml ``` * `--yes` skips the confirmation prompt. * `--max-cost` refuses a run estimated above the limit, and stops one that goes over. * `--report junit:` writes one test case per conversation, with failures and skipped conversations marked. Use `--report json:` for JSON. Repeat `--report` for both. ### Exit codes | Code | Meaning | | - | - | | `0` | Every conversation passed. | | `1` | At least one conversation failed. | | `2` | Invalid flags or `sims/` files. | | `3` | The run didn't finish: stopped, over budget, the local agent exited, nothing was judged, or an API error. | ## Review results ```bash theme={null} zeroruntime sim watch # live view of your runs zeroruntime sim runs list # recent runs zeroruntime sim show run_8b2d4e07f1 # a run and its conversations zeroruntime sim show run_8b2d4e07f1 --result failed zeroruntime sim show sims_2k8f1qd9xa --run run_8b2d4e07f1 --transcript ``` `sim watch` opens a live, full-screen view. Open a run to see its conversations, then a conversation to see its verdict, where it failed, the judge's suggested fixes, cost, and transcript as it happens. Use the arrow keys to move, **Enter** to open, **Esc** to go back, **r** to refresh, and **q** to quit. `sim show` prints a run's summary and conversations, or one conversation's verdict, where it failed, and the judge's suggestions. Add `--transcript` for the full conversation and `--json` for the raw response. You can open every run in the dashboard too. See [Review results](/agent-builder/simulation/review-results). ## Stop, re-run, and delete ```bash theme={null} zeroruntime sim stop run_8b2d4e07f1 # stop a running run zeroruntime sim rerun run_8b2d4e07f1 # run every conversation again zeroruntime sim rerun run_8b2d4e07f1 --failed # only failed, abandoned, or stopped ones zeroruntime sim delete run_8b2d4e07f1 # the run's simulation and all its runs zeroruntime sim delete sims_2k8f1qd9xa --run run_8b2d4e07f1 # one conversation ``` Stopping keeps the results of finished conversations. A re-run tests the version of the agent that's live when it starts, so you can fix the agent, run `zeroruntime up`, and re-run. `rerun` and `delete` ask first; `--yes` skips the prompt. ## Sync with the dashboard `sim run` uploads the scenarios and personas it uses. To sync without running: ```bash theme={null} zeroruntime sim push --all # upload every scenario and persona zeroruntime sim push scenarios 'close-*' # only these scenarios zeroruntime sim pull --all # write dashboard scenarios and personas that aren't in sims/ yet zeroruntime sim scenarios list # dashboard scenarios, and which have a local file zeroruntime sim scenarios delete close-account # delete from the dashboard; the file is kept ``` Files are matched to dashboard items by `name`. Renaming a file's `name` creates a new item on the next push. Default scenarios and personas are read-only: `push` skips them and `delete` refuses them. `pull` never overwrites a file. See the [CLI Reference](/runtime-sdk/deployments/cli-reference#sim) for every command and flag. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.