Benchmarks

Benchmarks let you test how your agents behave before they ever pick up a real call. Instead of checking a single question-and-answer, a benchmark runs your agent through a multi-step scenario: a back-and-forth conversation where each message builds on the last, so you can confirm the agent follows the whole flow from one instruction to the next, not just the first reply.

Who can use benchmarks

AI test suites and benchmarks are an Enterprise capability. Within an entitled workspace, users who can view agents can open and run them. Each benchmark is tied to one agent so you can re-run it whenever you change that agent's prompt, tools, or model.

Creating a benchmark

Open the Benchmarks page

Choose Benchmarks from the sidebar. The list shows every benchmark in your workspace, the agent it targets, how many scenarios it has, and the result of its last run.

Create a new benchmark

Click Create, give it a name (e.g. Customer Support Scenarios), add an optional description, and pick the agent you want to test. Press Create (or Ctrl/⌘ + Enter).

Open it and add scenarios

Click into the benchmark to add the scenarios that describe how a conversation should play out. A benchmark can hold as many scenarios as you like.

Scenarios & steps

A scenario is a single multi-turn conversation. It's made up of one or more steps, and each step is one thing the customer says plus what you expect back. Running the scenario replays those steps in order; the agent's real reply to step 1 stays in the conversation when step 2 is sent, so you're testing the agent's memory and flow, exactly like a live call.

Set an overall goal (optional)

The Goal describes what a successful conversation achieves overall, for example “Identify the order, confirm the reason for the refund, and tell the customer when to expect it.” The evaluator uses it to judge whether the agent completed the whole task, not just each step in isolation.

Add the conversation steps

For each turn, type the user message the customer would send. Add Add step for the next turn, and reorder or remove steps as you refine the flow. The steps run top to bottom.

Describe what you expect (optional)

Per step you can add an expected response (roughly what the agent should say) and an expected behaviour (what it should do, e.g. “ask for the order number” or “call the refund tool”). Both are optional; leave them blank for steps where you only care that the conversation keeps moving.

Mirror a real call

The most useful scenarios read like a transcript of a real customer. Start with the opening request, then layer in the follow-ups, edge cases, and curveballs a person would actually throw: “actually, can you make that two?”, a wrong order number, an out-of-scope question, so you find out where the flow breaks before a customer does.

How scoring works

When a benchmark runs, every scenario is played out turn by turn and scored two ways:

Similarity: each agent reply is compared against your expected response for that step to gauge how closely it matches.
AI judge: an evaluator model reviews the conversation against your expected behaviour for each step and the scenario's overall goal, and decides whether each step passed and whether the whole flow was handled well. It also writes a short summary explaining the verdict.

A scenario passes when its steps come through cleanly and the goal is met. Steps without expectations still run; they keep the conversation going so later steps have realistic context; they just aren't graded.

Running benchmarks & reading results

Run all tests

Click Run All Tests on the benchmark. Every scenario is executed and you get an overall pass rate plus a per-scenario breakdown.

Open a scenario result

Each result shows steps passed, the agent's actual reply next to your expectation for every turn, and the judge's summary of what went right or wrong, so a failure points you straight at the turn that broke.

Compare runs over time

The Test Runs tab keeps a history of every run. After you tweak a prompt, tool, or model, re-run the benchmark and compare the pass rate to confirm the change actually helped, and didn't regress anything else.

Benchmarks vs. live testing

Benchmarks are for repeatable, scripted checks across many scenarios at once. To improvise a single conversation by voice or chat, use the Test Agent and Preview Changes buttons on the agent editor instead.