On this page
Test a change before release
A change to the equipment-service agent has introduced a mistake: it shares service details before checking Alex’s access to the site. Save a caller scenario, test the changed publication and compare it with the version that verified access first.
The cases, results and costs in this example are fictional.
Save the scenario and assertions
The caller asks about the service request without immediately confirming identity.
Use a node assertion to check the visited steps and a judged criterion to check what the agent disclosed.
Open the agent's Tests tab and create a suite. For “Verify before service details”:
- Give the simulated caller a goal: ask about a service request without immediately confirming identity.
- Write the caller behavior: ask who is calling; confirm identity only after the purpose is explained.
- Add a node assertion for the relevant verification and lookup nodes. Visiting nodes is a structural check; it is not, alone, proof that they occurred in the correct business order.
- Add a judged criterion: “Did the agent verify identity before sharing service details?” Define a threshold and inspect the judge's reasoning when it fails.
- Mock
lookup_service_requestwith a fixed example response so the test does not depend on a production service record. - Save the case and validate it against the target draft or publication.
Mock each customer HTTP action the case needs. An unmocked customer HTTP action fails with unmocked_action before sending a request. Built-in actions have simulated defaults.
Use a separate case for each meaningful failure path. A useful starting suite includes caller refusal, wrong person, missing service record, lookup timeout, unavailable transfer and a request outside policy. Use deterministic assertions where possible and review model-judged failures against the conversation.
Read test suite creation, suite edits and suite validation for the current case, assertion and mock fields.
Run the exact candidate
The summary identifies the publication and suite revision. Open the failed case to inspect its trial.
In Studio, selecting the draft in the run dialog first publishes it as a test version with live: false, then estimates and starts the suite against that version. Through the API, call Publish an agent with live: false yourself, then pass the returned version to test_runs.estimate and test_runs.create. Review the trial count and estimate before starting, and follow the test run until it settles.
Save the earlier test-run ID as the baseline. Compare the same cases, mocks, judge settings and trial count across versions. If the suite changed too, review that difference before interpreting a higher score as an agent improvement.
The estimate, create, results and trial endpoints support the same workflow from code. Build and test recipes shows the publication-first sequence.
Inspect the failing evidence
Open the failed “Verify before service details” case. The conversation shows service details appearing before site access was verified.
Open the failing case, choose a trial and select the assertion. Review its evidence turns, node path, action calls and judge rationale. Determine whether the failure is in the agent, the scenario, the assertion or the judge before editing the flow.
For this case, move service disclosure after verification and cover the refusal path explicitly. Publish a new test version and run the same suite again. Add additional trials when you need to understand variability; one passing result does not establish a reliable pass rate.
Decide whether to promote
A candidate published with live: false can only run with test: true. To use the reviewed behavior for regular calls or tasks, publish it with live: true; moving a test version to the head does not change that restriction. Publication and recovery covers the exact requests.
A passing suite means these cases met their assertions. Before releasing a phone agent, also test its speech, number routing and real tool endpoints:
| Check | What it establishes |
|---|---|
| Draft validation | The definition meets compiler checks |
| Simulation assertions and evidence | The tested conversational behaviors under the configured scenarios |
| Browser voice test | Behavior through a browser voice session |
| Authorized carrier call | Real number routing and phone/media behavior |
| Real integration acceptance | The intended downstream service works with the actual credentials and business contract |
Simulation mocks do not prove live tool delivery. Text-based simulations do not prove audio quality or carrier routing. Release the reviewed graph for ordinary execution only after the acceptance checks appropriate to your use case pass, then inspect real run evidence.
Keep the test-run ID with your release review. It links the publication, suite revision and trial evidence you used to approve the change.




