CumulusDocs
Open Studio
On this page

Test a change before release

A change to the equipment-service agent has introduced a mistake: it shares service details before checking Alex’s access to the site. Save a caller scenario, test the changed publication and compare it with the version that verified access first.

The cases, results and costs in this example are fictional.

Save the scenario and assertions

Test a change before release
Download video

A saved test case with a caller goal and persona for an unverified service request

The caller asks about the service request without immediately confirming identity.

The test case assertions specifying verification and lookup nodes plus a judged identity criterion

Use a node assertion to check the visited steps and a judged criterion to check what the agent disclosed.

Open the agent's Tests tab and create a suite. For “Verify before service details”:

  1. Give the simulated caller a goal: ask about a service request without immediately confirming identity.
  2. Write the caller behavior: ask who is calling; confirm identity only after the purpose is explained.
  3. Add a node assertion for the relevant verification and lookup nodes. Visiting nodes is a structural check; it is not, alone, proof that they occurred in the correct business order.
  4. Add a judged criterion: “Did the agent verify identity before sharing service details?” Define a threshold and inspect the judge's reasoning when it fails.
  5. Mock lookup_service_request with a fixed example response so the test does not depend on a production service record.
  6. Save the case and validate it against the target draft or publication.

Mock each customer HTTP action the case needs. An unmocked customer HTTP action fails with unmocked_action before sending a request. Built-in actions have simulated defaults.

Use a separate case for each meaningful failure path. A useful starting suite includes caller refusal, wrong person, missing service record, lookup timeout, unavailable transfer and a request outside policy. Use deterministic assertions where possible and review model-judged failures against the conversation.

Read test suite creation, suite edits and suite validation for the current case, assertion and mock fields.

Run the exact candidate

A completed test run with its publication, summary, cost and failed case

The summary identifies the publication and suite revision. Open the failed case to inspect its trial.

In Studio, selecting the draft in the run dialog first publishes it as a test version with live: false, then estimates and starts the suite against that version. Through the API, call Publish an agent with live: false yourself, then pass the returned version to test_runs.estimate and test_runs.create. Review the trial count and estimate before starting, and follow the test run until it settles.

Save the earlier test-run ID as the baseline. Compare the same cases, mocks, judge settings and trial count across versions. If the suite changed too, review that difference before interpreting a higher score as an agent improvement.

The estimate, create, results and trial endpoints support the same workflow from code. Build and test recipes shows the publication-first sequence.

Inspect the failing evidence

Open the failed “Verify before service details” case. The conversation shows service details appearing before site access was verified.

A failed trial with an assertion trace, highlighted conversation evidence and the judge's explanation

Open the failing case, choose a trial and select the assertion. Review its evidence turns, node path, action calls and judge rationale. Determine whether the failure is in the agent, the scenario, the assertion or the judge before editing the flow.

The same service case compared between baseline and candidate

For this case, move service disclosure after verification and cover the refusal path explicitly. Publish a new test version and run the same suite again. Add additional trials when you need to understand variability; one passing result does not establish a reliable pass rate.

Decide whether to promote

A candidate published with live: false can only run with test: true. To use the reviewed behavior for regular calls or tasks, publish it with live: true; moving a test version to the head does not change that restriction. Publication and recovery covers the exact requests.

A passing suite means these cases met their assertions. Before releasing a phone agent, also test its speech, number routing and real tool endpoints:

CheckWhat it establishes
Draft validationThe definition meets compiler checks
Simulation assertions and evidenceThe tested conversational behaviors under the configured scenarios
Browser voice testBehavior through a browser voice session
Authorized carrier callReal number routing and phone/media behavior
Real integration acceptanceThe intended downstream service works with the actual credentials and business contract

Simulation mocks do not prove live tool delivery. Text-based simulations do not prove audio quality or carrier routing. Release the reviewed graph for ordinary execution only after the acceptance checks appropriate to your use case pass, then inspect real run evidence.

Keep the test-run ID with your release review. It links the publication, suite revision and trial evidence you used to approve the change.

Content version 705b95adCustomer SDKMarkdown source
Test a change before release · Cumulus developer docs