Test an agent

Test the behavior you expect before changing the live agent. For the equipment-service agent, check that it verifies Alex’s access before discussing the request, books only an agreed slot and handles unavailable dispatchers.

Start with a simulation: give a fictional caller a goal, provide mock responses for your tools and assert what the agent should do. Then test spoken interaction in the browser and number routing with a controlled phone call. The table below explains what each method executes.

Prerequisites

Use a member or admin key for the intended environment and publish a test version with live: false. Run admission must be enabled even for simulations. See workspace setup.

Testing methods

A phone run places a real call even with test: true.

Runs marked as tests are excluded from Cumulus billing, customer lifecycle webhooks and event triggers. Carrier charges remain your carrier's responsibility. live: false keeps a publication separate from the live head; it does not replace its business endpoints. Use synthetic records and an isolated agent with sandbox endpoints for browser and carrier tests.

If you author in TypeScript, local validation checks the definition without contacting Cumulus. A simulation runs the conversation against your saved case; browser and phone tests also use the configured speech services.

A suite contains cases, each with a simulated caller and assertions. Use a persona for a caller with a goal or a script for fixed lines. Text is the default. Where enabled, an audio case uses a scripted caller with a configured voice and joins the agent over LiveKit; it does not reach a carrier. Check audio worker and provider readiness separately from text simulation.

Call POST /api/v1/test-runs with the publication version to test. For a draft, publish with "live": false first. Simulations need no trunk or phone number, but Cumulus must enable run admission. Closed admission returns tenant_not_configured or tenant_admission_closed.

Create a simulation suite and run it

Replace the agent, node and action IDs below with IDs from your draft, and supply all dynamic variables the publication requires. The mock returns a successful authorization lookup; mock every other customer HTTP action the case can reach. Use the publication version returned when you published the test version, and the suite ID returned by suite creation. Send mutation requests with an Idempotency-Key.

POST /api/v1/test-suites
{
  "agent_id": "equipment-service",
  "name": "Service access verification",
  "spec": { "kind": "simulation", "cases": [{
    "case_id": "identity-confirms", "name": "The caller confirms their identity", "language": "en",
    "caller": { "kind": "persona", "goal": "Confirm you are Alex and ask what the service call is about",
                "brief": "You manage the warehouse. Ask who is calling before confirming your identity." },
    "dynamic_variables": { "contact_name": "Alex", "service_request_id": "SR-DEMO-24" },
    "mocks": [{ "action_id": "lookup_service_request", "respond": {
      "kind": "response", "status": 200,
      "body": { "authorized": true, "equipment_summary": "Loading-bay equipment" }
    } }],
    "assertions": [
      { "type": "node_visited", "id": "identity-checked", "node_ids": ["authorization"] },
      { "type": "agent_says", "id": "no-equipment-details-before-id", "expect": "never", "match": { "patterns": ["loading-bay equipment"] },
        "scope": { "until_node": "authorization" } }
    ]
  }] }
}

POST /api/v1/test-runs
{ "agent_id": "equipment-service", "version": 3, "suite_ids": ["<suite_id>"] }

• Deterministic assertions check the path and the words: node_visited, node_not_visited, final_node, path, agent_says, tool_called, variable, ended, transfer, disposition. A rubric assertion uses a model to judge your criteria. Review its reasoning against the conversation.
• mocks answer customer HTTP actions by action_id. An unmocked customer HTTP action fails with unmocked_action before sending; built-in actions have simulated defaults.
• Each case runs trials times (default 1, at most 10) and passes by pass_policy (majority or all). POST /api/v1/test-runs/estimate gives the trials, cost and minutes first.
• Poll GET /api/v1/test-runs/{test_run_id} until state is completed or cancelled; …/results lists every case and assertion verdict, and …/cases/{case_id}/trials/{trial} shows the transcript a trial was judged on.
• POST /api/v1/test-suites/{suite_id}/validate checks a suite's node, action and variable references against a version or the open draft.

Private keypad tests

verify_identity and tokenize_private nodes execute only on telephone runs. Browser, cloud and simulation runs containing these nodes return capability_unavailable. Test them with a dedicated telephone number and synthetic values instead.

Validate the policy locally, then test it with a development agent, a dedicated number, synthetic values and a sandbox vault. The recipient must consent to the call. # submits a field; * cancels. A timeout, disconnect, malformed input or stale activation fails closed. There is no automatic recapture of another guess or automatic replay of an uncertain vault send.

Check the selected success or failure route and the tool receipt. For tokenization, inspect the returned token and permitted last-four suffix. Confirm that the entered digits do not appear in model context, transcript, public receipts or recording. See private workflow reference.

Next step

Follow the regression-testing example to inspect a failed trial and compare versions. Fix the draft, publish a new test version and repeat. Use the release guide to release the reviewed graph for regular use.