Test an agent

Test a pinned publication before making it live. Use text simulations or browser voice sessions before placing real phone calls.

Prerequisites

Publish a test version, choose your sandbox, and ensure run admission is open.

Testing methods

A test suite holds cases: a simulated caller (a persona with a goal, or a script of lines) talks to the agent in text, and assertions check what happened. POST /api/v1/test-runs runs suites against one published version, live or test, so a draft is tested by publishing it with "live": false first. No phone call is placed, and no trunk or number is needed; the environment's run admission must be open (Cumulus staff open it; until then a test run is refused with tenant_not_configured or tenant_admission_closed).

Create a suite and run it

POST /api/v1/test-suites
{
  "agent_id": "fairway-rpc-outbound",
  "name": "Right-party contact",
  "spec": { "kind": "simulation", "cases": [{
    "case_id": "rpc-confirms", "name": "The debtor confirms who they are", "language": "en",
    "caller": { "kind": "persona", "goal": "Confirm you are Jane Doe and ask what the call is about",
                "brief": "You are Jane Doe. You answer briefly and politely." },
    "dynamic_variables": { "debtorName": "Jane Doe", "debtorFirstName": "Jane" },
    "assertions": [
      { "type": "node_visited", "id": "identity-checked", "node_ids": ["<a node id from the draft>"] },
      { "type": "agent_says", "id": "no-balance-before-id", "expect": "never", "match": { "patterns": ["balance", "you owe"] },
        "scope": { "until_node": "<the node that confirms identity>" } }
    ]
  }] }
}

POST /api/v1/test-runs
{ "agent_id": "fairway-rpc-outbound", "version": 0, "suite_ids": ["<suite_id>"] }

• Deterministic assertions check the path and the words: node_visited, node_not_visited, final_node, path, agent_says, tool_called, variable, ended, transfer, disposition. A rubric assertion is judged by a model against your criteria.
• mocks answer the agent's HTTP actions by action_id, so a test never reaches your systems.
• Each case runs trials times (default 1, at most 10) and passes by pass_policy (majority or all). POST /api/v1/test-runs/estimate gives the trials, cost and minutes first.
• Poll GET /api/v1/test-runs/{test_run_id} until state is completed; …/results lists every case and assertion verdict, and …/cases/{case_id}/trials/{trial} shows the transcript a trial was judged on.
• POST /api/v1/test-suites/{suite_id}/validate checks a suite's node, action and variable references against a version or the open draft.

Next step

Read the results, fix failures in a new draft and repeat. Promote the tested publication only when you are ready to go live. See recipes.