CumulusDocs
Open Studio
On this page

Release and operate a tested agent

Suppose you change an agent so it checks account access before discussing an invoice. Save that change as a test publication, run the access-check cases, and inspect any failure before releasing it. This guide follows that candidate through review, live publication and number assignment.

The screenshots use fictional support-call examples.

Keep the candidate separate from live behavior

Published versions showing live, test and previous states with the available actions

Edit the draft, validate it and publish a test version with live: false. Existing runs continue with their original publication. New runs without an explicit version keep using the live version until you change it.

Promotion changes the head of one agent in its current environment. Sandbox and production have separate agents, secrets, tests and phone resources; promoting a sandbox publication does not move it into production. Confirm the selected environment and validate and test the production agent with its own resource references before release.

Use the version list to identify the live publication and the candidate you intend to test.

A candidate publication expanded to show behavioral differences from the earlier version

Open the change summary and review prompt, action, condition and provider changes. In this example, the candidate changes how it describes the outcome. Check that it only claims completion after the business action is confirmed.

Choose the test cases

A saved persona case asking about an account without immediately confirming identity

Node and judged assertions for verification before account disclosure

Choose cases that exercise the behavior you changed and the routes beside it. For order resolution, include failed verification, missing account data, ineligible returns, caller refusal, stale data, uncertain writes and unavailable transfers.

Use structural assertions for the path and actions, and judged criteria for conversational requirements. State the expected behavior, such as “verify identity before sharing invoice details,” in each case.

Mock customer HTTP actions to give the cases repeatable responses. Unmocked customer HTTP actions fail with unmocked_action before sending. Test the real credentials, routing and business endpoints separately.

Review the tested version

A test run summary with its pinned version, suite revision, result and comparison context

Check both the agent publication and the suite revision. If the suite changed, the results may no longer measure the same behavior. Check the result type: a failed assertion, invalid case, execution error and flaky pass need different fixes.

For a case error, inspect the test setup and execution failure. For a failed criterion, read the evidence and judge rationale before deciding whether the agent needs a change.

Inspect a failed trial

A failed trial showing the missing verification node and the judge's evidence turn

Open a trial and select the assertion. Use the highlighted turn, node path and action record to determine whether the fault is in the agent behavior, test setup, criterion or judge.

In this example, the candidate shared invoice details before verification. Fix the sequence, publish another test version and rerun the same case. Preserve the failed trial as evidence of what the earlier candidate did.

Compare the candidate with its baseline

Aligned baseline and candidate conversations with the point of path divergence

Use the same case across versions and compare the trial evidence, not only the summary score. The baseline in this illustration waited for verification; the candidate took the lookup path too early. The aligned view helps locate that behavioral difference.

Compare the same suite revision, caller behavior, mocks, judge settings and trial count. If you changed the test setup too, record that difference in the release review.

Record a judge disagreement

The human feedback control for a judged assertion

If the judge is wrong, record the human verdict through the feedback control and retain the supporting explanation in your review process. This feedback is an evaluation of the trial; it does not rewrite the original test result.

Review judge scores against cases your team has assessed. Use deterministic assertions for requirements you can check directly.

Set the information your operators will see

Configured post-run analysis fields for request type, resolution and the next step

Define analysis fields that tell an operator what happened and what to do next—for example, request type, resolution and review reason. Confirm transaction success from the action result; an extracted resolved flag does not override a failed receipt.

The per-agent data-storage setting

Review workspace content retention and publication diagnostics. Validation reports unsupported per-agent storage or privacy options; resolve those errors before publishing. Without retained transcripts or recordings, operators cannot review those parts of older calls. Read workspace retention before deciding how long to keep review material.

Publish live and assign the number

Refresh and validate the reviewed draft, then use publication creation with live: true for regular calls or tasks. Preserve both the tested candidate and released publication version/digest in the release review. The release must be a live publication: selecting a test version as head does not make it usable for regular runs. Where a suite is configured as required, review the applicable publication checks and their results.

For incoming calls, ask an admin to reassign each number with PATCH /api/v1/telephony/numbers/{number_id} and {"inbound_agent_id": "<agent_id>"} after live publication. Check the returned inbound_agent_version: the number pins a publication and does not follow the head automatically. To roll back, select the earlier live publication and repeat the number assignment, then verify the pin and an incoming run's version.

After live publication and routing, check the first authorized live run. Confirm its publication version, phone routing and audio where applicable, business-action response and webhook or trigger delivery. Use a number and business record your team controls for the first phone check.

Use run review to confirm which publication executed and what the business service returned.

Content version 2a6a4f39Customer SDKMarkdown source
Release and operate a tested agent · Cumulus developer docs