CumulusDocs
Open Studio
On this page

Start a test run

Run suites against one publication version. Follow progress, then inspect case verdicts and trial evidence.

Operation ID: test_runs.create.

Request

POST /api/v1/test-runs

  • TypeScript SDK: client.test_runs.create (path arguments use path).
  • MCP tool: sdk_client_call with operationId: "test_runs.create" (path arguments use input.path). Legacy alias: test_runs_create. Connect to your authenticated /mcp endpoint with an API key for the same environment.
  • Minimum API key role: member.
  • Idempotency: Required. Send Idempotency-Key (TypeScript SDK: idempotencyKey) and keep it for retries.

Read resource IDs from earlier responses and use a key for the same environment.

Request example

Use a published version and suite IDs returned by test_suites.create or test_suites.list. Poll test_runs.get, then inspect test_runs.results.

Set CUMULUS_BASE_URL to your workspace API URL and CUMULUS_API_KEY to a key for that environment. Set IDEMPOTENCY_KEY to a new UUID for this action and keep it for retries. Example values are synthetic.

cURL

curl
curl --fail-with-body -sS -X POST "${CUMULUS_BASE_URL%/}/api/v1/test-runs" \
  -H "Authorization: Bearer $CUMULUS_API_KEY" \
  -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
  -H "Content-Type: application/json" \
  --data '{
  "agent_id": "ticket-triage",
  "version": 0,
  "suite_ids": [
    "11111111-1111-4111-8111-111111111111"
  ]
}'

TypeScript

typescript
import { Cumulus } from "@cumulus/sdk";

const baseUrl = process.env.CUMULUS_BASE_URL;
const apiKey = process.env.CUMULUS_API_KEY;
if (!baseUrl || !apiKey) throw new Error("Set CUMULUS_BASE_URL and CUMULUS_API_KEY");
const client = new Cumulus({ baseUrl, apiKey });

const idempotencyKey = process.env.IDEMPOTENCY_KEY;
if (!idempotencyKey) throw new Error("Set IDEMPOTENCY_KEY for this request");

const response = await client.test_runs.create({
  "body": {
    "agent_id": "ticket-triage",
    "version": 0,
    "suite_ids": [
      "11111111-1111-4111-8111-111111111111"
    ]
  },
  idempotencyKey,
});

Integration notes

Send agent_id, a published version and suite_ids. Use case_ids or trials to select cases or override the trial count. Call test_runs_estimate first to review trial count, cost and duration.

Follow test_runs_get until completion or cancellation. Read test_runs_results for case and assertion verdicts, then test_runs_trial for the transcript and evidence behind a verdict.

Input schema

Path, query and header parameters

Idempotency-Key

header parameter · Required.

Type: string. minLength: 1. maxLength: 256.

JSON request body

FieldType or valuesRequiredDescription and limits
agent_idstringRequiredAgent identifier in the environment selected by your API key. minLength: 1. maxLength: 256. pattern: ^[A-Za-z0-9][A-Za-z0-9_.:-]*$.
versionintegerRequiredAt least 0. At most 2147483647.
suite_idsarray of string (uuid)RequiredIdentifiers of the test suites to execute. minItems: 1. maxItems: 10.
case_idsarray of stringOptionalminItems: 1. maxItems: 100.
trialsintegerOptionalAt least 1. At most 10.
baseline_test_run_idstring (uuid)Optionalpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
Complete OpenAPI schema
OpenAPI JSON
{
  "parameters": [
    {
      "name": "Idempotency-Key",
      "in": "header",
      "required": true,
      "schema": {
        "type": "string",
        "minLength": 1,
        "maxLength": 256
      }
    }
  ],
  "requestBody": {
    "required": true,
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#/components/schemas/TestRunCreateRequest"
        }
      }
    }
  }
}

Responses

HTTP 200

Run test suites against one publication version (201; an idempotent replay returns 200)

application/json

FieldType or valuesRequiredDescription and limits
owner_idstringRequiredminLength: 1.
environmentstringRequiredminLength: 1.
tenant_idstring (uuid)Requiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
test_run_idstring (uuid)RequiredIdentifier of the test run whose progress or results you are reading. pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
agent_idstringRequiredAgent identifier in the environment selected by your API key. minLength: 1. maxLength: 256. pattern: ^[A-Za-z0-9][A-Za-z0-9_.:-]*$.
publicationobjectRequired—
publication.agent_idstringRequiredAgent identifier in the environment selected by your API key. minLength: 1. maxLength: 256. pattern: ^[A-Za-z0-9][A-Za-z0-9_.:-]*$.
publication.versionintegerRequiredAt least 0. At most 2147483647.
publication.digeststringRequiredpattern: ^sha256:[a-f0-9]{64}$.
purpose"manual" · "publish_gate" · "schedule" · "release"Required—
suitesarray of objectRequiredminItems: 1. maxItems: 10.
suites[].suite_idstring (uuid)RequiredIdentifier of the test suite. pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
suites[].revisionintegerRequiredGreater than 0. At most 2147483647.
suites[].digeststringRequiredpattern: ^sha256:[a-f0-9]{64}$.
state"queued" · "running" · "completed" · "cancelled"Required—
verdict"passed" · "failed" · "errored" or nullRequired—
progressobjectRequired—
progress.trialsintegerRequiredAt least 0. At most 9007199254740991.
progress.admittedintegerRequiredAt least 0. At most 9007199254740991.
progress.settledintegerRequiredAt least 0. At most 9007199254740991.
summaryobject or nullRequired—
summary.casesintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.passedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.flaky_passedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.failedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.erroredintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.invalidintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.trialsintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.regressedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.fixedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.guard_eventsintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
baseline_test_run_idstring (uuid) or nullRequiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
judge_modelobject or nullRequired—
judge_model · combined section 1object—Fields for this schema branch.
judge_model.model_idstringRequired in a non-null parentminLength: 1. pattern: \S.
judge_model.providerstringRequired in a non-null parentminLength: 1. pattern: \S.
judge_model.tier"frontier" · "cumulus-served" · "customer-gpu"Required in a non-null parent—
costobjectRequired—
cost.estimated_usdnumberRequiredAt least 0.
cost.measured_usdnumber or nullRequiredAt least 0.
created_byobjectRequired—
created_by · variant 1object—Fields for this schema branch.
created_by.owner_idstringRequiredminLength: 1.
created_by.environmentstringRequiredminLength: 1.
created_by.kind"user"Required—
created_by.idstring (uuid)Requiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
created_by · variant 2object—Fields for this schema branch.
created_by.owner_idstringRequiredminLength: 1.
created_by.environmentstringRequiredminLength: 1.
created_by.kind"api_key"Required—
created_by.idstring (uuid)Requiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
created_by · variant 3object—Fields for this schema branch.
created_by.owner_idstringRequiredminLength: 1.
created_by.environmentstringRequiredminLength: 1.
created_by.kind"system"Required—
created_by.idstringRequiredminLength: 1. maxLength: 256.
created_atstring (date-time)Requiredpattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:.\d+)?(?:Z))$.
finished_atstring (date-time) or nullRequiredpattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:.\d+)?(?:Z))$.

HTTP 201

Run test suites against one publication version (201; an idempotent replay returns 200)

application/json

FieldType or valuesRequiredDescription and limits
owner_idstringRequiredminLength: 1.
environmentstringRequiredminLength: 1.
tenant_idstring (uuid)Requiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
test_run_idstring (uuid)RequiredIdentifier of the test run whose progress or results you are reading. pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
agent_idstringRequiredAgent identifier in the environment selected by your API key. minLength: 1. maxLength: 256. pattern: ^[A-Za-z0-9][A-Za-z0-9_.:-]*$.
publicationobjectRequired—
publication.agent_idstringRequiredAgent identifier in the environment selected by your API key. minLength: 1. maxLength: 256. pattern: ^[A-Za-z0-9][A-Za-z0-9_.:-]*$.
publication.versionintegerRequiredAt least 0. At most 2147483647.
publication.digeststringRequiredpattern: ^sha256:[a-f0-9]{64}$.
purpose"manual" · "publish_gate" · "schedule" · "release"Required—
suitesarray of objectRequiredminItems: 1. maxItems: 10.
suites[].suite_idstring (uuid)RequiredIdentifier of the test suite. pattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
suites[].revisionintegerRequiredGreater than 0. At most 2147483647.
suites[].digeststringRequiredpattern: ^sha256:[a-f0-9]{64}$.
state"queued" · "running" · "completed" · "cancelled"Required—
verdict"passed" · "failed" · "errored" or nullRequired—
progressobjectRequired—
progress.trialsintegerRequiredAt least 0. At most 9007199254740991.
progress.admittedintegerRequiredAt least 0. At most 9007199254740991.
progress.settledintegerRequiredAt least 0. At most 9007199254740991.
summaryobject or nullRequired—
summary.casesintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.passedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.flaky_passedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.failedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.erroredintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.invalidintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.trialsintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.regressedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.fixedintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
summary.guard_eventsintegerRequired in a non-null parentAt least 0. At most 9007199254740991.
baseline_test_run_idstring (uuid) or nullRequiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
judge_modelobject or nullRequired—
judge_model · combined section 1object—Fields for this schema branch.
judge_model.model_idstringRequired in a non-null parentminLength: 1. pattern: \S.
judge_model.providerstringRequired in a non-null parentminLength: 1. pattern: \S.
judge_model.tier"frontier" · "cumulus-served" · "customer-gpu"Required in a non-null parent—
costobjectRequired—
cost.estimated_usdnumberRequiredAt least 0.
cost.measured_usdnumber or nullRequiredAt least 0.
created_byobjectRequired—
created_by · variant 1object—Fields for this schema branch.
created_by.owner_idstringRequiredminLength: 1.
created_by.environmentstringRequiredminLength: 1.
created_by.kind"user"Required—
created_by.idstring (uuid)Requiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
created_by · variant 2object—Fields for this schema branch.
created_by.owner_idstringRequiredminLength: 1.
created_by.environmentstringRequiredminLength: 1.
created_by.kind"api_key"Required—
created_by.idstring (uuid)Requiredpattern: ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$.
created_by · variant 3object—Fields for this schema branch.
created_by.owner_idstringRequiredminLength: 1.
created_by.environmentstringRequiredminLength: 1.
created_by.kind"system"Required—
created_by.idstringRequiredminLength: 1. maxLength: 256.
created_atstring (date-time)Requiredpattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:.\d+)?(?:Z))$.
finished_atstring (date-time) or nullRequiredpattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:.\d+)?(?:Z))$.

Error responses

HTTPDescription
4XXTyped error (ApiError)
5XXTyped error (ApiError)

Errors use the shared error envelope. Inspect error.code, retryable, outcome and details before retrying.

Complete OpenAPI schema
OpenAPI JSON
{
  "200": {
    "description": "Run test suites against one publication version (201; an idempotent replay returns 200)",
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#/components/schemas/TestRun"
        }
      }
    }
  },
  "201": {
    "description": "Run test suites against one publication version (201; an idempotent replay returns 200)",
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#/components/schemas/TestRun"
        }
      }
    }
  },
  "4XX": {
    "description": "Typed error (ApiError)",
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#/components/schemas/ApiError"
        }
      }
    }
  },
  "5XX": {
    "description": "Typed error (ApiError)",
    "content": {
      "application/json": {
        "schema": {
          "$ref": "#/components/schemas/ApiError"
        }
      }
    }
  }
}

Referenced schemas

Expand the complete OpenAPI definitions for integrations that generate their own clients.

Complete OpenAPI schema
OpenAPI JSON
{
  "components": {
    "schemas": {
      "TestRunCreateRequest": {
        "type": "object",
        "properties": {
          "agent_id": {
            "type": "string",
            "minLength": 1,
            "maxLength": 256,
            "pattern": "^[A-Za-z0-9][A-Za-z0-9_.:-]*$"
          },
          "version": {
            "type": "integer",
            "minimum": 0,
            "maximum": 2147483647
          },
          "suite_ids": {
            "minItems": 1,
            "maxItems": 10,
            "type": "array",
            "items": {
              "type": "string",
              "format": "uuid",
              "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
            }
          },
          "case_ids": {
            "minItems": 1,
            "maxItems": 100,
            "type": "array",
            "items": {
              "type": "string",
              "minLength": 1,
              "maxLength": 64,
              "pattern": "^[a-z0-9][a-z0-9_-]*$"
            }
          },
          "trials": {
            "type": "integer",
            "minimum": 1,
            "maximum": 10
          },
          "baseline_test_run_id": {
            "type": "string",
            "format": "uuid",
            "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
          }
        },
        "required": [
          "agent_id",
          "version",
          "suite_ids"
        ],
        "additionalProperties": false
      },
      "TestRun": {
        "type": "object",
        "properties": {
          "owner_id": {
            "type": "string",
            "minLength": 1
          },
          "environment": {
            "type": "string",
            "minLength": 1
          },
          "tenant_id": {
            "type": "string",
            "format": "uuid",
            "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
          },
          "test_run_id": {
            "type": "string",
            "format": "uuid",
            "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
          },
          "agent_id": {
            "type": "string",
            "minLength": 1,
            "maxLength": 256,
            "pattern": "^[A-Za-z0-9][A-Za-z0-9_.:-]*$"
          },
          "publication": {
            "$ref": "#/components/schemas/PublicationRef"
          },
          "purpose": {
            "type": "string",
            "enum": [
              "manual",
              "publish_gate",
              "schedule",
              "release"
            ]
          },
          "suites": {
            "minItems": 1,
            "maxItems": 10,
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "suite_id": {
                  "type": "string",
                  "format": "uuid",
                  "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
                },
                "revision": {
                  "type": "integer",
                  "minimum": 0,
                  "exclusiveMinimum": true,
                  "maximum": 2147483647
                },
                "digest": {
                  "type": "string",
                  "pattern": "^sha256:[a-f0-9]{64}$"
                }
              },
              "required": [
                "suite_id",
                "revision",
                "digest"
              ],
              "additionalProperties": false
            }
          },
          "state": {
            "type": "string",
            "enum": [
              "queued",
              "running",
              "completed",
              "cancelled"
            ]
          },
          "verdict": {
            "nullable": true,
            "type": "string",
            "enum": [
              "passed",
              "failed",
              "errored"
            ]
          },
          "progress": {
            "type": "object",
            "properties": {
              "trials": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "admitted": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "settled": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              }
            },
            "required": [
              "trials",
              "admitted",
              "settled"
            ],
            "additionalProperties": false
          },
          "summary": {
            "nullable": true,
            "type": "object",
            "properties": {
              "cases": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "passed": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "flaky_passed": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "failed": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "errored": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "invalid": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "trials": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "regressed": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "fixed": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              },
              "guard_events": {
                "type": "integer",
                "minimum": 0,
                "maximum": 9007199254740991
              }
            },
            "required": [
              "cases",
              "passed",
              "flaky_passed",
              "failed",
              "errored",
              "invalid",
              "trials",
              "regressed",
              "fixed",
              "guard_events"
            ],
            "additionalProperties": false
          },
          "baseline_test_run_id": {
            "nullable": true,
            "type": "string",
            "format": "uuid",
            "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
          },
          "judge_model": {
            "nullable": true,
            "allOf": [
              {
                "$ref": "#/components/schemas/ModelIdentity"
              }
            ]
          },
          "cost": {
            "type": "object",
            "properties": {
              "estimated_usd": {
                "type": "number",
                "minimum": 0
              },
              "measured_usd": {
                "nullable": true,
                "type": "number",
                "minimum": 0
              }
            },
            "required": [
              "estimated_usd",
              "measured_usd"
            ],
            "additionalProperties": false
          },
          "created_by": {
            "$ref": "#/components/schemas/PrincipalRef"
          },
          "created_at": {
            "type": "string",
            "format": "date-time",
            "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z))$"
          },
          "finished_at": {
            "nullable": true,
            "type": "string",
            "format": "date-time",
            "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z))$"
          }
        },
        "required": [
          "owner_id",
          "environment",
          "tenant_id",
          "test_run_id",
          "agent_id",
          "publication",
          "purpose",
          "suites",
          "state",
          "verdict",
          "progress",
          "summary",
          "baseline_test_run_id",
          "judge_model",
          "cost",
          "created_by",
          "created_at",
          "finished_at"
        ],
        "additionalProperties": false
      },
      "PublicationRef": {
        "type": "object",
        "properties": {
          "agent_id": {
            "type": "string",
            "minLength": 1,
            "maxLength": 256,
            "pattern": "^[A-Za-z0-9][A-Za-z0-9_.:-]*$"
          },
          "version": {
            "type": "integer",
            "minimum": 0,
            "maximum": 2147483647
          },
          "digest": {
            "type": "string",
            "pattern": "^sha256:[a-f0-9]{64}$"
          }
        },
        "required": [
          "agent_id",
          "version",
          "digest"
        ],
        "additionalProperties": false
      },
      "ModelIdentity": {
        "type": "object",
        "properties": {
          "model_id": {
            "type": "string",
            "minLength": 1,
            "pattern": "\\S"
          },
          "provider": {
            "type": "string",
            "minLength": 1,
            "pattern": "\\S"
          },
          "tier": {
            "type": "string",
            "enum": [
              "frontier",
              "cumulus-served",
              "customer-gpu"
            ]
          }
        },
        "required": [
          "model_id",
          "provider",
          "tier"
        ],
        "additionalProperties": false
      },
      "PrincipalRef": {
        "oneOf": [
          {
            "type": "object",
            "properties": {
              "owner_id": {
                "type": "string",
                "minLength": 1
              },
              "environment": {
                "type": "string",
                "minLength": 1
              },
              "kind": {
                "type": "string",
                "enum": [
                  "user"
                ]
              },
              "id": {
                "type": "string",
                "format": "uuid",
                "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
              }
            },
            "required": [
              "owner_id",
              "environment",
              "kind",
              "id"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "owner_id": {
                "type": "string",
                "minLength": 1
              },
              "environment": {
                "type": "string",
                "minLength": 1
              },
              "kind": {
                "type": "string",
                "enum": [
                  "api_key"
                ]
              },
              "id": {
                "type": "string",
                "format": "uuid",
                "pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})$"
              }
            },
            "required": [
              "owner_id",
              "environment",
              "kind",
              "id"
            ],
            "additionalProperties": false
          },
          {
            "type": "object",
            "properties": {
              "owner_id": {
                "type": "string",
                "minLength": 1
              },
              "environment": {
                "type": "string",
                "minLength": 1
              },
              "kind": {
                "type": "string",
                "enum": [
                  "system"
                ]
              },
              "id": {
                "type": "string",
                "minLength": 1,
                "maxLength": 256
              }
            },
            "required": [
              "owner_id",
              "environment",
              "kind",
              "id"
            ],
            "additionalProperties": false
          }
        ]
      },
      "ApiError": {
        "type": "object",
        "properties": {
          "error": {
            "type": "object",
            "properties": {
              "code": {
                "type": "string",
                "enum": [
                  "unauthenticated",
                  "forbidden",
                  "not_found",
                  "conflict",
                  "invalid_request",
                  "idempotency_conflict",
                  "policy_denied",
                  "capability_unavailable",
                  "provider_unavailable",
                  "knowledge_unavailable",
                  "publication_invalid",
                  "run_not_active",
                  "rate_limited",
                  "outcome_unknown",
                  "internal"
                ]
              },
              "message": {
                "type": "string"
              },
              "retryable": {
                "type": "boolean"
              },
              "outcome": {
                "type": "string",
                "enum": [
                  "none",
                  "unknown"
                ]
              },
              "details": {
                "type": "object",
                "additionalProperties": {}
              }
            },
            "required": [
              "code",
              "message",
              "retryable",
              "outcome"
            ],
            "additionalProperties": false
          }
        },
        "required": [
          "error"
        ],
        "additionalProperties": false
      }
    }
  }
}

Errors and recovery

See API error handling and Troubleshooting.

Content version 65213341Customer SDKMarkdown source
Start a test run · Cumulus developer docs