Skip to content

Agent evaluation API

Status: Current | Last reviewed: 2026-10-20

The additive testing workspace uses /api/agents/{agentId}/testing. Legacy endpoints below remain available. Reading plans/results and starting runs/comparisons require AIManager:RunAgents. Authoring definitions and saving comparison points require AIManager:ManageAgents. All requests enforce the active tenant and owning agent.

Method and suffix under /testing Behavior
GET /improvement?version=draft&baseline_id={uuid} Saved comparison points, changed configuration, changed test scope, history and actual attention findings
POST /baselines {name, version, request_key} saves an immutable comparison point; returns 201 {baseline}
POST /comparisons {baseline_id, version, mode, test_ids?, repetitions?, hypothesis?, request_key, consent:true} atomically starts both runs; returns 202 {id}
GET /comparisons/{id} Matched original results, first-attempt/recovery counts, retained attempts and scope changes
GET /tests, GET /requirements Versioned scenario and success criterion inventory
GET /tool-responses?test_id={uuid}&revision={n} Current operation schemas, saved response branches and reusable examples for this test; omit both parameters for a new test
POST /test-groups ManageAgents-only {name, tests: [{id, revision}]} creates a group from selected active tests; existing names and stale selections return 409
POST /tests, PATCH /tests/{id} Definitions and assignments can include new_criteria and tool_responses; criteria and immutable response revisions save atomically with this test
PATCH /test-groups ManageAgents-only {from: string \| null, to: string, tests: [{id, revision}]} atomically renames all active group members; preserves old revisions
GET /runs, GET /runs/{id} New-model run history and execution/attempt evidence
POST /runs Run explicitly selected situations and customer profiles without a before/after comparison
PATCH /runs/{id} ManageAgents-only {action:"cancel"} or {action:"hide"}; hiding does not delete evidence
GET /results, GET /results/{id} Immutable contracts and original or appended assessments

new_criteria accepts at most 50 entries with nonblank statements of up to 16,000 characters. enforcement is required (default) or informational. For a new test, submit {definition, new_criteria, requirements?}; editing uses the existing {action:"edit", revision, definition, requirements?, new_criteria?} shape. The test and local criteria commit together. Active runs/assessments and stale test revisions reject edits before new criteria are written. These are standard versioned semantic criteria, with no automatic global assignment.

tool_responses accepts up to 50 operation edits: {tool_ref, operation_id, contract_hash, fixture_revision_id, variant, responses: [{rule_index, result}], example_input?}. Use rule_index: null for the variant fallback or a zero-based rule index for a conditional response. fixture_revision_id is the current test-owned revision, or null when adding a response to a test without one. New responses use variant scenario, rule_index: null and checked example_input values. The editor copies reusable examples into result; it does not save a live pointer to a shared default.

Both new-test wrappers and edit requests accept this optional field alongside requirements and new_criteria. Input conditions, state updates and sequence guards cannot be changed through this endpoint. Invalid output schemas or conflicting final expected actions/criterion bindings return 400; stale test/fixture/contract references return 409. The enclosing transaction rolls back every write on failure. Tool-response reads require RunAgents and writes require ManageAgents. No live service calls occur while reading or saving.

Group renames require the complete current membership and revisions, even when the UI is filtered. Stale membership/revisions and an existing destination group return 409; invalid or blank names return 400. A group name is the test definition's optional user_journey field. No schema migration is required.

Comparison mode is quick or full; repetitions are 1–5. Explicit test IDs must be unique members of the saved plan. The comparison limit is 2,000 total conversations across both versions. Versions accept draft, published or v<number>. request_key is a UUID: identical retries return the existing record; reusing it for different inputs returns 409.

Changed harness versions, tool contracts or inconsistent service conditions return 409 before either run is queued. A changed active test plan is reported separately and does not overwrite the baseline's expectations.

The application endpoints are documented here rather than added to the public /api/v1 integration OpenAPI.

These are authenticated ai-manager application endpoints. They use the active tenant and AIManager:ManageAgents or AIManager:ManageTools permission. They are separate from the public /api/v1 integration API.

Method and path Behavior
GET /api/agents/{agentId}/performance/test-cases List up to 1,000 saved cases and their latest results; ?id={id} returns one full definition
POST /api/agents/{agentId}/performance/test-cases Save a caller scenario and expected outcome
PATCH /api/agents/{agentId}/performance/test-cases Replace a definition using id, current revision and the full input; stale revision returns 409
POST /api/agents/{agentId}/performance/simulations Validate and pin a target, persist jobs, return 202 with {batch}
GET /api/agents/{agentId}/performance/simulations List the latest 100 batches
GET /api/agents/{agentId}/performance/simulations?view=tests Compact latest 50 runs with aggregate product statuses
GET /api/agents/{agentId}/performance/simulations/{batchId}?after={ordinal} Return batch, job/review progress and at most 100 jobs; continue with next_after
GET /api/agents/{agentId}/performance/simulations/{batchId}?view=tests&after={ordinal} Compact progress with latest agent replies and result rationale, 100 jobs per page
POST /api/agents/{agentId}/performance/simulations/{batchId} {action:"cancel"} revokes unfinished jobs; {action:"retry_reviews"} retries failed scoring
DELETE /api/agents/{agentId}/performance/simulations {ids:[...]} deletes finished batch records; conversations and reviews remain
GET /api/tools/{id}/testing Return operation contracts, current hashes, latest mock revisions and staleness
POST /api/tools/{id}/testing Generate a mock draft or validate and save an immutable revision
POST /api/tools/testing Generate/check inline tool mocks before saving their agent/workflow

A simulation request carries a UUID Idempotency-Key header. Repeating it with the same specification returns the existing batch; reusing it for a different specification returns 409.

The saved-test picker submits the following shape. test_case_ids contains 1–1,000 distinct IDs owned by the current tenant and agent; disabled cases are rejected. Definitions, revisions and linked goldens are snapshotted during submission. An idempotent retry returns the original snapshot even if a case was subsequently edited.

{
  "consent": true,
  "test_case_ids": ["b791523e-f452-4bd2-888e-e064c7d59b01"],
  "version": "draft",
  "mode": "auto",
  "sandbox": true,
  "concurrency": 10,
  "repetitions": 1
}

Case input includes name, optional description, persona (required name and goal; optional context, opening line, details, language and emotional state), expected_outcome, optional owned golden_id, max_turns (1–40, default 20) and enabled (default true). Selected-case requests are capped at 2 MB after snapshots and 10,000 conversations including repetitions.

The existing caller-persona API remains available:

{
  "consent": true,
  "version": "draft",
  "profile": "default",
  "mode": "auto",
  "sandbox": true,
  "concurrency": 100,
  "repetitions": 1000,
  "max_turns": 12,
  "personas": [
    {
      "name": "Booking caller",
      "goal": "Confirm a booking and receive its reference",
      "emotional_state": "neutral",
      "language": "en-US"
    }
  ],
  "variants": {},
  "initial_state": {}
}

version accepts draft, published or v<number>. mode accepts auto, text, transcript or audio. GPT Live requires audio. At most 25 personas, 10,000 total conversations and 1–1,000 requested concurrent conversations are accepted. Shared worker/provider limits still govern admission.

An agent needs a scoring criterion (a saved case's expected outcome, a Goal or Golden) and a tenant OpenAI integration for the caller/judge. In sandbox mode, every external operation needs a checked mock matching its current input/output contract. Validation failures return 400 with a concrete error and a product-facing message before jobs are submitted.

Compact statuses distinguish waiting, starting, running, checking, passed, failed, error, unscored and stopped. Counts cover the entire run, not just the visible page. A saved-case verdict uses its explicit expected outcome and all must-goals, not overall score alone. test_case_revision on the latest result lets the picker distinguish an edited case from the definition previously run.

For tool mocks, submit {action:"generate", operation_id, instructions?} to generate a draft. Save with {action:"save", operation_id, contract_hash, definition}. A changed contract returns 409; invalid fixtures/examples return 400. See tool mock format.

Provider and harness errors belong to attempt status/evidence. Reaching a turn limit still produces a scored conversation. Scoring has its own lease and can be retried without starting another chat. See worker operations.