Agent testing workspace

Status: Current | Last reviewed: 2026-10-20

The Vanguard testing workspace provides saved scenario tests and optional version comparisons on the live Athens test engine. Test in the agent header or preview opens /agents/{id}:draft/testing directly. Pending draft edits are saved before navigation; a save failure leaves the editor open. The original Performance view remains available through an explicit link.

The header, navigation and every tab share WorkspaceLayout.module.css: a 90rem (1,440px) outer maximum with the same responsive gutters. Individual tabs no longer apply narrower page-level width limits.

Author workflow

1. Overview defaults to testing the selected agent version at any time. Choose Quick run (one calm customer per scenario) or Full run (all four customer styles), optionally select scenarios/repetitions, then Run tests. No saved comparison point is required. Run progress opens under Results.
2. Tests → Scenarios brings each customer scenario together with its assigned success criteria. Expand the criteria in a row or edit the test to change its name, customer instructions, group and criterion assignments. Add criterion for this test stages a plain-language criterion with required or informational enforcement; saving creates it with a single-test assignment.
3. Tests → Shared success criteria manages reusable expectations and whether they apply to selected tests or every test. Preparation is explicit; editing an agent does not silently regenerate its expectations.
4. To compare versions, enable Compare with a saved version on Overview. Choose or save a named comparison point. This stores the agent, dependency snapshots, test plan, service conditions, customer profiles and assessment policy without executing conversations. The displayed count now includes both versions and repetitions. Choose Run comparison.
5. Results uses a full-width run picker above the conversation list and evidence. Previous comparisons on Overview retains paired outcomes. Review failed/uncertain conversations and follow the checks to their evidence.

Criteria authored inside a test

Test POST/PATCH accepts optional new_criteria: [{statement, enforcement}] (up to 50). saveTestWithRequirements commits the test, existing assignment changes, new criterion records and their single-test applications in the same tenant transaction. Each criterion uses the existing semantic evaluator and versioned result pins. It remains local even if the agent has only one active test; there is no automatic promotion to a global assignment. The library can later edit or explicitly reuse it. No new entity type or schema migration is required.

The existing run/assessment lock and optimistic test revision apply before writes. A stale edit does not create orphaned criteria; retries against an old revision fail before another criterion is created. Old results and baselines retain their original pins. Cancelling a client draft does not persist anything.

Tool responses authored inside a test

The test editor displays actual frozen fixture responses under Tool responses; the old link to shared tool defaults is replaced by local response controls. GET /testing/tool-responses?test_id={id}&revision={n} reads current tool contracts, the test's pinned responses, checked tool examples and response examples from this agent's current active tests. Omitting both query parameters provides the catalog for a new test. Reads never call external tools or decrypt integration credentials. An unavailable or changed contract is reported rather than silently replacing a fixture.

Test POST/PATCH accepts tool_responses edits. The service checks the test revision, operation, contract hash and source fixture ID under the agent lock, then inserts new immutable tool_mock_revisions with purpose='test_fixture'. The enclosing tenant transaction commits those revisions, the updated test and any criterion changes together. Other tests, defaults, historical test versions and comparison points keep their existing IDs. Cancel makes no write; an invalid response, stale edit or conflicting final check rolls back the transaction.

An edit targets a result within the selected variant, preserving input conditions, state transitions, delays, response order and exhaustion guards. The visual editor preserves dynamic input expressions until the user explicitly chooses a fixed value. Schema-derived response-type choices support common discriminated outputs. The backend validates result schemas and checked examples, including edited expressions on later sequence branches. Adding a response to a test that has no fixture creates a static response with checked sample inputs. The UI does not author new conditional rules or sequences.

Expected actions are displayed separately inside Success criteria. Authors can explicitly edit their expected input/result selectors. Response edits never rewrite expected actions or criterion bindings; validation checks the final assignments after all staged criterion changes. A conflict prevents saving and asks the author to review the scenario. Changing expected values does not alter tool order or remove checks.

The workspace continues to submit sandbox runs. External catalog/inline calls use frozen mock responses; missing mocks do not fall back to real services. Agent/caller/evaluator model calls run normally, and native actions and workflow logic keep their existing simulation-runtime behavior. This change adds no live-service toggle.

Test groups

A group is the saved TestDefinition.user_journey name shared by member tests; there is no separate group object. Groups organize/select scenarios and have no effect on scoring. Athens preparation supplied Order status for all 20 tests, so the default All tests view omits that redundant grouping. By group exposes the retained labels and group-wide selection/rename controls.

Group selected creates a named group for an explicit, revision-pinned selection. The form freezes those members when opened and includes tests hidden by search. Group (optional) on a test can move it to an existing/new group or leave it ungrouped. An unused name stops appearing when no active tests use it; old revisions and run pins remain.

POST /api/agents/{id}/testing/test-groups creates a group from {name, tests}; PATCH renames a complete group using {from, to, tests}. Both require ManageAgents, lock agent/preparation state, validate every member before writing, and create new revisions in one tenant transaction. Stale revisions or renamed group membership return 409. Existing destination names also return 409 to avoid an accidental merge. Manual edits retain protection from automatic regeneration.

The initial Vanguard pilot is agent 4982, Testing Workspace (Athens), copied from live Athens agent 4210. Vanguard's existing agent 4210 and its history are preserved. The imported draft and published v1 were not edited to produce a favorable result. Conflicting catalog tools were copied under separate identities and names. Existing Vanguard integration credentials are reused; no credentials were copied from Athens.

What is fixed in a comparison

agent_test_baselines stores an immutable PreparedRun and the original agent configuration. agent_test_comparisons references this baseline, the two runs, the exact request, its test-contract hash and observed scope/configuration changes. Tenant/agent foreign keys and RLS protect the relationship. Update/delete guards retain comparison evidence while its owning agent exists.

Both runs are committed together. The server freezes identical situation revisions, required/informational expectations, selected customer profiles, repetitions, fixture responses, initial state, template variables, customer/verifier models, assessment model/prompt and harness version. Each version uses its own agent implementation and dependency snapshot. Changed tool contracts or incompatible service conditions reject submission before either run is queued.

Saving a new comparison point explicitly adopts the current test plan. The author can choose the old agent version for that new point and rerun both versions under the revised expectations. A changed test plan is never reported as an improvement in the agent.

Only matched, completed original pass/fail results from first attempts produce improved, regressed or unchanged. Retries, unresolved evidence, execution failures and assessment errors are uncertain. Successful recovery and first-attempt passes are separate counts. Attempt history and its evidence remain available in run progress; reassessments cannot rewrite original results.

Additive integration

The captured source is Athens feature/test-voice-agents at 5d9ce0661c3c47131378f9cabb144b8202b9467e, plus its captured working changes. apps/ai-manager/lib/testing/athens-source.json records the initial file hashes. The implementation lives under lib/testing, components/testing and /api/agents/{id}/testing. It preserves Vanguard's existing lib/performance and lib/agent-testing consumers.

Migration 076_agent_testing_workspace.sql adds canonical test, requirement, fixture, conversation-content and result records. Preparation uses the separate agent_test_preparations table. 077_agent_testing_comparisons.sql adds comparison points and paired checks. Apply these forward migrations to an existing database; do not apply Athens' legacy retirement/consolidation migrations or reset a city DB. Both migrations are registered in the city runner and folded into the fresh-boot baseline.

Shared simulation storage remains. New workers accept only spec.entity_model=2 and publish to agent_testing_conversations; legacy workers exclude that model. The shared Python runtime accepts both old case identities and new execution/test identities. It selects per-situation fixtures without changing the saved snapshot. Run the additional workers as described in worker operations.

apps/ai-manager/scripts/import-athens-testing-agent.ts performs a transactional, insert-only import from a private export. It allocates a new agent ID, remaps owned UUIDs and colliding tool names, recalculates fixture hashes, maps integrations without copying secrets, and writes a private receipt. It aborts if the chosen copy name already exists. Keep source exports, receipts and database backups outside Git.

Scope and limitations

This implements the first two phases of the Athens review. Automatic repair/generation loops and production monitoring are later phases. Quick run reduces customer styles, rather than inferring a smaller affected-situation set. Model usage is incurred; the interface does not invent a time or monetary estimate.

Voice adapters retain their declared text, spoken-reply and audio modes. Their assessments cover captured words and events, not speech quality, interruption handling or carrier/device routing. Copilot replay retains its existing adapter; this pilot's live execution validation uses chat.

These are application APIs, separate from the public integration OpenAPI: endpoint contracts.

Vanguard pilot verification (2026-10-20)

The migration was applied twice to a separate PostgreSQL container and the fresh-boot schema drift check passed. The live Vanguard database was backed up before either forward migration. Import verification retained all source test versions and checked that existing Vanguard 4210 still identifies MedTech Order Concierge (Chat). Private backups, export, receipt and logs are under /tmp/athens-vanguard-phase12-20261020; they are not repository artifacts.

Two paired checks exercised six live chat conversations: order delivered, verification declined, and an initial request to skip verification followed by cooperation. Both versions passed the delivered situation. Both failed the two other situations on their first attempts. One finding requires the exact masked phone string rather than its suffix; the other reports an omitted Northwind Retail greeting. These are saved automated findings for review, not a claim that identity verification was bypassed or that the agent is qualified for production. No prompts or expectations were edited to improve these observed results.

The checks also exercised draft-save synchronization, idempotent submission, rejection of mismatched request keys and foreign-agent baselines, saved result navigation, attempt history and mobile layout. Focused TypeScript tests passed 123 cases (45 database-gated cases skipped); Python runtime tests passed 45 (1 database-gated case skipped). Full-app typechecking, scoped typechecking and lint passed without errors; inherited lint warnings remain. The final browser walkthrough reported no page errors and followed a failed check to its highlighted conversation evidence.

The October 20 UX revision was verified through the live agent editor's Test entry, ordinary/paired conversation counts, inline scenario and group edits, conflict feedback, criterion navigation, and saved conversation evidence at 1440px and 390px widths. The group API rejected stale membership and blank names; a no-op rename retained every revision. Focused tests passed 31 cases. A single Quick run of the delivered-order scenario completed and passed on its first attempt, exercising the new Overview → Results path without a comparison point. Existing test definitions and historical runs were not altered by this check.

The subsequent width/authoring refinement passed 33 focused tests and four PostgreSQL integration tests in a disposable database. These verify local-only criteria (including the first-test case), future-test isolation, retained pins, stale retries, run locks and selected-group isolation. Browser checks measured identical tab widths at 390px, 1440px and 1920px and exercised authoring controls with intercepted writes, preserving the pilot's saved plan.

The per-test Tool responses refinement adds focused tests for immutable response revisions, same-agent example reuse, test isolation, schema/expression validation, stale/foreign pins, transactional rollback, explicit expected-value changes and new-test authoring. Browser verification covers existing response selection, typed customization, preserved input references, conflict feedback, cancellation and desktop/mobile layouts. Authoring checks use intercepted browser writes and a disposable PostgreSQL fixture so the imported plan remains unchanged. Validation passed 40 focused unit checks, three PostgreSQL integration checks, full-app typechecking and scoped lint. Live API rejection checks retained the pilot's 2,253 test versions and 416 fixture revisions, including after a response/expected-action conflict rolled back. Both existing-test and new-test browser flows passed; new-test authoring combined a typed service-error response with an ad-hoc criterion in one intercepted save.