Agent evaluation¶

Status: Current | Last reviewed: 2026-10-06

The 2026-09-30-decoded-tool-evidence reviewer presentation includes a readable copy of JSON encoded inside tool arguments. Literal quotes can refer to that displayed copy; the raw archive remains unchanged. This fixes rejected quotes of actual A2UI properties without accepting paraphrases or invented fields. Results retain the reviewer-policy version, and a qualification run must not combine policies silently.

Automatic mode uses synthetic audio for cascaded profiles containing STT and TTS, as well as GPT Live. This exercises provider turn detection and playback alongside agent behavior. Explicit text diagnostics are not acoustic parity.

Saved cases can provide E.164 caller/dialed numbers and a fresh Connect identity through testing.call_context. The worker owns identity generation and credentials. Studio run options support repeated cases and explicit connected tools, while sandbox responses remain the default. See demo qualification for the repeat-run and city parity procedure.

Passive Copilots use saved testing.copilot_replay turns and component assertions rather than the voice caller loop. The worker sends ordered indexes; the authenticated runtime supplies the pinned speaker and text. UI render, suggestion and worker-completion evidence is retained with each attempt. The target pins catalog schemas and tool_ids dependencies. Component checks cannot pass on a later render, absent checkpoint, malformed payload, unfinished worker or silently failed provider request. Replay checks published component state; it excludes audio transcription, browser layout and manual dismissal. The editor exposes startup/turn waits, present/absent component checks, optional text matching and exact field subsets. testing.tool_variants can select a saved tool-owned variant for an individual sandbox case.

testing.supervisor.delivery="chat" exercises a Salesforce-style escalation workflow that finishes before the human responds. The worker requires the configured workflow to resolve in the same tenant, attempt and root conversation, then sends the authored human message through the real supervisor agent. Successful workflow dispatch alone cannot create a decision. The usual waiting-workflow callback remains the default. Chat delivery requires agent mode; duplicate/late callback tests remain specific to workflow delivery.

Sandbox streams attach a headless visual surface, suppressing the multimodal share-link SMS while retaining authentication consent checks. Provider errors are captured at the pipeline worker, including errors travelling upstream, so a provider rejection fails promptly with its cause instead of a reply timeout. Recognized provider failures in sandbox runs receive one fresh-conversation retry with a 10–15 second delay and the same pinned settings. The test window shows Retrying, then the new attempt, and keeps Attempt history even when the retry succeeds. Failed attempts remain separate from scored results. See worker recovery policy for failure classification and limits. Nova startup notices remain in the initial context until the provider has opened its prompt and audio input. A stream object and allocated prompt name alone do not authorize live text: sending it earlier caused deterministic No open prompt found failures before either rerun reached its first reply. The end-session handler also registers a real provider callback. Nova checks registration before broadcasting tool frames, so a frame-only handler could not execute its goodbye tool. Callback-backed sessions use one result path; the frame fallback remains for session-switch routing and providers without callback registration.

RunTestsDialog is shared by Performance → Simulations and the editor's Preview headers (sidebar, full-screen, recording and expanded-node layouts). Preview seeds draft or the exact vN being viewed. Its draft submission awaits a successful collab flush before the simulation API snapshots the target; a failed flush leaves the picker retryable and creates no batch. Starting a batch keeps the dialog open on live progress without remounting the manual preview.

RunTestsWorkspace lists reusable agent_test_cases, supports selection and Play, and restores an active run when reopened. Completed runs appear in a latest-results snapshot and the Runs tab. The picker supplies sandbox defaults and automatic interaction mode; runtime profiles and fixture JSON are not part of this product flow. TestCaseEditor asks for a caller request and expected outcome, with optional caller details.

Each started test exposes View conversation. TestConversationViewer mounts the existing Surface App with a signed, conversation-bound embed token: hideInput, preserveConversation, hideContext and hideToolCalls are true. It starts on the normal transcript and retains the multimodal switcher for voice agents. Surface App replays history and subscribes to live events; the viewer does not create, send to, interrupt or end a session. Visual controls are read only, including supervisor cards; passive embeds skip avatar startup. Viewer tokens request a one-hour lifetime through the existing bounded mint API, since the session cookie cannot outlive its signed token. Reload chat mints a fresh token for the same conversation. View supervisor resolves the child conversation's actual owner when minting, and branding requests carry that owner hint through the shared-cookie proxy.

Progress polling continues while a conversation is open. An automatic retry leaves the viewer on its selected attempt and offers View current conversation. Back to test progress restores the run; Reset view clears local selection, search and navigation and returns to the test list. Neither affects execution. Closing unmounts the viewer; reopening restores active progress or the saved results snapshot. Historical attempts remain viewable.

Capability responses and supervisor conversations¶

Cases now have a testing object: named capability scenario selections, optional supervisor/authentication/payment/UI responses, and require_end_session. Responses during the test exposes the common choices in the case editor. Named scenarios and inline tool fixtures are authored in the capability's Studio testing.json; saved tools continue to own their operation mocks. Submission validates contracts and pins the expanded capability manifests, scenarios, workflow and supervisor agent alongside the main agent. The pinned internal-tool policy includes set_variables when an agent declares mutable variables: its real session-state handler runs without an external mock. An external operation cannot reuse that internal handler's name. This policy is part of the target hash. Procedure complete_when.tool values describe the completion of selected tools; they are not additional database tool IDs.

Supervisor decision mode runs the real request workflow, suspends its wait, and simulates the external decision. Agent mode additionally starts the configured supervisor through a second real orchestrator stream and the normal transcript relay. The simulated human speaks in that chat; the supervisor's inject_message_to_conversation tool must deliver the decision before the workflow resumes. Approval, rejection, clarification then approval, and expiry are supported. Repeated callbacks use one durable request identity. A closed, cancelled or expired attempt rejects later callbacks.

Authentication simulates the device ceremony while using the normal challenge, assurance composition and session-variable publication. Payment simulates the provider while retaining the caller-authorization gate, amount parsing, pinned node policy and normal settlement publication. No payment capture or charge reaches a provider. UI scenarios publish normal appointment/photo actions; photo fixtures create a synthetic attachment in the test conversation. These adapters require a live authenticated attempt and cannot target arbitrary conversations.

agent_sim_events stores callbacks; agent_sim_participants stores additional streams. Full supervisor tests reserve two active slots and share global, tenant and integration startup/rate limits. The worker polls pending events in bounded batches. Waits resume with LangGraph Command(resume=...), preserving the checkpoint and earlier side effects.

The result includes deterministic checks for workflow completion, expected decision/amount, supervisor completion, duplicate/late responses and actual end_session evidence. A failed check overrides a favorable model verdict. For appointment agents with record_copay, deterministic checks also compare records with completed payment events and reject missing, incorrect or repeated transaction records. The judge receives recorded execution arguments after capability bindings alongside the model's original request. Golden sample dates and IDs are illustrative; actual tool results govern factual correctness. Identical ingress/provider captions tied to one recorded caller input count once for the judge; identical answers on different inputs remain separate. When session closure is required, goal completion triggers a real goodbye and a bounded wait; disconnecting the test transport does not count as success. Case titles are excluded from caller knowledge, since labels can reveal an expected amount. The opening line still applies when the agent greets first. Configured startup greetings get up to 45 seconds to finish before the caller speaks; a missing greeting is a startup failure rather than a fabricated caller interruption. Lifecycle output mappings come from the same frozen workflow revision as execution. For multi-step caller goals, approval or an offer alone does not complete the task: the caller must answer requests to finish it. The reviewer treats the normal final acknowledgment after end_session as part of that closing response, before the runtime terminates the session.

Submission resolves selected IDs within the tenant and agent, snapshots their revisions and linked golden examples, and creates jobs through the existing durable queue. Editing a case cannot change an existing run. The scorer uses the case's own golden and expected outcome alongside the agent's general goals. A pass requires an overall score and expected-outcome score of at least 0.8, all must-goal scores at least 0.8, and no major or critical issues. An unexercised required outcome fails. Execution and scoring errors remain distinct from judged failures.

An empty, filtered or truncated judge completion is a checking error, never an agent-quality verdict. Incomplete or missing goal assessments receive one review-only retry against the same evidence; a second invalid response remains a checking error. The local HealthPlus reviewer is calibrated against recorded consent, bound arguments and completed transactions; see the worker profile.

The Vanguard HealthPlus example (3032) uses a shared synthetic patient across authentication, appointment and adverse-event fixtures. Appointment dates are relative to the lookup's from_date; provider/location remain deliberately absent to expose invented details. Confirmation and document responses retain the submitted references and photo URL; copay responses reflect the submitted transaction, amount, currency, status and timestamp. Adverse-event tools return their corresponding resource types, not generic success objects. The refusal persona declines verification without an appended instruction to consent. Retry/no-match cases select the same patient scenario in both appointment and adverse-event capabilities; switching subagents cannot reveal a contradictory patient record. Their transient lookup state is shared within the attempt. Successful fixture qualification does not certify the agent's prompts or real external integrations; the saved conversations and their verdicts do that separately.

Reliability experiments outside the application¶

Run controlled iterations against a saved suite before adding automated optimization. Retain the user direction, exact prompts, config snapshots and hashes, hypothesis, batch IDs, transcripts, tool evidence and verdicts for every candidate. The local 3032 journal lives under .devcontainer/local/evaluations/3032/reliability-loops/; it is private experiment data, not an application feature.

Keep cases, goals, goldens, callers, reviewers and fixtures fixed while comparing agent changes. Report strict first-attempt passes separately from retries, mean quality scores, session completion and execution/checking errors. A successful backend call or a 0.85 average score does not imply an 85% pass rate. Repeat the chosen candidate across the whole suite; do not select the best attempt of each case from different candidates.

Retain operator-interrupted trials and harness failures with their cause. A harness repair starts a documented measurement boundary; repeat affected cases under the unchanged agent before attributing improvement to a prompt. Reviewer allegations must be checked against evidence too. Record disputed judgments separately without silently replacing the frozen score or teaching the agent to satisfy a mistaken allegation.

The compact ?view=tests APIs return aggregate counts and paginated job status, latest agent messages and verdict explanations. The active panel polls every 1.5 seconds (5 seconds in a hidden tab); the picker refreshes every 4 seconds (10 seconds hidden). Closing the dialog aborts polling, not execution.

The prompt editor and golden tool picker share the pure helpers in apps/ai-manager/lib/agent-tools-list.ts. MCP operations retain their input metadata and owning tool reference while using callable operation names; workflow names retain the runtime's normalization. The reference query applies the configured MCP operation filters without returning raw tool configuration.

How TrueServe assesses whether an agent is performing properly — the emerging cross-platform standard, the eval-ish systems already in this repo, and the Performance Copilot that bootstraps the assess loop. Read this before touching anything under apps/ai-manager/lib/performance/ or the Performance surface at /agents/[agentId]/performance.

The north star: make it as easy as possible for a user to simulate and test their agents (voice and chat) and let AI judge whether they behave. The Performance surface owns the assess half. Its blocker is a cold start — its rubric (Goals) begins as a blank text box, and with zero Goals nothing downstream (reviews, continuous review, or any future simulation) has anything to score against. The Copilot removes that cold start.

Agent Test design and durable adapters¶

The normal testing UI is Lynn's Overview / Run Tests / Results dialog. The original Performance page remains available only by direct URL, /agents/<agent-id>/performance; editor/Preview controls open the new dialog.

GET /api/agents/:id/performance/test-suite?version=draft|published|vN returns current configured behaviors, saved cases, generation metadata and staleness. POST explicitly generates/refreshes scenarios; it accepts version, the last suite revision, optional behavior_ids and caller_types. The UI generates base scenarios and chooses caller variations when running. PATCH acknowledges review or suppresses a generated case with test_id, suite revision, version and action=reviewed|remove. Reads require RunAgents; generation and metadata edits require ManageAgents plus agent edit access.

Generation runs outside the database transaction. A config hash and optimistic suite revision are checked again before commit. Saved-case revisions distinguish unedited generated cases from authored edits; edited cases are retained, and removed generated cases are not recreated. Behavior fingerprints flag changed sources. Coverage measures linked tests for identified configuration behaviors; it does not establish successful execution or complete semantic coverage.

The existing simulations POST additionally accepts an optional name and caller_types (authored, calm, frustrated, confused, urgent, neutral). It expands saved scenarios × caller types × repetitions, validates the full 10,000-conversation limit, and pins aligned case snapshots and personas. The idempotency comparison includes run choices, excluding derived source hashes and snapshots. Simple-form saves retain language, context, golden association, turn limits, capability fixtures, caller identity, supervisor and replay settings. Concurrent case edits are rejected through the displayed revision.

The compact ?view=tests read model includes run name, source config hash, mean reviewed score, caller type and each job's pinned expected outcome. GET /api/agents/:id/performance/simulations/:batchId/cases/:jobId loads only the selected case's evidence with explicit tenant/agent/batch ownership checks. It uses the same evidence normalization as the evaluator, so highlighted turn indices refer to the dialogue and tool evidence actually judged. Results never substitute the current scenario's expected outcome. Execution errors, pending checks, failed checks and stopped cases remain distinct from passed/failed assertions. The new UI does not provide legacy fixture editing, goals/goldens, retry evidence or continuous-review controls; those remain on the direct-URL Performance page by product decision.

The emerging standard¶

Five commonalities recur across the assistant-evaluation platforms worth learning from (Sierra, Fin.ai, Google Gemini Enterprise CX, Decagon, Cresta). These are the bar to design toward, not features to copy wholesale.

Test cases are auto-generated from real business data, not hand-authored — SOPs, knowledge bases, coaching transcripts, and production feedback seed them. Hand-writing hundreds of tests is not expected; generating them is table stakes.
The simulated user is an AI persona with deliberate variation — across languages, tones, tech-comfort, and edge cases — and the persona set evolves with the real customer base.
An independent LLM judge scores against explicit criteria and explains why. The explanation (inspectable pass/fail rationale) is the feature, not a footnote.
The unit of evaluation is the full conversation / task completion, not the individual turn (Google layers turn, tool, and conversation levels).
Suites persist and gate releases — simulations re-run on every change, releases gate on them like unit tests, and scheduled runs alert on drift.

Two specifics are worth adopting deliberately:

Cresta's judge calibration — evaluators built from binary, verifiable criteria and calibrated against human review. This is the answer to "why trust the AI verdict."
Google CX's fidelity modes (stable replay / naive replay / tool-fake), its metric taxonomy (semantic match, tool correctness, hallucination, goal satisfaction, task completion), and its clean Scenario vs Golden test-case split.

Current state in this repo¶

This is a unification play, not greenfield. "Evals" is not one concept here; it is fragmented across three overlapping systems plus one dead scaffold.

#	System	Runs against	Scored how	Lives in
1	Agent Performance / Continuous Review	real production conversations	LLM judge vs Goldens + Goals; clusters issues → proposes config fixes	`apps/ai-manager/lib/performance/`; tables `agent_goldens` / `agent_goals` / `agent_reviews` / `agent_review_suggestions` / `agent_review_subscriptions`
2	Scale Testing / Agent Validation	simulated agent-vs-agent conversations generated from uploaded recordings	LLM assessment vs per-turn criteria	`apps/ai-manager/lib/scale-testing/`; tables `test_jobs` / `test_scripts` / `test_runs` / `test_assessments`
3	Interactive Simulation	live manual single sessions (chat / voice / telephony)	not scored — sandbox	`apps/ai-manager/app/(main)/testing/page.tsx`
4	"Eval Publish Gate" (dead)	—	—	Postgres fn `_eval_gate_check` reading `eval_runs` / `eval_results` / `evaluators` — tables never created

Three things to keep in mind:

The pieces do not share a dataset, rubric, or run entity. There are two test-case notions (Goldens vs Scripts) and two run notions (reviews vs test runs) with no common model between them.
The anchor was already sketched. The dead _eval_gate_check plus the phantom eval_runs / eval_results / evaluators schema signal the intended direction: offline evaluators and eval runs gating publish.
The fidelity trap. Today's scale-testing "simulation" does not run the agent through the real orchestrator — agent-invocation.ts pulls the system prompt and model out of config and calls OpenAI / Anthropic directly. No tools, no RAG, no workflow branching, no voice pipeline. For a voice-first platform that is a large asterisk on any "we simulated it" claim.

The Performance Copilot¶

A floating conversational widget on the Performance surface that bootstraps and operates on the existing tabs (it does not replace them). It reuses the editor assistant chassis end to end and proposes; it does not auto-write — the AI stops at a proposal, and the human accepts (or edits first), which is the one path that writes.

One shared session per agent¶

The Performance surface is another view of the same agent, not a different entity — unlike the agent-vs-workflow split that the assistant's kind already distinguishes. So the Copilot is the same kind='agent' assistant session the editor uses, extended with a small goal tool set, not a new session kind. Both mounts pass the same normalized base agent id as targetId (editor: pipelineId.split(':')[0]; Performance: baseAgentId), so they resolve to the one active (user, 'agent', baseAgentId, 'edit') session in assistant_sessions — shared history, shared claude --resume context, shared tool set and system prompt. That sharing is the whole point; the two mounts must pass an identical targetId or they fork into two sessions. A clean thread is one "New chat" away.

The Performance mount uses the same standard chat bubble and assistant header as the agent editor.

Phase 1 — draft Goals (shipped)¶

"Create some goals for me" → the Copilot reads the current agent version's behaviour prompt (and, when useful, real transcripts) → proposes candidate goals (text + severity + one-line rationale) as accept/edit chips → accepted goals write through the existing Goals API. Propose-don't-auto-write.

The only genuinely new capability is a two-tool set in performance-tools.ts, added to the agent session's tools in assistant-mcp/tools.ts:

list_agent_goals (read) → listGoals so the model avoids proposing duplicates.
propose_goals({ goals: [{text, severity, rationale}] }) — non-writing. It validates and echoes the proposal; its tool_result payload is what the widget renders. Severity normalizes to should when missing/invalid (GoalSeverity = 'must' | 'should' | 'nice_to_have').

The behaviour prompt and real traffic are already reachable through the agent session's existing tools (get_current_draft returns the full config incl. the prompt; list_agent_conversations / get_conversation_transcript read traffic), so no new reader was needed. The runner system prompt in assistant.py _assistant_agents_md gains a Performance section (agent-kind only) instructing the model to read the prompt, list existing goals, propose 5–8 focused checkable goals with severity + rationale, and never claim it created a goal (there is no goal-writing tool).

On the frontend, AssistantPanel correlates each streamed tool_result back to its tool_call by id (CLI tool_result events carry a generic name), and for propose_goals renders a GoalProposalCard: per goal, an editable text input + severity select + Accept/Dismiss. Accept POSTs {text, severity} to /api/agents/[agentId]/performance/goals (permission-checked, one write path) and shows the accepted chip; dismiss is local.

Writing on accept goes through the existing /goals route, not a new MCP effector — that keeps "propose don't auto-write" and avoids a second write path. agent_goals.agent_id is the base agent id, so a versioned targetId (4209:draft) is reduced to base before the write.

On-demand reviews from the Reviews tab¶

Continuous review only samples new conversations through the orchestrator sampler, so an existing conversation had no UI path to being scored — the POST …/performance/reviews route (mode manual) existed but was reachable only by hand. agent-review-worker's own docstring names the gap: its agent_reviews:pending queue is "the same Redis list … that on-demand UI buttons push onto."

ReviewsTable now has a Review conversations button that opens ReviewConversationsDialog — a multi-select picker (filter + "select all shown") that fans out one POST per selected conversation to that existing route (no batch endpoint, no new auth surface). Each review UPSERTs on conversation_id, so re-reviewing updates the same row. This is a manual, permission-checked write (AIManager:ManageAgents), distinct from the Copilot's propose-only stance on goals.

Reviews can be deleted singly or in bulk. The item route (DELETE …/performance/reviews/:reviewId) removes one; the collection route (DELETE …/performance/reviews with { ids: string[] }) removes up to 100 selected rows in one agent-scoped query. Both use the same AIManager:ManageAgents gate. No child table references agent_reviews, and the conversations are untouched, so later auto/manual reviews simply recreate the rows. ReviewsTable exposes per-row checkboxes, Select all shown, Delete selected, and a trash action; ReviewDetail retains its single Delete action.

Goals and goldens can also be deleted in bulk. The collection routes each gained a DELETE — DELETE …/performance/goals and DELETE …/performance/goldens (same AIManager:ManageAgents gate) — bodied { ids: string[] } (delete those rows) OR { all: true } (delete every row for the agent), returning { deleted }. An empty/missing selection is a 400, never a silent delete-all. The DAL helpers (deleteGoals/deleteGoldens in lib/performance/db.ts) are scoped by agent_id so a stray id from another agent is never removed, and ids === null (from { all: true }) is the only path that clears the whole set; an empty array deletes nothing. GoalsTable and GoldensList drive this with a per-row checkbox + a Select-all/Delete selected toolbar (confirm dialog, selection cleared on every reload). This is the bulk undo for Copilot-drafted goals/goldens — deleting never touches conversations or existing reviews.

Scoring is async (a worker drains the Redis queue), so both ReviewsTable and ReviewDetail poll silently rather than making the user reload. Each holds a stable setInterval that fetches (in silent mode — no "Loading…" flip, no error toast) only while something is in flight: a row is pending/scoring, or we're inside a short post-enqueue window (pollUntilRef, ~45 s) that covers the enqueue → worker-pickup race before any row has visibly flipped to scoring. The table shows an "Updating…" pulse while polling; ReviewDetail.rescore() opens the same window and refreshes in place so a failed → scoring → done transition lands without a reload. rowsRef/reviewRef let the interval read current state without resubscribing every render.

The review drawer (ReviewDetail) leads with a verdict band (score + qualitative word + rationale, colored by scoreTone), then Goals / Golden matches / Flagged moments / Transcript. Two things it does that the raw review row can't: it resolves goal_id/golden_id back to text by fetching /performance/goals + /performance/goldens and mapping by id (the judge output only carries ids), and it anchors every cited turn_index to a clickable "Turn N" chip that scrolls the transcript to that turn and briefly highlights it. All machine enum values (issue kind, severity, deviation kind, goal severity) render through the generic humanizeLabel — goal_violation → "Goal violation" — so a new enum value is human-readable without any per-value code. The status chip is only shown when actionable (scoring/failed); a finished review is represented by its score, not a "done" badge.

A review's agent_version_id is attribution, not "what was tested." The judge never runs the agent — it scores an existing transcript against the Goals/Goldens rubric — so the version field only tags which agent version the review is attributed to, for splitting scores by version over time (the versionId filter + agent_reviews_agent_version_score_idx). Continuous review stamps the currently-published version encoded as "<agentId>:v<n>" (sampler.py _resolve_agent_version_id); on-demand reviews store null (the manual route doesn't resolve a version). versionLabel renders the v<n> tail for the former and "Unversioned" for the latter — deliberately NOT "Draft", which the old ?? 'draft' fallback wrongly implied (nothing about the draft config is tested; the field is simply unset).

Proactive goals on prompt change¶

The Copilot doesn't only wait to be asked. When a user edits an agent's behaviour prompt and closes the stage edit-node drawer, the system auto-generates Goals from the new prompt, writes them directly (accepted by default), and then tells the user what happened — no "craft some goals" request required. This is the endorsed direction for proactive evaluation work: the AI should do more of the testing/evals/performance groundwork without being asked, then inform the user so they can edit or delete anything they don't want. It lands first on the highest-value surface — the rubric's cold start.

Trigger — drawer close, dirty-guarded. Generation fires from handleCloseInspector in the agent editor page (the single "drawer closed" callback — there is no separate blur event), and only if the behaviour prompt text actually changed since the drawer opened. The editor snapshots the stage's prompt at open (readStageBehaviorPrompt in lib/stage-prompt.ts — reads config.template.inline, then messagesToPrompt(config.messages), exactly as SystemPromptSection does, not worker.ts's extractSystemPrompt, which can't see template.inline) and diffs it at close via the pure shouldTriggerGoalGeneration (whitespace-normalized inequality; a non-empty new prompt only). It is restricted to prompt-bearing stages (stageHasEditablePrompt), is deliberately not per-keystroke, and a close with no prompt change — or a pure-reflow edit — is a no-op. An in-flight guard keeps two quick closes from double-POSTing, and the server dedup makes the write idempotent regardless. (A "Save button" was considered; on-close won for lower friction — the editor already autosaves via collab, so a save control would be new UI.)
Autonomy — auto-write, then notify. Goals are written straight to Performance → Goals, deduped against existing goals so repeated prompt edits don't pile up duplicates. The safety valve is an easy undo — the user deletes any unwanted goal in the Goals tab. The write is server-controlled, not a model-callable tool: a dedicated route POST /api/agents/[agentId]/performance/generate-goals (perm AIManager:ManageAgents) generates candidates (lib/performance/generate-goals.ts — a single OpenAI json_object call mirroring the judge, gracefully no-op'ing when OPENAI_API_KEY is unset), dedups via createGoalsDeduped (goal-dedup.ts: same-severity + equal-or-similar text), writes the net-new ones, and returns { created, skipped }. Routing through a server route rather than the shared chat assistant is deliberate — prompt-level gating of the model is unreliable, and a CLI assistant turn would hit the per-session lock (409 against a live user turn). The chat-driven "craft some goals" flow keeps its propose-then-accept stance (a UX choice, since the user is already in a conversation); only the proactive trigger auto-writes.
Notification — a system-initiated assistant turn. After writing, the route persists an assistant message that nobody typed — the first turn not initiated by the user — into the shared (user, 'agent', baseAgentId, 'edit') session, via appendAssistantNotification (db-assistant-sessions.ts), which finds the active session or creates a bare row (no runner dispatch, no token) when the panel hasn't lazily created one yet. The message (buildGoalNotificationMessage) summarises what was added (e.g. "I added 4 goals from your prompt change"). New messages omit links to the direct-URL-only Performance page. The editor page then calls the popup's imperative open() (a forwardRef handle on AssistantChatPopup → AssistantPanel.refresh()), which auto-opens if collapsed and re-reads the persisted history so the message shows immediately; because it is persisted server-side it also survives reload and appears the next time the assistant is opened even if the popup isn't mounted (assistant disabled / template). The bare row is safe — the first real user turn self-heals the missing workspace through the messages route's 404-recovery.
Scope. Goals only for now. The same trigger → auto-write → notify pattern is a likely follow-on for Goldens (Phase 2). Whether it extends to synthetic transcripts at scale (Phase 3+) is undecided.

Phase 2 — draft Goldens (shipped)¶

A Golden is a canonical "this is what good looks like" conversation the judge scores future conversations against — a turn list, each turn tagged essential (must match) / flexible (spirit must match) / ignore (skipped). Goldens mirror goals exactly across the two Copilot paths: propose-then-accept when you ask in chat, auto-write-and-notify when you edit the prompt.

Chat path (ask for it) — propose, don't auto-write. One non-writing tool on the shared agent session (performance-tools.ts):

propose_golden({name, description?, conversation_id?, turn_tags?, turns?}) — proposes a golden two ways. Pass a conversation_id to curate a real conversation: the tool reads its transcript server-side (a read, not a write — ownership-guarded via getConversationDetail, base-agent match, RLS is tenant-not-target) and applies any per-index turn_tags overrides so the card can show the real turns. Or pass from-scratch turns (role + content + tag) for a scenario the user describes.

Like propose_goals, this does not save — it echoes {proposed, name, description, source_conversation_id?, turns}, which GoldenProposalCard renders as the turn list with editable tag selects and an Accept / Dismiss control. Accept writes through the existing permission-checked POST /api/agents/[agentId]/performance/goldens (plain createGolden, no dedup — accepting is an explicit choice). Because it's non-writing, propose_golden is not in MUTATING_ASSISTANT_TOOLS, so it's safe in a read-mode (ask) session. AssistantPanel detects it by correlating the tool_result to its tool_call id, exactly as for propose_goals.

Proactive path (on prompt change) — auto-write + dedup + notify. On the same edit-drawer close that triggers proactive goals, after the goals write the editor also POSTs /api/agents/[agentId]/performance/generate-golden (generate-golden.ts drafts the ideal conversation from the new prompt, mirroring generate-goals.ts — OTEL usage span, no_api_key → 503 no-op). It writes through createGoldenDeduped (dedup keyed on source conversation then normalized name — golden-dedup.ts), so repeated prompt edits don't pile up near-identical goldens; on a dedup skip it returns golden: null (200) and the trigger surfaces nothing. On a real create it posts a system-authored notification (golden-notification.ts) into the shared assistant session without a legacy Performance link, and the page opens the popup when either the goals or the golden write landed. The proactive trigger is triggerProactivePerformance in the agent editor page (goals then golden, each best-effort and independent).

Naming the tool a turn should call. A golden turn can pin which tool the agent should call, not just what it should say — the judge scores a wrong_tool deviation against it. This rides GoldenTurn.tool_calls: [{name, arguments?, result?, error?}] (JSONB); name is the load-bearing field. Every authoring surface populates it through one pure normalizer, golden-turns.ts normalizeToolCalls, so the shape can't drift between them:

Manual editor (GoldenEditorDialog) — each turn has a Tool called picker whose options come from GET /api/agents/[agentId]/performance/tools. That route resolves the agent's config.tools / config.inline_tools across every stage through the same buildAgentToolsList the prompt editor's {{TOOL:}} dropdown uses, so a golden can only pin a tool the agent can actually call. Two ref kinds expand into SEVERAL names — a builtin:<slug> capability (one entry per real tool the capability registers), and an MCP tool_configs ref, which the orchestrator exposes as one callable per operation (tool_filters ∩ the discovered schema), so the picker lists the operation names (send_otp, verify_otp) and never the config display name ("OTP Verification"), which is the name of none of the runtime tools. The MCP operations are resolved by resolveMcpOperationNames (populated onto each ref by listToolReferences({ withMcpOperations: true })). A tool a curated turn already references but that the current config no longer wires up stays selectable (labelled not in config) so editing never silently drops it — this is why an older golden that pinned an MCP tool by its display name shows as not in config until it is re-picked against an operation.
Chat propose_golden and proactive generate-golden — the from-scratch turn schema accepts tool_calls: [{name, arguments?}]; generation is given the agent's real configured tool names in its prompt and told to name only those.
Curation from a real conversation (buildTurnsFromConversation) is the ONE path that fills result/error — those are runtime data the agent doesn't author (there is no wrong_result deviation kind), so from-scratch authoring and generation never set them. Tool-call events (conversation_events, event_type='tool_call') merge into the turn timeline by timestamp as tool turns carrying the real {name, arguments, result}.

Background conversation tests in Vanguard¶

Performance → Simulations → Run tests and the Copilot proposal card submit POST /api/agents/:id/performance/simulations. Submission requires AIManager:ManageAgents, explicit consent: true, and a scoring criterion: a saved case's expected outcome, an agent goal or a golden. A transaction pins the requested draft/published/numbered version, explicit profile, nested definitions, rubric, and tool mock revisions, then inserts durable jobs with a set-based fan-out. Up to 1,000 selected saved cases, or 25 manually supplied personas, multiplied by repetitions may produce 10,000 jobs per batch. An optional UUID Idempotency-Key deduplicates identical requests; reusing it for a different specification fails.

The API returns 202. It does not run conversations in a Next.js background task. The database queue/outbox survives web restarts. A confirm publisher sends only job/tenant identifiers to durable RabbitMQ queue agent_simulations. Lost or unacknowledged publications are safe to replay. Conversation and scoring workers are separate roles in scripts/run-simulation-worker.ts.

Profile runtime	Auto mode	What is exercised
Text LLM and complete STT → LLM → TTS cascades	Text	Actual configured LLM, prompts, tools and orchestration; speech stages are bypassed
OpenAI Realtime and ElevenLabs conversational profiles	Text	Native message delivery to the configured S2S service
Gemini Live, Nova Sonic, Deepgram Voice Agent	Transcript	Native user-message delivery with the voice session and output active
GPT Live, including Responses and client delegation	Audio	Caller TTS into a paced virtual microphone; actual Live/delegation session
Other S2S providers	Audio	The same synthetic microphone adapter; provider support still needs a smoke test

Mode is recorded with results. GPT Live rejects text/transcript mode because a backend-only text call does not exercise its speech/delegation behavior. Explicit audio mode can also test cascades. Selecting a named profile is explicit; tests never silently substitute a chat model. Semantic tests do not establish ASR accuracy, voice quality, interruption behavior or real telephone latency.

Each attempt owns one VoiceAgent.Stream gRPC connection to real ai-orchestrator. No browser, telephone call or per-turn database polling is needed. Session setup is authenticated by a random capability in gRPC metadata; only its hash is stored. Public test_mode/sandbox_tools metadata cannot grant mocking privileges. The runtime claims an attempt once, checks its live lease, and uses pinned configuration. Internal MCP/HTTP/broker calls propagate the capability and check it again before running nested work.

The stream carries sequenced canonical transcript revisions, tool events, backend activity and finalization. Every stream is continuously drained. GPT Live uses 20 ms PCM16 frames, including silence while the caller or backend thinks. TTS audio is resampled to the negotiated format off the main audio event loop; a bounded cache shares repeated clips within the tenant. Audible playback, transcript activity and outstanding tools/delegations determine reply boundaries. A timeout, missing finalization, provider error or audio starvation fails the attempt instead of reporting a completed conversation.

Tool mocks belong to tools. Tool definitions expose an output schema and a Testing panel. Supported schemas generate a checked default at creation; authors can edit variants/state/examples or request a draft from instructions. Saving validates examples and appends an immutable revision. Agent-local Python/API/ Knowledge tools carry the same contract inline. Preflight rejects missing or stale mocks. Runtime evaluates data-only rules against isolated attempt state, validates rendered results, and records the revision/result separately from the unmodified tool response. Input repair/retries use the same dispatcher. See Tool testing for the contract.

Internal conversation controls and supported workflow control flow execute normally. External calls need mocks in sandbox mode. Direct effect nodes such as HTTP, notification and arbitrary code must be moved behind a tool definition or run with the explicit real-tools policy. Provider-hosted search and unresolved capability expansions are also rejected rather than given fabricated success. Real-tools runs can cause external effects; they remain an explicit run choice.

Admission counts active sessions, sessions still starting and recent starts across global, tenant, integration and batch scopes. Local worker concurrency and separate audio slots bound process resources. Lease renewal, fencing, cancellation and recovery prevent two owners from completing the same attempt. An interrupted attempt gets a fresh conversation, never a guessed replay of hidden provider state. The conversation-api startup registry also shares a readiness task for duplicate requests without holding its global lock across network I/O; independent sessions can start together.

Completed attempts are scored from their finalized evidence and pinned goals/ goldens. Scoring has its own lease; retrying failed scoring never redials a caller. The UI shows queued/starting/active/completed/failed/cancelled counts, job errors, paged jobs and review links. Cancel remaining stops queued work and revokes active leases. See Operating evaluation workers for limits, local commands, recovery and validation.

Future consideration — configurable autonomy¶

The chat-driven Copilot stops the AI at propose_goals (the user accepts before anything is written); the proactive goals on prompt change feature above is the auto-write counterpart, implemented as a server-controlled generate-goals route rather than a model-callable effector. With both write paths in place, the natural next increment is a per-tenant/per-user setting that chooses, per surface, whether the Copilot writes directly or routes through the human accept step — including a toggle to turn the proactive behaviour off (or downgrade it to propose-only). Auto-write-then-notify stays the default for proactive work.