Athens testing review and evolution proposal¶

Status: Draft | Last reviewed: 2026-10-20

Use this review to evolve the live Athens Testing workspace into a reliable way to decide whether an agent change improves its ability to do its job. Preserve the execution and evidence engine, make comparisons reproducible, and present customer situations, outcomes and next actions before evaluation configuration.

This document records the analysis and proposal. The subsequent Vanguard implementation of phases 1–2 is documented in Agent testing workspace; Athens agent 4210 was copied as Vanguard agent 4982 without replacing Vanguard's existing 4210.

Evidence and scope¶

The review targets agent 4210, Testing Workspace, at the Athens Performance route with tab=tests. Read-only SSM access confirmed the running city's checkout:

Branch: feature/test-voice-agents.
HEAD: 5d9ce0661c3c47131378f9cabb144b8202b9467e, October 19, 2026.
The checkout had 43 changed/untracked status entries. The review captured 41 relevant source and documentation files and verified their hashes against the live files.
The refreshed origin/athens branch ends at 540583e4, September 23. It is historical context, not the basis for the current findings.
Source inspection uses the isolated checkout at /tmp/trueserve-athens-live-review-20261020. Source links below refer to that captured working copy.
Live database reads ran inside read-only transactions. The public page was reachable but required login; this review did not use an authenticated visual walkthrough or launch new agent conversations.
Vanguard evidence includes its current generation/evaluation implementation and the saved HealthPlus 3032 experiment journals. Historical journal results describe those experiments, not current platform reliability.

The live agent inventory at capture:

Item	Observation	Meaning
Tests	20, grouped under Order status	One journey with different business outcomes and customer choices
Requirements	26 active	Includes generated obligations and protected manual content
Assignment coverage	25 of 26, displayed as 96%	Requirements have test associations; this is not a success measure
Default caller types	Calm, Frustrated, Confused, In a hurry	Communication styles applied across selected Tests
Default full run	20 × 4 × 1 = 80 conversations	Repetitions increase that to as many as 400
Latest saved run	October 16; 2 Tests × 4 types = 8 conversations	A subset of the current 20-Test inventory
Latest saved verdicts	5 passed, 3 failed, all first attempts	62.5%, displayed as 63%; all three failures concern the opening greeting

The latest run is c7c40dd6-14d9-4fc9-8fac-78e256ed169a. It tested the order-delivered situation and a customer who initially asks to skip verification, then cooperates. The three failed Results report that the first assistant response omitted the Northwind Retail greeting. These are the saved assessments, not a new qualification under the October 19/20 code.

The preceding run used a different second situation, order-shipped, and recorded 7/8 passes. Those two percentages are not a controlled before/after comparison.

The unassigned Requirement is a protected manual “Hello.” entry. It is not evidence of a generated omission; its presence also shows why coverage percentages require interpretation.

How the live implementation works¶

Prepare tests. Read the current draft's instructions, selected default runtime profile, referenced agents/workflows and supported tool contracts. Extract Requirements, then create Tests from customer journeys, service outcomes and caller decisions. Preparation does not execute the agent.
Create the test setup. Each Test owns caller instructions, customer facts, saved variables, checked simulated service responses and, for generated tool journeys, an expected execution path. General Requirements apply to every Test; conditional Requirements are assigned where the planned situation should exercise them.
Choose and run. Select Tests, shared caller styles, agent version and repetitions. Voice agents also expose supported conversation modes. A draft run waits for collaborative saving. Starting a Run freezes the selected Test, Requirement, caller, fixture, evaluator and agent/dependency versions.
Simulate conversations. A customer model follows the saved request, facts and style. The actual agent runtime responds. External services normally return checked simulated responses; applicable native state and ending behavior still execute. Durable workers own progress and bounded execution retries.
Assess the evidence. Deterministic checks examine tool calls, correlated results, ordering, capability events and closure. A model assesses semantic duties against the captured conversation. It first determines whether a conditional duty applied. The Test's expected path independently checks that required execution occurred.
Show Results. Any failed applicable Required check, including the expected path, fails the Result. Missing required evidence makes it inconclusive when there is no proven failure. Feedback-only findings do not change the verdict. Results preserve the historical contract and link checks to evidence.

The run pass percentage counts confirmed passes out of all requested conversations once settled. The per-conversation check percentage counts applicable Required checks and the expected path equally; missing or unresolved required checks suppress that score. The run's average check score uses scored conversations only. These are different questions and denominators.

Sources: live workflow guide, execution pins and modes, verdict logic, check-score calculation.

What deserves to be retained¶

The live work has already corrected several limitations of September's version:

Draft and published-version execution are supported; tests are not always redirected to the draft.
Manual edits and deletion decisions survive regeneration.
Runs retain immutable inputs and evidence, so later edits cannot rewrite their history.
The actual agent executes against isolated, checked service responses.
Deterministic failures cannot be overridden by a favorable semantic judge.
Execution errors, assessment errors, unresolved evidence and behavioral failures have distinct states.
Customer instructions do not expose expected service results or the grading checklist.
Voice mode descriptions explicitly describe the evidence each mode can establish.

The current model also removes several earlier duplicate concepts: a Test is one runnable situation, its journey is a grouping label, and Requirements own their evaluation meaning. The proposal should improve that model's presentation rather than introduce another parallel scenario/check inventory.

Sources: current authoring model, canonical contracts, manual-edit preservation.

Where the experience is confusing¶

Finding	Consequence	Recommended change
Overview leads with Tests, Coverage and Last run	Counts and percentages arrive before a conclusion about the agent	Lead with what changed, failed and remains untested
Coverage counts assignments	96% can look like broad validation despite a latest run covering only two situations	Label it “Requirements with tests”; show executed scope and freshness separately
Tests expose name, Description and Caller instructions	Description is display copy; changing it does not change execution	Present situation, customer behavior, service conditions and success criteria together
Prepared facts and service setup are largely hidden/read-only	An author must leave the Test, edit shared tool defaults and regenerate to change data	Provide typed per-Test data/outcome controls with explicit shared defaults
Requirements require scope, assignment and enforcement choices	A business author has to understand the checking configuration	Generate these bindings behind plain-language expected outcomes; make advanced editing optional
The panel says “Tests update automatically as you change the agent”	Live preparation actually requires explicit regeneration	Correct the copy and show a concrete list of affected tests after an edit
“What needs attention” and suggestion insets contain lorem ipsum	The places expected to explain problems offer placeholder content	Replace them with actual failures or an honest empty state
All 20 Tests and all four styles are selected by default	A simple prompt check starts at 80 conversations without prioritizing relevant situations	Offer Quick check, Full check and explicit repeat settings
Results default to descending check score	The strongest results appear before the failures an author needs to investigate	Default to new failures and unresolved problems
Run history lacks a controlled comparison workflow	“Run again” can use new Test revisions and the currently selected agent version	Add an explicit baseline and compare compatible pinned cohorts

The editable Test form preserves the existing setup and expected path. This is good for avoiding accidental data loss, but changing caller behavior can leave an author with an old success-path setup that no longer expresses the new situation. The form needs an explicit compatibility check or a proposed setup change.

Regeneration also updates or retires unprotected generated Requirements as instructions change. That is useful maintenance, but it can move the measurement criteria during an improvement experiment. Immutable historical Results preserve what happened; they do not by themselves make a new Run comparable.

A recent saved assessment provides a calibration example: one conversation was failed for saying a phone suffix instead of the literal masked phone string, while the finding acknowledged that the full number was not disclosed. Review the intended policy and evidence before treating that as an agent defect. If exact phrasing is required, state it clearly; if protecting the full number is the requirement, grade that behavior.

Sources: overview copy and placeholders, Test form and preserved setup, saved inputs, result suggestion placeholder, rerun semantics, result sorting.

The proposed author experience¶

Keep Test as the entry point. Make its main action Check my changes, with Run full check and Edit test plan alongside it. The default view should answer:

What did this change improve?
What became worse?
Which important situations have not been checked?
What should I do next?

For a first agent, lead with Check this agent and establish the first baseline. Afterward, default the comparison target to the last accepted baseline or published version, with the choice visible.

Present Tests as customer situations with a concise, editable contract. For example:

Field	Example
Situation	Customer refuses identity verification
Customer behavior	Wants order status but declines the code
Service conditions	Account exists; verification has not succeeded
Success	Explain why order details cannot be shared and close appropriately
Must not happen	Reveal order details or perform the protected lookup
Variations	Calm; frustrated; volunteers an order number early

Business conditions belong to the situation. Communication styles vary how the customer expresses it. Neither belongs in a hidden instruction that the author cannot review.

Translate plain-language outcomes into the existing Requirements and assertions. Show critical tool ordering or exact values when the author needs to make a decision, using readable labels and typed fields. Preserve JSON as an advanced inspection/export format.

Generate the initial plan from the business brief and agreed policies, supplemented by implementation discovery. Ask authors to review ambiguous assumptions and consequential success conditions. An omission in the agent prompt must not automatically become an omission in the test plan.

Use three execution choices:

Quick check: affected situations plus the critical regression set. Use a representative customer style by default, with targeted variation where relevant.
Full check: the complete agreed suite and selected communication variations.
Check consistency: repeat selected situations and show every attempt, including recoveries and failures.

Show the conversation count and a measured time/usage estimate before starting. A Quick check must say what it omitted. Reducing runtime should never silently turn a partial run into a full-agent claim.

Results should group evidence by business problem, such as “Opening greeting missing in three customer styles” or “Order lookup attempted before verification.” Opening a problem should show the expected outcome, actual behavior, exact supporting evidence, relevant configuration change and proposed next action.

Keep a distinct action for The assessment looks wrong. Preserve the original Result, record the dispute and append any reassessment. Do not train the agent to satisfy an unverified grading allegation.

Make comparisons trustworthy¶

Add a comparison record above the existing Runs. It should identify:

Baseline and candidate agent snapshots.
One approved Test/Requirement set and identical fixture/input revisions.
Caller model, profiles, grading policy/model and runtime/channel versions.
Requested repetitions, original attempts, retries and exclusions.
The candidate change and the hypothesis it was intended to test.

Compare matched situations and customer variations. Show improved, regressed, unchanged and uncertain outcomes. Added or removed Tests are scope changes, not improvements in agent performance.

Changing business intent is a separate operation from improving the implementation. If the test contract changes, either rerun both agent versions against the new contract or compare only the explicitly unchanged portion. If the runtime, mocks, caller or judge changes, mark a new measurement boundary and rerun the necessary control.

Keep required business and safety checks decisive. Report task completion, policy failures, communication quality and execution/checking failures separately. A favorable average should not hide a critical failure; a presentational failure should still be explained accurately rather than making the whole score look like a diagnosis.

Expose first-attempt performance separately from recovery after a provider retry. Preserve all attempts in ordinary Results, with the technical detail collapsed. Small synthetic samples describe observed behavior; they do not establish a production reliability percentage.

Maintain a small holdout set of customer situations and variants that the repair loop does not optimize against. Add new cases from actual failures and surprising conversations after review and removal of unnecessary personal information. Mark novel AI-generated stress cases as exploratory until their setup and expected outcomes have been checked.

Use the same evaluation service during generation¶

The generated agent and the author's later edits should use the same saved Tests, runtime adapters, checks and Results. Avoid a separate generation-only evaluator with a different definition of success.

The bounded loop should be:

Extract the business contract from the brief, including required outcomes, prohibitions, assumptions and supported channels.
Produce the draft agent, reusable Tests and consistent simulated service data.
Validate configuration, dependency resolution and the test setup before interpreting conversation failures.
Run the relevant core situations against the candidate.
Classify failures: agent behavior, tool/runtime behavior, inconsistent test data, unrealistic customer behavior, or an assessment problem.
Propose and apply a focused candidate change within the agreed authoring scope. Retain the previous candidate and the reason for the change.
Recheck affected situations and critical regressions; evaluate the selected candidate on the full suite and holdout set.
Return the draft with tested revisions, improvements, unresolved failures and the evidence supporting the recommendation.

Use explicit time, model-usage and iteration budgets, with stop conditions for no progress or repeated setup failure. Once the user starts an improvement run, routine sandbox iterations can proceed within that scope. Keep the business contract stable throughout the experiment.

Athens already has assistant tools to read Results and propose Requirements, Tests and Runs. Its Studio handoff saves canonical test artifacts. Proposal tools do not execute Runs. The missing product capability is the measured orchestration and comparison around these pieces.

Source: existing assistant and handoff support.

What Vanguard's experiments contribute¶

The retained HealthPlus journals show why “iterate until the score improves” is insufficient:

In the first round, concise prompts raised the mean reviewer score from 0.380 to 0.610 while strict first-attempt passes remained 0/10.
An unchanged candidate later produced 7/10 and then 5/10 automated passes.
The round-five checkpoint reported 9/10 automated passes but 6/10 first-attempt passes; three conversations required provider-error retries.
Manual audit found both missed defects and unsupported failure allegations.
Some progress required runtime/harness repairs. One early prompt rewrite also removed existing procedure blocks, which a later round restored.
A text comparison changed model and transport together, so it could not isolate model quality.

These are concrete reasons to retain candidate diffs, preserve structured behavior, separate failure causes, record retries, calibrate graders and repeat controls after infrastructure changes. None of these historical cohorts supports a claim about today's overall agent reliability.

Sources: initial frozen comparison, journal and procedure correction, round-five measurements, current experiment guidance.

Ongoing testing, voice and Copilots¶

After authoring, run affected checks when prompts, tools, models, knowledge or relevant runtime dependencies change. Schedule broader suites within a usage budget. Reuse the agreed rules to sample real conversations, surface recurring failures and propose new regression situations.

Synthetic results and production observations should remain separate views with their own denominators and source/version attribution. Passing a mocked service test does not qualify a changed live integration. Add focused connected-system qualification where that dependency matters.

Athens already retains automatic assessment of completed non-Test conversations and recurring-failure suggestion machinery. V1 hides the settings/suggestions surfaces. Bringing a concise, evidence-linked version into the workflow is an extension of existing capability.

Channel	Reusable business checks	Additional evidence needed
Chat	Fulfilled request, correct facts, tool use, refusal, recovery and escalation	Rendering/actions where relevant; connected integration behavior
Voice	The same business situations and obligations	Speech recognition, accents/languages, noise, interruptions, silence, audible closure, latency and channel handoffs
Copilot	Correct facts, permission boundaries, suggested actions and relevant policies	Speaker-aware replay, timing, useful/absent suggestions, card state, duplicate suppression and human interaction

Athens' current voice modes are useful, but their scope matters. Audio mode sends synthesized speech through the actual voice pipeline; assessment reads captured words and events, not audio. The harness is sequential, uses the configured synthetic voice and does not intentionally test barge-in. It does not traverse SIP, WebRTC, carrier routing or real devices. Text and typed-input/spoken-output modes exercise smaller portions of the stack.

Treat voice qualification as business behavior, speech interaction, then real-channel checks. Do not convert a successful text test into a voice-quality claim.

Vanguard already has passive Copilot replay and component assertions, and the Athens transport recognizes that mode. Extend that route with representative/customer turns, workspace context and expected suggestions. Correct silence is a valid outcome; a passive Copilot should not be graded as an autonomous customer-facing agent. Historical replay cannot alone prove that a different suggestion improves the eventual human/customer outcome.

Sources: voice scope, saved mode labels, Vanguard Copilot replay, existing monitoring and suggestions.

Delivery order and pilot¶

Make current results understandable. Correct automatic-update copy, remove placeholders, surface real attention items, clarify percentages and show the tested scope/version. Present saved success conditions and service setup together. Default Results to failures and unresolved problems.
Add dependable comparison. Save an approved baseline, freeze the comparison contract, provide Quick/Full checks, identify changed scope and show matched improvements/regressions with all attempts retained.
Connect generation and improvement. Use the shared Run APIs for bounded candidate loops, failure classification, targeted repair, full-suite rechecks and holdout validation.
Extend qualification and monitoring. Add voice interaction/channel packs, strengthen Copilot replay and expose production-derived regression suggestions.

Start with agent 4210. Freeze its meaningful order-status obligations, customer situations and grading policy; review the unassigned manual placeholder separately. Make one opening-greeting change. Run the affected situations and critical verification/refusal checks, then the unchanged full suite. Compare matched Results and demonstrate that improved greetings did not weaken identity verification or unavailable-action handling.

Include a deliberately broken verification candidate to verify that the test plan catches an important regression. Run an unchanged control to expose natural variation. Then use a voice/Copilot example to validate the shared contract across adapters.

Measure time to a first understandable result, time to identify the right repair, conversation count/usage per iteration, detection of seeded regressions, and agreement between automated findings and human evidence review. Treat these as pilot measurements, not preclaimed targets.

Athens has a coordinated Performance schema cutover in migrations 068–069 and refuses unexpected legacy records. Adopting the work in Vanguard requires a deliberate data/consumer migration; copying the UI or replacing the current tables is not an integration plan.

Verification performed for this review: 24/24 existing focused unit tests passed in the isolated live-source copy, covering Requirement verdicts, check-score denominators, actual tool-result evidence and session closure. These do not test live model quality, browser usability or production channels. No live settings, agent definitions, test records or services were changed.