Skip to content

Demo qualification and city parity

Status: Current | Last reviewed: 2026-09-30

Current release status

The September 30 rollout goal is now functional demo parity, at the user's request: make the current demos available in both cities, check their main journeys and integrations, and fix failures that block those journeys. Repeated 90% qualification is no longer a release prerequisite. The unstarted long-run queues were withdrawn; every completed attempt and failure remains in Studio.

Historical qualification is per saved case: at least ten repetitions, with at least 90% strict first-attempt passes for every case. Execution and scoring failures remain in the denominator. A passing pilot or a high aggregate score is not a statistical qualification. The failed candidates and their evidence remain in Agent Studio; none are retrospectively rescored into passes.

Agent Vanguard Macau
Takeda Copilot 4450 Published v17; three native startup cases passed Published v3; the same three native startup cases passed
Allianz Copilot 4446 Published v2; revised document triggers passed 50/50, then 5/5 publication checks Published v2 after the same 50/50, then 5/5 publication checks
Takeda Voice 4453 Published v15; connected handoff and prompt-extracted end field verified Published v3; matching end-field configuration and native Salesforce rehearsal
Consumer Cellular 4451 Published v1; received-payment and pending-AutoPay checks passed Published v1; both focused checks passed
Allianz Voice 4444 Published v2; specialist stream, review submission and approval recap verified; automated caller-stop check failed Published v1; focused handoff smoke passed
Allianz Specialist 4445 Published v10; exercised by the voice handoff Published v1; exercised by the voice handoff
Capabilities Live 4343 Published v1; standard appointment, headshot and copay check passed Published v1; same focused check passed
Capabilities Haiku 4306 Published v2; focused check passed with an explicit extraction lane Published v2; the same focused check passed
Capabilities Nova 3032 Provider rejects appointment sessions; unqualified Copied as draft 5001; existing Macau ADK agent 3032 preserved

The voice dependency import staged eight agents, four workflows, five rebuilt catalogs and 55 saved tests. Reference chat 3026 maps to Macau 5002; colliding source workflow 1324 maps to 5003. Integration credentials remain city-owned. The five imported Takeda saved cases use Macau's +18338518680; their assertions and caller identity are unchanged. Haiku's auxiliary extraction/judge lane is explicitly pinned to the working Live demo's city-local lane with pin-demo-logic-lane.ts 4306 4343. Its primary Haiku model stays the same. An implicit auxiliary default left Macau patient lookups waiting on extraction; explicit selection prevents city-default drift. All eight published main agents passed runtime preflight in both cities after the native Takeda release. Nova remains a separate provider-blocked draft.

Takeda's screen1.svg through screen9.svg apply to the Takeda human-assistant journey. They do not redefine shared Salesforce styling or other demo catalogs. The center workspace and Agent Assist use the same evolving case state. The three patient cards can be present at human answer only when the voice handoff contains those already-discussed patient Cases; a fresh call starts with greeting coaching and reveals patients as they are discussed.

The current native Takeda rehearsal passed in both cities with real Salesforce parent/child Cases, synthetic stereo audio, a transfer card before caller speech, LMS appeal actions, headless field capture and Case/Task saves. The native Studio startup batches passed 3/3 without retries: Vanguard 05f59ae9-9841-41ec-87e5-d4da8b4577c4 and Macau a60afef8-14e3-4351-aa39-6d597a43ed85. The historical nine-case composed-center suite remains disabled with its results retained. These checks do not exercise PSTN/Kinesis or establish a repeated-run consistency percentage.

Both source checkouts include the native release through 42dda18a, including the earlier runtime and import fixes. A source audit verified the same head in both cities. Authoring-script changes do not require restarting calls. The integration branch also includes Macau's newly arrived PrototypeUIAgent merge (8759024b and its two supporting commits). Its Studio dashboard, notifications and test-dialog changes passed 18 focused tests, the full manager typecheck and lint without errors. They do not change the Takeda-specific journey designs. Macau's modified Genesys binary remains unchanged (SHA-256 286e4e88bac89769225845d6af450afe47ba9093d0ffa1ea56a0aadea6f8ef3c). The September 30 fast-forward was preceded by a verified private source/Git/ configuration backup, an idle-call check and a recovery branch. The source diff required no schema or dependency migration. Vanguard's later rollout preserved coeditor changes byte-for-byte and followed an idle-call check. Macau's combined source was fast-forwarded from a checksum-verified Git bundle after its Git remote authentication failed before any source mutation. Its HTTP health and gRPC listener were verified before starting the next tests. Responses ordering, tool-input repair and empty topic receipts are now deployed in both cities. Scoped startup capture is disabled. The original failures remain in Studio. Focused functional checks now replace the repeated qualification gate.

Macau now reuses the retired AutoDesk number +1 833 851 8680 and inbound flow 28bc20f0-d399-460a-a7b2-b685f22a1258, explicitly authorized by the user. The old flow, number metadata and disabled connector configuration are privately backed up. Connector f8zxqbwmfsbvkp2bqxlpug now targets Macau, avoiding the pending quota increases. Following the connected three-patient smoke, the published route now starts Voice 4453 and transfers to Erik and Copilot 4450. The older reserved +1 770 821 9575 remains unassigned. No Vanguard route is used as Macau parity. The new number passed the real Lambda authentication/unknown-environment checks, published-flow comparison, and an idempotent Salesforce initializer probe for Denise Warren / Mayo Clinic. These checks do not constitute a live phone test.

Qualified evidence

Allianz Copilot's revised document triggers passed 50/50 first attempts in batch 6718c1e1-5173-45c3-b544-a3ca7dc30ec9, without retries or execution/ scoring errors. The configuration hash is 1a1b4752c3b2df96171dab2801c8d22870c595723261e42c26a143140f5d2984. The qualified bundle checksum is 89bdb6c955a12a913afb76957bd76457cf082db59cee2456faaa7c3d07621cee; that exact candidate is published as Vanguard 4446:v2. Publication batch 8a7621ee-93ac-4bc7-9fe9-a1c658a4d2d8 passed 5/5. Macau draft batch b14f0296-89a0-4b68-b038-ca4e4cd987d7 passed the same five saved cases at ten repetitions, 50/50 without retries or errors. Its existing approved answer bodies and destination integrations were preserved; only the trigger phrases changed. Its configuration hash is 19fb1d84a779fb618e474199dba55d5c18ae4db150c1bb7352d2ac982196c054. The qualified destination bundle checksum is f40d1103d5576f33181b8d625b3bd214cb368358b0c296d9d5249133faadb17b; that exact candidate is published as Macau 4446:v2. Publication batch 0a9d902b-0906-4060-8be8-a038338bb792 passed 5/5 without retries or errors. Takeda's nine-case Macau runtime check also passed in 6881a6d0-403a-4bd6-bc46-ea0308c2415f. On the later c80e2cc7 runtime, Macau Takeda passed 9/9 in 64aa43bf-28ce-4d26-82a2-89b4b18c1005 and Allianz passed 5/5 in aaceb1d5-34b0-4017-b47f-faebbac2de5a, without retries or errors.

Takeda Voice batch c098aae3-7c9e-421d-a70c-9fe53bc74de0 passed 47/50 first attempts: insurance update and Renee-only escalation each 10/10; corrected identifier, Thomas appeal and three-patient handoff each 9/10. There were no retries, execution errors or scoring errors. The configuration hash is abf5c525df0c01758921364d21189b8df0f5d0b06d3966483587cba746ad11cc. Its qualified export checksum is e4e649462ccf9395b99eb2e3a18c317a526cee7d555726fd52647b2c3bbd1129, published as Vanguard 4453:v11.

The three failed calls remain failures: an unsupported claim that the assistant submitted Thomas's earlier appeal; one incorrectly transcribed corrected member identifier; and a failed handoff after another Salesforce user deleted the active call's parent Case. The deletion was established through queryAll and the modifying user, rather than inferred from transcript timing. The deleted Case was not restored or replaced. Keep generated interaction Cases in place during connected qualification.

The historical v11 publication check repeated the appeal attribution defect: the task backend copied the caller's "we submitted" into its own answer. The revised draft explicitly attributes that action to the practice, with a conditional example grounded in the returned approval and closed status. The original qualified batch and failed publication check remain unchanged. Later functional handoff checks support the current release; the old qualification result does not qualify a newer configuration.

Takeda Vanguard batch 873c0079-bb91-44eb-9485-6b9838765a24 passed 90/90, with no retries, execution errors or scoring errors. Its tested configuration hash is f8326a20c653d2b67ac23d35785b7b8ed4a414fc5bdd6906657fb60a6b234b6f. Macau batch b77c6ae4-4084-4f8e-ab95-9eb15da53ce1 passed the same 90 cases; its remapped configuration hash is cd580951484e7cf5bd8e1b7db14224dbd3bdbe847b1741c4761f3064a1024d9b.

Allianz Copilot's original publication batch is 33a28a16-b88f-4668-a7b1-76b933058b96 (50/50). The subsequent Vanguard batch cc4deb79-3c75-4f41-82fd-f85439762c17 passed 49/50, with every case at or above 9/10. Its one failed caller-speech interpretation remains a failure.

Earlier six-case Takeda batches, older published versions and trials using an older reviewer policy remain historical measurements. Do not combine their best individual results with current candidates. The original run files and configuration snapshots remain private; Git history retains prior investigation notes without making them the current release status.

Retained tuning evidence

  • Allianz Copilot's approved document answer now matches requests such as "what documents" or "do I need a claimant statement." Mentioning an existing death certificate or an unfinished statement during intake no longer triggers generic document, tax or payment guidance. Sources and approved answer text remain unchanged; both cities published the tested v2 candidate.
  • Haiku's appointment delivery states each new confirmation or supervisor outcome once and proceeds to the next required question. A pilot generated approximately 83 seconds of repeated speech after one caller turn and hit the unchanged reply deadline. The revised draft preserves procedures, payment authorization and all saved expectations.
  • Takeda's numeric group parser accepts a spoken “o” token as zero alongside “oh” and “zero.” Candidate eb26685e-0f15-4120-bad8-8b4a94b89e7f exposed a partial parse of “six o one four” as “6”; its unstarted jobs were withdrawn and its failures retained. The fix requires a fresh connected qualification. The subsequent connected batch found a second boundary: a comma after “the group number is” caused the latest correction to be missed, leaving an earlier partial “six” from “six hundred fourteen.” The parser now accepts that punctuation and requests individual digits for ambiguous grouped number words. It never falls back to an older value after a newer ambiguous correction. The exact failed exchange is covered by a regression test; connected qualification must start again after deployment.
  • Allianz Voice's final recap now uses one explicit fact checklist in both business and native speech instructions. The older instruction permitting only a short farewell after a proactive recap is replaced. Saved tests, required qualifications and the caller's right to skip the recap are unchanged. Its next pilot passed two calls but timed out while speaking a pending update followed by a full recap after specialist completion. That same call ended without a caller closing request. The revised guidance limits interim updates and treats the specialist message as a task result; the caller still decides whether to end. The existing reply deadline and failed evidence are retained.
  • Allianz Voice's review setup explicitly starts the transcript stream before dispatching the escalation workflow. A pilot queued the workflow without connecting the supervisor chat, so no human decision could arrive. The revised guidance also recovers an already queued request by starting only the missing stream; it never treats either setup receipt as approval.
  • Allianz Voice's next candidate distinguishes a requested illustration from assumptions mentioned during an estimate question. Preparation means that review is needed, not that it has started. Tool-owned receipts track the prepared request, connected specialist stream and queued workflow; the review dispatch gate requires verified identity, a prepared request and a successful stream. An estimate retains the explicit “not a final surrender quote” qualification. Batch f01e6b4f-bca2-4b60-b762-2784983cb472 exposed these gaps; its started attempts remain recorded and its unstarted work was withdrawn. The saved tests and reviewer policy are unchanged.
  • Live's opt-in delegation_on_user_turn dispatches completed caller captions to the existing client worker. Native and automatic dispatch claim the same input revision; independent application receipts keep their own jobs. This retains native GPT-Live speech, with no separate transcription/TTS cascade.
  • Takeda returns an unsaved insurance draft and a field-by-field spoken readback. Corrections require a new readback and confirmation. A bounded 2.5-second receipt settlement read never repeats the Salesforce write.
  • Capabilities procedures keep continuation instructions within the runtime's 900-character limit. The supervisor decision step covers the decision; appointment confirmation, actual photo attachment and separate charge permission remain ordered prerequisites. Already supplied identity and authorization are reused only where the real tool gates allow them.
  • The next Capabilities candidate checks the actual screen action before asking again. appointment_done confirms that displayed appointment; photos_captured with its stored attachment ends the camera wait. Neither needs another spoken confirmation. A delayed statement about uploading does not undo an already successful attachment. Charge permission still needs a separate caller answer, and a rejected authorization still requires renewal. Haiku's render step now explicitly binds doneAction to appointment_done. The previous step listed the card's data fields but omitted this event binding; several traces used the business tool name instead. The rejected batch and its review errors remain recorded. Private diagnostic reviews are not written back to Studio or counted toward qualification. Haiku's next batch then rebuilt the appointment from generic standard-catalog cards, leaving the required AppointmentSummary step incomplete. Three first-attempt failures in the payment-failure case made that candidate unqualified. Its replacement restores explicit component references and a bounded render instruction naming the selected healthcare catalog. The appointment and camera schemas expand outside the 900-character continuation hint; the actual component requirement and saved tests remain unchanged.
  • Authentication, appointment confirmation, photo attachment, payment and reaction-recording receipts are tool-owned. The model cannot create them with variable writes. An empty search does not undo verified identity, and the two-search limit still applies.
  • Allianz keeps the original illustration request open while review is pending. A queued escalation workflow is not submission. Only the related human specialist's explicit completed-submission message supports its reference. A full spoken recap retains valuation date and commission qualification. GPT-4.1 mini failed this quality contract; subsequent backend comparisons remain unqualified.
  • The latest Responses adapter fix preserves later trusted updates in their conversation position instead of moving them into opening instructions. Tool-input repair also retains the real caller answer and sends internal guidance as an application notice. These changes need deployed-runtime qualification; unit tests alone do not establish agent consistency.
  • Nova's appointment request is rejected even in a fresh provider session with its actual tools. The diagnostic report retains request IDs. Prompt consolidation preserves procedure blocks and consent gates; no provider filter is disabled and these failures are not counted as passes.

Run and inspect saved cases

apps/ai-manager/scripts/demo-qualification/seed.ts saves the maintained Consumer Cellular, Takeda voice/Copilot and Allianz Copilot cases through the same repository used by the Studio editor. It is idempotent for unchanged definitions. Existing Allianz voice and capability cases remain in Studio; record explicit revisions when correcting a test contract.

run.ts run <agent> <draft|published|vN> <repetitions> [name filter] pins cases, configuration, dependencies, goals and fixtures, then uses the ordinary durable workers. --mode=audio|text|auto can select a supported transport for diagnostics; qualification must keep that mode fixed. --concurrency=1..4 records the per-batch concurrency explicitly for load and startup-latency comparisons. It does not change deadlines or excuse failures. Pass --connected-tools only for the intended integration suite. Takeda's connected tests create synthetic Salesforce parent and child Cases and can update Marisol's demo Member Plan. They do not call a payer or place PSTN calls.

Run these scripts from the repository root through with-env.sh and the ai-manager tsx --tsconfig apps/ai-manager/tsconfig.json executable. Results are written privately under .local-data/demo-qualification; override the directory with DEMO_QUALIFICATION_OUTPUT and tenant with DEMO_TENANT_ID. run.ts report <batch UUID> retains all attempts and reports strict first attempt passes per requested case, with retries separate. No average review score substitutes for the repeat-run success rate.

withdraw.ts <batch UUID> <reason> retires only unstarted jobs for a rejected candidate. It retains every attempt and lets active calls finish.

The September 30 comparison uses SIMULATION_JUDGE_MODEL=gpt-5.6-terra, SIMULATION_CALLER_MODEL=gpt-4.1 and the default GPT-4.1 stop verifier in both cities. Macau's first import run used the default reviewer and exceeded some startup checkpoints; it was withdrawn, with its failures retained. Do not compare runs using different evaluator models as environment parity. Literal stop evidence tolerates whitespace from a caption split inside an English contraction, such as you / ’d. It still rejects missing words, paraphrases and evidence from another indexed turn.

export-assets.ts <private directory> <batch UUID> [...] requires ten or more repetitions and at least 90% first-attempt passes for every selected case. Every requested job and first-attempt review must finish before export; nine passes with the tenth still queued or awaiting review do not qualify. The export records one consistent reviewer/evidence/caller policy per batch. It re-pins the current authoring revision and dependencies and refuses drift from the tested configuration hash. --prepare skips only the qualification threshold for migration preparation; its output is explicitly unqualified. For the user-authorized functional rollout, import/publication additionally accept --demo-readiness. They retain qualified: false and identify the release as functional demo readiness rather than statistical qualification. Checksums, unchanged source configuration, integration identity and destination publication validation still apply. --draft-only-agent=<source ID> stages a known-blocked root without publishing it; use this for Nova while the provider rejection remains unresolved. The export selects the tested agent/workflow revisions and mock revisions, includes catalog artifacts and declared integration/secret requirements, and excludes credential values. It also compares current saved cases and goldens with their pinned snapshots, and carries the tested tool configuration, including resolved MCP schemas.

The payment-consent smoke regression accepts spoken US, U S, U.S. and USD in an explicit charge instruction. Previously “You may charge the twenty five US dollars for my co pay” was rejected repeatedly, stranding the standard appointment demo. The current instruction is consumed once; refusals, questions and conditional permission remain blocked.

Allianz's Voice backend disables parallel tool calls: starting supervisor chat and dispatching its dependent review concurrently caused the review's receipt gate to reject dispatch. Sequential calls retain that gate and let the supervisor connection finish before the workflow runs. The failed smoke remains in Studio; apply refine-allianz-tool-order.ts <agent ID> in each city. The custom Responses backend now forwards this option through the shared request adapter; previously the option only reached the default backend.

Migration planning preserves existing destination agents, workflows, tools, capabilities and catalogs. Conflicting identities receive separate destination IDs; catalog name changes require rebuilding the source and CSS namespace. Tools with an existing name/type pair receive a labelled imported name; their references still use the planned destination UUID, preserving the original tool. New destination tool tokens include the bundle fingerprint in their names, so an existing city's Takeda token is preserved. For archived Tailwind v4 catalogs, the importer removes only the exact unused empty tailwind.config.ts starter when styles.css already owns the Tailwind import and sources. Custom themes/plugins and explicit @config references remain intact and require a deliberate port. This fixes rebuilding the older financial-services archive under the current catalog builder. After a failed build, a retry restores that migration-owned workspace from its immutable archive before rebuilding. The catalog ownership marker is checked first; unrelated destination catalogs are never overwritten. Failed version rows remain available for diagnosis. A replacement build takes the next free patch version; explicit source version pins require remapping before that retry, while these demos use @latest catalog references. Keep the generated plan with the export and reuse it for retries. Integration IDs must resolve to active connections of the same type before import. Numeric identity remapping applies to reference fields, not amounts or member identifiers.

import-assets.ts <plan|catalogs|stage|publish> <private bundle directory> macau separates inventory planning, catalog rebuilding, transactional draft creation and native publication. Mutations require a qualified schema-2 bundle and DEMO_IMPORT_ENVIRONMENT=macau. The saved plan covers both agent and workflow numeric namespaces, surfaces, UUIDs and catalog URIs. Catalogs rebuild from their archived source under the destination namespace. Publication uses the target's normal governance and catalog pins; source release approvals are never copied. A failed phase retains its private checkpoint and must be inspected before retrying. Takeda and Allianz Copilot have completed this workflow in Macau; the remaining voice candidates are not qualified imports.

Missing connections use a separate transfer-integrations.ts envelope. Generate the recipient RSA key on Macau, retain its private key there, and pass only the public key to seal. The envelope uses AES-256-GCM with an RSA-OAEP-SHA256 wrapped key; its tenant, destination, expiry and recipient are authenticated metadata. It expires after 24 hours. import decrypts in memory and calls the destination's native encrypted secret store. It creates non-default connections and writes an ID map; it does not overwrite existing connections. Ordinary asset exports remain credential free. Envelope tests cover recipient/tenant/city mismatch, expiry and tampering.

The September 30 envelope for Macau's three missing connections was imported through the destination secret store. The temporary transfer object and recipient private key were removed after successful import. Existing Macau connections were preserved; the private ID map records the created identities.

Start with one repetition to find configuration and behavior problems. Freeze the cases, caller, reviewer, tool fixtures and goals before agent tuning. For the selected candidate, run at least ten repetitions of every case. Require at least nine strict first-attempt passes for each case; include execution and scoring failures in the denominator and show their counts. Record configuration hashes and test revisions. Do not select each case's best historical result from different candidates.

Studio test call context lives in testing.call_context: bounded E.164 caller_number, dialed_number, and a connect boolean. The worker maps these to ANI/DNIS and derives contact IDs from its fresh conversation UUID. Arbitrary metadata, tenant IDs and credentials are not authorable here.

Copilot cases use testing.copilot_replay: ordered customer/representative turns, bounded wait times and component assertions at startup (after_turn=0) or after a numbered turn. Runtime input carries only an ordered index; the authenticated snapshot supplies the words and speaker. Component schemas and Copilot tool_ids dependencies are pinned with the agent. The scorer checks published component state, exact/subset properties and optional text. Late cards do not satisfy an earlier checkpoint. Missing checkpoints, malformed render events, unfinished workers and provider errors cannot pass a negative assertion. These tests exclude STT, browser layout and manual card dismissal. Optional startup_events send frozen host context through the normal Copilot event route before the startup checkpoint. Each contains name, data and delivery (stream by default). This supports an empty greeting, a single patient or multiple supplied patients without a fake caller utterance. seed.ts takeda-native-copilot 4450 saves those three native Takeda tests and disables the historical composed-center cases while retaining their revisions and run history. Native Salesforce/LMS and real Case/Task saves are exercised by native-workspace-rehearsal.ts.

testing.tool_variants selects an existing tool-owned response by operation name. It does not define a new mock or alter connected-tool execution. seed-takeda-mocks.ts <context tool ID> saves checked decoded MCP responses for no handoff, three patients, Renee only and the criteria question. Fixture links use salesforce.example.test; connected voice tests separately verify the real Salesforce data model.

Allianz supervisor chat cases set testing.supervisor.delivery="chat". The driver observes the pinned escalation workflow's successful completion, then sends the saved human decision through the configured supervisor agent. The workflow completion itself is never counted as approval or submission.

Finish runtime edits and verify both HTTP health and the gRPC listener before admitting or releasing queued work. Keep that check and queue release in one sequential operation that stops on failure. The September 30 capture-removal restart released jobs before gRPC was ready; five Capabilities starts received connection-refused errors. Those failed attempts remain in their original pilot and require a fresh run. Vanguard's development watcher restarts active streams on Python edits. Retain interrupted development pilots as errors and start a new batch; do not rewrite their outcomes or include them as completed agent conversations. Audit every Overmind evaluation pool, including alternate sockets: a pool started before a runtime change may still consume the same RabbitMQ jobs. Restart all participating conversation and review workers from the candidate. The September 29 diagnostics found two older pools and retained their mixed runtime pilots separately. Human representative replay captions are excluded from AI speech normalization; the indexed replay evidence preserves their words for review.

Publish and migrate

For the already migrated Allianz Copilot, refresh-allianz-knowledge.ts accepts a qualified checksum-verified source bundle, the existing destination agent ID and its reviewed draft revision. With DEMO_IMPORT_ENVIRONMENT=macau, it updates only approved trigger phrases and refuses changed answer text, citations, speaker scope or concurrent destination edits. It preserves the destination's integrations, catalogs, other workers and identity, saves a private rollback snapshot, and leaves publication pending the same saved Macau qualification tests.

publish-qualified.ts <private bundle directory> uses native automated publication and records a checksum. It does not require Vanguard's browser-test user in another city. DEMO_PUBLISHER_USERNAME can attribute the release to an active local user. Qualification, current configuration and native publication checks still apply.

Preserve Macau's committed history and uncommitted work using the city update runbook. Source synchronization is separate from asset import, secret transfer, database migration and phone routing. Back up and validate the database before any required migration. Preserve destination identities and retain the import maps. Existing credentials, phone routes and Salesforce records must not be overwritten to force parity.

After migration, run the same frozen cases with the same caller/reviewer policy and tools, then separately test destination-specific Salesforce and phone paths. Keep simulation, connected synthetic-audio, browser and PSTN results distinct. Remove temporary browser authentication files and transfer material after their last required check; retain private evidence and verified recovery backups.