Allianz demo validation¶
Status: Historical September 28 validation | Last reviewed: 2026-09-30
September 30 phone and console deployment: the published Vanguard agents now have a DevSDO Salesforce console and Connect route. See the current configuration and validation scope. The draft-only and unconnected-phone statements below describe the historical run.
Current voice and styling checks: see the September 30 refinement. The standalone raw-CSS component test below did not exercise native CSS scoping or blob-loaded fonts. It is superseded by the real surface-app iframe checks of producer catalog 1.0.1, including loaded fonts, logo, responsive layout and actions.
Supervision changed on September 29. The workflow/approval results below belong to the original review-desk revision. They do not validate the corrected Salesforce action and supervisor-chat response. See the current supervision checks.
The local demo is installed with voice agent 4444, supervisor chat 4445, beneficiary Copilot 4446, workflow 1847, two published component catalogs and the Allianz surface brand. Use the runbook and the demo walkthrough. The voice and Copilot remain drafts available in Preview. No production call routing or insurer system was changed.
Voice¶
The latest recorded runs for all 7 listed scenario variants passed. These are per-scenario results across different draft revisions, not a full-suite run of one identical revision. Earlier failed attempts remain in the evidence. The rehearsal findings record the subsequent spoken-recap checks and runtime fix.
| Scenario | Latest outcome | Native review score |
|---|---|---|
| precise estimate, no illustration | Passed | 100% |
| complete policy review and specialist approval | Passed | 92% |
| supervisor requests clarification | Passed | 98% |
| policy source unavailable | Passed | 100% |
| supervisor rejects submission | Passed | 100% |
| failed authentication | Passed | 98% |
| follow-ups and spoken wrap-up | Passed | 100% |
Every listed run finalized its evaluation stream and passed its configured platform checks. Additional assertions verify zero policy-business calls after failed authentication, zero illustration submissions in the rejection and values-only cases, and exactly one submission returning IFR-291640 in the complete journey. Human review participants used the real supervisor chat agent, including the clarification question and a new caller answer.
The native tests use actual GPT Live audio and Responses delegation, with sandbox responses for insurance operations and simulated authentication/human decisions. Separate workflow tests execute the actual synthetic Python submission. This does not verify SMS delivery, a real producer identity, an insurer API, or production telephone transport.
Copilot and supervisor latency¶
| Measurement | Result | Sample/conditions |
|---|---|---|
| Correct Copilot cues | 12/12 within 1 second | Six phrases, repeated twice; session connected six seconds before speech |
| Speech end to server card | Median 405 ms, p95 479 ms | Real TTS audio, stereo 16 kHz input and Deepgram transcription |
| Earlier startup probe | 11/12 within 1 second; first cue 3,025 ms after clip end | 1.5-second connection lead; retains trailing TTS silence |
| Supervisor first text | Median 778 ms; range 462–1,265 ms | Three real chat sessions |
| Supervisor response completion | 1,166–3,453 ms | Same three sessions; transcript monitoring stayed silent |
| Browser approval to observed result | 882 ms | One authenticated Chromium run, including result polling |
| Actual workflow approval/rejection | 1029 / 524 ms | One each; polling contributes up to 500 ms |
The primary Copilot measurement removes trailing synthetic silence using the last 10 ms PCM window at or above -54 dBFS. Audio is paced in 20 ms frames. This is an approximate acoustic boundary to server publication, not browser paint. The samples are small and do not establish a production p95 guarantee. Connect the Copilot before the caller starts; fresh account lookups, intake and summary model calls have separate timing.
The live-preview assertions also verify exact gross benefit and included rider amounts, hidden data without trusted verification, unchanged facts rendered once, retained beneficiary/policy intake values and correct approved knowledge sources. Unmatched wording remains outside the bounded approved-phrase pack.
Runtime, workflow and browser checks¶
- 313 Python runtime tests passed, covering Live behavior, consent, Copilot builder/parallel delivery, final-transcript intake and exact tool-result projection.
- 189 focused runtime tests passed during the wrap-up refinement, including long-closing completion and bounded stalls. The final real voice/browser pending-review check spoke through to its farewell 49,011 ms after the end-tool request, then ended once without submitting. See the rehearsal evidence.
- 14 TypeScript tests passed, covering review eligibility/duplicate decisions and simulation target behavior.
- 7 executable fixture tests passed, covering authorization, decimal arithmetic, review gating and knowledge.
- Actual workflow approval and rejection passed. No submit node ran before approval or on rejection.
- Authenticated browser approval passed; unauthenticated reads returned 401 and duplicate decisions returned 409.
- Real catalog React components rendered on desktop and at 390 px width, with no horizontal overflow. The illustration action carried the changed assumption exactly once; the logo decoded successfully and no browser page errors occurred.
- Application and demo-script TypeScript checks passed. Targeted ESLint had no errors and one advisory about the static SVG image element.
- Both catalogs passed native validation/compilation and were published as 1.0.0. Service schemas and generation references were regenerated.
The workflow/browser checks create synthetic active conversation records; their notification subscriber count is zero. Delivery into an actual voice stream is covered by the native complete-journey test.
Evidence and repeatability¶
Raw transcripts, events, screenshots, script output and all historical attempts are in the gitignored .local-data/allianz-demo/ directory. preview-validation.json records the acoustic and earlier clip-based timings. write-report.ts refuses to write this report unless all latest voice variants pass and its critical submission/authentication assertions hold.
| Scenario | Batch | Conversation |
|---|---|---|
| precise estimate, no illustration | 3722a36b-3be4-4847-9e0d-025324863fdc |
d574f1ea-9953-4a06-a558-09f188244706 |
| complete policy review and specialist approval | 35150493-e834-413a-950f-571fc6ac73f1 |
798686a1-e538-4d3c-8212-313f79fff934 |
| supervisor requests clarification | 3722a36b-3be4-4847-9e0d-025324863fdc |
6e52fbc5-ef1a-441b-8bcc-0b9cecbaebf2 |
| policy source unavailable | 3722a36b-3be4-4847-9e0d-025324863fdc |
e37530f9-a9d2-43cf-820d-0a2befa4052c |
| supervisor rejects submission | 94e6f3b5-6f0c-43c8-9cde-d9aa8f1bc31f |
ca39cc96-9c33-4c54-9174-d969be6bbd82 |
| failed authentication | 3722a36b-3be4-4847-9e0d-025324863fdc |
a835c88e-a11a-468c-9384-8efd4b253143 |
| follow-ups and spoken wrap-up | e9518781-321a-4d86-b5d8-ce9c6f7baeb0 |
96215572-0405-4da2-90b1-743e1224e3d1 |
Copilot acoustic timing: 8abc0c06-c395-4f54-8cbf-62b9e746d980. Verified facts/intake: 1eff0d3a-e8a9-40d2-b003-c356e4379725. Browser screenshots: specialist-review-desktop.png, allianz-components-desktop.png, allianz-components-mobile.png.
Remaining integration work¶
Connect approved policy/claims knowledge, producer authorization, CRM data and the intended CCaaS channels before production use. Add durable external idempotency and invalidate outstanding reviews when live assumptions change. Human assistance here demonstrates chat supervision and workflow approval; a real telephony transfer is not connected. The earlier client-delegation teardown issue remains outside the selected managed-delegation configuration.