Allianz Script v2 validation¶

Status: Historical baseline | Last reviewed: 2026-10-10

For the October 10 DCC human-service integration, see Allianz DCC human service.

For the subsequent self-service migration of agent 4873, see Allianz self-service v2. The baseline below tests the older 4444 voice agent and the separate 4870 human-assisted journey. Its results do not describe the revised 4873 pipeline.

At the October 9 baseline, the published Allianz agents did not satisfy Allianz Script v2.docx throughout the producer journey. Use this report for the document's automated call, human-assisted call and native DevSDO wrap-up. The voice test scores 40% against the script. Native illustration capture, saving and wrap-up work; the Copilot has gaps in qualifications, native state synchronization and the proactive signature reminder.

This is a test baseline, not a publication. No agent configuration, catalogue or Salesforce application metadata was changed by these tests.

Source and tested assets¶

The frozen script preserves all 28 speaker turns and the document's tables and stage directions. Its source document SHA-256 is d0094cfbfc91c552aa6d981dc0d3b1662ec7b279d3072f4bea8ecca4b9143937. Stage directions and the written calculation disclosure are not synthesized as spoken dialogue.

October 12 valuation and October 13 verification timestamps in the supplied fixtures are future authored demo dates relative to this October 9 test. They are neither the test date nor live financial valuations. November 1 and November 14 are also authored scenario dates.

Asset	Published version	Coverage
4444 — Allianz Producer Services	4	Actual voice simulation, original tool fixtures, simulated verification
4870 — Allianz Illustration Assist	17	Exact human-side dialogue, real stereo STT and DevSDO browser
Allianz Illustration Assist catalogue	4.0.5	Published catalogue used by the current experience
4445 — Allianz Producer Specialist	11	Configuration audit only; the script requests a live representative, not specialist chat review
4446 — Allianz Beneficiary Assist	3	Configuration audit only; the separate beneficiary caller scenario is absent from this document

The earlier SVG workspace validation tested 4870 v11 and catalogue 3.0.4. Those results do not establish compatibility of the subsequently published v17/4.0.5 pair with the native host.

The two call legs are tested separately. The human replay starts with the authorized native fixture; it does not receive a real telephony handoff from the voice simulation. This report does not claim a continuous PSTN → SCV call.

Automated call results¶

Saved Agent Test case: c59d7de7-d967-4bd8-8bed-e6523485f696, revision 2. Batch: 8d324246-9607-4884-8c94-6abb97ed3f45. Conversation: 2bd9a09a-bf21-4a93-96ad-83e64baa512b. Pinned evaluation configuration hash: 11cb84722aa7c913a80bfdbcce01777635a07446acf39f469d4cb18016cc2ef7.

The native evaluator completes with 40%, failed. This is one scripted evaluation, not an aggregate reliability estimate. Manual inspection of the transcript, tool calls and terminal transfer event confirms the principal findings.

The simulated caller uses the document's value request, “tell me more,” and representative request, while answering extra identity or clarification questions when necessary. Expected October/November facts are assertions; they are not injected into the agent's tool responses or taught by the caller.

Document requirement	Result	Observed behavior
Personalized Producer Services greeting	Mismatch	Generic Allianz Life / Taylor greeting; collects producer, agency and contract later
Consent and secure verification before lookup	Pass in simulation	Requests consent, starts passkey verification, then retrieves the policy
Explicit Producer of Record confirmation	Missing	Does not explicitly confirm that authorization to David
$184,320.14 value and $181,555.34 estimate	Pass	Speaks both amounts and creates the policy A2UI surface
October 12 fixture valuation date	Fail	Existing policy source returns and agent speaks September 28
Good Order and market-performance qualification	Fail	Gives only before-tax/adjustment and non-final-quote qualifications
November 14, 5% → 4%, $552.96 saving	Fail	Missing from both the existing source and spoken answer
$13,824 free-withdrawal privilege / partial option	Fail	Missing; follow-up explains subtraction and the crediting rate
Initiate representative handoff with summary	Pass at agent boundary	`end_session` uses `transfer_requested`, `session_escalated=true`; terminal event has `is_transfer=true` and the summary
Actual device/webchat and downstream queue transfer	Unverified	Test device/authentication and attached visual client are simulated

The policy fixture itself lacks the new alternatives. Changing the prompt alone would not supply approved facts. The agent correctly avoids inventing October/November data, but that still leaves the new script unmet.

The first exploratory batch, 9c3cb5e9-b221-46c8-aab5-4e58262b7c00, scored 45% and exposed the same content gaps. Its saved test also contained irrelevant supervisor-chat checks. Revision 2 removes those checks and is the reported voice baseline; the earlier evidence is retained.

Human-assisted call results¶

The first full replay uses conversation 998f18f7-f931-4c25-b842-ec831b472cd6, Case 00023178 (500DW00000Gav9RYAR), and receipt DEMO-IFR-CF61C6374C. It reaches the final wrap-up despite individual failed checks. A repeat on the same published Copilot also captures the form correctly but shows inconsistent benefit, clarity and setup cues; a single successful cue is not a reliable pass.

The corrected repeat uses conversation 22fe95ee-51d6-4d53-b5e3-1a54ea50f3cf, Case 00023181 (500DW00000GavAFYAZ), and receipt DEMO-IFR-55713D5B3F. Its pinned 4870 configuration hash matches the audit: fe2b2380e8cb8077699f2c53e8847c7fa929c450c136aac8c3e1341a75f9e727. Both replays finish the journey and fail overall acceptance.

Document requirement	Result	Evidence
Benefit question before clarity interruption	Inconsistent	Correct sequence in the first replay; opening guidance persists during the repeat
Benefit amounts and proportional reduction	Correct when shown	First replay displays all five exact amounts; repeat misses the benefit checkpoint
Complete written benefit qualification	Fail	Card omits Good Order, taxes, free-withdrawal eligibility and explicit MAV-remains-controlling condition
Clarity cue and enhancement action	Inconsistent	First cue/button work and unmatched call is rejected; repeat misses clarity cue
Enhancement reflects actual audio acknowledgement	Fail for native integration	Later speech causes simulated “on” state despite the rejected call match
Measured WPM available to components	Pass	Original STT publishes separate agent/customer timing surfaces
Open native illustration form	Inconsistent	Copilot opens it in first replay; repeat requires the native Illustrations tab
Capture amount/frequency, date and duration	Pass	$1,000 monthly, November 1, 2026, five years; three of five groups
Question alone does not confirm assumptions	Pass	Allocation/purchase-payment question leaves fields unconfirmed
Caller agreement completes five groups	Pass	Current allocation and no future purchase payments captured; submission remains gated
Proactive complete modeling read-back prompt	Fail	Sidebar still asks for already captured assumptions; exact level/tax/proportional/carrier-approved prompt is absent
Save and submit reviewed native request	Pass	Real Apex save returns a durable demo receipt
Copilot immediately follows saved receipt	Fail	Remains in illustration state for the 25-second check, advances after further speech
Delivery claims follow actual delivery evidence	Fail	Intermediate model output says the document is available/delivered while `deliveryStatus=not_sent_demo`
Proactive Jack Monahan reminder	Inconsistent	First replay loses supplied Jack data; repeat shows the correct reminder
Closing after reminder acknowledgement	Pass	Closing guidance uses the actual receipt and qualifies delivery as unconfirmed
Copilot opens existing wrap-up	Pass on corrected repeat	Complete call opens the native receipt-backed summary
Existing DCC summary and Complete action	Pass	Saves notes, closes Case, completes Voice Task and creates reminder Task

The enhancement card also displays “Demo preview. Audio is unchanged.” The failure is a native/demo mode and status mismatch, not evidence that audio was actually processed. Audible before/after conversion still needs a matching SCV call.

The native form and sidebar disagree after capture and submission. For example, the native form has all five groups while the sidebar shows allocation as “Listening” and purchase payments as “Pending.” The pending-action fixture has Jack's contract, request and next step; the first replay's rendered Copilot payload instead contains an empty pending object. The repeat retains it. These are observed native host/Copilot integration failures; their exact context-loss mechanism requires a separate implementation fix.

Two initial browser assertions were harness errors: Salesforce's inner date input displays Nov 1, 2026, and catalogue 4.x renamed the closing action to Complete call. The maintained rehearsal reads the public ISO date value and recognizes both closing labels. WPM and audio status, and native receipt save and sidebar receipt state, have separate assertions.

Remaining end-to-end boundaries¶

A real authentication device, SMS link and customer webchat were not exercised.
The agent emits a transfer, but Connect queue routing and verified context across the actual handoff were not tested.
Human-side calculation sharing into the customer webchat was not exercised.
No illustration document is generated or delivered to webchat or producer portal. The actual receipt records a demo request. The document's promise of delivery “in a moment” is not supported by that operation.
XYZ remains an unresolved purpose placeholder in the document's signature reminder. The authorized SVG/native fixture identifies a beneficiary change for Jack; the model should use that fact rather than invent a purpose.

Follow-up before a scripted demonstration¶

Align 4444's policy source, capability fixtures, customer card and spoken requirements with the document's authored valuation and withdrawal options. Retain verification and the actual transfer signal.
Reconcile the current catalogue/Copilot contract with the native host. Progress, pending action, receipt and audio status must come from the same trusted Case snapshot. Keep native audio mode distinct from the independent demo's simulated control.
Carry the complete written benefit qualification and modeling read-back. Preserve the selected delivery route without claiming document delivery.
Repeat the saved tests, then exercise a connected SCV call and the intended customer webchat/document delivery path to cover the remaining boundaries.

Repeat the tests¶

Use the local dev stack, its simulation workers, an authenticated devsdo Salesforce alias and the existing clean producer fixture. The scripts save test definitions/results and create synthetic Cases/Tasks. They do not publish agents or deploy metadata.

Run from the repository root:

cd apps/ai-manager
bash ../../.devcontainer/scripts/with-env.sh node_modules/.bin/tsx scripts/allianz-demo/script-v2-tests.ts audit
bash ../../.devcontainer/scripts/with-env.sh node_modules/.bin/tsx scripts/allianz-demo/script-v2-tests.ts voice-start
# Export the returned batch's status, review and transcript.
bash ../../.devcontainer/scripts/with-env.sh node_modules/.bin/tsx scripts/allianz-demo/script-v2-tests.ts BATCH_UUID
bash ../../.devcontainer/scripts/with-env.sh node_modules/.bin/tsx scripts/allianz-demo/producer-workspace-rehearsal.ts --script-v2
# Run after the rehearsal reaches a saved receipt, including when semantic checks fail.
bash ../../.devcontainer/scripts/with-env.sh node_modules/.bin/tsx scripts/allianz-demo/producer-wrapup-test.ts --script-v2

The rehearsal pins the published Copilot version at startup, uses the exact human dialogue through stereo transcription, and continues after a failed semantic checkpoint. A nonzero exit can therefore accompany a completed journey; inspect checks and journeyCompleted separately. It retains the original SVG rehearsal when --script-v2 is absent.

Private transcripts, per-check screenshots, pinned configurations, native review output and Salesforce save evidence live in .local-data/allianz-script-v2/. Keep raw evidence, Salesforce session URLs, credentials and source DOCX outside committed reports.

Harness validation also checks the source document hash, all 28 ordered turns, critical fixture text and documentation links. Targeted ESLint reports zero errors and six existing explicit-any warnings in the transport/browser helpers.