Analytics migration validation

Status: Current | Last reviewed: 2026-10-02

The Analytics implementation is running on Vanguard with six published agents. The original CCIQ application was also run locally against its original demo sources for comparison. This validates the local migration; it is not an official Dev/QA deployment or a production load certification.

Configurable composition verification — 2026-10-02

Verification used a disposable Builder and a cloned agent on Vanguard, with the existing 999-contact v7 dataset and its existing chat model. The six published demo configurations and their data were preserved.

• Browser: selected AlertTile from the published Cross-Industry catalogue, changed composition guidance and intent drift, waited for collaboration save, and reloaded successfully.
• Real chat: queried all conversations, returned 999 and displayed the selected published catalogue component in Preview. No browser errors.
• Real briefing worker: used a full custom briefing prompt, prompt-selected metrics, a 15-minute on-open interval and three initially expanded secondary findings. The selected catalogue card and all three sections rendered. Reopening reused the same ready briefing ID.
• Isolated PostgreSQL: 25 focused tests passed without skips, including exact catalogue version/tenant checks, renderer namespace isolation, unknown metric handling, report hierarchy and evidence. Intent-share SQL ran through the actual Python/PostgreSQL bridge and produced the expected ±100-point shifts; a targeted 50% deflection scenario produced the expected 4 EUR estimate.
• Python: 15 runtime/configuration tests passed. Typecheck and scoped lint passed. Both documentation sites built; schemas, shared A2UI package and API specs were regenerated.

This verifies configuration and rendering behavior, not a fresh run of the historical 52-turn CallSight parity benchmark. Published catalogue components display analytical evidence; their data tools and business actions are not automatically attached. See composition.

Authoring redesign verification — 2026-10-02

The new authoring route is /agents/{id}/analytics; published consumption remains under Analytics. Verification used disposable Builder/Viewer accounts and cloned agents on Vanguard, plus an isolated PostgreSQL schema for ingestion tests.

• Browser: four-node canvas, bounded viewport, dataset/transcript explorer, valid downloaded CSV example, two simultaneous editors merging independent fields, persisted autosave, slide-out Preview, publishing, Viewer consumption without edit controls and Builder return-to-editor controls. No page errors.
• Runtime: all five briefing-enabled original demos returned ready briefings. Their existing GPT-5.4 models and imported evidence were preserved.
• Providers: live OpenAI generation; live Bedrock Sonnet 4 tool request/result loop and schema-validated annotation; live Titan v2 embeddings at 512 dimensions. The encoded Bedrock model path uses the SDK SigV4 signer. Structured tasks use a forced result tool and local schema validation.
• Google: request/schema/dimension and thought-signature round trips passed with controlled provider responses. Live testing remains blocked by the existing integration's invalid service-account JWT signature; no live Vertex success is claimed.
• Source fidelity: default trend and memory prompts are byte-for-byte identical to the pre-redesign functions for representative inputs. Custom placeholder substitution also passed. Chat and briefing task prompts retain their existing defaults and now support per-agent overrides.
• Focused suite: 69 Analytics/collaboration tests passed against the isolated database, with no skips. Coverage includes quoted/multiline CSV, role mapping, adding/removing retained sources, old-version preservation, cross-tenant denial, encryption/redaction, provider budgets and compiler/report recovery. Gateway: 36 tests passed. Shared PII: 15 tests passed, including Unicode offsets.
• Type checks passed for ai-manager and collab-gateway; scoped lint passed. Config schemas and API specs were regenerated. The wider ai-manager suite passed 1,561 tests but reported three unrelated registry/prompt/cost-catalogue assertions, reproduced on unchanged cbf00e0d, plus one missing-REDIS_URL setup failure that passed when rerun with the normal city environment.

The earlier timing comparisons below remain historical evidence. The authoring redesign does not establish a new performance benchmark or a new production load limit. Original files, model credentials and private chat captures are not included in this document.

Sources and migrated agents

Original repository revision: 720c2a48c3d7599ec2d987cdff0283e5fea1d83b. The archived CallSight comparison below used reference date 2026-06-04, UTC. Current authoring always uses today UTC; repeat this historical comparison with explicit date ranges instead of a configured reference date. Current-week questions cover May 29 through June 4 inclusive; the preceding week covers May 22–28. Explicit dates override those windows.

Vanguard agent	Source	Prepared contacts	Published reports
Insurance FNOL Analytics, 4546	fnol-v1	2	Original interval report and memory pages
Delta Seed Analytics, 4547	delta-airline-v1	250	Native hierarchy and memory pages
Netomi Delta Analytics, 4548	netomi-delta-v1	2,830	Native hierarchy and memory pages
Deloitte Conversation Analytics, 4549	cciq-demo-v7	999	Original imported hierarchy and memory pages
Delta Conversation Analytics, 4550	cciq-demo-v7	999	Same imported snapshot, independent agent settings
FlightStar Conversation Analytics, 4551	cciq-demo-v7	999	Same imported snapshot, independent agent settings

The v7 import includes 4,000 customers, 120 agents, 30 queues, 370 surveys, 202 reservations, 300 proactive messages, four operational changes and 118 flight-disruption events. It preserves 999 annotations, 2,705 annotation evidence rows, 14 memory versions, 101 trends, 421 citations and 115 artifact bodies. Native report rebuilds for FNOL and v7 are also available without replacing their published original snapshots.

The older Delta and Netomi operational tables were empty in the original warehouse. Delta had 250 compiler conversations; Netomi had only five compiler conversations although its source files contained 2,830. The new agents compile all source conversations. This expands coverage; it is not evidence of matching original operational results for those datasets.

Canonical v7 input contains 19,215 speech turns. Its legacy compiler table has 20,505 rows: 1,290 additional legacy utterances are retained separately and redacted, with an import manifest recording the difference. They are not silently added to canonical transcripts. Original SMS channels and non-speech source positions are preserved where required for imported evidence references.

Executed comparisons

The main run exercised 32 scenarios / 52 turns against original CallSight and the published Delta/FNOL agents. All 52 completed without API fallback or broken conversation dependencies. Cases cover:

• Dataset summaries and hourly/daily/weekly reports.
• Driver, friction, AHT, escalation, disposition and baggage comparisons.
• Themes, one/five examples, filtered searches and contact follow-ups.
• Taxonomy/goals, reported-versus-observed intent and reverse citations.
• All six anomaly metrics, operational changes and external disruptions.
• Surveys, customer/queue cohorts, reservations and proactive callbacks.
• All four what-if levers, recommendations and nonstandard date windows.

HTTP success is only a capture check. scripts/validate-analytics-parity.py independently aggregates the original export to check chart values, populations, comparison windows, transcript selection and original report versions. It rejects missing output shapes instead of treating them as passes. All 31 source-fact checks pass after substituting the explicitly identified repair runs. Narrative review remains separate. The sanitized results artifact records each check and keeps the timings of different model configurations separate.

The first GPT-5.4 pass returned correct core chart values but included two inconsistent ranking sentences and an incomplete survey comparison. Evidence review was enabled and strengthened to check comparative claims, requested metrics, denominators and unrequested date filters. The affected conversation sequences were rerun: 13/13 turns across five scenarios completed. Policy count/rate rankings are now distinct; category-goal claims agree with their chart; the survey comparison includes all three measures and all six hub denominators over the full snapshot. The operational diagnosis was independently checked against source counts, including Atlanta's 38 versus 25 policy-confusion contacts and 11 versus one checked-bag-fee inquiry.

Four additional fresh original/native turns cover the Deloitte and FlightStar profiles. All four completed and preserved the selected original report. Separately, all six final published agents passed a live count-and-report check (12 answers), and all five enabled briefings returned ready.

Original responses for the main and repair runs were captured from the running source application, then reused only when source identity and the complete question history matched. Reuse is explicitly recorded; original model requests were not repeated for each repair. The capture specification's old free-text modelNotes names the earlier GPT-4.1 candidate and is retained for baseline identity. Actual tested bindings are recorded here:

• Main 52-turn candidate: GPT-5.4, low reasoning, review off.
• Published final configuration: GPT-5.4, low reasoning, review on.
• Native annotation/report narrative: GPT-4.1 mini.
• Native embeddings: text-embedding-3-small, 512 dimensions.
• Original chat: source default Bedrock Claude Sonnet 4. Imported v7 annotations identify Claude Sonnet 4.5; these are different stages.

See prompt/configuration mapping. The model changes are explicit; no claim of identical original model selection is made.

Timing

The main 52-turn run used a sequential workload, one observation per turn:

Completion time	Original CallSight	Studio, GPT-5.4 review off
Median	11.60 s	11.66 s
Mean	15.20 s	15.12 s
Observed 95th percentile	35.60 s	39.20 s

This run shows similar overall completion time, not a universal speedup. The 13-turn review rerun had a 19.89 s median versus 17.38 s for those same original turns (24.07 s versus 20.72 s mean). Evidence review adds a model iteration; this targeted subset contains harder questions and is not directly comparable with the full-run median. The small mixed-question sample is not a stable production p95, and the concurrently running development environment is not a controlled load benchmark. Tool-status updates are streamed; the final composed answer appears after model completion. First useful text/fully painted chart latency and sustained concurrency still need a dedicated benchmark.

Direct paired tool calls isolate the data-path improvement. One full-population count/AHT query took about 2.26 s in original Athena and 13 ms in PostgreSQL; original weekly report reads took about 5.1 s versus 12 ms locally. These are individual observations, including different cache/storage paths. Exact cosine search remains suitable for these demo sizes; larger datasets need a separate index/warehouse benchmark.

Browser, processing and isolation checks

Three Playwright tests pass against the shared renderer and live Vanguard API: all nine chart forms plus tables; a live report → collapsed details → five-contact picker → on-demand transcript drawer → history/new-chat sequence; and failed briefing → Retry briefing → ready. The chart matrix and retry failure are explicit fixtures; report/evidence/history use real persisted data.

The recorded demo's library/briefing/four-question flow was also exercised. Original and Studio screenshots were inspected at the supplied video's 1296×806 viewport. The source screenshot replays its captured response in the unmodified original UI; the Studio screenshot shows its live persisted answer. This is a layout and interaction comparison, not a pixel-equality assertion between independently generated responses.

The source layout is retained inside the Studio branding: 768px conversation column, 1152px briefing with 300px operating-context rail, compact report framing, collapsed secondary findings/details/tool activity, five initial picker rows, and a right-side transcript drawer. The eight Analytics response components use the shared A2UI registry. They are built-in components, not external catalog-builder bundles. Mock source audio is not presented as a real recording.

Additional executed checks:

• All 16 source tools, through 17 direct calls; original survey SQL and intent drift failures are explicit source defects, with native regression coverage.
• All four citation metric resolvers, current authorization, cross-tenant rejection, immutable snapshots and private history.
• Five database integration tests against an isolated PostgreSQL/pgvector database with migrations 068–071, including restartable compiler checkpoints.
• Four native hierarchy builds, each with 36 configured interval/daily/weekly windows and two independently synthesized dataset memory pages.
• Real-provider transcription/redaction, annotation and embedding preparation, plus resumable bulk ingestion and failed-stage recovery.
• Node regressions, eight Python SQL/privacy/runtime tests, TypeScript and targeted ESLint; configuration-schema and database-schema drift checks.
• Regenerated OpenAPI and SDK documentation: no errors, six existing SDK documentation warnings.
• Analytics worker image build and feature-disabled startup smoke; the final smoke mounts current runtime code. No image was deployed outside Vanguard.

Analytics routes pass the authentication audit, including both briefing methods. The repository-wide audit still reports four existing GET/POST handler findings in allianz-demo/reviews and embed/takeda/interactions/[conversationId]. Those baseline findings keep the global audit red.

The briefing incident was caused by a stale worker rejecting new strict configuration fields. Both services were restarted, failed work was requeued, and all five enabled briefings became ready. Retry briefing now offers recovery without leaving the selected agent. FNOL deliberately has no opening briefing; request its latest interval report rather than a nonexistent weekly source report.

Intentional differences and remaining limits

Studio corrects unequal-duration volume comparisons, distinguishes observed transfer flags from annotation triggers, uses the actual survey NPS formula, and treats peer z-scores as descriptive rather than statistical significance. Source evidence wording is preserved, including original brand names; FlightStar does not rewrite quoted Delta testimony.

Generated narrative and representative semantic results may differ with the model. Evidence review reduces errors but does not establish universal semantic equivalence. Imported report prose may contain illustrative legacy claims; specific metrics use structured population queries. The acceptance catalogue's 38 scenarios / 59 turns includes compiler/browser/failure cases in addition to the 32 chat scenarios. Inventory coverage alone is not a passing functional test.

Runtime model bindings currently support OpenAI integrations. Warehouse imports are bounded typed snapshots, not arbitrary database connectors. Native hierarchy windows use configurable UTC alignment and fixed durations, not calendar-aware local-time calendars. Source deletion scheduling, generalized large-warehouse adapters, production concurrency/load certification and additional model-provider adapters remain outside this delivered implementation.

Reproduce

Keep raw source exports, full responses, login/session credentials and any real transcript screenshots in private files. Public validation output contains only aggregate checks and timing:

python scripts/audit-analytics-parity.py /path/to/trueserve-cciq
cd apps/ai-manager
node --import tsx scripts/compare-analytics.ts SPEC.json AUTH.json PRIVATE-RESULTS.json
node --import tsx --test scripts/analytics.test.ts scripts/analytics-preparation.test.ts \
  scripts/analytics-comparison.test.ts scripts/analytics-reports.test.ts
python -m unittest analytics_runtime.test_runtime
npm run typecheck
npx eslint components/analytics lib/analytics app/api/analytics
# From the repo root:
python scripts/validate-analytics-parity.py SOURCE-BUNDLE.json PRIVATE-RESULTS.json AGGREGATE-CHECKS.json
python scripts/render-analytics-comparison.py PRIVATE-RESULTS.json PRIVATE-REVIEW.html

The standalone review shows original/native answer text and expandable component payloads, with no external scripts or model-authored HTML. It is private evidence, not a replacement renderer or a public handbook page.

For browser tests, set E2E_BASE_URL, ANALYTICS_PARITY_AUTH_FILE and ANALYTICS_PARITY_AGENT_ID, then run npx playwright test e2e/analytics-parity.spec.ts --project=chromium --no-deps. Use the prepared ANALYTICS_PYTHON environment for Python tests. Database integration tests require an isolated validation database; never point their reset/fixture setup at a shared tenant database.