Analytics question and result parity¶

Status: Current | Last reviewed: 2026-10-02

Use this acceptance contract for the CallSight migration. Completion requires equivalent question coverage, substantive results, evidence, follow-ups and presentation across the original demos. The full analytical runtime implements this contract; measured coverage and remaining source-data differences are recorded in validation. Higher performance is an additional target measured against equivalent work; reduced answer scope does not count as an optimization.

The acceptance catalogue contains 38 scenarios, 59 question turns before expansion across profiles, all 16 registered source tools, eight response shapes and all nine chart types plus tables. It includes API conversations, browser actions, compiler checks and failure injection. It is a coverage definition, not a record of passing end-to-end tests.

What must remain equivalent¶

Dimension	Acceptance requirement
Question coverage	Answer the same question families and combinations of supported filters, metrics, dimensions and follow-ups
Facts	Same source scope, metric meaning, population, units, ordering, comparison windows and authoritative numbers
Findings	Comparable substantive drivers, themes, explanations and actionable recommendations, supported by evidence
Evidence	Correct contact/utterance/artifact versions, accessible citations and consistent representative-example selection
Conversation	Retain topic, metric, filters, source window, selected entities and prior result IDs; handle explicit topic changes
Presentation	Match original card selection, order, text depth, chart semantics, message dimensions and disclosure defaults
Responsiveness	No regression in accepted answer quality; measure first useful content, completion and evidence interaction separately
Configuration	Reproduce the original through versioned presets while allowing alternate domains, providers, processing stages and presentation

Exact prose need not match. Tool names and deployment services can change. Preserve the analytical capability behind them. For imported annotations and fixed rankings, match the source records and ordering. For independently rebuilt annotations or semantic retrieval, evaluate label quality, evidence relevance and substantive findings separately; a different model's label is not an authoritative correction merely because it differs.

Use held-out paraphrases and new combinations of supported dimensions/windows as well as the named cases. A collection of special responses to the recorded questions does not satisfy this contract.

Complete source tool inventory¶

Source: trueserve-cciq revision 720c2a48c3d7599ec2d987cdff0283e5fea1d83b. Tool paths below are under contact-center-iq/backend/tools/; the registry is __init__.py.

Source tools	Behavior to preserve
`query_data_lake`	Flexible read-only aggregation, filtering, joins, comparisons, ranking and record lists; validation/error correction; distinct empty-result behavior
`get_intent_taxonomy`	Fuzzy language to profile-specific intent leaves, hierarchy/flat taxonomy, containment goals and domain vocabulary
`search_transcripts`	Semantic discovery with intent/category/hub/date/driver/root-cause filters; annotation summaries and source excerpts
`get_contact`	Correct full transcript and timestamps, customer/agent/queue context, outcome, observed/reported intent, summary and metadata
`detect_anomalies`	All six metrics, supported dimensions, baseline changes, peer outliers and change-point explanation
`detect_intent_drift`	Reported/source versus observed/model classification mismatches, filters, frequency ranking and contact drill-down
`run_whatif`	AI-first routing, second-auth removal, billing deflection and containment uplift; explicit assumptions, units and directional-estimate qualification
`list_memory_pages`, `read_memory_page`	Discover/read dataset-level executive summaries, active or explicit versions, KPIs and evidence
`list_trends`, `read_trend`	Discover/read hourly, daily and weekly reports with exact windows, versions, hierarchy and evidence
`list_citations`, `resolve_citation`	Source/target evidence lookup, all four metric resolvers, contextual contact pickers and deterministic ranking
`make_chart`	Line, area, bar, horizontal bar, donut, composed, scatter, radial, treemap and table; axes, units, series and grounded highlights
`make_stat_tile`, `make_metric_strip`	One hero KPI or 2–4 related metrics, with correct formatting, comparison and contextual placement

Tool availability is broader than the video. Treat original what-if, classification disagreement, operational diagnosis and prescriptive questions as required migration scope. A configurable replacement can generalize the airline-specific scenario parameters without removing those capabilities.

Inspect handlers as well as their descriptions. For example, the current get_contact handler removed survey/reservation/event queries from its hot path; those datasets remain queryable by the analyst. Its advertised audio player is a mock. Do not claim a functioning recording exists from that flag alone.

Data and analytical semantics¶

The original business substrate includes contacts/transcripts, customers, agents, queues, surveys, reservations, proactive messages, internal operational changes and external flight-disruption events. Import supported source fields and their relationships, not only transcript objects.

Required contact dimensions include reported intent/category, hub, queue, agent, customer/tier, channel, handling mode, disposition, operational containment/escalation, AHT and repeat contact. The canonical import preserves the original SMS channel alongside voice and chat. Retain missingness, source provenance and join cardinality.

Metric definitions must include AHT, containment, escalation, repeat-call rate, CSAT, top-two-box satisfaction, NPS and proactive-message outcomes. Survey metrics use respondent populations; transcript sentiment cannot substitute for CSAT. Goals such as expected containment are configuration, not observed performance. Reservation outcomes such as rebooked/flew cannot be inferred from a transcript's resolution label.

The compiler contract supplies rich annotations, taxonomies, evidence, representative-example scores, hourly/daily/weekly reports, dataset summaries and dependency-aware rebuilds. Preserve primary driver, secondary mentions, friction and escalation as separate measures.

Every aggregate needs an explicit numerator/denominator, filter, time window, timezone, units and release. Deduplicate joins at the appropriate fact grain. Represent absent values as unknown, and report sparse/partial coverage. Keep the original reference date and explicit historical window boundaries when comparing demo answers.

Question routing and investigation¶

The original runtime supports multiple model/tool rounds and parallel independent tools. Its prompt deliberately selects different paths:

Broad dataset narrative → executive summary, with supporting queries.
Time-window narrative → matching compiled trend, with a freshness check.
Specific count/rate/comparison → authoritative structured metrics.
Diagnosis → metrics, segment drill-down, transcript evidence and the applicable operational-change or external-disruption context.
Customer themes → search and summarize; open individual transcripts when requested rather than displaying every retrieved excerpt.
Record listing → exact rows with contact IDs, timestamps and requested order.
Prescriptive question → establish the problem, scope the opportunity, recommend a concrete move and identify ownership and a measurable outcome.
What-if question → supported scenario calculation with explicit assumptions.

The source permits exact structured artifact aggregates for a matching metric and window, but illustrative prose numbers are never a numeric authority. A report outside the requested window does not substitute for the query. Correlated changes are hypotheses supported by evidence, not proof of causality.

Full analysis uses the original 16-tool loop over scoped PostgreSQL relations, with bounded query repair, versioned attachments and an optional answer review. Basic transcript mode remains available for small deployments.

Use composable typed analytical operations or an equivalently constrained query planner over authorized views. Support the original combinations of joins, groupings, date windows, rankings and comparisons; a small menu of hardcoded queries is insufficient. Allow bounded validation/error repair and parallel independent work while preserving result and attachment order.

Conversation state must retain structured tool results and their IDs, not only truncated answer prose. Full analysis retains eight complete tool turns and their attachment IDs in a private session bound to the published agent version and immutable snapshot. Direct citation/contact opening should avoid an extra model turn.

Recommendations and presentation¶

Preserve the original recommendation categories: deflection, containment uplift, first-call resolution, handle-time reduction, root-cause elimination, agent enablement, digital-channel improvements and proactive communication. Ground quantitative benefit in source metrics or an explicit scenario calculation. Expose important assumptions and what evidence would change the recommendation. Recommendations are advice; they do not execute operational changes.

Follow the catalogue and recorded-demo contract for exact composition and disclosure defaults. Include the full library → briefing → chat → evidence journey. Ordinary answers, explicit reports, themes, one contact and multiple contacts have different shapes.

Test chart/metric/table selection, contextual recommendation/chip placement, normal and diagnostic prose length, no repeated facts across cards, meaningful focus explanations, highlighted chart rows, correct singular/plural contact counts and no dropped attachment anchors. Test nested expanders, table previews, old-message interaction, drawer navigation and scroll restoration. Replay identical saved response payloads in both renderers before introducing model variability.

Known source bugs should have explicit correction cases: scope drifting after a briefing, stale dependency caches, incomplete child windows, unequal-duration volume baselines, and inconsistent operational transfer flags. Preserve source evidence while documenting corrections. Brand aliases may change display copy; they must not rewrite quoted source testimony.

Baselines and release gates¶

Run applicable cases across FNOL, the original Delta seed, Netomi Delta and CallSight. Test the Deloitte, Delta and FlightStar brand configurations separately even when they share data. The catalogue's delta-original and netomi-delta profiles are local validation bindings, not shipped source profiles. Their original operational warehouse was empty during the October 2 comparison; Delta had 250 compiler records and Netomi only five despite 2,830 source conversations. Native compilation covers the full migrated source. These are coverage expansions, not matching operational baselines. An empty-source answer is not a passing numerical comparison.

Capture source revision, data/operational export manifests, agent versions, reference date/timezone, annotation/taxonomy/metric/artifact versions, models and runtime limits with each comparison. Preserve raw responses and evidence privately. The original saved sweeps supply useful prompts and structural regressions but contain permissive checks such as plausible numeric ranges; use exact source-backed expectations for migration acceptance.

Track every required scenario as pending, blocked, failed or accepted, with evidence. Require:

Source and native data manifests reconcile; missing source relationships are enumerated rather than silently omitted.
Deterministic facts, windows, denominators, ranks and evidence pointers match.
Findings, diagnosis, recommendations, uncertainty and follow-ups satisfy semantic review against the same evidence.
Original response composition and recorded browser interactions pass.
Alternate profile configuration and published/preview behavior pass without custom runtime code for each demo.
Performance meets the agreed target on the same accepted workload.

Do not declare parity from HTTP success, a passing compiler unit test or a matching contact count. Annotation agreement does not prove causal correctness. Security and Studio lifecycle checks remain required improvements alongside the original behavior: SSO, tenant/audience scope, revocation, feature gates, history persistence and pinned evidence.

Measure cold and warm behavior, p50/p95 completion, time to first useful content, charts ready and citation/drawer latency, plus errors, fallback rate, provider rounds/tokens, query time and processing cost. Compare equivalent work at realistic concurrency; sequential runs with two repetitions cannot establish p95 or load capacity. Report preprocessing cost separately from chat. Cached or degraded answers are identified explicitly.

Optimize indexed metrics, compiled report reads, evidence lookups, shared artifacts, deduplicated briefing refresh, provider caching and independent tool concurrency. Preserve facts and useful response depth while reducing work. The earlier pilot latency comparison covers fewer capabilities and remains an initial measurement, not a parity performance result.

Coverage and comparison tooling¶

Validate inventory coverage without cloud access or model calls:

python scripts/audit-analytics-parity.py /path/to/trueserve-cciq

The audit reads the catalogue's pinned Git revision and checks every registered tool, chart variant and declared response shape has a scenario. It does not execute questions or certify behavior. Use the catalogue to select applicable cases and populate private run specifications with the actual agent bindings and source-backed expectations.

The comparison runner accepts the existing single question or an ordered turns array per case. Both apps reuse their own session within a sequence; repetitions start fresh. Per-turn checks support exact native JSON values and JSON Pointer comparisons with normalized original text, attachments or tools. Specify an absolute numeric tolerance only when the metric's rounding requires it.

{
  "id": "themes-to-examples",
  "agentId": "4550",
  "profile": "airline-delta",
  "turns": [
    {"question": "What are customers saying about vouchers?"},
    {"question": "Show a few representative calls"}
  ]
}

Run with private specification/authentication/output files:

cd apps/ai-manager
npm run analytics:compare -- SPEC.json AUTH.json PRIVATE-RESULTS.json
node --import tsx --test scripts/analytics-comparison.test.ts

An optional baseline contains cciqRevision, snapshotId, referenceDate, timezone and modelNotes. These identify declared inputs; the runner does not independently verify the warehouse snapshot. reuseCciqResults requires matching source inputs and full conversation history. Legacy one-question captures without identity metadata are labelled legacy_unverified and cannot be reused when a baseline is declared.

Incomplete original SSE, malformed events, missing native sessions and fallback warnings fail capture; dependent turns become blocked. Independent cases continue. Failed captures or explicit checks return a nonzero exit. Results are checkpointed atomically with private file permissions. Full answers stay in the file; terminal output contains statuses and timings only. qualityReview: pending remains explicit even when all automated checks pass. The harness measures original first text separately; browser first-useful-content, visual review, briefing refresh and load measurement require their own runs.