Conversation Intelligence Architecture¶

Status: Current | Last reviewed: 2026-06-05

This document describes the AWS Conversation Intelligence Compiler deployed in account 465572510980, region us-east-1. The compiler turns redacted Amazon Connect Contact Lens analysis JSON and isolated synthetic demo datasets into Athena Iceberg tables, Bedrock-style annotations, evidence documents, rolling memory pages, and Codex-readable dashboards.

Executive Summary¶

The system is a serverless analytics compiler for contact-center conversations. It reads only redacted Contact Lens VOICE JSON, normalizes transcript and signal data into Iceberg tables, annotates conversations with convintel.annotation.v1, compiles time-window memory pages, and exposes query/chart tools through a Codex skill.

The current Delta synthetic demo dataset is delta-airline-v1. It contains 250 conversations, 7,027 utterances, 250 annotations, and one compiled driver-summary memory page.

Source¶

Terraform stack: infra/stacks/aws-conversation-intelligence
Compiler Lambda source: compiler_handler.py
Glue loader source: load_staged_to_iceberg.py
Synthetic and workbook data factory: tools/conversation-intelligence
Codex skill: .agents/skills/conversation-intelligence
Operations summary: Conversation intelligence compiler
Architecture diagram source: conversation-intelligence-architecture.mmd
Data model diagram source: conversation-intelligence-data-model.mmd
Visual microsite: conversation-intelligence-microsite.html

AWS Resource Links¶

Area	Resource	Link
Step Functions	`tsf-convintel-dev-compiler`	state machine
EventBridge	`tsf-convintel-dev-every-15-min`	scheduled rule
Lambda	`tsf-convintel-dev-compiler`	function
Glue	`tsf-convintel-dev-load-staged-to-iceberg`	job
Glue Catalog	`trueserve_conversation_intelligence`	database
Athena	`tsf_convintel`	workgroup
S3 warehouse	`trueserve-converse-analytics-warehouse/conversation-intelligence/`	warehouse prefix
S3 synthetic source	`conversation-intelligence/synthetic/source/`	synthetic datasets
S3 memory pages	`conversation-intelligence/memory/`	memory prefix
S3 real source	`dcc-dev-main-sls-awc-call-recordings-742c3e2b503100fa/Analysis/Voice/Redacted/`	redacted Contact Lens source
CloudWatch Logs	`/aws/lambda/tsf-convintel-dev-compiler`	Lambda logs
CloudWatch Logs	`/aws/vendedlogs/states/tsf-convintel-dev-compiler`	State Machine logs

Architecture Diagram¶

flowchart LR
  subgraph Sources
    CL["Amazon Connect Contact Lens<br/>redacted VOICE JSON<br/>read-only source bucket"]
    SYN["Synthetic data factory<br/>Contact Lens-shaped JSON<br/>dataset_id isolated"]
  end

  subgraph Warehouse["S3 warehouse: trueserve-converse-analytics-warehouse/conversation-intelligence"]
    STAGED["staged run bundles<br/>JSONL"]
    ICEBERG["Athena Iceberg tables<br/>Parquet"]
    EVIDENCE["evidence documents"]
    MEMORY["versioned memory pages"]
    STATE["incremental watermarks"]
  end

  EB["EventBridge<br/>rate(15 minutes)"]
  SFN["Step Functions compiler<br/>tsf-convintel-dev-compiler"]
  LAMBDA["Compiler Lambda<br/>discover / normalize / annotate / compile_memory / sync_kb"]
  GLUE["Glue loader job<br/>MERGE staged rows into Iceberg"]
  BEDROCK["Bedrock model<br/>convintel.annotation.v1"]
  KB["Bedrock Knowledge Base<br/>optional, currently gated"]
  ATHENA["Athena workgroup<br/>tsf_convintel"]
  CODEX["Codex skill + helper<br/>metrics, search, charts, dashboards"]

  CL --> EB
  EB --> SFN
  SYN --> SFN
  SFN --> LAMBDA
  LAMBDA --> STAGED
  LAMBDA --> STATE
  LAMBDA --> BEDROCK
  BEDROCK --> LAMBDA
  SFN --> GLUE
  GLUE --> ICEBERG
  LAMBDA --> EVIDENCE
  LAMBDA --> MEMORY
  EVIDENCE -. optional sync .-> KB
  ICEBERG --> ATHENA
  MEMORY --> CODEX
  EVIDENCE --> CODEX
  ATHENA --> CODEX

Runtime Flow¶

Schedule or manual run: EventBridge starts the Step Functions compiler every 15 minutes. Synthetic datasets can also be forced through the same state machine.
Discover: the compiler Lambda lists new .json objects under a redacted Contact Lens prefix. It skips non-redacted real objects and uses conversation-intelligence/state/<scope>/watermark.json for incremental runs.
Normalize: the Lambda validates VOICE Contact Lens JSON, derives conversation_id, maps participants to CUSTOMER or AGENT, and writes staged JSONL bundles for conversations, utterances, and Contact Lens signals.
Load normalized data: the Glue job creates Iceberg tables if needed and MERGEs staged rows into the catalog tables.
Annotate: synthetic data can carry expected labels for demo/evaluation. Normal imported and real data is annotated through Bedrock with the convintel.annotation.v1 schema. When a dataset has a taxonomy.json, the Lambda gives Bedrock that L2/L3/SubCategory label space as guidance, but the row-level workbook ground truth stays outside the prompt. If Bedrock fails, the Lambda emits a low-confidence fallback annotation instead of dropping the conversation.
Load annotations: Glue merges annotations and evidence spans into Iceberg.
Compile memory: the Lambda summarizes labeled calls into Markdown memory pages under conversation-intelligence/memory/ and writes version rows. The MVP writes a run-level driver page; the target model is rolling hour, day, and week summaries that capture issue mix, changes, representative evidence, and context useful to downstream LLM responses.
Knowledge Base sync: the state machine calls sync_kb. The current deployment skips this because enable_bedrock_knowledge_base=false; the Terraform is present and gated on a compatible OpenSearch Serverless vector index.
Analysis: analysts and Codex use Athena, memory pages, evidence URIs, and the packaged helper to answer questions and generate charts/dashboards.

Deployed Configuration¶

Setting	Value
Account	`465572510980`
Region	`us-east-1`
Resource prefix	`tsf-convintel-dev`
Real source bucket	`dcc-dev-main-sls-awc-call-recordings-742c3e2b503100fa`
Real source prefix	`Analysis/Voice/Redacted/`
Existing Connect instance	`dcc-dev-main-connect` / `9156b961-eb34-4d3a-92ad-f770d8f8b282`
Warehouse bucket	`trueserve-converse-analytics-warehouse`
Warehouse root prefix	`conversation-intelligence/`
Glue database	`trueserve_conversation_intelligence`
Athena workgroup	`tsf_convintel`
Compiler Lambda	`tsf-convintel-dev-compiler`, Python 3.12, 1024 MB, 300 sec timeout
Glue loader	`tsf-convintel-dev-load-staged-to-iceberg`, Glue 5.0, 2 x `G.1X`, max concurrency 5
State machine	`arn:aws:states:us-east-1:465572510980:stateMachine:tsf-convintel-dev-compiler`
Schedule	`rate(15 minutes)`, enabled
Annotation model	`us.anthropic.claude-sonnet-4-5-20250929-v1:0`
Embedding model for optional KB	`amazon.titan-embed-text-v2:0`

S3 Layout¶

Prefix	Purpose
`conversation-intelligence/staged/<run_id>/`	Normalized run bundles such as `conversation.jsonl`, `utterance.jsonl`, and `contact_lens_signal.jsonl`.
`conversation-intelligence/annotations/<run_id>/`	Annotation and evidence JSONL bundles before Glue loads them.
`conversation-intelligence/memory/<dataset_or_real>/<page_id>/<version_id>.md`	Human-readable compiled memory pages. The MVP uses run-level driver summaries; the intended pattern is hour/day/week issue memory for dashboards and LLM context.
`conversation-intelligence/memory-index/<run_id>/`	Memory page version rows staged for Iceberg.
`conversation-intelligence/iceberg/<table>/`	Athena Iceberg table storage in Parquet.
`conversation-intelligence/evidence/`	Evidence documents used by KB/Codex evidence retrieval.
`conversation-intelligence/synthetic/source/<dataset_id>/Analysis/Voice/Redacted/`	Synthetic Contact Lens-like redacted JSON.
`conversation-intelligence/synthetic/source/<dataset_id>/taxonomy.json`	Dataset-specific taxonomy used by the Bedrock annotation prompt.
`conversation-intelligence/synthetic/source/<dataset_id>/ground_truth.jsonl`	Evaluation labels from imported/generated data; not used during normal annotation.
`conversation-intelligence/athena-results/`	Athena query output.
`conversation-intelligence/state/<dataset_or_real>/watermark.json`	Incremental discovery watermark.

Data Model Diagram¶

erDiagram
  CI_CONVERSATION ||--o{ CI_UTTERANCE : contains
  CI_CONVERSATION ||--o{ CI_CONTACT_LENS_SIGNAL : has
  CI_CONVERSATION ||--|| CI_LLM_ANNOTATION : annotated_by
  CI_LLM_ANNOTATION ||--o{ CI_ANNOTATION_EVIDENCE : cites
  CI_CONVERSATION }o--o{ CI_CLAIM : supports
  CI_MEMORY_PAGE_VERSION }o--o{ CI_CLAIM : summarizes
  CI_INGESTION_RUN ||--o{ CI_CONVERSATION : processes
  CI_TAXONOMY_VERSION ||--o{ CI_MEMORY_PAGE_VERSION : classifies
  CI_PROMPT_VERSION ||--o{ CI_LLM_ANNOTATION : prompts
  CI_PROMPT_VERSION ||--o{ CI_MEMORY_PAGE_VERSION : compiles

  CI_CONVERSATION {
    string conversation_id PK
    string contact_id
    string dataset_id
    boolean synthetic
    string source_uri
    timestamp started_at
  }

  CI_UTTERANCE {
    string conversation_id FK
    string utterance_id PK
    int sequence_index
    string participant_role
    string content
  }

  CI_LLM_ANNOTATION {
    string conversation_id PK
    string schema_version
    string primary_driver
    string root_cause
    double confidence
  }

  CI_ANNOTATION_EVIDENCE {
    string conversation_id FK
    string utterance_id FK
    string quote
    string rationale
  }

  CI_MEMORY_PAGE_VERSION {
    string page_id
    string version_id PK
    string dataset_id
    string page_type
    string s3_uri
  }

  CI_INGESTION_RUN {
    string run_id PK
    string mode
    string dataset_id
    string status
  }

  CI_CLAIM {
    string claim_id PK
    string dataset_id
    string claim_text
    string metric_sql
  }

  CI_TREND_SNAPSHOT {
    string snapshot_id PK
    string dataset_id
    string driver
    string root_cause
  }

  CI_CONTACT_LENS_SIGNAL {
    string conversation_id FK
    string signal_type
    string signal_name
    string value
  }

  CI_TAXONOMY_VERSION {
    string taxonomy_version PK
    string taxonomy_json
  }

  CI_PROMPT_VERSION {
    string prompt_version PK
    string prompt_name
    string model_id
  }

Table Definitions¶

All tables live in Glue database trueserve_conversation_intelligence and use Athena Iceberg storage under s3://trueserve-converse-analytics-warehouse/conversation-intelligence/iceberg/.

Table	Grain	Merge key	Purpose
`ci_conversation`	One row per Contact Lens contact/conversation	`conversation_id`	Conversation header, dataset boundary, source URI, queue, agent, start time, and duration.
`ci_utterance`	One row per transcript turn	`conversation_id`, `utterance_id`	Redacted utterance text, participant role, offsets, sentiment, confidence.
`ci_contact_lens_signal`	One row per matched Contact Lens category/signal	`conversation_id`, `signal_type`, `signal_name`, `value`	Contact Lens category and signal facts.
`ci_llm_annotation`	One row per conversation annotation	`conversation_id`	Structured Bedrock/synthetic annotation with driver, root cause, confidence, and evidence metadata.
`ci_annotation_evidence`	One row per cited evidence span	`conversation_id`, `utterance_id`, `rationale`	Evidence spans linking annotations back to utterances.
`ci_trend_snapshot`	One row per trend window and driver/root cause	`snapshot_id`	Reserved/materialized trend summary table for faster dashboards and LLM context retrieval.
`ci_claim`	One row per analyst/memory claim	`claim_id`	Stores claim text, metric SQL, contact evidence IDs, and claim confidence.
`ci_memory_page_version`	One row per compiled memory page version	`version_id`	Versioned Markdown insight pages, rolling time-window summaries, and lineage metadata.
`ci_ingestion_run`	One row per compiler run	`run_id`	Run metadata, source prefix, status, and processed object counts.
`ci_taxonomy_version`	One row per taxonomy version	`taxonomy_version`	JSON taxonomy used to interpret driver/root-cause labels.
`ci_prompt_version`	One row per prompt version	`prompt_version`	Prompt text and model metadata for annotation and memory generation.

`ci_conversation`¶

Column	Type	Notes
`conversation_id`	`string`	Stable hash of Contact Lens contact ID.
`contact_id`	`string`	Contact Lens contact ID from `ContactId` or `CustomerMetadata.ContactId`.
`dataset_id`	`string`	Synthetic dataset ID or real-data grouping.
`synthetic`	`boolean`	True for generated demo data.
`source_uri`	`string`	S3 source object.
`channel`	`string`	MVP supports `VOICE`.
`language_code`	`string`	Contact Lens language.
`instance_id`	`string`	Connect instance ID when present.
`queue`	`string`	Queue from Contact Lens/customer metadata.
`agent_name`	`string`	Agent name from metadata.
`started_at`	`timestamp`	Conversation start timestamp.
`duration_ms`	`bigint`	Contact Lens total conversation duration.
`ingested_at`	`timestamp`	Compiler ingestion timestamp.

`ci_utterance`¶

Column	Type	Notes
`conversation_id`	`string`	Joins to `ci_conversation`.
`utterance_id`	`string`	Contact Lens transcript turn ID.
`sequence_index`	`int`	Turn order.
`participant_role`	`string`	`CUSTOMER` or `AGENT`.
`content`	`string`	Redacted utterance text only.
`begin_ms`	`bigint`	Turn begin offset.
`end_ms`	`bigint`	Turn end offset.
`sentiment`	`string`	Contact Lens or synthetic turn sentiment.
`confidence`	`double`	Turn confidence.

`ci_llm_annotation`¶

Column	Type	Notes
`conversation_id`	`string`	One annotation row per conversation.
`schema_version`	`string`	`convintel.annotation.v1`.
`primary_driver`	`string`	Top-level driver.
`secondary_drivers`	`array<string>`	Additional drivers.
`intent`	`string`	Customer intent label.
`root_cause`	`string`	Specific root cause.
`resolution_status`	`string`	Resolved, unresolved, escalated, etc.
`friction_points`	`array<string>`	Identified pain points.
`escalation_triggers`	`array<string>`	Escalation factors.
`channel_conflict`	`boolean`	True when another channel contradicted or failed.
`novelty_score`	`double`	Novelty signal for emerging issues.
`confidence`	`double`	Annotation confidence.
`model_id`	`string`	Bedrock model ID used.
`prompt_version`	`string`	Prompt version.
`created_at`	`timestamp`	Annotation timestamp.

Supporting Tables¶

Table	Columns
`ci_contact_lens_signal`	`conversation_id`, `signal_type`, `signal_name`, `value`, `observed_at`
`ci_annotation_evidence`	`conversation_id`, `utterance_id`, `quote`, `rationale`, `annotation_created_at`
`ci_trend_snapshot`	`snapshot_id`, `dataset_id`, `window_start`, `window_end`, `driver`, `root_cause`, `contact_count`, `previous_contact_count`, `delta_pct`, `created_at`
`ci_claim`	`claim_id`, `dataset_id`, `claim_text`, `metric_sql`, `evidence_contact_ids`, `confidence`, `created_at`
`ci_memory_page_version`	`page_id`, `version_id`, `previous_version_id`, `dataset_id`, `page_type`, `title`, `page_status`, `s3_uri`, `model_id`, `prompt_version`, `taxonomy_version`, `created_at`
`ci_ingestion_run`	`run_id`, `mode`, `dataset_id`, `source_prefix`, `objects_discovered`, `objects_processed`, `status`, `started_at`, `completed_at`
`ci_taxonomy_version`	`taxonomy_version`, `taxonomy_json`, `created_at`
`ci_prompt_version`	`prompt_version`, `prompt_name`, `prompt_text`, `model_id`, `created_at`

Analyst and Dashboard Access¶

The Codex skill ships with scripts/convintel.py, which can query Athena, search representative contacts, read memory pages, and generate charts or a single-file dashboard.

python3 "${CODEX_HOME:-$HOME/.codex}/skills/conversation-intelligence/scripts/convintel.py" dashboard \
  --dataset-id delta-airline-v1 \
  --output convintel-dashboard.html

The helper defaults to AWS profile default when AWS_PROFILE and explicit credential environment variables are absent. Set AWS_PROFILE=dccdev when running from the deployment workspace if that profile has the required account permissions. If live AWS access fails for delta-airline-v1, the helper falls back to the bundled offline synthetic snapshot and labels the result.

Security and Data Boundaries¶

Real data ingestion is limited to redacted Contact Lens JSON under Analysis/Voice/Redacted/.
The compiler rejects real objects that are not under a redacted path.
Original audio and unredacted transcripts are not read by this stack.
Synthetic data is isolated by dataset_id and synthetic=true.
Reset/reseed operations are scoped by dataset_id; S3 purges require an explicit destructive flag in the CLI.
The existing Connect analysis bucket is treated as read-only input.

Current Limitations¶

Bedrock Knowledge Base resources are present in Terraform but disabled until a compatible OpenSearch Serverless vector collection and index are selected.
The MVP covers voice Contact Lens JSON only; chat and audio generation are future extensions.
ci_trend_snapshot, ci_claim, and ci_memory_page_version are the path to rolling hour/day/week issue memory. The current dashboard helper can compute trends directly from ci_conversation and ci_llm_annotation; the next iteration should materialize those windows so dashboards and LLM responses can share the same distilled operating context.