Skip to content

Conversation Intelligence Compiler

Status: Draft | Last reviewed: 2026-06-05

AWS-native compiler for Contact Lens analytics. It normalizes redacted conversation JSON, enriches calls with Bedrock annotations, stores facts in Athena/Iceberg-shaped tables, writes evidence documents, compiles memory pages, and exposes analyst tools through MCP/Codex.

Source

AWS Shape

Component Value
Account 465572510980
Region us-east-1
Existing Connect instance dcc-dev-main-connect / 9156b961-eb34-4d3a-92ad-f770d8f8b282
Real source Redacted Contact Lens JSON under Analysis/Voice/Redacted/
Warehouse s3://trueserve-converse-analytics-warehouse/conversation-intelligence/
Glue database trueserve_conversation_intelligence
Athena workgroup tsf_convintel
Schedule EventBridge rate(15 minutes)

The compiler treats the existing Connect bucket as read-only. Synthetic demo data is written to an isolated warehouse prefix and identified by dataset_id.

Runtime Flow

  1. EventBridge starts the Step Functions compiler every 15 minutes.
  2. discover lists new redacted Contact Lens JSON objects, using a warehouse watermark so empty runs exit quickly.
  3. normalize stages conversation, utterance, and Contact Lens signal JSONL bundles for the run, then Glue appends them to Iceberg.
  4. annotate uses synthetic expected labels when evaluating generated data, or Bedrock annotation for real transcripts.
  5. Glue appends annotation and evidence rows to Iceberg.
  6. compile_memory writes versioned Markdown pages under the memory prefix and Glue appends page-version rows.
  7. sync_kb starts Bedrock Knowledge Base ingestion when KB resources are enabled.

The Glue loader creates the Iceberg tables if they do not already exist and upserts core fact rows with Iceberg MERGE, so replayed runs do not duplicate contacts. The Step Functions Glue states retry transient concurrency errors. Athena named DDL queries are still created for inspection and manual repair.

Synthetic Demo Data

The reusable generator defaults to 250 conversations with 18-36 utterances, 4-12 minute simulated durations, repeat contacts, escalations, sentiment movement, and configurable trend spikes.

cd tools/conversation-intelligence
python -m conversation_intelligence.cli generate \
  --config examples/airline-demo.yaml \
  --dataset-id airline-v1 \
  --output s3://trueserve-converse-analytics-warehouse/conversation-intelligence/synthetic/source

Use reset to write a tombstone. Add --purge-s3 only when intentionally deleting generated synthetic objects.

Netomi Workbook Adapter

The Netomi workbook adapter turns Netomi Call Transcript Extract.xlsx into a synthetic-source dataset that still uses the real compiler path. It extracts the transcript and summary columns, repairs rows where those two columns are swapped, redacts [*masked_*] markers to [REDACTED], emits Contact Lens-shaped JSON under Analysis/Voice/Redacted/, and writes the workbook L2, L3 Category, and L3 SubCategory values as a reusable taxonomy.

cd tools/conversation-intelligence
python -m conversation_intelligence.cli import-netomi-xlsx \
  --input ../../Netomi\ Call\ Transcript\ Extract.xlsx \
  --dataset-id netomi-delta-v1 \
  --brand Delta \
  --industry airline \
  --output s3://trueserve-converse-analytics-warehouse/conversation-intelligence/synthetic/source

For smoke testing the full Bedrock path, import a small dataset first:

python -m conversation_intelligence.cli import-netomi-xlsx \
  --input ../../Netomi\ Call\ Transcript\ Extract.xlsx \
  --dataset-id netomi-delta-smoke-v1 \
  --limit 25 \
  --output s3://trueserve-converse-analytics-warehouse/conversation-intelligence/synthetic/source

The adapter writes taxonomy.json and taxonomy.md next to the dataset. During annotation, the compiler loads conversation-intelligence/synthetic/source/<dataset_id>/taxonomy.json and asks Bedrock to prefer those taxonomy IDs when the transcript supports them. The row-level labels are stored in ground_truth.jsonl for evaluation only and are not embedded into normal Contact Lens documents or evidence files.

The full workbook import creates 2,830 Contact Lens-shaped JSON files. Keep production-like imports bounded and run the Netomi dataset with the batch wrapper:

cd tools/conversation-intelligence
AWS_PROFILE=dccdev scripts/ingest-netomi-delta-batches.sh

The wrapper processes netomi-delta-v1 in 100-call batches by default, writes progress under batch-ingestion/netomi-delta-v1/<batch_run_id>/, and compiles a final dataset rollup memory page from Athena. Use --dry-run to preview the plan, --max-batches 1 for a first live batch, and --start-batch N --batch-run-id <id> to resume.

Operations

cd infra/stacks/aws-conversation-intelligence
terraform init
terraform plan -out convintel.tfplan
terraform apply convintel.tfplan

For local validation:

cd tools/conversation-intelligence
python -m pytest
python -m conversation_intelligence.cli generate --dataset-id smoke-v1 --count 3 --output ./out
python -m conversation_intelligence.cli validate --dataset-id smoke-v1 --input ./out

Notes

  • The Bedrock Knowledge Base is optional in Terraform until an OpenSearch Serverless collection and vector index are selected.
  • The MVP covers voice Contact Lens redacted JSON only.
  • No userguide update is required because this change adds no UI.