Operations¶
Status: Current | Last reviewed: 2026-10-13
Troubleshooting trees and runbooks. The shape: each "troubleshooting" doc is a fault tree for a class of failures; each "runbook" is a step-by-step recovery for a known recurring task.
Troubleshooting¶
| Doc | When to use |
|---|---|
| Troubleshooting voice | SIP setup, audio failure, call drops |
| Troubleshooting translation | Bilingual / interpretation pipeline misrouting |
| Troubleshooting deploy | Helm install fails, pods crash, ingress wrong |
| Recover an Amazon Connect AI assistant association | AI agents page denies GetAssistant or ListAIAgents; check tags and deleted assistant references |
Runbooks¶
| Runbook | When to run |
|---|---|
| Workflow startup and recovery | Configure reserved worker capacity, inspect durable dispatch, or reproduce startup measurements |
| Rotate master encryption key | Annual rotation or after suspected compromise |
| Rebuild Codespace | Devcontainer wedged; need a clean slate |
| Reset local DB | Schema diverged from init-scripts/ (DESTRUCTIVE — throwaway DBs only) |
| Migrate a populated DB to the squashed schema | Bring vanguard/city DBs to the cleaned-up schema without losing data |
| Migrate a city from MinIO to the S3 storage seam | Bring a pre-cutover city onto the STORAGE_* seam — usually code + env only, the shared-MinIO data already moved |
| Copy workflow templates between environments | Replicate the workflow-template rows from one environment (e.g. richmond) into another (e.g. dev) |
| Recover a clobbered built-in catalog | A built-in A2UI catalog renders blank or shows only ExampleCard after a seed run published the empty scaffold over it |
| Verify multimodal "send link" SMS end-to-end | After a multimodal-SMS fix, or when a customer reports the visual link never arrived — proves the whole path, not just the conversation-api validator |
| Migrate object storage to S3 | Move the shared-MinIO data to the Agent Studio S3 buckets (freeze → sync → flip) |
See also¶
- Core services security audit and fix plan
- AI Manager CVE-2026-94545 assessment
- Richmond AI Manager CVE-2026-94545 assessment
- Status page
-
Agent evaluation workers — durable background tests, capacity, recovery and provider smoke tests.