Observability

Status: Current | Last reviewed: 2026-09-17

Logs, traces, metrics, and the status page.

Tracing: two streams, one mechanism

All tracing is generic OTLP — endpoint + protocol + headers. There is no backend selection switch: Arize Phoenix, LangFuse, and OTel collectors are all just OTLP destinations (see recipes). The shared bootstrap lives in libs/telemetry/ (Python, consumed by ai-orchestrator and async-agent via their lib/telemetry.py / app/telemetry.py shims) and apps/ai-manager/lib/telemetry/routing.ts (TypeScript mirror).

Spans are partitioned into two independently routable streams:

Stream	Contents	Destination env vars
infra	HTTP/gRPC server spans, worker/job spans, framework auto-instrumentation	OTEL_EXPORTER_OTLP_ENDPOINT / _TRACES_ENDPOINT / _PROTOCOL / _HEADERS
ai	Conversation/session spans, per-turn spans, LLM/STT/TTS/S2S usage spans, workflow-run + node + provider spans	OTEL_AI_EXPORTER_OTLP_ENDPOINT / _PROTOCOL / _HEADERS

Fallback: when OTEL_AI_EXPORTER_OTLP_ENDPOINT is unset, AI spans go to the infra destination — a single-destination deployment needs zero changes. When it IS set, the streams are disjoint partitions.

Classification happens at span end in libs/telemetry/classify.py, by ordered rule: explicit telemetry.stream attribute → instrumentation-scope allowlist (pipecat, ai-orchestrator.conversation, ai-orchestrator.service-usage, ai-manager.llm, …) → attribute heuristics (openinference.span.kind, gen_ai.*, span.type=llm_request) → default infra. A collector can also split on the telemetry.stream span attribute — see the routing-connector reference in otel-collector/values.yaml.

Key environment variables:

# enablement
ENABLE_TRACING=true                     # or ENABLE_OTEL, or implicitly by setting an endpoint
OTEL_SERVICE_NAME=ai-orchestrator
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=sentinel,service.namespace=trueserve

# infra stream
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

# AI stream (unset = falls back to the infra endpoint)
OTEL_AI_EXPORTER_OTLP_ENDPOINT=http://phoenix:4317
OTEL_AI_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_AI_EXPORTER_OTLP_HEADERS=Authorization=Basic <token>
OTEL_AI_PROJECT_NAME=conversations              # -> openinference.project.name
OTEL_AI_TENANT_PROJECT_TEMPLATE=conversations-{tenant_id}   # optional per-tenant project split

Full table: env-vars reference.

ai-orchestrator and async-agent batch trace export by default (OTEL_BATCH_EXPORT=true) so ending spans never block realtime model/audio handlers or HTTP request responses. Use synchronous export only for targeted debugging.

In Vanguard, otel-collector is not run by default. Point OTEL_AI_EXPORTER_OTLP_ENDPOINT at the local Phoenix container (http://localhost:4317, grpc) for AI traces; infra traces need a local Tempo/Jaeger or are silently dropped.

Removed legacy variables

OTEL_BACKEND, PHOENIX_COLLECTOR_ENDPOINT, PHOENIX_PROJECT_NAME, PHOENIX_PROTOCOL, ARIZE_PHOENIX_PROJECT, and LANGFUSE_HOST/_PUBLIC_KEY/ _SECRET_KEY are removed and ignored — startup logs an ERROR naming the replacement when one is set. Migrate with the recipes below.

For KMS dev and QA, configure the ai-orchestrator and async-agent project names with OTEL_AI_PROJECT_NAME in values-kms-dev.yaml and values-kms-qa.yaml. These values are supplied by Helm, not Secrets Manager. Keep the existing OTLP endpoint and protocol; remove the retired variables. Promote the change through the deployment branches and let ArgoCD sync it so new pods receive the updated environment.

Destination recipes

Arize Phoenix (self-hosted container or cloud):

OTEL_AI_EXPORTER_OTLP_ENDPOINT=http://phoenix:4317
OTEL_AI_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_AI_PROJECT_NAME=conversations

LangFuse (cloud or self-hosted; Basic auth is base64 of publicKey:secretKey):

OTEL_AI_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel
OTEL_AI_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_AI_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(pk:sk)>

One collector, two backends: send everything to a collector on the infra vars and route there on attributes["telemetry.stream"] == "ai" — reference config in otel-collector/values.yaml.

Tenant-level telemetry

Tenant admins can route their own AI telemetry from ai-manager → Settings → Tenant Settings → Telemetry (the Tenant Settings group is visible only to tenant admins and platform admins). This runs in addition to the system-level env config: AI spans carrying tenant.id are additionally exported to the tenant's destination.

• The destination is configured inline on the Telemetry tab — endpoint, protocol, optional project name, and a list of headers. A tenant needs only one AI destination, so there is no integration row; the whole config lives in the tenant_settings key telemetry ({ai: {enabled, endpoint, protocol, project_name, headers[]}}), saved by apps/ai-manager/lib/telemetry-settings.ts via PUT /api/settings/telemetry.
• Each header can be flagged secret. Secret header values are encrypted at rest with AES-256-GCM under MASTER_ENCRYPTION_KEY (same envelope as integration_secrets, so the Python runtimes decrypt them with their existing helpers), stored as an enc:{ciphertext,iv,authTag} object inside the setting JSON. Secrets go out to the client masked; saving a masked value keeps the stored secret unchanged.
• Runtimes (ai-orchestrator, async-agent) refresh the config on a ~60s TTL (TENANT_TELEMETRY_CACHE_TTL), so changes take effect without a restart. Disable the whole mechanism with TENANT_TELEMETRY_ENABLED=false.
• A mid-session config change splits that session's spans across the old and new destinations — accepted consequence of immediate effect.
• On the system AI destination, OTEL_AI_TENANT_PROJECT_TEMPLATE (e.g. conversations-{tenant_id}) rewrites openinference.project.name per span so one destination can segregate tenants into per-tenant projects.

Runtime pieces: libs/telemetry/tenant_routing.py (TTL-cached per-tenant exporters + trace-id→tenant inheritance for spans that don't carry tenant.id themselves) and libs/telemetry/tenant_store.py (reads the telemetry setting + decrypts secret headers inline), attached at service startup via enable_tenant_telemetry().

PII scrubbing at export

Every exported span — system infra/AI sinks and every tenant destination — is scrubbed by libs/telemetry/scrub.py before it leaves the process. Content attributes (input.value, output.value, transcript.*, llm.input_messages.*, llm.output_messages.*, gen_ai.prompt*/gen_ai.completion*) are run through the shared trueserve_pii tier-1 rules detector (checksummed cards/IBANs, SSNs, contextual identifiers) and redacted in a span proxy — non-content attributes and clean spans pass untouched. On by default; TELEMETRY_PII_SCRUB=off disables it. A per-span detector fault withholds the content attribute rather than exporting it raw. Design context: data protection, layer L3.

AI span coverage

Every AI span carries tenant.id, session.id (voice/chat) or workflow.run_id (workflows), and telemetry.stream=ai:

• Session/turn spans — ConversationTracer (linear + custom S2S pipelines) and RealtimeEventTracer (OpenAI Realtime) root a session span per conversation with per-turn child spans; tool calls are mirrored onto the current turn span by ConversationAPIPublisher.
• Per-service usage spans — every registry-built service (Deepgram, ElevenLabs, GECX/CES, OpenAI, Google, AWS, Moshi, Agentforce, MavenAGI, Palabra, …) gets a session-anchored Service Usage Summary span from service_usage.py (audio ms, characters, tokens). Custom services report provider-side usage (latencies, provider conversation/session ids) via report_service_usage().
• Workflow spans — async-agent roots a Workflow Run span with per-node child spans; LLM calls, OCR, PII detection, and transcription provider calls each get their own AI span.
• ai-manager spans — judge/suggest, scale-testing, and TTS-preview LLM calls via lib/otel-usage.ts.

Metrics

Each app exposes Prometheus metrics at /metrics on its main HTTP port (or a sidecar port for gRPC services). The collector scrapes them and forwards to the cluster's Prometheus / Cloud Monitoring / CloudWatch.

Logs

In Vanguard, logs go to overmind connect <service> for host processes, docker logs tsf-local-<container> for compose services. There is no log aggregation locally.

In cluster, logs go to whatever the cluster's log shipper does — Cloud Logging on GKE, CloudWatch on EKS, Azure Monitor on AKS. The application code does not configure shippers; it just writes to stdout. OTLP log export (infra stream only) is opt-in via OTEL_LOGS_EXPORTER=otlp.

Official Dev/QA: Loki through Grafana

KMS dev exposes Grafana at https://grafana.kms.dev.trueserve.deloitte.com. Its log datasource UID is loki; loki-tempo accesses the same logging backend with trace-link configuration. Prometheus and Tempo are separate datasource types. The Grafana datasource proxy provides read access to Loki without ArgoCD CLI SSO: /api/datasources/proxy/uid/loki/loki/api/v1/.

Use the city tunnel from a laptop to reach the target Grafana; a shell on the city host already uses that network path. The repository's grafana-read.py is GET-only. It reads an authorized environment-specific service-account token from ~/.local/state/trueserve/grafana-kms-dev.token or grafana-kms-qa.token (mode 0600), refuses redirects, and redacts common credential formats. Tokens remain machine-local; the relay installer does not distribute them. See official-environment debugging for the Dev/QA endpoint mapping and source-branch rules.

python3 scripts/studio-access/grafana-read.py --environment dev /api/datasources

python3 scripts/studio-access/grafana-read.py --environment dev \
  /api/datasources/proxy/uid/loki/loki/api/v1/query_range \
  --param 'query={namespace="trueserve",app="webex-app-manager"} |= "Background token refresh failed"' \
  --param since=1h --param limit=50 --param direction=backward

For QA use --environment qa and discover the datasource UID before querying. Do not use local city Overmind/Docker logs as evidence about official Dev/QA.

Useful stream labels are namespace, app, container, and pod; most Python and Node services use the generic container name app, so group by the app label to identify services. Discover labels with /loki/api/v1/labels, then bound queries by namespace, service, time range, and result limit.

Counts of lines containing “error” are not counts of failed requests or crashes. Inspect nearby lines and deduplicate multiline exceptions. TURN authentication challenges can precede successful allocation, and socket closures or media EOF can accompany a normal hangup. The periodic Opus “no data supplied” message has previously been identified as DTX/silence behavior in the KMS voice investigation.

Status page

apps/status-page is a Python service that renders deployment health. It runs in two modes:

• Kubernetes mode — reads the in-cluster Kubernetes API (kubectl get pods --all-namespaces-equivalent), surfaces pod counts, image versions, and rollout state.
• Docker mode — reads docker ps and overmind status, surfaces Compose container state and Overmind process state. Used in Vanguard.

Mode is auto-detected at startup via apps/status-page/docker_mode.py. Override with STATUS_PAGE_MODE=docker|kubernetes.

Visit it at http://localhost:8090 in Vanguard, or whatever ingress hostname your cluster maps to it.

Sentry

Frontend apps (ai-manager, surface-app, crm-dashboard) report errors to Sentry. Configured per-app via NEXT_PUBLIC_SENTRY_DSN. In Vanguard, leave the DSN unset to skip reporting.

See also

• Env-vars reference (telemetry table)
• Secret storage (inline secret headers)
• Networking
• Status page service
• Troubleshooting deploys