Troubleshooting voice¶
Status: Current | Last reviewed: 2026-10-07
Fault tree for voice failures.
Symptom: ElevenLabs HTTP agent says it will transfer but the call stays open¶
Check runtime logs after the accepted-handoff action. On October 7, agent
4463:v33 invoked end-session through its authored rule and emitted EndFrame,
but the ElevenLabs HTTP stage raised:
ElevenLabsHTTPTTSService._stop() takes 1 positional argument but 2 were given
Its private HTTP cleanup helper shadowed Pipecat's _stop(frame) dispatcher.
Renaming it to _close_http_client restores the inherited dispatch;
stop(frame) drains queued audio and then closes HTTP resources. Cancellation
retains the same cleanup. Start a new call after loading the runtime fix.
A recorded handoff receipt or CALL_ENDED event upstream of TTS does not prove
the phone connection received the end signal. Verify the downstream gRPC
TYPE_DISCONNECT and its session_escalated parameters, then the connector's
return to the contact-center flow. A rule-driven tool invocation may also lack
a model tool-call transcript row, so inspect the rule and handler logs before
concluding that the model failed to call end-session.
Symptom: Copilot test is silent and its sidebar reports an integrity mismatch¶
Check both the simulated voice agent and the catalog: these are independent
failures. On Vanguard on 2026-09-18, agent 3018:draft used voice agent
4108:draft. The voice receive loop dropped on its first completed response
because _text_mode_reprompt was initialized only in cancel, rather than in
the Realtime service constructor. Separately, replacement context aggregators
lost Pipecat's assistant-to-user reference, raising “no paired user aggregator”
when realtime mode activated. The shared context-pair factory now wires the
active forwarding user aggregator into the assistant for both S2S and cascaded
pipelines. Restart ai-orchestrator after applying these Python fixes.
The sidebar's copilot-cards/1.0.0/bundle.css object had changed while the
database retained its original SHA-384. The saved local build matched that
original hash exactly. Recovery uploaded those verified bytes under an
integrity-qualified key and updated only matching local version rows, retaining
their hashes and leaving the shared object untouched. Never accept the new
download's hash merely to suppress an integrity error. New catalog publications
include a publication UUID in object paths to separate same-version builds from
different city databases; older paths remain valid. CSS URLs and hashes also
participate in the manifest ETag so a repair invalidates cached manifests.
Reload the test page and start a new session after recovery.
Symptom: Copilot transcripts repeat or later speech appears at the top¶
On 2026-09-18, normalized STT finals reused one message key per
speaker for the entire call, overwriting prior utterances while retaining their
first audible timestamp. The test stream also merged -standin transcripts,
duplicating speech and exposing voice-style annotations. Copilot normalizers now
assign per-utterance identities and timestamps, enhancement preserves them, and
the combined stream excludes partner transcript events. These changes correct
new utterances; already-overwritten historical message rows require recovery
from retained event evidence, rather than a UI sort change.
Symptom: Copilot keeps transcribing after hangup or emits no cards¶
Compare the voice and -assist session end times. A copilot sidecar has no
Asterisk channel; ARI must save its allocation and release it during teardown.
On 2026-09-18, the missing release left agent 3018's sidecar alive for about
43 seconds after its voice session ended.
For missing cards, distinguish gate decisions from worker errors. The same incident had OpenAI 400 errors for non-default temperature in the gate, coaching, and Suggested Answers finalizer. The gate failed open, so it was not filtering those turns out. The shared request helper now omits temperature for reasoning models. A knowledge response saying the connection does not own a dataset is a separate dataset/connection selection issue; do not bypass its access check. An invalid Bedrock enhancement model is also separate from raw STT delivery.
Symptom: SIP INVITE returns 404¶
| Cause | How to verify | Fix |
|---|---|---|
Trunk ACL doesn't include shared NAT egress (54.92.142.217) |
Genesys / vendor portal trunk page | Add the IP, save |
| Per-city Kamailio cert mismatch | Check CN against vendor's expected cert | Re-issue + restart kamailio |
| Wrong dialplan extension | tail -f /var/log/asterisk/full while dialing |
Add the extension to extensions.conf |
Symptom: Call answers, then hangs up immediately with Q.850 cause=16¶
This is almost always func_odbc.conf querying the dropped agent_configs view.
docker exec -it tsf-local-asterisk grep -n agent_configs /etc/asterisk/func_odbc.conf
If you see hits, replace with a query against agent_versions. See data-model.md.
Symptom: Call connects but no audio¶
Most common on AWS + Global Accelerator: rtpengine is single-interface and GA's DNAT delivers packets to LAN_IP while rtpengine only listens on PUBLIC_IP.
docker exec -it tsf-local-rtpengine cat /etc/rtpengine/rtpengine.conf | grep interface
Should look like:
interface = pub/${PUBLIC_IP}!${PUBLIC_IP};priv/${LAN_IP}
See rtpengine.md.
Symptom: Every agent goes silent and the orchestrator log says could not translate host name "localhost"¶
Take the DNS error at face value and you'll chase a networking ghost. The real
cause is file-descriptor exhaustion in the ai-orchestrator process: at the
RLIMIT_NOFILE cap every new socket fails, including the UDP socket the
resolver needs, so a plain Postgres connect surfaces as a name-resolution
failure. No DB → db_pipeline_store returns None → CRITICAL ERROR: Pipeline
ID <id>:draft not found in database → no pipeline is built and the agent never
speaks. gRPC signaling and the SDP answer still succeed, so it looks like a
media bug.
Confirm it — a count pinned at (or just under) the soft limit is the tell:
P=$(ss -lntp | grep :50051 | grep -oE 'pid=[0-9]+' | head -1 | cut -d= -f2)
ls /proc/$P/fd | wc -l # e.g. 1024
grep 'Max open files' /proc/$P/limits # soft / hard
ls -l /proc/$P/fd | grep -c socket # sockets dominate when leaking
ss -tanp state all | grep "pid=$P" | awk '{print $5}' | grep -oE ':[0-9]+$' \
| sort | uniq -c | sort -rn | head # :443 = leaked provider sockets
Hundreds of ESTAB sockets to :443 that outlive their calls mean a session
failed to tear down — a long-running Nova Sonic session continuation loop
retrying a dead AWS stream is one known source (Session continuation: forced
transition failed: ... AWS_ERROR_SYS_CALL_FAILURE repeating for hours/days).
Fix now: overmind restart ai-orchestrator, then re-check with
POST /preflight {agent_id, allow_draft} — {"ok": true} means it will build.
Prevention: start-ai-orchestrator.sh
raises the soft limit to 65536 (AI_ORCHESTRATOR_NOFILE overrides). Overmind
children otherwise inherit a 1024 soft limit against a 524288 hard limit,
which this service can exhaust legitimately — each live session holds provider
sockets plus an epoll+eventfd pair per event loop. Raising the ceiling buys
headroom; it does not fix a leak, so a process climbing back toward 65536 is a
real socket leak worth tracing.
Symptom: GPT Live hears the caller but restarts its greeting¶
Check for caller TranscriptionFrame messages and non-silent gRPC input before
assuming the microphone is disconnected. Check the media-proxy log for
TYPE_INTERRUPTION and flushed audio frames to distinguish a transport
interruption from the model changing its speech.
In the agent 4343 incident (2026-09-11), the caller's "Can you hear me" reached
GPT Live, but InstructionRefresher repeatedly rebuilt a roughly 26,000-character
business prompt after assistant turns. The adapter appended that entire prompt
to the voice engine, including its opening instructions. The provider then
repeated the greeting without a media-proxy interruption.
The adapter now skips unchanged instruction refreshes and sends changed
business rules only to the delegation backend. Keep voice instructions
separate: session.instructions.append is an explicit steering command that
can interrupt speech, not a replacement for updating the backend prompt.
After updating the adapter, start a new Preview call so its voice context does
not retain the previously appended scripts.
See OpenAI GPT Live for the protocol and regression tests.
Symptom: GPT Live ends mid-sentence after a tool failure¶
Check the final end_session call and its reason before diagnosing a process
crash. Agent 4343's 2026-09-11 13:08–13:10 UTC Preview ended with
human_assistance_needed: appointment management had no FHIR base URL or OAuth
integration. Its expanded URL still contained a capability-config placeholder.
The tool-input classifier incorrectly turned the URL error into a missing
Request field, spending about 21 seconds waiting for unrelated caller input.
The private TTS lane also inherited the Live-only gleam voice and produced no
audio. A transport stop event then ended the call during “I hit a snag.” The
fixes are separate: configure the capability connection; reject unconfigured
URLs before execution; infer repairable fields only from the tool schema; map
unsupported private voices to marin; and have Live hangup wait for its native
transcript turn to finish. A private voice override remains available through
the auxiliary-service configuration.
Regression coverage lives in test_openai_live_service.py,
test_end_session_tool_handler.py, test_aux_resolution.py,
test_tool_input_healing.py, and test_saved_api_tools.py. A successful
preflight alone does not prove that an external capability endpoint responds;
use a read-only lookup to verify the configured connection.
Symptom: Test call auto-ends with SIP 488 Not Acceptable Here¶
If Asterisk logs couldn't allocate a port for RTP instance, its RTP socket
range is full. In local dev this can happen when Asterisk retains RTP sockets
after failed/noisy calls even though core show channels reports 0 active
calls.
bash .devcontainer/scripts/asterisk-rtp-watchdog.sh --once
bash .devcontainer/scripts/asterisk-rtp-watchdog.sh --once --prune
The local Overmind stack runs asterisk-rtp-watchdog continuously. It restarts
Asterisk only when RTP usage crosses the high-water mark and there are no active
calls, then restarts ari-controller so ARI reconnects. If you change
configs/asterisk/rtp.conf, re-sync/restart Asterisk or run wake-up.sh.
Also check that unknown initial SIP INVITEs are being stopped by Kamailio
instead of reaching Asterisk. The expected log line is:
docker logs tsf-local-kamailio --since 10m | grep 'Rejecting unknown initial INVITE'
If a new provider or test route is legitimate, add a specific Kamailio matcher
for its headers/domain/user pattern. Do not restore a broad fallback that relays
all unmatched INVITEs to the dispatcher.
Symptom: "Asterisk ARI not ready after 60s"¶
ari-controller raced Asterisk boot. Once Asterisk is up at :8087:
overmind restart ari-controller
Note: :8087 is ARI. :8088 is Janus admin — they're easy to confuse.
Symptom: Chrome 502 on SIP registration in browser¶
Nginx /ws location dropped the WebSocket upgrade headers when stripping cookies. Re-declare them:
location /ws {
proxy_set_header Cookie "";
proxy_set_header User-Agent "";
proxy_set_header Referer "";
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://janus:8188;
}
Symptom: Outbound transfer to BYOC vendor fails¶
Genesys-specific: 404 after RFC3261 REFER indicates the SIP Route header was not preserved on the ACK for the original call. Kamailio config must keep the route set intact. See kamailio.md.