media-proxy¶

Status: Current | Last reviewed: 2026-08-30

The Go service that sits between telephony transports and ai-orchestrator. It terminates RTP/SRTP/WebRTC, converts codecs, mixes streams, and ducks audio.

Source¶

apps/media-proxy

Runtime¶

Aspect	Value
Language	Go
Transport	RTP, SRTP, WebRTC, gRPC bridge to ai-orchestrator
Codecs	Opus, PCMU/PCMA, G.722

What it does¶

Accepts inbound RTP from kamailio/asterisk or WebRTC from janus.
Decodes to PCM, optionally resamples.
Streams to ai-orchestrator via gRPC.
Receives TTS audio back; encodes; sends to the transport.
Records transport-owned audio when orchestrator asks it to, including translation RTP/SIP sessions, as stereo conversation audio: caller on the left channel, agent audio on the right channel. Normal non-translation calls use orchestrator-side recording instead so generated agent audio is captured before transport encoding.
Mixes hold music, on-hold messages, translation feeds, advisor sidetone.
Records direct WebRTC caller/agent PCM for Activity Log playback when the transport path owns audio capture; ai-orchestrator can disable that recorder per session through gRPC metadata when a pipeline-local recorder is already attached.

Mixing¶

internal/audio/mix.go implements three primary modes:

Function	Use
`MixUnderbed`	Talk track full volume; bed at 25% gain underneath
`MixTranslationStyle`	Original speaker ducked to ~15% while translation plays at full
`MixWithBackground`	Constant low-volume background (hold, ambient)

Math: per-sample linear gain with 50ms log-curve fade-ins at transitions. See audio-ducking.md for details.

The consult mix bus¶

internal/audio/mixbus.go is the second, separate mixer: the one that lets a browser caller take part in an External Consult. It is pure data — no I/O, no session types — so the browser session and the RTP B-leg read the same table and cannot disagree about what "merged" means.

ConsultMatrix is eight gain cells, consultMatrices maps each phase to one, written out literally so a wrong cell is a visible wrong number:

Phase	The caller hears	The specialist hears	The origin agent hears
`off`	the agent	— (no consult)	the caller
`hold`	the comfort tone	the dialer agent (the brief)	nothing — hold-time speech must not land as a caller turn
`hold_listening`	the agent	the dialer agent	the caller
`merged`	the specialist	the caller	nothing — both agents are out
`merged_with_agent`	the specialist + the agent	the caller	the caller + the specialist
`merged_with_both`	the specialist + the agent	the caller + the dialer agent	the caller + the specialist

MatrixFor falls back to Off for an unknown phase because every other default silences a live call; IsKnownConsultPhase is what the HTTP layer uses to reject a controller typo as a 400 rather than mixing something wrong; IsConsultActive keeps off/hold_listening on the single-source fast path, so an ordinary call pays nothing for any of this.

The dialer agent's leg is retired by gain, not by hangup. On SIP, mergeCalls hangs that channel up. Here it is the only wire the specialist's audio arrives on — the caller has no Asterisk channel to bridge the specialist into — so the leg stays up for the whole call and is silenced by dropping ConsultFromDialer/DialerFromConsult. That is what merged_with_both is for.

Hold audio is generated, not shipped (internal/audio/hold.go): 425 Hz (ITU-T E.180 — 440 Hz reads as a test tone), 1 s on / 3 s off, 25 ms fades, ~6% amplitude, optionally over the agent's ambience bed. Dead air reads as a dropped call; a music asset means licensing.

The link itself is owned by the Manager (internal/proxy/consult_link.go), because the caller lives in webrtcSessions and the B-leg in the RTP port table and neither map can find the other. LinkConsultToWebRTC installs both sinks before any phase is set — install one first and the missing direction is one-way audio that reads as a codec problem. Merged prefill is 40 ms rather than the usual 200: buffering between two humans is permanent one-way delay.

Three requireAPIKey routes, all quoting only the caller's session id:

POST /v1/webrtc/consult/link    {"caller_session_id":"…","consult_session_id":"…"}
POST /v1/webrtc/consult/phase   {"caller_session_id":"…","phase":"hold"}
POST /v1/webrtc/consult/unlink  {"caller_session_id":"…"}

ari-controller drives them from MediaProxyConsultProvider. Full flow: warm-handoff.md.

Dependencies¶

Service	Purpose
ai-orchestrator	gRPC audio stream sink/source
kamailio + rtpengine	Inbound RTP path
janus	Inbound WebRTC path

Vanguard¶

overmind restart media-proxy

Default port :18080 (gRPC) plus dynamically allocated RTP ports.

Production¶

Helm chart at charts/media-proxy. Runs as a Deployment with HPA on concurrent sessions. Not host-networked — it talks to rtpengine which holds the public socket.

WebRTC startup optimizations (WARP)¶

Direct WebRTC (browser ↔ media-proxy) is the only WebRTC leg where TrueServe owns both endpoints, so it is the only one that can adopt WARP — the WebRTC Abridged Roundtrip Protocol work OpenAI is standardizing through IETF TSVWG to collapse handshake setup from ~6 round trips to 1. Every other leg (janus SIP, LiveKit avatar, Genesys/Amazon Connect edges) peers with a third-party stack, and telephony isn't WebRTC at all.

Two of WARP's four parts are live here:

Part	Status	Where
SNAP — pre-negotiated SCTP handshake via the `a=sctp-init` SDP attribute (saves 2 RTTs)	On, but inert until the browser enables it (see below)	`SettingEngine.EnableSctpSnap(true)` in `internal/webrtc/peer.go` `NewPeer`
Pre-negotiated data channel — skip the in-band DCEP `OPEN`/`ACK` exchange	On, opt-in per call	`Peer.EnableNegotiatedVideoChannel` (peer.go) ← `SessionConfig.NegotiatedVideoChannel` ← `negotiated_video_channel` on `POST /offer`
SPED — DTLS handshake piggybacked inside STUN/ICE	Blocked upstream	pion/webrtc #3362, pion/dtls #766
DTLS 1.3	Blocked upstream	pion/dtls #727 (MVP tracking)

Pion's WARP umbrella tracking issue is pion/webrtc#3335. SNAP requires pion/webrtc ≥ v4.2.18 (PR #3327 plus the #3443 renegotiation fix) — the pin in go.mod is load-bearing, and downgrading silently drops back to the classic 2-RTT SCTP handshake.

Both parts are backward compatible, but in different ways, and the difference matters:

SNAP is symmetric and automatic. Pion emits a=sctp-init on the answer only when the offer already carried one, so a browser without SNAP support negotiates the classic handshake with no code path of ours involved. Safe to leave on unconditionally.

But it's browser-gated today, so measure before claiming the win. Chrome 149 ships SNAP behind the WebRTC-Sctp-Snap field trial, which is off by default — a stock Chrome offer carries no a=sctp-init, our answer correctly carries none either, and the session silently uses the classic SCTP handshake. Verified against the live stack: stock Chrome 149 → chromeOffersSnap:false, answerSnap:false; the same Chrome launched with --force-fieldtrials="WebRTC-Sctp-Snap/Enabled" → chromeOffersSnap:true, answerSnap:true. So the server half is done and correct, and TrueServe picks the win up automatically as Chrome enables the trial — but don't attribute a startup improvement to SNAP without checking a=sctp-init is actually in both SDPs (chrome://webrtc-internals, or the WebRTC.PeerConnection.NegotiatedSctpSnap histogram). The pre-negotiated data channel below, by contrast, works on stock Chrome right now. - The pre-negotiated channel is not automatic — it's opt-in per call. A pre-negotiated channel carries no in-band handshake, so both ends must independently agree on the SCTP stream id, and neither end learns the other got it wrong. The browser sets negotiated_video_channel: true on the offer and media-proxy pre-creates its side only then; the OnDataChannel (DCEP) handler stays untouched for clients that haven't opted in — cached SPA bundles, embedded snippets on customer pages. Making it opt-in also means an un-rebuilt client can never end up with a dangling channel media-proxy is not reading.

Why id 0 is safe to hard-code, and the assumption behind it. RFC 8832 splits the SCTP stream id space by DTLS role: the DTLS client uses even ids, the DTLS server odd ones. media-proxy answers a=setup:active, so media-proxy is the DTLS client (even ids → our pre-negotiated channel takes 0) and the browser is the DTLS server (odd ids → its DCEP channels start at 1, which the live logs confirm: data channel opened (DCEP) label=video id=1). That parity split is what keeps id 0 collision-free — not the opt-in flag. If the answering role ever flips to passive (SettingEngine.SetAnsweringDTLSRole), the browser becomes the DTLS client, its first DCEP channel lands on id 0, and it would collide; TestWARPNegotiatedIDCannotCollideWithDCEPClient asserts the role and the odd-id parity so that change fails loudly instead of silently corrupting frames.

The browser half is contexts/auth-context.tsx: NEGOTIATED_VIDEO_CHANNEL_ID must equal negotiatedVideoChannelID in peer.go. The audio-only callers (hooks/use-webrtc-call.ts, shared/phone-emulator) create no data channel and send no flag, so they keep the DCEP-free audio path unchanged. Note that creating a data channel on the answerer does not inject a spurious m=application section into an audio-only answer — pion's answer path only matches sections present in the remote offer — which peer_warp_test.go locks in alongside both channel paths and both SNAP directions.

Independent of WARP, the offer/answer exchange no longer blocks long on ICE gathering: the wait is 150 ms on both ends (webrtcGatherWait in peer.go, ICE_GATHERING_WAIT_MS in the three browser call sites). The SDP already carries the ICE credentials and remaining candidates trickle through the ICE agent, so a longer wait only buys a fuller candidate list at the cost of call-startup latency. In practice this is the largest of the three wins here: measured on the vanguard stack, Chrome's gathering finished at ~114–118 ms (inside the new cap, previously it would have kept waiting toward 500 ms) and a Pion client gathered in 5–11 ms.

Measured against the live stack (agent 3017, real Chrome 149 and a Pion client, agent audio confirmed flowing in every run):

Client	SNAP offered → answered	`video` channel	Channel open	Agent audio
Chrome 149, stock	no → no	pre-negotiated id 0	+216 ms, same ms as `connected`	33 KB inbound RTP
Chrome 149, `WebRTC-Sctp-Snap/Enabled`	yes → yes	pre-negotiated id 0	+172 ms, same ms as `connected`	30 KB inbound RTP
Pion, SNAP + pre-negotiated	yes → yes	pre-negotiated id 0, `OnDataChannel` never fired	+26 ms, same ms as `connected`	398 packets
Pion, legacy (no SNAP, DCEP, no opt-in flag)	no → no	DCEP, id 1	+13 ms	410 packets
Pion, audio-only (no data channel)	no → no	none; answer has no `m=application`	—	416 packets

The load-bearing observation is that the data channel opens in the same millisecond the peer connection reaches connected — the DCEP OPEN/ACK exchange is gone rather than merely faster.

Session teardown (graceful disconnect)¶

When the agent ends a session — e.g. an end_session / transfer-to-human tool — ai-orchestrator sends a Control.TYPE_DISCONNECT over the gRPC stream. media-proxy must drain any queued TTS, then close the transport so the final utterance ("connecting you to a specialist…") plays out before the connection drops. Both transport paths now do this:

Path	Handler	On `TYPE_DISCONNECT`
RTP/SIP	`internal/proxy/session.go`	`drainPendingTTSBeforeClose(8s)` → notify ARI → `close()` (sends SIP BYE)
Direct WebRTC	`internal/webrtc/session.go`	`drainOpusBuf(8s)` → `Close()` → `pc.Close()`

The WebRTC path closing pc.Close() is what flips the browser's RTCPeerConnection to closed, which the AI Manager test-call UI (contexts/auth-context.tsx onconnectionstatechange) listens for to end the call and show "Call ended". If media-proxy only returns from its gRPC receive loop without closing the peer, the browser never sees a state change and the test-call UI stays stuck on "connected" forever — this was the symptom of the 2026-05-29 bug where end_session left the UI connected.

Mid-call migration suppression window (`TYPE_SESSION_MIGRATING`)¶

A tier-C mid-call migration (see runtime-profiles) tears the orchestrator's pipecat chain down behind a live transport, and the dying chain's own teardown emits the same TYPE_DISCONNECT a real session end does. Control.TYPE_SESSION_MIGRATING (value {"window_seconds": N}, default 10 s, capped 30 s by media-proxy) arms a bounded suppression window on the RTP/SIP session (internal/proxy/migration.go):

a TYPE_DISCONNECT arriving inside the window is held, not acted on — and if the window expires without TYPE_SESSION_ACTIVE, the held teardown (params included) is resumed exactly as it would have run, so a failed migration never strands a leg;
TYPE_SESSION_ACTIVE disarms the window and discards a held disconnect — the migrated chain owns the same stream;
the idle reaper reads a migrating session as not-idle for the window (session.isIdle), so the quiet rebuild interval can't be reaped; the window's 30 s cap keeps this strictly a delay, never a bypass;
a silent gRPC stream end during the window needs no special handling here: stream end has never been an immediate teardown on this path (the reaper is), and the reaper suppression above covers the window.

ari-controller is untouched: it hangs a leg up only when media-proxy's session-end callback fires, and its channel reconciler keys on Redis-session presence, which a migrating session keeps. Regression coverage: internal/proxy/migration_test.go.

The drain duration is a real teardown delay the caller experiences. The WebRTC path logs how long it took (TTS drained in Nms) and warns if it hits the 8s cap (farewell clipped). It self-terminates as soon as the queue empties, so a long delay means a lot of TTS was queued — not a fixed wait. ai-orchestrator's end_session_tool_handler sends the EndFrame immediately after the bot stops speaking (no artificial pre-sleep), since audio frames are already ordered ahead of it and the transports drain before closing. Any teardown lag beyond the logged drain time is browser-side jitter buffer.

Keypad digits into a direct WebRTC session¶

A SIP call's digits arrive as RFC 4733 / SIP INFO and reach the gRPC stream through ari-controller's POST /session/{sessionId}/dtmf-in, which resolves the session with FindSessionByID — a walk of the RTP port table. Direct WebRTC sessions are held in their own map, so that lookup never sees one and dtmf-in 404s for every browser test call. There is also no in-band alternative: a direct WebRTC leg has no SIP dialog at all.

POST /v1/webrtc/dtmf ({session_id, digit, duration_ms}, API-key + webrtcCORS, 204 on success, 404 for an unknown session) is that path. Manager.HandleWebRTCDTMF looks the id up in webrtcSessions and calls Session.SendDTMF, which sends a ClientMessage_Dtmf carrying a DtmfEvent — the same message the SIP path produces, so ai-orchestrator's PaymentModeController sees an identical InputDTMFFrame either way. duration_ms defaults to 250. Digit-charset validation lives in the browser-facing ai-manager route, not here; this handler checks presence only, matching the dtmf-in contract.

This is why Session has a sendMu. grpc-go permits only one concurrent Send on a stream, and the WebRTC session now has three independent goroutines that can send: the audio pump, the data-channel callback, and an HTTP-driven DTMF injection. Before DTMF there were two and they were effectively serialized by the audio loop; a keypress arriving mid-frame made concurrent sends real. Every Send on that stream goes through Session.sendToGRPC, which holds sendMu — a Send that bypasses it corrupts the stream for the whole session, not just the racing message. TestSessionSendToGRPCSerializesConcurrentSends asserts that no two sends are ever in flight at once.

Verifying the path without a browser. A dev container has no browser, so use the dtmfprobe command in the media-proxy module: it opens a real (silent) pion WebRTC session through /v1/webrtc/offer and POSTs each digit to /v1/webrtc/dtmf, exactly as a keypad does.

export MEDIA_PROXY_API_KEY=...   # from the running process's env
go run ./cmd/dtmfprobe -agent 4215 -digits '159#'

Each digit answering 204 only proves media-proxy accepted it; confirm the far side with [GRPCClient] _pump_inbound received DTMF digit=… followed by Pushing InputDTMFFrame button=… in the ai-orchestrator log.

Which surface uses which transport, and how each of them managed to fail silently, is in Payments (PCI).

Global Accelerator dual-interface caveat¶

In EC2 + GA setups, rtpengine (not media-proxy) needs both LAN_IP and PUBLIC_IP interfaces because GA DNATs PUBLIC_IP → LAN_IP before delivery. media-proxy itself only sees the LAN-side socket. See rtpengine.md.