Real-time translation¶
Status: Current | Last reviewed: 2026-10-14
How a bilingual call works end to end.
What is "real-time translation" here¶
Two parties speak different languages. Each hears the other in their own language, with TTS playing over a ducked version of the original audio. Latency target: under 1.5 seconds end-to-end.
The pipeline subtype is translation. It is built by apps/ai-orchestrator/lib/pipecat/pipeline/builders/translation_builder.py.
Pipeline shape¶
Caller A audio (lang A)
→ STT (lang A)
→ translate (LLM or dedicated translator)
→ TTS (lang B)
→ mixed with caller A's original audio at reduced volume (underbed)
→ played to caller B
The reverse path runs concurrently for caller B → caller A. Two STTs, two translation calls, two TTSs, both mixed into the opposite caller's outbound audio.
Amazon Voice Translation¶
Select AWS → Voice Translation in a Speech-to-Speech Translation stage. Both directions use streaming Transcribe → Translate → Polly, with a choice of PCM response streaming (Phase 1) or bidirectional Polly streaming (Phase 2). The stage sends PCM as it arrives and avoids the standalone TTS prefill buffer. Use the AWS runtime and comparison guide for configuration, credentials, measured latency and SCV diagnostics. The latency target above is a target; the synthetic AWS smoke runs did not meet it.
Krisp Voice Translation¶
Agent Studio registers Krisp as provider=krisp,
type=s2s_translation, service=Voice Translation. Each direction is a
separate Krisp session, matching the existing two-flow translation runtime.
In each incoming or outgoing flow, add a Speech-to-Speech Translation stage,
then select Krisp → Voice Translation.
Krisp Voice Translation has no Behavior tab; selecting the stage opens its
Properties directly. Properties renders the full definition in five groups:
Languages, Voice, Vocabulary and Glossary, Audio Processing, and Output Control.
The stage supports dynamic BCP-47 source/target languages, male/female output,
vocabulary terms, source→target dictionary pairs, direction-local cloud BVC,
and independent audio/source-transcript/translated-text outputs.
The server resolves the selected encrypted Krisp integration key (or
KRISP_VT_API_KEY), mints a short-lived session token, converts input to mono
PCM16 at 16 kHz, and sends exact 20 ms / 640-byte frames. Krisp SDK callbacks
run on a vendor thread; the adapter transfers them to Pipecat's event loop
before emitting frames. Startup retries once for token/transient failures.
A runtime failure ends translation so SCV restores its stable enhanced audio
paths rather than leaving an inactive synthesized track installed.
On SCV, incoming Krisp VT consumes the local NC-only customer branch. Inbound Accent Conversion changes only what the agent hears. Krisp VT cloud BVC can add cleanup inside one translation direction, but it does not alter the customer's original underbed audio.
Key files¶
| File | What it does |
|---|---|
translation_builder.py |
Constructs the bidirectional pipeline |
translation_test_builder.py |
Offline harness for testing translation pairs |
translation_normalizer.py |
Turns each flow's two raw text channels into normalized frames the publisher handles. Inserted last in the flow, one per flow (not the per-stage registry lookup), and owns fragment spacing via text_granularity |
translation_frames.py |
TranslationTranscriptFrame — the accumulated transcript snapshot, a DataFrame so it can't be mistaken for pipeline text |
apps/media-proxy/internal/audio/mix.go |
MixTranslationStyle — the audio mixing function |
apps/ai-manager/lib/pipeline-types.ts |
Defines the TRANSLATION subtype + TranslationFlowConfig / flows.{incoming,outgoing} |
apps/connect-scv-translator/src/translation/orchestrator-grpc/ |
Browser-side per-direction consumer (Salesforce SCV Live Translation) |
How direction works¶
config.translation.flows holds exactly two named flows — incoming (caller → agent, what the agent hears) and outgoing (agent → caller, what the caller hears). Each is a TranslationFlowConfig carrying its own STT / translator / TTS or S2ST stages. The language pair lives on the owning stage (target_language on S2ST/translation, language on STT, voice_id on TTS); the flow-level source_language/target_language/voice_config fields are deprecated fallbacks.
build_processors has two modes, chosen by whether flow_direction is present in runtime_context (directly, or under session_metadata):
flow_direction |
What gets built |
|---|---|
| absent | both flows in a ParallelPipeline; audio is routed by speaker identity from the carrier — left vs right channel for stereo G.711, or per-leg streams for Opus |
incoming or outgoing |
only that one flow (_build_single_flow) |
The single-flow mode is what per-direction callers use: ari-controller → media-proxy allocates two sessions per translated call (A2B and B2A), and connect-scv-translator opens two gRPC streams from the browser. One stream cannot carry both legs — AudioChunk.track_id exists in proto/voice.agent.proto but the orchestrator neither reads it on input nor sets it on output, so both legs would be conflated and translated by whichever single flow was selected. See connect-scv-translator.
Transcripts: two channels per flow¶
A translation flow emits text on two channels, and both are fragment streams:
| Channel | TextEvent.kind on the wire |
What it is |
|---|---|---|
| source | KIND_TRANSCRIPT (0) |
what the speaker said, in their own language |
| translation | KIND_RESPONSE (1) |
what the listener hears |
The speaker is a property of the flow, not of the kind. On an outgoing flow both
channels belong to the agent. TranslationNormalizer maps flow_direction →
normalized_role for exactly this reason; a consumer deriving the speaker from kind
attributes every agent utterance to the customer.
Neither channel gets a "done" event — there is no LLMFullResponseEndFrame on a
translation flow, and providers stream phrases (Google) or sub-word deltas (OpenAI).
The division of labour that follows is load-bearing:
TranslationNormalizerowns spacing and marks every fragmentaccumulation_mode="append", inserting separators per the flow'stext_granularity(token= the provider's own whitespace is authoritative;phrase= whitespace-free, needs one added;auto= infer). It is inserted last in the flow and there is one per flow, not the per-stage registry lookupuniversal_builderdoes — a cascaded flow starts withdeepgram/stt, whose registered normalizer has no translated-text branch and would drop that channel.ConversationAPIPublisherowns accumulation and turn boundaries (an idle timer), because it is the only component that knows both the full text and when the utterance ended. It emitsTranslationTranscriptFrame— aDataFrame, deliberately not aTextFrame, carrying the accumulated snapshot pluschannel/is_final/turn_id.
Without the normalizer the publisher never recognizes either channel and a translated
call produces no conversation_messages rows at all, silently. Two related traps:
the gRPC transport drops bare TranscriptionFrame/InterimTranscriptionFrame so
fragments can't leak to a client, and TranscriptionFrame is a TextFrame
subclass — an unguarded isinstance(frame, TextFrame) on the text-out path ships
source ASR as though it were the translation. text_out_enabled is therefore
has_text or uses_s2s_translation, while text_in_enabled stays keyed on has_text
alone (a translation flow emits text but doesn't accept it).
Consumer-side handling of the pair — including the Salesforce single-text-field constraint — is in connect-scv-translator.
Audio mixing (MixTranslationStyle)¶
When TTS for caller B is ready to play to caller A, the original lang-B audio is still flowing. Playing both at full volume produces unintelligible noise.
MixTranslationStyle in apps/media-proxy/internal/audio/mix.go handles this:
- Original audio plays at
echoVolume— a linear 0..1 gain, default0.10. This is its idle level, not full volume: simultaneous interpretation needs a little of the original underneath for prosody and turn-taking, but not enough to compete. - When TTS arrives, the original ducks to
TranslationDuckedFloor(0.01) over atranslationFadeTimeMslinear ramp (20 ms) — fast enough to land with speech onset. - TTS plays at full volume.
- When TTS ends, the original ramps back up to
echoVolume.
echoVolume is therefore the level of the other party's untranslated voice in this listener's ear, not a self-echo suppressor. 0.0 mutes the original entirely; 1.0 disables ducking. It is authored per direction as config.translation.flows.<direction>.echo_volume and never ducked up: if the authored value is below the floor, the floor is ignored (if target > echoVolume { target = echoVolume }) — an author who asked for less original voice must not get more.
baseDb/duckExtraDbbelong toMixUnderbed/MixWithBackground(the underbed + hold-music primitives), not toMixTranslationStyle. See audio-ducking.md.
echo_volume on the wire is a PERCENTAGE, and the browser honours it too¶
The orchestrator forwards the authored value verbatim as a string in ServerHello metadata — hello_metadata["echo_volume"] = str(ev) in main.py, defaulting to 10 — and the consumer divides by 100 (s.echoVolume = echoVol / 100.0, session.go). So "10" means 10%, and reading it as a fraction requests 1000% gain. Both consumers must keep the /100.
connect-scv-translator used to document this key as a media-proxy-only directive and ignore it, which made the browser path fail in opposite directions on the two legs — see Echo ducking is per-leg. It now applies the same contract client-side. skip_recording remains media-proxy-only.
Three authoring/runtime divergences on this setting (pre-existing, unfixed)¶
Worth knowing before trusting the editor, or "fixing" one of them in isolation:
| Divergence | Detail |
|---|---|
| Default is 25 in the UI, 10 at runtime | TranslationPipelineEditor.tsx renders flow.echo_volume ?? 25 and the generated schema says "default 25", but an unset flow actually runs at 10 (ev = echo_volume if echo_volume is not None else 10 in main.py, matching media-proxy's echoVolume: 0.10). So the slider shows a value the call is not using until someone saves. Changing either side is a behaviour change for existing agents — pick deliberately, don't "align" it as a typo. |
echo_original_audio has no runtime reader at all |
A grep across apps/**/*.py + apps/**/*.go finds nothing; only TS types, the collab shape, and generated schemas. The switch gates the slider's visibility in the editor and nothing else — turning it off does not stop the original voice being mixed. (Its silver lining: the orchestrator never mixes the original into the translated return stream, so adding it at the listener cannot double-count it.) |
| The UI range is 5–50, the consumers accept 0–100 | The slider clamps to 5–50 in steps of 5; MixTranslationStyle and parseEchoVolume both clamp 0..1, so 0 (fully mute the original) and values above 50 are reachable by API/DB but not from the editor. |
Translation provider options¶
Configurable per direction in the agent's agent_versions.config:
- LLM translation — pass through GPT-4o / Claude / Gemini with a translation prompt. High quality, higher latency.
- Dedicated translator — Google Translate, AWS Translate, DeepL. Lower latency, less context-aware.
- Speech-to-speech — OpenAI Realtime or Gemini Live with built-in translation. Lowest latency, single API call.
The provider is chosen per direction; you can have asymmetric setups (LLM for outbound English-to-Spanish, dedicated translator for inbound Spanish-to-English).
Google S2ST (Gemini Live Translate)¶
GoogleS2STService (s2st_service.py, wrapped by google_s2st_services.py) uses the public Generative Language API, not Vertex AI:
- Endpoint:
wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=API_KEY(the key rides in the query string; there is no bearer header). - Auth: a Gemini API key resolved from the
googleintegration'sapi_key(or theGOOGLE_API_KEYenv, or a per-nodeapi_key_override) — not a service account /service_account_json. - Model:
gemini-3.5-live-translate-preview. Configs pinning the old private-beta Vertex model (gemini-2.5-flash-s2st-exp-11-2025) are overridden to this default at runtime with a warning. - Setup message (camelCase):
generationConfig.{responseModalities, translationConfig.{targetLanguageCode, echoTargetLanguage}}, withinputAudioTranscription/outputAudioTranscriptionandrealtimeInputConfig.automaticActivityDetectionat the top level ofsetup— the API rejects the transcription fields if nested insidegenerationConfig. - Audio: input 16-bit PCM @ 16 kHz mono (declared via the
audio/pcm;rate=16000mimeType — no resampling); output 24 kHz mono. 70+ BCP-47 languages (Google docs). - Constraints: single target language per session (each direction is its own session — matches the dual-flow builder), audio-only input, and no voice selection (the translated audio uses a fixed model voice). To control the output voice, set
output_audio: falseand route the translatedTextFramethrough a downstream TTS (e.g. ElevenLabs).
targetLanguageCode must be a code Google accepts verbatim — or the socket 1007s on the first audio frame¶
translationConfig.targetLanguageCode is matched against a fixed list of exact, case-sensitive strings. An unlisted value does not fail the handshake: the API answers setupComplete normally, and then closes the WebSocket with 1007 (invalid frame payload data) Request contains an invalid argument. the moment the first audio frame arrives. The session therefore connects, logs healthily, and dies as soon as anybody speaks — surfacing as Error processing frame: received 1007 … plus a flood of [GoogleS2ST] Got audio frame but not connected! 640 bytes dropped, far from the actual cause.
Region-qualified tags are accepted only for pt-BR, pt-PT, zh-CN, zh-TW, zh-Hans, zh-Hant. Every other lang-REGION form is rejected — including the ones authoring UIs naturally produce (en-US, en-GB, es-ES, es-MX, es-US, es-419, fr-FR, de-DE, ja-JP, hi-IN, nl-BE, pt-AO, zh-HK, cmn-CN, yue-HK), as are wrong-case variants (ES, zh-tw, pt-br) and non-region subtags (es-Latn). Bare language subtags all work (es, en, fr, de, ja, hi, ar, ru, ko, it, tr, pl, vi, id, th, nl, cmn, yue, pt, zh).
normalize_target_language() in s2st_service.py is the single gate, applied where the setup message is built: it canonicalizes BCP-47 casing (so pt-br becomes the accepted pt-BR rather than 1007ing), keeps the six accepted regional codes verbatim, and reduces everything else to its bare language subtag with an INFO log. Underscore-separated Salesforce locales (en_US) normalize too. This matters because a stage-level target_language is not the only input — media-proxy and the SCV connector both send per-call metadata.target_language overrides that beat the authored value (_resolve_flow_languages_with_override), and locale-derived overrides are exactly the region-qualified form. Normalizing in the service covers every caller; don't move the fix upstream into one of them. Test coverage: test_google_s2st_target_language.py (both code sets verified against the live API 2026-08-04).
Dial & Translate embed demo¶
A page embedding the phone emulator can let its viewer dial a real phone number, pick "my language" / "their language", and talk to the dialled party through Live Translation. The demo reuses the pre-existing dial-through translation flow — no new runtime path in ari-controller, media-proxy, or the orchestrator.
Embed iframe (embed cookie from /api/emulator/embed/exchange)
dial pad: number + caller/callee language
→ POST /api/emulator/embed/dial-grant
server picks dest = the tenant's consult ROUTING NUMBER
server builds the UUI and signs it into X-Internal-Leg-Sig
→ Janus INVITE sip:<translation-agent-id>@asterisk
User-to-User (pipe-encoded), X-Internal-Leg-Sig,
X-Call-Source: emulator-embed, X-Conversation-Id
→ [from-extensions] _X. → [agent-router] → Stasis(ai-bridge, …)
→ StasisStart: signature verified → trusted internal leg, mode=translation
→ addHuman(caller) → originateDialThrough(dest) with
X-Routing-Type: external, X-Target-DNIS: <number the viewer typed>
→ the tenant's Genesys Architect flow dials that number externally
→ B-leg answers → buildTranslation: two bridges, two ExternalMedia, cross-routed
→ media-proxy: two gRPC sessions (flow_direction incoming/outgoing) carrying
per-direction source_language / target_language
→ translation_builder: session metadata beats the authored stage languages
The number the viewer types is a routing instruction, never a dial target¶
The SIP leg always dials the tenant's External Consult routing number (consult_routing_number on an active Genesys integration with external_consults_enabled). The number the viewer typed rides the UUI as routeto, which ari-controller forwards on the originated B-leg as X-Target-DNIS for the customer's Architect flow to dial. So the trunk, and the number that fronts the dial-out, stay ours.
UUI contract¶
/api/emulator/embed/dial-grant is the only thing that builds this. Values are encodeURIComponent-encoded and pairs joined with ;:
| Field | Value | Consumed by |
|---|---|---|
pipeline, conv |
the translation agent id, a fresh randomUUID() |
StasisStart session keying |
dest |
the tenant's consult routing number | session.uuiDestination → triggers originateDialThrough |
routingtype |
external |
X-Routing-Type on the B-leg |
routeto |
the E.164 number the viewer dialled | X-Target-DNIS on the B-leg |
fqdn |
the integration's byoc_trunk_fqdn (optional) |
session.ccaasByocFqdn — honoured only on a trusted internal leg |
lang / peer_lang |
caller / callee language (BCP-47) | session.uuiLanguage / uuiPeerLanguage → the media-proxy language override |
ani, caller_name |
the config's caller-id fields (optional) | caller presentation |
Anything non-standard lands in uuiCustomFields and is forwarded as session_variables.
[agent-router] does not read X-Target-FQDN / X-Caller-Id-* (only [ai-manager-outbound] does), which is why ANI, caller name, and FQDN travel in the UUI on this path rather than as SIP headers.
The grant returns the UUI pipe-encoded (;→|): ; is the parameter delimiter in a SIP header value, and the dialplan hands the UUI to Stasis as a single arg. ari-controller decodes |→; before parsing and before verifying the signature, so the HMAC is over the semicolon form.
The signature binds the UUI¶
X-Internal-Leg-Sig is HMAC-SHA256(secret, "callSource:convId:pipelineId:sha256hex(uui)"), base64url. Before this feature it covered only the call source and ids, which would have let a browser edit routeto after the grant and place arbitrary PSTN calls on the tenant's trunk. The legacy (UUI-less) payload form is still accepted, but only when the leg carries no UUI — see ari-controller. Signing lives in apps/ai-manager/lib/internal-leg-sig.ts and apps/ari-controller-node/src/utils/internal-leg-sig.ts; keep them in lockstep.
Language override precedence¶
Highest wins:
- What the viewer picked on the dial screen (rejected outright if the config pins
selectableLanguagesand the request is outside it — that's tampering, not a preference). - The embed config's
translation_defaults.callerLanguage/calleeLanguage. - The agent's authored stage languages (
target_languageon S2ST/translation,languageon STT).
The pair is all or nothing at every hop: the grant omits lang/peer_lang unless both resolve, buildTranslation posts callerLanguage/agentLanguage only as a pair, and media-proxy injects the per-direction gRPC metadata only when both are non-empty. A half pair silently does nothing and the call runs on the authored languages with no signal why. auto is normalized away for the same reason — it's the UUI parser's default for an absent field, not a language.
buildTranslation must post callerLanguage/agentLanguage — the keys allocator.go actually reads. The sourceLanguage/targetLanguage keys in the same meta object have no Go reader and exist for log parity only.
Configuration¶
- Genesys integration (Settings → Integrations): External Consults enabled + a Routing Number, whose Architect flow dials
X-Target-DNIS. Optionally a BYOC trunk FQDN. - A published dual-flow translation agent (subtype
translation). - An embed config (the Embed admin drawer): transport SIP, dial mode Dial & Translate, that agent, and optional default / selectable languages.
dial_translatehas nodestination— the DB CHECK is relaxed for exactly that case (026_emulator_embed_dial_translate.sql) — and noauto_connect, since there's nothing to dial on mount.
The embed viewer is authenticated by the short-lived, HMAC-signed embed cookie plus the config having explicitly opted into dial_mode='dial_translate'; there is no anonymous SIP provisioning. Dial grants are rate-limited to 5 per 60 s per config.
Testing it¶
No-config path: the /emulator admin page has a Translate toggle (upper right) that swaps the dial pad for the Dial & Translate screen with an agent picker — no embed configuration needed. It exchanges the dial for a grant at POST /api/emulator/dial-translate-grant (session + AIManager:ManageAgents, API tokens rejected — the same bar as /api/internal-leg-sig), which validates the picked agent is a published translation agent in the caller's tenant, resolves the same consult routing, builds the same UUI, and signs with call source ai-manager-test. The shared grant pieces (normalizeE164/buildUui/resolveLanguage/resolveConsultRouting) live in lib/dial-translate-grant.ts so this route and the embed grant can't drift; the embed route re-exports them for its tests. Rate limit is per user (rl:emulator-dial:<userId>, 5/min) rather than per config.
Embed path: the Embed configurations drawer (/emulator → Embed) is the harness for the real embed flow: selecting a config mints a 5-minute token via GET /api/emulator/embed-configs/:id/share-link and renders /embed/phone?t=… in an inline iframe, which is the real embed path (same token exchange, same dial-grant call) rather than a mock. Two operational notes that cost debugging time:
- The preview requires AI Manager's own origin to be in the config's
allowed_origins— theGETshare-link handler mints fornew URL(request.url).originand 403s (Preview requires the current origin to be in allowed_origins) otherwise. A config destined for a third-party site needs both origins listed. assertTranslationAgentInTenantchecksagents.subtype === 'translation'but not that a published version exists. An unpublished translation agent therefore saves cleanly and fails at call time. If you tighten this, tighten it there.
Per-hop, for a dial that misbehaves:
| Hop | What to look for |
|---|---|
| ai-manager | the dial-grant response — uui (pipe-encoded), conversationId, and languages (null means only one side resolved) |
| Asterisk | [agent-router] HEADER-DEBUG shows CALL_SOURCE='emulator-embed' and a non-empty User-to-User |
| ari-controller | trusted internal leg accepted, mode=translation, then Originating dial-through call carrying X-Routing-Type: external + X-Target-DNIS: <the number the viewer typed> |
| Genesys | the Routing Number's Architect flow places the outbound call; B-leg answers → translation: dial-through B-leg connected |
| media-proxy | two sessions with distinct flow_direction, each with source_language/target_language set |
| ai-orchestrator | translation_builder logs the per-flow override (language_override=True) |
The user-facing setup walkthrough is userguide/surfaces/dial-and-translate-setup.md.
Failure modes specific to this path¶
| Symptom | Cause |
|---|---|
| Dial screen shows "No Genesys integration with External Consults enabled…" (409) | No active Genesys integration with both external_consults_enabled and consult_routing_number. |
| Dial screen shows "This embed is not configured for Dial & Translate" (403) | The config is WebRTC transport or still dial_mode='fixed_destination'. |
| Call rings but ari-controller logs a signature mismatch | The UUI reaching Stasis isn't the one that was signed — a tampered routeto, or an ai-manager/ari-controller version skew (deploy ari-controller first). |
| Both parties connect but hear untranslated audio | Only one language resolved. Check the grant's languages field and ari-controller's propagating language override log line. |
UI¶
The FlowEditor in ai-manager renders translation pipelines as parallel pairs of stages, one per direction. The inspector lets you set source/target language pairs and pick providers.
A "translation-assist" mode shows the human agent both raw transcript and translation in real time without speaking the translation; the customer hears nothing translated. Useful for monolingual agents handling multilingual callers. See the archived translation-assist.md for the UX history.
Failure modes¶
| Symptom | Likely cause |
|---|---|
| Audio cuts in/out | Mixer ducking too aggressive. Tune duckExtraDb lower (underbed modes) or raise echo_volume (translation). |
| Original untranslated voice at full volume under the translation | echo_volume not being applied at the listener. On the browser (SCV) path check the echo: agentEar=…% customerEar=…% line in the [OrchestratorGrpcProvider] started log, and any [EchoDucker] … no media element is rendering the original voice warning. |
| Nothing but translated phrases, dead silence between them | The listener's echo_volume is 0, or (browser path) the original voice was never added to that leg — the customer leg has to add it, since replaceTrack removes the agent's real voice outright. |
| Muted agent still audible to the customer (quietly) | Browser path: mute must gate the echo path as well as the capture. It bypasses the sender track that CCP/Genesys mute acts on. |
| Translation lags by 5+ seconds | LLM translation provider choice. Try a dedicated translator. |
| Translation is fine on short turns but ends seconds late on long ones, and gets worse the longer someone talks | A rate error, not a delay — something per-chunk. The wire carries ~40 ms chunks (audio_out_10ms_chunks=4, unoverridden), so any per-chunk cost accumulates. On the browser (SCV) path this was playback chained on onended; see playback scheduling. Don't assume PhraseAwareBuffer is smoothing the output — it is inserted only after a service_type == "tts" stage, and a single-stage S2S flow has none. |
| Muted or held, but the tail of the translation still plays out | The listener is scheduling audio ahead of the clock, so there is no queue to empty — pending sources have to be stop()ed. |
| Salesforce transcript shows garbled part-words, one line per fragment | Raw provider text frames reached the wire instead of the publisher's accumulated TranslationTranscriptFrame. Both channels are fragment streams with no done event. See transcripts. |
| Each utterance appears several times, source text CUMULATIVE but translation a non-overlapping DELTA | Two wire producers, not a client-side join — the transport's text-chat branch caught the re-pushed fragment beside the publisher's accumulation. Gate is the session param text_out_translation_only. The asymmetry is the tell: one joined string cannot produce it. See transcripts. |
| The translation is crammed under the original in one Salesforce chat bubble | The pre-split newline form. An utterance in two languages is two messages (splitTranscriptMessages), each needing its own messageId and its own slice of the time window. |
| Source-language ASR appears in the transcript as if it were the translation | An unguarded isinstance(frame, TextFrame) on the text-out path — TranscriptionFrame is a TextFrame subclass. |
| Every agent utterance attributed to the customer | Speaker derived from TextEvent.kind instead of from the stream. On the outgoing stream both kinds belong to the agent. |
| One direction works, other does not | Per-flow config asymmetry. Check both flows.incoming and flows.outgoing. For a per-direction caller, check that two sessions were opened with distinct flow_direction values. |
| Wrong-language TTS | STT language autodetect fired in the wrong mode. Force the source language. |
| Google S2ST silent / WS closes immediately | Missing or invalid Gemini API key. Set api_key on the google integration or GOOGLE_API_KEY. The key goes in the WS query string, not a header. |
Google S2ST connects, then 1007 … Request contains an invalid argument. on first speech (+ repeated Got audio frame but not connected!) |
An unaccepted targetLanguageCode — a region-qualified code such as en-US/es-ES. normalize_target_language() handles this; if it recurs, check the Setup sent: target_language=… (sent as '…') log line for what actually went on the wire. See above. |
See troubleshooting-translation.md for diagnosis steps.
See also¶
- Universal pipeline builder
- Audio ducking
- Voice request lifecycle
- Troubleshooting translation
- media-proxy service doc
- connect-scv-translator — browser-side Live Translation on a Salesforce SCV call