AWS real-time translation¶
Status: Current | Last reviewed: 2026-10-14
Use Amazon Voice Translation to compare Transcribe → Translate → Polly with the existing speech-to-speech providers in Salesforce Live Translation. Both AWS modes use the same orchestrator stage and SCV audio routing.
Select a mode¶
Add Speech-to-Speech Translation → AWS → Voice Translation to each translation flow. Its Properties contain the complete configuration; it has no Behavior prompt.
| Setting | Phase 1 | Phase 2 |
|---|---|---|
| Polly streaming mode | PCM response streaming (standard) |
Bidirectional streaming (bidirectional) |
| Polly operation | SynthesizeSpeech |
StartSpeechSynthesisStream |
| Text input | One committed phrase per request | Multiple committed phrases per utterance stream |
| Audio output | PCM bytes as the response arrives | PCM audio events while more text arrives |
| Engines | Standard, Neural, Generative | Generative |
Both modes default to Generative so an A/B test changes only the streaming
protocol. Pin the same region, voice, language pair, phrase settings and input
audio. voice_id=auto resolves an available voice during startup; a benchmark
pins that first selected voice for subsequent runs. An unavailable
voice/language/engine combination fails instead of substituting another engine.
For customer Spanish and agent English, configure incoming es-US → en-US
and outgoing en-US → es-US. SCV's language pickers override these stage
languages for the current call. Base languages are normalized to default
locales, including en → en-US and es → es-US. A pinned voice must also
support the overridden target locale; use auto when testing several pairs.
The supported pairs are the intersection of Transcribe streaming, Translate,
and the chosen Polly engine/region.
Runtime path¶
The Python adapter downmixes and resamples input to mono PCM16 at 16 kHz. A private Node.js worker sends 100 ms chunks to Transcribe Streaming, commits stable text prefixes, translates those phrases, and emits Polly PCM in chunks of at most 40 ms. Capture, translation and synthesis run concurrently. Neither mode waits to download the complete audio response.
The worker is shared by directions and calls on one orchestrator event loop. Each direction has its own AWS clients, credentials, streams and bounded queues. There is no additional server, port or browser AWS credential. The worker exits when its last direction closes; the hard cap is 128 active directions per worker, independent of AWS account quotas.
AWSVoiceTranslationService registers as s2s_translation. It bypasses the
generic 800 ms PhraseAwareBuffer added after standalone TTS stages. SCV still
uses its existing two gRPC streams, original-audio underbed, playback scheduler,
ducking and interruption handling. Source and translated phrases pass through
the existing per-flow normalizer; timing messages use a separate system envelope.
Transcribe returns cumulative partial results. The committer tracks a cursor
per ResultId and speaks only Stable prefixes. It commits at punctuation,
or after max_phrase_ms of source audio with at least three tokens. A final
result flushes the remaining suffix once. The duration setting is a boundary
hint, not a latency guarantee: unstable words wait for more recognition context.
Translating short prefixes can lose context needed for word order or meaning;
listen for those effects alongside measuring speed.
Phase 2 keeps one Polly stream per Transcribe utterance. It sends translated
phrases with FlushStreamConfiguration.Force when Flush each translated
phrase is enabled, and sends CloseStreamEvent at the utterance boundary.
Empty text events every two seconds keep the stream alive during pauses.
Streams rotate after nine minutes to stay below Polly's ten-minute limit.
HTTP-200 event-stream exceptions are treated as failures.
Pending work older than max_queue_ms fails the affected direction, allowing
SCV to restore its original call audio. Queue overflow also fails visibly.
This limit applies to work waiting for translation/synthesis; it is not
a bound on browser playback delay or on an active synthesis request.
Interruption cancels the direction's streams and starts fresh recognition state.
There is no automatic speech replay after a provider failure.
Install and authorize¶
The orchestrator image includes Node.js 22 and the locked worker dependencies. The devcontainer launcher installs them when the lockfile changes. For a manual orchestrator launch, run from the repository root:
npm ci --prefix apps/ai-orchestrator/aws-translation --ignore-scripts --no-audit --no-fund
Choose an active Amazon integration in the same tenant and enable its Translation capability. The runtime reads its encrypted access key, secret key and optional session token. Without explicit keys, it uses the AWS SDK's workload credential chain. Omitting the integration also uses that chain. An unknown or cross-tenant integration fails; it cannot select the workload identity by failing a lookup. Credentials embedded directly in stage config are ignored.
Region precedence is stage AWS region, integration region, AWS_REGION,
AWS_DEFAULT_REGION, then us-east-1. Grant the selected identity:
| IAM action | Purpose |
|---|---|
transcribe:StartStreamTranscription |
Streaming recognition |
translate:TranslateText |
Phrase translation |
polly:DescribeVoices |
Validate/select the target voice |
polly:SynthesizeSpeech |
Phase 1 |
polly:StartSpeechSynthesisStream |
Phase 2 |
No Connect media-streaming permission is added by this stage: SCV already supplies the call audio through the orchestrator transport. Enable access to all three service endpoints in the chosen region.
Compare live AWS replays¶
Run the same synthetic or consented recording through both modes:
npm run compare --prefix apps/ai-orchestrator/aws-translation -- \
--audio /tmp/sample-16k-mono.wav \
--output /tmp/aws-translation-comparison \
--source en-US --target es-US --region us-east-1 --voice Lupe --repeats 3
The input must be a 16 kHz mono PCM16 WAV. The command uses the workload
credential chain, sends audio in real time, alternates mode order between
repeats, and writes one output WAV per successful run plus comparison.json.
The JSON records the input PCM hash, settings, individual measurements,
failures and p50/p95 summaries. Any failed run gives a nonzero exit status.
Output recordings contain the translated speech; store them as call data when
using customer recordings.
Polly's bidirectional audio events have no phrase markers. Summaries therefore
compare only phrase_index=0 for each ASR utterance in both modes. Later
standard-mode phrase measurements remain in the raw records. A p95 from three
samples is the maximum observed sample, not a production tail estimate.
The retained English→Spanish
and Spanish→English
smoke results used three replays per mode, Generative, us-east-1, Lupe for
Spanish output and Danielle for English output. The two language-direction
comparisons ran concurrently. These are short synthetic utterances, not a call
quality or load benchmark.
| Direction | Phase 1 Polly first audio p50 | Phase 2 Polly first audio p50 | Phase 1 source→audio p50 | Phase 2 source→audio p50 |
|---|---|---|---|---|
| English→Spanish | 154 ms | 158 ms | 2.29 s | 2.30 s |
| Spanish→English | 157 ms | 128 ms | 5.52 s | 5.51 s |
Transcribe stable-prefix commitment dominated these runs. The data does not
establish a material overall latency advantage for either Polly protocol.
Reproduce the inputs with espeak, then convert with ffmpeg:
espeak -s 175 -w /tmp/en.wav "Hello. I need help with my order. Can you tell me when it will arrive?"
espeak -v es -s 145 -w /tmp/es.wav "Hola. Necesito ayuda con mi pedido. Puede decirme cuándo llegará?"
ffmpeg -i /tmp/en.wav -ar 16000 -ac 1 -c:a pcm_s16le /tmp/en-16k.wav
ffmpeg -i /tmp/es.wav -ar 16000 -ac 1 -c:a pcm_s16le /tmp/es-16k.wav
Inspect SCV timing¶
The orchestrator logs [AWSTranslationMetric]. SCV emits a provider
performance event and logs [AWSTranslationPerformance] with the direction
and buffered_playback_ms. Filter the browser console by that label during
test calls. Timing envelopes contain no transcript text and are not sent to
Salesforce as utterances.
| Field | Measurement |
|---|---|
asr_commit_ms |
Last source sample in the first phrase reaching the worker → phrase commitment |
translate_ms |
Translate request duration |
tts_queue_ms |
Translated phrase ready → synthesis request |
polly_first_audio_ms |
Synthesis request → first PCM chunk |
commit_to_first_audio_ms |
Phrase commitment → first PCM chunk |
source_to_first_audio_ms |
Last source sample in the first phrase reaching the worker → first PCM chunk |
buffered_playback_ms |
SCV scheduler occupancy when the metric arrives |
Worker timings exclude capture before worker ingress, transport to the browser, and device playback. Browser occupancy is a separate observation; summing it with worker timing does not measure mouth-to-ear latency. For an actual SCV comparison, use the two published profiles with the same recorded phrases, record what the listener hears, and measure corresponding source/playback boundaries. Include pauses, interruptions, long turns and overlapping speakers.