ElevenLabs voices and half-cascade agents¶
Status: Current | Last reviewed: 2026-10-07
Use this guide to pair a streaming text generator with ElevenLabs v4, configure spoken delivery, and distinguish text-first pipelines from native speech agents.
Create and configure¶
In the new-agent wizard, choose Voice agent → Pick a stack → OpenAI Realtime + ElevenLabs v4. Select the OpenAI and ElevenLabs integrations. Open the resulting ElevenLabs node and choose a voice. The Design assistant also offers Realtime + ElevenLabs v4 in its voice-stack question.
The stack uses openai-elevenlabs, with Realtime output_audio: false
(the runtime requests output_modalities: ["text"]), followed by ElevenLabs Text to Dialogue using
eleven_v4_turbo. It enables speech_format: expressive. Existing agents and
model selections are not automatically upgraded.
Keep the business prompt, tools, procedures, response guards and subagents on the generating stage. Choose Speech Formatting on the ElevenLabs stage:
| Value | Effect |
|---|---|
off |
Use the authored prompt. This is the default for existing stages. |
natural |
Request conversational text, sensible punctuation and unambiguous spoken numbers, without audio tags or markup. |
expressive |
Add sparse vocal delivery tags to the natural-text guidance. Requires a v3/v4 model. |
Advanced → Additional Delivery Guidance (speech_instructions) adds optional
style, pacing and pronunciation instructions when formatting is enabled.
The guidance asks the generator to preserve factual values, required disclosures,
consent questions and tool-result meaning. It leaves tool calls in their structured
channel. It uses auditory tags, punctuation and ellipses rather than SSML; v4 does
not support SSML.
This is model prompting, not a deterministic rewrite or an enforcement gate. No additional model invocation is inserted. Author response guards separately when a reply needs checking before speech; those checks can add delay. Explicit scripted phrases and guard replacement text are not rewritten. Author those phrases in the desired spoken format.
Model discovery and transport¶
GET /api/models/elevenlabs?integration_id=… requires authenticated agent/tool
management access, applies rate limiting, resolves tenant-authorized credentials,
and calls ElevenLabs GET /v1/models. Credentials remain on the server.
The response is not cached across tenants or requests. The UI scopes its last
loaded catalog to the active tenant and integration and supports Refresh models.
The model picker filters TTS and voice-conversion capabilities separately. Documented options remain available when discovery fails; the UI reports the failure and retains a saved selection even if the catalog no longer lists it. Discovery does not prove that a particular voice, model or plan can synthesize. Play sample uses the selected model, and changing it cancels an old preview. The reviewed public pricing catalog does not yet cover the v4 selectors, so their cost estimates have incomplete coverage until reviewed rates are added.
| Studio service | Models and behavior |
|---|---|
| WebSocket API | Traditional TTS models such as Flash v2.5. Rejects v3/v4. |
| Text to Dialogue | v3/v4 families; sends text incrementally and flushes the last fragment at turn completion. |
| HTTP Streaming | Sends a complete text segment and streams its audio response. Includes v4. |
V4 sends only Stability and Similarity voice settings. Older Style, Speed and Speaker Boost values may remain stored after a model change but are hidden in the editor and omitted from v4 requests.
HTTP Streaming closes its client through stop(frame) and cancel(frame),
after the base TTS service drains or cancels queued audio. Keep its private
HTTP cleanup helper separate from AIService._stop(frame): overriding that
dispatcher with a no-argument method drops EndFrame with a signature error.
The October 7 Takeda handoff exposed this after its closing line. Regression
checks must send EndFrame through process_frame, verify farewell audio
precedes it, and preserve the transfer parameters; calling stop directly
does not exercise the failing dispatch path.
The documented v4 endpoint is /v1/text-to-dialogue/stream-input. The existing
multi-context endpoint still documents v3-only support. Therefore v4 opens one
single-stream socket per turn, stamps received events with the originating
Pipecat context ID, and drains the final audio before closing. Barge-in closes
the interrupted socket without flushing unheard text. Errors finish their own
audio context. V3 retains the existing multi-context transport.
For v4, a sentence-ending text fragment schedules a non-closing flush after
120 ms without further text. This releases short sentence tails held by the
provider's text threshold. Ordinary word fragments never trigger this timer:
Live sends them at speech cadence, and flushing every inter-fragment pause
breaks natural word grouping. Further text cancels the pending flush; explicit
flush, turn close, interruption, provider completion and disconnect also cancel
it. Turn completion still owns the final audio and socket close.
The v4 transport also implements Pipecat's wait_closed contract. Disconnect
waits for all turn sockets/readers to close, including repeated cleanup; a
successful disconnect must not report a missing-method transport error.
A v4 turn therefore incurs a socket handshake and consumes a dialogue session
while open; measure latency and concurrency with your account and voices.
Other generators¶
| Generator | ElevenLabs path |
|---|---|
| OpenAI Realtime | Audio input with text output, followed by ElevenLabs. Included in the new stack. |
| Streaming text LLM | STT → LLM → ElevenLabs. Existing OpenAI, Gemini and other registered streaming text services can feed the same TTS stage. |
| Google ADK Live | Current ADK Live documentation requires AUDIO; it is not a text-only starting option. For a Google text-first path, use STT → streaming Gemini → ElevenLabs. |
| GPT Live | With native playback disabled, incremental output captions feed the downstream ElevenLabs stage. Live still generates native audio and owns conversation timing; disabling playback does not request text-only generation. |
For GPT Live with ElevenLabs v4, Text to Dialogue with sentence aggregation disabled sends caption fragments into the provider's streaming input immediately. HTTP Streaming instead waits for a completed text segment before starting each request. Keep the model, voice and output format fixed when comparing them: dialogue input can improve longer responses while its handshake and internal buffer can make a very short opening slower than HTTP. It is not a latency guarantee. No STT is needed between Live captions and ElevenLabs. Retain Live's caller transcript settlement window so partial consent is not treated as a completed caller turn.
The Consumer Cellular 4451 greeting exposed a 1.076-second delivery gap after the opening audio had already started. Direct gRPC capture reproduced it without a browser or Mock Caller playback. The sentence-tail flush addresses that buffer wait; it does not remove Live's initial connection/greeting latency or change the selected voice.
Formatting is derived from the first downstream TTS on the same linear branch. It does not cross another generator, a parallel-group boundary or an output-lane branch. Place the generator and its TTS together for this option. The builder also seeds native session instructions with the delivery suffix, so Realtime receives it even when greeting is disabled and it answers before its first conversation-context frame. It appends the same suffix after prompt resolution and before the instruction refresher captures appended guidance, preserving it during variable refreshes and subagent takeovers. A standalone stage prompt preview does not have the downstream pipeline context needed to derive this suffix.
Verification¶
The tests cover discovery parsing, transport filtering, stack/profile derivation, v4 wire messages, split audio tags, final audio, interrupted and failed sockets, HTTP context IDs and completion after multi-sentence, empty or failed responses, and delivery guidance across variable and subagent changes. They mock provider connections. Before rollout, listen with your actual voice, exercise barge-in and short final phrases, and check first-audio latency and procedure outcomes. Model availability and audio quality require a live account.
Implementation: model discovery, delivery prompt, dialogue service, v4 sockets.
Provider references, checked 2026-09-29: models, list models, v4 settings, speech prompting and Enhance guidance, dialogue endpoints, multi-context reference, Realtime text output, GPT Live captions, ADK Live configuration.